@Sal I never bother removing the h5 section as it is not affecting the code (as it is never executed). But if you think that the code will be cleaner I can remove it.
@Pierre
- I was aware of this issue and I fixed it already 2 weeks ago (commit). I didn't push it on PyPI yet (I usually don't do it at every commit otherwise I need to increase the version) but will do it asap. In the meanwhile you can use the test code in the github repository, which works fine.
- can you elaborate more on "the BioBrillouin website is unable to open it"? Is it giving an error or just not showing anything? Could you send the file, so we can check what is the problem?
- as we extensively discussed in the past, the brimfile format is not limited to 4D. I created a function to specifically save 4D arrays as I believe this is the most common scenario in imaging (timepoints can be saved by creating multiple data groups, i.e. calling the function create_data_group multiple times). For now the possibility of saving additional dimensions in a single PSD array (e.g. for angle resolved measurements) can only be done through the function create_data_group_raw. I plan to improve this once people start to use the library and give feedback (N.B the way the data is saved in the file will not change, I will only add helper functions to facilitate reading and writing to the file).
- adding a checksum to the file is a good idea, but I am not sure how to implement it technically. It might be done at the level of individual chunks (as discussed here). At the level of the whole file it might be problematic, as I am not sure that given the same input, all the zipping tools would produce the same binary output (i.e. compression level, metadata, order of the entries, etc.. might differ), so just unzipping and zipping again a file will not pass the check, even though the file is valid. Additionally there is always the issue of where to store the hash: I am not sure if appending it to the end of the file will make the zip unreadable
Talk to you tomorrow.
Best,
Carlo
On Wed, Aug 27, 2025 at 09:30, Pierre Bouvet via Software <software@biobrillouin.org> wrote:
Hi everyone,@ SalI would suggest adding it to the backend with a dedicated function of the wrapper class with a header looking like:“def export_brim(self, path, filepath)” and then adding the menu buttons to the “show_treeview_context_menu” of the main file of the HDF5_BLS_GUI package, ideally with a bit of logic to ensure only datasets compatible with your module can be exported to brim. The question is then do you want to make your module an independent package or to join it to either Carlo’s or mine. I would suggest adding it to HDF5_BLS because I’m sure there will be no fundamental limitations. I can do it if you want :)Enjoy Banyuls!@ Carlo & SebastianThe code on supplementary for Brim still doesn’t work. I went to dig in the sources and the problem comes from your use of the Enum class: you’re mixing literal expressions and patterns. I think your code also lacks commenting, as is it’s really hard to read and fix.After modifying your library, I tried exporting a synthetic BLS image to Brim using the code you provided. I do get a .zip file or .zarr file where I am supposed to have it, but I cannot say if it works or not since the viewer on the BioBrillouin website is unable to open it and there are no compatible softwares out there. I think it’s important this is fixed, and I would suggest doing as I did in supplementary, by showing an example of integration of your code to an existing process.Next, from what I could understand from your code, you can only add 4D PSD arrays to Brim, I think this is non intuitive and could easily be improved by adding silent dimensions in your process. As I’ve already mentioned before, this is in any case extremely limiting for a number of applications, starting with all TFP measures and time-series.Last problem I see, but that’s proper to zarr: you end up saving a zipped file, which can be corrupted easily by the user if he unzips and then re-zips his file. To limit the risks, I would strongly recommend adding a hash key to check file integrity, generated after each addition or modification to the file, and compared to the computed hash key at opening.Best,PierrePierre Bouvet, PhDPost-doctoral FellowMedical University ViennaDepartment of Anatomy and Cell BiologyWahringer Straße 13, 1090 Wien, AustriaOn 26/8/25, at 17:29, Sal La Cavera Iii <Salvatore.Lacaveraiii@nottingham.ac.uk> wrote:Hi guys,Sorry I am a little slow on communications right now because I am at a summer school that I have been helping organise in Banyuls sur Mer this week and last week.Thanks for the initial feedback on the conversion library! 2 follow ups:
Sent: 11 August 2025 12:41
To: software@biobrillouin.org <software@biobrillouin.org>
Subject: [Software] Re: new link for meeting nowHi Sal,thanks a lot for sending the figures and the brim_converter module. I will try to run it in the next days and give you feedback. In the meanwhile, if you want to test it on more brim files, I am uploading on an S3 bucket a few example brim files from different modalities in our lab; you can find the link to them in the `Example brim files` section in the Supplementary of the paper draft.Regarding HDF5 support, I initially design the brimfile package to easily support different file formats (that's why I introduced the abstract class `FileAbstraction`).Later I thought that I don't see a clear advantage of HDF5 over zarr (apart from being more mature and thus more tools being available) and if we want to advocate for the use of Zarr it is better to not support both in the initial phase, otherwise people would start using both and force us to keep supporting HDF5 as well.If in the future we realize that supporting HDF5 will be beneficial, we can easily do it (as you have already experienced).Regarding the use of R2 and RMSE, you can just ignore them if they are not on Pierre's end, as they are optional. I still believe that it makes sense to keep them in the definition of the file format though, as they are commonly used metrics to estimate the goodness of fit (even though I agree that they are by no means complete or informative about the error on the fit parameters).Regarding the covariance matrix, I feel that that is the quantity that you get directly from the fit and the actual errors on the parameters can be easily computed from there by the visualization software. The only drawback is an increase of the file size, which, in any case, won't be much (it is ~10 additional numbers for each fit while a spectrum has typically several tens of points; also, if it is indeed a purely diagonal matrix, compression should work very well). Instead the advantage of storing it would be that it helps troubleshooting the fit (e.g. in case it turns out not to be diagonal).Regarding distributing the conversion script, as you are only importing brimfile and not HDF5_BLS, I would propose to make it a submodule of brimfile (i.e. a folder `converter` inside `brimfile` so that you can import it by `import brimfile.converter`). I would include any additional dependencies you need (e.g. h5py) into an optional dependency `converter`, so people can do `pip install brimfile[converter]` only if they need it. Anyways, even if you need to use it from the HDF5_BLS library, you need to install brimfile, so by making it a submodule you are not forcing to install any unnecessary dependency.If you go for this option you can write the documentation into an '__init__.py' file inside the 'brimfile/converter' folder and it will be automatically added to the online documentation of the brimfile library (I am using a github action which triggers pdoc to compile the documentation and upload it online at every push). The docstring of each function will also be automatically compiled to the documentation.If you think that is a good option, you can just make a pull request to the brimfile repository. Let me know what you think about it and if you want to discuss some technical detail.Best,CarloOn Fri, Aug 8, 2025 at 21:21, Pierre Bouvet via Software <software@biobrillouin.org> wrote:This message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please contact the sender and delete the email and attachment. Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham. Email communications with the University of Nottingham may be monitored where permitted by law.Hi Sal,In v 1.0 there are no Stokes and anti-Stokes specific attributes, to combine data from different types of peaks my go-to solution for now is to add dimensions to the shift and linewidth array (if I fit 3 peaks in a mapping of 50 by 50 elements, shift and linewidth will have the following shape: 50x50x3). I do think you have a point there, and it needs to be integrated in the next version of the format, but as is I think it’s better to leave it for the future, mainly because we don’t expect changes in Stokes and anti-Stokes parameters (at least with the way we’re using BLS).For unit extraction, for now it’s the text that is between the parenthesis after the last underscore. It’s not super elegant I agree, but it can accommodate almost anything and is relatively easy to use on the spreadsheet. Maybe we could think about updating that on the next version of the library?Regarding _std datasets, I’ve done a full conversion to a more general “_err” dataset (linewidth_std -> linewidth_err), which I plan to upload after submission (or maybe I can directly correct for that on your code if you agree? I honestly don’t remember if I have already pushed the changes or not). The idea is to patch the problem you underline, and allow other estimators of errors to be stored in the format. This being said, R2 and RMSE cannot quantify an error on both the shift and linewidth as they are essentially scalar values. This is why I advocate for the use of the standard deviation as the standard error estimator, obtained from the covariance matrix estimated from the Hessian and Jacobian matrices that are derived during the fit. Storing the covariance matrix is also not super interesting as the fitted parameters are independent (if they’re not, it means we’re essentially fitting noise). That means that the covariance matrix will always be essentially a diagonal matrix where the diagonal is the variance on the fitting of individual parameters and the covariance of any two different parameters is negligible. I think we already had this discussion a few months back though and the reason why Carlo wanted to keep R2 and RMSE is that people have used it in the past, but here again I think it’s mathematically incorrect to use them as error estimators.For integration, I propose to add it in the sub-menu “Export -> Brim” that opens when you right click on an element of the file when opened with the GUI, and to the main menu under “Actions -> Export -> Brim”. I have not implemented the dragging from the GUI to the desktop (or any other file viewer) but when I’ll get to that, I will also integrate it in the export choices.Best,PierrePierre Bouvet, PhDPost-doctoral FellowMedical University ViennaDepartment of Anatomy and Cell BiologyWahringer Straße 13, 1090 Wien, AustriaOn 8/8/25, at 20:01, Sal La Cavera Iii via Software <software@biobrillouin.org> wrote:Hi everyone,Carlo/Sebastian, I've attached the svg file for the draft of fig 1a, the zip also contains the png output images from Blender. If you need anything, e.g., higher resolution anything, just let me know.I think the conversion library is pretty much finished!In the attached brim_converter.zip, the test run script is called master_BrimConverter_test.py. This allows the user (us for now) to define the input source file (either a brim or brimx file) and the desired output file, both with the correct/desired file extensions. Then both of these are passed to the BrimConverter class and the conversion mode is specified ('brim2brimX' or vice versa). I've added some user-friendly features to the run script, that aren't really needed, but the main syntax to use the conversion library is in general:file_in = ... # string of the filepathfile_out = ... # string of the filepathconvert_this = BrimConverter(file_in, file_out, mode='brimX2brim')convert_this.convert()I have tested this using Carlo's drosophila_LSBM.brim.zarr file as the test brimfile, and one of Pierre's more recent brimx files called Measures.h5 I think. However, Pierre's required a little tmp hardcoding to reshape the PSD since the spatial dimensions of the PSD didn't match the shift dataset.I have given fairly rigorous testing on the above: for example, producing a file the brimX2brim way around, and then passing that back through the brim2brimX way around (and vice versa) and so far so good.A couple things for Carlo/Pierre:Carlo, I see in the manuscript google doc that you guys are discontinuing HDF5 compatibility for brimfile. No worries there. To get everything to play nicely in the BrimConverter library I slightly modified file.py and file_abstraction.py in the following ways:
Sent: 31 July 2025 09:25
To: software@biobrillouin.org <software@biobrillouin.org>
Subject: [Software] Re: [EXTERN] Re: new link for meeting nowHi,Following yesterday’s meeting, here are some screenshots of the HDF5_BLS GUI for the C pane of the figure.I’d probably use the 5th figure which is the most representative of what the GUI does.Best,PierrePierre Bouvet, PhDPost-doctoral FellowMedical University ViennaDepartment of Anatomy and Cell BiologyWahringer Straße 13, 1090 Wien, AustriaOn 30/7/25, at 17:19, Kareem Elsayad via Software <software@biobrillouin.org> wrote:Hi All,Sorry for not making meeting today.Pierre gave me an update of all that was discussed. For the most part I agree with everything concluded.One comment on the data for the Consensus I think was brought up. The labs generally did not provide spectra (except a couple that provided one or two representative spectra shown in figures and SM). Now we could ask people to send, but this would probably not go quickly (also given that it is summer). I also fear processing these may be quite a bit of work (different spectral ranges, degrees of elastic suppression, and naming strategies for e.g. temp sweeps). So I would suggest to not make more work than needed and to keep things in a reasonable time frame, to not pursue this, but happy to if the majority think that this would add significant value.I should over next days also be able to look at text so-far.All the best,KareemFrom: Pierre Bouvet via Software <software@biobrillouin.org>
Reply to: Pierre Bouvet <pierre.bouvet@meduniwien.ac.at>
Date: Wednesday, 30. July 2025 at 16:42
To: <software@biobrillouin.org>
Subject: [EXTERN] [Software] Re: new link for meeting nowHi,This is the website I was mentioning to scan for viruses in files: https://www.virustotal.com/gui/home/uploadFun fact: I remember first hearing of it after they unintentionally leaked thousands of private addresses from US security officials (which got me into trusting them, kind of paradoxical).Best,PierrePierre Bouvet, PhDPost-doctoral FellowMedical University ViennaDepartment of Anatomy and Cell BiologyWahringer Straße 13, 1090 Wien, Austria_______________________________________________ Software mailing list -- software@biobrillouin.org To unsubscribe send an email to software-leave@biobrillouin.org_______________________________________________
Software mailing list -- software@biobrillouin.org
To unsubscribe send an email to software-leave@biobrillouin.orgThis message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please contact the sender and delete the email and attachment. Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham. Email communications with the University of Nottingham may be monitored where permitted by law.<fig1a_sal.zip><brim_converter.zip><file_abstraction.zip>_______________________________________________
Software mailing list -- software@biobrillouin.org
To unsubscribe send an email to software-leave@biobrillouin.org