Hi Sal, In v 1.0 there are no Stokes and anti-Stokes specific attributes, to combine data from different types of peaks my go-to solution for now is to add dimensions to the shift and linewidth array (if I fit 3 peaks in a mapping of 50 by 50 elements, shift and linewidth will have the following shape: 50x50x3). I do think you have a point there, and it needs to be integrated in the next version of the format, but as is I think it’s better to leave it for the future, mainly because we don’t expect changes in Stokes and anti-Stokes parameters (at least with the way we’re using BLS). For unit extraction, for now it’s the text that is between the parenthesis after the last underscore. It’s not super elegant I agree, but it can accommodate almost anything and is relatively easy to use on the spreadsheet. Maybe we could think about updating that on the next version of the library? Regarding _std datasets, I’ve done a full conversion to a more general “_err” dataset (linewidth_std -> linewidth_err), which I plan to upload after submission (or maybe I can directly correct for that on your code if you agree? I honestly don’t remember if I have already pushed the changes or not). The idea is to patch the problem you underline, and allow other estimators of errors to be stored in the format. This being said, R2 and RMSE cannot quantify an error on both the shift and linewidth as they are essentially scalar values. This is why I advocate for the use of the standard deviation as the standard error estimator, obtained from the covariance matrix estimated from the Hessian and Jacobian matrices that are derived during the fit. Storing the covariance matrix is also not super interesting as the fitted parameters are independent (if they’re not, it means we’re essentially fitting noise). That means that the covariance matrix will always be essentially a diagonal matrix where the diagonal is the variance on the fitting of individual parameters and the covariance of any two different parameters is negligible. I think we already had this discussion a few months back though and the reason why Carlo wanted to keep R2 and RMSE is that people have used it in the past, but here again I think it’s mathematically incorrect to use them as error estimators. For integration, I propose to add it in the sub-menu “Export -> Brim” that opens when you right click on an element of the file when opened with the GUI, and to the main menu under “Actions -> Export -> Brim”. I have not implemented the dragging from the GUI to the desktop (or any other file viewer) but when I’ll get to that, I will also integrate it in the export choices. Best, Pierre Pierre Bouvet, PhD Post-doctoral Fellow Medical University Vienna Department of Anatomy and Cell Biology Wahringer Straße 13, 1090 Wien, Austria
On 8/8/25, at 20:01, Sal La Cavera Iii via Software <software@biobrillouin.org> wrote:
Hi everyone,
Carlo/Sebastian, I've attached the svg file for the draft of fig 1a, the zip also contains the png output images from Blender. If you need anything, e.g., higher resolution anything, just let me know.
I think the conversion library is pretty much finished!
In the attached brim_converter.zip, the test run script is called master_BrimConverter_test.py. This allows the user (us for now) to define the input source file (either a brim or brimx file) and the desired output file, both with the correct/desired file extensions. Then both of these are passed to the BrimConverter class and the conversion mode is specified ('brim2brimX' or vice versa). I've added some user-friendly features to the run script, that aren't really needed, but the main syntax to use the conversion library is in general:
file_in = ... # string of the filepath file_out = ... # string of the filepath convert_this = BrimConverter(file_in, file_out, mode='brimX2brim') convert_this.convert()
I have tested this using Carlo's drosophila_LSBM.brim.zarr file as the test brimfile, and one of Pierre's more recent brimx files called Measures.h5 I think. However, Pierre's required a little tmp hardcoding to reshape the PSD since the spatial dimensions of the PSD didn't match the shift dataset.
I have given fairly rigorous testing on the above: for example, producing a file the brimX2brim way around, and then passing that back through the brim2brimX way around (and vice versa) and so far so good.
A couple things for Carlo/Pierre:
Carlo, I see in the manuscript google doc that you guys are discontinuing HDF5 compatibility for brimfile. No worries there. To get everything to play nicely in the BrimConverter library I slightly modified file.py and file_abstraction.py in the following ways: in file_abstraction.py you had a hardcoded flag for whether h5 mode was activated or not, use_h5 = False or something. Then you had an if statement later checking the state of this flag. And by being False this always activated the path containing Zarr etc. This prevented using file_abstraction.py for both h5 files and non-h5 files (e.g. zarr) dynamically. Switching to =True when h5 is wanted isn't a viable long term solution. I fixed this by relocating the StoreType(Enum) class definition to the beginning of file_abstraction.py and adding HDF5 = 'hdf5' as one of the defined StoreTypes (see attached file in the other zip). Then I could delete the use_h5 flag and the related if statements. _h5File class is now H5File class in file_abstraction.py is mainly unchanged (it is outside of any if statements now), except I needed to modify the create_attr() function definition to make sure that data types, and None/NaN's were dealt with correctly. That's also in the attached file_abstraction.py file. Moved the Compression class into FileAbstraction For file.py the main change is not using the single line in the __init__ self._file = _AbstractFile( filename, mode=mode, store_type=store_type) Nothing beyond these 2 files needs to be modified I'm pretty sure.
instead I take the store_type passed as input, and then load the relevant file type's class (each defined in file_abstraction still). H5File for h5 and then ZarrFile for everything else as you previously had it with your if statement flagged by use_h5.
All of the above might not be of any use anymore if you're removing hdf5 compatibility, but using the above/attached fixes I was able to get file_abstraction.py and file.py working for both h5, zarr, and zips, whereas previously it wouldn't work for h5. So now I can create .brim.h5 files and everything plays nicely in all directions.
Pierre, is there scope to have a Brillouin_type attribute for both AS and S measurements? If I pass a brimfile to BrimConverter to produce a brimX, and that brim has AS and S, currently only one of those is passed to brimX because there isn't a distinction between the 2 in the Brillouin_type attributes list. Or is there a different solution you already have for managing this? Do you have a clean way of extracting units (e.g. having them as attributes?). All I could find on the github guide was that you chuck them onto the ends of things like MEASURE.Exposure_(s) which makes scraping a little clunky.
Things to note for everyone: I haven't dealt with metadata yet, as that is lower priority atm. It is on my to-do list though, though I reckon this can be sorted during submission/review etc. Currently AS vs S distinction is lost when converting to brimX BLT and all _std's on Pierre's end don't map to anything on Carlo's end. And R2, RMSE, and Cov_matrix on Carlo's end don't map to anything on Pierre's end.
Do people have opinions on how to roll out the conversion library? Integrated into brimfile and HDF5_BLS libraries? Currently it's pretty straightforward to use in scripting, but perhaps we could integrate it as a button in Pierre's GUI?
I have integrated relevant explanations into the manuscript, but have not formally written documentation for the library as I'd prefer to wait for any feedback. I don't see any point in making online documentation that is separate to the brimfile and/or HDF5_BLS documentation, so would hope to port it to your guys' sites. HDF5_BLS one is easy enough b/c it's on github.
Any questions/comments/edits just let me know. Hope everyone has a nice weekend,
Best,
Sal
--------------------------------------------------------------- Salvatore La Cavera III Royal Academy of Engineering Research Fellow Nottingham Research Fellow Optics and Photonics Group University of Nottingham Email: salvatore.lacaveraiii@nottingham.ac.uk <mailto:salvatore.lacaveraiii@nottingham.ac.uk> ORCID iD: 0000-0003-0210-3102 <image.png> <https://outlook.office.com/bookwithme/user/6a3f960a8e89429cb6fc693c01d10119@...> Book a Coffee and Research chat with me! From: Pierre Bouvet via Software <software@biobrillouin.org> Sent: 31 July 2025 09:25 To: software@biobrillouin.org <software@biobrillouin.org> Subject: [Software] Re: [EXTERN] Re: new link for meeting now
Hi,
Following yesterday’s meeting, here are some screenshots of the HDF5_BLS GUI for the C pane of the figure. I’d probably use the 5th figure which is the most representative of what the GUI does.
Best,
Pierre
Pierre Bouvet, PhD Post-doctoral Fellow Medical University Vienna Department of Anatomy and Cell Biology Wahringer Straße 13, 1090 Wien, Austria
On 30/7/25, at 17:19, Kareem Elsayad via Software <software@biobrillouin.org> wrote:
Hi All,
Sorry for not making meeting today.
Pierre gave me an update of all that was discussed. For the most part I agree with everything concluded.
One comment on the data for the Consensus I think was brought up. The labs generally did not provide spectra (except a couple that provided one or two representative spectra shown in figures and SM). Now we could ask people to send, but this would probably not go quickly (also given that it is summer). I also fear processing these may be quite a bit of work (different spectral ranges, degrees of elastic suppression, and naming strategies for e.g. temp sweeps). So I would suggest to not make more work than needed and to keep things in a reasonable time frame, to not pursue this, but happy to if the majority think that this would add significant value.
I should over next days also be able to look at text so-far.
All the best, Kareem
From: Pierre Bouvet via Software <software@biobrillouin.org <mailto:software@biobrillouin.org>> Reply to: Pierre Bouvet <pierre.bouvet@meduniwien.ac.at <mailto:pierre.bouvet@meduniwien.ac.at>> Date: Wednesday, 30. July 2025 at 16:42 To: <software@biobrillouin.org> Subject: [EXTERN] [Software] Re: new link for meeting now
Hi,
This is the website I was mentioning to scan for viruses in files: https://www.virustotal.com/gui/home/upload Fun fact: I remember first hearing of it after they unintentionally leaked thousands of private addresses from US security officials (which got me into trusting them, kind of paradoxical).
Best,
Pierre
Pierre Bouvet, PhD Post-doctoral Fellow Medical University Vienna Department of Anatomy and Cell Biology Wahringer Straße 13, 1090 Wien, Austria
_______________________________________________ Software mailing list -- software@biobrillouin.org <mailto:software@biobrillouin.org> To unsubscribe send an email to software-leave@biobrillouin.org <mailto:software-leave@biobrillouin.org> _______________________________________________ Software mailing list -- software@biobrillouin.org <mailto:software@biobrillouin.org> To unsubscribe send an email to software-leave@biobrillouin.org <mailto:software-leave@biobrillouin.org>
This message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please contact the sender and delete the email and attachment. Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham. Email communications with the University of Nottingham may be monitored where permitted by law.<fig1a_sal.zip><brim_converter.zip><file_abstraction.zip>_______________________________________________ Software mailing list -- software@biobrillouin.org <mailto:software@biobrillouin.org> To unsubscribe send an email to software-leave@biobrillouin.org <mailto:software-leave@biobrillouin.org>