- Software - lists.biobrillouin.org

Proposition of paragraph for next Newsletter
by Pierre Bouvet Sept. 1, 2025

Sept. 1, 2025

Hi everyone, Following this afternoon’s meeting, here is a proposition for a paragraph for the next newsletter: The “contact us” sends to a new email sent to “software(a)biobrillouin.org <mailto:software@biobrillouin.org>” (the address I’m sending this email to), the links to GitHub send to the two GitHub repositories. This is a “by default” version, but if you’d want to change or add things, just tell me, I’m planning on sending the newsletter on Monday. Best, Pierre PS: sorry if I was grumpy during the meeting, I’ve been trying to convince a spectrometer to work for weeks and I am starting to think it has a will of its own and that its will is entirely focused on making sure it does NOT work… I guess that’s what makes research “fun” Pierre Bouvet, PhD Post-doctoral Fellow Medical University Vienna Department of Anatomy and Cell Biology Wahringer Straße 13, 1090 Wien, Austria

4 7

Re: [EXTERN] Re: new link for meeting now
by Pierre Bouvet Aug. 27, 2025

Aug. 27, 2025

Hi everyone, @ Sal I would suggest adding it to the backend with a dedicated function of the wrapper class with a header looking like: “def export_brim(self, path, filepath)” and then adding the menu buttons to the “show_treeview_context_menu” of the main file of the HDF5_BLS_GUI package, ideally with a bit of logic to ensure only datasets compatible with your module can be exported to brim. The question is then do you want to make your module an independent package or to join it to either Carlo’s or mine. I would suggest adding it to HDF5_BLS because I’m sure there will be no fundamental limitations. I can do it if you want :) Enjoy Banyuls! @ Carlo & Sebastian The code on supplementary for Brim still doesn’t work. I went to dig in the sources and the problem comes from your use of the Enum class: you’re mixing literal expressions and patterns. I think your code also lacks commenting, as is it’s really hard to read and fix. After modifying your library, I tried exporting a synthetic BLS image to Brim using the code you provided. I do get a .zip file or .zarr file where I am supposed to have it, but I cannot say if it works or not since the viewer on the BioBrillouin website is unable to open it and there are no compatible softwares out there. I think it’s important this is fixed, and I would suggest doing as I did in supplementary, by showing an example of integration of your code to an existing process. Next, from what I could understand from your code, you can only add 4D PSD arrays to Brim, I think this is non intuitive and could easily be improved by adding silent dimensions in your process. As I’ve already mentioned before, this is in any case extremely limiting for a number of applications, starting with all TFP measures and time-series. Last problem I see, but that’s proper to zarr: you end up saving a zipped file, which can be corrupted easily by the user if he unzips and then re-zips his file. To limit the risks, I would strongly recommend adding a hash key to check file integrity, generated after each addition or modification to the file, and compared to the computed hash key at opening. Best, Pierre Pierre Bouvet, PhD Post-doctoral Fellow Medical University Vienna Department of Anatomy and Cell Biology Wahringer Straße 13, 1090 Wien, Austria > On 26/8/25, at 17:29, Sal La Cavera Iii <Salvatore.Lacaveraiii(a)nottingham.ac.uk> wrote: > > Hi guys, > > Sorry I am a little slow on communications right now because I am at a summer school that I have been helping organise in Banyuls sur Mer this week and last week. > > Thanks for the initial feedback on the conversion library! 2 follow ups: > Pierre, I've been meaning to check out where to insert the BrimConverter to HDF5_BLS based on your suggestions (Export and Actions-Export buttons) but I am currently unable to clone the HDF5_BLS repo because of the venue's wifi firewall. I can clone using my phone's hotspot but mobile network is really slow and is going to eat all my data. In essence it should be really easy though right? BrimConverter() is just a 1-liner and I imagine clicking on it should just bring up a window to select the input file and define the output file name. > Carlo, I've recently upgraded my brimfile version just to check compatibility from the BrimConverter end. It looks like that use_h5 hardcoded flag is still in file_abstraction.py. Probably it would be best to remove that and the related if-statements that query use_h5's value if you're going to deactivate h5 compatibilty? If you ever did want to bring hdf5 back I have some updated versions of file_abstraction.py that play nicely with h5 in a generalised way if you ever needed/were curious. > > See you guys on Thursday, > > Cheers, > > Sal > > --------------------------------------------------------------- > Salvatore La Cavera III > Royal Academy of Engineering Research Fellow > Nottingham Research Fellow > Optics and Photonics Group > University of Nottingham > Email: salvatore.lacaveraiii(a)nottingham.ac.uk <mailto:salvatore.lacaveraiii@nottingham.ac.uk> > ORCID iD: 0000-0003-0210-3102 > <https://outlook.office.com/bookwithme/user/6a3f960a8e89429cb6fc693c01d10119…> > Book a Coffee and Research chat with me! > From: Carlo Bevilacqua via Software <software(a)biobrillouin.org <mailto:software@biobrillouin.org>> > Sent: 11 August 2025 12:41 > To: software(a)biobrillouin.org <mailto:software@biobrillouin.org> <software(a)biobrillouin.org <mailto:software@biobrillouin.org>> > Subject: [Software] Re: new link for meeting now > > Hi Sal, > thanks a lot for sending the figures and the brim_converter module. I will try to run it in the next days and give you feedback. In the meanwhile, if you want to test it on more brim files, I am uploading on an S3 bucket a few example brim files from different modalities in our lab; you can find the link to them in the `Example brim files` section in the Supplementary of the paper draft <https://docs.google.com/document/d/1Qg6gwRQ8EzmyfbjBKin8atMTCTtewu1THTUlb_z…>. > > Regarding HDF5 support, I initially design the brimfile package to easily support different file formats (that's why I introduced the abstract class `FileAbstraction`). > Later I thought that I don't see a clear advantage of HDF5 over zarr (apart from being more mature and thus more tools being available) and if we want to advocate for the use of Zarr it is better to not support both in the initial phase, otherwise people would start using both and force us to keep supporting HDF5 as well. > If in the future we realize that supporting HDF5 will be beneficial, we can easily do it (as you have already experienced). > > Regarding the use of R2 and RMSE, you can just ignore them if they are not on Pierre's end, as they are optional. I still believe that it makes sense to keep them in the definition of the file format though, as they are commonly used metrics to estimate the goodness of fit (even though I agree that they are by no means complete or informative about the error on the fit parameters). > Regarding the covariance matrix, I feel that that is the quantity that you get directly from the fit and the actual errors on the parameters can be easily computed from there by the visualization software. The only drawback is an increase of the file size, which, in any case, won't be much (it is ~10 additional numbers for each fit while a spectrum has typically several tens of points; also, if it is indeed a purely diagonal matrix, compression should work very well). Instead the advantage of storing it would be that it helps troubleshooting the fit (e.g. in case it turns out not to be diagonal). > > Regarding distributing the conversion script, as you are only importing brimfile and not HDF5_BLS, I would propose to make it a submodule of brimfile (i.e. a folder `converter` inside `brimfile` so that you can import it by `import brimfile.converter`). I would include any additional dependencies you need (e.g. h5py) into an optional dependency <https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#depen…> `converter`, so people can do `pip install brimfile[converter]` only if they need it. Anyways, even if you need to use it from the HDF5_BLS library, you need to install brimfile, so by making it a submodule you are not forcing to install any unnecessary dependency. > If you go for this option you can write the documentation into an '__init__.py' file inside the 'brimfile/converter' folder and it will be automatically added to the online documentation of the brimfile library (I am using a github action which triggers pdoc <https://pdoc.dev/> to compile the documentation and upload it online at every push). The docstring of each function will also be automatically compiled to the documentation. > If you think that is a good option, you can just make a pull request to the brimfile repository. Let me know what you think about it and if you want to discuss some technical detail. > > > Best, > Carlo > > > > On Fri, Aug 8, 2025 at 21:21, Pierre Bouvet via Software <software(a)biobrillouin.org <mailto:software@biobrillouin.org>> wrote: > Hi Sal, > > In v 1.0 there are no Stokes and anti-Stokes specific attributes, to combine data from different types of peaks my go-to solution for now is to add dimensions to the shift and linewidth array (if I fit 3 peaks in a mapping of 50 by 50 elements, shift and linewidth will have the following shape: 50x50x3). I do think you have a point there, and it needs to be integrated in the next version of the format, but as is I think it’s better to leave it for the future, mainly because we don’t expect changes in Stokes and anti-Stokes parameters (at least with the way we’re using BLS). > For unit extraction, for now it’s the text that is between the parenthesis after the last underscore. It’s not super elegant I agree, but it can accommodate almost anything and is relatively easy to use on the spreadsheet. Maybe we could think about updating that on the next version of the library? > Regarding _std datasets, I’ve done a full conversion to a more general “_err” dataset (linewidth_std -> linewidth_err), which I plan to upload after submission (or maybe I can directly correct for that on your code if you agree? I honestly don’t remember if I have already pushed the changes or not). The idea is to patch the problem you underline, and allow other estimators of errors to be stored in the format. This being said, R2 and RMSE cannot quantify an error on both the shift and linewidth as they are essentially scalar values. This is why I advocate for the use of the standard deviation as the standard error estimator, obtained from the covariance matrix estimated from the Hessian and Jacobian matrices that are derived during the fit. Storing the covariance matrix is also not super interesting as the fitted parameters are independent (if they’re not, it means we’re essentially fitting noise). That means that the covariance matrix will always be essentially a diagonal matrix where the diagonal is the variance on the fitting of individual parameters and the covariance of any two different parameters is negligible. I think we already had this discussion a few months back though and the reason why Carlo wanted to keep R2 and RMSE is that people have used it in the past, but here again I think it’s mathematically incorrect to use them as error estimators. > For integration, I propose to add it in the sub-menu “Export -> Brim” that opens when you right click on an element of the file when opened with the GUI, and to the main menu under “Actions -> Export -> Brim”. I have not implemented the dragging from the GUI to the desktop (or any other file viewer) but when I’ll get to that, I will also integrate it in the export choices. > > Best, > > Pierre > > Pierre Bouvet, PhD > Post-doctoral Fellow > Medical University Vienna > Department of Anatomy and Cell Biology > Wahringer Straße 13, 1090 Wien, Austria > > > > > > On 8/8/25, at 20:01, Sal La Cavera Iii via Software <software(a)biobrillouin.org <mailto:software@biobrillouin.org>> wrote: > > Hi everyone, > > Carlo/Sebastian, I've attached the svg file for the draft of fig 1a, the zip also contains the png output images from Blender. If you need anything, e.g., higher resolution anything, just let me know. > > I think the conversion library is pretty much finished! > > In the attached brim_converter.zip, the test run script is called master_BrimConverter_test.py. This allows the user (us for now) to define the input source file (either a brim or brimx file) and the desired output file, both with the correct/desired file extensions. Then both of these are passed to the BrimConverter class and the conversion mode is specified ('brim2brimX' or vice versa). I've added some user-friendly features to the run script, that aren't really needed, but the main syntax to use the conversion library is in general: > > file_in = ... # string of the filepath > file_out = ... # string of the filepath > convert_this = BrimConverter(file_in, file_out, mode='brimX2brim') > convert_this.convert() > > I have tested this using Carlo's drosophila_LSBM.brim.zarr file as the test brimfile, and one of Pierre's more recent brimx files called Measures.h5 I think. However, Pierre's required a little tmp hardcoding to reshape the PSD since the spatial dimensions of the PSD didn't match the shift dataset. > > I have given fairly rigorous testing on the above: for example, producing a file the brimX2brim way around, and then passing that back through the brim2brimX way around (and vice versa) and so far so good. > > A couple things for Carlo/Pierre: > > Carlo, I see in the manuscript google doc that you guys are discontinuing HDF5 compatibility for brimfile. No worries there. To get everything to play nicely in the BrimConverter library I slightly modified file.py and file_abstraction.py in the following ways: > in file_abstraction.py you had a hardcoded flag for whether h5 mode was activated or not, use_h5 = False or something. Then you had an if statement later checking the state of this flag. And by being False this always activated the path containing Zarr etc. This prevented using file_abstraction.py for both h5 files and non-h5 files (e.g. zarr) dynamically. Switching to =True when h5 is wanted isn't a viable long term solution. > I fixed this by relocating the StoreType(Enum) class definition to the beginning of file_abstraction.py and adding HDF5 = 'hdf5' as one of the defined StoreTypes (see attached file in the other zip). Then I could delete the use_h5 flag and the related if statements. > _h5File class is now H5File class in file_abstraction.py is mainly unchanged (it is outside of any if statements now), except I needed to modify the create_attr() function definition to make sure that data types, and None/NaN's were dealt with correctly. That's also in the attached file_abstraction.py file. > Moved the Compression class into FileAbstraction > For file.py the main change is not using the single line in the __init__ > self._file = _AbstractFile( filename, mode=mode, store_type=store_type) > Nothing beyond these 2 files needs to be modified I'm pretty sure. > > instead I take the store_type passed as input, and then load the relevant file type's class (each defined in file_abstraction still). H5File for h5 and then ZarrFile for everything else as you previously had it with your if statement flagged by use_h5. > > All of the above might not be of any use anymore if you're removing hdf5 compatibility, but using the above/attached fixes I was able to get file_abstraction.py and file.py working for both h5, zarr, and zips, whereas previously it wouldn't work for h5. So now I can create .brim.h5 files and everything plays nicely in all directions. > > Pierre, > is there scope to have a Brillouin_type attribute for both AS and S measurements? If I pass a brimfile to BrimConverter to produce a brimX, and that brim has AS and S, currently only one of those is passed to brimX because there isn't a distinction between the 2 in the Brillouin_type attributes list. Or is there a different solution you already have for managing this? > Do you have a clean way of extracting units (e.g. having them as attributes?). All I could find on the github guide was that you chuck them onto the ends of things like MEASURE.Exposure_(s) which makes scraping a little clunky. > > Things to note for everyone: > I haven't dealt with metadata yet, as that is lower priority atm. It is on my to-do list though, though I reckon this can be sorted during submission/review etc. > Currently AS vs S distinction is lost when converting to brimX > BLT and all _std's on Pierre's end don't map to anything on Carlo's end. And R2, RMSE, and Cov_matrix on Carlo's end don't map to anything on Pierre's end. > > Do people have opinions on how to roll out the conversion library? Integrated into brimfile and HDF5_BLS libraries? Currently it's pretty straightforward to use in scripting, but perhaps we could integrate it as a button in Pierre's GUI? > > I have integrated relevant explanations into the manuscript, but have not formally written documentation for the library as I'd prefer to wait for any feedback. I don't see any point in making online documentation that is separate to the brimfile and/or HDF5_BLS documentation, so would hope to port it to your guys' sites. HDF5_BLS one is easy enough b/c it's on github. > > Any questions/comments/edits just let me know. Hope everyone has a nice weekend, > > Best, > > Sal > > --------------------------------------------------------------- > Salvatore La Cavera III > Royal Academy of Engineering Research Fellow > Nottingham Research Fellow > Optics and Photonics Group > University of Nottingham > Email: salvatore.lacaveraiii(a)nottingham.ac.uk <mailto:salvatore.lacaveraiii@nottingham.ac.uk> > ORCID iD: 0000-0003-0210-3102 > <image.png> <https://outlook.office.com/bookwithme/user/6a3f960a8e89429cb6fc693c01d10119…> > Book a Coffee and Research chat with me! > From: Pierre Bouvet via Software <software(a)biobrillouin.org <mailto:software@biobrillouin.org>> > Sent: 31 July 2025 09:25 > To: software(a)biobrillouin.org <mailto:software@biobrillouin.org> <software(a)biobrillouin.org <mailto:software@biobrillouin.org>> > Subject: [Software] Re: [EXTERN] Re: new link for meeting now > > Hi, > > Following yesterday’s meeting, here are some screenshots of the HDF5_BLS GUI for the C pane of the figure. > I’d probably use the 5th figure which is the most representative of what the GUI does. > > Best, > > Pierre > > Pierre Bouvet, PhD > Post-doctoral Fellow > Medical University Vienna > Department of Anatomy and Cell Biology > Wahringer Straße 13, 1090 Wien, Austria > > > > > On 30/7/25, at 17:19, Kareem Elsayad via Software <software(a)biobrillouin.org <mailto:software@biobrillouin.org>> wrote: > > Hi All, > > Sorry for not making meeting today. > > Pierre gave me an update of all that was discussed. For the most part I agree with everything concluded. > > One comment on the data for the Consensus I think was brought up. The labs generally did not provide spectra (except a couple that provided one or two representative spectra shown in figures and SM). Now we could ask people to send, but this would probably not go quickly (also given that it is summer). I also fear processing these may be quite a bit of work (different spectral ranges, degrees of elastic suppression, and naming strategies for e.g. temp sweeps). So I would suggest to not make more work than needed and to keep things in a reasonable time frame, to not pursue this, but happy to if the majority think that this would add significant value. > > I should over next days also be able to look at text so-far. > > All the best, > Kareem > > > > > From: Pierre Bouvet via Software <software(a)biobrillouin.org <mailto:software@biobrillouin.org>> > Reply to: Pierre Bouvet <pierre.bouvet(a)meduniwien.ac.at <mailto:pierre.bouvet@meduniwien.ac.at>> > Date: Wednesday, 30. July 2025 at 16:42 > To: <software(a)biobrillouin.org <mailto:software@biobrillouin.org>> > Subject: [EXTERN] [Software] Re: new link for meeting now > > Hi, > > This is the website I was mentioning to scan for viruses in files: https://www.virustotal.com/gui/home/upload > Fun fact: I remember first hearing of it after they unintentionally leaked thousands of private addresses from US security officials (which got me into trusting them, kind of paradoxical). > > Best, > > Pierre > > Pierre Bouvet, PhD > Post-doctoral Fellow > Medical University Vienna > Department of Anatomy and Cell Biology > Wahringer Straße 13, 1090 Wien, Austria > > > > _______________________________________________ Software mailing list -- software(a)biobrillouin.org <mailto:software@biobrillouin.org> To unsubscribe send an email to software-leave(a)biobrillouin.org <mailto:software-leave@biobrillouin.org> > _______________________________________________ > Software mailing list -- software(a)biobrillouin.org <mailto:software@biobrillouin.org> > To unsubscribe send an email to software-leave(a)biobrillouin.org <mailto:software-leave@biobrillouin.org> > > This message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please contact the sender and delete the email and attachment. Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham. Email communications with the University of Nottingham may be monitored where permitted by law.<fig1a_sal.zip><brim_converter.zip><file_abstraction.zip>_______________________________________________ > Software mailing list -- software(a)biobrillouin.org <mailto:software@biobrillouin.org> > To unsubscribe send an email to software-leave(a)biobrillouin.org <mailto:software-leave@biobrillouin.org> > > This message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please contact the sender and delete the email and attachment. Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham. Email communications with the University of Nottingham may be monitored where permitted by law.

2 1

new link for meeting now
by Robert Prevedel Aug. 11, 2025

Aug. 11, 2025

https://embl-org.zoom.us/j/8640411259?pwd=NkZsdDJGRWk4U1JMVzVTMWtZcjFodz09 -- Dr. Robert Prevedel Group Leader and Senior Scientist Cell Biology and Biophysics Unit European Molecular Biology Laboratory Meyerhofstr. 1 69117 Heidelberg, Germany Phone: +49 6221 387-8722 Email: robert.prevedel(a)embl.de http://www.prevedel.embl.de

5 7

outline of the manuscript
by Carlo Bevilacqua July 14, 2025

July 14, 2025

Dear all, we put together an outline of the paper here (https://docs.google.com/document/d/1HHOcbm4111wyifnk-Jr36vdAycFjAnFU3lIaKEI…) (I added it to a google doc I shared with you some time ago). We outlined the main content and the work division. We can use this as a basis for discussion tomorrow. I think it is important that we reach an agreement on this in tomorrow's meeting, so that we can start with the actual writing. Talk to you tomorrow. Best regards, Carlo

3 2

name for the webapp
by Carlo Bevilacqua July 14, 2025

July 14, 2025

Hi all, Sebastian is working on a first version of the webapp which we could have online for people to test. We are thus deciding a name for it. Since the file format is called brim, we haveshortlisted two options for the name (with the corresponding logo attached): * BrimView (or BrimVue) (descriptive and scientific name) * SeeBrim (more fun and catchy name that still convey the idea of looking at brim files) Would you have a preference for one of the two? Best regards, Carlo

4 3

Re: [EXTERN] Practice dataset
by Carlo Bevilacqua May 22, 2025

May 22, 2025

Hi Sal, thanks a lot for writing the script. Actually the specs changed a bit and here (https://github.com/prevedel-lab/Brillouin-standard-file/blob/main/docs/BLS_…) is the most updated version (nothing major, just better defining some fields). The main difference though is that we are now using Zarr file (as we discussed in our last meeting). In the meanwhile I wrote a library which exposes an intuitive interface to read/write to our file format (it is not online yet, but will be soon). I think the best way would be to use that to actually export Pierre's hdf5 to ours. Also Sebastian progressed quite a bit on the GUI side. Shall we have a meeting to update on each other's status? Would Monday 26th in the afternoon (e.g. 3pm) work for you? Best, Carlo On Wed, May 21, 2025 at 17:44, Sal La Cavera Iii wrote: Hi guys, Hope everyone's doing well! Found some time to get this hierarchical-to-flat conversion script off the ground. In the below wetransfer link (3 days until it expires, so let me know if you need me to send it again) there's a python script that should be run with Pierre's multiparameter cell imaging dataset (h5 file) in the same directory. It doesn't map to every sub-group in Carlo's file_spec document (https://github.com/bio-brillouin/HDF5_BLS/blob/main/Bh5_file_spec.md) but it does the basic ones like Raw_data, Frequency, Shift, Linewidth, *_std, etc. Additional datasets/attributes should be fairly easy to add, but probably this will require a different test file that contains some of these other parameters (e.g. PSD, Amplitude, IRF, Calibration, etc). It will probably require some rigidity on behalf of the end-user, e.g., when they're using Pierre's software to create an h5 file, the user shouldn't have the option to save the Frequency data as 'freq' or 'f' it'll have to be 'Frequency.' Also, this version doesn't accommodate multiple signal processing runs of the same dataset (Carlo parametrised this as /Analysis_{m} in the file spec), but again this should be pretty straight forward to implement (having a test file that contained {m+1} would be useful). Right now Analysis_0 is hardcoded in. Carlo/Sebastian does this file format trend towards what you're looking for for handing over Pierre's h5 file to your plotting side of things? https://we.tl/t-TszCgBzwfE (https://we.tl/t-TszCgBzwfE) Any feedback etc just let me know, happy to chat further, Cheers, Sal --------------------------------------------------------------- Salvatore La Cavera III Royal Academy of Engineering Research Fellow Nottingham Research Fellow Optics and Photonics Group University of Nottingham Email: salvatore.lacaveraiii(a)nottingham.ac.uk (mailto:salvatore.lacaveraiii@nottingham.ac.uk) ORCID iD: 0000-0003-0210-3102 (https://outlook.office.com/bookwithme/user/6a3f960a8e89429cb6fc693c01d10119…) Book a Coffee and Research chat with me! ------------------------------------ From: Pierre Bouvet Sent: 17 April 2025 12:35 To: Sebastian Hambura Cc: Sal La Cavera iii (staff) ; Carlo Bevilacqua Subject: Re: [EXTERN] Practice dataset Hi Sebastian, Was just writing to you to say that in my hurry, I didn’t added the treated data for cell.h5. I’ll do that whenever I get a bit of time or when the GUI will be able to do it (most likely option B except if you want it fast then I can spend a few minutes doing that by hand, you tell me :) ). Your question is good, the way you’d do it for these data is to recover the index on the azimuthal and polar calibration datasets. It’s indeed impractical and very specific to this kind of data (I had the same problem with accessing the time axis with time-domain datasets). For now however, I’d say that it’s sufficient since - that I know of - I have the only spectrometer able to do that, and there are technique-specific ways to access this information. Since I guess you also want to be able to convert these information into your data format and not have to create technique-specific access function, one way to avoid altogether this issue could be to define all the treated datasets as 2D arrays with size - (512x512) in this example. Then you would have the shift[i, j] associated with PSD[i, j, :]. If you have other ideas, I’m open to suggestions :) Best, Pierre Pierre Bouvet, PhD Post-doctoral Fellow Medical University Vienna Department of Anatomy and Cell Biology Wahringer Straße 13, 1090 Wien, Austria On 17/4/25, at 13:19, Sebastian Hambura wrote: Hi Pierre, Ok, this clarifies a bit the whole thing. But then concretly, what is the spectra (as intensity = f(frequency) ) used for a given treated point ? For example, if I wanted to see the underlaying spectra (or acquired data + fit) of /Brillouin/Mock anisotropic sample/Treated/Shift[1000], what part of /Brillouin/Mock anisotropic sample/Frequency and /Brillouin/Mock anisotropic sample/Intensity would I need to display ? Best, Sebastian Sebastian Hambura Software Engineer Robert Prevedel Group Cell Biology and Biophysics EMBL Heidelberg Meyerhofstraße 1 69117 Heidelberg, Germany Am 17.04.2025 um 11:50 schrieb Pierre Bouvet: Hi Sebastian, Sorry for the lack of details, I’ll try to explain it as best I can For the Frequency and Intensity datasets, there are the size of my detector (for now a 512x512 but hopefully soon a 2048x2048 if we can get the money ^^). By nature, I am entangling 3 hyper parameters in my measures: shift frequency, azimuthal angle of observation, polar angle of observation (with errors on each of them that I would add later on but let’s say my device is perfect for now). Now the VIPA spectrometer creates constructive interferences for specific frequencies so after treatment, I will only get treated values for the points where constructive interference happened. For each of these points I will therefore have a shift, a linewidth, a value for the azimuthal angle and a value for the polar angle. The most direct way of storing the results is therefore to keep everything 1D and then, when trying to represent things, use a color mesh with the azimuthal and polar abscissa as the x and y positions of the shift and linewidth. For the calibration now, this is different because we can consider that the angular response of water follows the theory for homogeneous and isotropic solutions, meaning that we will essentially have no dependence of the shift and linewidth on the polar angles but a dependence of sin(theta/2) on the azimuthal angles. Therefore, it is possible to reconstruct a 2D image for the shift and linewidth on all the points of the detector (which are themselves associated with a value for azimuthal and polar angles, as you can see in the calibration group). That’s weird for cells.h5, H5Viewer told me nothing on VS, but on the other hand I might have my answer for why the file was super heavy, just need to figure out how to delete groups and datasets without truncating the file. Here is the same file but made without deleting files from the HDF5 file (I tried it on myHDF5 it seems to work): https://we.tl/t-WlmabmsXHo (https://we.tl/t-WlmabmsXHo) Best, Pierre Pierre Bouvet, PhD Post-doctoral Fellow Medical University Vienna Department of Anatomy and Cell Biology Wahringer Straße 13, 1090 Wien, Austria On 17/4/25, at 11:27, Sebastian Hambura (mailto:sebastian.hambura@embl.de) wrote: Hi Pierre, I'm currently looking at your angle_resolved_dataset.h5, and maybe it's because I'm not familiar with your specific setup but I'm a bit confused: - Frequency and Intensity are 512*512 for all 3 experiments/datagroups - Treated for 'Mock anisotropic sample' is 3584(*1) : are you somehow using the same row of your PSD for multiple treated datapoints ? Or are you doing some 'subpixel' rows ? - Treated for '/Brillouin/Water spectrum for angle calibration' is 512*512: how does that work from your PSD data ? For the cells.h5 file, I tried opening it with https://myhdf5.hdfgroup.org (https://myhdf5.hdfgroup.org/) to explore it, and the tool refusing to open it and report this error: (...) unable to read superblock major: File accessibility minor: Read failed #006: /__w/libhdf5-wasm/libhdf5-wasm/build/1.14.2/_deps/hdf5-src/src/H5Fsuper.c line 603 in H5F__super_read(): truncated file: eof = 309329920, sblock->base_addr = 0, stored_eof = 537790152 major: File accessibility minor: File has been truncated I'll see if other tools might still be able to read it Cheers, Sebastian Sebastian Hambura Software Engineer Robert Prevedel Group Cell Biology and Biophysics EMBL Heidelberg Meyerhofstraße 1 69117 Heidelberg, Germany Am 14.04.2025 um 17:56 schrieb Pierre Bouvet: Sure, here is the link for the ar-VIPA mock H5 file: https://we.tl/t-Sd3ZC8wIXn (https://we.tl/t-Sd3ZC8wIXn) For a more classic file, here are some measures of cells (I only kept 2 cell types, 2 days of measure and 2 samples per day of measure for a total of 8 spectra + their frequency/treated arrays). I treated them using a custom code so I didn’t export the treatment steps in the JSON file I mentioned @ last meeting. Some of the measures are rubish but I guess you don’t care about it ^^ One problem I observed though is that I created this file by deleting elements from a larger file but it seems the memory is still allocated to the HDF5 file. So we get a 500MB file when a classic size for 8 spectra et al is usually under 100Mb. I’ll dig into that when I get time. Here is the link: https://we.tl/t-HUAf2lR9AF (https://we.tl/t-HUAf2lR9AF) Best, Pierre Pierre Bouvet, PhD Post-doctoral Fellow Medical University Vienna Department of Anatomy and Cell Biology Wahringer Straße 13, 1090 Wien, Austria On 14/4/25, at 17:04, Sal La Cavera Iii (mailto:Salvatore.Lacaveraiii@nottingham.ac.uk) wrote: Hi both, Pierre I think that WeTransfer link expired! Any chance you can regenerate it? Potentially the simulated angle-resolved dataset will be a good late-stage-dev to test the software's generalisability? In the first instance should we just use a basic 2D scan of a cell or zebrafish, c. elegans, etc? From a 2-stage VIPA e.g. Something we can treat as the MNIST or ImageNet dataset in the Deep Learning community. Carlo, what about something from your 2019 zebra fish paper? (unless you guys have something else you've already both been using for dev purposes). Safe travels guys! Enjoy that Colomba pasquale Carlo! My favourite are the pistachio ones :) Cheers, Sal --------------------------------------------------------------- Salvatore La Cavera III Royal Academy of Engineering Research Fellow Nottingham Research Fellow Optics and Photonics Group University of Nottingham Email: salvatore.lacaveraiii(a)nottingham.ac.uk (mailto:salvatore.lacaveraiii@nottingham.ac.uk) ORCID iD: 0000-0003-0210-3102 (https://outlook.office.com/bookwithme/user/6a3f960a8e89429cb6fc693c01d10119…) Book a Coffee and Research chat with me! ------------------------------------ From: Pierre Bouvet Sent: 14 April 2025 10:04 To: Carlo Bevilacqua Cc: Sal La Cavera iii (staff) ; Sebastian Hambura Subject: Re: [EXTERN] Practice dataset Hi, That sounds awesome, wish I could come with you to Italy, must be wonderful at this time of year!! Perfect for legal department @ EMBL. To be clear, I think your local approach is robust by design but I’m always scared of having things online (that’s why I never used GitHub before this project ^^). In particular here, my concern is mainly on people being able to have somehow access to the data that will be opened or treated (I’m still traumatized by how stupidly complex it was to work with human samples in France legally speaking) and people being able to modify the code so it downloads whatever. I’ve looked a bit this weekend and people @ HDF have done things with safety in mind so there are normally no SQL-like attacks possible and no possibility of saving compiled programs so I guess just adding a layer between your code and the user to filter all non-HDF5 files coming in or out, and adding hashing to the original data to confirm that no corruption will occur is enough but I’m no specialist in cybersecurity. For third-party attacks it’s trickier. I looked into the H5Web GitHub page to try and understand how they did it, but I’m not super familiar with TypeScript so aside the HTTPS protocol, I don’t really know how they do it. I might be over-worrying here (kind of my specialty) but I think if this project is successful and BLS develops enough, it will become a real problem so might as well do it now. Have a nice time in Italy! Best, Pierre Pierre Bouvet, PhD Post-doctoral Fellow Medical University Vienna Department of Anatomy and Cell Biology Wahringer Straße 13, 1090 Wien, Austria On 13/4/25, at 13:18, Carlo Bevilacqua (mailto:carlo.bevilacqua@embl.de) wrote: Hi guys, thanks for getting started the process of generating a standard dataset. Sorry for the late reply but in the last days I was focusing more on lab work as I want to finish something before flying to Italy next Thursday. I will do home office when in Italy and I will try to generate a file with some experimental data. @Pierre regarding your concerns about GDPR, we could for sure double check with the legal department at EMBL at a later stage, but I want to highlight that the data is not being transfered to any external server, it is just being read by the browser (basically the browser is acting like your Qt GUI and in principle it could run completely offline once the website is loaded). I think we should put a disclaimer on the website about this and maybe the legal department could help on the exact phrasing. Best, Carlo On Tue, Apr 8, 2025 at 18:11, Pierre Bouvet (mailto:pierre.bouvet@meduniwien.ac.at) wrote: Hi everyone, I’ve created a mock file that would correspond to an angle-resolved measurement with calibration in angle and impulse response function. It’s a little heavy so I prefer to send it using wetransfer (all the data are simulated): https://we.tl/t-Om7OrG9Mnu (https://we.tl/t-Om7OrG9Mnu) I’m sending you this one first because it is the most exotic BLS technique in terms of the needs for data storage (I think). It asked me to revise the format a lot when I started working on it. The trickiest conceptual limitation is that I place the detector in the Fourier plane of the sample, meaning that I entangle the frequency axis, the azimuthal angles of collections, the polar angle of collections, and the positions on the detector, leading to the need of multiple abscissa defined on the same axis (I promise I'll try explaining it better with images and all when I’ll start writing the article). This technique will at one point (when we get the green light to buy a new detector) be used for imagery, by extracting the c_ii coefficients and plotting them, but I kept things as simple as possible for now as I’m still brainstorming a lot the ways to use this method effectively in biological samples. I have to underline that with the few devices I’m working with here, I find that in general, each type of spectrometer leads to a slightly different but specific representation of data that falls under the description of the format I made (for example, in time domain, you need an abscissa for the time axis, in TFP you never have raw data, in 1-VIPA you usually bin the signal on the detector, in 2-VIPA you usually have a dimensionality reduction to perform to pass from raw data to PSD…) so I’ll try to send you a file for each technique I use at the lab (plus time-resolved with the data you sent me) as soon as possible so you can cross-check compatibility. I’m currently doing two long acquisitions for two groups we collaborate with so I don’t know how long it’ll take, but I am confident all other techniques will be much easier to implement in Carlo’s structure. Another detail : I found an unexpected limitation in the HDF5_BLS library that doesn’t allow me to add attributes directly to the top group so for now each group has its own set of attributes. I’ve added that on the list of things to do ^^ For the time-domain compatibility, I forgot my parents where coming to Vienna this weekend so I unfortunately couldn’t do it, but it’s close to the top of the list now so it’ll be done soon :) Final point, this time more for Carlo & Sebastian: I realized that for some samples (human tissue for example), we might have to face some legal challenges. Since you’re not only proposing to read the file but also write in it using a web-based application, I’m scared we will at one point have to face the infamous GDPR, get a green light from an Ethics committee or get some kind of legal validation from a lawyer. So before putting anything on the server, do you know people that could either advise us on these safety points or better, overview them at EMBL? This is one of the few computer things I’ve never really studied because I always had the magic “just don’t put it online” joker - and also because I kind of flee legislation stuff whenever I can - but I guess this is not an option anymore so we need to do it properly (plus if we don’t, from what I understand we can get in huge troubles). Tell me what you think :) Best, Pierre Pierre Bouvet, PhD Post-doctoral Fellow Medical University Vienna Department of Anatomy and Cell Biology Wahringer Straße 13, 1090 Wien, Austria On 7/4/25, at 10:39, Sal La Cavera Iii wrote: Good morning guys, Hope you had a nice sunny weekend! Just wanted to follow up from Friday's meeting to see if you had a test dataset we can use as the standard for this software project. I think I heard "smilies" and then also one with cells? Preferably Pierre an .h5 with your preferred structure etc, even if it's a work in project / likely to be modified a little bit in the future. If not I can whip something up, or repurpose something, but would prefer to work with a dataset that is most-representative and most useful for your guys! Cheers, Sal --------------------------------------------------------------- Salvatore La Cavera III Royal Academy of Engineering Research Fellow Nottingham Research Fellow Optics and Photonics Group University of Nottingham Email: salvatore.lacaveraiii(a)nottingham.ac.uk (mailto:salvatore.lacaveraiii@nottingham.ac.uk) ORCID iD: 0000-0003-0210-3102 (https://outlook.office.com/bookwithme/user/6a3f960a8e89429cb6fc693c01d10119…) Book a Coffee and Research chat with me! This message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please contact the sender and delete the email and attachment. Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham. Email communications with the University of Nottingham may be monitored where permitted by law. This message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please contact the sender and delete the email and attachment. Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham. Email communications with the University of Nottingham may be monitored where permitted by law. This message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please contact the sender and delete the email and attachment. Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham. Email communications with the University of Nottingham may be monitored where permitted by law.

5 6

(no subject)
by Kareem Elsayad May 7, 2025

May 7, 2025

1 0

catch-up meeting
by Carlo Bevilacqua May 2, 2025

May 2, 2025

Dear all, I just realized that we set a meeting for today at 3pm. Unfortunately both me and Robert won't have time and we are still working on the GUI, thus there will not be much update on our side. In general we feel it would be better to work separately and reconvene at a later time once we are at a more advanced stage. Sorry for the short notice, hope it is not much trouble for you. Best regards, Carlo

2 1

Software/data format catch up meeting on Fri...
by Kareem Elsayad March 31, 2025

March 31, 2025

Hope we’re all doing well… Just quick reminder we have our next brief update/touch-base meeting scheduled on Fri 4th @15:00 Austria/Germany time (14:00 in UK).. Can use same Zoom link as last time: https://us02web.zoom.us/j/5191046969?pwd=alpzaldoZEd3N2ZEQ0hYZU1RR1dOdz09 Meeting ID: 519 104 6969 Passcode: jY3zH8 See you then hopefully! Kareem

1 0

Software manuscript / BLS microscopy
by Kareem Elsayad March 7, 2025

March 7, 2025

Hi Robert, Carlo, Sal, Pierre, Think it would be good to follow up on software project. Pierre has made some progress here and would be good to try and define tasks a little bit clearer to make progress… There is always the potential issue of having “too many cooks in the kitchen” (that have different recipes for same thing) to move forward efficiently, something that I noticed can get quite confusing/frustrating when writing software together with people. So would be good to clearly assign tasks. I talked to Pierre today and he would be happy to integrate things in framework we have to try tie things together. What would foremost be needed would be ways of treating data, meaning code that takes a raw spectral image and meta-data and converts it into “standard” format (spectral representation) that can then be fitted. Then also “plugins” that serve a specific purpose in the analysis/rendering that can be included in framework. The way I see it (and please comment if you see differently), there are ~4 steps here: Take raw data (in .tif,, .dat, txt, etc. format) and meta data (in .cvs, xlsx, .dat, .txt, etc.) and render a standard spectral presentation. Also take provided instrument response in one of these formats and extract key parameters from this Fit the data with drop-down menu list of functions, that will include different functional dependences and functions corrected for instrument response. Generate/display a visual representation of results (frequency shift(s) and linewidth(s)), that is ideally interactive to some extent (and maybe has some funky features like looking at spectra at different points. These can be spatial maps and/or evolution with some other parameter (time, temperature, angle, etc.). Also be able to display maps of relative peak intensities in case of multiple peak fits, and whatever else useful you can think of. Extract “mechanical” parameters given assigned refractive indices and densities I think the idea of fitting modified functions (e.g. corrected based on instrument response) vs. deconvolving spectra makes more sense (as can account for more complex corrections due to non-optical anomalies in future –ultimately even functional variations in vicinity of e.g. phase transitions). It is also less error prone, as systematically doing decon with non-ideal registration data can really throw you off the cliff, so to speak. My understanding is that we kind of agreed on initial meta-data reporting format. Getting from 1 to 2 will no doubt be most challenging as it is very instrument specific. So instructions will need to be written for different BLS implementations. This is a huge project if we want it to be all inclusive.. so I would suggest to focus on making it work for just a couple of modalities first would be good (e.g. crossed VIPA, time-resolved, anisotropy, and maybe some time or temperature course of one of these). Extensions should then be more easy to navigate. At one point think would be good to involve SBS specific considerations also. Think would be good to discuss a while per email to gather thoughts and opinions (and already start to share codes), and then plan a meeting beginning of March -- how does first week of March look for everyone? I created this mailing list (software(a)biobrillouin.org) we can use for discussion. You should all be able to post to (and it makes it easier if we bring anyone else in along the way). At moment on this mailing list is Robert, Carlo, Sal, Pierre and myself. Let me know if I should add anyone. All the best, Kareem

5 32