You don't often get email from software@biobrillouin.org. Learn why this is important

Hi Kareem,

thanks for restarting this and sorry for my silence, I just came back from US and was planning to start working on this again.

Could you also add Sebastian (in CC) to the mailing list?

As you outlined, I would split the project into two parts: 1) getting from the raw data to some standard "processed' spectra and 2) do data analysis/visualization on that. For the second part the way I envision it is:

the most updated definition of the file format from Pierre is this one, correct? In addition to this document I think it would be good to have a more structured description of the file (like this), where each field is clearly defined in terms of data type and dimensions. Sebastian was also suggesting that the document should contain the reasoning behind each specific choice in the specs and also things that we considered but decided had some issues (so in future we can look back at it). I still believe that the file format should contain the "processed" spectra (i.e. after FT and baseline subtraction for impulsive, or after taking a line profile and possibly linearization with VIPA,...) so we can apply standard data processing or visualization which is independent on the actual underlaying technique (e.g. VIPA, FP, stimulated, time domain, ...)
agree on an API to read the data from our file format (most likely a Python class). For that we should: 1) decide on which information is important to extract (spectral information, spatial coordinates, hyper-parameters, metadata,...) 2) implement an interface to read the data (e.g. readSpectrumAtIndex(...), readImage(...), ...)
build a GUI that use the previously defined API to show and process the data.

I would say that, once we have a very solid description of the file format as in step 1, step 2 will come naturally and we can divide the actual implementation between us. Step 3 can also easily be implemented and divided between us once we have an API (and I am happy to work on the GUI myself).

The first half of the project is the trickiest and I know Pierre has already done a lot of work in that direction. We should definitely agree on which extent we can define a standard to store the raw data, given the variability between labs, (and probably we should do it for common techniques like FP or VIPA) and how to implement the treatments, leaving to possibility to load some custom code in the pipeline to do the conversion.

Let me know what you all think about this.

If you agree I would start by making a document which clearly defines the file format in a structured way (as in my step 1 before). @Pierre could you write a new document or modify the document I originally made to reflect the latest structure you implemented for the file format? The metadata can still be defined in a separated excel sheet, as long as the data type and format is well defined there.

Best regards,

Carlo

On Wed, Feb 12, 2025 at 02:19, Kareem Elsayad via Software <software@biobrillouin.org> wrote:

Hi Robert, Carlo, Sal, Pierre,

Think it would be good to follow up on software project. Pierre has made some progress here and would be good to try and define tasks a little bit clearer to make progress…

There is always the potential issue of having “too many cooks in the kitchen” (that have different recipes for same thing) to move forward efficiently, something that I noticed can get quite confusing/frustrating when writing software together with people. So would be good to clearly assign tasks. I talked to Pierre today and he would be happy to integrate things in framework we have to try tie things together. What would foremost be needed would be ways of treating data, meaning code that takes a raw spectral image and meta-data and converts it into “standard” format (spectral representation) that can then be fitted. Then also “plugins” that serve a specific purpose in the analysis/rendering that can be included in framework.

The way I see it (and please comment if you see differently), there are ~4 steps here:

Take raw data (in .tif,, .dat, txt, etc. format) and meta data (in .cvs, xlsx, .dat, .txt, etc.) and render a standard spectral presentation. Also take provided instrument response in one of these formats and extract key parameters from this

Fit the data with drop-down menu list of functions, that will include different functional dependences and functions corrected for instrument response.

Generate/display a visual representation of results (frequency shift(s) and linewidth(s)), that is ideally interactive to some extent (and maybe has some funky features like looking at spectra at different points. These can be spatial maps and/or evolution with some other parameter (time, temperature, angle, etc.). Also be able to display maps of relative peak intensities in case of multiple peak fits, and whatever else useful you can think of.

Extract “mechanical” parameters given assigned refractive indices and densities

I think the idea of fitting modified functions (e.g. corrected based on instrument response) vs. deconvolving spectra makes more sense (as can account for more complex corrections due to non-optical anomalies in future –ultimately even functional variations in vicinity of e.g. phase transitions). It is also less error prone, as systematically doing decon with non-ideal registration data can really throw you off the cliff, so to speak.

My understanding is that we kind of agreed on initial meta-data reporting format. Getting from 1 to 2 will no doubt be most challenging as it is very instrument specific. So instructions will need to be written for different BLS implementations.

This is a huge project if we want it to be all inclusive.. so I would suggest to focus on making it work for just a couple of modalities first would be good (e.g. crossed VIPA, time-resolved, anisotropy, and maybe some time or temperature course of one of these). Extensions should then be more easy to navigate. At one point think would be good to involve SBS specific considerations also.

Think would be good to discuss a while per email to gather thoughts and opinions (and already start to share codes), and then plan a meeting beginning of March -- how does first week of March look for everyone?

I created this mailing list (software@biobrillouin.org) we can use for discussion. You should all be able to post to (and it makes it easier if we bring anyone else in along the way).

At moment on this mailing list is Robert, Carlo, Sal, Pierre and myself. Let me know if I should add anyone.

All the best,

Kareem