Feature proposal: mzML reader

[david-bouyssie]

I would propose to implement an mzML reader.
We could create a first implementation based on Rust code like https://github.com/lazear/sage/blob/master/crates/sage-cloudpath/src/mzml.rs or wrap the complete mzdata one https://github.com/mobiusklein/mzdata/blob/main/src/io/mzml/reader.rs

[lazear]

I will say that one of them has been tested on over 100,000 mzML files and is substantially less complex :). mzdata supports many more features of the format and CV stuff though. I think if the lifetimes and phantom types could be removed it would make sense to use (I just don't think these are ergonomic for an API... particularly considering that most contributers/users of rusteomics will probably be novice Rust programmers).

[david-bouyssie]

So we could maybe start by using the Sage implementation and gradually Implement some mzdata features?
Also during the today's meeting we discussed about the benefits of sync vs async APIs.
We were tempted to offer an async API for performance reason, but would prefer to stick to sync for the sake of simplicity.
Any opinion about this @lazear?

[lazear]

I would vote for sync - only reason Sage is using async here is for streaming files from S3 and slightly increased perf. Async adds some substantial complexity and mental overhead

[mobiusklein]

I agree with @lazear that it's probably better to write a synchronous version of the implementation in Sage. mzdata's high level API doesn't actually involve lifetimes on the produced spectra, only for internal data structures involved in building the spectra, and the handling of unsafe views of temporary buffers, but it definitely was over-built for different co-existing signal/peak type channels, and to round-trip as much information as possible. And certainly hasn't seen the amount of testing in the wild that @lazear's implementation has. A lot of the complexity I built out hasn't really been "used" much either, and I don't know whether I'll have trapped myself in a corner with the type system until I do.

Please do steal features though, for example the spectrum index and random accessors are very handy when you only want to read a subset of the spectra, and is independent of the broader monstrosity.

Another pro for synchronous APIs is that not all IO will work with async Rust. Anything not built on Rust-native byte streams will either not be async-compatible, or you'll need to write an abstraction over it to make it look async compatible while doing even more due diligence that what you're doing is "safe". This would apply to any FFI-based bindings from Rust to .NET or plain C. Just to add that to the pile of other issues with async Rust.

[david-bouyssie]

This would apply to any FFI-based bindings from Rust to .NET or plain C.

@mobiusklein I was hacking around that yesterday and I made some progress.
Since this point is actually more connected to another discussion (https://github.com/orgs/rusteomics/discussions/4), I will write it there.

[julianu]

Just as a comment, similar to the one in the mzid discussion:
I would definitely be for taking the ideas of how a spectrum is defined in Rusteomics from mzML for the fact, that it could most probably hold all information we will ever encounter. This approach instead of re-inventing something or just taking some information that one tool needs, will be a little overhead in the beginning, but does not need much adding later on.

[lazear]

How do you plan on handling TIMS data if you treat everything as mzML spectra? Bruker has been very kind to release a native Rust package for reading their data formats natively... https://github.com/MannLabs/timsrust

[julianu]

I just would take it as a basis, that seems to be able to handle most things and be able to address the base parameters of any spectrum in the same way.
To be honest, I never really worked with ion mobility data, but it seems as if mzML is also able to handle it, as stated in Status here https://www.psidev.info/mzML.

[david-bouyssie]

Unfortunately, I fear @lazear is right as TIMS data have to be treated in a specific way. And this direct and specific treatment enables possibilities of optimizations. Even if some other vendors support ion mobility, TIMS data are kind of unique in many aspects. So I may not try to stick to a common representation, even if it could be provided for reference.