BIAFLOWS: A collaborative framework to reproducibly deploy and benchmark bioimage analysis workflows
- Abstract number
- 84
- Presentation Form
- Poster
- DOI
- 10.22443/rms.elmi2021.84
- Corresponding Email
- [email protected]
- Session
- Poster Session 2
- Authors
- Ulysse Rubens (13), Romain Mormont (13), Lassi Paavolainen (4), Volker Bäcker (9), Benjamin Pavie (15), Leandro A. Scholz (12), Gino Michiels (5), Martin Maška (8), Devrim Ünay (3), Graeme Ball (2), Renaud Hoyoux (1), Rémy Vandaele (13), Ofra Golani (7), Stefan G. Stanciu (10), Natasa Sladoje (14), Perrine Paul-Gilloteaux (11), Raphaël Marée (13), Sébastien Tosi (6)
- Affiliations
-
1. Cytomine SCRL FS
2. Dundee Imaging Facility, School of Life Sciences, University of Dundee
3. Faculty of Engineering İzmir, Demokrasi University
4. FIMM, HiLIFE, University of Helsinki
5. HEPL, University of Liège
6. Institute for Research in Biomedicine, IRB Barcelona, Barcelona Institute of Science and Technology, BIST
7. Life Sciences Core Facilities, Weizmann Institute of Science
8. Masaryk University
9. MRI, BioCampus Montpellier
10. Politehnica Bucarest
11. Structure Fédérative de Recherche François Bonamy, Université de Nantes, CNRS, INSERM
12. Universidade Federal do Paraná
13. University of Liège
14. Uppsala University
15. VIB BioImaging Core
- Keywords
Bioimage analysis workflows, public benchmarking, webtool, microscopy
- Abstract text
Image analysis has become a ubiquitous tool for research in biology and efficient algorithms addressing important problems such as automated cell segmentation and nuclei tracking are now often published as the main outputs of scientific articles. As more options are available to the end users, picking the right image analysis method is becoming increasingly daunting. Furthermore, state of the art methods are still sometimes released as source code requiring specific software environments and complex procedures to run, making reproducing results sometimes a project of its own. Finally, comparing methods by simple visual inspection is biased by nature while quantitatively comparing methods across platforms is time consuming as it typically requires to re-implement the same evaluation framework in each environment or to deal with different data formats.
BIAFLOWS (Rubens et al., Cell Patterns, 2020) is an open-source web platform extending Cytomine (Marée et al., Bioinformatics, 2016) that has been developed to overcome all these challenges by defining standard data formats for a broad range of bioimage problems and by storing image datasets, image analysis workflows and associated functional parameters in the same platform. Once integrated to BIAFLOWS, workflows written in any programming language or targeting any existing bioimage analysis software can be run remotely on all the images of a dataset stored in the system, and all the results can be co-visualized through a user-friendly web interface. Additionally, when ground truth annotations are available, a set of problem dependent benchmark metrics assessing the accuracy of the workflows is automatically computed and available as consolidated statistics. All data is stored in unified databases and can be easily explored and shared through the web for effective collaboration and to ensure reproducibility.
A curated instance of BIAFLOWS with a strong focus on multidimensional microscopy bioimages is publicly accessible1 and, as a community effort2, is being populated with reference multidimensional scientific image datasets (2D, 3D, 2D+t, 3D+t,...) and associated image analysis workflows using computer vision or deep learning algorithms (currently over 40 workflows, targeting for instance ImageJ, Icy, Ilastik, CellProfiler, Python, Keras/Tensorflow,...). The datasets and workflows are selected to recapitulate important bioimage analysis problems organized as nine problem classes (object segmentation, pixel classification, object counting, object detection, filament tree tracing, filament networks tracing, landmark detection, particle tracking, object tracking), each with its own set of associated benchmark metrics.
New image analysis workflows can be versioned and added through a framework3 leveraging online managed web services and the source code of all the workflows is publicly available through the system. For this purpose, a sandbox server4 is available to test the integration of new workflows or to test existing workflows on user uploaded images. Finally, besides being a public platform for continuous benchmarking, BIAFLOWS can also be installed as a local solution for image management and analysis into which existing workflows can easily be imported.
- References
Rubens U, Mormont R, [...], Tosi S. BIAFLOWS: A Collaborative Framework to Reproducibly Deploy and Benchmark Bioimage Analysis Workflows, Patterns (Cell Press) 1(3): 100040. 2020.
Marée, R., Rollus, L., Stévens, B., Hoyoux, R., Louppe, G., Vandaele, R., Begon, J.M., Kainz, P., Geurts, P., and Wehenkel, L. (2016). Collaborative analysis of multi-gigapixel imaging data with Cytomine. Bioinformatics 32, 1395–1401.