Performance benchmarking of microbubble-localization algorithms for ultrasound localization microscopy

We have compiled algorithms and datasets into an open platform for ultrasound localization microscopy and introduce quantitative and qualitative metrics to benchmark localization strategies.
Performance benchmarking of microbubble-localization algorithms for ultrasound localization microscopy
Like

A decade ago, following in the footsteps of revolutionary techniques in optics, ultrasound super-resolution was demonstrated. Conventional ultrasound is limited in penetration by absorption and in resolution by diffraction, both phenomena governed by the wavelength of the acoustic wave transmitted. The revolutionary idea was to use intravascular sub-wavelength contrast agents to produce super-wavelength bright echoes in the vascular system and to image them while they are carried around by flow. When a low concentration of contrast agents, these bright echoes can be isolated and the true position can be localized with sub-wavelength accuracy. Recording and accumulating millions of these events thanks to ultrafast ultrasound led to the rupture of the penetration-diffraction limit and allowed ultrasound to peer deep in the microvasculature (figure 1.a).

a)      Ultrasound Localization Microscopy reveals microvasculaure deep in organs in vivo with a  resolution. Compared to microangio-CT which takes place post-mortem, and Power Doppler which is limited in resolution by the penetration-diffraction trade-off it opens up exciting avenues in pre-clinical and clinical imaging

b)      In this article, a collection of datasets is provided with accompanying algorithms and simulations to benchmark, quantify errors and introduces metrics to equip the researchers with the tools needed to assess their own implementations.

In our team, we have always been eager to apply the development of ultrasound localization microscopy (ULM) in vivo whether in the brain or in other organs with fascinating vascular perspective like kidneys, spine, or liver. As such, we are familiar with widely different experimental conditions and frequently run into the necessities of adapting our algorithms to different signal to noise ratios, to conditions imposed by a new experimental paradigm, or to new imaging modes such as 4D ultrafast ultrasound. The key part of the algorithm in ULM processing is the localization part and we saw that each experimental condition would benefit from slightly different algorithms. For example, our recent endeavors into 3D ULM showed that the algorithms we were accustomed to were simply too complex and too computationally expensive to be of use in that setting. The lower signal to noise ratio associated with transcranial ultrasound also induced a shift for us to go towards algorithm that were more robust to noise.

This project was born when we started broadcasting our results to the biology and medical communities who were excited with the idea of deep microvascular imaging of organs but were put off by the complexity of the different algorithms and the extensive work needed to master each step of the whole pipeline. At the same time, we saw a lot of new teams investigating the new possibilities offered by ULM, coupling it with deep learning methods, adding new localization algorithms of their own making, investigating new ultrasound pulse sequences. However, the lack of metrics and of common datasets rendered comparison of results complicated, particularly for resolution measurements. We thought that it would be a valuable resource if one of the teams could compile the most common algorithms used for localization and come up with metrics to benchmark them together. This would allow an extensive review of the choices at hand for the less-technically inclined willing to endeavor in ULM. In the same process, collecting datasets from different organs and experimental paradigms would allow researchers in from other fields to apply new processing techniques such as deep learning approaches. Considering the amount of datasets we had in our possession and our familiarity with the different implementation of ULM, in the end we decided to rise to the challenge and came up with this open platform for ultrasound localization microscopy with processing routine, datasets, and benchmarking tools.

In this view, we wished to propose a complete and detailed method that can be implemented in all laboratories equipped with ultrasound scanners or directly on the provided data (figure 1.b). We provide five of the most used ultrasound localization algorithms and two new ones that we implemented in our quest towards 3D ULM. The scripts for post-processing of ULM data are available both as a Github or as a toolbox with a graphical interface that we have named LOTUS (LOcalization and tracking Toolbox for Ultrasound Superresolution). The toolbox will help to kickstart researchers and projects with an accessible interface to ULM. In supplementary materials we wrote down the detailed protocols for ULM processing, acquisition and additional results on complementary datasets with enlarged figures to really dive deep into the microvasculature. We also deposited large in silico and in vivo datasets on a public online repository for other scientists to learn, explore and apply ULM on their setups. The framework can also be implemented on human data using the same codes provided in supplementary materials. Additionally, we have developed 11 metrics to compare and benchmark any localization algorithms of the community. Indeed, each step of our methodology can be improved upon and, with the metrics detailed in this article and its datasets, any scientist can assess the progress of his new algorithms and share it with everyone.

We envision that these datasets and algorithms could become a nucleus for a broader endeavor toward an open field in super-resolution ultrasound imaging. The collection of tracks of microbubbles flowing within the microvasculature in different organs of different species are a new form of imaging data that could be shared easily. They can be exploited as a tool for the radiologist or the biologist, as a field of study for the biophysicist or as a geometrical problem for the mathematician. These different perspectives of this kind of data could lead to better diagnosis and further understanding of the micro-anatomy.