Data Analysis
At the D2AM beamline, multidimensional data sets are usually recorded and treated directly after each measurement, allowing one to drive the experiment on the fly. Python-based short scripts were written by beamline scientists to perform specific treatments, such as the conversion from angular to reciprocal space, azimuthal integrations and correction of distortions induced by the detectors’ geometries, data visualization (3D maps, cuts), and fittings. Software developed at the ESRF and at the beamline by the D2AM scientists are routinely used to perform such treatments: Pymca, PyFAI (Fast Azimuthal Integration using Python) suite, XSOCS, SILXS.
Due to the specificity and complexity of the conducted experiments, these scripts undergo continuous minor or major modifications depending on the experimental configuration and the information that needs to be accessed. The Jupyter Notebooks platform is used to keep track and document the data treatment process. The beamline is, thus, building a modular python-based software that can be adapted to each experiment, creating a data treatment electronic logbook that helps to process the data in future similar experiments. A machine is installed at the beamline, dedicated for data treatment and analysis. Users can hereafter connect to the D2AM Jupyter Notebook fastly in order to perform data treatment fastly, document their scripts or use scritps developped by beamline scientists.
With the variety of experimental configurations and experiments that are being conducted at the beamline, this modular approach appears to be best suited for performing fast and efficient online data analysis and to meet the challenges of big data.