10.11.2020
ESCAPE DIOS: Development and application of an automatic workflow for the replication of scientific data
« The Major Atmospheric Gamma Imaging Cherenkov (MAGIC) Telescopes (from the Cherenkov Telescope Array – CTA – one of the ESFRIs supported by ESCAPE) are dedicated to the observation of gamma rays from galactic and extragalactic sources in the very high energy range (from 20 GeV to beyond 100 TeV). MAGIC data is replicated to a variety of Tier-1 or Tier-2 facilities, and to smaller Tier-3 or 4 facilities managed by partner institutions. Currently, Port d’Informació Científica (PIC) receives a huge amount of data from the MAGIC experiments, which in turn is distributed in real time to scientific data centers (also called datalake).
Thus, we would like to develop a suitable workflow to handle large data sets produced by the gamma ray telescope, and continuously stream these files to the datalake for permanent storage and access, while keeping free space in the data source center. For this, Rucio is an open-source software framework that provides scientific collaborations with the functionality to organize, manage, and access their data at scale. Also, Rucio can trigger the automatic deletion of files once they have been successfully replicated to its destination. (…) »
source > projectescape.eu, 26 octobre 2020
