Data Resources
Scientific data in HEP and RA are massive, distributed and federated. The Data Continuum is a federated infrastructure that enables scientific data to be stored, moved, discovered, and preserved across geographically distributed and technologically heterogeneous sites. For data-intensive disciplines such as HEP and RA, where exabyte-scale datasets are generated and shared globally, the Data Continuum is not a peripheral service but a central enabler of scientific discovery. It transforms fragmented storage silos into a seamless, policy-compliant, and performance-optimized ecosystem.
Storage
The continuum relies on a distributed storage capability spanning many sites, which together present a unified namespace to users and applications. Experiments in HEP such as ATLAS and CMS now manage more than an exabyte of data across over 120 storage centers using systems like Rucio. A critical aspect of this capability is data replication and integrity management. Replication rules ensure that each dataset is stored at multiple geographically distinct sites, balancing performance, resilience, and cost. Integrity checks, based on checksums or cryptographic hashes, are performed during transfers and periodically verified in storage to detect corruption or loss. Automated repair mechanisms can re-replicate damaged or missing files from verified copies elsewhere in the federation, maintaining long-term data reliability. Furthermore, access controls and accounting mechanisms are embedded at the storage level to enforce data governance and monitor resource usage across the federation. In practice, HEP and RA communities already share similar approaches: for example, the ESCAPE project demonstrated the applicability of Rucio for managing distributed data from the SKAO, CTA, and Vera C. Rubin Observatory, where Rucio is now deployed operationally. Within the continuum, this federated storage capability presents a coherent, policy-driven view of data—users simply request a dataset, and the system locates and retrieves it seamlessly, regardless of where the bytes physically reside. Alternative solutions for Federated Storage are iRODS which is planned to be included in the EuroHPC Federation middleware and Onedata powering the EGI DataHub.
Data Transfer
For both HEP and RA, data movement at scale is as critical as compute cycles. Moving multi-petabyte datasets between facilities requires high-performance data movement services and a performant underpinning network. Dedicated transfer nodes and optimized protocols (e.g. XRootD, HTTP TPC) are used to shuttle data reliably. The transfer layer orchestrates transfers, schedules bulk copies to avoid saturating links and queues, and automatically retries failed transfers. For example, current LHC experiments regularly transfer tens of petabytes per day between CERN and remote Tier-1 centers, coordinating thousands of files in flight. Together with dedicated research network backbones such as GÉANT, LHCOPN, and LHCONE, the high-performance fiber infrastructure could ensure that raw and processed data flow efficiently across the continuum.
Data Discovery and Cataloging
A critical need for scientists is to find and understand data. The continuum needs to include metadata catalogs and registries to make data (including software) findable. Each dataset should be tagged with rich, standardized metadata (source, date, conditions, format, analysis provenance, processing chain, etc.). Global catalog services keep track of all registered datasets and their physical locations. For example, Rucio provides lookup services and has been used in HEP for years while RA is starting to adopt it. RA also has its own services such as, e.g., the distributed LOFAR long-term archive that provides rich, metadata-based data discovery. More broadly, astronomers make use of the International Virtual Observatory Alliance (IVOA), who have developed standards to make astronomical data and tools interoperable. Provenance tracking is integrated: as data move and transform, metadata records the lineage (which original files were processed by which code version to produce current datasets). This supports the FAIR principles increasingly mandated in research.
Data Lifecycle Management
Scientific datasets have long lifespans and varying usage patterns. The continuum enforces lifecycle policies. For example, raw data might be retained for a fixed period or indefinitely, depending on the need for future reprocessings, intermediate formats deleted sooner, and analysis-ready data preserved indefinitely or refreshed on an annual basis. Automated policies may move data between storage tiers based on access frequency and cost. Quality assurance monitors periodically verify data integrity (checksums, error rates) and rebuild lost fragments if needed. An example in HEP is the ATLAS Data Carousel project, which retrieves data from tape to disk regularly to allow for scheduled analyses, and then removes them until next reprocessing. For archival preservation, the system includes curation metadata such as data format, documentation, and software environment, to ensure future usability. In some HEP contexts, these policies are encoded in Rucio rules that replicate datasets N times and expire older replicas when space is needed. RA projects similarly define retention for different data products (e.g. calibrated images vs. raw voltages). The lifecycle block ensures that valuable data remain accessible and trustworthy over decades, while minimizing costs and respecting any legal or safety requirements.