Data Management and Transport

Strategic Rationale and Evidence Base

Strategic Importance

SPECTRUM use cases are based on the acquisition and processing of very large amounts of data. The structure of the data (their architecture) and their management are fundamental to European e-Infrastructure competitiveness. The capability to store data and efficiently transport them to HPC centres is essential for HEP and RA communities, whilst native management according to FAIR principles enables cross-domain research and Open Science.

Current State

The management of large amounts of data around HTC computing centres is reasonably addressed by existing tools within each domain. Pan-European research networks (GÉANT) and dedicated overlays (e.g. LHCONE, LHCOPN) provide high-capacity transport, though sustained investment is needed for exabyte-scale data movement. Gaps exist primarily between different types of infrastructures (e.g., data infrastructures and HPC facilities) and across different domains (e.g., standardisation of FAIR storage) (D6.1).

Community Evidence

Three main gaps have been identified in the data domain (D6.1):

• Gap #6 (Data federation and FAIR principles): No unified architectural approach or core implementation for FAIR storage across domains, including standardised metadata and provenance tracking
• Gap #7 (Data transfer and staging): Transfer of large datasets between long-term archives and transient compute resources is highly fragmented and often relies on ad-hoc tools. Remote access in some HPC is not possible as outside connections from nodes is blocked for security reasons.
• Gap #9 (Sensitive data handling): Complexity of integrating GDPR-sensitive and special category data threatens to impact usability and performance for open scientific data use cases

Technical Foundation

D6.1 4.2 (Data Resources) defines four capabilities comprising the Data Continuum (D6.1):

• Federated Storage: Distributed storage presenting unified namespace to users and applications, with data replication, integrity management, and embedded access controls
• Data Transfer: High-performance data movement services (e.g., FTS, Rucio, XRootD) orchestrating bulk transfers across research network backbones (e.g. GÉANT, LHCONE)
• Data Discovery and Cataloguing: Metadata catalogues and registries enabling data findability, with provenance tracking supporting FAIR principles
• Data Lifecycle Management: Automated policies for retention, tiered storage migration, integrity verification, and long-term preservation

These capabilities transform fragmented storage silos into a seamless, policy-compliant ecosystem for exabyte-scale scientific data.

European Alignment

ESFRI identifies data sharing across Research Infrastructures as essential for complex scientific challenges. EOSC provides the policy framework through its federated infrastructure model, whilst EuroHPC requires efficient data access to exploit European compute capacity for data-intensive workloads.

Target Outcomes and Impact Assessment

Strategic Outcomes

Outcome 1: Federated FAIR Data Management

Cross-domain standards for federated FAIR data management enable Research Infrastructures to share data whilst maintaining custodian control. This includes common metadata catalogues, provenance tracking, and access interfaces built on progressive maturity frameworks. Flexible architectures decouple sensitive and non-sensitive data repositories, enabling GDPR compliance without impacting performance for open scientific use cases.

Outcome 2: Automated Data Movement at Scale

Large-scale data movement infrastructure operates with advance planning capabilities across HTC facilities, HPC centres, and cloud providers. High-performance transfer services (e.g., FTS, Rucio) integrate with HPC storage systems, enabling autonomous data staging for workflows spanning federated European infrastructure. Multi-protocol support ensures compatibility with diverse storage backends.

Success Indicators

• FAIR adoption: Research Infrastructures implementing common FAIR data management frameworks
• Transfer automation: Production workflows with fully automated data pre-staging across infrastructure types
• Integration breadth: HPC centres, cloud providers, and HTC sites integrated via common data movement protocols
• Data accessibility: Representative datasets available for analysis within target timeframes
• Interoperability validation: Cross-domain workflows demonstrated using federated data services

European Impact

Federated FAIR data management strengthens European data sovereignty by enabling Research Infrastructures to share data under common governance whilst retaining custodian control. Automated data movement maximises return on EuroHPC investment by enabling data-intensive workloads to exploit European compute capacity. Shared data infrastructure reduces duplication across Research Infrastructures, improving cost efficiency and sustainability.

Implementation Pathway and Stakeholder Coordination

Implementation Phases

Short term (1-3 years):

• Perform a gap analysis to select pilot use cases related to FAIR storage, high‑performance data movement and access, and sensitive data handling, with sufficient coverage for different topics.
• Identify critical interfaces between:
• Research Infrastructures (RI) generating or stewarding large datasets
• e-Infrastructures providing compute, network and storage services
• Community and cross-domain data-access interfaces / APIs (including metadata)
• Identify and evaluate viable architectures for federated FAIR storage, metadata cataloguing, and provenance tracking.
• Pilot one or more representative use cases, such as for example:
• FAIR-compliant cross‑RI data federation
• Automated data staging from archives into HPC environments
• Handling of mixed sensitive/non-sensitive datasets

Mid term (3-5 years):

• Prototype and demonstrate integrated FAIR data management across multiple RIs and e-Infrastructure providers.
• Validate that architectures are fit for purpose for exabyte-scale data volumes.
• Expand scope of supported use cases:
• High-frequency data ingestion (e.g., radio astronomy streaming or GW low-latency data)
• Multi-protocol access to federated storage
• GDPR-sensitive workflows using secure enclaves or logically separated data zones
• Develop standards and practices governing:
• FAIR metadata schemas
• Access control & identity integration
• Operational interfaces for cross-domain data movement
• Compliance requirements for RIs and e-Infrastructure providers
• Draft a Blueprint Architecture for the European Data Management & Transfer ecosystem.

Long term (5 years):

• Roll-out federated data management and automated transport capabilities across European e-Infrastructures.
• Onboard more communities adopting federated FAIR data services, including new scientific domains.
• Ensure sustainability and continual evolution through collective governance, coordinated investment, and community-driven standards.

Stakeholder Roles

• Scientific Communities, Research Infrastructures

Responsibilities:

• Co-design federated FAIR data architectures, including interoperable metadata standards and provenance models.
• Provide domain-specific requirements for data retention, curation, sensitivity, and access.
• Adopt harmonised interfaces and participate in pilot and validation campaigns.

Benefits:

• Seamless access to distributed data holdings with uniform FAIR-compliant interfaces.
• Increased scientific productivity due to reduced time-to-access and improved dataset reuse.
• Greater visibility, interoperability, and cross-domain collaboration.
• Computing & Data Service Providers / e-Infrastructures

Responsibilities:

• Deploy scalable federated storage systems with unified namespaces and policy-controlled access.
• Provide high-performance, automated data-transfer capabilities integrated with workflows.
• Adopt high-level data-access APIs
• Provide secure environments for sensitive and special-category data, aligned with GDPR and ethical frameworks.
• Monitoring services

Benefits:

• Maximised utilisation of storage, network, and compute resources through coordinated data movement.
• Reduced operational overhead thanks to automation and standards-based interfaces.
• Broader user base as more communities adopt shared data services.
• Policy Makers / Funding Bodies

Responsibilities:

• Ensure continued investment in high-capacity pan-European research networks (e.g., GÉANT).
• Support development of FAIR-aligned infrastructure and long-term preservation facilities.
• Coordinate regulatory and governance frameworks for cross-border data management (GDPR, FAIR metrics, EOSC policies).

Benefits:

• Strengthened European data sovereignty and reduced duplication across scientific domains.
• More efficient use of public infrastructure investments.

Resource Needs

• Compute, network, and storage resources scaling across phases from pilot to exabyte-level deployment.
• Persistent high-speed connectivity through GÉANT and domain overlays (LHCOPN, LHCONE, or equivalents).
• Monitoring services
• Long-term archival and nearline storage; data orchestration tools including intelligent tiering systems; metadata catalogues.
• Engineering resources within RIs and e-Infrastructures to develop, port, test, and integrate FAIR and data-transfer capabilities.
• Governance structures for standards, certification, and service evolution.

Critical Dependencies

• Commitment from infrastructure providers to expose harmonised and interoperable data interfaces.
• Adoption of common metadata standards, provenance models, and access protocols across domains.

Cross-priority dependencies with:

• P01 (Cross-Infrastructure Governance)
• P02 (Federated Resource Allocation), P03 (Access, Security & Trust)
• P04 (Heterogeneous Computing Integration)