Challenge
European research infrastructures operate across heterogeneous computing environments that were not designed to work together. A scientific workflow may require GPU clusters for machine learning training, HPC systems for large-scale simulation, and distributed storage for multi-petabyte datasets. Currently, each resource type often requires separate interfaces, data formats, and operational procedures.
Three interconnected challenges prevent the compute and data continuum from functioning as an integrated technical capability for research:
- Computing integration fragmentation: HTC and Cloud communities have established federation within their domains (WLCG federates 170 sites; EGI provides brokering across European e-Infrastructures). The HPC federation is still developing, with the EuroHPC Federation Platform under development. Each HPC centre uses its own portals, schedulers, and software stacks, requiring experiments to custom-adapt to each site. Integration across the different resource types remains limited and standardised integration policies are needed to enable unified orchestration across HPC, HTC, Cloud, and emerging quantum resources (D6.1, Gaps #3, #5).
- Data management fragmentation: Exabyte-scale workflows require moving data between long-term archives and transient compute resources, yet transfer mechanisms are fragmented and often rely on ad-hoc tools. No unified architectural approach exists for FAIR storage across domains, including standardised metadata and provenance tracking. Between large Tier-0/1 sites (CERN, the main WLCG and SRCNet hubs) the network backbone is generally adequate, with GÉANT and dedicated overlays (LHCONE, LHCOPN) providing high-capacity transport (D6.1 Section 4.2). Reaching beyond those core sites is the binding constraint: D6.1 Section 5.4 identifies network connectivity as a persistent technical constraint and notes that "at minimum, HPC sites need strong connectivity to WLCG centres and SRCNet nodes", with realistic demand validation required at non-core centres.
- Workflow orchestration limitations: Scientific workflows increasingly span multiple facilities, yet orchestration systems remain fragmented across communities. Cross-facility workflows lack resilience and provenance capture. Resource-aware scheduling that considers data locality, compute availability, energy and cost optimisation remains immature.
Priorities Overview
Pillar 2 establishes the technical machinery enabling heterogeneous computing integration, data federation and transport, and workflow orchestration across distributed resources. Three priorities address complementary layers of the technical stack: computing abstraction (P04), data management (P05), and workflow coordination (P06).
P04: Heterogeneous Computing Integration. Provide unified access to heterogeneous computing resources (HPC, HTC, cloud, edge, and emerging quantum) so that researchers can compose workflows across providers without per-stack glue. This requires standardised integration policies adopted across the major European federations, EuroHPC Federation Platform, EGI Federation, WLCG, SRCNet, alongside national pre-exascale and exascale centres. Move beyond reliance on CLI-based batch interfaces toward API-based access mechanisms compatible with workflow engines including the LEXIS Platform (based on Apache Airflow), Common Workflow Language, Dask and HTCondor; introduce shared ontologies (with reference to Numpex) as a politically tractable first-stage requirement. Integration must connect with data infrastructures and data providers, including the European Space Agency (ESA) initiative serving Earth observation and astronomy communities, since heterogeneous compute is rarely useful without proximate data. (Detail: Annex A, P04)
P05: Data Management and Transport. Establish cross-domain standards for federated FAIR data management, automated large-scale data movement with advance-planning and event-driven capabilities, and flexible architectures for sensitive data handling (a transversal need across data-intensive science beyond HEP/RA). For datasets too large to move repeatedly, remote computational access (where supported by the target site) complements bulk transfer rather than substituting for it, since many HPC sites block outbound connections from compute nodes. Convergence on common transfer protocols supporting both POSIX-like and object-storage access patterns, deployment of proven movement services at HPC centres, and operational observability of throughput, latency, error rates, and end-to-end staging time complete the operational picture. Network capacity underlies all of this: transport between core sites is provided by GÉANT, the NRENs and dedicated overlays (e.g., LHCONE, LHCOPN), but assured end-to-end capacity beyond the Tier-0/1 core remains a binding constraint (D6.1 Section 5.4), and a dedicated assessment of exabyte-era network requirements is flagged as a follow-up action (Section 8). (Detail: Annex A, P05)
P06: Workflow Orchestration and Management. Evolve workflow orchestration into a cross-facility capability with built-in resilience, provenance capture, and support for interactive compute use cases, mixed workloads combining embarrassingly parallel tasks with tightly coupled AI processes, and unattended execution of long-running workflows. Build on D6.1's Orchestration and Workflows capability (Workflow Management, Resource Orchestration, Task Scheduling, Provenance) and on the European workflow-management projects catalogued in the Technical Blueprint already advancing this space. Cross-priority dependencies on P01-P03 (governance, allocation, identity) carry the orchestration prerequisites identified in the D6.1 section 4.4 cross-references. (Detail: Annex A, P06)
Interdependencies: The three priorities form a layered structure. Workflow orchestration (P06) depends on heterogeneous computing integration (P04) to execute across diverse resources and on data management (P05) to stage data appropriately. Effective data transport requires understanding of compute and storage resource availability; efficient computing requires data locality awareness. Existing projects demonstrate this integration: interTwin delivers workflow orchestration spanning HPC and HTC resources, whilst Rucio (the WLCG-originated distributed data-management framework, now adopted across multiple experiments) provides data management across the WLCG and SRCNet federation.
Recommendations
For Scientific Communities and Research Infrastructures
- Down-select to a small set of supported interfaces, data-management tools, and workflow engines so that e-Infrastructures can build against a stable target (P04, P05, P06)
- Ensure the capability to execute on diverse compute platforms via code porting and/or the use of high level programming frameworks (P04)
- Contribute to use cases that drive data management and transport requirements (P05)
- Engage in co-design of workflow orchestration capabilities through community working groups and with service providers (P06)
For e-Infrastructures and Service Providers
- Implement standardised interfaces for heterogeneous resource description, discovery, and access (P04)
- Deploy interoperable data management services with FAIR-compliant metadata (P05)
- Support cross-facility workflow execution through common orchestration frameworks (P06)
For Policy Makers and Funding Bodies
- Fund architecture-ecosystem development across European digital infrastructures (EuroHPC, EOSC, EGI) (P01, P04)
- Fund uptake of federated identity and access management for workflows and the compute, data, and network components those workflows interface with (P03, P04)
- Fund development of integration and deployment testbeds enabling validation across HPC/HTC/Cloud resources (P04)
- Support (smart) network capacity investments enabling exabyte-scale data movement (P05)
- Require interoperability standards in infrastructure procurement (P04, P05, P06)