Heterogeneous Computing Integration

Strategic Rationale and Evidence Base

Strategic Importance

The compute continuum (seamless integration of HPC, HTC, Cloud, edge, and emerging quantum resources) is essential for data-intensive science operating at exabyte scales. HL-LHC will generate 1 EB/year requiring processing across heterogeneous resources (D5.1), whilst SKA Observatory distributes 700 PB/year to regional centres requiring diverse computational capabilities (D5.1). Scientific workflows increasingly require combinations of high-throughput batch processing, high-performance simulation, interactive analysis, and accelerated computing that no single resource type can optimally provide.

European research communities face a fragmented landscape where HPC, HTC, and Cloud resources operate under different allocation models, access policies, and technical interfaces. HEP communities have historically optimised for HTC workloads (e.g. WLCG sustained 7.4B HTC CPU hours in 2024), whilst emerging requirements for ML training, simulation, and real-time processing demand HPC and GPU capabilities that require new integration approaches.

This priority describes the importance of an integration of heterogeneous computing (continuum computing) including HPC, HTC, cloud, edge and emerging QC systems on the example of the HEP and Astronomy community but is equally relevant and important for many other communities, e.g. numerical weather prediction and foundation model training as well.

Current State

D6.1 Section 5.1 (Standardization of Interfaces) identifies the core integration challenge: "Today, each supercomputing site uses its own portals, schedulers and software stacks. Therefore, an experiment wishing to run on multiple HPC centres must custom-adapt to each... Without a standardized approach, scaling the number of participating HPC sites remains a labor-intensive challenge."

D6.1 Section 2.3.2.5 (Compute Federation Services) documents the compute federation landscape from D5.3 (D5.3): "It should be mentioned that none of these [federation platforms] support end user access privileges/quotas across all participating systems, and that automatic routing of accesses/compute tasks to the best suited systems is not available."

Current landscape shows partial federation within resource types, but limited cross-type integration:

• HPC federation: EuroHPC Federation Platform currently under development, will integrate EuroHPC supercomputing facilities, AI factories and Quantum Computers) and will provide AARC-BPA-compliant AAI across EuroHPC systems, with interoperable services for cross-site workflows. Integration with HTC federations is not yet addressed and how Gigafactories will be incorporated remains also unclear.
• HTC/Cloud federation: WLCG federates HTC resources for workflow execution via experiment-specific layers, with centres providing access points mediated by batch systems (D6.1). EGI Federation federates HTC and Cloud resources across European e-Infrastructures via common AAI, accounting, and service catalogues. Similarly to WLCG, SRCnet is currently being built, the federated digital infrastructure to process, store and distribute SKA Telescopes scientific data.

The JENA Computing White Paper identifies seven technical gaps between HPC and HTC: edge services, federated access and AAI, workflow management, wide-area networking, data management, software deployment, and programming models. The JENA HPC Working Group specifies six technical areas requiring standardised policies across HPC centres: AAI, data transfers, networking, edge services, container support, and software deployment.

Significant progress has been made through European projects. The interTwin project delivered working HPC federation solutions across Cloud, HTC, and HPC testbeds with 10 pilot Digital Twin applications, now serving as "foundation for EGI Federation's HPC integration strategy". EGI's strategic transition explicitly commits to moving from "compute-focused roots" to "data-and-compute-centric infrastructure", with plans to formalise HPC as a fully-fledged offering. The EXTRACT project has been working on solutions for workflow execution and orchestration.

EuroHPC's Federation Platform addresses unified access to 23+ heterogeneous systems including 8 pre-exascale/exascale HPC systems, 8 quantum computers, and (at the time of writing) 13 AI Factories. However, integration with existing HTC federations (e.g., WLCG with 160 sites across 40+ countries) and emerging Cloud resources remains incomplete.

In the context of BDVA, DataLabs initiative brings together key stakeholders in the digital ecosystems for the implementation of the European Data Union Strategy following the AI Continent Action Plan.

Community Evidence

D6.1 Section 2.5 consolidates requirements from WP5 analysis, identifying key gaps relevant to heterogeneous computing integration:

• Gap #3: Interoperable allocation and accounting. Allocations and quotas are bound to specific HPC centres, limiting flexibility. Accounting practices differ across infrastructures. Recommendation: Enable e-Infrastructure-wide allocations and harmonised accounting, so projects can burst across systems whilst ensuring fair contributions and usage transparency (D6.1).
• Gap #4: High-level and portable software environments. Software stacks differ widely between sites; portability across CPUs, GPUs, and accelerators is inconsistent. Scientists often face brittle interfaces. Recommendation: Provide standardised, efficient software stacks (containers, CVMFS) and high-level domain-specific interfaces, enabling reproducibility and accelerator portability (D6.1).
• Gap #5: Workflow execution and orchestration. Workflow engines are fragmented across communities (e.g., HTCondor in HEP vs LEXIS Platform in EuroHPC). Job granularity and scheduling flexibility represent fundamental differences between HTC and HPC that must be addressed. Cross-facility workflows lack resilience and provenance capture. Recommendation: Co-design and converge on common APIs for workflow systems, with support for unattended execution, provenance tracking, and cross-infrastructure orchestration (D6.1).
• Gap #14: Interactive access to significant-scale computing. Interactive access to small/medium partitions is possible, but problems include delays in availability and potential underuse of resources. Recommendation: Extend scheduling/orchestration to support interactive compute use cases (D6.1).

Technical Foundation

D6.1 Section 4.1 (Compute Resources) provides the architectural framework: "The Compute Resources capability is a foundational layer of the European Compute and Data Continuum, providing seamless access to heterogeneous computational resources distributed across HPC centres, community and commercial clouds, edge infrastructures, and emerging quantum platforms."

D6.1 Section 5.5 (Workflow Adaptation and Optimization) recommends: "Integrate existing application specific workflow systems with HPC resource managers to enable native submission, data staging, and monitoring across federated infrastructures to allow for orchestration of complex workflows across grids and supercomputers" (D6.1).

Multiple approaches exist for addressing these gaps, including meta-schedulers and platform-level orchestration as well as API-based access mechanisms are essential for workflow engine integration.

European Alignment

The ESFRI Landscape Analysis 2024 identifies that "systems of linked RIs are likely to be necessary" for complex scientific challenges, with distributed RIs requiring "central coordination" and European Partnerships expected to "reduce fragmentation of the research and innovation landscape". This provides policy-level justification for heterogeneous computing integration.

EuroHPC targets federation across its investment in pre-exascale and exascale systems, with the Federation Platform (under development) designed to provide unified access whilst preserving national sovereignty. EGI's 2025+ strategic direction commits to the data-compute continuum transition, aligning with EOSC's federated infrastructure model. Both infrastructure providers and user communities have responsibilities in achieving federation goals.

JENA identifies standardised policies across HPC centres for AAI, data transfers, networking, edge services, and container support as prerequisites for effective technical integration. In order to create the compute continuum, policies should focus on integrating existing initiatives rather than forcing individual HPC or HTC centres to change.

In addition it is also critical to introduce a shared language and ontologies in order to enable resource sharing. This is a first-stage requirement that enables many other technical solutions. Ontologies are politically easier to agree on as they do not change existing habits, only require precise definition of needs.

Target Outcomes and Impact Assessment

Strategic Outcomes

Heterogeneous computing integration delivers value across two dimensions: seamless workflow execution and standardised integration infrastructure.

Outcome 1: Standardised Integration Policies Across European HPC, HTC Centres and Cloud Providers

European pre-exascale and exascale centres implement standardised policies for AAI, data transfers, networking, edge services, and container support, enabling plug-and-play integration for data-intensive science workflows. Cross-platform accounting metrics extend established benchmark methodologies (e.g., HEPSCORE) to unified CPU+GPU+quantum measurement, enabling comparable resource tracking across heterogeneous infrastructure. Policies also need to connect with Data Infrastructures and Data Providers

Outcome 2: Seamless Workflow Execution Across Compute Continuum

Researchers submit workflows that automatically decompose across optimal resource types (HTC for throughput, HPC for tightly-coupled simulation, Cloud for burst capacity, GPU for ML training) without manual resource management. The unified submission interface supports automatic resource selection across federated sites, enabling scientific workflows to span organisational boundaries whilst maintaining data locality optimisation and provenance tracking.

Success Indicators

For outcome 1:

• Integration coverage: Sites meeting standardised integration criteria (AAI, containers, data staging, network access).
• Workflow validation: Representative HEP and RA workflows demonstrated on multiple resource types.
• Orchestration adoption: Production workflows using cross-infrastructure orchestration.
• Accounting interoperability: Common metrics (e.g., HEPSCORE-based) reported across HPC/HTC/Cloud sites.

For outcome 2:

• Workflow portability: Percentage of scientific workflows executable across multiple resource types without modification.
• Resource utilisation efficiency: Utilisation rate of heterogeneous resources under unified allocation.
• Time-to-science: Reduction in time from workflow submission to results for multi-resource workflows.
• Integration coverage: Number of sites implementing standardised integration policies.
• Accounting harmonisation: Common accounting framework operational across federated HPC/HTC/Cloud resources.

European Impact

Heterogeneous computing integration maximises return on Europe's distributed infrastructure investments. Rather than requiring researchers to choose between HPC for simulation, HTC for throughput, or Cloud for flexibility, integrated infrastructure enables optimal resource matching for each workflow component. This increases utilisation efficiency, reduces time-to-science, and enables new computational approaches combining traditional physics simulation with ML-accelerated analysis.

Implementation Pathway and Stakeholder Coordination

Implementation Phases

• Short term (1-3 years):
• Agree on a common (descriptive) language - ontologies - over the domain specific ontologies. Once having them, then they can be mapped to HPC systems.
• Reliance on and starting with existing initiatives, with e.g. EuroHPC Federation Platform (EFP) establishing a European federated and secure supercomputing, AI and quantum computing service.
• Start discussion about policies around existing initiatives.
• Analyse the established policies, develop suitable prototypes and start with the implementation.
• Mid term (3-5 years):
• Show first implementations and API-based access across initiatives EuroHPC, EGI and Cloud providers.
• Demonstrate different use cases including HTC, batch execution, AI training, and inference with different requirements.
• Defining next steps based on lessons learned.
• Long term (5+ years):
• Unified federation across different HPC, HTC and cloud providers based on lessons learned from earlier phases.

Stakeholder Roles

• Scientific Communities, Research Infrastructures
• Responsibilities: Community workflows and codes must be adapted for the efficient use of a federated compute continuum infrastructure, especially HPC resources. y
• Benefits: Provide full ability to use the compute continuum.
• Computing & Data Service Providers / e-Infrastructures
• Responsibilities: Align policies and protocols, ensure federation fits funding models.
• Benefits: Enable the compute continuum, offer services to a wide range of users.
• Policy Makers / Funding Bodies
• Responsibilities: Provide political agreement, legal framework, and funding support.
• Benefits: Most efficient use of funding, to help the EU keep being competitive. Realise a federated compute and data infrastructure.

Resource Needs

• A federated European compute continuum involving all relevant services providers requires first support by politics and funding agencies.
• An efficient approach to establish the federated compute continuum would be dedicated funded projects both for service providers and scientific communities.
• Both compute service providers and scientific communities need to agree and engage in the joint projects. To build the legal framework and technical solutions is a basic requirement, however, compute resources (HPC, HTC, Cloud, etc) for scientific communities and research infrastructures are needed as well in order to allow the federated use of compute resources.

Critical Dependencies

First, a common ontology for the different stakeholders (scientific communities, research infrastructures and services providers) needs to be defined. Then, the legal & political framework can be elaborated and defined. Also, a political discussion and decision with a firm funding commitment is needed. The continuum computing priority has of course also cross-dependencies to almost all other priorities in this document.

Cross-priority dependencies with:

• P01: Cross-Infrastructure Governance
• P02: Federated Resource Allocation
• P03: Federated Identity and Trust
• P05: Data Management and Transport
• P06: Workflow Orchestration and Management.
• P08: Software Portability.
• P11: Community Collaboration and Co-Design.