Federated Resource Allocation
Europe invests substantially in research computing and data infrastructure through EuroHPC, national programmes, and federated e-Infrastructures. Yet allocation mechanisms fragment this investment. Scientific computing is increasingly data-intensive, with exploitation of exabyte-scale datasets relying on a diverse and heterogeneous compute continuum spanning HPC, HTC, cloud, and emerging accelerator systems. Research infrastructures increasingly operate on multi-decade timelines, yet face annual application cycles, system-specific allocations that prevent workload-appropriate resource selection, and accounting frameworks that cannot track usage across governance boundaries. The result is less utilised infrastructures, repetitive submission burden, extra overhead e.g. in proposal writing, data collection, steering, and planning uncertainty that undermines long-term scientific programmes.
Strategic Rationale and Evidence Base
Strategic Importance
Europe invests substantially in research computing and data infrastructure through EuroHPC, national programmes, and federated e-Infrastructures. Yet allocation mechanisms fragment this investment. Scientific computing is increasingly data-intensive, with exploitation of exabyte-scale datasets relying on a diverse and heterogeneous compute continuum spanning HPC, HTC, cloud, and emerging accelerator systems. Research infrastructures increasingly operate on multi-decade timelines, yet face annual application cycles, system-specific allocations that prevent workload-appropriate resource selection, and accounting frameworks that cannot track usage across governance boundaries. The result is less utilised infrastructures, repetitive submission burden, extra overhead e.g. in proposal writing, data collection, steering, and planning uncertainty that undermines long-term scientific programmes.
Federated resource allocation addresses how compute and data resources are allocated, accounted, and funded across the continuum:
- Compute resources: CPU cycles, GPU hours, emerging quantum computing allocations across HPC, HTC, and cloud systems, and associated requirements in terms of I/O and memory
- Data resources: Storage capacity (HL-LHC requires 4-6 EB tape archival by mid-2030s),
- Network: bandwidth for data transfer (SKA and WLCG distributes several hundreds of petabytes per year to regional centres)
- Data lifecycle management, including preservation and curation costs.
Balanced and flexible allocation across Europe is essential: governance coordination and federated identity enable access, but without allocation mechanisms that match scientific planning horizons and support diverse user communities, researchers cannot reliably integrate heterogeneous resources into sustainable workflows.
Current Status
Science is outcome driven, whereas allocations are project driven; consequently, predictability in resource allocation is difficult. Whilst a substantial part of the scientific community has (WLCG) or plans for multi-annual, MoU-based access allocations, a significant part of European e-Infrastructures generally provide grant-based access with short-term allocation cycles (months to one year)[^1]. This creates several problems:
- Repetitive submission burden: Researchers submit repetitive allocation requests, creating administrative overhead for both users and review panels
- Planning uncertainty: Multi-decade experiments cannot reliably plan workflows when allocation is uncertain beyond 12 months
- Fragmentation of resource allocation and accounting: The allocations are generally valid for specific hosts/sites instead of the whole e-Infrastructure; this limits the overall resource utilisation
- Utilisation inefficiency: Short-term allocations discourage investment in workflow optimisation for specific systems
Community Evidence
The analysis of the interoperable access policies (D5.2) and the technical blueprint for compute and data continuum (D6.1) prepared by the SPECTRUM project identifies two gaps:.
#1: Long-term assured access and planning of resource allocations. Long-term (3-5 years) evolution and planning are not substantially informed by the needs of research communities that have been involved in the SPECTRUM project.
#3: Resource allocation and accounting that are valid across the whole e-Infrastructure. In many cases (such as EuroHPC JU), allocations are good for specific hosting sites/systems only, while users have voiced the necessity to use resources across an e-Infrastructure to profit from burst capacity, maximise the efficiency of systems for parts of their workload, and increase resiliency.
Technical Foundation
D6.1 Section 5.8 provides the technical framework: "Multi-year or renewable annual guaranteed allocations would enable experiments to fully leverage such investments and integrate HPC resources more effectively into their computing models."
D6.1 4.6 (Resource Federation) specifies requirements for federated access and allocation: "Usage data, such as CPU/GPU hours, storage consumption, and data transfer volumes, must be collected and reported in a consistent way across all sites. This not only supports fair allocation and equitable participation but also enables funding bodies and infrastructure providers to verify contributions and assess impact."
Key technical requirements:
- Standardised and transparent accounting framework for heterogeneous resources (CPU, GPU, storage, network) utilisation across governance boundaries
- Cross-infrastructure resource discovery and allocation
European Alignment
Resource allocation management is critical to broader European strategies. For EuroHPC, multi-year allocations for storage and compute would enable better integration of HPC resources into long-term scientific planning, with efficient resource allocation mechanisms minimising the risk of underutilising the resources. A federated resource allocation mechanism benefits interoperability, harmonisation, integration and synergies among the ESFRI research infrastructures, as well as within the EOSC web of fair data and services.
Target Outcomes and Impact Assessment
Strategic Outcomes
- Adopt long-term, flexible resource allocation processes: processes are implemented that allow scientific communities and research infrastructures to obtain long-term access privileges and quotas in a flexible and scalable manner, federated across infrastructures, with transparent accounting and monitoring. [D5.2, R2]
- Enable e-Infrastructure-wide use of resource allocations and quotas: Enable e-Infrastructure-wide allocations and harmonized accounting, so projects and users can burst across systems and providers while ensuring fair contributions and usage transparency. [D5.2, R1]
- Align ESFRI roadmaps with e-Infrastructure roadmaps
- Align e-Infrastructures with the European Data Strategy: interconnect sectoral data spaces by deploying robust e-Infrastructures and advanced interoperability standards
Success Indicators
- Metrics and checkpoints are established for multi-year allocations
- Standardised accounting framework deployed across major European facilities
- Multi-year allocation agreements operational for data-intensive research infrastructures
European Impact
The European strategy for data aims at creating a single market for data that will ensure Europe’s global competitiveness and data sovereignty. This will lead to the creation of Common European Data Spaces. The European leadership in AI requires cooperation and a seamless, integrated ecosystem combining AI Factories, Data Spaces and HPC resources. In this context, Federated resource allocation and standardised resource accounting systems enables research communities to access compute and data infrastructures across organisational boundaries, increasing utilisation of European investments and reducing reliance on non-European providers. Standardised accounting enables to measure the impact on science of EU investments on computing, allowing future better alignment between investments and needs. This strengthens European digital sovereignty by making European infrastructure easier to use than commercial alternatives.
Implementation Pathway and Stakeholder Coordination
Implementation Phases
Short term (Years 1-3) :
- Continue the conversations between stakeholders, in order to close the identified gaps ( Long-term assured access and planning of resource allocations; Resource allocation and accounting that are valid across the whole e-Infrastructure)
- get stakeholders involved to establish an interoperability framework for resource accounting; set up and agree on relevant metrics and checkpoints; identify tasks and responsibilities for the implementation of the federated accounting system; deploy first testbeds and feasibility studies
- Identify appropriate EC calls to get funding for the design and prototyping phases of the federated accounting system (e.g. ENSURE project in INFRA-TECH-01)
- Identify roles of research communities/RIs, resource providers/e-Infrastructures, funding bodies/policy makers in establishing memoranda of understanding
Mid term (Years 3-5):
- Pilot studies of e-Infrastructure-wide use of resource allocations and quotas
- Testbeds for the accounting system are deployed and progressively integrated into resource providers, infrastructures, research communities.
- Memoranda of Understanding are put into place
Long term (Years 5+):
- A federated, flexible resource allocation process is widely adopted
- The standardised resource accounting system is routinely used in production
Stakeholder Roles
- Scientific Communities, Research Infrastructures
- Responsibilities: establish clear computing resource needs over a timescale of several years, in terms of workflows to execute, storage needed, data access patterns, local- and wide-area network requirements; consume the information aggregated by the accounting system
- Benefits: researchers obtain long-term access privileges and quotas in a flexible and scalable manner
- Computing & Data Service Providers / e-Infrastructures
- Responsibilities: Enable e-Infrastructure-wide use of resource allocations and quotas; establish MoUs with funding bodies and scientific communities/RIs; agree with Scientific Communities and Research Infrastructures on federated accounting metrics and checkpoints, implement the associated sensors and aggregate information
- Benefits: underutilisation of resources is minimised
- Policy Makers / Funding Bodies
- Responsibilities: establish clear and multi-annual funding plans with the goal of matching eInfrastructure technological evolution with scientific needs; establish MoUs with computing & data service providers and scientific communities/RIs
- Benefits: agencies contributing funding to e-Infrastructures can gauge their return on investment by a more efficient use of the invested funding tailored to scientific community needs
Resource Needs
- Funding is needed for
- the technical aspects of the accounting system and the multi-annual, federated resource allocation mechanism
- developing and finalising the policy aspects of federated resource allocation (a pool of people from all stakeholders)
- Funding timeline:
- Short-term, project-related funds for designing/prototyping/testbeds/pilots
- medium-long term structural funds for commissioning and operations
Critical Dependencies
Cross-priority dependencies with
- P01-Cross-Infrastructure Governance,
- P03- Federated Identity and Trust,
- P09-Open Science and Reproducibility