Software Portability

Strategic Rationale and Evidence Base

Strategic Importance

SPECTRUM envisions a European Compute and Data Continuum, which will enable research infrastructures and researchers to access and use the most appropriate data centre for any given task in hand. For example, the choice of centre may depend on data locality, connectivity, availability of specialist hardware, or simply on the distribution of load across the continuum. Users will benefit from seamless access to the most effective and efficient e-Infrastructures to address their needs.

Taking advantage of these capabilities requires that the software tools and workflows required by the user are available on the appropriate e-Infrastructure. This requires that they be portable across centres: tools developed in one centre should run without modification in other centres, and, where possible, should automatically take advantage of the special capabilities each centre offers, such as accelerator hardware.

In this way, the SPECTRUM Compute and Data Continuum will maximise the efficiency of the use of compute resources while simultaneously eliminating the overhead imposed on researchers and research infrastructures by a need to modify software depending on locality and improving reproducibility and sustainability of the software by decoupling it from specific execution environments.

Current State

We identify two essential aspects of the software portability ecosystem: whether the code itself is engineered in such a way as to make it easily portable, and whether there exist software delivery mechanisms that make it possible to deploy, manage and execute the software on multiple platforms. We discuss these aspects independently.

With respect to the first aspect, software engineering, we note that most contemporary HEP and RA software packages are based on large and highly-specialised codebases that have been developed over many years for x86_64 CPU architecture as a primary target. Software is typically optimized at compile-time to the specific characteristics of the system on which it will be executed, and is dependent on specific features of the execution environment, such as system libraries or configuration files, and portability has not been prioritised. In many cases, work is now underway to adapt existing code to accelerator hardware, notably GPUs, to take advantage of their superior energy efficiency and computational throughput. While this work is a step towards portability, it often targets specific hardware environments that are not generically available.

Similarly, although individual compute and data centres generally provide ways, such as modules or containers, for users to configure their execution environment and choose from available software, there is little consistency across the ecosystem. The variety of different approaches and technologies easily leads to brittle environments, poor software portability, and concomitant confusion.

However, we note that this situation is beginning to change. The scientific community is maturing in its approach to software development. In particular, we note the success of projects such as EVERSE that are promulgating best practices in research software engineering across the European scientific landscape, while the FAIR Principles for Research Software (FAIR4RS) provide a roadmap for continued improvement.

At the same time, a range of tools that can help are becoming available. These range from low-level abstractions over HPC hardware (e.g. Kokkos) through containerized systems that encapsulate software together with its dependencies (e.g. Apptainer) to systems like CernVM-FS and EESSI that can distribute and manage complex software environments.

We therefore identify the coming years as a critical period: the principles and tools that can address the current challenges are available, but coordinated, focused effort across the scientific and technical community is necessary to fully take advantage of them.

Community Evidence

The SPECTRUM Technical Blueprint specifically identifies portability as a key requirement (D6.1 Section 2.4 recommendation 4). This requirement is derived from D5.3 Requirements 5 (“Enable portability of compute tasks across accelerated compute platforms”) and 7 (“Establish common SW stacks for compute applications”).

Technical Foundation

This priority is captured in the following SPECTRUM Compute and Data Continuum capabilities (D6.1 Section 4):

Software & Execution;
Compute Continuum.

European Alignment

Software portability is critical to the effective use of HPC/HTC resources at European scale. For example, EESSI is supported through the EuroHPC Centre-of-Excellence MultiXscale, and is used by the European Federation Platform as the basis for its Federated Software Catalogue. Meanwhile, EOSC specifically supports the deployment of containerized applications through its Cloud Container Platform.

The issues surrounding software engineering quality and its impact on portability are also widely acknowledged through major European projects, including, for example, EVERSE and EXTRACT, while the ESCAPE Project’s Open-source Scientific software and Service Repository demonstrated the potential of centralized software distribution systems.

Target Outcomes and Impact Assessment

Strategic Outcomes

Flexible transfer of software and workflows between data and compute centres with minimal adaptation.
Minimal implementation effort for software to take advantage of a wide range of compute accelerators.
Increases overall efficiency of the Compute and Data Continuum and reduces implementation overhead for software authors.

Success Indicators

Number of standardized software repositories.
Existence of commonly agreed and applied standards enabling portability of software across multiple data centres.
Number of codes published making use of hardware abstraction layers like Kokkos.
Number of software packages made available through tools like CernVM-FS and EESSI.
Number of data centres on which commonly used HEP and RA processing software can be executed without modification.
Number of software packages following FAIR4RS guidelines.
Number of ESFRIs and other research infrastructures including software portability in their coding standards or developer guides.

European Impact

Increased homogeneity and portability over different compute systems and architectures increases the overall efficiency of the scientific process, at every level from code implementation to workflow execution. The development of common standards here will spread to other domains, leading to increased reuse and efficiency across the European scientific ecosystem

Implementation Pathway and Stakeholder Roles

Implementation Phases

Short term

Gap analysis of existing tools and systems supporting portability.
Establish and document common software stacks for major research infrastructures.
Increase adoption of well established technologies such as containerization for software distribution.

Medium term

Based on the gap analysis, ensure that adequate tooling is available to support appropriate abstractions over hardware at all levels.
Document commonly-accepted standards for portability over the SPECTRUM Compute and Data Continuum.
Publish software stacks through common software repositories.
Provide training to researchers and software engineers on adopting the new technologies.
Establish a governance model in support of sustainable long-term support for portable software practices by research infrastructures and computing & data service providers.

Long term

Require and expect that all appropriate software is published through the established standards to the relevant repositories.
Ensure a process is in place for modernizing and updating standards as well as the supporting services and capabilities with time.
Provide long-term, sustained support for development and maintenance of key enabling technologies.

Stakeholder Roles

Scientific Communities

Responsibilities: Define scientific requirements and use cases. Implement software tools following best practices for portability and reuse. Distribute software through standardized repositories.
Benefits: Reduced overhead in software implementation across multiple e-Infrastructures. Seamless access to compute and accelerator capacity through the Compute and Data Continuum.

Research Infrastructures

Responsibilities: Incentivize and support the adoption and development and adoption of portable software. Provide training. Invest in appropriate support infrastructures such as software repositories.
Benefits: Able to make use of the federated compute infrastructure for efficient processing of data from the RI. Enhanced scientific impact.

Computing & Data Service Providers / e-Infrastructures

Responsibilities: Facilitate the development of common standards and tools for software portability. Ensure their service offering complies with relevant standards and supports standard environments.
Benefits: A common standard for both interfacing with community software and exchanging software with peer facilities reduces support overhead and increases efficiency, ultimately leading to reduced costs. Make the e-Infrastructure more accessible for end users.

Policy Makers / Funding Bodies

Responsibilities: Define and incentivize strategic goals that recognize the importance of software portability. Ensure the availability of structural funding for key enabling technologies.
Benefits: Increased impact of resources spent on computing due to improved efficiency. More transparent scientific process due to benefits for reusability and reproducibility.

Resource Needs

Funding for the development of key standards and conventions as well as the necessary supporting and enabling technologies. Infrastructure for establishing software repositories and building and storing software and other artefacts. Development resources where the gap analysis shows existing tools to be inadequate.

Critical Dependencies

P04 Heterogeneous Computing Integration
P09 Open Science and Reproducibility
P10 Software Preservation and Sustainability
P11 Community Collaboration and Co-Design
P13 Skills, Careers and Training