Software Distribution and Execution
The software, compiler, and library ecosystem is the critical abstraction layer that bridges scientific applications with the heterogeneous resources of the Compute and Data Continuum. While the Compute and Data resources provide the raw infrastructure, it is the software stack that makes these resources usable, and enables portable, and reproducible workflows for global collaborations. For communities such as HEP and RA, whose workflows must run consistently across hundreds of sites and over decades, a robust and harmonized software and execution environment is indispensable.
A central feature of the continuum is the ability to distribute scientific software consistently across federated sites. This is achieved through mechanisms such as software registries, container registries, and virtual file systems.
- Software and compiler registries act as catalogues of validated applications, frameworks, and libraries, annotated with metadata on authorship, licensing, and hardware compatibility.
- Container registries store portable images (e.g., Docker, Apptainer/Singularity), allowing scientists to encapsulate complex environments and run them reproducibly across HPC, cloud, and edge.
- Read Only Virtual file systems, such as CVMFS, enable lightweight, versioned, and up to date distribution of software environments to thousands of nodes, a model already proven in WLCG and elsewhere (e.g. the Vera C. Rubin Observatory)
Together, these services ensure that researchers can access the correct software stack regardless of the underlying hardware or site-specific configuration.
To ensure flexibility, the continuum supports dynamic runtime configuration across heterogeneous systems. This includes environment module systems for selecting and switching software stacks, automated dependency resolution to prevent conflicts, cross-architecture support for x86, ARM, RISC-V, GPU nodes, and other emerging accelerators. Although environment modules are a de facto standard in HPC, they are not yet widely adopted in the HEP/RA communities, which rely more heavily on container-based workflows. Within the European HPC ecosystem, EasyBuild has emerged as the de facto standard for module generation, with EESSI, distributed via CVMFS and mandated for EuroHPC systems through the EFP, providing an optimised software compilation and distribution layer. Strengthening alignment between HEP/RA software practices and these EuroHPC approaches represents a concrete, actionable step toward improving interoperability and reducing duplication across the continuum.
The execution environment must also support runtime flexibility, enabling users and workflow systems to request specific software stacks, compilers, or libraries declaratively. This includes selecting between different MPI implementations, choosing accelerator-enabled maths libraries, or loading architecture-appropriate builds of AI frameworks such as PyTorch, TensorFlow, or JAX. Exposing these capabilities through stable APIs is essential not only for seamless integration with workflow orchestrators, but also for ensuring long-term reproducibility and reuse. In this perspective, execution environments should be versioned, preservable, and discoverable, so that analyses can be re-executed or adapted over time despite the evolution of hardware and software ecosystems, while avoiding unnecessary redevelopment and duplication of effort.
For both HEP and RA, portable and well-preserved execution environments are key to ensuring that complex reconstruction frameworks, data reduction pipelines, AI/ML models, and simulation codes can run on globally distributed resources with consistent behaviour over long time periods. Technologies such as CVMFS, containerisation, and workflow provenance already play a central role in research software ecosystems and provide a foundation for this continuity. Extending their use to support systematic preservation, reuse, and cross-community sharing of software and its execution context can reduce heterogeneity, improve efficiency, and contribute to sustainability by limiting redundant software development and enabling more effective use of existing computing infrastructures.