Workflow Orchestration and Management
Scientific workflows are becoming increasingly sophisticated. Advanced workflow management systems capable of orchestrating multi-step processes across distributed resources and domains, ensuring data locality, and optimizing resource utilization, are therefore needed. Unifying workflow environments is crucial for sustaining and expanding innovative approaches such as AI-based workflows. Automated workflow tools are needed to streamline updates and deployments of e.g. AI models in real-time applications. Mixed workloads that combine embarrassingly parallel tasks with tightly coupled AI/ML processes must also be supported. Modern systems must adapt to support this diversity by providing heterogeneous architectures that integrate CPUs, GPUs, and high-speed interconnects. Streamlining workflow software and using automated workflow management for orchestrating, monitoring, accounting, is also important to reduce operational overhead and maximise resource utilisation
Strategic Rationale and Evidence Base
Strategic Importance
Scientific workflows are becoming increasingly sophisticated. Advanced workflow management systems capable of orchestrating multi-step processes across distributed resources and domains, ensuring data locality, and optimizing resource utilization, are therefore needed. Unifying workflow environments is crucial for sustaining and expanding innovative approaches such as AI-based workflows. Automated workflow tools are needed to streamline updates and deployments of e.g. AI models in real-time applications. Mixed workloads that combine embarrassingly parallel tasks with tightly coupled AI/ML processes must also be supported. Modern systems must adapt to support this diversity by providing heterogeneous architectures that integrate CPUs, GPUs, and high-speed interconnects. Streamlining workflow software and using automated workflow management for orchestrating, monitoring, accounting, is also important to reduce operational overhead and maximise resource utilisation
Current State
Well-developed, automated tools enable workflow management for complex tasks including submission, monitoring, accounting of workloads, and their data management. Integrating these management systems into HPC centres has required careful tailoring to match the computational resources, configuration, and policies. Standard mechanisms for integrating HPC centres into workload management systems are required to maximize HPC adoption. This has been tackled by european projects such as EXTRACT project
Community Evidence
The analysis of the interoperable access policies (D5.2) and the technical blueprint for compute and data continuum (D6.1) prepared by the SPECTRUM project identifies following gaps related to workflow orchestration:
- #5: Workflow execution and orchestration: Workflow engines are fragmented across communities. Cross-facility workflows lack resilience and robust, standardised provenance capture.
- #15: High-level end-user interfaces: Most e-Infrastructures do not offer high-level, abstracted interfaces. Domain scientists should not need to know system details and interact with brittle application interfaces.
- #16: Unattended execution of long-running workflows Ensure reliable and unattended execution of long-running workflows and user-operated services.
Technical Foundation
Workload Management relies on the following supporting SPECTRUM Compute and Data Continuum capabilities (see D6.1 Section 4.4): Workflow Management, Resource Orchestration, Task Scheduling, Provenance
European Alignment
Workflow management is crucial to broader European strategies for sustaining and expanding innovative approaches such as AI-based workflows, and mixed workloads that combine embarrassingly parallel tasks with tightly coupled processes, such as those used for the application of AI/ML. Resource providers will benefit from automated workflow management that makes efficient usage of their resources and their exposure to user communities. The EOSC web of fair data and service benefits from a unified workflow management that is independent of the implementation details and that enables seamless access for European researchers to compute and data resources.
Target Outcomes and Impact Assessment
Strategic Outcomes
- Workflow orchestration evolves into a cross-facility capability with built-in resilience and capturing of provenance.
- Workflow scheduling and orchestration supports interactive compute use cases
- High-level general and domain-specific user interfaces are deployed to lower the threshold to a broader research community.
Success Indicators
- Number of use cases/domains supported by a standardised workflow execution environment (to be tracked per environment and framework)
- Number of resource providers enabling cross-facility capabilities
- Number of supported use cases and associated number of users
European Impact
Standardised workflow management enables researchers to access compute resources and execute their workflows efficiently and independently on the implementation details, thus decreasing the time-to-completion of their research projects and boosting the competitiveness of the European Research Area.
Implementation Pathway and Stakeholder Roles
Implementation Phases
- Short-term (Years 1-3).
- Continuation of the gap analysis that started with the landscape analysis of SPECTRUM for a selection of use cases.
- Identification of critical interfaces with 1) research infrastructures, 2) e-Infrastructures, 3) (general) user interfaces/API’s.
- Identification and continuation of work for viable orchestration & workflow management frameworks for supporting cross-facility & cross-domain scientific workflows.
- Pilot one or more use cases (HEP, SRCnet,LOFAR).
- Medium-term (Years 3-5)
- Prototyping & demonstration of selected use cases with selected e-Infrastructure providers.
- Validation of being fit-for-purpose.
- Exploration of scope of use cases that can/should be supported (possibly resulting in multiple classes of cases, each requiring a specific solution.
- Drafting standards and practices that (each) stakeholder must adhere to (e.g. Blueprint Architecture).
- Long-term (Years 5+)
- Roll-out across European e-Infrastructures, grow communities & research infrastructures that build on/benefit.
Stakeholder Roles
- Scientific Communities, Research Infrastructures, Computing & Data Service Providers / e-Infrastructures
- Responsibilities: Co-design and converge on a limited set of interoperable workflow systems, with support for unattended execution, provenance tracking, and cross-infrastructure orchestration.
- Benefits: (for scientific communities / research infrastructures) scientific output is accelerated; (for Computing & Data Service Providers / e-Infrastructures) resource utilization is optimised.
- Scientific Communities, Research Infrastructures, Computing & Data Service Providers / e-Infrastructures
- Responsibilities: Extend scheduling/orchestration to support interactive compute use cases.
- Benefits: (for scientific communities): being able to perform interactive data analysis tasks on a scalable infrastructure, with predictable, resilient behaviour and sufficiently low latency/performance; (for Computing & Data Service Providers / e-Infrastructures) efficient use of owned resources.
- Computing & Data Service Providers / e-Infrastructures
- Responsibilities: support deployment of/integration with domain-specific user interfaces by e.g. providing service hosting and integration APIs.
- Benefits: diverse use of infrastructure; (for users) the high-level part of the scientific workflows can be scaled up to ever-increasing dataset dimensions
Resource Needs
- Compute, network, (temporary) storage resources scaling up through the phases (pilot/demonstration/full deployment).
- Engineering resources for porting/building/demonstrating capabilities (embedded in research infrastructures & e-Infrastructure providers.
- Long-term resources for supporting & maintaining shared capabilities.
Critical Dependencies
- Coordination between research infrastructures & e-Infrastructures towards achieving fit-for-purpose unification of orchestration & workflow frameworks.
- Commitment from e-Infrastructure providers to provide the required unified capabilities through harmonized interfaces.
Cross-priority dependencies with:
- P01 (Cross-Infrastructure Governance),
- P02 (Federated Resource Allocation),
- P03 (Access, Security & Trust),
- P04 (Heterogeneous Computing Integration),
- P05 (Data Management and Transport),
- P08 (Portability - Software Portability)