Run a subflow for each value in parallel and wait for their completion — recommended pattern to iterate over hundreds or thousands of list items
About this blueprint
Parallel
First, create the following flow that we'll use as a parametrized subflow:
id: my_subflow
namespace: company.team
inputs:
- id: my_input
type: STRING
defaults: kestra
tasks:
- id: super_task
type: io.kestra.plugin.core.debug.Return
format: hi from {{ flow.id }} using input {{ inputs.my_input }}
Then, create the flow subflow_for_each_value.
This flow will trigger multiple executions of the flow my_subflow. Each execution will be triggered in parallel using input from the list of values. In this example, you should see three executions of the subflow, one with the input user1, another with the input user2 and yet another execution with the input user3.
This pattern is particularly useful if the list of values you iterate over is large. As explained in the Flow best practices documentation, it's better to execute thousands of flows, each running one task, than to trigger one flow with thousands of parallel tasks. We recommend this pattern with subflows as it's easier to scale and it makes your workflows more modular and easier to maintain.
id: subflow_for_each_value
namespace: company.team
tasks:
- id: parallel
type: io.kestra.plugin.core.flow.ForEach
concurrencyLimit: 50
values: "{{ range(1, 400) }}"
tasks:
- id: subflow
type: io.kestra.plugin.core.flow.Subflow
flowId: my_subflow
namespace: company.team
inputs:
my_input: "{{ taskrun.value }}"
More Related Blueprints