About this blueprint
Outputs Parallel
This example demonstrates how to process files in parallel.
In the bash task, we generate multiple files, and store them in the internal storage.
Next, in the each task, we process these files in parallel, where the tasks path and contents run for each of the 4 output files, resulting in 8 parallel task runs.
Instead of the bash script, you may have a Python/R/Node.js script that generates such files.
yaml
id: parallel_files
namespace: company.team
tasks:
- id: bash
type: io.kestra.plugin.scripts.shell.Commands
runner: PROCESS
outputFiles:
- "out/**"
commands:
- mkdir -p out
- echo "Hello from 1" >> out/output1.txt
- echo "Hello from 2" >> out/output2.txt
- echo "Hello from 3" >> out/output3.txt
- echo "Hello from 4" >> out/output4.txt
- id: each
type: io.kestra.plugin.core.flow.EachParallel
value: "{{ outputs.bash.outputFiles | jq('.[]') }}"
tasks:
- id: path
type: io.kestra.plugin.core.debug.Return
format: "{{ taskrun.value }}"
- id: contents
type: io.kestra.plugin.scripts.shell.Commands
runner: PROCESS
commands:
- cat "{{ taskrun.value }}"