36

Currently, I'm going to need an implementation that must find all files within a directory and start a parallel task for every file found.

Is it possible to achieve this using declarative pipelines?

pipeline {
    agent any
    stages {
        stage("test") {
            steps {
                dir ("file_path") {
                    // find all files with complete path
                    parallel (
                        // execute parallel tasks for each file found.
                        // this must be dynamic
                        }
                    }
                }
            }
        }
    }
}
Pierre.Vriens
  • 7,225
  • 14
  • 39
  • 84
thclpr
  • 871
  • 1
  • 7
  • 9

5 Answers5

41

Managed to solve it with the following code:

pipeline {
    agent { label "master"}
    stages {
        stage('1') {
            steps {
                script {
                    def tests = [:]
                    for (f in findFiles(glob: '**/html/*.html')) {
                        tests["${f}"] = {
                            node {
                                stage("${f}") {
                                    echo '${f}'
                                }
                            }
                        }
                    }
                    parallel tests
                }
            }
        }       
    }
}
Tensibai
  • 11,416
  • 2
  • 37
  • 63
thclpr
  • 871
  • 1
  • 7
  • 9
15

This also works, if you want to stay within the Declarative Pipeline space

// declare our vars outside the pipeline
def tests = [:]
def files

pipeline {
    agent any
    stages {
        stage('1') {
            steps {
                script {
                    // we've declared the variable, now we give it the values
                    files = findFiles(glob: '**/html/*.html')
                    // Loop through them
                    files.each { f ->
                        // add each object from the 'files' loop to the 'tests' array
                        tests[f] = {
                            // we're already in the script{} block, so do our advanced stuff here
                            echo f.toString()
                        }
                    }
                    // Still within the 'Script' block, run the parallel array object
                    parallel tests
                }
            }
        }       
    }
}
primetheus
  • 191
  • 2
  • 8
8

Be aware, that dynamic build steps could cause some problems in some build steps, e.g. when you call an other job:

pipeline {
    stages {
        stage('Test') {
            steps {
                script {
                    def tests = [:]
                    for (f in findFiles(glob: '**/html/*.html')) {
                        // Create temp variable, otherwise the name will be the last value of the for loop
                        def name = f
                        tests["${name}"] = {
                            build job: "${name}"
                        }
                    }
                    parallel tests
                }
            }
        }       
    }
}
Christian Gripp
  • 181
  • 1
  • 1
2

It's much easier to use scripted Pipelines to do this since you can use arbitrary Groovy, but you should still be able to do this with declarative Pipelines using the findFiles step.

jayhendren
  • 3,022
  • 8
  • 16
1

Here is a solution where you can setup the maximum number of concurrent tasks running at once with collate(n).

pipeline {
  stages {
    stage("Parallel") {
      steps {
        script {
          def list = ["alpha", "beta", "gamma", "delta", "epsilon", "zeta", "eta", "theta", "iota", "kappa"]

      // Split the list into batches of 4
      def parallelBatches = list.collate(4);
      // [["alpha", "beta", "gamma", "delta"], ["epsilon", "zeta", "eta", "theta"], ["iota", "kappa"]]

      // This allows to run a maximum of 4 tasks in parallel (at once)
      for (parallelBatch in parallelBatches) {
        def tasks = [:]

        for (task in parallelBatch) {
          tasks["${task}"] = {
            stage("${task}") {
              echo '${task}'
            }
          }
        }
        parallel tasks

        // Once the batch is over, the outer loop move onto the next batch
      }
    }
  }
}



}
}
shellwhale
  • 111
  • 2