Is there a way to run with_items loops in parallel in Ansible?

Question

I'm running Ansible 2.2, but can upgrade if it helps.

I saw this and was pretty excited, but it doesn't seem to be in this (or any) version of Ansible documentation.

The problem I'm trying to solve is I've got 1000 users that I need to manage on a Centos box.

It takes quite a while to run this task serially. And even more annoying, everything shows up as changed because the "expires" command on the user module always marks the thing as changed.

this also looked promising, but it took the same amount of time to run each command in the with_items loop and didn't go any faster (I never bothered to wait long enough to get to the end).

Skipping tasks is fast now (a lot faster than it was in Ansible 2.0), if I can't figure out how to make this work in parallel, I think I'll go back and figure out how to skip pointless tasks and if all else fails, I'll write my own module. But it seems like I should be able to do all this faster in Ansible.

This is what I want to run in parallel, host_authorizations is a list of usernames and other data.

  - name: Create/modify OS user accounts
    user: name={{ item.username }} group=sshusers shell=/bin/bash home="/home/selinux-modules/{{ item.username }}" state=present expires={{item.expiredate|default(omit)}}
    with_items: "{{ host_authorizations }}"
    tags: full_maintenance

score 35 · Answer 1 · edited Oct 13 '21 at 18:03

As @webKnja mentioned this is possible with async mode. I have recently discovered it myself and learned that you can use it in 3 different ways depending on your needs.

Execute and poll the results, notice the poll:5, This will poll the results every 5 seconds. You may save some time with this method.

 - name: My long runing task
   some_module_name:
     ip: "{{item.fabric}}"
     username: "{{user}}"
     password: "{{password}}"
     secret: "{{secret}}"
   loop: "{{zoning_list}}"
   register: _alias_vc_0
   async: 60
   poll: 5

Fire and forget poll: 0, This is very quick option since Ansible it's just shooting out those tasks. The down side is that we don't know what was the outcome of the task i.e. changed: True/False. Of course it's a downside if you care about the feedback ;).
```
 name: My long runing task
 some_module_name:
   ip: "{{item.fabric}}"
   username: "{{user}}"
   password: "{{password}}"
   secret: "{{secret}}"
 loop: "{{zoning_list}}"
 register: _alias_vc_0
 async: 60
 poll: 0
```
Fire and forget with async_status, the syntax for the task is the same as example 2 whowever it will require additional task async_status. This is my favorite since it's relatively fast (faster then normal looping or the execute and poll) and allows you to capture the feedback although will need to deal with new register for your async_task.

retries: 20 -- how many attempts before failing.

delay: 2 -- how many second to wait between polls.

    - name: My long runing task
      some_module_name:
        ip: "{{item.fabric}}"
        username: "{{user}}"
        password: "{{password}}"
        secret: "{{secret}}"
      loop: "{{zoning_list}}"
      register: _alias_vc_0
      async: 60
      poll: 0
- name: Wait for My long running task to finish
  async_status:
    id: &quot;{{ item.ansible_job_id }}&quot;
    #jid: &quot;{{ item.ansible_job_id }}&quot; # ansible version &gt; 2.8
  register: _jobs_alias_vc_0
  retries: 20
  delay: 2
  until: _jobs_alias_vc_0.finished
  loop: &quot;{{_alias_vc_0.results}}&quot;

A word of caution, depending on the task yo may not be able to use the async option. I had examples where I was interacting with system which was not able to handle multiple requests for the same resource. I found async option best working if I have to perform the same task across multiple hosts. That's where I was able to "save" the most time.

Since you posted the link to Ansible documentation in the question I'm not going to do that.

score 6 · Answer 2 · edited Apr 11 '19 at 15:39

It is possible to achieve this using async mode. Please find some references for how to do this below.

Refs:

---

- name: Run tasks in parallel
  hosts: localhost
  connection: local
  gather_facts: no
  tasks:
    - name: Pretend to create instances
      command: "sleep {{ item }}"  # Instead of calling a long running operation at a cloud provider, we just sleep.
      with_items:
        - 6
        - 8
        - 7
      register: _create_instances
      async: 600  # Maximum runtime in seconds. Adjust as needed.
      poll: 0  # Fire and continue (never poll)

    - name: Wait for creation to finish
      async_status:
        jid: "{{ item.ansible_job_id }}"
      register: _jobs
      until: _jobs.finished
      delay: 5  # Check every 5 seconds. Adjust as you like.
      retries: 10  # Retry up to 10 times. Adjust as needed.
      with_items: "{{ _create_instances.results }}"

score 3 · Answer 3 · edited Apr 22 '21 at 05:17

To answer your question: No, as of now Ansible can't run loops in parallel.

I'd use newusers instead, which is made for bulk user creation. Create a file with all users in it, copy it over to the host, and run newusers /path/to/user/list in a command or shell task.

For example:

- name: Create multiple users
  shell: newusers /path/to/user/list && touch /path/to/user/ansible-users-added
  args:
    creates: /path/to/user/ansible-users-added

Is there a way to run with_items loops in parallel in Ansible?

3 Answers3