Ansible: How to optimize running speed of a playbook?

Question

Current Setup

CPU/RAM 60c/64GB ~2000 servers Time consuming ~1 hour

Ansible playbook actions

• Create folders for each server on localhost and filling them up with templates (vars taken from host inventory's)
• Encrypting the folders
• and all this folders are pushed to 2 Remote Servers

The most time consuming step is the one which creates the folders for each config file. And so on for 2k servers.

What I have tried

• Increasing from 50 > 100 > 400> 800 forks

  • There was improvement in about 5% no more.

• Switching to free strategy takes even more time that linear.

Any ideas how to make it at least x2 faster?

The heaviest task

- name: Create blablabla
  file:
    path: "{{ item.value }}"
    state: directory
    mode: "{{ item.mode | default('0755') }}"
  with_items:
    - { value: "{{ some_dir }}/{{ inventory_hostname }}/XXX/ZZZZ.d", create: "{{ not WWWW }}" }
    - { value: "{{ some_dir }}/{{ inventory_hostname }}/XXX", create: "{{ not WWWW }}" }
    - { value: "{{ some_dir }}/{{ inventory_hostname }}/XXX/EEEE/QQQQ-versions.d" }
    - { value: "{{ some_dir }}/{{ inventory_hostname }}/XXX.d/SSS" }
    - { value: "{{ some_dir }}/{{ inventory_hostname }}/XXX.d/EEE/WEWEW" }
    - { value: "{{ some_dir }}/{{ inventory_hostname }}/XXX/EEE" }
    - { value: "{{ some_dir }}/{{ inventory_hostname }}/XXXX/WWW" }
    - { value: "{{ some_dir }}/{{ inventory_hostname }}/XXXX/DD" }
    - { value: "{{ some_dir }}/{{ inventory_hostname }}/XXXX/SSS/DDD-QQQQ.d" }
    - { value: "{{ some_dir }}/{{ inventory_hostname }}/XXXX/DD" }
    - { value: "{{ some_dir }}/{{ inventory_hostname }}/XXXX/DDD/conf" }
    - { value: "{{ some_dir }}/{{ inventory_hostname }}/XXX/DDD/.docker" }
    - { value: "{{ some_dir }}/{{ inventory_hostname }}/XXX/DD/gpg", mode: "0700" }
    - { value: "{{ some_dir }}/{{ inventory_hostname }}/XXX/DDD.gpg.d", mode: "0755" }

Answer 1

Most time consuming step is the one which creates the folders for each config file.

Right. This is the expected behavior since the file module can't work with a list for parameter path and therefore has to create a single connection for each parameter. This is causing a significant overhead.

Any ideas how to make it at least x2 faster?

I understand that you like to know "How to increase the overall playbook performance and decrease the execution time of specific tasks?" and furthermore you do not want to loop over the directories, instead providing a list of directories.

Yes. Since your problem is quite common and other users experience the same, including myself, I've wrote up a possible solution approach under Ansible - with_items pointed directly at vars rather than expanding them first. It shows specifically the test results and differences in execution time.

Further Similiar Q&A are

• How do I optimize performance of Ansible playbook with regards to SSH connections?
• Parallel execution of localhost tasks in Ansible?

As one can see from the results it can become easily 4 times faster and even much more. It also shows why using async can make the issue worst under certain circumstances.

To summarize, the simplest, quick and lazy approach can be just to use the command or shell module with cmd: mkdir -p ... and as shown here already.

---

Another approach and depending on your infrastructure and other capabilities could be Creating an own Custom Module in Bash or Shell.

Examples can be find under

• File search between two dates with Ansible
• How to execute a shell script on a Remote Server using Ansible?

In general it would just take a list of directories in one string as path parameter

path: "/this,/is,/my,/list,/of,/directories,/to,/create"

and perform a kind of

mkdir -p {$path}

within the module.

Answer 2

Growing the number of forks will help running on several hosts at a time but not running several concurrent tasks on the same host in parallel like when looping. This is where async tasks come into play.

You may notice more performance gain trying the following pseudo/untested playbook. Meanwhile you should not expect too much as the individual task in each loop iteration (i.e. create a folder) is rather fast. So the gain of launching them in parallel will be low and you still need to check and cleanup after yourself which takes some extra time. You're the only judge.

Anyhow, here is the example:

- name: stripped down playbook
  hosts: all_relevant_hosts

  vars:
    # vars only for the example. Stripped down
    some_dir: /path/to/your/base
    config_folders:
      - { value: "{{ some_dir }}/{{ inventory_hostname }}/XXX/EEEE/QQQQ-versions.d" }
      - { value: "{{ some_dir }}/{{ inventory_hostname }}/XXX.d/SSS" }
      - { value: "{{ some_dir }}/{{ inventory_hostname }}/XXX.d/EEE/WEWEW" }
      
  tasks:
    - name: Create configuration folders asynchronously 
      ansible.builtin.file:
        path: "{{ item.value }}"
        state: directory
        mode: "{{ item.mode | default('0755') }}"
      loop: "{{ config_folders }}"
      # This should be more than enough for a directory creation
      async: 60
      poll: 0
      register: create_folder

    - name: Wait until all creations are done
      ansible.builtin.async_status:
        jid: "{{ item.ansible_job_id }}"
      register: async_poll_result
      until: async_poll_result.finished
      # Hopefully this will succeed on first try.
      retries: 60
      delay: 1
      loop: "{{ create_folder.results }}"

    - name: Clean async job cache
      async_status:
        jid: "{{ item.ansible_job_id }}"
        mode: cleanup
      loop: "{{ create_folder.results }}"

Answer 3

Short answer: Instead of creating the items one by one; complexity O(mn) where m is the number of remote hosts and n is the number of items (files, directories, links, ...), create, transfer, and unpack an archive(s); complexity O(m). As a hint, see the comparison below (each set repeated 3 times)

• m=2 n=10 (16.68s, 16.37s, 16.59s) (example 3; 10 directories)
• m=2 n=20 (33.82s, 34.95s, 33.95s) (no example; 20 directories)
• m=2 n=1 (7.06s, 6.93s, 6.46s) (example 4; 1 directory and 1 archive)
• m=1 n=10 (15.33s, 15.03s, 14.89s) (example 5; 10 directories)
• m=1 n=20 (28.77s, 29.21s, 29.83s) (no example; 20 directories)
• m=1 n=1 (6.27s, 5.99s, 6.26s) (example 6; 1 directory and 1 archive)

^{(The remote directories were always deleted before the test.)}

Test serial/fork and different matrix of hosts/items on your own.

---

Details:

1. Create data for testing

Given the simplified list of files for testing

  my_files:
    - value: X/d0
    - value: X/d1
    - value: X/d2
    - value: X/d3
    - value: X/d4
    - value: X/d5
    - value: X/d6
    - value: X/d7
    - {value: X/d8, mode: "0700"}
    - {value: X/d9, mode: "0755"}

Declare the directory for the files and the path for the archive. For example,

  tar_file: /tmp/tar_files.gz
  some_dir: /tmp/test_files

Create the files and the archive on the localhost

    - name: Create local files
      file:
        state: directory
        path: "{{ some_dir }}/{{ item.value }}"
        mode: "{{ item.mode|default('0755') }}"
      loop: "{{ my_files }}"
      delegate_to: localhost
      run_once: true
      when: create_local_files|d(false)|bool

    - name: Create local tar
      archive:
        path: "{{ some_dir }}/*"
        dest: "{{ tar_file }}"
      delegate_to: localhost
      run_once: true
      when: create_local_tar|d(false)|bool

gives

shell> tree /tmp/test_files/
/tmp/test_files/
└── X
    ├── d0
    ├── d1
    ├── d2
    ├── d3
    ├── d4
    ├── d5
    ├── d6
    ├── d7
    ├── d8
    └── d9

11 directories, 0 files

shell> tar tvf /tmp/tar_files.gz
drwxr-xr-x admin/admin       0 2022-12-11 16:20 X/d5/
drwxr-xr-x admin/admin       0 2022-12-11 16:20 X/d2/
drwxr-xr-x admin/admin       0 2022-12-11 16:20 X/d1/
drwxr-xr-x admin/admin       0 2022-12-11 16:20 X/d3/
drwx------ admin/admin       0 2022-12-11 16:20 X/d8/
drwxr-xr-x admin/admin       0 2022-12-11 16:20 X/d9/
drwxr-xr-x admin/admin       0 2022-12-11 16:20 X/d0/
drwxr-xr-x admin/admin       0 2022-12-11 16:20 X/d7/
drwxr-xr-x admin/admin       0 2022-12-11 16:20 X/d4/
drwxr-xr-x admin/admin       0 2022-12-11 16:20 X/d6/

The above tasks are idempotent

shell> ansible-playbook pb.yml -e create_local_files=true
shell> ansible-playbook pb.yml -e create_local_tar=true

---

2. Create the playbook for testing (2 remote hosts, 10 directories)

- hosts: test_11,test_13

  vars:

    tar_file: /tmp/tar_files.gz
    some_dir: /tmp/test_files

    my_files:
    - value: X/d0
    - value: X/d1
    - value: X/d2
    - value: X/d3
    - value: X/d4
    - value: X/d5
    - value: X/d6
    - value: X/d7
    - {value: X/d8, mode: "0700"}
    - {value: X/d9, mode: "0755"}

  tasks:

    - name: Create local files
      file:
        state: directory
        path: "{{ some_dir }}/{{ item.value }}"
        mode: "{{ item.mode|default('0755') }}"
      loop: "{{ my_files }}"
      delegate_to: localhost
      run_once: true
      when: create_local_files|d(false)|bool

    - name: Create local tar
      archive:
        path: "{{ some_dir }}/*"
        dest: "{{ tar_file }}"
      delegate_to: localhost
      run_once: true
      when: create_local_tar|d(false)|bool

    - name: Unpack remote files
      block:
        - file:
            state: directory
            path: "{{ some_dir }}"
        - unarchive:
            src: "{{ tar_file }}"
            dest: "{{ some_dir }}"
      when: unpack_remote_files|d(false)|bool

    - name: Create remote files
      file:
        state: directory
        path: "{{ some_dir }}/{{ item.value }}"
        mode: "{{ item.mode | default('0755') }}"
      loop: "{{ my_files }}"
      when: create_remote_files|d(false)|bool

---

3. Create remote files one by one. Profile the tasks

shell> ANSIBLE_STDOUT_CALLBACK=ansible.posix.profile_tasks ansible-playbook pb.yml -e create_remote_files=true
Sunday 11 December 2022  20:47:32 +0100 (0:00:00.036)       0:00:00.036 ******* 
Sunday 11 December 2022  20:47:32 +0100 (0:00:00.083)       0:00:00.120 ******* 
Sunday 11 December 2022  20:47:32 +0100 (0:00:00.037)       0:00:00.158 ******* 
Sunday 11 December 2022  20:47:32 +0100 (0:00:00.049)       0:00:00.208 ******* 
Sunday 11 December 2022  20:47:32 +0100 (0:00:00.052)       0:00:00.261 ******* 
Sunday 11 December 2022  20:47:49 +0100 (0:00:16.682)       0:00:16.943 ******* 
=============================================================================== 
Create remote files ------------------------------------------------------------------ 16.68s
Create local files -------------------------------------------------------------------- 0.08s
unarchive ----------------------------------------------------------------------------- 0.05s
file ---------------------------------------------------------------------------------- 0.05s
Create local tar ---------------------------------------------------------------------- 0.04s

shell> ssh admin@test_11 ls -la /tmp/test_files/X
total 14
drwxr-xr-x  12 root  wheel  12 Dec 11 19:47 .
drwxr-xr-x   3 root  wheel   3 Dec 11 19:47 ..
drwxr-xr-x   2 root  wheel   2 Dec 11 19:47 d0
drwxr-xr-x   2 root  wheel   2 Dec 11 19:47 d1
drwxr-xr-x   2 root  wheel   2 Dec 11 19:47 d2
drwxr-xr-x   2 root  wheel   2 Dec 11 19:47 d3
drwxr-xr-x   2 root  wheel   2 Dec 11 19:47 d4
drwxr-xr-x   2 root  wheel   2 Dec 11 19:47 d5
drwxr-xr-x   2 root  wheel   2 Dec 11 19:47 d6
drwxr-xr-x   2 root  wheel   2 Dec 11 19:47 d7
drwx------   2 root  wheel   2 Dec 11 19:47 d8
drwxr-xr-x   2 root  wheel   2 Dec 11 19:47 d9
shell> ssh admin@test_13 ls -la /tmp/test_files/X
...

The directories were created in 16.68s (repeated 2 more times in 16.37s and 16.59s).

---

4. Unarchive remote files. Profile the tasks

(remove the remote files /tmp/test_files)

shell> ANSIBLE_STDOUT_CALLBACK=ansible.posix.profile_tasks ansible-playbook pb.yml -e unpack_remote_files=true
Sunday 11 December 2022  20:53:26 +0100 (0:00:00.034)       0:00:00.034 ******* 
Sunday 11 December 2022  20:53:26 +0100 (0:00:00.082)       0:00:00.116 ******* 
Sunday 11 December 2022  20:53:26 +0100 (0:00:00.038)       0:00:00.155 ******* 
Sunday 11 December 2022  20:53:28 +0100 (0:00:02.084)       0:00:02.239 ******* 
Sunday 11 December 2022  20:53:33 +0100 (0:00:04.979)       0:00:07.219 ******* 
Sunday 11 December 2022  20:53:33 +0100 (0:00:00.111)       0:00:07.331 ******* 
=============================================================================== 
unarchive ----------------------------------------------------------------------------- 4.98s
file ---------------------------------------------------------------------------------- 2.08s
Create remote files ------------------------------------------------------------------- 0.11s
Create local files -------------------------------------------------------------------- 0.08s
Create local tar ---------------------------------------------------------------------- 0.04s

The directories were created and unarchived in 7.06s (2.08s + 4.98s) (repeated 2 more times in 6.93s (2.04s + 4.89s) and 6.46s (1.76s + 4.70s)).

---

5. Create remote files one by one. Single remote host - hosts: test_11 Profile the tasks.

The directories were created in 15.33s (repeated 2 more times in 15.03s and 14.89s).

---

6. Unarchive remote files. Single remote host - hosts: test_11 Profile the tasks.

The directories were created and unarchived in 6.27s (1.93s + 4.34s) (repeated 2 more times in 5.99s (1.59s + 4.40s) and 6.26s (1.91s + 4.35s)).