3

My corporate firewall policy allows only 20 connections per minute 60 seconds between the same source and destinations.

Owing to this the ansible play hangs after a while.

I would like multiple tasks to use the same ssh session rather than creating new sessions. For this purpose i set the below pipelining = True in the local folder ansible.cfg as well as in the command line.

cat /opt/automation/startservices/ansible.cfg

[defaults]
host_key_checking = False
gathering = smart
[ssh_connection]
ssh_args = -o ControlMaster=auto -o ControlPersist=600s
control_path = %(directory)s/%%h-%%r
pipelining = True

ANSIBLE_SSH_PIPELINING=0 ansible-playbook -i /opt/automation/startservices/finalallmw.hosts /opt/automation/startservices/va_action.yml -e '{ dest_host: myremotehost7 }' -e dest_user=oracle

The playbook is too big to be shared here but it is this task which loops and this is where it hangs due to more than 20 ssh connections in 60 seconds.

 - name: Copying from "{{ inventory_hostname }}" to this ansible server.
   synchronize:
     src: "{{ item.path }}"
     dest: "{{ playbook_dir }}/homedirbackup/{{ inventory_hostname }}/{{ dtime }}/"
     mode: pull
     copy_links: yes
   with_items:
     - "{{ to_copy.files }}"

With the pipelining settings set, my play still hangs after 20 connections.

Below are the playbook settings:

   hosts: "{{ groups['dest_nodes'] | default(groups['all']) }}"
   user: "{{ USER | default(dest_user) }}"
   any_errors_fatal: True
   gather_facts: false
   tags: always

   vars:
     ansible_host_key_checking: false
     ansible_ssh_extra_args: -o StrictHostKeyChecking=no  -o ConnectionAttempts=5

Post suggestions this far on this thread the issue persists. Below is my local directory ansible.cfg

$ cat /opt/automation/startservices/ansible.cfg
# config file for ansible -- http://ansible.com/
# ==============================================

# nearly all parameters can be overridden in ansible-playbook
# or with command line flags. ansible will read ANSIBLE_CONFIG,
# ansible.cfg in the current working directory, .ansible.cfg in
# the home directory or /etc/ansible/ansible.cfg, whichever it
# finds first

[defaults]
host_key_checking = False
roles_path = roles/
gathering = smart
[ssh_connection]
pipelining = True
ssh_args = -o ControlMaster=auto -o ControlPersist=1200s  -o ControlPath=~/.ansible/cp/%r@%h:%p

[persistent_connection]
control_path_dir = ~/.ansible/cp
$

Can you please suggest any solution to the issue on the ansible side where all tasks use the same ssh session and is pipelining not working here?

Ashar
  • 2,942
  • 10
  • 58
  • 122
  • `-o ControlMaster=auto -o ControlPersist=600s` is already supposed to reuse the same master session. Pipelining is yet an other optimization to reduce the number of operations but does still uses several connections if needed without master session set. I strongly suspect your original options are overridden for some reason and that you are not using the master session at all. You can check that running the playbook in debug mode looking at the actual ssh commands being issued. If my previous guess is wrong, `synchronize` might consume connections in a specific way (never debugged that myself) – Zeitounator Jul 23 '21 at 08:58
  • The problem is when the target executes the modules and configuration in ansible.cfg has no effect on that by default. See `synchronize` [use_ssh_args](https://docs.ansible.com/ansible/latest/collections/ansible/posix/synchronize_module.html#parameter-use_ssh_args) and [ssh_connection_multiplexing](https://docs.ansible.com/ansible/latest/collections/ansible/posix/synchronize_module.html#parameter-ssh_connection_multiplexing) – Zeitounator Jul 23 '21 at 10:41
  • @Zeitounator `ssh_connection_multiplexing` is not supported in the old and latest version of ansible. Error here: `"msg": "Unsupported parameters for (synchronize) module: ssh_connection_multiplexing Supported parameters include: _local_rsync_path,_substitute_controller,archive,checksum,compress,copy_links,delete,dest,dest_port,dirs,existing_only,group,links,mode,owner,partial,perms,private_key,recursive,rsync_opts,rsync_path,rsync_timeout,set_remote_user,src,ssh_args,times,verify_host"}` – Ashar Jul 23 '21 at 13:21
  • The documentation I linked is the latest module `ansible.posix.synchronize` from the `ansible.posix` collection which should be installed and used with ansible >= 2.10 – Zeitounator Jul 23 '21 at 14:55
  • @Zeitounator @toydarian It worked. However, I'm more concerned than before. The solution works only when using `ansible.posix.synchronize` and not plain `synchronize` along with both `use_ssh_args: yes` & `ssh_connection_multiplexing: yes` attributes. This means if I face the same issue of ssh connections count with any other module say `lineinfile` I will have no solutions like how i had for `ansible.posix.synchronize`. I would also like to understand why is this solution not introduced or applicable to the default `synchronize` module ? – Ashar Jul 24 '21 at 05:17
  • if using `synchronize` and `ansible.posix.synchronize` as a name lead to different result, I would be very concerned about my ansible installation: the same module should be execudted. Please read ansible documentation to understand how modules have been refactored in v>=2.10. Your above comment actually does not make much sense. There is nothing as a "default `synchronize` module`. Either you use ansilble <2.10 and it is bundled or >=2.10 and it is packaged in the ansible.posix collection. Several collections are included when installing ansible through pip. Good luck. – Zeitounator Jul 24 '21 at 08:09
  • @Zeitounator In the latest ansible 2.9.x when I use `synchronize` i get error `Unsupported parameters for (synchronize) module: ssh_connection_multiplexing` however, when I mention `ansible.posix.synchronize` it works with `ssh_connection_multiplexing: yes` and `use_ssh_args: yes` – Ashar Jul 24 '21 at 09:26
  • Please **actually read** the messages and research the proposed documentations. If using 2.9, `synchronized` is bundled inside the package. You can decide to install the collection to have the latest module but I strongly suggest you upgrade to ansible > 2.10 and you can use synchronize as before with the latest options available. This is not a chat so this will be my last message on the subject. You have all the info to make your own research. – Zeitounator Jul 24 '21 at 09:38
  • @Zeitounator cool but what about my query regarding reuse of ssh connections and not creating new one work for other modules line lineinfile or others work ? The other modules do not have `ssh_connection_multiplexing: yes` and `use_ssh_args: yes` which helped here or do they ? – Ashar Jul 24 '21 at 09:41
  • @Zeitounator i m using latest redhat 8 and ansible v2.10 is not available for my OS. ansible v2.9 is the best redhat offers. Secondly, i would like to know if `ssh_connection_multiplexing: yes` and `use_ssh_args: yes` attributes are present for all other modules where i wish to reuse the same ssh for all my tasks? Can you please answer? – Ashar Jul 27 '21 at 19:01
  • Why would you want options such as `ssh_connection_multiplexing` and `use_ssh_args` for modules which are not internally using ssh? This was only usefull for `syncrhonize` in your case. But if you want to know where those options could exist [just help yourself](https://docs.ansible.com/ansible/latest/collections/index_module.html). About ansible version, just remove the one installed with yum and install through pip. I use ansible 2.11.x (ansible-core version distributed with ansible package 4.2.0) on dozens of machine everyday still running centos/rhel 7 without any problems. – Zeitounator Jul 27 '21 at 22:32
  • @Zeitounator can you post an answer with all options that helped for me to accept? – Ashar Jul 28 '21 at 05:48

2 Answers2

8

First: pipelining = True does not do what you are looking for. It reduces the number of network operations, but not the number of ssh connections. Check the docs for more information.
Imho, it is still a good thing to use, as it will speed up your playbooks.

What you want to use is the "persistent control mode" which is a feature of OpenSSH to keep a connection open.

You could for example do this in your ansible.cfg:

[ssh_connection]
pipelining = True
ssh_args = -o ControlMaster=auto -o ControlPersist=1200

This will keep the connection open for 1200 seconds.

toydarian
  • 4,246
  • 5
  • 23
  • 35
  • Note that OP seems to have master session already set up. My guess is that the config is overridden/not used for some reason. Note also that it is preferable to not set `ControlPath` in the ssh args directly but through the `control_path_dir` and `control_path` options you can find in the [ansible config file](https://github.com/ansible/ansible/blob/4d78b58540dafd818a5e75ec390f9f03f5367ed9/examples/ansible.cfg#L382) – Zeitounator Jul 23 '21 at 08:53
  • @toydarian i tried your suggestions but i continue to see the issue... after 20 connections it does not come back. I think your suggestion persists ssh to 1200 seconds however, it does not ascertain reusability of that ssh connection for other tasks!! Note that a sleep of 5 seconds / pause for 5 seconds between tasks helps overcome the issue but this is a workaround not a solution. Kindly suggest. – Ashar Jul 23 '21 at 09:09
  • 1
    @Zeitounator weird, in the [documentation](https://docs.ansible.com/ansible/latest/reference_appendices/config.html), I can't find `control_path` and `control_path_dir` is supposed to be under `persistent_connection` and not under `ssh_connection`. – toydarian Jul 23 '21 at 09:16
  • @Ashar maybe you have a `ansible.cfg` in a different location or some ENV variables that override your config? Did you check if the sockets are created in `control_path_dir` ? – toydarian Jul 23 '21 at 09:19
  • 1
    @toydarian +1 for the section I totally went through it. Regarding the `control_path` setting, you came to the same conclusion as myself: it is not documented. This mainly why I always check the sample cfg file directly on github for easteregg features ;) – Zeitounator Jul 23 '21 at 09:22
  • I dont have the permission to edit the ansible.cfg that gets placed at the time of ansible installation. i created a fresh ansible.cfg in the directory where this playbook runs. There should be a way to override the global ansible.cfg setting for each play. – Ashar Jul 23 '21 at 09:23
  • @Zeitounator I tried it out, it belongs to `ssh_connection`, so the example was right. I will check that in the future instead of the docs. Fixed my answer accordingly. – toydarian Jul 23 '21 at 09:30
  • @Ashar the file in the directory should override the global one. But you never know what they did to force you using their file. Try `export ANSIBLE_CONFIG=/path/to/your/ansible.cfg` before running your playbook and check the `control_path_dir` if the socket is created. If that is not the case, something is overriding your settings – toydarian Jul 23 '21 at 09:32
  • There seems to be a big mess between the doc and the examples file. I went back to ansible version 1.5 and those settings where always set under `ssh_connection` in the example config. Who will be quicker to open a ticket ? ;) @ashar putting an ansible.cfg at the same level as your playbook **is** the way to override the settings for your local project. – Zeitounator Jul 23 '21 at 09:33
  • I tried exporting but seems the configuration is picked but not a solution to the issue i face. Here: `export ANSIBLE_CONFIG=/opt/automation/startservices/ansible.cfg ls -ltr /opt/automation/startservices/ansible.cfg -rwxr-xr-- 1 localuser localuser 661 Jul 23 04:18 /opt/automation/startservices/ansible.cfg` – Ashar Jul 23 '21 at 09:39
  • @toydarian here is the `control_path_dir` check: `$ cd ~/.ansible/cp $ ls -ltr total 0 srw------- 1 localuser localuser 0 Jul 23 04:34 oracle@myremotehost7:22 ` – Ashar Jul 23 '21 at 09:47
  • 1
    @toydarian: https://github.com/ansible/ansible/issues/75309 ;) – Zeitounator Jul 23 '21 at 09:49
  • @Zeitounator & @toydarian are you saying that with the ssh settings proposed by you the `synchronize` module will use a single ssh connection for all the number of times the `synchronize` module runs in that loop? I m doubting that to be the case as the issue persists and the suggestions have not helped. – Ashar Jul 23 '21 at 09:59
  • I said you should check that in my comment right below your question. There are also [options you can set](https://docs.ansible.com/ansible/latest/collections/ansible/posix/synchronize_module.html#parameter-ssh_connection_multiplexing) for that directly inside the module. The above suggestions are for executing the module itself (i.e. dialog between controller and target), not necessarily for what happens inside the module (i.e. when the target executes it really) – Zeitounator Jul 23 '21 at 10:37
  • @Ashar try setting `use_ssh_args: yes` on your `syncronize` task and make sure you have `-o ControlPath=~/.ansible/cp/%r@%h:%p` on your `ssh_args` in `ansible.cfg` – toydarian Jul 23 '21 at 11:54
  • @toydarian i mentioned `use_ssh_args: yes` in the synchronize module and `ControlPath` in the ansible.cfg but it did not help. It seems to be creating new ssh connection for each iteration of the synchronize loop. – Ashar Jul 23 '21 at 13:18
1

The problem is not with ansible controller running the module(s) (i.e. copying the necessary temporary AnsibleZ files to your target and execute them - you already have the correct options for that in ansible.cfg to use master sessions) but with the synchronize module itself which needs to spawn its own ssh connections to transfer files between relevant servers while it is running on the target.

The latest synchronize module version is now part of the ansible.posix collection and has recently gained 2 options that will help you work around your problem to apply the use of master sessions to the module itself while using rsync.

Although it is possible to install this collection in ansible 2.9 to override the older stock module version (which does not have those options), I strongly suggest you use ansible version 2.10 or 2.11. My personal preferred installation method for ansible is through pip as it will let you install any ansible version on any OS for any user in any number of (virtual) environment.

Regarding pip, the versioning has changed (and is quite a mess IMO...)

  • ansible is now a meta package with its own independent versioning.
  • the usual ansible binaries (ansible, ansible-playbook, ....) are packaged in ansible-core which has the version the corresponding to what you get when running ansible -v
  • the meta ansible package installs a set of collections by default (including the ansible.posix one if I'm not wrong)

=> To get ansible -v => 2.10 you want to install ansible pip package 3.x

=> To get ansible -v => 2.11 you want to install ansible pip package 4.x

You will have to remove any previous version installed via pip in your current environment before proceeding.

Zeitounator
  • 38,476
  • 7
  • 53
  • 66
  • thank you for the answer. However, installing `ansible -v => 2.10` or greater using pip is a challenge in itself. i had raised the query here but seems to be closed. Will open it on unix stack overflow -> https://stackoverflow.com/questions/68510040/unable-to-upgrade-ansible-using-yum-or-pip – Ashar Jul 28 '21 at 09:46