How to efficiently collect Remote Host information in Ansible or Python?

Question

I want to collect some information of the Remote Host, I try to use Ansible to execute some shell commands, but when executed on some machines, I found that its efficiency is a bit slow and consumes a lot of resources.

Here are some shell commands I wrote to collect the information I want.

# Startup status of some services
chkconfig --list ip6tables| awk '{$1=""; print $0}' | sed 's/\s\s*/ /g'
# Some kernel parameters, such as vm.swappiness
sysctl vm.swappiness|awk '{print $3}'
# Other parameters that need to read the file
cat /etc/logrotate.conf |grep -i '^rotate'|awk '{print $2}'
# some database information
lsnrctl status | grep -v 'Connecting'|grep -Po '(?<=HOST=)[^)]+|(?<=PORT=)\d+'|sed 'N;s/\n/:/'  

I use the shell module of Ansible to execute, but on some machines, it executes slowly, about 55 similar commands, and the execution takes more than 100 seconds.

Is there any other way to increase speed and reduce resource consumption?

Answer 1

According your current vague description without any measurements and further details

I try to use Anisble to execute some shell commands, but when executed on some machines, I found that its efficiency is a bit slow and consumes a lot of resources. ... I use the shell module of Ansible to execute, but on some machines, it executes slowly, about 55 similar commands, and the execution takes more than 100 seconds.

this seems to be the expected behavior since for every command a new connection is created, a small program is sent over to the Remote Node for execution and the result is collected and sent back over the wire to the Control Node.

According

Is there any other way to increase speed and reduce resource consumption?

I understand that you are interested in a performance gain. The short answer is yes, of course. For how to achieve that you may need to take certain problem domains into account. And it is way to complex to provide a single on the point answer here.

1. Get an idea about how the shell module is working and why you are experience such behavior. A first starting point can be How do I optimize performance of Ansible playbook with regards to SSH connections?

2. Get an idea about Distributed Computing, Local Execution, Remote Execution, etc. Where to generate and collect the data and information? How to generate and collect the data and information? It might turn out that you are interested in Custom facts about the Remote Nodes which could be generated by themselfs upfront. To do so, Adding Custom Facts to the Remote Nodes and just collect all results via gather_facts, the setup module in single run and without any further playbook code.

3. Get an idea or full list of what kind of information you are interested in. It might turn that most Metrics are already covered by Monitoring Tools, in example such as Prometheus Node Exporter and the Collectors. From a very high level view it works somehow similar as Ansible local facts.

4. Depending on your not further described use case you can probably take advantage of using available Ansible modules or implementing own Custom Modules, in example for the database information.

After research, reading and taking certain problem domains into account and addressing them appropriate, it will be possible to decrease the runtime to 1/4 up to 1/10 easily, whereby I've seen already result of up to 1/100.

Some Further Readings

additionally to the already linked ...

• Ansible: How to optimize running speed of a playbook?

• Parallel execution of localhost tasks in Ansible

• Ansible - 'with_items' pointed directly at 'vars' rather than expanding them first

• Ansible Issue #76380 - "Improve Ansible performance"

• How to search for a string in a remote file using Ansible?

• How to check if a file is of type Human readable in Ansible?

It is recommend to read them and even follow the links there too.

Summary

To summarize, avoid using the shell module for a long list of single commands and your specific use case at all.