2

I have an Ansible playbook to perform the following task:

Access a remote host via ssh and create a text file that lists all the backups on the remote host filesystem that can be inserted into an email body. I am looking for a more efficient way to write this or to find a best practice to implement this.

Any and all feedback is greatly appreciated!

Below is the current iteration in use to locate backups on the remote host:

- name: Create backup log from Remote Host
  shell: |
    cd /mnt/bntestmnt; echo -e "Backup server: [ipv4 address goes here] \nBackup directory: /export
    /Backups/ESG \n\nConfluence Backups:" > BackupStatus.txt
    ls -td Confluence/daily/* | xargs du -sh >> BackupStatus.txt; echo " " >> BackupStatus.txt; 
    ls -td Confluence/weekly/* | xargs du -sh >> BackupStatus.txt; echo -e "\nGitlab Backups:" >> 
    BackupStatus.txt
    ls -td GitLab/daily/* | xargs du -sh >> BackupStatus.txt; echo " " >> BackupStatus.txt
    ls -td GitLab/weekly/* | xargs du -sh >> BackupStatus.txt; echo -e "\nJira Backups:" >> 
    BackupStatus.txt
    ls -td Jira/daily/* | xargs du -sh >> BackupStatus.txt; echo " " >> BackupStatus.txt
    ls -td Jira/weekly/* | xargs du -sh >> BackupStatus.txt; cat BackupStatus.txt
  register: Backup_Status
  delegate_to: [remote host name goes here]

Because '>>' redirects the command output to a text file, Ansible cannot see the result, so 'cat' is used to display the final product so it can be logged as a variable, to be inserted into the email body.

This is the expected output:

    Backup server: [ipv4 address goes here] 
    Backup directory: /export/Backups/ESG 
    
    Confluence Backups:
    4.1G    Confluence/daily/Thu-23Mar23
    4.0G    Confluence/daily/Wed-22Mar23
    4.0G    Confluence/daily/Tue-21Mar23
    4.0G    Confluence/daily/Mon-20Mar23
    4.0G    Confluence/daily/Sun-19Mar23
    3.9G    Confluence/daily/Sat-18Mar23
    3.9G    Confluence/daily/Fri-17Mar23
     
    3.7G    Confluence/weekly/Fri-10Mar23
    3.5G    Confluence/weekly/Fri-03Mar23
    3.4G    Confluence/weekly/Fri-24Feb23
    3.3G    Confluence/weekly/Sat-18Feb23
    
    Gitlab Backups:
    2.5G    GitLab/daily/Thu-23Mar23
    2.5G    GitLab/daily/Wed-22Mar23
    2.4G    GitLab/daily/Tue-21Mar23
    2.4G    GitLab/daily/Mon-20Mar23
    2.4G    GitLab/daily/Sun-19Mar23
    2.3G    GitLab/daily/Sat-18Mar23
    2.3G    GitLab/daily/Fri-17Mar23
     
    2.2G    GitLab/weekly/Fri-10Mar23
    2.1G    GitLab/weekly/Fri-03Mar23
    2.2G    GitLab/weekly/Fri-24Feb23
    
    Jira Backups:
    2.6G    Jira/daily/Thu-23Mar23
    2.6G    Jira/daily/Wed-22Mar23
    2.6G    Jira/daily/Tue-21Mar23
    2.6G    Jira/daily/Mon-20Mar23
    2.6G    Jira/daily/Sun-19Mar23
    2.6G    Jira/daily/Sat-18Mar23
    2.6G    Jira/daily/Fri-17Mar23
     
    2.6G    Jira/weekly/Fri-10Mar23
    2.6G    Jira/weekly/Fri-03Mar23
    2.6G    Jira/weekly/Fri-24Feb23
    2.6G    Jira/weekly/Fri-17Feb23

The list within each application has a newline between the last instance of 'daily' and the first instance of 'weekly'

Some alternate formats I have tried to use are below. I ran into issues in trying to format the output to achieve the desired result.

`ls -td Confluence/{daily,weekly}/* | xargs du -sh`

or

`find Confluence -mindepth 2 -type d -exec du -sh {} \;`
(This version seems to sort the list incorrectly, as the size/dates are out of order)

I've also looked into modifying the task to utilize Jinja2 to create a template, but feel that I may be out of my depth with that method. See below for the attempted format:

- name: Create backup log from Remote Host
  template:
    src: backup_report.j2
    dest: /mnt/bntestmnt/BackupStatus.txt
  vars:
    confluence_backups: [variable goes here]
    gitlab_backups: [variable goes here]
    jira_backups: [variable goes here]

I'm still trying to figure out how I would write the contents of "backup_report.j2", but I have the following inside:

Backup server: [ipv4 address]
Backup directory: /export/Backups/ESG

Confluence Backups: 
{% for backup in confluence_backups %}
{{ backup }}
{% endfor %}

Gitlab Backups: 
{% for backup in gitlab_backups %}
{{ backup }}
{% endfor %}

Jira Backups: 
{% for backup in jira_backups %}
{{ backup }}
{% endfor %}
Eridith
  • 23
  • 4
  • 1
    Hi Eridith welcome to SO. Your question contains the "expected output" but doesn't say what _problem_ you're having aside from "ran into issues" which is almost a given since you're here asking questions. Please [edit your question](https://stackoverflow.com/posts/75871703/edit) and make it an [MCVE](https://stackoverflow.com/help/mcve). Good luck! – mdaniel Mar 29 '23 at 01:39

3 Answers3

2

Given the simplified tree both on the host test_11 and test_13

shell> ssh admin@test_11 find /tmp/bntestmnt
/tmp/bntestmnt
/tmp/bntestmnt/Gitlab
/tmp/bntestmnt/Gitlab/weekly
/tmp/bntestmnt/Gitlab/weekly/Fri-10Mar23
/tmp/bntestmnt/Gitlab/weekly/Fri-03Mar23
/tmp/bntestmnt/Gitlab/daily
/tmp/bntestmnt/Gitlab/daily/Sat-18Mar23
/tmp/bntestmnt/Gitlab/daily/Sun-19Mar23
/tmp/bntestmnt/Gitlab/daily/Fri-17Mar23
/tmp/bntestmnt/Confluence
/tmp/bntestmnt/Confluence/weekly
/tmp/bntestmnt/Confluence/weekly/Fri-03Mar23
/tmp/bntestmnt/Confluence/weekly/Fri-10Mar23
/tmp/bntestmnt/Confluence/daily
/tmp/bntestmnt/Confluence/daily/Sat-18Mar23
/tmp/bntestmnt/Confluence/daily/Sun-19Mar23
/tmp/bntestmnt/Confluence/daily/Fri-17Mar23
/tmp/bntestmnt/Jira
/tmp/bntestmnt/Jira/weekly
/tmp/bntestmnt/Jira/weekly/Fri-10Mar23
/tmp/bntestmnt/Jira/weekly/Fri-03Mar23
/tmp/bntestmnt/Jira/daily
/tmp/bntestmnt/Jira/daily/Sun-19Mar23
/tmp/bntestmnt/Jira/daily/Sat-18Mar23
/tmp/bntestmnt/Jira/daily/Fri-17Mar23

The play

- hosts: all

  vars:

    bck_remote_dir: /tmp/bntestmnt
    bck_dir: /export/Backups/ESG
    bck:
      Confluence: [daily, weekly]
      Gitlab: [daily, weekly]
      Jira: [daily, weekly]

  tasks:

    - shell:
        cmd: "ls -td {{ item.0.key }}/{{ item.1 }}/* | xargs du -sh"
        chdir: "{{ bck_remote_dir}}"
      with_subelements:
        - "{{ bck|dict2items }}"
        - value
      register: bck_status
    - debug:
        var: bck_status
      when: debug|d(false)|bool

    - copy:
        dest: /tmp/BackupStatus.txt
        content: |
          Backup server: {{ ansible_host }}
          Backup directory: {{ bck_dir }}

          {% for i in bck_status.results %}
          {{ i.item.0.key }} {{ i.item.1 }} Backups:
          {% for j in i.stdout_lines %}
          {{ j }}
          {% endfor %}

          {% endfor %}

gives (the same for test_13)

shell> ssh admin@test_11 cat /tmp/BackupStatus.txt
Backup server: 10.1.0.61
Backup directory: /export/Backups/ESG

Confluence daily Backups:
4.5K    Confluence/daily/Sun-19Mar23
4.5K    Confluence/daily/Sat-18Mar23
4.5K    Confluence/daily/Fri-17Mar23

Confluence weekly Backups:
4.5K    Confluence/weekly/Fri-10Mar23
4.5K    Confluence/weekly/Fri-03Mar23

Gitlab daily Backups:
4.5K    Gitlab/daily/Sun-19Mar23
4.5K    Gitlab/daily/Sat-18Mar23
4.5K    Gitlab/daily/Fri-17Mar23

Gitlab weekly Backups:
4.5K    Gitlab/weekly/Fri-10Mar23
4.5K    Gitlab/weekly/Fri-03Mar23

Jira daily Backups:
4.5K    Jira/daily/Sun-19Mar23
4.5K    Jira/daily/Sat-18Mar23
4.5K    Jira/daily/Fri-17Mar23

Jira weekly Backups:
4.5K    Jira/weekly/Fri-10Mar23
4.5K    Jira/weekly/Fri-03Mar23

Example of a complete playbook to create the tree for testing

shell> cat pb-create.yml
- hosts: all

  vars:

    bck_remote_dir: /tmp/bntestmnt
    bck_files:
      daily: [Fri-17Mar23, Sat-18Mar23, Sun-19Mar23]
      weekly: [Fri-03Mar23, Fri-10Mar23]
    bck:
      Confluence: [daily, weekly]
      Gitlab: [daily, weekly]
      Jira: [daily, weekly]

  tasks:

    - file:
        state: directory
        path: "{{ bck_remote_dir }}"

    - file:
        state: directory
        path: "{{ bck_remote_dir }}/{{ item.0.key }}/{{ item.1 }}"
      with_subelements:
        - "{{ bck|dict2items }}"
        - value

    - shell: "for file in {{ files }}; do cp /etc/passwd $file; done"
      with_subelements:
        - "{{ bck|dict2items }}"
        - value
      vars:
        dir: "{{ bck_remote_dir }}/{{ item.0.key }}/{{ item.1 }}"
        files: "{{ [dir]|product(bck_files[item.1])|
                         map('join', '/')|
                         join(' ') }}"
Vladimir Botka
  • 58,131
  • 4
  • 32
  • 63
  • 1
    Your suggestion works! I was able to implement the playbook with the requisite adjustments to make it fit the parameters I have in place. I will post the final result in an answer to the post. Thanks for the help! – Eridith Mar 29 '23 at 17:50
0

As pointed out by user Vladimir Botka, this implementation provides an approach via templates to create the file. Here is the final product:

    ---
    - name: StackOverflow Test Playbook
      hosts: localhost
      become: true
      gather_facts: false
      vars:
        bck_remote_dir: /mnt/bntestmnt
        bck_dir: /export/Backups/ESG
        bck:
          Confluence: [daily, weekly]
          GitLab: [daily, weekly]
          Jira: [daily, weekly]
    
      tasks:
        - name: Retrieve Backup data
          shell:
            cmd: "ls -td {{ item.0.key }}/{{ item.1 }}/* | xargs du -sh"
            chdir: "{{ bck_remote_dir }}"
          with_subelements:
            - "{{ bck|dict2items }}"
            - value
          register: bck_status
          delegate_to: (remote hostname is here)
        - debug:
            var: bck_status
          when: debug|d(false)|bool
    
        - name: Create backup log from Remote Host
          ansible.builtin.copy:
            dest: /mnt/bntestmnt/results.txt
            mode: 0755
            content: |
              Backup server: (plaintext ipv4 address is here)
              Backup directory: /export/Backups/ESG
    
              {% for i in bck_status.results %}
              {{ i.item.0.key }} {{ i.item.1 }} Backups:
              {% for j in i.stdout_lines %}
              {{ j }}
              {% endfor %}
    
              {% endfor %}
          delegate_to: (remote hostname is here)
    ...

The output of the .txt file is as follows:

Backup server: 10.5.5.50
Backup directory: /export/Backups/ESG

Confluence daily Backups:
4.4G    Confluence/daily/Wed-29Mar23
4.4G    Confluence/daily/Tue-28Mar23
4.3G    Confluence/daily/Mon-27Mar23
4.3G    Confluence/daily/Sun-26Mar23
4.2G    Confluence/daily/Sat-25Mar23
4.1G    Confluence/daily/Fri-24Mar23
4.1G    Confluence/daily/Thu-23Mar23

Confluence weekly Backups:
3.9G    Confluence/weekly/Fri-17Mar23
3.7G    Confluence/weekly/Fri-10Mar23
3.5G    Confluence/weekly/Fri-03Mar23
3.4G    Confluence/weekly/Fri-24Feb23

Jira daily Backups:
2.6G    Jira/daily/Wed-29Mar23
2.6G    Jira/daily/Tue-28Mar23
2.6G    Jira/daily/Mon-27Mar23
2.6G    Jira/daily/Sun-26Mar23
2.6G    Jira/daily/Sat-25Mar23
2.6G    Jira/daily/Fri-24Mar23
2.6G    Jira/daily/Thu-23Mar23

Jira weekly Backups:
2.6G    Jira/weekly/Fri-17Mar23
2.6G    Jira/weekly/Fri-10Mar23
2.6G    Jira/weekly/Fri-03Mar23
2.6G    Jira/weekly/Fri-24Feb23

GitLab daily Backups:
2.7G    GitLab/daily/Wed-29Mar23
2.7G    GitLab/daily/Tue-28Mar23
2.6G    GitLab/daily/Mon-27Mar23
2.6G    GitLab/daily/Sun-26Mar23
2.6G    GitLab/daily/Sat-25Mar23
2.6G    GitLab/daily/Fri-24Mar23
2.5G    GitLab/daily/Thu-23Mar23

GitLab weekly Backups:
2.3G    GitLab/weekly/Fri-17Mar23
2.2G    GitLab/weekly/Fri-10Mar23
2.1G    GitLab/weekly/Fri-03Mar23
2.2G    GitLab/weekly/Fri-24Feb23
Eridith
  • 23
  • 4
0

Any and all feedback is greatly appreciated!

Let's have a look how this goes ...

I am looking for a more efficient way to write this or to find a best practice to implement this.

Even if the other answers are working properly, provide results, fulfill the requirements, in my opinion there are more efficient ways to write this, depending on how one defines that and The Best Practice would be to use Custom Facts in order to decouple the creation of the backup status (files).


How to implement and use a Custom Fact?

I am referencing a simple minimal example of "How to implement and use a Custom Fact?" here only and instead of providing a full implementation. And one would need to utilize it in example with


A decoupled and distributed approach has some advantages above others.

Access a Remote Host via SSH and create a text file that lists all the backups on the Remote Host filesystem ...

In my opinion it is neither necessary nor a Best Practice to create facts about a Remote Node out of an Ansible playbook itself and during runtime. The Remote Node can run such scripts and generate the required data by hisself without depending on an Ansible Control Node or even Ansible at all. The Remote Node can do a lot of pre-processing and format the necessary data (output). Gathering the facts should just be enough that later

that can be inserted into an email body.

By doing this you achieve that your systems are loosely coupled.

No need to use the shell module, but still using Shell scripts which are completely separated from the Ansible scripts. They can be maintained separately, even by different persons or teams or be written in an other language. The provided output format can already be JSON and be better post-processed in Ansible.

Background Information

and

  • Ansible Best Pratices: Roles and Modules, p. 7

    THINK DECLARATIVELY. ... If you're trying to "write code" in your plays and roles, you're setting yourself up for failure. Our YAML-based playbooks were never meant to be for programming.

U880D
  • 8,601
  • 6
  • 24
  • 40