Not working wait_for in starting JBoss server via Ansible based on data from logs

Question

I tried to start JBoss service via Ansible and use wai_for module to wait till the JBoss started. However, the JBoss service started, however, the wait_for still progressing until there is time out then hit error. Below is my code

- name: Get the contents of the last line from log
  command: "tail -n 1 /home/nityo/application.log"
  register: tail_output

- name: Create a variable with a meaningful name, just for clarity
  set_fact:
    last_line_of_the_log_file: "{{ tail_output.stdout }}"

- name: JBoss service starting
  service:
    name: "jboss.service"
    state: started
  become: yes
  become_user: root

- name:Wait for server started
  wait_for:
    path: "//home/nityo/application.log"
    search_regex: "{{ last_line_of_the_log_file }}\r(.*\r)*.*JBoss EAP.*started.*"
    timeout: 600

Besides that can we combine all thsi into 1 task rrather than having multiple task

Sample output log

2020-10-11 01:13:42,009 INFO  [org.jboss.as] JBoss EAP 7.2 (WildFly Core) running in 100281ms - service to be running
2020-10-11 01:13:42,005 INFO  [org.jboss.as] processing data
2020-10-11 01:13:43,009 INFO  [org.jboss.as] JBoss EAP 7.2 (WildFly Core) stopped in 100281ms - service to be stopped
-
-
-
-
-
-
-
2020-10-11 01:13:48,009 INFO  [org.jboss.as] JBoss EAP 7.2 (WildFly Core) started in 100281ms - service to be started

Answer 1

It will never match in the last task, because you are searching for the last logfile line plus some extra stuff. You are using a variable in your regex, which contains a string that never matches if you add some other staff at the end of it.

So skip the first and second task and change your search_regex Parameter of the last task.

You could use the ansible fact ansible_date_time (described in Ansible date variable) and use parts of date, time and/or epoch-seconds to build your regex via set_fact module. So your regex will match the logfile line with a timestamp not older then 10 minutes.

With date you could create a timestamp for a specific hour. The playbook could look like this:

---
- name: check logfile last 10 Minutes
  hosts:
  - localhost
  gather_facts: no
  tasks:
  - name: set now timestring
    command: 'date +"%Y-%m-%d %H"'
    register: date_now
  - name: set_now
    set_fact:
      now: "{{ date_now.stdout }}"
  - name: set 10 minutes ago timestring
    command: 'date +"%Y-%m-%d %H" -d "10 minutes ago"'
    register: date_10min_ago
  - name: set 10min_ago
    set_fact:
      ten_min_ago: "{{ date_10min_ago.stdout }}"
  - name: debug
    debug:
      msg: "({{ now }}|{{ ten_min_ago }}).*JBoss EAP.*started.*"
  - name: Wait for server started
    wait_for:
      path: "//home/nityo/application.log"
      search_regex: "({{ now }}|{{ ten_min_ago }}).*JBoss EAP.*started"
      timeout: 600

The Timestamps are totally different if the daliy limit is reached. In the middle of an hour, it seems superfluous. Debug-Output how the regex looks like this:

TASK [debug] **********************
ok: [localhost] => {
    "msg": "(2020-10-12 08|2020-10-12 08).*JBoss EAP.*started.*"
}

Answer 2

What you are storing in last_line_of_the_log_file variable might not be getting matched at the default interval in which wait_for module checks by the time it reaches this task.

Checking service status by parsing a log file can be tricky. However I think the cleanest way will be to match the pattern JBoss EAP.*started.* in the log file. However for this to work reliably, it would be better to start with an empty log file before starting the service.

# Backup the log file with date/time stamp
- shell: cat /home/nityo/application.log >> /home/nityo/application-$(date %b-%d-%H-%M).log

# Empty the file contents before starting service
- command: truncate -s 0 /home/nityo/application.log

- wait_for:
    path: "/home/nityo/application.log"
    search_regex: "JBoss EAP.*started.*"
    timeout: 600

Parsing log files can be hit-or-miss if the logs generate differently in a given span of time. You can tweak the time, retries, and search pattern to get more consistent results.

Update:

It would be good to wait for some time to roll beyond the default lines (10) fetched by tail command, to avoid matching a previous startup. Then use egrep:

# Pause and wait for logs to roll beyond 10 lines fetched by 'tail'
- pause:
    seconds: 30

- shell: tail /home/nityo/application.log | grep -e "JBoss EAP.*started"
  register: file_tail
  until: file_tail is success
  # use appropriate values as per the rate of logging
  retries: 30
  delay: 20