Trace of executed programs called by a Bash script

Question

A script is misbehaving. I need to know who calls that script, and who calls the calling script, and so on, only by modifying the misbehaving script.

This is similar to a stack-trace, but I am not interested in a call stack of function calls within a single bash script. Instead, I need the chain of executed programs/scripts that is initiated by my script.

Answer 1

A simple script I wrote some days ago...

# FILE       : sctrace.sh
# LICENSE    : GPL v2.0 (only)
# PURPOSE    : print the recursive callers' list for a script
#              (sort of a process backtrace)
# USAGE      : [in a script] source sctrace.sh
#
# TESTED ON  :
# - Linux, x86 32-bit, Bash 3.2.39(1)-release

# REFERENCES:
# [1]: http://tldp.org/LDP/abs/html/internalvariables.html#PROCCID
# [2]: http://linux.die.net/man/5/proc
# [3]: http://linux.about.com/library/cmd/blcmdl1_tac.htm

#! /bin/bash

TRACE=""
CP=$$ # PID of the script itself [1]

while true # safe because "all starts with init..."
do
        CMDLINE=$(cat /proc/$CP/cmdline)
        PP=$(grep PPid /proc/$CP/status | awk '{ print $2; }') # [2]
        TRACE="$TRACE [$CP]:$CMDLINE\n"
        if [ "$CP" == "1" ]; then # we reach 'init' [PID 1] => backtrace end
                break
        fi
        CP=$PP
done
echo "Backtrace of '$0'"
echo -en "$TRACE" | tac | grep -n ":" # using tac to "print in reverse" [3]

... and a simple test.

I hope you like it.

Answer 2

You can use Bash Debugger http://bashdb.sourceforge.net/

Or, as mentioned in the previous comments, the caller bash built-in. See: http://wiki.bash-hackers.org/commands/builtin/caller

i=0; while caller $i ;do ((i++)) ;done

Or as a bash function:

dump_stack(){
    local i=0
    local line_no
    local function_name
    local file_name
    while caller $i ;do ((i++)) ;done | while read line_no function_name file_name;do echo -e "\t$file_name:$line_no\t$function_name" ;done >&2
}

Another way to do it is to change PS4 and enable xtrace:

PS4='+$(date "+%F %T") ${FUNCNAME[0]}() $BASH_SOURCE:${BASH_LINENO[0]}+ '
set -o xtrace    # Comment this line to disable tracing.

Answer 3

~$ help caller
caller: caller [EXPR]
    Returns the context of the current subroutine call.

    Without EXPR, returns "$line $filename".  With EXPR,
    returns "$line $subroutine $filename"; this extra information
    can be used to provide a stack trace.

    The value of EXPR indicates how many call frames to go back before the
    current one; the top frame is frame 0.

Answer 4

Since you say you can edit the script itself, simply put a:

ps -ef >/tmp/bash_stack_trace.$$

in it, where the problem is occurring.

This will create a number of files in your tmp directory that show the entire process list at the time it happened.

You can then work out which process called which other process by examining this output. This can either be done manually, or automated with something like awk, since the output is regular - you just use those PID and PPID columns to work out the relationships between all the processes you're interested in.

You'll need to keep an eye on the files, since you'll get one per process so they may have to be managed. Since this is something that should only be done during debugging, most of the time that line will be commented out (preceded by #), so the files won't be created.

To clean them up, you can simply do:

rm /tmp/bash_stack_trace.*

Answer 5

UPDATE: The code below should work. Now I have a newer answer with a newer code version that allows a message inserted in the stacktrace.

IIRC I just couldn't find this answer to update it as well at the time. But now decided code is better kept in git so latest version of the above should be in this gist.

original code-corrected answer below:

There was another answer about this somewhere but here is a function to use for getting stack trace in the sense used for example in the java programming language. You call the function and it puts the stack trace into the variable $STACK. It show the code points that led to get_stack being called. This is mostly useful for complicated execution where single shell sources multiple script snippets and nesting.

function get_stack () {
   STACK=""
   # to avoid noise we start with 1 to skip get_stack caller
   local i
   local stack_size=${#FUNCNAME[@]}
   for (( i=1; i<$stack_size ; i++ )); do
      local func="${FUNCNAME[$i]}"
      [ x$func = x ] && func=MAIN
      local linen="${BASH_LINENO[(( i - 1 ))]}"
      local src="${BASH_SOURCE[$i]}"
      [ x"$src" = x ] && src=non_file_source

      STACK+=$'\n'"   "$func" "$src" "$linen
   done
}

Answer 6

The simplest script which returns a stack trace with all callers:

i=0; while caller $i ;do ((i++)) ;done

Answer 7

adding pstree -p -u `whoami` >>output in your script will probably get you the information you need.

Answer 8

You could try something like

strace -f -e execve script.sh