1

After about 20 years of intimate life with Linux I barely dare to ask such a basic question - is it somehow possible log everything what's happening when I run a process regarding working directories and process calls (with arguments)?

I have the following scenario: I run a build tool which runs processes which run processes and one of these produces an error. Even with environment variables and the usual --verbose argument I don't get enough information to manually 'replay' what's happening in order to even identify the misbehaving process.

I know set -x but as far as I know it only affects the current bash/sh instance, while the process I start runs a Python script which runs a Python script which would run configure at some place, maybe make, etc. At least it didn't help me much yet.

I also know strace which might be of some help but is there a way to extract process calls and working directories? In my experience it produces so much output that it's barely possible to find the spot I have to start greping around..

What I'm currently dreaming of would be something like

precord python3 buildme.py

which would give me at least something like

python3 buildme.py --verbose (/home/me/project/root)
├── /usr/bin/mkdir build (/home/me/project/root)
├── make all (/home/me/project/root/build)
│   ├── g++ -O2 -o bla.o bla.cpp
.   ├── ...
. 

(please ignore the fact that the process tree above would not make any sense - it's about the idea)

This question is not about solving only this riddle but I'm looking for a general approach.

I'm desperately hoping that this was one of the first Unix tools being invented for obvious reasons and somehow I just missed it until now..

frans
  • 8,868
  • 11
  • 58
  • 132
  • Something like [pstree](http://www.linfo.org/pstree.html)? – user1934428 Nov 30 '20 at 07:51
  • BTW, your question seems to me to belong to [superuser](https://superuser.com/) instead of StackOverflow. – user1934428 Nov 30 '20 at 07:52
  • @user1934428: `pstree` just gives me a snapshot and with no directories the processes have been started in. so it's no help to me, when the call I want to investigate has been terminated already – frans Nov 30 '20 at 07:59
  • Ah, right. Do you need just the initial working directory, when the process starts, or do you want to track any change of the working directory inside the process as well? – user1934428 Nov 30 '20 at 08:00
  • also, yes, you might be right regarding SuperUser, but I'm investigating in context of a build-chain as a developer. So if there is an approach which applies only to develoment environments, I'm ok, too. – frans Nov 30 '20 at 08:02
  • I need everything to manually playback what has happened - and for now this is every process call together with arguments and the working directory, since running e.g. `make` could be done in dozens of directories and I'd like to have an unambiguous call – frans Nov 30 '20 at 08:04
  • I do not know whether it is a universal property in Linux that the working directory is always reflected in the environment variable `PWD`, or whether this is just how bash (ans some other shells) are doing it, but if this is the case, you can for a process, where you know the PID, get the value of PWD by `strings /proc/$pid/environ | grep ^PWD=`. – user1934428 Nov 30 '20 at 08:07
  • For exactly playing back what was happening, this may not be enough. Parameters can be passed to a process either by its arguments or via environment variables, so for your goal to achieve, you would also have to monitor the full environment of the processes, and not just `PWD`. – user1934428 Nov 30 '20 at 08:12
  • as far as I know `/proc/$pid` is available as long as the process is running. If you run `./build.sh` and get s.th. like `execution of './configure' failed: returned exit code 3` it's of no help, I'm afraid. I then want to go to the directory `configure` was started in and re-run it with all command line arguments. Unfortunately even `strace` truncates command line arguments and strings (at least by default), which makes this approach even harder. – frans Nov 30 '20 at 08:30
  • 5
    `strace -f -o strace.out -e trace=process,chdir,fchdir ...` and then write a script to parse the output file? – pynexj Nov 30 '20 at 08:35
  • @frans : Correct. But after the processes are finished, everything is gone anyway. You could at best have another process running in parallel and collect this information, at the price that you may miss one subprocess or the other. Perhaps instead of searching for **general** solution of this problem, it might make more sense to focus on your concrete application case and instrument the involved parties (Makefile, Python programs etc) to write the information you need to some logfile. – user1934428 Nov 30 '20 at 08:40
  • `is it somehow possible log everything what's happening when I run a process regarding working directories and process calls (with arguments)?` Just `strace`. `is there a way to extract process calls and working directories?` Yes, just write that script. – KamilCuk Nov 30 '20 at 08:59
  • @pynex - `trace=chdir,fchdir` was what I was missing, adding `-v` and `-s 65536` turns it into something useful. If you add an answer I can accept it! – frans Nov 30 '20 at 10:16
  • let's keep it as a comment. it's far from the tree like output you want. :) – pynexj Nov 30 '20 at 12:11

2 Answers2

0

Sorry, no reputation with this account yet: https://rr-project.org/ should help as it can record program runs and trees of program runs and you can use rr replay with gdb interface/scripts or rr ps to analyze the trace.

God bless you and I hope you find an useful solution!

aivanov-atpco
  • 1
0

strace-process-tree now exists, which allows this:

$ strace -f -e trace=process -s 1024 -o /tmp/trace.out make binary-package
...

$ strace-process-tree /tmp/trace.out
25510 make binary-package
  ├─25511 /bin/sh -c 'dpkg-parsechangelog | awk '\''$1 == "Source:" { print $2 }'\'''
  │   ├─25512 dpkg-parsechangelog
  │   │   └─25514 tail -n 40 debian/changelog
  │   └─25513 awk '$1 == "Source:" { print $2 }'
  ├─25515 /bin/sh -c 'dpkg-parsechangelog | awk '\''$1 == "Version:" { print $2 }'\'''
  │   ├─25516 dpkg-parsechangelog
  │   │   └─25518 tail -n 40 debian/changelog
  │   └─25517 awk '$1 == "Version:" { print $2 }'
  ├─25519 /bin/sh -c 'dpkg-parsechangelog | grep ^Date: | cut -d: -f 2- | date --date="$(cat)" +%Y-%m-%d'
  │   ├─25520 dpkg-parsechangelog
  │   │   └─25525 tail -n 40 debian/changelog
  │   ├─25521 grep ^Date:
  │   ├─25522 cut -d: -f 2-
  │   └─25523 date --date=" Thu, 18 Jan 2018 23:39:51 +0200" +%Y-%m-%d
  │       └─25524 cat
  └─25526 /bin/sh -c 'dpkg-parsechangelog | awk '\''$1 == "Distribution:" { print $2 }'\'''
      ├─25527 dpkg-parsechangelog
      │   └─25529 tail -n 40 debian/changelog
      └─25528 awk '$1 == "Distribution:" { print $2 }'
Jan Tojnar
  • 5,306
  • 3
  • 29
  • 49
  • And there is apparently a [Perl script for this in strace repo](https://github.com/strace/strace/blob/master/src/strace-graph), as mentioned in https://stackoverflow.com/a/16572546/160386. But I did not manage to make it print argv. – Jan Tojnar May 24 '23 at 00:50
  • 1
    Funnily I'm working on something quite similar.. https://projects.om-office.de/frans/ttrace – frans May 25 '23 at 11:47