5

I'd like to use sed or any command line tool to replace parts of lines by the output of shell commands. For example:

  • Replace linux epochs by human-readable timestamps, by calling date
  • Replace hexa dumps of a specific protocol packets by their decoded counterparts, by calling an in-house decoder

sed seems best fitted because it allows to match patterns and reformat other things too, like moving bits of matches around, but is not mandatory.

Here is a simplified example:

echo "timestamp = 1234567890" | sed "s/timestamp = \(.*\)/timestamp = $(date -u --d @\1 "+%Y-%m-%d %T")/g"

Of course, the $(...) thing does not work. As far as I understand, that's for environment variables.

So what would the proper syntax be? Is sed recommended in this case ? I've spent several hours searching... Is sed even capable of this ? Are there other tools better suited?

Edit

I need...

  • Pattern matching. The log is full of other things, so I need to be able to pinpoint the strings I want to replace based on context (text before and after, on the same line). This excludes column-position-based matching like awk '{$3...
  • In-place replacement, so that the reste of the line, "Timestamp = " or whatever, remains unchanged. This exclused sed's 'e' command.
Gabriel
  • 2,841
  • 4
  • 33
  • 43
  • 1
    Awk, Perl or, Python would probably be better suited if you have more than just a few lines of `sed`. They are more versatile and much more readable. – tripleee Jan 07 '15 at 16:37
  • 1
    read this: http://stackoverflow.com/questions/14102504/replace-strings-with-evaluated-string-based-on-matched-group-elegant-way-not-u – Kent Jan 07 '15 at 21:25

2 Answers2

4

To run an external command in sed you need to use e. See an example:

$ echo "timestamp = 1234567890" | sed "s#timestamp = \(.*\)#date -u --d @\1 "\+\%Y"#e"
2009

With the full format:

$ sed "s#timestamp = \(.*\)#echo timestamp; date -u --d @\1 '\+\%Y-\%m-\%d \%T'#e" <<< "timestamp = 1234567890"
timestamp
2009-02-13 23:31:30

This catches the timestamp and converts it into +%Y format.

From man sed:

e

This command allows one to pipe input from a shell command into pattern space. If a substitution was made, the command that is found in pattern space is executed and pattern space is replaced with its output. A trailing newline is suppressed; results are undefined if the command to be executed contains a nul character. This is a GNU sed extension.

However, you see it is a bit "ugly". Depending on what you want to do, you'd better use a regular while loop to fetch the values and then use date normally. For example, if the file is like:

timestamp = 1234567890

Then you can say:

while IFS="=" read -r a b
do
  echo "$b"
done < file

this will make you have $b as the timestamp and then you can perform a date ....

fedorqui
  • 275,237
  • 103
  • 548
  • 598
  • wrong documentation reference, use https://www.gnu.org/software/sed/manual/sed.html#The-_0022s_0022-Command instead – glenn jackman Jan 07 '15 at 18:50
  • 2
    The problem is, the whole line is replaced by the output of the command. I'd like just the timestamp to be replaced (or the hexa by its decoded value), as sed usually does. – Gabriel Jan 08 '15 at 14:54
  • Perfect, Thanks! I had to adapt it only little to cover only lines with timestamps, e.g. #1606748699: cat ~/.bash_history | sed "s/^#\([0-9]\+\)$/echo -n '#'; date -u --d @\1 '\+\%Y-\%m-\%d \%T'/e" | less – stefan123t Dec 14 '20 at 13:37
2

As commented, use a language with built-in time functions. For example:

$ echo "timestamp = 1234567890" | gawk '{$3 = strftime("%F %T", $3)} 1'
timestamp = 2009-02-13 18:31:30

$ echo "timestamp = 1234567890" | perl -MTime::Piece -pe 's/(\d+)/ localtime($1)->strftime("%F %T") /e'
timestamp = 2009-02-13 18:31:30
glenn jackman
  • 238,783
  • 38
  • 220
  • 352