Using 'sed' to remove newlines for specific characters

Question

I know this question has been asked and answered in different variations. But mine focuses on why sed is not behaving as I would expect vi does.

For a given threaddump file, I need to remove the newlines before every line that is " Locked ownable synchronizers" as shown below.

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
    at com.project.tools.threads.NamedThread.run(NamedThread.java:37)

   Locked ownable synchronizers:
        - None

I can do this using vi:

:g/^M   Locked ownable synchronizers/s//   Locked ownable synchronizers/g

^^^ the ^M is ctrl-M. The above vi command works, ie, it successfully removes the newline before Locked. However, when I try to used it in sed, none of the following work (I tried multiple ways to represent the newline character but none worked).

sed -i'' -e 's/^M   Locked ownable synchronizers/   Locked ownable synchronizers/g' file.threaddump
sed -i'' -e 's/\n   Locked ownable synchronizers/   Locked ownable synchronizers/g' file.threaddump
sed -i'' -e 's/\r   Locked ownable synchronizers/   Locked ownable synchronizers/g' file.threaddump
sed -i'' -e 's/\r\n   Locked ownable synchronizers/   Locked ownable synchronizers/g' file.threaddump

As I understand, vi commands work in sed (and they have been). Why doesn't this one work????

Thank you

PS: The solution that worked was using perl:

perl -0pe 's/\n   Locked ownable synchronizers:/   Locked ownable synchronizers:/g' < file.threaddump

but I want to figure out why the sed didn't work!

weibeld · Accepted Answer · 2017-06-26T12:06:27.123

First of all, a simpler ex command which works in vim is:

:%s/\n\(   Locked\)/\1/

Now, it's not straightforward to replace newlines with sed, because sed reads its input line by line, and, for sed, a line by itself doesn't contain a newline character (the newline characters are just the separators between lines). So, a sed pattern including \n will not match anything by default.

The question of replacing newlines with sed has been asked before, and we can adopt this answer to your case:

sed -e :a -e N -e '$!ba' -e 's/\n\(   Locked\)/\1/' file.threaddump

The above solution is quite complicated for such a simple task, because it tries to make sed do something it is not meant to do, namely matching newline characters.

The moral of the story is: choose the right tool for the job.

In this case, a better tool is one which allows to redefine the record separator so that the newline character appears as an ordinary character in the string, and is not treated as the record separator.

Sed doesn't allow to do this, as it is specifically designed to process lines and the newline character is hardcoded as the record separator.

However, as you have already seen, Perl allows to do this with the -0 switch:

perl -0 -p -e 's/\n(   Locked)/$1/' file.threaddump

The -0 switch (without arguments) basically sets the record separator to the empty string, which results in treating the entire input as a single record. You can then match the newline character \n like any other character in the s/// command.

Note:

In case you want to remove the carriage return \r (U+000D) instead of the newline \n (U+000A), you should be able to replace the \n in above code by \r.

score 1 · Answer 2 · answered Jun 25 '17 at 11:27

sed reads and operates on 1 line at a time. Lines are separated by newlines. Therefore you can't remove a newline from the line being operated on by sed because it doesn't contain a newline.

Since sed is only for simple substitutions on individual lines you should be using awk instead anyway:

awk -i inplace -v RS='^$' '{print gensub(/\n(\s*Locked ownable synchronizers)/,"\\1","g")}' file.threaddump

The above uses GNU awk instead of GNU sed for inplace editing and other syntactic sugar.

Your explanation makes sense and I have a better understanding of sed.Thank you! — mbsf, Jun 26 '17 at 03:49

score 0 · Answer 3 · answered Jun 25 '17 at 11:53

This might work for you (GNU sed):

sed -i ':a;N;s/\n\(\s*Locked ownable synchronizers:\)/\1/;ta;P;D' file

Append a second line to the first and if that line begins with the required string, remove the preceeding newline and repeat. Otherwise print the first line, delete it and repeat.

Using 'sed' to remove newlines for specific characters

3 Answers3