Bash trim or sed newline

Question

Have log file that has SOME new lines to remove (not all); have tried many ways of trim and sed with no luck (usually ends up one big line).

I think i need to have a specific search, when trimming or sedding, but not sure.

example file:

JOBV2|MISC: PURCHASE_DENIED|Purchasemode = sync ; iErrorCode = 992 ; terminalId = XXXXXXXXXXXX; offeringId = XXXXXX; rentalType = ps ; price = 1649 ; rentalWindowHrs = 0 ; IsAdult = False ; IsRedeemTitle = False ; TransactionID = IsUG=N;TId=X;VID=XXXX;Price=16.49;Pid=0;;CP=0 ; RedeemPoints =  ; PointsRatio =  ; TotalMRPoints =  ; NewTransactionID = IsUG=N;TId=XXXXXXXXXXXX;VID=XXXX;Pid=0;RetPrice=16.49;DisPrice=0.00;NetPrice=16.49;MRPoints=0;P2DVal=0;TotalPoints=0; ; |2015-10-07 00:00:14.320
JOBV2|MISC: Non-Purchase_Denied Error string: BILLING SYSTEM IS DOWN|^@|2015-10-07 00:02:13.457
JOBV3|MISC: PURCHASE FAILED: PURCHASE_DENIED   at STRING.JOBV2(PurchaseParams objPurchaseParams)
   at STRING.JOBV3(String strInputParams)|strInputParams = strUnitAddress=XXXXXXXXXXXX,terminalid=XXXXXXXXXXXX,offeringid=XXXXXX,rentalType=PS,price=1649,rentalWindowHrs=0,IsAdult=false,IsRedeemTitle=false,promotionType=,isUpgrade=N,catalogprice=0,strpromotionid=0; ; |2015-10-07 00:00:14.320
JOBV2|MISC: Non-Purchase_Denied Error string: BILLING SYSTEM IS DOWN|^@|2015-10-07 00:02:13.457
JOBV3|MISC: PURCHASE FAILED: PURCHASE_DENIED   at STRING.JOBV2(PurchaseParams objPurchaseParams)
   at STRING.JOBV3(String strInputParams)|strInputParams = strUnitAddress=XXXXXXXXXXXX,terminalid=XXXXXXXXXXXX,offeringid=XXXXXX,rentalType=PS,price=1649,rentalWindowHrs=0,IsAdult=false,IsRedeemTitle=false,promotionType=,isUpgrade=N,catalogprice=0,strpromotionid=0; ; |2015-10-07 00:00:14.320
JOBV2|MISC: PURCHASE_DENIED|Purchasemode = sync ; iErrorCode = 992 ; terminalId = XXXXXXXXXXXX ; offeringId = XXXXXX; rentalType = ps ; price = 1649 ; rentalWindowHrs = 0 ; IsAdult = False ; IsRedeemTitle = False ; TransactionID = IsUG=N;TId=XXXXXXXXXXXX;VID=XXXX;Price=16.49;Pid=0;;CP=0 ; RedeemPoints =  ; PointsRatio =  ; TotalMRPoints =  ; NewTransactionID = IsUG=N;TId=XXXXXXXXXXXX:VID=XXXX;Pid=0;RetPrice=16.49;DisPrice=0.00;NetPrice=16.49;MRPoints=0;P2DVal=0;TotalPoints=0; ; |2015-10-07 00:00:09.313
JOBV2|MISC: Non-Purchase_Denied Error string: BILLING SYSTEM IS DOWN|^@|2015-10-07 00:02:13.457

Need out:

JOBV2|MISC: PURCHASE_DENIED|Purchasemode = sync ; iErrorCode = 992 ; terminalId = XXXXXXXXXXXX; offeringId = XXXXXX; rentalType = ps ; price = 1649 ; rentalWindowHrs = 0 ; IsAdult = False ; IsRedeemTitle = False ; TransactionID = IsUG=N;TId=X;VID=XXXX;Price=16.49;Pid=0;;CP=0 ; RedeemPoints =  ; PointsRatio =  ; TotalMRPoints =  ; NewTransactionID = IsUG=N;TId=XXXXXXXXXXXX;VID=XXXX;Pid=0;RetPrice=16.49;DisPrice=0.00;NetPrice=16.49;MRPoints=0;P2DVal=0;TotalPoints=0; ; |2015-10-07 00:00:14.320
JOBV2|MISC: Non-Purchase_Denied Error string: BILLING SYSTEM IS DOWN|^@|2015-10-07 00:02:13.457
JOBV3|MISC: PURCHASE FAILED: PURCHASE_DENIED   at STRING.JOBV2(PurchaseParams objPurchaseParams) at STRING.JOBV3(String strInputParams)|strInputParams = strUnitAddress=XXXXXXXXXXXX,terminalid=XXXXXXXXXXXX,offeringid=XXXXXX,rentalType=PS,price=1649,rentalWindowHrs=0,IsAdult=false,IsRedeemTitle=false,promotionType=,isUpgrade=N,catalogprice=0,strpromotionid=0; ; |2015-10-07 00:00:14.320
JOBV2|MISC: Non-Purchase_Denied Error string: BILLING SYSTEM IS DOWN|^@|2015-10-07 00:02:13.457
JOBV3|MISC: PURCHASE FAILED: PURCHASE_DENIED   at STRING.JOBV2(PurchaseParams objPurchaseParams) at STRING.JOBV3(String strInputParams)|strInputParams = strUnitAddress=XXXXXXXXXXXX,terminalid=XXXXXXXXXXXX,offeringid=XXXXXX,rentalType=PS,price=1649,rentalWindowHrs=0,IsAdult=false,IsRedeemTitle=false,promotionType=,isUpgrade=N,catalogprice=0,strpromotionid=0; ; |2015-10-07 00:00:14.320
JOBV2|MISC: PURCHASE_DENIED|Purchasemode = sync ; iErrorCode = 992 ; terminalId = XXXXXXXXXXXX ; offeringId = XXXXXX; rentalType = ps ; price = 1649 ; rentalWindowHrs = 0 ; IsAdult = False ; IsRedeemTitle = False ; TransactionID = IsUG=N;TId=XXXXXXXXXXXX;VID=XXXX;Price=16.49;Pid=0;;CP=0 ; RedeemPoints =  ; PointsRatio =  ; TotalMRPoints =  ; NewTransactionID = IsUG=N;TId=XXXXXXXXXXXX:VID=XXXX;Pid=0;RetPrice=16.49;DisPrice=0.00;NetPrice=16.49;MRPoints=0;P2DVal=0;TotalPoints=0; ; |2015-10-07 00:00:09.313
JOBV2|MISC: Non-Purchase_Denied Error string: BILLING SYSTEM IS DOWN|^@|2015-10-07 00:02:13.457

Any help greatly apprecaited!

Do you just need to remove the newline on the end of any line that ends with a `)`? — Etan Reisner, Oct 13 '15 at 17:14
Or perhaps join any two lines where the second line starts with whitespace? — Graham, Oct 13 '15 at 19:08

jayant · Accepted Answer · 2015-10-13T19:03:41.403

You can do this with awk:

awk 'BEGIN {ORS=""} /^ /{ print $0; next;} {print "\n"$0;}' input_file | sed 's/\r//g'

which gives

JOBV2|MISC: PURCHASE_DENIED|Puchasemode = sync ; iEoCode = 992 ; teminalId = XXXXXXXXXXXX; offeingId = XXXXXX; entalType = ps ; pice = 1649 ; entalWindowHs = 0 ; IsAdult = False ; IsRedeemTitle = False ; TansactionID = IsUG=N;TId=X;VID=XXXX;Pice=16.49;Pid=0;;CP=0 ; RedeemPoints =  ; PointsRatio =  ; TotalMRPoints =  ; NewTansactionID = IsUG=N;TId=XXXXXXXXXXXX;VID=XXXX;Pid=0;RetPice=16.49;DisPice=0.00;NetPice=16.49;MRPoints=0;P2DVal=0;TotalPoints=0; ; |2015-10-07 00:00:14.320
JOBV2|MISC: Non-Puchase_Denied Eo sting: BILLING SYSTEM IS DOWN|^@|2015-10-07 00:02:13.457
JOBV3|MISC: PURCHASE FAILED: PURCHASE_DENIED   at STRING.JOBV2(PuchasePaams objPuchasePaams)   at STRING.JOBV3(Sting stInputPaams)|stInputPaams = stUnitAddess=XXXXXXXXXXXX,teminalid=XXXXXXXXXXXX,offeingid=XXXXXX,entalType=PS,pice=1649,entalWindowHs=0,IsAdult=false,IsRedeemTitle=false,pomotionType=,isUpgade=N,catalogpice=0,stpomotionid=0; ; |2015-10-07 00:00:14.320
JOBV2|MISC: Non-Puchase_Denied Eo sting: BILLING SYSTEM IS DOWN|^@|2015-10-07 00:02:13.457
JOBV3|MISC: PURCHASE FAILED: PURCHASE_DENIED   at STRING.JOBV2(PuchasePaams objPuchasePaams)   at STRING.JOBV3(Sting stInputPaams)|stInputPaams = stUnitAddess=XXXXXXXXXXXX,teminalid=XXXXXXXXXXXX,offeingid=XXXXXX,entalType=PS,pice=1649,entalWindowHs=0,IsAdult=false,IsRedeemTitle=false,pomotionType=,isUpgade=N,catalogpice=0,stpomotionid=0; ; |2015-10-07 00:00:14.320
JOBV2|MISC: PURCHASE_DENIED|Puchasemode = sync ; iEoCode = 992 ; teminalId = XXXXXXXXXXXX ; offeingId = XXXXXX; entalType = ps ; pice = 1649 ; entalWindowHs = 0 ; IsAdult = False ; IsRedeemTitle = False ; TansactionID = IsUG=N;TId=XXXXXXXXXXXX;VID=XXXX;Pice=16.49;Pid=0;;CP=0 ; RedeemPoints =  ; PointsRatio =  ; TotalMRPoints =  ; NewTansactionID = IsUG=N;TId=XXXXXXXXXXXX:VID=XXXX;Pid=0;RetPice=16.49;DisPice=0.00;NetPice=16.49;MRPoints=0;P2DVal=0;TotalPoints=0; ; |2015-10-07 00:00:09.313
JOBV2|MISC: Non-Puchase_Denied Eo sting: BILLING SYSTEM IS DOWN|^@|2015-10-07 00:02:13.457

Alternative method:

awk 'BEGIN {ORS=""} /\)$/{print $0; next;} {print $0"\n";}' input_file | sed 's/\r//g'

Thank you jayant; I have over simplified the sample data originally, which what you provided grabbed some of the non-new line lines....my apologies. I updated the sample file data. — kwanx, Oct 13 '15 at 18:08
Very close jayant - When I look at the file in VI, they show ^M where the two lines were joined; which cat/head are interpreting as line breaks: JOBV3|MISC: PURCHASE FAILED: PURCHASE_DENIED at STRING.JOBV2(PurchaseParams objPurchaseParams)^M at STRING.JOBV3(String strInputParams)|strInputParams = strUnitAddress=XXXXXXXXXXXX,terminalid=XXXXXXXXXXXX,offeringid=XXXXXX,rentalType=PS,price=1649,rentalWindowHrs=0,IsAdult=false,IsRedeemTitle=false,promotionType=,isUpgrade=N,catalogprice=0,strpromotionid=0; ; |2015-10-07 00:00:14.320^M — kwanx, Oct 13 '15 at 18:46
Which platform are you on? Does it work better if you replace the `\n` in the command with `\r\n`? Otherwise can you try [this](http://stackoverflow.com/a/5843561/1517864) to the output of the awk? — jayant, Oct 13 '15 at 18:52
I've updated my answer with an alternative solution and a sed command to take care of the ^M — jayant, Oct 13 '15 at 19:04
Thank you jayant, this one seems to be the winner: awk 'BEGIN {ORS=""} /^ /{ print $0; next;} {print "\n"$0;}' input_file | sed 's/\r//g'.....I will test more today en masse. — kwanx, Oct 15 '15 at 14:18
This ran automagically last night and everything is perfect this morning, thank you! awk 'BEGIN {ORS=""} /^ /{ print $0; next;} {print "\n"$0;}' input_file | sed 's/\r//g' — kwanx, Oct 16 '15 at 13:15

Jose Ricardo Bustos M. · Answer 2 · 2015-10-13T18:43:11.840

Another solution, you can use sed

sed -r ':a;N;$!ba;s/\n\s+/ /g' file

or, without to use extended regular expressions option -r

sed ':a;N;$!ba;s/\n[ \t][ \t]*/ /g' file

you get

JOBV2|MISC: PURCHASE_DENIED|Purchasemode = sync ; iErrorCode = 992 ; terminalId = XXXXXXXXXXXX; offeringId = XXXXXX; rentalType = ps ; price = 1649 ; rentalWindowHrs = 0 ; IsAdult = False ; IsRedeemTitle = False ; TransactionID = IsUG=N;TId=X;VID=XXXX;Price=16.49;Pid=0;;CP=0 ; RedeemPoints =  ; PointsRatio =  ; TotalMRPoints =  ; NewTransactionID = IsUG=N;TId=XXXXXXXXXXXX;VID=XXXX;Pid=0;RetPrice=16.49;DisPrice=0.00;NetPrice=16.49;MRPoints=0;P2DVal=0;TotalPoints=0; ; |2015-10-07 00:00:14.320
JOBV2|MISC: Non-Purchase_Denied Error string: BILLING SYSTEM IS DOWN|^@|2015-10-07 00:02:13.457
JOBV3|MISC: PURCHASE FAILED: PURCHASE_DENIED   at STRING.JOBV2(PurchaseParams objPurchaseParams) at STRING.JOBV3(String strInputParams)|strInputParams = strUnitAddress=XXXXXXXXXXXX,terminalid=XXXXXXXXXXXX,offeringid=XXXXXX,rentalType=PS,price=1649,rentalWindowHrs=0,IsAdult=false,IsRedeemTitle=false,promotionType=,isUpgrade=N,catalogprice=0,strpromotionid=0; ; |2015-10-07 00:00:14.320
JOBV2|MISC: Non-Purchase_Denied Error string: BILLING SYSTEM IS DOWN|^@|2015-10-07 00:02:13.457
JOBV3|MISC: PURCHASE FAILED: PURCHASE_DENIED   at STRING.JOBV2(PurchaseParams objPurchaseParams) at STRING.JOBV3(String strInputParams)|strInputParams = strUnitAddress=XXXXXXXXXXXX,terminalid=XXXXXXXXXXXX,offeringid=XXXXXX,rentalType=PS,price=1649,rentalWindowHrs=0,IsAdult=false,IsRedeemTitle=false,promotionType=,isUpgrade=N,catalogprice=0,strpromotionid=0; ; |2015-10-07 00:00:14.320
JOBV2|MISC: PURCHASE_DENIED|Purchasemode = sync ; iErrorCode = 992 ; terminalId = XXXXXXXXXXXX ; offeringId = XXXXXX; rentalType = ps ; price = 1649 ; rentalWindowHrs = 0 ; IsAdult = False ; IsRedeemTitle = False ; TransactionID = IsUG=N;TId=XXXXXXXXXXXX;VID=XXXX;Price=16.49;Pid=0;;CP=0 ; RedeemPoints =  ; PointsRatio =  ; TotalMRPoints =  ; NewTransactionID = IsUG=N;TId=XXXXXXXXXXXX:VID=XXXX;Pid=0;RetPrice=16.49;DisPrice=0.00;NetPrice=16.49;MRPoints=0;P2DVal=0;TotalPoints=0; ; |2015-10-07 00:00:09.313
JOBV2|MISC: Non-Purchase_Denied Error string: BILLING SYSTEM IS DOWN|^@|2015-10-07 00:02:13.457

Thank you Jose; I tried both methods you suggested. sed ':a;N;$!ba;s/\n[ \t][ \t]*/ /g' file.txt > out2.txt When vi out2.txt, I can see "^M" where the lines were joined. When head/Cat out2.txt, this ^M is interpreted as line break and does not show the lines joined on the screen. — kwanx, Oct 15 '15 at 14:11

score 0 · Answer 3 · answered Oct 13 '15 at 19:18

0

If what you're doing is joining lines that start with whitespace to the previous line, you could do this in bash.

last=""
while IFS="" read line; do
  if [[ "$line" =~ ^[[:space:]] ]]; then
    echo -n "$last "
  else
    echo "$last"
  fi
  last="$line"
done < inp
echo "$line"

Or even in POSIX (non-bash) shell:

last=""
while IFS="" read line; do
  if expr "$line" : '[[:space:]]' >/dev/null; then
    echo -n "$last "
  else
    echo "$last"
  fi
  last="$line"
done < inp
echo "$line"

answered Oct 13 '15 at 19:18

Graham

1,631
14
23

1

On the first, I would suggest `read -r`, on both I would suggest `printf` instead of `echo`, on the last I would suggest `>/dev/null 2>&1` (if you really want to mask all potential complaints). Kudos for thinking about both solutions. – David C. Rankin Oct 13 '15 at 19:52

Bash trim or sed newline

3 Answers3