Jenkins pipeline failing for sed command

Question

I'm running a sed command in pipeline but it is failing with an error

sed: file - line 2: unterminated `s' command

here is my code for sed in pipeline:

sh '''sed '/^[[:blank:]]*$/d;s|^|s/%%|;s|:|%%/|;s|$|/|' key.txt | sed -f - file1.txt > file2.txt'''

I if I will run only first sed command, Jenkins gives very strange output, here it is from Jenkins logs its adding extra lines with characters:

+ sed '/^[[:blank:]]*$/d;s|^|s@%%|;s|:|%%@|;s|$|@|' keys.txt
s@%%sql_server_name%%@test_seqserver_1234
@
s@%%
@
s@%%sql_login_name%%@test_login_name
@
s@%%
@
s@%%password%%@test_password 
@
s@%%
@

If run following command this is what Jenkins output looks like

sh '''sed '/^[[:blank:]]*$/d;/:/!d;s|^|s/%%|;s|:|%%/|;s|$|/|' 
keys.txt'''

Jenkins output:

+ sed '/^[[:blank:]]*$/d;/:/!d;s|^|s/%%|;s|:|%%/|;s|$|/|' keys.txt
s/%%sql_server_name%%/test_seqserver_1234
/
s/%%sql_login_name%%/test_login_name
/
s/%%password%%/test_password 
/
s/%%SID%%/123456 
/

I ran the new command:

sh '''sed -e '/:/!d;s|^\\([^:]*\\):\\(.*\\)$|s/%%\\1%%/\\2/|' -e 'N;s|\\n/|/|' keys.txt'''

Here is the output:

Running shell script
+ sed -e '/:/!d;s|^\([^:]*\):\(.*\)$|s/%%\1%%/\2/|' -e 'N;s|\n/|/|' 
keys.txt
s/%%sql_server_name%%/test_seqserver_1234
/

s/%%sql_login_name%%/test_login_name
/

s/%%password%%/test_password 
/

s/%%SID%%/123456

Here is xxd output for text file:

Running shell script
+ xxd keys.txt
0000000: 7371 6c5f 7365 7276 6572 5f6e 616d 653a  sql_server_name:
0000010: 7465 7374 5f73 6571 7365 7276 6572 5f31  test_seqserver_1
0000020: 3233 340d 0a0d 0a73 716c 5f6c 6f67 696e  234....sql_login
0000030: 5f6e 616d 653a 7465 7374 5f6c 6f67 696e  _name:test_login
0000040: 5f6e 616d 650d 0a0d 0a70 6173 7377 6f72  _name....passwor
0000050: 643a 7465 7374 5f70 6173 7377 6f72 6420  d:test_password 
0000060: 0d0a 0d0a 5349 443a 3132 3334 3536 200d  ....SID:123456 .
0000070: 0a0d 0a64 6566 6175 6c74 5f64 6174 6162  ...default_datab
0000080: 6173 653a 7465 6d70 6462 0d0a 0d0a 6465  ase:tempdb....de
0000090: 6661 756c 745f 6c61 6e67 7561 6765 3a75  fault_language:u
00000a0: 735f 656e 676c 6973 680d 0a0d 0a63 6865  s_english....che
00000b0: 636b 5f65 7870 6972 6174 696f 6e3a 4f46  ck_expiration:OF
00000c0: 460d 0a0d 0a63 6865 636b 5f70 6f6c 6963  F....check_polic
00000d0: 793a 4f46 460d 0a0d 0a64 656c 6976 6572  y:OFF....deliver
00000e0: 7974 7970 653a 7363 6865 6475 6c65 640d  ytype:scheduled.
00000f0: 0a0d 0a73 6368 6564 756c 6564 5f64 656c  ...scheduled_del
0000100: 6976 6572 7964 6174 653a 3035 2d33 302d  iverydate:05-30-
0000110: 3230 3939 0d0a 0d0a 7363 6865 6475 6c65  2099....schedule
0000120: 645f 6465 6c69 7665 7279 5f32 3468 725f  d_delivery_24hr_
0000130: 6365 6e74 7261 6c5f 7469 6d65 3a31 3135  central_time:115
0000140: 3920 0d0a 0d0a 0d0a 0d0a 0d0a 0d0a 0d0a  9 ..............

This error depends on contents of key.txt. What is your key.txt? — Andriy Makukha, May 10 '18 at 05:24
See also the [previous question](https://stackoverflow.com/q/50262468/3266847) for context. — Benjamin W., May 10 '18 at 13:27

Andriy Makukha · Accepted Answer · 2018-05-10T17:06:29.867

1

It looks like your key.txt file has incorrect value. Judging from the first sed command:

sed '/^[[:blank:]]*$/d; s|^|s/%%|; s|:|%%/|; s|$|/|' key.txt

it expects each line to contain a semicolon. Then it forms sed code for the second sed command:

sed -f - file1.txt > file2.txt

If your key.txt contains non-empty lines without a semicolon, you will get the error unterminated 's' command.

Ensure that key.txt is correct, or at least add /:/!d; into your pipeline. Like this:

sh '''sed '/^[[:blank:]]*$/d;/:/!d;s|^|s/%%|;s|:|%%/|;s|$|/|' key.txt | sed -f - file1.txt > file2.txt'''

For example, correct key.txt contents:

username:server1

Incorrect key.txt:

username server2

There is no semicolon in this line, so it will cause error.

You might try to replace your first sed command with a simpler one:

sed '/:/!d;s|^\([^:]*\):\(.*\)$|s/%%\1%%/\2/|' key.txt

or better:

sed -e '/:/!d;s|^\([^:]*\):\(.*\)$|s/%%\1%%/\2/|' -e 'N;s|\n/|/|' key.txt

If that doesn't help, run xxd key.txt or hexdump -C key.txt and post the output.

After you added hex contents of your key.txt file, I finally could replicate the issue on my machine. The problem could be solved by this command:

sed -e '/:/!d;s|^\([^:]*\):\(.*\)\r|s/%%\1%%/\2/|' key.txt

So the trick is to use \r instead of $ in the first sed command. If it still doesn't work for you (it might, if you use MacOS), you can just remove carriage return from the key.txt file with a tool of your choice (like dos2unix) and then your original code should work.

edited May 10 '18 at 17:06

answered May 10 '18 at 05:37

Andriy Makukha

7,580
1
38
49

my key.txt file is a key value file, where : is the separator. When I run my command in Jenkins its adding extra 3 lines. I'm going to edit my question and show you the output from first sed command, very strange output by jenkins – dino110 May 10 '18 at 06:01
if I run only first sed statement, I get a extra line with a / only. I'm pasting output in my question with your command. – dino110 May 10 '18 at 06:14
@dino110, looks like your are dealing with some unusual sed implementation, which interprets `s|$|/|` differently from GNU sed. – Andriy Makukha May 10 '18 at 06:26
getting an errro ....org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed: WorkflowScript: 17: unexpected char: '\' @ line 17, column 29. sh '''sed '/:/!d;s|^$[^:]*$:$.*$$|s/%%\1%%/\2/|' key.txt''' – dino110 May 10 '18 at 06:42
Update your answer with my output – dino110 May 10 '18 at 06:49
@dino110, you are not supposed to post your comments in my answer. I updated the answer with another solution and debugging suggestion. – Andriy Makukha May 10 '18 at 07:03
My apologies, I have updated my question with new output and debug output. – dino110 May 10 '18 at 14:06
@dino110, that helped. Updated my answer. – Andriy Makukha May 10 '18 at 17:07

Jenkins pipeline failing for sed command

1 Answers1