Jenkins pipeline failing for sed command - jenkins

Question

I'm running a sed command in pipeline but it is failing with an error
sed: file - line 2: unterminated `s' command
here is my code for sed in pipeline:
sh '''sed '/^[[:blank:]]*$/d;s|^|s/%%|;s|:|%%/|;s|$|/|' key.txt | sed -f - file1.txt > file2.txt'''
I if I will run only first sed command, Jenkins gives very strange output, here it is from Jenkins logs its adding extra lines with characters:
+ sed '/^[[:blank:]]*$/d;s|^|s#%%|;s|:|%%#|;s|$|#|' keys.txt
s#%%sql_server_name%%#test_seqserver_1234
#
s#%%
#
s#%%sql_login_name%%#test_login_name
#
s#%%
#
s#%%password%%#test_password
#
s#%%
#
If run following command this is what Jenkins output looks like
sh '''sed '/^[[:blank:]]*$/d;/:/!d;s|^|s/%%|;s|:|%%/|;s|$|/|'
keys.txt'''
Jenkins output:
+ sed '/^[[:blank:]]*$/d;/:/!d;s|^|s/%%|;s|:|%%/|;s|$|/|' keys.txt
s/%%sql_server_name%%/test_seqserver_1234
/
s/%%sql_login_name%%/test_login_name
/
s/%%password%%/test_password
/
s/%%SID%%/123456
/
I ran the new command:
sh '''sed -e '/:/!d;s|^\\([^:]*\\):\\(.*\\)$|s/%%\\1%%/\\2/|' -e 'N;s|\\n/|/|' keys.txt'''
Here is the output:
Running shell script
+ sed -e '/:/!d;s|^\([^:]*\):\(.*\)$|s/%%\1%%/\2/|' -e 'N;s|\n/|/|'
keys.txt
s/%%sql_server_name%%/test_seqserver_1234
/
s/%%sql_login_name%%/test_login_name
/
s/%%password%%/test_password
/
s/%%SID%%/123456
Here is xxd output for text file:
Running shell script
+ xxd keys.txt
0000000: 7371 6c5f 7365 7276 6572 5f6e 616d 653a sql_server_name:
0000010: 7465 7374 5f73 6571 7365 7276 6572 5f31 test_seqserver_1
0000020: 3233 340d 0a0d 0a73 716c 5f6c 6f67 696e 234....sql_login
0000030: 5f6e 616d 653a 7465 7374 5f6c 6f67 696e _name:test_login
0000040: 5f6e 616d 650d 0a0d 0a70 6173 7377 6f72 _name....passwor
0000050: 643a 7465 7374 5f70 6173 7377 6f72 6420 d:test_password
0000060: 0d0a 0d0a 5349 443a 3132 3334 3536 200d ....SID:123456 .
0000070: 0a0d 0a64 6566 6175 6c74 5f64 6174 6162 ...default_datab
0000080: 6173 653a 7465 6d70 6462 0d0a 0d0a 6465 ase:tempdb....de
0000090: 6661 756c 745f 6c61 6e67 7561 6765 3a75 fault_language:u
00000a0: 735f 656e 676c 6973 680d 0a0d 0a63 6865 s_english....che
00000b0: 636b 5f65 7870 6972 6174 696f 6e3a 4f46 ck_expiration:OF
00000c0: 460d 0a0d 0a63 6865 636b 5f70 6f6c 6963 F....check_polic
00000d0: 793a 4f46 460d 0a0d 0a64 656c 6976 6572 y:OFF....deliver
00000e0: 7974 7970 653a 7363 6865 6475 6c65 640d ytype:scheduled.
00000f0: 0a0d 0a73 6368 6564 756c 6564 5f64 656c ...scheduled_del
0000100: 6976 6572 7964 6174 653a 3035 2d33 302d iverydate:05-30-
0000110: 3230 3939 0d0a 0d0a 7363 6865 6475 6c65 2099....schedule
0000120: 645f 6465 6c69 7665 7279 5f32 3468 725f d_delivery_24hr_
0000130: 6365 6e74 7261 6c5f 7469 6d65 3a31 3135 central_time:115
0000140: 3920 0d0a 0d0a 0d0a 0d0a 0d0a 0d0a 0d0a 9 ..............

Answer 1

It looks like your key.txt file has incorrect value. Judging from the first sed command:
sed '/^[[:blank:]]*$/d; s|^|s/%%|; s|:|%%/|; s|$|/|' key.txt
it expects each line to contain a semicolon. Then it forms sed code for the second sed command:
sed -f - file1.txt > file2.txt
If your key.txt contains non-empty lines without a semicolon, you will get the error unterminated 's' command.
Ensure that key.txt is correct, or at least add /:/!d; into your pipeline. Like this:
sh '''sed '/^[[:blank:]]*$/d;/:/!d;s|^|s/%%|;s|:|%%/|;s|$|/|' key.txt | sed -f - file1.txt > file2.txt'''
For example, correct key.txt contents:
username:server1
Incorrect key.txt:
username server2
There is no semicolon in this line, so it will cause error.
You might try to replace your first sed command with a simpler one:
sed '/:/!d;s|^\([^:]*\):\(.*\)$|s/%%\1%%/\2/|' key.txt
or better:
sed -e '/:/!d;s|^\([^:]*\):\(.*\)$|s/%%\1%%/\2/|' -e 'N;s|\n/|/|' key.txt
If that doesn't help, run xxd key.txt or hexdump -C key.txt and post the output.
After you added hex contents of your key.txt file, I finally could replicate the issue on my machine. The problem could be solved by this command:
sed -e '/:/!d;s|^\([^:]*\):\(.*\)\r|s/%%\1%%/\2/|' key.txt
So the trick is to use \r instead of $ in the first sed command. If it still doesn't work for you (it might, if you use MacOS), you can just remove carriage return from the key.txt file with a tool of your choice (like dos2unix) and then your original code should work.