0

I am getting crazy with simple shell script, I am testing in wsl (ubuntu 2020.4) under win10, the shell script shall be eventually run in a gitlab pipeline.

I want to process a json file in a convert.sh script that I want to call like this: #./convert.sh source.json destination.json

my convert.sh reads jq 'my filter' $1 > $2

if I run in terminal #jq 'my filter' source.json > destination.json

the destination.json is as expected.

but the convert.sh script adds a \m at the end of the filename, I eventually get a destination.json\m file: why is this stuff adding a CR char at the end of the filename ?

I also tried cat $1 | jq 'my filter' | sponge $2 but it generates the same wrong filename.

as said, writing directly the jq cmd in the terminal works flawless.

Any idea?

eventually, I would like to apply 2 filters on my file, and write back to the same filename. it is possible to combine filters?

EricBDev
  • 1,279
  • 13
  • 21
  • Have you tried jq's `--compact-output / -c` option? – Barry Carter Aug 16 '22 at 15:32
  • This means one of your files was saved as a DOS/Windows text file instead of UNIX text files, and the extra content has leaked into your script's output. DOS uses two-character `\r\n` line separators; UNIX uses `\n` line terminators. (The difference between a separator and a terminator is that a terminator needs to be on the end of _every_ line, including the last one in the file; whereas a separator only goes at boundaries _between_ lines; so on Windows text files don't have a newline at the end, whereas to UNIX tools if they don't they're invalid and may behave badly). – Charles Duffy Aug 16 '22 at 16:52

1 Answers1

1

The \r character is in the file containing the invocation of convert, so it becomes part of the second argument. Then, when convert.sh is executed, $2 is expanded with the \r.

Make sure your script files use Unix line endings.

rici
  • 234,347
  • 28
  • 237
  • 341
  • nothing to do with the content of the json files, the issue was indeed the LF of the sh file! switching to unix line endings in my sh file in notepad++ solved the issue. – EricBDev Aug 16 '22 at 18:03