-1

I have a couple of 100 source files and 22,000 variable names to replace. Made a sed script with the replace variables, in file ReplaceScript, like this:

#! /usr/bin/sed -f
s/foo1/bar1/g
s/foo2/bar2/g
s/foo3/bar3/g

Need to replace variable names in all .c files in a tree. In the base directory to be searched, I've used the following command:

find . -type f -name "*.c" -exec sed -i '' -f ./ReplaceScript {} +
sed: 1: ./ReplaceScript: bad flag in substitute command: 's'

Single replacement command did work as intended:

find . -type f -name "*.c" -exec sed -i '' 's/foo1/bar1/g' {} +

My question: how do I fix my bad flag?

Update:

Using GNU sed results in a different error:

find . -type f -name "*.c" -exec /usr/local/bin/sed -i -f ./ReplaceScript {} +
/usr/local/bin/sed: file ./ReplaceScript line 1: unknown option to `s'

Update

It turns out to be two problems: 1. DOS carriage returns instead of unix ones. 2. little endian UTF 16 file format, must be ASCII for sed to parse correctly.

Update 2 A duplicate? The same answer to a different question, with an extra UTF 16 problem on top?

Koyovis
  • 111
  • 8
  • 3
    Do you have a `/` in your patterns in `ReplaceScript`? Pasting an actual sample could help. – codeforester Apr 11 '17 at 03:08
  • Seems to work for me, could it be a whitespace or line ending issue? – jerry Apr 11 '17 at 03:11
  • 1
    I can't reproduce it either. Apart from that, you don't need that hashbang line - that's if you want to create a sed executable that can be called like `./script` instead of `sed -f script`, but that couldn't be combined with `-i`, so I don't think you want that. – Benjamin W. Apr 11 '17 at 03:13
  • What happens when you execute the script without the surrounding `find` command, e.g. `sed -i '' -f ./ReplaceScript file1 file1`? – Ed Morton Apr 11 '17 at 03:14
  • Note that OP is on OS X so is likely using BSD sed, not GNU sed. I don't know the reason for the error, but a quick fix would be to delete the shebang line. – Jordan Running Apr 11 '17 at 03:16
  • 1
    The attempted shebang line is just a comment as far as sed is concerned, it's doing no good but neither is it doing any harm. Chances are `foo1` is really `/home/foo/bar` or something. – Ed Morton Apr 11 '17 at 03:18
  • @EdMorton That's a sensible conclusion, of course, but it doesn't help explain why sed is reporting an error on line 1. – Jordan Running Apr 11 '17 at 03:20
  • @Jordan I suspect the OPs real script file that's producing the error message doesn't even have the attempted shebang line like in the made-up one the OP posted. – Ed Morton Apr 11 '17 at 03:25
  • With BSD sed, use the `-e` flag if you're using a blank in-place flag. – Todd A. Jacobs Apr 11 '17 at 04:07
  • Yes am on OSX. Oh it does work on Linux does it? A common theme here seems to be that OS X needs an empty backup quote after -i. @codeforester there are many / in the c-source files as comments. – Koyovis Apr 11 '17 at 05:26
  • Have added the first bit of one of the c-sources. – Koyovis Apr 11 '17 at 05:43
  • @Ed Morton when executing the script without the find the same thing happens: a bad flag. The script file is as per the one I'm using, only the variable names are replaced. – Koyovis Apr 11 '17 at 05:48
  • @Benjamin W Have tried without shebang - she's still flagging :( – Koyovis Apr 11 '17 at 05:55
  • I strongly recommend [installing GNU coreutils on OS X](http://apple.stackexchange.com/questions/69223/how-to-replace-mac-os-x-utilities-with-gnu-core-utilities) to avoid the inevitable head-scratching that comes with using OS X's quirky BSD-derived tools. – Jordan Running Apr 11 '17 at 05:56
  • When I try `sed -i '' -f ./ReplaceScript` I get `sed: can't read : No such file or directory` and I am pretty sure that it is because sed tries to process the file `''`. What happens on your machine? What happens if you try `sed -i.bak -f ./ReplaceScript` instead? – Yunnosch Apr 11 '17 at 06:03
  • @Jordan did that, it's still complaining about an s on line 1. Deleted the first line comment in ReplaceScript, still no joy. – Koyovis Apr 11 '17 at 06:24
  • @Yunnosch same thing - bad flag – Koyovis Apr 11 '17 at 06:45
  • Please consider making an answer to your question. Q/A pairs are nicer and more useful than Q/commetn pairs. In my opinion it is even an interesting one, i.e. it could attract upvotes. Since excel seems to be an important part of the problem, editing the title and maybe adding a tag would seem appropriate to me. – Yunnosch Apr 11 '17 at 08:08
  • @Yunnosch did, thx for the suggestion. – Koyovis Apr 11 '17 at 11:05

2 Answers2

1

Based on this comment , the file is UTF-16 encoded, in little endian format with a byte order mark. Try running the following command on it to convert it to ASCII:

iconv -f UTF-16 -t ASCII ReplaceScript > ReplaceScript-new

I notice that it does also have carriage returns. sed seems to be okay with them on my machine, but you can easily get rid of them with dos2unix if they cause problems for you.

Community
  • 1
  • 1
jerry
  • 2,581
  • 1
  • 21
  • 32
  • @ jerry Yes that works, thx. Cannot up vote your answer at the moment, not enough credits being a new user. – Koyovis Apr 14 '17 at 02:57
0

The problem was in the DOS carriage returns in the script file. Had exported Space Delimited Text .prn from an Excel sheet, which turned out to use DOS/Windows carriage returns. These don't show up in Text Editor, only when you vi the script file does it show up as ^M. And those little mad hatters cause problems when unix sed interprets them.

Solutions posted elsewhere on this forum: install dos2unix; instruct vim to convert to unix with command file format; replace patterns; they all did not work for me. I now still export form Excel and paste into TextWrangler.

The comments suggesting that the code I posted did not show any problems, were what led to the solution. Thanks everyone for contributing to my first post on this forum.

Koyovis
  • 111
  • 8
  • Something seems off. Obviously, per my comment above, I considered the possibility that it was line ending related. I tested with various line endings on OS X and could not reproduce. Between that and the fact that you can't fix the problem with some of the standard tools, seems to me that there's something missing from the explanation. – jerry Apr 12 '17 at 15:53
  • I agree. Something else seems to be wrong but I cannot identify it: after exporting from Excel all seems to be in order when loaded in vim, but sed still sees a '?' in line 1: – Koyovis Apr 12 '17 at 20:35
  • `sed: 1: ./Replace_8_7: invalid command code ?`. Both OSX and GNU sed are the only ones seeing this question mark, I cannot identify or visualise it with any tool. – Koyovis Apr 12 '17 at 20:43
  • If only I could find out how, I could post a link to a sample of an Excel output file. Copy-Paste as in my original question does not recreate the problem. – Koyovis Apr 12 '17 at 21:36
  • You could run `xxd -p ` and post the output. – jerry Apr 13 '17 at 01:52
  • This is the sample sed replace script `s/APUGenFault/APUGenFault_A/g s/APUGenLoad_N_1/APUGenLoad_N_A/g s/APUGenLoadCurrPerPhase/APUGenLoadCurrPerPhase_A/g s/BattChargerDisableRelay/BattChargerDisableRelay_A/g`. – Koyovis Apr 13 '17 at 05:58
  • Had to edit the above and use the first two only otherwise this post would be too long. Checked if it threw a sed question mark, it did. This is the output of xxd -p: `fffe73002f00410050005500470065006e004600610075006c0074002f00 410050005500470065006e004600610075006c0074005f0041002f006700 0d000a0073002f00410050005500470065006e004c006f00610064005f00 4e005f0031002f00410050005500470065006e004c006f00610064005f00 4e005f0041002f0067000d000a00` – Koyovis Apr 13 '17 at 06:05
  • That's definitely little endian UTF-16, that explains why `sed` chokes on it. – jerry Apr 13 '17 at 13:54