Bash: Can't append a string to an existing string - it appears to overwrite the start of that string instead

Question

This is a script that helps us in creating a dhcpd.conf file.

sample inputs (ex. mac-tab-;-tab-IP)

DC:D3:21:75:61:90   ;   10.25.131.17
    ;

expected outputs

Host 27-48 { hardware ethernet DC:D3:21:75:61:90 ; fixed-address 10.25.131.17 ; }

#host 27-48 { hardware ethernet ; fixed-address ; }

Currently the line outputted is this:

Host 27-48 { hardware ethernet 00:16:6B:C8:3D:C9 ; fixed-address 10.25.129.185

Specific line in code I'm stuck on

outputLine="Host $((names[i]))-$((startingNumber+counter)) { hardware ethernet $first ; fixed-address $second"

If I try adding ; }

outputLine="Host $((names[i]))-$((startingNumber+counter)) { hardware ethernet $first ; fixed-address $second ; }"

I get this:

; } 27-48 { hardware ethernet 00:16:6B:C8:3D:C9 ; fixed-address 10.25.129.185

The issue is, whenever I append " ; }" to the end of the above line, it overwrites the beginning of the line. I've tried a few tricks to work around it, such as writing the above line to a string, and then trying to append to the string, but the same issue occurs. I had an idea to export the entire contents to a file, and re-reload the file into an array just so I can append, but it seems a little overkill.

for ((j=1; j<=${sizes[i]}; j++ )); do

    #split line, read split as two entries for an arrIN
    IN=(${line[counter+1]})
    arrIN=(${IN//;/ })
    first="${arrIN[0]}"
    second=${arrIN[1]}


    if [ ${lineSize[counter+1]} -gt 5 ]
    then
        #sed 's/$/ ; }/' $outputLine > newoutputLine
        outputLine="Host $((names[i]))-$((startingNumber+counter)) { hardware ethernet $first ; fixed-address $second"
        echo $outputLine
    else
        echo "#host $((names[i])) $((startingNumber+counter)) { hardware ethernet ; fixed-address ; }"
    fi
    counter=$((counter+1))
done

Try printing `$second` -- may be there's something funny at the end of `${arrIN[1]}` — mustaccio, Jan 19 '17 at 16:15
State your input variables clearly and provide an exact expected output. Is your `bash` code creating the `dhcpd.conf` file or you just want to edit that file in-place with the `; }` at end? — Inian, Jan 19 '17 at 16:17
updated sample input. Bash code is not creating dhcpd.conf, but just a section of it which will ultimately be merged with a subnet portion (i.e. copy and pasted). One output has already been provided, will update the output for the other "blank case" — thistleknot, Jan 19 '17 at 16:29
just a hunch, I would change the `echo $outputLine` into `echo "$outputLine"` — Aaron, Jan 19 '17 at 16:42
There must be a carriage return sneaking in (that is, your input must have Windows line endings). Try running, say, `second="${second//$'\r'/}"` to snip it out. — ruakh, Jan 19 '17 at 16:46
the input were columns in excel, pasted into either notepad or notepad++. I'll try that, thanks. That just might be it. Edit: That was it! Thank you so much! If you add this as an answer, I will upvote it. I thought it was a "special characters" issue. — thistleknot, Jan 19 '17 at 16:53

score 6 · Accepted Answer · edited May 23 '17 at 12:33

As ruakh explains in a comment on the question, the problem was a CR (0xD, \r) character at the end of the value of variable $second, which can be removed with the following parameter expansion: second="${second//$'\r'/}".

The rest of this answer explains the symptom and provides background information.

The issue is, whenever I append " ; }" to the end of the above line, it overwrites the beginning of the line.

"overwriting the beginning of a line" almost always points to an embedded CR (0xD, \r) character, which, when a string is printed to the terminal, gives the appearance of overwriting the start of the line:

$ printf 'abc\rd'  # `printf '\r'` produces a CR 
dbc  # It LOOKS LIKE 'd' replaced 'a', but that's an artifact of printing to the terminal.

It is only because the terminal interprets the CR (\r) as "place the cursor at the start of the line" that d - the remaining string after \r - appears to "overwrite" the start of the already-printed part of the string.

You can visualize embedded CRs (and other control characters) with cat -et, which represents them in caret notation as ^M:

$ printf 'abc\rd' | cat -et
abc^Md   # ^M represents a CR (\r)

As you can see, the d didn't actually overwrite a, the start of the string.

CR (\r) instances are rarely used in the world of Unix text processing. If they do appear as part of text data, it is usually from Windows-originated sources that use CRLF (\r\n) line endings rather than the Unix LF-only (\n) line endings.

Often, the simplest approach is to simply remove the CR (\r) instances before using the data with Unix utilities.

There are many, many existing answers that cover this territory, often recommending use of third-party utility dos2unix, easily installable via the package managers of many platforms (e.g., sudo apt install dos2unix on Ubunutu, or brew install dos2unix on macOS, via Homebrew).

Alternatively,

for text already stored in a variable, use ruakh's approach based on parameter expansion; e.g., second="${second//$'\r'/}"
for files, solutions using standard utilities can act as a makeshift dos2unix.

Bash: Can't append a string to an existing string - it appears to overwrite the start of that string instead

1 Answers1