How to use locale variable and 'delete last line' in sed

Question

I am trying to delete the last three lines of a file in a shell bash script.

Since I am using local variables in combination with the Regex syntax in sed the answer proposed in How to use sed to remove the last n lines of a file does not cover this case. On the contrary, the cases covered deal with sed in a terminal and does not cover syntax in shell scripts, neither does it cover the use of variables in sed expressions.

The commands I have available is limited, since I am not on a Linux but use a MINGW64 for it. sed does a create job so far, but deleting the last three lines gives me some headaches in relation of how to format the expression.

I use wc to be aware of how many lines the file has and subtract then with expr three lines.

n=$(wc -l < "$distribution_area")
rel=$(expr $n - 3)

The start point for deleting lines is defined by rel but accessing the local variable happens through the $ and unfortunately the syntax of sed is using the $ to define the end of file. Hence,

sed -i "$rel,$d" "$distribution_area"

won't work, and what ever variant of combinations e.g. '"'"$rel"'",$d' gives me sed: -e expression #1, char 1: unknown command: `"' or something similar.

Can somebody show me how to combine the variable with the $d regex syntax of sed?

Answer 1

sed -i "$rel,$d" "$distribution_area"

Here you're missing the variable name (n) for the second arg.

Consider the following example on a file called test that contains 1-10:

n=$(wc -l < test)
rel=$(($n - 3))

sed "$rel,$n d" test

Result:

1
2
3
4
5
6

To make sure the d will not interfere with the $n, you can add a space instead of escaping.

---

If you have a recent head available, I'd recommend something like:

head -n -3 test

Answer 2

Can somebody show me how to combine the variable with the $d regex syntax of sed?

$d expands to a varibale d, you have to escape it.

"$rel,\$d"

or:

"$rel"',$d'

But I would use:

head -n -3 "$distribution_area" > "$distribution_area".tmp
mv "$distribution_area".tmp "$distribution_area"

Answer 3

You can remove the last N lines using only pure Bash, without forking additional processes (such as sed). Such scripts look ugly, but they would work in any environment where only Bash runs and nothing else is available, no other binaries like sed, awk etc.

If the entire file fits in RAM, a straightforward solution is to split it by lines and print all but the N trailing ones:

delete_last_n_lines() {
  local -ir n="$1"
  local -a lines
  readarray lines
  ((${#lines[@]} > n)) || return 0
  printf '%s' "${lines[@]::${#lines[@]} - n}"
}

If the file does not fit in RAM, you can keep a FIFO buffer that stores N lines (N + 1 in the “implementation” below, but that’s just a technical detail), let the file (arbitrarily large) flow through the buffer and, after reaching the end of the file, not print out what remains in the buffer (the last N lines to remove).

delete_last_n_lines() {
  local -ir n="$1 + 1"
  local -a lines
  local -i pos i
  for ((i = 0; i < n; ++i)); do
    IFS= read -r lines[i] || return 0
  done
  printf '%s\n' "${lines[pos]}"
  while IFS= read -r lines[pos++]; do
    ((pos %= n))
    printf '%s\n' "${lines[pos]}"
  done
}

The following example gets 10 lines of input, 0 to 9, but prints out only 0 to 6, removing 7, 8 and 9 as desired:

printf '%s' {0..9}$'\n' | delete_last_n_lines 3

Last but not least, this simple hack lacks sed’s -i option to edit files in-place. That could be implemented (e.g.) using a temporary file to store the output and then renaming the temporary file to the original. (A more sophisticated approach would be needed to avoid storing the temporary copy altogether. I don’t think Bash exposes an interface like lseek() to read files “backwards”, so this cannot be done in Bash alone.)