4

I would like to edit the first and last line in a very huge file (~500GB). How can do so? For example, in the first line I have:

-flag </begin>

and I would like to omit the "-flag". I tried using sed (as shown) to edit the first line but I didn't work:

sed -i '1s/-flag <begin>/<begin>/g' file.txt
timgeb
  • 76,762
  • 20
  • 123
  • 145
NewToAndroid
  • 581
  • 7
  • 25

4 Answers4

4

I can't think of a way you can do this in-place (I'd be interested to hear one!)

Hardly a one-liner but you could give this a try:

# substitute the first line and exit
sed '1s/-flag \(.*\)/\1/;q' file > new        
# add the rest of the file (probably quicker than sed)
tail -n +2 file >> new    
# cut off the last line of the file
truncate -s $(( $(stat -c "%s" new) - $(tail -n 1 new | wc -c) )) new
# substitute the last line                             
tail -n 1 file | sed 's/-flag \(.*\)/\1/' >> new

This assumes you have a couple of tools like truncate and that you can do arithmetic in your shell (my shell is bash).

The truncate -s removes the last line by taking the difference between the total file size stat -c "%s" and the length of the last line in bytes.

I'm not sure what you were trying to remove from the last line but I assumed that it was the same as the first (remove -flag from the start of the line).

Suggested modifications are welcome.

Tom Fenech
  • 72,334
  • 12
  • 107
  • 141
  • You could do it in place only if you were replacing existing text with exactly the same amount of text, so that the offsets in the rest of the file did not change. For that, you could use `dd`. Since the OPs example is deleting text, the entire rest of the file needs to be shifted... – twalberg Jun 10 '14 at 18:45
  • In the last step when you're substituting the last line, what do you mean by file? – NewToAndroid Jun 11 '14 at 04:51
  • I have a 1To file where I did "echo "" >> my1tofile. I need to remove this last line (which is a huge mistake). Which one of your solution is the fastest? – Olivier Pons Aug 14 '19 at 12:13
  • 1
    @OlivierPons you should use one of the approaches here https://stackoverflow.com/a/17794626/2088135 (the link shows the answer which will perform well on large files) – Tom Fenech Aug 14 '19 at 13:21
2

if you only want to get rid of -flag (note the trailing blank) in the first line:

sed -i '1s/-flag //' file

If you want to completely replace the content of the first line, you can issue

sed -i '1s/.*/new first line/' file

To do the same to the last line (I'm providing this as an example because you did not say what you want to do with the last line), you'd do

sed -i '$s/.*/new last line/' file
timgeb
  • 76,762
  • 20
  • 123
  • 145
  • Isn't this supposed to be a fast procedure since I'm specifying the line number? However, I can see that sed has been working for a while since I fired the command. – NewToAndroid Jun 10 '14 at 17:06
  • A dash has no special meaning in regex (except in the character class sublanguage), and thus does not need to be escaped. – tripleee Jun 10 '14 at 17:06
  • The file will be processed start to end. There is no way to rewrite only the beginning of a file. – tripleee Jun 10 '14 at 17:07
1

In general, changing the beginning of a file requires to rewrite the file entirely - for reasons linked to how file-systems work that are well explained here.

But there is a trick if you rewrite the line while keeping exactly the same length: in-place editing.

A simple commandline tool to do this task would be hexedit on linux (see its shortcuts there). It is blazing fast as only the changed bytes need to be written to disk. However, it requires the new line to have the same number of characters, which is not always possible.

In the OP case, it is probably sufficient to replace -flag with 5 "space" characters, but other cases might be more tricky to handle.

Community
  • 1
  • 1
Jealie
  • 6,157
  • 2
  • 33
  • 36
0

Found that sed has the -i option which changes file in place. And that you can remove a line number n with flag nd, being 1d for first line. And the $d for last line

to remove first line

sed -i 1d yourfile.txt

to remove last line

sed -i '$d' yourfile.txt

more info here: https://stackoverflow.com/a/53433208/9475713

Caina K. Campos
  • 41
  • 3