0

I have a large file, several gig of binary data, with an ASCII header at the top. I need to make a few small changes to the ASCII header. sed does the job, but it takes a fair bit of time since the file is so large. vi/vim is slow too. Is there any linux utility that can just go into the file, make the change at the top, and then get out quickly?

An example might be a header that looks like:

Code Rev:   3.5
Platform:   platform1
Run Date:   12/13/16
Data source:  whatever
Restart:    False
followed by a large amount of binary data ....

and then I might need to, for example, edit an error in "Data source".

bob.sacamento
  • 6,283
  • 10
  • 56
  • 115
  • @Inian No, not the first line. Near the top of the file, but not the absolute top. I provided a short example. Thanks. – bob.sacamento Feb 06 '18 at 14:39
  • So `Data source:` just occurs once in the entire file and you want to modify its value to some other value? – Inian Feb 06 '18 at 14:39
  • why don't you use 'head' a linux utility just to do what you want – varnit Feb 06 '18 at 14:40
  • @Inian Yes, that's it. Like I said, I know of a couple of ways to do it, but nothing that works fast. – bob.sacamento Feb 06 '18 at 14:40
  • @varnit I've never seen "head" used for anything except pure display of contents. It can edit? – bob.sacamento Feb 06 '18 at 14:41
  • 4
    If the new data is *exactly* the same length as the old data, then you can modify the bytes in the file directly. But if the new data is smaller or larger that's not possible and you have to rewrite the whole file. – Some programmer dude Feb 06 '18 at 14:42
  • How about `rpl -R from_word to_word .` – khelili miliana Feb 06 '18 at 14:44
  • i'm sorry about head actually you need head to find the length of header and then use dd to overwrite that bytes 'head -n1 file.csv | ./do-some-processing | dd of=file.csv bs=1 conv=notrunc' – varnit Feb 06 '18 at 14:46
  • @Inian Sorry, I am trying, but the command just returns a blank file. I am not very adept with awk. However, I am copying and pasting your command, so I don't know what I could be doing wrong. – bob.sacamento Feb 06 '18 at 14:49
  • @Anonymousmiliana Is rpl a common utility? I've looked for it on a couple of different platforms but can't find it. – bob.sacamento Feb 06 '18 at 14:52
  • @bob.sacamento, install it before `sudo apt-get install rpl` – khelili miliana Feb 06 '18 at 14:57
  • The following links might be useful : [Replace text quickly in very large file](https://unix.stackexchange.com/questions/255373/replace-text-quickly-in-very-large-file) and [Working with huge files in vim](https://stackoverflow.com/questions/1591723/working-with-huge-files-in-vim) and [Editing a huge file - Vim or something else?](https://superuser.com/questions/364012/editing-a-huge-file-vim-or-something-else) – kvantour Feb 06 '18 at 16:28

1 Answers1

3

Provided that you know that your header is less than X bytes, you can use dd. (!) But it only works if both strings have the same length (!)

Lets say, that the header is less that 4096 bytes

dd if=/path/to/file bs=4096 count=1 | sed 's/XXX/YYY/' | dd of=/path/to/file conv=notrunc

You can also do it programmatically, using languages like C,Python,PHP,JAVA etc. The idea is to open the file, read the header, fix it, and write it back.

Alexander Miroch
  • 101
  • 4