25

In Linux, I have a rather large file with some extraneous information tacked on to the end of it. Let's say for example I know there are 314 bytes of extraneous data at the end of a 1.6GB file.

Of course it is very easy and efficient to add more data to the end of a file, but what can I do to remove it without having to copy the first portion of that file into another (or overwrite said file)?

Edit

I'm seeing some good advice on doing this in C. I was hoping to script it from the commandline, but failing that I would be more inclined to doing it in python than C.

I see that python has a truncate method on its file object but it seems to be demolishing my file no matter how i use it--I should be able to figure this one out, but of course answers are more than welcome still.

andyortlieb
  • 2,326
  • 3
  • 21
  • 33

4 Answers4

34

use the function truncate

http://linux.die.net/man/2/truncate

int truncate(const char *path, off_t length);
int ftruncate(int fd, off_t length);

truncate takes the file name
ftruncate takes an open file descriptor

both of these set the file length to length so it either truncates or elongates (in the latter case, the rest of the file will be filled with NULL/ZERO)

[edit]
truncate (linux shell command) will work also

**SYNTAX**

truncate -s integer <filename>  
**OPTIONS**

-s number specify the new file length. If the new length is smaller than the current filelength data is lost. If the new length is greater the file is padded with 0. You can specify a magnitude character to ease large numbers:
b or B size is bytes.
k size is 1000 bytes.
K size is 1024 bytes.
m size is 10^6 bytes.
M size is 1024^2 bytes.
g size is 10^9 bytes.
G size is 1024^3 bytes.


**EXAMPLES**

To shrink a file to 10 bytes:

truncate -s 10 /tmp/foo

To enlarge or shrink a file to 345 Megabytes:

truncate -s 345M /tmp/foo

[/edit]

KevinDTimm
  • 14,226
  • 3
  • 42
  • 60
  • ugh! /usr/bin/truncate is in coreutils in debian squeeze, but not lenny (which I cannot upgrade) – andyortlieb Sep 12 '11 at 19:57
  • grab the source, make it yourself :) – KevinDTimm Sep 12 '11 at 20:08
  • 1
    That is in progress. Damn their stability! – andyortlieb Sep 12 '11 at 20:16
  • Apparently when you use truncate -s to reduce the file by 1 byte, then it removes the newline character from the end? That is what happened to me on my Ubuntu machine. – IgorGanapolsky Sep 27 '11 at 18:28
  • 1
    @IgorG. It should remove whatever the last character is in your file. If it's a newline, it removes the newline. What would you rather it do? – KevinDTimm Sep 27 '11 at 18:41
  • 1
    @KevinDTimm: I didn't know that a newline character is automatically inserted at the end of a line on Linux. I used vim, and typed a simple single-line sentence, so I expected a smaller filesize. I didn't expect the newline character. – IgorGanapolsky Sep 28 '11 at 14:00
21

Although there were plenty of references to the truncate function in this thread, no one really answered the OP's question about reducing a file by a fixed amount from a scripting environment. Kevin's answer used truncate to resize the target file to a fixed amount, but of course the correctness of this solution requires the user to first know the size of the target file minus the extraneous data at the end. So, we have: 

   -s, --size=SIZE
          set or adjust the file size by SIZE bytes

Truncate actually supports removing data from the end of a target file directly by prefixing SIZE with a - character.

For example, to resize a file by 314 bytes you can do:

truncate --size=-314 target_file.bin

harm
  • 10,045
  • 10
  • 36
  • 41
sherrellbc
  • 4,650
  • 9
  • 48
  • 77
  • Thats' exactly what I'm looking for. Thanks! – Robert Jan 02 '17 at 08:39
  • From the OP `Let's say for example I know there are 314 bytes of extraneous data at the end of a 1.6GB file.` Modify that to say `suppose I know there are X bytes of extraneous data at the end of a file of size Y`. OP's question was answered. – KevinDTimm Jan 03 '17 at 19:55
5

Using "truncate" is best way, I just post some examples:

  1. I have a file "PNav-h.db", it has 50176 bytes.

    -rw-r--r--  1 user user 50176 Mar  8 23:43 PNav-h.db
 $truncate -s 1000 PNav-h.db

    it set the file size to 1000 byptes

     -rw-r--r--  1 user user  1000 Mar  9 00:02 PNav-h.db

  2. For your case, use $truncate --size=xxx xxfilename, using -<size number> to reduce the file size

    $truncate --size=-300 PNav-h.db
-rw-r--r--  1 user user   700 Mar  9 00:07 PNav-h.db

    final file size = 1000-300=700

  3. using +<size number> to increase the file size

    $truncate --size=+500 PNav-h.db
-rw-r--r--  1 user user  1200 Mar  9 00:09 PNav-h.db

    final file size = 700 + 500 = 1200

  4. if there is no - or +, it is to set the file size.

    $truncate --size=60000 PNav-h.db
-rw-r--r--  1 user user 60000 Mar  9 00:12 PNav-h.db

    final file size set to 60000

phuclv
  • 37,963
  • 15
  • 156
  • 475
Hannah Zhang
  • 489
  • 5
  • 5
2

In C on a POSIX-compliant system (or more generally, most Unix-like systems), you can use the truncate and ftruncate functions.

Tom Zych
  • 13,329
  • 9
  • 36
  • 53
  • there are neither `truncate` nor `ftruncate` in C standard. They're POSIX functions – phuclv Jan 07 '19 at 01:26
  • @phuclv: Actually, they’re system calls that originated in 4.2BSD. OP did specify Linux, so I feel safe in recommending generally available Unix system calls. But you have a point, I’ll edit the answer. Thanks. – Tom Zych Jan 09 '19 at 01:41