3

I have a simulation running and expect it to go on for atleast 10 more hours. I have directed the console out put to a .txt file using 

(binary) > out.txt

This out.txt is becoming too huge. I do not need a lot of contents in this file. How can I delete the older parts of this file without harming the writing process? The contents that will be written towards the end of the simulation is important to me.

akhil28288
  • 182
  • 1
  • 2
  • 14

4 Answers4

2

As Carl mentioned in the comments, you cannot really do this on an actively written log file. However, if the initial data is not relevant to you, you can do the following (though beware that you will loose all data)

> out.txt

For future, you can use a utility called logrotate(8)

jaypal singh
  • 74,723
  • 23
  • 102
  • 147
  • From what I've read on [this question](http://stackoverflow.com/questions/980283/truncating-a-file-while-its-being-used-linux) it sounds like this will work for reducing disk usage, but the file will start with a bunch of 0's (which likely isn't a problem in this case). – Brendan Long May 28 '13 at 03:29
  • @Jaypal, can you please explain your solution a little more? Did you mean, do > logfile.txt on the logfile.txt (out.txt in the example I have given in the question) that is being created? What will this do? Will it not disturb the writing process? – akhil28288 May 28 '13 at 03:37
  • 1
    @akhil28288 Running `> logfile.txt` command on your prompt will reduce your log file to zero size. This is helpful when you are running an application that cannot be restarted and if the data in the log is not critical. This will prevent your application from crashing due to unavailability of space. – jaypal singh May 28 '13 at 03:40
  • 1
    @akhil28288 Yes just do `> out.txt` on your prompt. You will loose all data in the log but your application will continue to run. This will just truncate your existing log file. – jaypal singh May 28 '13 at 03:41
  • @Jaypal, just to confirm, after I do this, the application will continue to write to this out.txt right? Sorry to be asking again. I just want to make sure. – akhil28288 May 28 '13 at 03:43
  • Yes, since it is still the same file/inode, it will continue to log into it. – jaypal singh May 28 '13 at 03:44
  • I did that. The file size reduced. However, I am not able to open that file in an editor anymore. I can see that the file size is growing. Do you have any idea why? – akhil28288 May 28 '13 at 03:57
  • 1
    let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/30720/discussion-between-jaypal-and-akhil28288) – jaypal singh May 28 '13 at 03:58
  • I execute "> out.txt" but the file size is not reduced. A process (kafka) is appending to this file but no locks are found on it. There are some "empty" chars at the beginning of this file and "cat out.txt" makes the screen blinking for a while. What happened? – David Oct 27 '22 at 03:11
0

You could use tail to only store the end of the file:

# Say you want to save the last 100 lines
your_binary | tail -n 100 > out.txt

This assumes that the output ends at some point.

Brendan Long
  • 53,280
  • 21
  • 146
  • 188
  • This answer only works if you restart the simulation, though. – Carl Norum May 28 '13 at 03:21
  • Ohh I can't afford to restart the simulation. The simulation has already run for almost 10 hours. I am guessing, I can't just open the file in an editor, delete lines and save it. I am hoping there is some other way. The file has become 10.7GB now. – akhil28288 May 28 '13 at 03:27
0

saw your comments - the file is 10 GB now ... try using sed -i to reduce the size so that it will work with the other tools, if you want to completely erase it then :> logfile.

tools can cope up with a file which is as big as their buffer , else they should be streamed ..... something like split wont work on a 4 GB file , dont know if they made a code adjustment for this , its been long since i had to work with a file that big.

two suggestions :

1

there were a few methods i could think off like using split ....but almost all were involving creation of a seperate file from the log (a reduced version) and renaming that or redirecting to that.

use split to break the log to smaller logs (split -l 100 ...) and just redirect the program output to the recent the last log found using ls -1. this seems to work fine .

2

Also i tried a second method to edit/truncate top 10 lines in the same file ......

Kaizen ~/shell_prac
$ cat zcntr.sh
## test truncate a log file

##set -xv
:> zcntr.log ;

## fxn
cntr_log()
{
limit=$1 ;
start=0 ;

 while [ $start -lt $limit ]
 do
  echo "count is $start" >> zcntr.log ;  ## generate a continuous log
  start=$(($start + 1));
  sleep 1;

  cnt=$(($start % 10)) ;
  if [  $cnt -eq 0 ]           ## check to truncate the top 10 lines using sed
   then
    echo "truncate at $start " >> zcntr.log ;
    sed -i "1,10d"  zcntr.log ;
  fi

done ;

}

## main cntrlr
  echo "enter a limit" ;
  read lmt ;
  cntr_log $lmt ;

this seems to work i tested it with a counter to print till value 25 output :

Kaizen ~/shell_prac
$  cat zcntr.log
count is 19 
truncate at 20
count is 20
count is 21
count is 22
count is 23
count is 24

i think either of the two will help.

let me know if there is something else on your mind !!

Community
  • 1
  • 1
Nitin4873
  • 16,804
  • 1
  • 13
  • 15
0

Truncate file with cat

> cat /dev/null > out.txt
Vlad Bezden
  • 83,883
  • 25
  • 248
  • 179