26

I have the following error while exporting pandas dataframe into csv file. I have enough space on my hard disk.

OSError: [Errno 28] No space left on device

What can be the reason for this? Many thanks in advance.

Andre Nevares
  • 711
  • 6
  • 21
Khalid
  • 621
  • 1
  • 7
  • 14
  • I suspect your tmp directory gets exhausted. – Géza Török Nov 24 '18 at 09:56
  • Thanks for your reply. How can I clean my tmp directory? – Khalid Nov 24 '18 at 09:57
  • 2
    @khalid, as pointed out earlier, clear your cache/tmp directory. For more clear understanding have a look here https://stackoverflow.com/questions/6998083/python-causing-ioerror-errno-28-no-space-left-on-device-results-32766-h – sync11 Nov 24 '18 at 10:00
  • What OS you are working on, if its Linux , First try to see the space on the terminal by executing `pwd; df -Ph .` and see `Avail` & `Use%` column, – Karn Kumar Nov 24 '18 at 10:00
  • Apologies, i'm non window user :( – Karn Kumar Nov 24 '18 at 10:02
  • @dataLeo I have looked at this link. Actually, it is not clear how to clean my directory from this link. There was a part, however it did not work. – Khalid Nov 24 '18 at 10:02
  • @Khalid Which Win OS are you using? – sync11 Nov 24 '18 at 10:04
  • @dataLeo WIndows 7 – Khalid Nov 24 '18 at 10:05
  • 4
    Open `Run` and type `%temp%`. This will lead to temp files folder. Select all and delete them. If some doesn't get delete, don't worry. Restart and try to save your CSV. If that doesn't help I advice you to do a disk defragmentation, steps of which can be found here https://support.microsoft.com/en-in/help/17126/windows-7-improve-performance-defragmenting-hard-disk – sync11 Nov 24 '18 at 10:11
  • @dataLeo That is the issue. I cant do it. I would like to ask you whether you know how to delete temp files by using python code. – Khalid Nov 24 '18 at 10:14
  • You can do it without python. Follow above steps which I've mentioned. If you insists to do it using python then change the path to above mentioned directory and delete all files. I suggest you follow first method. – sync11 Nov 24 '18 at 10:34
  • @dataLeo I have joined some powerful PC and work on it. Therefore, I cant to it. Regarding python. So, I should try the following method: cd /tmp rm /tmp . I try, however it did not delete it – Khalid Nov 24 '18 at 10:36
  • Does `%env JOBLIB_TEMP_FOLDER=/tmp` solve your problem? – Gonçalo Peres May 05 '21 at 08:07

5 Answers5

2

Late answer, but maybe usefull.

I encountered a similar issue :

OSError: [Errno 28] No space left on device:

But, I have enough space:

$ df -h
Sys. de fichiers          Taille Utilisé Dispo Uti% Monté sur
/dev/dm-0                   5,6G    3,2G  2,1G  62% /
udev                         10M       0   10M   0% /dev
tmpfs                       4,8G    481M  4,3G  10% /run
tmpfs                        12G    8,0K   12G   1% /dev/shm
tmpfs                       5,0M       0  5,0M   0% /run/lock
tmpfs                        12G       0   12G   0% /sys/fs/cgroup
tmp                         945M     23M  858M   3% /tmp
var                         5,6G    3,3G  2,0G  64% /var
data                         20G     11G  8,6G  56% /data

In fact, it's not a weight problem, it is an amount problem.

The evidence:

$ df -hi
Sys. de fichiers          Inœuds IUtil. ILibre IUti% Monté sur
/dev/dm-0                   367K   307K    61K   84% /
udev                        3,0M    365   3,0M    1% /dev
tmpfs                       3,0M    597   3,0M    1% /run
tmpfs                       3,0M      2   3,0M    1% /dev/shm
tmpfs                       3,0M      9   3,0M    1% /run/lock
tmpfs                       3,0M     13   3,0M    1% /sys/fs/cgroup
tmp                          61K   5,1K    56K    9% /tmp
var                         367K    95K   273K   26% /var
data                        1,3M   1,3M      0  100% /data

There is no more place for inodes (files).

And this is clearly due to the session management:

$ du --max-depth=1 --inodes -h /data/my-python-program/data/
1,2M    /data/base-eelv/data/sessions
178 /data/base-eelv/data/medias

The quick fix is to empty the session directory.

The clean fix is to auto-cleanup, but at this time, I have'nt search how to do this.

Bastien Ho
  • 777
  • 12
  • 23
0

I was working with docker containers and increasing its swap space bypassed the error for me.

0

Assuming you actually do have enough space on disk i.e. nothing funny is going on disk-space-wise (temp files deleted, OS correctly reporting amount of free space)...

This error while exporting with pandas specifically could be triggered because the dataframe in question is too large for the write operation to process it properly. You can try writing the dataframe to a CSV in batches instead. Manually this looks something like:

    INTERVAL = 50
    num_rows = len(df)
    final_index = num_rows - 1
    indices = list(range(0, final_index, INTERVAL))
    indices.append(final_index)
    

    for i in range(len(indices) - 1):
        start = indices[i]
        end = indices[i+1]        
        table_segment = df.iloc[start:end+1] 
        if i == 0:
            table_segment.to_csv("./test2.csv")
        else:
            table_segment.to_csv("./test2.csv", mode = 'a', header = False)

EDIT: Alternatively the (new?) chunksize argument in to_csv for pandas dataframes does the same thing:

df.to_csv(path_or_buf="./test3.csv", chunksize=INTERVAL)

Lastly, there is a good answer explaining this error generally here: Python causing: IOError: [Errno 28] No space left on device: '../results/32766.html' on disk with lots of space

Dev Arora
  • 139
  • 5
0

try truncating the .log files, /var/log

examples

sudo truncate -s 0 /var/log/celery/*log
sudo truncate -s 0 /var/www/html/app/stderr.log
cottontail
  • 10,268
  • 18
  • 50
  • 51
0

The ENOSPC ("No space left on device") error will be triggered in any situation in which the data or the metadata associated with an I/O operation can't be written down anywhere because of lack of space. This doesn't always mean disk space – it could mean physical disk space, logical space (e.g. maximum file length), space in a certain data structure or address space. For example you can get it if there isn't space in the directory table (vfat) or there aren't any inodes left. It roughly means “I can't find where to write this down”.

Source:

Python causing: IOError: [Errno 28] No space left on device: '../results/32766.html' on disk with lots of space

Also:

It turns out the best solution for me here was to just reformat the drive. Once reformatted all these problems were no longer problems.

Possible solutions:

  • Reformat your drive (OP of other question has not update his/her answer)
  • Delete the \tmp directory with rm -r /tmp/
DialFrost
  • 1,610
  • 1
  • 8
  • 28