0

Perhaps this is a simple problem and I'm overthinking it. I'm trying to retrieve a netcdf file from an FTP server as a temporary file and then access it using xarray. Currently, I can do this this by writing the file onto my hard drive, accessing, and deleting the file:

from ftplib import FTP
import xarray as xr

with FTP("ftp.ptree.jaxa.jp") as ftp:
    ftp.login(user = user, passwd = pw)
    ftp.cwd("jma/netcdf/202205/25")
    with open("temp.nc","wb") as f:
        ftp.retrbinary("RETR " + "NC_H08_20220525_0000_r14_FLDK.02701_02601.nc", f.write)
    ds = xr.open_dataset("temp.nc",engine='netcdf4') 
    ! rm temp.nc
    ftp.quit()

This works fine, but I've read that saving temporarily to memory makes the code significantly more efficient and that would be great because I plan to implement the method on a much larger scale. My attempt at doing so was as such:

from ftplib import FTP
import tempfile
import xarray as xe

with FTP("ftp.ptree.jaxa.jp") as ftp:
    ftp.login(user = user, passwd = pw)
    ftp.cwd("jma/netcdf/202205/25")
    with tempfile.NamedTemporaryFile(suffix='.nc',mode='wb') as f:
        ftp.retrbinary("RETR " + "NC_H08_20220525_0000_r14_FLDK.02701_02601.nc", f.write)
        ds = xr.open_dataset(f.name,engine='netcdf4')
    ftp.quit()

which leads to the following warning on the ds = xr.open_dataset... line:

OSError: [Errno -101] NetCDF: HDF error: b'/tmp/tmp2w59l_kr.nc'

I've tried a few workarounds, but none have worked. I'm thinking that when I retrieve the file via ftp.retrbinary, it doesn't actually write on the file. I'm open to any suggestions.

Ian Cornejo
  • 49
  • 1
  • 5
  • 1
    I do not have experience with the `NamedTemporaryFile`, but don't you have to seek the file to the beginning before you read it? `f.seek(0)` + `NamedTemporaryFile` is imo not for *"saving temporarily to memory"*. For that you can use `io.BytesIO`. For an example, see [Retrieve data from gz file on FTP server without writing it locally](https://stackoverflow.com/q/52990046/850848). – Martin Prikryl Feb 13 '23 at 07:27

0 Answers0