Strings are read incomplete after opening gzip files with fs.open() method

Asked Oct 29 '20 at 09:12

Active Oct 29 '20 at 09:18

Viewed 76 times

File format: GZIP, Stored in: GCP Storage

When I read the file with this piece of code:

fs = gcsfs.GCSFileSystem(project='myproject',token='mytoken')
with fs.open(file,'rt') as f:
   file_tmp = pd.read_csv(f, sep=",", header=0)
   print(file_tmp)

Some string fields come incomplete.

User A - IDFV (expected to be read): C305F2DB-56FC-404F-B6C1-BC52E0B680D8

User A - IDFV (actually read): C3

I have also tried the csv.reader() method, but same results. The only way that these fields come incomplete is when I do the import from GCS Storage to BigQuery.

Please I appreciate your insights on how to solve this problem. Thanks!

edited Oct 29 '20 at 09:18

asked Oct 29 '20 at 09:12

gonzathegreat

look at gzip module and also the file is binary in your example you are opening it in text mode – Sedy Vlk Oct 29 '20 at 09:14
thanks! will try now – gonzathegreat Oct 29 '20 at 09:19
Try opening the file with the gzip library, as described in this [community answer](https://stackoverflow.com/a/13083308/12857703) and it should work. – Ralemos Oct 29 '20 at 17:02
@gonzathegreat Did the solutions shared above fixed the issue you were facing? – Ralemos Nov 05 '20 at 10:29

Strings are read incomplete after opening gzip files with fs.open() method

0 Answers0