OSError: [Errno 22] Invalid argument for read() in Python

Asked Sep 27 '17 at 23:07

Active Sep 28 '17 at 06:46

Viewed 1,262 times

I am using Python to read a large file in macOS 10.13. It seems that everytime I try to read a large file it just simply fails. Code:

f = open('file_name', 'r')

x = f.read()

print(x)

The scripts work well when the file file_name is small, but fails when reading large files (in my case, 3.5GB):

/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/bin/python3.6 /Users/mengzhao/PycharmProjects/testread/readthisfile.py
Traceback (most recent call last):
  File "/Users/mengzhao/PycharmProjects/testread/readthisfile.py", line 3, in <module>
    x = f.read()
OSError: [Errno 22] Invalid argument

Process finished with exit code 1

I have 16GB memory and I am sure it is not because of the memory. And sys.version gives me this:

3.6.1 (default, Apr  4 2017, 09:40:21) 
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.38)]

And it does not matter whether I use 'rb' or 'r', I just have the same problem. Here are some debug information after the first line is executed. It seems that the open method is fine:

Besides, I changed to python 2.7.10 and the code works. It seems that the problem has something to do with Python3 on Mac. Both Python 3.5.2 and Python 3.6.1 does not work.

Any ideas on how to fix this?

edited Sep 28 '17 at 06:46

asked Sep 27 '17 at 23:07

Zhao

2,043
1
17
36

What does `import sys; print(sys.version)` give you? – juanpa.arrivillaga Sep 27 '17 at 23:14
How much memory do you have? – user2357112 Sep 27 '17 at 23:16
1

Also, what **exactly** are you passing to `f = open('file_name', 'r')` when it fails? – juanpa.arrivillaga Sep 27 '17 at 23:22
1

There's a million "OSError: [Errno 22] when I try to .read()" questions on SO already. What makes this one unique? Without exact repro steps and data, it's impossible to answer. – Ssswift Sep 27 '17 at 23:33
@juanpa.arrivillaga gives me this: 3.6.1 (default, Apr 4 2017, 09:40:21) [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.38)] – Zhao Sep 28 '17 at 06:31
@user2357112 I have 16GB of memory – Zhao Sep 28 '17 at 06:31
@I am passing a 3.58GB file to open. From debug information, it seems that the open method works just as fine. – Zhao Sep 28 '17 at 06:38
@Ssswift sorry for confusing. I have added more information. – Zhao Sep 28 '17 at 06:38
Acctually X_train.npy is the training set of a Kaggle project. The same scripts works well on Ubuntu 16.04. – Zhao Sep 28 '17 at 06:40
1

are you using 32bit versions of Python? – juanpa.arrivillaga Sep 28 '17 at 07:58
1

@juanpa.arrivillaga I am sure it is a 64bit version – Zhao Sep 28 '17 at 08:26
>>> import platform >>> print(platform.architecture()) ('64bit', '') – Zhao Sep 28 '17 at 08:27
I am running into the same error (3.6GB file) inside a Jupyter notebook. When I execute the file.read() in the python shell, the read operation is successful. – Shailesh Oct 26 '17 at 14:41
Having the exact same issue. – Daniel Aug 20 '18 at 17:16

OSError: [Errno 22] Invalid argument for read() in Python

0 Answers0

Linked