Can't load 'mnist-original' dataset using sklearn

Question

This question is similar to what asked here and here. Unfortunately, in my case the suggested solution didn't fix the problem.

I need to work with the MNIST dataset but I can't fetch it, even if I specify the address of the scikit_learn_data/mldata/ folder (see below). How can I fix this?

In case it might help, I'm using Anaconda.

Code:

from sklearn.datasets.mldata import fetch_mldata

dataset = fetch_mldata('mnist-original', data_home='/Users/michelangelo/scikit_learn_data/mldata/')
mnist = fetch_mldata('MNIST original')

Error:

---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
<ipython-input-5-dc4d45bc928e> in <module>()
----> 1 mnist = fetch_mldata('MNIST original')

/Users/michelangelo/anaconda2/lib/python2.7/site-packages/sklearn/datasets/mldata.pyc in fetch_mldata(dataname, target_name, data_name, transpose_data, data_home)
    168     # load dataset matlab file
    169     with open(filename, 'rb') as matlab_file:
--> 170         matlab_dict = io.loadmat(matlab_file, struct_as_record=True)
    171 
    172     # -- extract data from matlab_dict

/Users/michelangelo/anaconda2/lib/python2.7/site-packages/scipy/io/matlab/mio.pyc in loadmat(file_name, mdict, appendmat, **kwargs)
    134     variable_names = kwargs.pop('variable_names', None)
    135     MR = mat_reader_factory(file_name, appendmat, **kwargs)
--> 136     matfile_dict = MR.get_variables(variable_names)
    137     if mdict is not None:
    138         mdict.update(matfile_dict)

/Users/michelangelo/anaconda2/lib/python2.7/site-packages/scipy/io/matlab/mio5.pyc in get_variables(self, variable_names)
    290                 continue
    291             try:
--> 292                 res = self.read_var_array(hdr, process)
    293             except MatReadError as err:
    294                 warnings.warn(

/Users/michelangelo/anaconda2/lib/python2.7/site-packages/scipy/io/matlab/mio5.pyc in read_var_array(self, header, process)
    250            `process`.
    251         '''
--> 252         return self._matrix_reader.array_from_header(header, process)
    253 
    254     def get_variables(self, variable_names=None):

mio5_utils.pyx in scipy.io.matlab.mio5_utils.VarReader5.array_from_header()

mio5_utils.pyx in scipy.io.matlab.mio5_utils.VarReader5.array_from_header()

mio5_utils.pyx in scipy.io.matlab.mio5_utils.VarReader5.read_real_complex()

mio5_utils.pyx in scipy.io.matlab.mio5_utils.VarReader5.read_numeric()

mio5_utils.pyx in scipy.io.matlab.mio5_utils.VarReader5.read_element()

streams.pyx in scipy.io.matlab.streams.FileStream.read_string()

IOError: could not read bytes

If you type this: from sklearn.datasets import fetch_mldata, mnist = fetch_mldata('MNIST original') does it work ? — seralouk, Nov 16 '17 at 09:05
copy paste each command separately. 1) from sklearn.datasets import fetch_mldata 2) mnist = fetch_mldata('MNIST original') — seralouk, Nov 16 '17 at 09:09
what is your sklearn version ? use: import sklearn and sklearn.__version__ to print the version — seralouk, Nov 16 '17 at 09:14
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/159130/discussion-between-sera-and-albus-c). — seralouk, Nov 16 '17 at 09:17

score 21 · Answer 1 · edited Jul 16 '20 at 10:30

21

Unfortunately fetch_mldata() has been replaced in the latest version of sklearn as fetch_openml().

So, instead of using:

from sklearn.datasets import fetch_mldata
mnist = fetch_mldata('MNIST original')

You must use:

from sklearn.datasets import fetch_openml
mnist = fetch_openml('mnist_784')
x = mnist.data
y = mnist.target

shape of x will be = (70000,784)
shape of y will be = (70000,)

edited Jul 16 '20 at 10:30

sterne

645
1
6
15

answered Nov 27 '19 at 07:45

Saurabh Yadav

311
2
4

score 7 · Answer 2 · answered Mar 04 '19 at 15:21

A quick update for the question here:

mldata.org seems to still be down. Then scikit-learn will remove fetch_mldata.

Solution for the moment: Since using the lines above will create a empty folder a the place of data_home, find the copy of the data here: https://github.com/amplab/datascience-sp14/blob/master/lab7/mldata/mnist-original.mat and download it. Then place it the ~/sklearn_data/mldata/ which is empty.

It worked for me.

score 5 · Accepted Answer · answered Sep 12 '18 at 14:23

I just faced the same issue and it took me some time to find the problem. One reason is, data can be corrupted during the first download. Remove the cached data. Find the scikit data home dir as follows:

from sklearn.datasets.base import get_data_home 
print (get_data_home())

Clean the directory and redownload the dataset. This solution works for me. For reference: https://github.com/ageron/handson-ml/issues/143

This is also related with the following question: How to use datasets.fetch_mldata() in sklearn?

score 3 · Answer 4 · answered Nov 16 '17 at 09:09

3

Instead of :

from sklearn.datasets.mldata import fetch_mldata

use:

from sklearn.datasets import fetch_mldata

And then:

mnist = fetch_mldata('MNIST original')
X = mnist.data.astype('float64')
y = mnist.target

Please see this example:

http://scikit-learn.org/stable/auto_examples/linear_model/plot_sparse_logistic_regression_mnist.html#sphx-glr-auto-examples-linear-model-plot-sparse-logistic-regression-mnist-py

answered Nov 16 '17 at 09:09

Vivek Kumar

35,217
8
109
132

Thanks for the reply Vivek! I still get `IOError: could not read bytes` – albus_c Nov 16 '17 at 09:11
1

@albus_c Possibly the download is corrupt. Please check the size of the downloaded file in `scikit_learn_data/mldata`. It should be at least 52 MB. If not, delete and try again. – Vivek Kumar Nov 16 '17 at 09:17
1

@albus_c Precisely 52.9 MB. If still not successful, then please download the file from [this link](http://mldata.org/repository/data/download/matlab/mnist-original/) in a browser and replace the file in that folder. – Vivek Kumar Nov 16 '17 at 09:25
Most likely that was the problem. I will now try with the direct link. – albus_c Nov 16 '17 at 09:32
fetch_mldata is now deprecated : ( – Eric Nov 17 '18 at 03:06

score 3 · Answer 5 · answered Nov 16 '17 at 09:49

3

For people having the same issue: it was a connection problem. If you get a similar error, check that you have the entire mnist-original.mat file, as suggested by @vivek-kumar. Current file size: 55.4 MB.

answered Nov 16 '17 at 09:49

albus_c

6,292
14
36
77

1

Yes, in the system, it shows as 55.4 MB, but during download its shown as 52.9 MB. Please consider upvoting and accepting the answer if helped. – Vivek Kumar Nov 16 '17 at 10:29

score 2 · Answer 6 · edited Feb 25 '22 at 15:41

2

In the latest sklearn version (0.21) use this:

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_digits

digits = load_digits()

X = digits.data
y = digits.target

edited Feb 25 '22 at 15:41

desertnaut

57,590
26
140
166

answered Nov 16 '19 at 21:16

seralouk

30,938
9
118
133

score 2 · Answer 7 · edited May 08 '20 at 01:17

2

Just use these two lines:

from sklearn.datasets import fetch_openml
mnist = fetch_openml('mnist_784', version=1, cache=True)

edited May 08 '20 at 01:17

csaladenes

1,100
1
11
27

answered Apr 05 '20 at 21:16

Wahiduzzaman Khan

116
4

This now throws `urlopen error [Errno -3] Temporary failure in name resolution` – csaladenes May 07 '20 at 23:20

score 0 · Answer 8 · answered Nov 22 '21 at 21:08

try this:

print(sklearn.__version__)

try:
    from sklearn.datasets import fetch_openml
    mnist = fetch_openml('mnist_784', version=1, cache=True)
except ImportError:
    from sklearn.datasets import fetch_mldata
    mnist = fetch_mldata('MNIST original')

Puneet Kumar Rajan · Answer 9 · 2019-10-16T15:09:46.090

-1

Try this one, this will work.

from sklearn.datasets import fetch_mldata
mnist = fetch_mldata('MNIST original')

edited Oct 16 '19 at 15:09

answered Oct 16 '19 at 14:48

Puneet Kumar Rajan

1
1

2

Hi Puneet! Could you add an explanation as to why this is the correct answer? – Nathan Oct 16 '19 at 14:53

Can't load 'mnist-original' dataset using sklearn

9 Answers9

Linked

Related