Feature normalization on MNIST dataset

Question

I am working with a subset of MNIST dataset where I want to normalize the features of the samples in the dataset. I am trying to load the dataset in the form of .mat file. Can anyone please guide me on how I can convert .mat to numpy array so I can perform basic operations like mean and std. deviation on the feature vectors?

This is my code for loading .mat file and converting to numpy array:

import scipy.io
import numpy as np

train_0 = scipy.io.loadmat('data/training_data_0.mat')
train_1 = scipy.io.loadmat('data/training_data_1.mat')

test_0 = scipy.io.loadmat('data/testing_data_0.mat')
test_1 = scipy.io.loadmat('data/testing_data_1.mat')

# to return a group of the key-value
# pairs in the dictionary
result = train_0.items()

# Convert object to a list
data = list(result)

# Convert list to an array
numpyArray = np.array(data)

print(numpyArray.mean())

However after execution I am getting this error:

  numpyArray = np.array(data)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/mish/Work/ASU/Fall20/CSE 569/main.py", line 20, in <module>
    print(numpyArray.mean())
  File "/usr/local/lib/python3.8/site-packages/numpy/core/_methods.py", line 160, in _mean
    ret = umr_sum(arr, axis, dtype, out, keepdims)
TypeError: can only concatenate str (not "bytes") to str

score 0 · Answer 1 · answered Oct 15 '20 at 13:07

You passing a list of tuples (key, value) to the numpy.array, you have the numpy array already just use train_0['<some variable name here>']

To get the variable names just use: print(train_0.keys())

This is probably answers your question: Convert loaded mat file back to numpy array

The scipy.io.loadmat returns a dictionary:

Returns
    mat_dictdict

        dictionary with variable names as keys, and loaded matrices as values.

https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.loadmat.html

Feature normalization on MNIST dataset

1 Answers1