19

I was running the sketch_rnn.ipynb on my jupyter notebook, upon loading the environment to load the trained dataset, it returned an error 'Object arrays cannot be loaded when allow_pickle=False'

This is the code already used by google developers in developing the sketch_rnn algorithm that was even run in the google colab. In the past i have ran it myself on the google colab it worked but seems not to be working on my own jupyter notebook

from magenta.models.sketch_rnn.sketch_rnn_train import *
from magenta.models.sketch_rnn.model import *
from magenta.models.sketch_rnn.utils import *
from magenta.models.sketch_rnn.rnn import * 

model_params.batch_size = 1
eval_model_params = sketch_rnn_model.copy_hparams(model_params)
eval_model_params.use_input_dropout = 0
eval_model_params.use_recurrent_dropout = 0
eval_model_params.use_output_dropout = 0
eval_model_params.is_training = 0
sample_model_params = sketch_rnn_model.copy_hparams(eval_model_params)
sample_model_params.max_seq_len = 1
return [model_params, eval_model_params, sample_model_params]


[train_set, valid_set, test_set, hps_model, eval_hps_model, 
sample_hps_model] = load_env_compatible(data_dir, model_dir)

i expected the output to be 

INFO:tensorflow:Downloading http://github.com/hardmaru/sketch-rnn- 
datasets/raw/master/aaron_sheep/aaron_sheep.npz
INFO:tensorflow:Loaded 7400/300/300 from aaron_sheep.npz
INFO:tensorflow:Dataset combined: 8000 (7400/300/300), avg len 125
INFO:tensorflow:model_params.max_seq_len 250.
total images <= max_seq_len is 7400
total images <= max_seq_len is 300
total images <= max_seq_len is 300
INFO:tensorflow:normalizing_scale_factor 18.5198.

But it gave me

ValueError: Object arrays cannot be loaded when allow_pickle=False
Duncan Jerry
  • 190
  • 1
  • 1
  • 6
  • 1
    Same issue here, with the imdb text classification example. Leading to believe that this is on their end with the dataset. – Bryan W Apr 24 '19 at 08:02

3 Answers3

28

Use allow_pickle=True as one of the arguments to np.load().

Madhuparna Bhowmik
  • 2,090
  • 4
  • 12
  • 22
23

This code solved the problem at my side.

# Downgrate numpy to fix a problem
!pip install numpy==1.16.2
import numpy as np
print(np.__version__)

I just downgrade numpy as the problem is due to some internal conflict.

Salomon Kabongo
  • 549
  • 5
  • 15
  • 2
    @DuncanJerry You're welcome. you can flag it as the correct answer to help others to find it quickly. – Salomon Kabongo May 02 '19 at 00:18
  • 3
    Changed in version 1.16.3: Made default False in response to CVE-2019-6446. See: https://numpy.org/devdocs/reference/generated/numpy.load.html#numpy.load – Huan Oct 05 '20 at 11:39
12

So I believe this has just surfaced due to a change in numpy to load(), if you observe the line that the error occurs it references something like 

    with np.load(path) as f:
        x_train, labels_train = f['x_train'], f['y_train']
        x_test, labels_test = f['x_test'], f['y_test']

but the Keras source code, for example here at line 58: https://github.com/keras-team/keras/blob/master/keras/datasets/imdb.py

now uses 

    with np.load(path, allow_pickle=True) as f:
        x_train, labels_train = f['x_train'], f['y_train']
        x_test, labels_test = f['x_test'], f['y_test']

where np.load(path) becomes np.load(path, boolean)

From brief reading, the addition of pickles has to do with security, since pickles can contain arbitrary Python code that would be run when something is loaded. (Possibly similar to the way SQL injections are performed)

After updating np.load with the new param list, it's working for my project

Bryan W
  • 1,112
  • 15
  • 26
  • i tried the above code, but it returned [Errno 22] Invalid argument: 'http://github.com/hardmaru/sketch-rnn-datasets/raw/master/aaron_sheep.. – Duncan Jerry Apr 24 '19 at 11:27
  • @DuncanJerry, could you add the full error log? I'm guessing the culprit is `load_env_compatible(data_dir, model_dir)`, where the method definition for `load_env_compatible` calls some form of .load() that needs to be updated. – Bryan W Apr 25 '19 at 06:24
  • ----> 1 with np.load(data_dir, allow_pickle=True) as load_env_compatible: 2 train_set, test_set, valid_set, hps_model, eval_hps_model, sample_hps_model = load_env_compatible(data_dir, model_dir) [Errno 22] Invalid argument: 'http://github.com/hardmaru/sketch-rnn-datasets/raw/master/aaron_sheep' – Duncan Jerry Apr 26 '19 at 00:25
  • Numpy 1.16.3 was released few days ago. From release notes: "The functions np.load(), and np.lib.format.read_array() take an allow_pickle keyword which now defaults to False in response to CVE-2019-6446 ". Downgrading to 1.16.2 helped me, as the error occurred deep inside some library. – kristjan Apr 26 '19 at 14:00