Can't use collections.defaultdict() in google-app-engine

Question

Trying to use collections.defaultdict() to create an histogram in google-app-engine :

class myDS(ndb.Model):

    values = ndb.PickleProperty()
    hist = ndb.PickleProperty()

class Handler:
    my_ds = myDS()
    my_ds.values = {}   
    my_ds.hist = defaultdict(lambda : 0)

And got the error (from log)

File "/base/alloc/tmpfs/dynamic_runtimes/python27/277b61042b697c7a_unzipped/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 1331, in call
    newvalue = method(self, value)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27/277b61042b697c7a_unzipped/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 1862, in _to_base_type
    return pickle.dumps(value, pickle.HIGHEST_PROTOCOL)
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

Any way to solve this?

Note that `ndb` is only supported in the standard GAE environment, which only works with python 2.7. Donno if that's related to your problem or not. — Dan Cornilescu, Jun 13 '18 at 14:50
@DanCornilescu defaultdict should be included in python2.7.. https://docs.python.org/2/library/collections.html#collections.defaultdict — Cranjis, Jun 13 '18 at 15:09
@snakecharmerb How can I access the logs? I didn't know there are logs :) — Cranjis, Jun 13 '18 at 16:40
To view cloud logs, see [here](https://console.cloud.google.com/logs). The dev server also produces logs, which may be written to the terminal (the default) or file. See also docs [here](https://cloud.google.com/appengine/docs/standard/python/logs/) — snakecharmerb, Jun 13 '18 at 16:59
@snakecharmerb Amazing , had no idea I can see the log! I edited the post with the log error — Cranjis, Jun 13 '18 at 17:51
I don't think you can pickle a function, so putting that lambda inside the `defaultdict` might be what is causing the problem. — Carl Smith, Jun 13 '18 at 17:54
@CarlSmith this is the only way I know to init a defaultdict - is there any other wat I can use it here? — Cranjis, Jun 13 '18 at 17:55
I'm not sure I understand what you're asking. You can just call `defaultdict` with no args to initialise it. — Carl Smith, Jun 13 '18 at 17:58
Sorry, I don't actually know how `defaultdict` works, so ignore me if I'm being ignorant. I was really just mentioning that you cannot pickle a function, in case that helped. — Carl Smith, Jun 13 '18 at 18:00
@CarlSmith this is the way I know to init defaultdict - if there is another way I will use it.. — Cranjis, Jun 13 '18 at 18:01
Does it work if you do `my_ds.hist = defaultdict(int)`? This works for me locally, but I haven't tried in the cloud. — snakecharmerb, Jun 13 '18 at 19:14
@snakecharmerb No , I get a error TypeError: first argument must be callable — Cranjis, Jun 13 '18 at 19:17
@snakecharmerb yep.. https://stackoverflow.com/questions/42137849/defaultdict-first-argument-must-be-callable-or-none — Cranjis, Jun 14 '18 at 07:33
`int` *is* a callable, so either you aren't using the code I suggested or that exception is being raised somewhere else. — snakecharmerb, Jun 14 '18 at 08:05
@snakecharmerb you meant the word "int"? I thought you meant (to put a number there). How does python "knows" to put 0 when I write "int"? — Cranjis, Jun 14 '18 at 09:24
See the defaultdict examples in the [docs](https://docs.python.org/2/library/collections.html#defaultdict-examples) — snakecharmerb, Jun 14 '18 at 09:39

score 2 · Accepted Answer · answered Jun 14 '18 at 17:51

A PickleProperty field require a value that is serializable using Python's pickle protocol (see docs for more info):

PickleProperty: Value is a Python object (such as a list or a dict or a string) that is serializable using Python's pickle protocol; Cloud Datastore stores the pickle serialization as a blob. Unindexed by default. Optional keyword argument: compressed.

See also this answer from Martijn Pieters:

Pickle cannot handle lambdas; pickle only ever handles data, not code, and lambdas contain code. Functions can be pickled, but just like class definitions only if the function can be imported. A function defined at the module level can be imported. Pickle just stores a string in that case, the full 'path' of the function to be imported and referenced when unpickling again.

There are multiple options to work with default values, depending on your use case.

Can't use collections.defaultdict() in google-app-engine

1 Answers1