25

I have a queryset that I need to pickle lazily and I am having some serious troubles. cPickle.dumps(queryset.query) throws the following error:

Can't pickle <class 'myproject.myapp.models.myfile.QuerySet'>: it's not the same object as myproject.myapp.models.myfile.QuerySet

Strangely (or perhaps not so strangely), I only get that error when I call cPcikle from another method or a view, but not when I call it from the command line.

I made the method below after reading PicklingError: Can't pickle <class 'decimal.Decimal'>: it's not the same object as decimal.Decimal and Django mod_wsgi PicklingError while saving object:

def dump_queryset(queryset, model):
    from segment.segmentengine.models.segment import QuerySet
    memo = {}
    new_queryset = deepcopy(queryset, memo)
    memo = {}
    new_query = deepcopy(new_queryset.query, memo)
    queryset = QuerySet(model=model, query=new_query)    
    return cPickle.dumps(queryset.query)

As you can see, I am getting extremely desperate -- that method still yields the same error. Is there a known, non-hacky solution to this problem?

EDIT: Tried using --noreload running on the django development server, but to no avail.

EDIT2: I had a typo in the error I displayed above -- it was models.QuerySet, not models.mymodel.QuerySet that it was complaining about. There is another nuance here, which is that my models file is broken out into multiple modules, so the error is ACTUALLY:

 Can't pickle <class 'myproject.myapp.models.myfile.QuerySet'>: it's not the same object as myproject.myapp.models.myfile.QuerySet

Where myfile is one of the modules under models. I have an __ini__.py in models with the following line:

from myfile import *

I wonder if this is contributing to my issue. Is there some way to change my init to protect myself against this? Are there any other tests to try?

EDIT3: Here is a little more background on my use case: I have a model called Context that I use to populate a UI element with instances of mymodel. The user can add/remove/manipulate the objects on the UI side, changing their context, and when they return, they can keep their changes, because the context serialized everything. A context has a generic foreign key to different types of filters/ways the user can manipulate the object, all of which must implement a few methods that the context uses to figure out what it should display. One such filter takes a queryset that can be passed in and displays all of the objects in that queryset. This provides a way to pass in arbitrary querysets that are produced elsewhere and have them displayed in the UI element. The model that uses the Context is hierarchical (using mptt for this), and the UI element makes a request to get children each time the user clicks around, we can then take the children and determine if they should be displayed based on whether or not they are included in the Context. Hope that helps!

EDIT4: I am able to dump an empty queryset, but as soon as I add anything of value, it fails.

EDIT4: I am on Django 1.2.3

Community
  • 1
  • 1
Bacon
  • 2,155
  • 4
  • 23
  • 31
  • You're calling __magic_methods__ directly. You should import copy and use the copy.deepcopy function. – Chris Wesseling Jul 20 '11 at 07:50
  • But that won't solve your problem. You could backtrack how and when Django (re)loads your modules. And try to have them loaded in a predefined way in the top level. The django development server does all kinds of handy auto-reloading, that are not helpful to you now. Try running it with [--noreload](https://docs.djangoproject.com/en/dev/ref/django-admin/#django-admin-option---noreload). – Chris Wesseling Jul 20 '11 at 08:06
  • Maybe it could be useful if you can elaborate a bit more what the purpose of pickling the queryset is? – Bernhard Vallant Jul 20 '11 at 11:07
  • According to this post, pickle a queryset should be no problem. https://docs.djangoproject.com/en/dev/ref/models/querysets/ I guess the issue may contributing in other place. – lucemia Jul 20 '11 at 19:46
  • Well maybe you can verify two things: 1. Does the pickling work if you don't assign another query (query empty)? 2. Try to do the import like `from segment.segmentengine.models import segment`, then do `queryset = segment.QuerySet(model=model, query=new_query)` – Bernhard Vallant Jul 21 '11 at 00:12
  • I just tried it with the empty queryset and it worked! Unfortunately, no such luck with the relative import. This is progress! – Bacon Jul 21 '11 at 22:22
  • Could you post your complete stack trace and error message? – lucemia Jul 21 '11 at 23:59
  • What do you get if you do `id(QuerySet)` and `id queryset.__class__`? The same value? Different? For the minimal case where you trigger the error, repeat this for every class and instance you can find in your namespace and `QuerySet`'s internal namespace (recursive), both where you define `QuerySet` and where you attempt to pickle it. Let us know which don't match, how they're imported in each file, where that object is created and how, etc. – agf Jul 25 '11 at 05:51
  • Does this error only happen with cPickle? Have you tried it with standard Pickle? There are [other modules](http://pypi.python.org/pypi?:action=search&term=pickle&submit=search) that have similar functionality to Pickle/cPickle, such as [Gnosis Utilities](http://pypi.python.org/pypi/Gnosis%20Utilities)' Pickle, which has the benefit of generating standard XML output using the [gnosis.xml.pickle](http://gnosis.cx/download/gnosis/xml/pickle/doc/HOWTO) module. I'd try replacing cPickle with a different module to see if it has the same problem, or if it could be a bug in your cPickle implementatio – NoBugs Jul 24 '11 at 03:27

4 Answers4

7

This may not be the case for everyone, but I was using Ipython notebook and having a similar issue pickling my own class. The problem turned out to be from a reload call 

from dir.my_module import my_class    
reload(dir.my_module)

Removing the reload call and then re-running the import and the cell where the instance of that object was created then allowed it to be pickled.

J_Tuck
  • 141
  • 1
  • 5
1

not so elegant but perhaps it works: add the directory of the myfile -module to os.sys.path and use only import myfile in each module where you use myfile. (remove any from segment.segmentengine.models.segment import, anywhere in your project)

Remi
  • 20,619
  • 8
  • 57
  • 41
  • This was my problem - inconsistent usage of import statements (i imported partly from main module, partly as a local submodules). Adding main directory to `sys.path` and rewrite every `import` to start with main module solved the problem to me. – xolodec May 09 '15 at 14:26
0

According to this doc, pickling a QuerySet should be not a problem. Thus, the problem should come from other place.

Since you mentined:

EDIT2: I had a typo in the error I displayed above -- it was models.QuerySet, not models.mymodel.QuerySet that it was complaining about. There is another nuance here, which is that my models file is broken out into multiple modules, so the error is ACTUALLY:

  1. The second error message you provided look like the same as previous one, is that what you mean?
  2. The error message you provided looks weird. Since you are pickling "queryset.query", the error should related to the django.db.models.sql.Query class instead of the QuerySet class.

Some modules or classes may have the same name. They will override each other then cause this kind of issue. To make thing easier, I will recommend you to use "import ooo.xxx" instead of "from ooo import *".

lucemia
  • 6,349
  • 5
  • 42
  • 75
  • 1
    The doc you link specifies that when a queryset is pickled, it is first executed and then the results get serialized. The question is about saving the query so that it can be executed after deserialization. – Shai Berger Mar 15 '12 at 12:48
0

Your could also try 

import ooo.xxx as othername
Hassek
  • 8,715
  • 6
  • 47
  • 59