AttributeError: Can't pickle local object 'pre_datasets..' when implementing Pytorch framework

Question

I was trying to implement a pytorch framework on CNN.
I'm sure the code is right because it's from a tutorial and it works when I ran it on Jupyter Notebook on GoogleDrive.
But when I tried to localize it as a .py file, it suggest an error:
AttributeError: Can't pickle local object 'pre_datasets.<locals>.<lambda>' I know it's about inferencing objects outside a function, but what was the exact matter about this error?
And how should I fix it?

Here's the major part of the code.

def pre_datasets():
    TRAIN_TFM = transforms.Compose(
        [
            transforms.Resize(size=(128, 128)),
            # TODO
            transforms.ToTensor(),
        ]
    )
    train_set = DatasetFolder(
        root=CONFIG["train_set_path"],
        loader=lambda x: Image.open(x),
        extensions="jpg",
        transform=TRAIN_TFM,
    )
    train_loader = DataLoader(
        dataset=train_set,
        batch_size=CONFIG["batch_size"],
        shuffle=True,
        num_workers=CONFIG["num_workers"],
        pin_memory=True,
    )
    return train_loader

def train(train_loader):
    ...
    for epoch in range(CONFIG["num_epochs"]):
    ...
        for batch in train_loader: # error happened here
    ...

if __name__ == "__main__":
    train_loader = pre_datasets()
    train(train_loader)

Here's the error message:

Traceback (most recent call last):
  File "HW03_byCRZ.py", line 197, in <module>
    train(train_loader, valid_loader)
  File "HW03_byCRZ.py", line 157, in train
    for batch in train_loader:
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 355, in __iter__
    return self._get_iterator()
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 301, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 914, in __init__
    w.start()
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'pre_datasets.<locals>.<lambda>'

I've solved the problem. I avoid the problem by directly replacing the lambda x:Image.open(x) part with Image.open. — ceezous, Aug 11 '21 at 06:28

score 9 · Answer 1 · answered Dec 28 '21 at 03:55

9

I had a similar issue and I used dill like this:

import dill as pickle

and it worked out of the box!

answered Dec 28 '21 at 03:55

Anastasios Andronidis

6,310
4
30
53

2

Just to make this more clear, `dill` is serializing the lambda function out of the box where as `pickle` does not do this. This fixed it for me as well. – turtlehouse Oct 20 '22 at 23:19

D. ACAR · Answer 2 · 2021-08-06T11:06:20.180

5

Pickle'ing a lambda function requires extra modifications (serializing the function) see Can Python pickle lambda functions? for more information on how to do it.

edited Aug 06 '21 at 11:06

answered Aug 06 '21 at 11:00

D. ACAR

290
1
9

1

I've solved the problem. I avoid the problem by directly replacing the `lambda x:Image.open(x)` part with `Image.open`. But the pickle'ing a function part helped me a lot about understanding what's going on. – ceezous Aug 11 '21 at 06:26
2

Oh thats nice to hear. Also have you tried adding ```import dill``` to the beginning of your code. It would possibly solve that issue. – D. ACAR Aug 11 '21 at 06:38
Yeah, I tried that, but the console said that I had to do more changes. So I edit the code as the simpler way as I mentioned. – ceezous Aug 20 '21 at 07:03

AttributeError: Can't pickle local object 'pre_datasets..' when implementing Pytorch framework

2 Answers2

Linked