2

i write a application/Gui in PyQt5 and would like store large Images (>5000 RGB Images)

Now, I have a function, which store every Picture with cv2.imwrite, but this process takes a lot of time. So i read here in Stackoverflow, that I can do this with multiprocessing. But I´m very new in python.

My Multiprocessing Code: 

def SaveImages(self):
    jobs = []
    for i in range(5):
        p = multiprocessing.Process(target = self.SaveAllImages, args=self)
        jobs.append(p)
        p.start()

In Function SaveAllImages are the basic Code for storing Images per Frame. If I run this Code - there are a Error: 

p = multiprocessing.Process(target = SaveAllImages, args=self)
NameError: name 'SaveAllImages' is not defined

But SaveAllImages are definied: def SaveAllImages(self)

So my Question are:

  1. Why i became this error

  2. How can I implement a very simply multiprocessing for storing Images

eyllanesc
  • 235,170
  • 19
  • 170
  • 241
Mr. Fkcm95
  • 103
  • 1
  • 11
  • 1
    Hello and welcome to StackOverflow. Please provide us a [**minimal** and **reproducible** example](https://stackoverflow.com/help/minimal-reproducible-example) we can work with to better understand what is your problem. – musicamante Aug 23 '19 at 19:47
  • 3
    Multiprocessing is more for processing than storing... can you take a step further back and say what you are actually trying to achieve please? – Mark Setchell Aug 23 '19 at 21:16
  • Your task is I/O bound, not CPU bound, and so it is unlikely to be improved by multiprocessing. Multiprocessing also requires a copy of the process to be spawned for each new worker (which is expensive on Windows, since Python on Windows lacks a fork implementation), so it's likely to do more harm than good. – Alex Huszagh Aug 23 '19 at 22:04
  • 1
    The code block you show does not match the error you show - the former uses ``self.SaveAllImages``, the latter just ``SaveAllImages``. Which one is it? – MisterMiyagi Aug 24 '19 at 06:12

2 Answers2

4

The error you're seeing is because you're calling a method that doesn't exist, probably because it's not part of self.

You'll likely see better performance with multithreading than multiprocessing. Multiprocessing is best for CPU-bound tasks for the simple reason that Python uses a global lock for all operations. Multiprocessing is a hack to get around this lock. It's nastier to work with than threading and it's best to avoid it unless absolutely necessary.

Multithreading is likely enough for your use case and it won't create lots of gotchas for a new programmer. Here's a working sample setup using Python 3's Futures API that will easily scale your problem size, just add your arguments and actual save code in the marked places.

import concurrent.futures

# Save single image
def save_image(image_arg):
    # your image save code goes here
    print("Working on image {}...".format(image_arg))
    return True

# max_workers specifies the number of threads. If None then use 5x your CPU count
with concurrent.futures.ThreadPoolExecutor(max_workers=None) as executor:
    # Images we'll save. Depending on how you generate your images you might not
    # want to materialize a list like this to avoid running out of memory.
    image_args = ["image1", "image2", "image3"]

    # Submit futures to the executor pool.
    # Map each future back to the arguments used to create that future. That way
    # if one fails we know which image it was that failed.
    future_to_args = {executor.submit(save_image, image_arg): image_arg for image_arg in image_args}

    # Images are being saved in worker threads. They will complete in any order.
    for future in concurrent.futures.as_completed(future_to_args):
        image_arg = future_to_args[future]
        try:
            result = future.result()
        except Exception as exc:
            print("Saving image {} generated an exception: {}".format(image_arg, exc))
        else:
            print("Image {} saved successfully.".format(image_arg))

If you insist on multiprocessing, just use ProcessPoolExecutor instead. That might be worthwhile if you also want to generate your images in parallel.

Whether ThreadPoolExecutor or ProcessPoolExecutor is better depends a lot on what the rest of your workload is and how you structured it. Try both to see which works better for you. Note that multiprocessing places restrictions on communication and sharing state between workers, hence why I suggest trying threads first.

Adam
  • 16,808
  • 7
  • 52
  • 98
  • Multi-threading is mostly for pseudo-parallelism, not for performance improvements. Multi-threading and multiprocessing are unlikely to improve performance here (which is critical to what the OP wanted), since the task is fundamentally I/O-bound. – Alex Huszagh Aug 23 '19 at 22:42
  • 1
    @AlexanderHuszagh multithreading *is* likely to improve performance here precisely because OP's workload is I/O-bound. That's a major use case for multithreading in Python. The I/O calls release the GIL. – Adam Aug 23 '19 at 22:46
  • it's not to improve performance, it's to allow you to run other tasks while performing an I/O-bound task. Sure, the I/O releases the GIL, but writing to disk is fundamentally performance-limited regardless of how many threads are writing to disk. https://stackoverflow.com/questions/902425/does-multithreading-make-sense-for-io-bound-operations – Alex Huszagh Aug 23 '19 at 22:48
  • I/O tasks are intrinsically bound to a single, serialized operation, so you cannot improve performance with multiple threads in a single process, or multiple processes. If your fundamental limit is I/O-bound, think about reducing the amount of data you need to write to disk, or get a faster hard drive (SSD rather than HDD, for example). – Alex Huszagh Aug 23 '19 at 22:50
  • The only situation multiple threads or multiple processes would make sense is writing to multiple disks, since this is a workload that can be parallelized. – Alex Huszagh Aug 23 '19 at 22:51
  • if you're not going to listen to reason, how about a simple example? https://gist.github.com/Alexhuszagh/7a71cb348049c65d08296d68508cb3d6 Threads are great for speeding up processing before it is serialized to I/O, but cannot intrinsically speed up a serial task. Up the number of iterations and you will see there is no performance difference between the two. – Alex Huszagh Aug 23 '19 at 23:09
  • I would love to help the OP improve their code: part of that is avoiding complexity for the sake of complexity. Multithreading and multiprocessing adds complexity, and it does not improve I/O-bound tasks. – Alex Huszagh Aug 23 '19 at 23:11
  • 1
    @AlexanderHuszagh If that's what your benchmarks look like then no wonder you have wrong ideas. 8MB for testing I/O? writing the same file so it hits OS caches? And yet the parallel version is still 25% faster on my machine? And twice as fast for a larger buffer? – Adam Aug 23 '19 at 23:56
  • 1
    @AlexanderHuszagh this isn't complexity for the sake of complexity. OP is trying to do 5000 expensive things. Parallelism is definitely going to help them. They are struggling with getting something to work. Most likely they will parallelize their whole operation. Take your poor attitude and put it into your own answer if you like your strong opinions. I'm done with this. – Adam Aug 23 '19 at 23:58
  • I mean, it's a simple contrived example. And no, I ran it numerous times, it's not running 25% faster in parallel. If you want me to create a new file name every iteration, sure. But fundamentally, parallelism will only speed up mixed or CPU bound tasks. – Alex Huszagh Aug 23 '19 at 23:58
  • @Adam parallelism _**may**_ help you getting better performances, but it *doesn't* mean that you *always* get a *better performance*, as that depends on how the threading is implemented in the "framework", accordingly with how the OS works with it, and how the whole low level CPU/core/thread management is done, which also relates to (asynchronous) IO access. – musicamante Aug 24 '19 at 01:51
  • @musicamante I know that! I never said parallelism is always better! But OP asked how to do something, so I answered, with what the caveats are. But note that OP said they're using `cv2.imwrite`. That's not just "write bytes". I would bet very good money that they WILL see a speedup, even with simple threads, simply because writing bytes to the device is almost never the bottleneck. Formatting is. My example code will parallelize their formatting code, so I fully expect them to see an improvement even if they don't extend that to the rest of their program. – Adam Aug 24 '19 at 02:14
  • I've parallelized a lot of I/O programs with the above approach and it works very well. If you guys have a better way then please offer it. – Adam Aug 24 '19 at 02:15
  • 1
    @Adam I never wrote that parallelism is "worst" (and, AFAIU) neither did Alexander. But I feel that we always should be clear on how some approach is **or** could be better against another one, not only according to our experience and common knowledge, but also respecting what we can assume on somebody else's. We don't know their experience, background and knowledge, nor hw/sw backend or specific configuration. So, both theories *can* be true, but I think we should clarify both aspects/realities for the sake of the OP *and* every other user that will face a similar issue in the future. – musicamante Aug 24 '19 at 03:52
1

Before you try to improve, you should always measure performance.

Use a disk testing program to see what the maximum sustained write throughput of your disk is.

Then use a performance monitoring program to check the write throughput that your program generates (without multi- threading/processing). If your program can reach the same throughput as the test program most of the time, then there is little you can do.

Assuming that you are using a regular harddisk, the best way to improve write performance is to use an SSD instead.

Roland Smith
  • 42,427
  • 3
  • 64
  • 94