0

We have built a system where videos are stored in mongodb. The videos are each a couple of hundred megabytes in size. The system is built in python3 using mongoengine. The c extensions of pymongo and bson are installed.

The definition of the mongoengine documents is:

class VideoStore(Document, GeneralMixin):
    video = EmbeddedDocumentListField(SingleFrame)
    mutdat = DateTimeField()
    _collection = 'VideoStore'

    def gen_video(self):
        for one_frame in self.video:
            yield self._get_single_frame(one_frame)

    def _get_single_frame(self, one_frame):
        if one_frame.frame.tell() != 0:
            one_frame.frame.seek(0)
        return pickle.loads(one_frame.frame.read())


class SingleFrame(EmbeddedDocument):
    frame = FileField()

Reading a video in Linux takes about 3 to 4 seconds. However running the same code in Windows takes 13 to 17 seconds.

Does anyone out there have any experience with this problem and any kind of solution?

I have thought of and tested (to no avail):

  1. increasing the chunksize
  2. reading the video as a single blob without using yield
  3. Storing the file as a single blob (so no storing of separate frames)
Dick Kniep
  • 518
  • 4
  • 11
  • is this run as windows service? are you running this code on windows server or workstation? is NUMA enabled? – profesor79 May 30 '16 at 11:29
  • Does hardware on both machine is comparatively same? Windows version and disk format type? NTFS? – Saleem May 30 '16 at 11:55
  • I have tested this behaviour both on a laptop (Windows 7 Core i7 with normal harddrive) and on a virtual machine (virtualbox Windows 10) with an SSD. I did not know about NUMA, I will certainly give it a try. – Dick Kniep May 31 '16 at 09:29
  • It is not run as a windows service, the mongo daemon is started from the application. – Dick Kniep May 31 '16 at 09:30

1 Answers1

0

Use Linux, Windows is poorly supported. The use of "infinite" virtual memory among other things causes issues with Windows variants. This thread elaborates further: Why Mongodb performance better on Linux than on Windows?

Community
  • 1
  • 1
Indigo8
  • 590
  • 1
  • 4
  • 10
  • Unfortunately, the system is required to run on Windows, as most users will have a Windows machine and not a Linux system. If I could choose I would most certainly choose Linux. – Dick Kniep May 31 '16 at 09:31