MemoryError on large files with base64

Question

I am faced with a problem that I am hoping to get some insight on. I have encrypted videos in my web application that decrypts the videos in to a BytesIO memory object then base64 encoded, allowing the video to be played through an HTML5 video player for the user.

The video itself should never really be decrypted on the file system (per say). However if I attempt to load video files greater then 50MB it fails with a "MemoryError" message in web2py. How can I work around this issue?

(I believe it is failing in the base64.b64encode method)

I am developing this using Python27 and Web2py

import io
import base64

cry_upload = request.args(0)
cry = db(db.video_community.id == int(cry_upload)).select(db.video_community.ALL).first()
if cry:
    if auth.user and auth.user.id not in (cry.community.members):    
        return redirect(URL('default', 'cpanel'), client_side = True)

    filepath = os.path.join(request.folder, 'uploads/', '%s' % cry.file_upload)
    form = FORM(LABEL('The Associated Key'), INPUT(_name='key', value="", _type='text', _class='string'), INPUT(_type='submit', _class='btn btn-primary'), _class='form-horizontal')
    if form.accepts(request, session):
        outfile = io.BytesIO()
        with open(filepath, 'rb') as infile:
            key = str(form.vars.key).rstrip()
            try:
                decrypt(infile, outfile, key)
            except:
                session.flash = 'Decryption error. Please verify your key'
                return redirect(URL('default', 'cpanel'), client_side = True)
        outfile.seek(0)
        msg = base64.b64encode(outfile.getvalue().encode('utf-8'))
        outfile.close()
        response.flash = 'Success! Your video is now playable'
        return dict(form=form, cry=cry, msg=msg)

Any advice is greatly appreciated.

The decryption function

Courtesy of How to AES encrypt/decrypt files using Python/PyCrypto in an OpenSSL-compatible way?

from hashlib import md5
from Crypto.Cipher import AES
from Crypto import Random
from Crypto.Random import random

def derive_key_and_iv(password, salt, key_length, iv_length):
    d = d_i = ''
    while len(d) < key_length + iv_length:
        d_i = md5(d_i + password + salt).digest()
        d += d_i
    return d[:key_length], d[key_length:key_length+iv_length]



def decrypt(in_file, out_file, password, key_length=32):
    bs = AES.block_size
    salt = in_file.read(bs)[len('Salted__'):]
    key, iv = derive_key_and_iv(password, salt, key_length, bs)
    cipher = AES.new(key, AES.MODE_CBC, iv)
    next_chunk = ''
    finished = False
    while not finished:
        chunk, next_chunk = next_chunk, cipher.decrypt(in_file.read(1024 * bs))
        if len(next_chunk) == 0:
            padding_length = ord(chunk[-1])
            if padding_length < 1 or padding_length > bs:
               raise ValueError("bad decrypt pad (%d)" % padding_length)
            # all the pad-bytes must be the same
            if chunk[-padding_length:] != (padding_length * chr(padding_length)):
               # this is similar to the bad decrypt:evp_enc.c from openssl program
               raise ValueError("bad decrypt")
            chunk = chunk[:-padding_length]
            finished = True
        out_file.write(chunk)

Here is the video player in the view

{{if msg != None:}}
    <div class="row" style="margin-top:20px;">
        <div class="video-js-box">
            <video class="video-js vjs-default-skin" data-setup='{"controls": true, "autoplay": false, "preload": "none", "poster": "https://www.cryaboutcrypt.ninja/static/crying.png", "width": 720, "height": 406}'>
                    <source type='video/mp4; codecs="avc1.42E01E, mp4a.40.2"' src="data:video/mp4;base64,{{=msg}}" />

            </video>
       </div>
    </div>
{{pass}}

As noted I am passing the base64 encoded BytesIO data to the player. @Anthony - I am not 100% sure on what you are suggesting.

Here is the MemoryError that occurs

File "/usr/lib/python2.7/base64.py", line 53, in b64encode
    encoded = binascii.b2a_base64(s)[:-1]
MemoryError

How are you determining it's a memory issue? What is this hosted on? How much memory do you have available to your application? How many concurrent users do you have? — Demian Brecht, Nov 06 '15 at 05:12
Added the decryption method. @Demian - I receive a MemoryError when loading a video greater then suggested size. I am the only user of this web application. I don't know the specs of my server off hand. — Maurth, Nov 06 '15 at 05:12
Are you running your Python script via PHP? Because if I'm not mistaken PHP has restrictions configured for how much memory a subprocess can execute under. I quite `"This sets the maximum amount of memory in bytes that a script is allowed to allocate. This helps prevent poorly written scripts for eating up all available memory on a server."` — Torxed, Nov 06 '15 at 07:59
What are you doing with the encoded video data in the view? Rather than base64 encoding, could you instead stream the decrypted file object directly via a separate request (e.g., if the main view includes a video player, provide the player with a URL to request the streamed video)? — Anthony, Nov 06 '15 at 17:16
@Anthony - I have updated my information to show the videoplayer in the view. *cheers — Maurth, Nov 06 '15 at 20:25

score -1 · Answer 1 · answered Nov 06 '15 at 23:12

Instead of embedding the base 64 encoded video file in the HTML of the page, maybe provide a URL as the video player source and stream the file from that URL:

        <source type='video/mp4; codecs="avc1.42E01E, mp4a.40.2"'
         src="{{=URL('default', 'video', args=request.args(0))}}" />

Then most of your code would move to the video function in the default.py controller. Instead of base 64 encoding the video, simply return outfile via response.stream().

MemoryError on large files with base64

1 Answers1