Read an video file object from S3 and use it for further processing through Opencv

Question

import boto3

import cv2
import numpy as np
s3 = boto3.resource('s3')
vid = (s3.Object('bucketname', 'video.blob').get()['Body'].read())
cap = cv2.VideoCapture(vid)

This is my code. I have a video file in an s3 bucket. I want to do some processing on it with OpenCV and I don't want to download it. So I'm trying to store that video file into vid. Now the problem is that type(vid) is byte which is the reason to result in this error TypeError: an integer is required (got type bytes) on line 6. I tried converting it into an integer or a string but was unable to.

On an attempt to convert byte to an integer: I referred to this and was getting length issues. This is just a sample video file. The actual file I want to do processing on will be huge when converted to byte object.

On an attempt to get the object as a string and then convert it to an integer: I referred to this. Even this doesn't seem to work for me.

If anyone can help me solve this issue, I will be grateful. Please comment if anything is uncertain to you regarding my issue and I'll try to provide more details.

Since you seem to want to access the video from some web resource, you probably want some string containing a URL -- integer identifiers given to `VideoCapture` constructor are used to identify local cameras only. — Dan Mašek, Oct 04 '18 at 14:33
The url of the video file is a force download url. I'm trying to make it viewable by altering the settings. Even though If I could make it displayable, I cannot access it via OpenCv because `VideoCapture` is only working for `https://www.example.com/myimage.mp4` and for video links like `https://www.youtube.com/watch?v=jNQXAC9IVRw` it won't work. Any other possible solution please? — Vinay Varma, Oct 04 '18 at 15:48
https://stackoverflow.com/questions/37555195/is-it-possible-to-stream-video-from-https-e-g-youtube-into-python-with-ope ? — Dan Mašek, Oct 04 '18 at 16:51
Thank you for the help. This is relevant however it does not exactly solve my problem. — Vinay Varma, Oct 04 '18 at 18:48

score 3 · Answer 1 · answered Jun 22 '21 at 20:49

If streaming the video from a url is an acceptable solution, I think that is the easiest solution. You just need to generate a url to read the video from.

import boto3
import cv2

s3_client = boto3.client('s3')
        
bucket = 'bucketname'      
key = 'video.blob' 
            
url = s3_client.generate_presigned_url('get_object', 
                                       Params = {'Bucket': bucket, 'Key': key}, 
                                       ExpiresIn = 600) #this url will be available for 600 seconds
    
cap = cv2.VideoCapture(url)
       
ret, frame = cap.read()

You should see that you are able to read and process frames from that url.

score -6 · Answer 2 · answered Nov 29 '18 at 11:44

Refer below the useful code snippet to perform various operations on S3 bucket.

import boto3
s3 = boto3.resource('s3', region_name='us-east-2')

for listing buckets in s3

for bucket in s3.buckets.all():
    print(bucket.name)

bucket creation in s3

my_bucket=s3.create_bucket(Bucket='Bucket Name', CreateBucketConfiguration={
    'LocationConstraint': 'us-east-2'
})

listing down objects inside bucket

my_bucket = s3.Bucket('Bucket Name')
for file in my_bucket.objects.all():
    print (file.key)

Uploading a file from current directory

import os
print(os.getcwd())

fileName="B01.jpg"
bucketName="Bucket Name"
file = open(fileName)
s3.meta.client.upload_file(fileName, bucketName, 'test2.txt')

reading image/video from bucket

import matplotlib.pyplot as plt
s3 = boto3.resource('s3', region_name='us-east-2')
bucket = s3.Bucket('Bucket Name')   # bucket name
object = bucket.Object('maisie_williams.jpg')  # image name
object.download_file('B01.jpg')            #donwload image with this name
img=plt.imread('B01.jpg')              #read the downloaded image
imgplot = plt.imshow(img)              #plot the image
plt.show(imgplot)

Reading from one bucket and then dumping it to another

import boto3
s3 = boto3.resource('s3', region_name='us-east-2')
bucket = s3.Bucket('Bucket Name')   # bucket name
object = bucket.Object('maisie_williams.jpg')  # image name
object.download_file('B01.jpg')



fileName="B01.jpg"
bucketName="Bucket Name"
file = open(fileName)
s3.meta.client.upload_file(fileName, bucketName, 'testz.jpg')

If you have access keys then you can probably do the folowing

keys = pd.read_csv('accessKeys.csv')
        #creating Session for S3 buckets
        session = boto3.session.Session(aws_access_key_id=keys['Access key ID'][0],
            aws_secret_access_key=keys['Secret access key'][0])
        s3 = session.resource('s3')

        buck = s3.Bucket('Bucket Name')