0

Goal: I have an application that allows users to upload videos, and then I need to be able to stream those videos to other users immediately. My current setup works fine for this, but I now want to transcode the videos so they are optimized for mobile streaming. Looking for some best practices, please help!

Current setup: I'm using an S3 bucket to store the video asset itself. on successful upload I'm creating a separate video object elsewhere with all the metadata for that particular video, including the URL to retrieve it. *Note that in reality I'm streaming via AWS CloudFront, so the URL I'm saving as an attribute on the video object is actually the base URL for the CloudFront distribution that is hooked up to the S3 bucket.

Problem: Now I want to transcode the videos for mobile-optimized streaming. I can run a background job using elastic transcoder that takes them from one AWS bucket (call it "bucket1"), transcodes them, and drops them into another bucket designated for the optimized videos (call it "bucket2"). But my problem is, now when a user uploads a video, the flow looks like this: Upload to bucket1, and create a corresponding video object with metadata. The video object needs a URL so I can stream the video later. I set the URL associated with bucket2 (where the optimized video asset will eventually be placed). The problem is, say my batch job doesn't run for another hour, but a user requests that video one minute from now. Obviously no optimized version exists at the bucket2 URL yet, so I can't stream the video.

What I've tried: Recently I thought, what if instead of creating bucket2 for the transcoded videos, I set up my background job to take videos from bucket1, transcode them, and then drop them back into the same bucket again, with the same file name (overwriting the old non-optimized version). Then I only ever have 1 file per video, so worst case scenario if a user asks for a video before the job has run on that video they'll still get content, it'll just be the non-optimized version. One problem with this is that elastic transcoder won't let me overwrite files apparently (I could probably solve this by dropping them in a second bucket, and then doing the overwrite with a copyObject operation, per this post: aws transcoder overwrite files on s3). However, a bigger problem is that this all requires one file name, including extension. That doesn't seem possible if I want to convert files from .mov format (for instance) to an hls playlist with a .m3u8 extension.

If anyone has any best practice tips to get around the problem I've descried I would be very appreciative!

Cheers, Brendan

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
bkopp
  • 478
  • 1
  • 8
  • 20
  • Sounds like the problem is waiting for your batch process to kick off. Why not just kick off the transcoder job immediately after upload? Could do this directy from an api layer or via a lambda. – duhseekoh Aug 23 '17 at 22:19

2 Answers2

0

http://www.bitcodin.com is able to do faster than realtime transcoding with immediate output of the video to users. It can also be used in Amazon AWS, e.g. following this tutorial: http://www.bitcodin.com/blog/2015/02/create-mpeg-dash-hls-content-for-amazon-s3-and-cloudfront/

Stefan Lederer
  • 453
  • 2
  • 2
0

I know this is a couple of years old but wanted to add extra info. Hopefully you have found other solutions since your post 2 years ago, but you could use an online video platform or transcoding service, both of which are built to handle exactly what you describe, plus more. There are many in the market to choose from, from free through to paid. Also, when Stefan says "faster than realtime transcoding" this means if your video is 15 mins long, the transcoding would take less than 15 mins.

Mandy
  • 1,449
  • 1
  • 11
  • 10