Playing a video with captions in Jupyter notebook

Question

How to play a video with captions in Jupyter notebook?

With code snippets from these post, I've tried to play a video inside jupyter notebook:

from IPython.display import HTML

# Show video
compressed_path = 'team-rocket.video-compressed.mp4'
mp4 = open(compressed_path,'rb').read()
data_url = "data:video/mp4;base64," + b64encode(mp4).decode()


HTML("""
<video width=400 controls>
      <source src="%s" type="video/mp4">
</video>
""" % data_url)

[out]:

When I tried to add a `.vtt` file as caption, the option appears

from IPython.display import HTML

# Show video
compressed_path = 'team-rocket.video-compressed.mp4'
mp4 = open(compressed_path,'rb').read()
data_url = "data:video/mp4;base64," + b64encode(mp4).decode()


HTML("""
<video width=400 controls>
      <source src="%s" type="video/mp4">
      <track src="team-rocket.vtt" label="English" kind="captions" srclang="en" default >
</video>
""" % data_url)

[out]:

But the subtitles/captions are not present when playing the video. How to play a video with captions in Jupyter notebook?

The files used in the examples above are on:

team-rocket.vtt
team-rocket.video-compressed.mp4 file can be found on https://colab.research.google.com/drive/1IHgo9HWRc8tGqjmwWGunzxXDVhPaGay_?usp=sharing

Have you tried loading the VTT file the same way (via data url) you did with the video? — J. M. Arnold, Mar 24 '23 at 16:31
I've just tried it by streaming the vtt in with the same base64 encoding but it didn't work. https://colab.research.google.com/drive/1IHgo9HWRc8tGqjmwWGunzxXDVhPaGay_?usp=sharing — alvas, Mar 24 '23 at 18:33
@alvas it looks like Colab is not exposing the right path to the browser to load the vtt files: https://imgur.com/a/DAV7BlP — Atalay K., Mar 26 '23 at 03:08

score 3 · Answer 1 · edited Mar 27 '23 at 05:08

It is a combination of security issues. Modern browsers forbid using local data (local subtitles file in this case) and Colab prevents you rendering embedded videos through "insecure" http protocol. So in order to display a video with subtitles you will need to create an iframe and embed there a video that is served through https (both video and subtitles).

You can test this by creating a cell with the following content:

%%html
<iframe width="800" height="500" src="https://iandevlin.com/html5/webvtt-example.html"
frameborder="0" allow="encrypted-media;" allowfullscreen />

So as I see it you will need to create a self signed certificate, add it to your browser for it to become legit, run a TLS web server in a thread to not block your notebook, create a small HTML file with the <video> tag which (the HTML file) your web server will serve over HTTPS and its URL will be the source of the <iframe> in the notebook. Whoa, this is really complicated.

If we try to load an full HTML into an iframe that has the video + track assets stored non-locally, these works:

%%html
<iframe width="800" height="500" src="https://iandevlin.com/html5/webvtt-example.html"
frameborder="0" allow="encrypted-media;" allowfullscreen />

or

HTML("""<iframe width="800" height="500" src="https://iandevlin.com/html5/webvtt-example.html"
frameborder="0" allow="encrypted-media;" allowfullscreen /> """)

[out]:

But when you try to take the HTML from https://iandevlin.com/html5/webvtt-example.html and load it into the iframe through srcdoc as follows, it fails to load the captions/subtitles:

%%html
<iframe width="800" height="500" 
srcdoc='
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset=utf-8>
    <title>WebVTT Video Subtitles Example for Chrome</title>
    <meta name="keywords" content="webvtt,webvtt example,chrome,html5" />
    <meta name="description" content="Sample HTML5 code that shows how Chrome supports WebVTT subtitles" />
    <style>
        body { 
            padding:10px;
            font-size:80%;
            font-family:verdana, serif;
        }
        figure { margin:30px 0; }
    </style>
</head>
<body>
    <header>
        <h1>WebVTT Video Subtitles Example for Chrome</h1>
        <p><a href="http://www.iandevlin.com/blog/2012/06/html5/google-chrome-supports-webvtt-subtitles">Read the full article</a></p>
        <p>To view this in Google Chrome, you must first enable the <code>track</code> element:</p>
        <ol>
            <li>Go to the configuration tools in Chrome by typing <code>chrome:flags</code> in the address bar</li>
            <li>Search for "Enable the &lt;track&gt; element" and activate it</li>
            <li>Restart the browser</li>
        </ol>
    </header>
    <figure>
        <video controls preload="metadata" poster="http://html5multimedia.com/media/elephants-dream.title.jpg" width="422" height="253">
            <source src="http://html5multimedia.com/media/elephants-dream-medium.webm" type="video/webm">
            <track label="English subtitles" kind="subtitles" srclang="en" src="https://iandevlin.com/html5/english-subtitles.vtt" default>
        </video>
        <figcaption>
            <small><a href="http://www.elephantsdream.org">Elephant's Dream</a> &ndash; &copy; copyright 2006, Blender Foundation / Netherlands Media Art Institute</small>
        </figcaption>
    </figure>
</body>
</html>' />

And taking a closer look at the console following @atalay-k's comment, you will see:

Unsafe attempt to load URL https://iandevlin.com/html5/english-subtitles.vtt from frame with URL about:srcdoc. Domains, protocols and ports must match.

which looks like a pesky issue from Security error loading subtitles on HTML video

Try changing protocol of the video source to `https` in ``. – igrinis Mar 27 '23 at 15:46 — igrinis, Mar 27 '23 at 15:46

Mazhar · Accepted Answer · 2023-04-02T12:44:00.997

3

Try this:

from IPython.display import HTML
from base64 import b64encode

video_path = 'team-rocket.video-compressed.mp4'
captions_path = 'team-rocket.vtt'

with open(video_path, 'rb') as f:
    video_data = f.read()
    video_base64 = b64encode(video_data).decode()

with open(captions_path, 'r') as f:
    captions_data = f.read()
    captions_base64 = b64encode(captions_data.encode('utf-8')).decode()

video_html = f"""
<video width="640" height="360" controls>
    <source src="data:video/mp4;base64,{video_base64}" type="video/mp4">
    <track src="data:text/vtt;base64,{captions_base64}" kind="captions" srclang="en" label="English" default>
</video>
"""

HTML(video_html)

For some reason, explicitly streaming in the video + captions/subtitle while specifying the encoding would bypass the security issues that @igrinis' answer pointed out.

"data:video/mp4;base64,{video_base64}"
"data:text/vtt;base64,{captions_base64}"

edited Apr 02 '23 at 12:44

answered Mar 31 '23 at 04:46

Mazhar

1,044
6
11

You should add some explanation within your Answer. Otherwise it just looks like a code dump. For example think about: Is it tested and proven to work? How/Why is it better than the other's Answer's logic (you could explain it avoids needing a custom SSL certificate when using a direct VTT file url inside a `` tag... well that's If it even does that?)... Just some advice for a better Answer. Maybe you gave a correct solution but a random code dump with no context/explanation will be harder to win a bounty award. **PS:** Do not reply these words, I will delete this comment soon. – VC.One Mar 31 '23 at 10:01
This works! I'll be good to understand how the streaming + encoding specification bypass those security issues. – alvas Mar 31 '23 at 10:23

Playing a video with captions in Jupyter notebook

How to play a video with captions in Jupyter notebook?

When I tried to add a .vtt file as caption, the option appears

But the subtitles/captions are not present when playing the video. How to play a video with captions in Jupyter notebook?

2 Answers2

When I tried to add a `.vtt` file as caption, the option appears