Streaming video to iOS devices

It seems that neither iOS devices, nor Safari on OS X, support mp4; so if you’re trying to stream video, you need to provide another format.

The recommendation is to use HLS, which fortunately is supported by ffmpeg, you merely need to adjust your incantation:

app.get('/hls/video', function(req, res) {
    res.contentType('application/vnd.apple.mpegurl');
    var proc = ffmpeg();
    proc.stdout.pipe(res);
});
 
function ffmpeg() {
    var cmd = "ffmpeg";
    var filter = "some complex filter expr";
    var args = ["-i", "video1.mp4"];
    ...
    args.push(
        "-vcodec", "libx264",
        "-f", "hls",
        "-hls_time", "9",
        "-hls_list_size", "0",
        "-profile:v", "baseline",
        "-level", "3.0",
        "pipe:1"
    );
    return spawn(cmd, args);
}

I could probably use conneg to decide which format to return, rather than the uri, but I’m not convinced that my caching infrastructure (varnish and cloudfront now!) would handle that correctly.

Streaming video from ffmpeg

ffmpeg is a fantastic tool for converting, concatenating, or otherwise fiddling with video content. If you can generate what you need in advance, then you can upload it to s3 (or some other CDN like option); but sometimes you need to stream video on demand.

First, you need to get the ffmpeg incantation right:

ffmpeg -i video1.mp4 -i video2.mp4 ... \
       -filter_complex "something something" \
       -movflags "frag_keyframe+empty_moov"
       -f mp4 pipe:1

Using pipe:1 sends the output to stdout (as documented here), and the movflags are needed to allow streaming in mp4 format.

With this in place, it’s easy to pipe the output data to a browser using expressjs:

#!/usr/bin/env node
"use strict";

const express = require('express');
const { spawn } = require('child_process');

var app = express();

app.get('/video', function(req, res) {
    res.contentType('video/mp4');
    var proc = ffmpeg();
    proc.stdout.pipe(res);

    res.on("close", () => {
        proc.kill("SIGKILL");
    });
});

app.listen(4000);

function ffmpeg() {
    var cmd = "ffmpeg";
    var filter = "some complex filter expr";
    var args = ["-i", "video1.mp4"];
    ...
    args.push(
        "-filter_complex", filter,
        "-s", "1280x720",
        "-acodec", "aac",
        "-vcodec", "h264",
        "-movflags", "frag_keyframe+empty_moov",
        "-f", "mp4",
        "pipe:1"
    );
    return spawn(cmd, args);
}

I found I ended up with “zombie” ffmpeg processes, if the client connection had closed before the video ended (because it’s still trying to write data to the pipe?). There’s probably a neater way to solve that, but kill -9 is pretty effective!

At this point, you can stream some video; but this solution won’t scale, every request runs the ffmpeg command, which is pretty cpu intensive. It would be relatively simple to cache the output, either in memory or on disk; or you could put the node process behind a caching proxy.

However, the best solution for our needs seemed to be using this as an “origin server” for Cloudfront. This means that only the first request (of each type) hits our server, and the response is cached for as long as necessary.

If you need to render different videos for every request, then you’ll just have to throw money at it!