Quick start

This guide explains creating a basic compositor setup. We will take two input videos and produce an output video with both of them side-by-side.

Configure inputs and output

Start the compositor

React

import LiveCompositor from "@live-compositor/node"

async function start() {
  const compositor = new LiveCompositor();
  await compositor.init();
}

HTTP

Start the compositor server. Check out configuration page for available configuration options.

Membrane Framework

Following code snippets are implementing handle_init/2 or handle_setup/2 callbacks. Those are just examples, you can use any Membrane.Pipeline callbacks instead.

alias Membrane.LiveCompositor

def handle_init(ctx, opts) do
  spec = [
    ...

    child(:live_compositor, %LiveCompositor{
      framerate: {30, 1}
    }),

    ...
  ]
  {[spec: spec], %{}}
end

Register input stream input_1.

React

await compositor.registerInput("input_1", {
  type: "rtp_stream",
  transportProtocol: "tcp_server",
  port: 9001,
  video: {
    decoder: "ffmpeg_h264"
  },
  audio: {
    decoder: "opus"
  }
})

After registerInput call is done you can establish the connection and start sending the stream. Check out how to deliver input streams to learn more.

HTTP

POST: /api/input/input_1/register
Content-Type: application/json

{
  "type": "rtp_stream",
  "transport_protocol": "tcp_server",
  "port": 9001,
  "video": {
    "decoder": "ffmpeg_h264"
  }
  "audio": {
    "decoder": "opus"
  }
}

After receiving the response you can establish the connection and start sending the stream. Check out how to deliver input streams to learn more.

Membrane Framework

def handle_init(ctx, opts) do
  spec = [
    ...

    video_input_1_spec
    |> via_in(Pad.ref(:video_input, "video_input_1"))
    |> get_child(:live_compositor),

    audio_input_1_spec
    |> via_in(Pad.ref(:audio_input, "audio_input_1"))
    |> get_child(:live_compositor),

    ...
  ]
  {[spec: spec], %{}}
end

where video_input_1_spec and audio_input_1_spec are elements producing H264 video and Opus audio respectively.

Register input stream input_2.

React

await compositor.registerInput("input_2", {
  type: "rtp_stream",
  transportProtocol: "tcp_server",
  port: 9002,
  video: {
    decoder: "ffmpeg_h264"
  },
  audio: {
    decoder: "opus"
  }
})

After registerInput call is done you can establish the connection and start sending the stream. Check out how to deliver input streams to learn more.

HTTP

POST: /api/input/input_2/register
Content-Type: application/json

{
  "type": "rtp_stream",
  "transport_protocol": "tcp_server",
  "port": 9002,
  "video": {
    "decoder": "ffmpeg_h264"
  },
  "audio": {
    "decoder": "opus"
  }
}

After receiving the response you can establish the connection and start sending the stream. Check out how to deliver input streams to learn more.

Membrane Framework

def handle_init(ctx, opts) do
  spec = [
    ...

    video_input_2_spec
    |> via_in(Pad.ref(:video_input, "video_input_2"))
    |> get_child(:live_compositor),

    audio_input_2_spec
    |> via_in(Pad.ref(:audio_input, "audio_input_2"))
    |> get_child(:live_compositor),

    ...
  ]
  {[spec: spec], %{}}
end

where video_input_2_spec and audio_input_2_spec are elements that produce H264 video and Opus audio respectively.

Register output stream output_1.

Configure it to:

• render an empty View component with a background color set to #4d4d4d (gray)
• produce silent audio

React

function App() {
  return <View backgroundColor="#4d4d4d"/>
}

async function start() {
  // init code from previous steps

  await compositor.registerOutput("output_1", {
    type: "rtp_stream",
    transportProtocol: "tcp_server",
    port: 9003,
    video: {
      resolution: { width: 1280, height: 720 },
      encoder": {
        type: "ffmpeg_h264",
        preset: "ultrafast"
      },
      root: <App />
    },
    audio: {
      encoder: {
        type: "opus",
        channels: "stereo"
      },
    }
  })
}

After registerOutput is done you can establish the connection and start listening for the stream. Check out how to receive output streams to learn more.

View component does not have any children, so on the output you should see just a blank screen of a specified color as shown below. There are no InputStream components in the scene and useAudioInput hook was not used, so output audio will be silent.

HTTP

POST: /api/output/output_1/register
Content-Type: application/json

{
  "type": "rtp_stream",
  "transport_protocol": "tcp_server",
  "port": 9003,
  "video": {
    "resolution": { "width": 1280, "height": 720 },
    "encoder": {
      "type": "ffmpeg_h264",
      "preset": "ultrafast"
    },
    "initial": {
      "root": {
        "type": "view",
        "background_color_rgba": "#4d4d4dff"
      }
    }
  },
  "audio": {
    "encoder": {
      "type": "opus",
      "channels": "stereo"
    },
    "initial": {
      "inputs": []
    }
  }
}

After receiving the response you can establish the connection and start listening for the stream. Check out how to receive output streams to learn more.

View component does not have any children, so on the output you should see just a blank screen of a specified color as shown below. The initial.inputs list in audio config is empty, so the output audio will be silent.

Membrane Framework

def handle_init(ctx, opts) do
  spec = [
    ...

    get_child(:live_compositor),
    |> via_out(Pad.ref(:video_output, "video_output_1"), options: [
      width: 1280,
      height: 720,
      encoder: %LiveCompositor.Encoder.FFmpegH264{
        preset: :ultrafast
      },
      initial: %{
        root: %{
          type: :view,
          background_color_rgba: "#4d4d4dff",
        },
      }
    ])
    |> video_output_1_spec,

    get_child(:live_compositor)
    |> via_out(Pad.ref(:audio_output, "audio_output_1"), options: [
      encoder: LiveCompositor.Encoder.Opus.t(),
      initial: %{
        inputs: []
      }
    ])
    |> audio_output_1_spec

    ...
  ]
  {[spec: spec], %{}}
end

where video_output_1_spec and audio_output_1_spec are elements that can consume H264 video and Opus audio respectively.

You can configure output framerate and sample rate using framerate and output_sample_rate bin options.

View component does not have any children, so on the output you should see just a blank screen of a specified color as shown below. The initial.inputs list in audio config is empty, so the output audio will be silent.

Output stream

Update output

Configure it to:

• Show input streams input_1 and input_2 using Tiles component.
• Mix audio from input streams input_1 and input_2, where input_1 volume is slightly lowered.

React

function App() {
  return (
    <Tiles backgroundColor="#4d4d4d">
      <InputStream inputId="input_1" volume={0.9} />
      <InputStream inputId="input_2" />
    </Tiles>
  )
}

HTTP

POST: /api/output/output_1/update
Content-Type: application/json

{
  "video": {
    "root": {
      "type": "tiles",
      "background_color_rgba": "#4d4d4dff",
      "children": [
        { "type": "input_stream", "input_id": "input_1" },
        { "type": "input_stream", "input_id": "input_2" }
      ]
    }
  },
  "audio": {
    "inputs": [
      { "input_id": "input_1", volume: 0.9 },
      { "input_id": "input_2" }
    ]
  }
}

Membrane Framework

def handle_setup(ctx, state) do
  video_request = %LiveCompositor.Request.UpdateVideoOutput{
    output_id: "video_output_1",
    root: %{
      type: :tiles,
      children: [
        %{ type: :input_stream, input_id: :input_1 },
        %{ type: :input_stream, input_id: :input_2 }
      ]
    }
  }
  audio_request = %LiveCompositor.Request.UpdateAudioOutput{
    output_id: "audio_output_1",
    inputs: [
      %{ input_id: "input_1", volume: 0.9 },
      %{ input_id: "input_2" }
    ]
  }

  events = [
    notify_child: {:live_compositor, video_request},
    notify_child: {:live_compositor, audio_request}
  ]

  {events, state}
end

Output stream