Chapter 20. Working with Audio and Video

Table of Contents

Media Backend Services
GStreamer Application
GStreamer Pipeline
FFmpeg Plugin

Storyboard Suite includes a media plugin for Storyboard Engine that supports the playback of several different types of audio and video formats. The media plugin provides a common action interface for creating and controlling media playback using a variety of different backend media player services.

It is the backend services that perform the decoding of content. The display of that content may be to a Storyboard External Buffer render extension which will be composited into the Storyboard application, or the service may have the ability to render to discrete hardware layers available on the platform and composited to the Storyboard application directly in hardware.

External media backend services are provided for particular operating system and hardware based on their underlying media support. Currently Storyboard provides two external media backends, which are described in the section called “Media Backend Services”.

Note

Audio and Video support is not universally available on all embedded target platforms. It is however available on MacOS, Linux and Windows desktop simulation platforms.

Video and Audio playback and control are accomplished by using a set of defined actions. When creating a media application you must include these media actions in your Storyboard project. The media actions must be installed on a project by project basis and are defined as Action Templates. The media action template is located in your Storyboard Suite installation root under the directory Samples/ActionDefinitions/media.sbat. Copy this file to the templates directory of your project in order to be able to use media actions in your project. Once you have copied the file over, you will have to close and re-open your application to see the media actions.

In order to create a Storyboard application that will play video, you must first add a control to your project containing an External Render Extension. This render extension will be the display target where the video content will be rendered, so size and position it appropriately in your application. In the properties of the external render extension you must set the Buffer Name and Object Path values. These values will be required to play the video properly and will be used to point to where the video content should be displayed.

To initiate video playback, you will use one of the media actions. You will need to add the predefined media action gra.media.new.video to your application, which will tell the media plugin to play your video. As discussed in Chapter 8, Connecting Events to Actions this action can be configured to be a response to any event.

In the action parameters, you will specify the Channel_name parameter, which will be used as the name of the Storyboard IO channel that will be created for communication with the media plugin. The channel name should not conflict with the application name and should be relevant to its use, a good suggestion might be media. Specify the project relative name of the video file to play in the parameter Media_name, for example video/myvideo.webm. Additionally fill in the parameters that will link the media command to the external buffer where the playback is to occur. For this you configure the External_buffer_name with the same name as the Buffer Name used in the external render extension and the Object_name with the same name as the Object Path. The Output_width and Output_height should also match the dimensions of the external buffer. Finally the parameter Output_depth should be set to 4.

Other media actions allow you to control the video playback operation once it has been started, including providing pause, resume, stop and seek action commands. Storyboard Designer includes a media application in its samples named Media. For more information on loading samples, refer to the section called “New Project from a Storyboard Sample”.

Media Backend Services

The media backend service does the work of decoding, playing and controlling the media based on requests from the Storyboard application over a Storyboard IO channel. The default Storyboard IO channel name is com.crank.media_backend. This value can be overwritten by setting the SBMEDIA_CHANNEL_NAME environment variable to a new value. The FFmpeg plugin is loaded automatically when it is present in the Storyboard plugins directory. If the gstreamer-backend is going to be used to play media, the FFmpeg plugin needs to be removed from the Storyboard plugins directory or else both backends will compete to service media requests.

GStreamer Application

This media backend uses the gstreamer framework to play and control audio and video files. In order to use this backend the platform must have gstreamer and the required plugins installed. It is a good idea to try and play content with the “gst-launch” application to ensure a proper installation before running gstreamer-backend. This backend also uses Storyboard IO for communication with the Storyboard application so please ensure Storyboard IO is functional and the application has the “greio” plugin loaded.

Options:

-e : Render the video content with an external buffer

-p pipeline: Use the defined gstreamer pipeline to play the media

-v: increase verbosity, debug output
                        

Action Data

The “new.audio” and “new.video” actions take an extra_data argument. This argument is a string which can contain the following options which must be separated by a “;”.

GStreamer Pipeline

You can specify the gstreamer pipeline used to play the particular media by either passing it on the command line to gstreamer-backend with the -p option or by passing it to the actions. The pipeline can be passed in as:

“pipeline:[your pipeline]”

This pipeline can be similar to the one used with the gst-launch application with a few minor modifications. In order to allow the changing of the media file the first part of the pipeline must contain a named filesrc element as follows:

“pipeline:filesrc location=video.mov name=media-src”

Doing this will allow the code to find the named element and replace the location with a new video file.

FFmpeg Plugin

This is a plugin to Storyboard which uses the FFmpeg libraries to play and control audio and video files. In order to use this backend, the plugin must be included with in the runtime engine. You can play a video from the Storyboard Designer Simulator, as well as using the Storyboard Engine on supported platforms. Note that at this time, FFmpeg ships with only WebM video format support and Ogg audio format support.

Action Data

The “new.audio” and “new.video” actions take an extra_data argument. When using FFmpeg, no extra data is required and this argument can be empty.