Add graphic overlays to a live stream

This article explains how to overlay images onto live streams using the Wowa Streaming Engine™ media server software Transcoder and Wowza Streaming Engine Java API. It provides the steps to create a custom Transcoder module, add a static image to the video, add text to the video, fade the image and text, and animate the image and text. The examples and classes provide a starting point for development. More elaborate overlays are possible through custom development by expanding the examples or by creating custom classes.

Prerequisites

This article is intended for developers who are familiar with Wowza Streaming Engine and are experienced with Java programming and XML. Before proceeding, do the following:

Optimize your Wowza Streaming Engine configuration. For information, see Tune Wowza Streaming Engine for optimal performance.
Set up and run Transcoder in Wowza Streaming Engine. Be sure to test basic live setup and playback of a single transcoded 360p stream by completing Test #1 and Test #2 in the "Troubleshooting tests" section of that tutorial. The overlay example in this article uses the stream name myStream.
Download the sample overlay images and helper classes used in this article: Download TranscoderOverlayExampleFiles.zip. Copy the files in the content directory that is included in TranscoderOverlayExampleFiles.zip to the content directory of your Wowza Streaming Engine installation ([install-dir]/content).

Note: If you create your own overlay images, be sure to review Overlay image requirements for the Transcoder in Wowza Streaming Engine.

Create a new Transcoder module

To overlay an image or text on a live stream, a Transcoder module must be created. This example uses the Wowza IDE.

Create a new project.
1. Use the Wowza IDE to create a new Wowza Streaming Engine Project:
  - Project name – ModuleTranscoderOverlayExample
  - Wowza Streaming Engine location – Select the install directory where Wowza Streaming Engine is installed.
  - Package – com.wowza.wms.plugin.transcoderoverlays
  - Name – ModuleTranscoderOverlayExample
2. Under Run > Run Configurations, select ModuleTranscoderOverlayExample.
3. On the Arguments tab, add the following to the VM arguments:
  
  -Dcom.wowza.wms.native.base="win"

Set up the configuration files by add the following code to [install-dir]/conf/[application-name]/Application.xml:

<Module>
    <Name>ModuleTranscoderOverlayExample</Name>
    <Description>Example Overlay</Description>
    <Class>com.wowza.wms.plugin.transcoderoverlays.ModuleTranscoderOverlayExample</Class>
</Module>

Start Wowza Streaming Engine and check for the message, "INFO server comment - onAppStart: myApplication/_definst_" in the default message log or set a break point on onAppStart.

Create a Transcoder module

The overlayExample uses the following Wowza Streaming Engine Java API base classes and interface to overlay a graphic on the video:

IliveStreamTranscoderNotify
LiveStreamTranscoderActionNotifyBase
TranscoderVideoDecoderNotifyBase

The example also includes the OverlayImage and AnimationEvents classes for quick drawing and animation. You must add these two classes to the project.

Modify the ModuleTranscoderOverlayExample class.

Import the following:

import java.awt.Color;
import java.awt.Font;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

import com.wowza.util.SystemUtils;
import com.wowza.wms.application.*;
import com.wowza.wms.amf.*;
import com.wowza.wms.client.*;
import com.wowza.wms.module.*;
import com.wowza.wms.request.*;
import com.wowza.wms.stream.*;
import com.wowza.wms.stream.livetranscoder.ILiveStreamTranscoder;
import com.wowza.wms.stream.livetranscoder.ILiveStreamTranscoderNotify;
import com.wowza.wms.transcoder.model.LiveStreamTranscoder;
import com.wowza.wms.transcoder.model.LiveStreamTranscoderActionNotifyBase;
import com.wowza.wms.transcoder.model.TranscoderSession;
import com.wowza.wms.transcoder.model.TranscoderSessionVideo;
import com.wowza.wms.transcoder.model.TranscoderSessionVideoEncode;
import com.wowza.wms.transcoder.model.TranscoderStream;
import com.wowza.wms.transcoder.model.TranscoderStreamDestination;
import com.wowza.wms.transcoder.model.TranscoderStreamDestinationVideo;
import com.wowza.wms.transcoder.model.TranscoderStreamSourceVideo;
import com.wowza.wms.transcoder.model.TranscoderVideoDecoderNotifyBase;
import com.wowza.wms.transcoder.model.TranscoderVideoOverlayFrame;

Add these member variables to the class.
```
String graphicName = "logo_${com.wowza.wms.plugin.transcoderoverlays.overlayimage.step}.png";
int overlayIndex = 1;

private IApplicationInstance appInstance = null;
private String basePath = null;
private Object lock = new Object();
```
overlayIndex
The overlayIndex defines how the overlay image is drawn relative to an overlay image defined in transcode/transcode.xml. The overlay image is drawn in the following order:
1. Encode/Source with a lower number.
2. Encode/Source with a higher number.
3. Decode/Destination with a lower number.
4. Decode/Destination with a higher number.
Where:
- Encode relates to the Root/Transcode/Encodes/Encode/Video/Overlays/Overlay/Index element in transcode/transcode.xml.
- Source relates to encodeSource=true in the module source code.
- Decode relates to the Root/Transcode/Decode/Video/Overlays/Overlay/Index element.
- Destination relates to encodeSource=false in the module source code.
A decode/destination with the highest number will be on top. An encode/source with a value of 0 will be on the bottom.

The overlay image defined in transcode/transcode.xml will not be displayed if the overlayIndex value specified in transcode/transcode.xml is the same as the overlayIndex value in the module and both are for the encode/source or both are for the decode/destination.

basePath
The basePath variable is used to store the full path name of the content directory where the graphics are located.

Modify the onAppStart method of the class.

public void onAppStart(IApplicationInstance appInstance)
{
    String fullname = appInstance.getApplication().getName() + "/" + appInstance.getName();
    getLogger().info("onAppStart: " + fullname);
    this.appInstance = appInstance;
    String artworkPath = "${com.wowza.wms.context.VHostConfigHome}/content/" + appInstance.getApplication().getName();
    Map<String, String> envMap = new HashMap<String, String>();
    if (appInstance.getVHost() != null)
    {
        envMap.put("com.wowza.wms.context.VHost", appInstance.getVHost().getName());
        envMap.put("com.wowza.wms.context.VHostConfigHome", appInstance.getVHost().getHomePath());
    }
    envMap.put("com.wowza.wms.context.Application", appInstance.getApplication().getName());
    if (this != null)
        envMap.put("com.wowza.wms.context.ApplicationInstance", appInstance.getName());
    this.basePath =  SystemUtils.expandEnvironmentVariables(artworkPath, envMap);
    this.basePath = this.basePath.replace("\", "/");
    if (!this.basePath.endsWith("/"))
        this.basePath = this.basePath+"/";
    this.appInstance.addLiveStreamTranscoderListener(new TranscoderCreateNotifierExample());
}

Create EncoderInfo as a nested class. This class is used to bundle the input and output video streams for each encoder.

class EncoderInfo
{
    public String encodeName;
    public TranscoderSessionVideoEncode sessionVideoEncode = null;
    public TranscoderStreamDestinationVideo destinationVideo = null;
    public int[] videoPadding = new int[4];
    public EncoderInfo(String name, TranscoderSessionVideoEncode sessionVideoEncode, TranscoderStreamDestinationVideo destinationVideo)
    {
        this.encodeName = name;
        this.sessionVideoEncode = sessionVideoEncode;
        this.destinationVideo = destinationVideo;
    }
}

Create the TranscoderCreateNotifierExample class as a nested class.

Create the class.

class TranscoderCreateNotifierExample implements ILiveStreamTranscoderNotify
{
}

Add required functions.

@Override
public void onLiveStreamTranscoderCreate(ILiveStreamTranscoder liveStreamTranscoder, IMediaStream stream) 
{
}

@Override
public void onLiveStreamTranscoderDestroy(ILiveStreamTranscoder arg0, IMediaStream arg1) 
{
}

@Override
public void onLiveStreamTranscoderInit(ILiveStreamTranscoder arg0, IMediaStream arg1) 
{
}

Modify the onLiveStreamTranscoderCreate method.

public void onLiveStreamTranscoderCreate (ILiveStreamTranscoder liveStreamTranscoder, IMediaStream stream)
{
    getLogger().info("ModuleTranscoderOverlayExample#TranscoderCreateNotifierExample.onLiveStreamTranscoderCreate["+appInstance.getContextStr()+"]: "+stream.getName());
    ((LiveStreamTranscoder)liveStreamTranscoder).addActionListener(new TranscoderActionNotifierExample());
}

Create the TranscoderActionNotifierExample class as a nested class.

Create the class.

class TranscoderActionNotifierExample extends LiveStreamTranscoderActionNotifyBase
{
    TranscoderVideoDecoderNotifyExample transcoder=null;
}

Create the onSessionVideoEncodeSetup method. This will create the TranscoderVideoDecoderNotifyExample class, which extends the TranscoderVideoDecoderNotifier class. This class will invoke its onBeforeScaleFrame method for each frame that passes through the system.

public void onSessionVideoEncodeSetup(LiveStreamTranscoder liveStreamTranscoder, TranscoderSessionVideoEncode sessionVideoEncode)
{
    getLogger().info("ModuleTranscoderOverlayExample#TranscoderActionNotifierExample.onSessionVideoEncodeSetup["+appInstance.getContextStr()+"]");
    TranscoderStream transcoderStream = liveStreamTranscoder.getTranscodingStream();
    if (transcoderStream != null && transcoder==null)
    {
        TranscoderSession transcoderSession = liveStreamTranscoder.getTranscodingSession();
        TranscoderSessionVideo transcoderVideoSession = transcoderSession.getSessionVideo();
        List<TranscoderStreamDestination> alltrans = transcoderStream.getDestinations();

        int w = transcoderVideoSession.getDecoderWidth();
        int h = transcoderVideoSession.getDecoderHeight();
        transcoder = new TranscoderVideoDecoderNotifyExample(w,h);
        transcoderVideoSession.addFrameListener(transcoder);

        //apply an overlay to all outputs
        for(TranscoderStreamDestination destination:alltrans)
        {
            //TranscoderSessionVideoEncode sessionVideoEncode = transcoderVideoSession.getEncode(destination.getName());
            TranscoderStreamDestinationVideo videoDestination = destination.getVideo();
            System.out.println("sessionVideoEncode:"+sessionVideoEncode);
            System.out.println("videoDestination:"+videoDestination);
            if (sessionVideoEncode != null && videoDestination !=null)
            {
                transcoder.addEncoder(destination.getName(),sessionVideoEncode,videoDestination);
            } 
        }
    }
    return;
}

Create the TranscoderVideoDecoderNotifyExample class as a nested class.

Create the class.

class TranscoderVideoDecoderNotifyExample extends TranscoderVideoDecoderNotifyBase
{
}

Add member variables.

private OverlayImage mainImage=null;private OverlayImage wowzaImage=null;
private OverlayImage wowzaText = null;
private OverlayImage wowzaTextShadow = null;
List<EncoderInfo> encoderInfoList = new ArrayList<EncoderInfo>();
AnimationEvents videoBottomPadding = new AnimationEvents();

Add the TranscoderVideoDecoderNotifyExample constructor.

public TranscoderVideoDecoderNotifyExample (int srcWidth, int srcHeight)
{
    int lowerThirdHeight = 70;
}

Add the addEncoder method.

public void addEncoder(String name, TranscoderSessionVideoEncode sessionVideoEncode, TranscoderStreamDestinationVideo destinationVideo)
{
    encoderInfoList.add(new EncoderInfo(name, sessionVideoEncode,destinationVideo));
}

Add the onBeforeScaleFrame method. This method is called for each frame that is ingested by the transcoder (source). Here you can add a graphic either to the source or to each destination.

public void onBeforeScaleFrame(TranscoderSessionVideo sessionVideo, TranscoderStreamSourceVideo sourceVideo, long frameCount)
{
    boolean encodeSource=false;
    boolean showTime=false;
    double scalingFactor=1.0;
    synchronized(lock)
    {
        if (mainImage != null)
        {
            //does not need to be done for a static graphic, but left here to build on (transparency/animation)
            videoBottomPadding.step();
            mainImage.step();
            int sourceHeight = sessionVideo.getDecoderHeight();
            int sourceWidth = sessionVideo.getDecoderWidth();
            if(showTime)
            {
                Date dNow = new Date( );
                SimpleDateFormat ft = new SimpleDateFormat("hh:mm:ss");
                wowzaText.SetText(ft.format(dNow));
                wowzaTextShadow.SetText(ft.format(dNow));
            }
            if(encodeSource)
            {
                //put the image onto the source
                scalingFactor = 1.0;
                TranscoderVideoOverlayFrame overlay = new TranscoderVideoOverlayFrame(mainImage.GetWidth(scalingFactor),
                    mainImage.GetHeight(scalingFactor), mainImage.GetBuffer(scalingFactor));
                overlay.setDstX(mainImage.GetxPos(scalingFactor));
                overlay.setDstY(mainImage.GetyPos(scalingFactor));
                sourceVideo.addOverlay(overlayIndex, overlay);
            } 
            else	
            {
                ///put the image onto each destination but scaled to fit
                for(EncoderInfo encoderInfo: encoderInfoList)
                {
                    if (!encoderInfo.destinationVideo.isPassThrough())
                    {
                        int destinationHeight = encoderInfo.destinationVideo.getFrameSizeHeight();
                        scalingFactor = (double)destinationHeight/(double)sourceHeight;
                        TranscoderVideoOverlayFrame overlay = new TranscoderVideoOverlayFrame(mainImage.GetWidth(scalingFactor),
                            mainImage.GetHeight(scalingFactor), mainImage.GetBuffer(scalingFactor));
                        overlay.setDstX(mainImage.GetxPos(scalingFactor));
                        overlay.setDstY(mainImage.GetyPos(scalingFactor));
                        encoderInfo.destinationVideo.addOverlay(overlayIndex,	overlay);
                        //Add padding to the destination video i.e. pinch
                        encoderInfo.videoPadding[0] = 0; // left
                        encoderInfo.videoPadding[1] = 0; // top
                        encoderInfo.videoPadding[2] = 0; // right
                        encoderInfo.videoPadding[3] = (int)(((double)videoBottomPadding.getStepValue())*scalingFactor); // bottom
                        encoderInfo.destinationVideo.setPadding(encoderInfo.videoPadding);
                    }
                }
            }
        } 
    } 
    return; 
}

encodeSource
With the module, you can set the encodeSource variable to place an overlay image either on the source video before it's encoded to different destination videos (encodeSource=true) or on each destination video (encodeSource=false). Both methods have benefits and drawbacks.

Source (encodeSource=true):

Benefits: Only one graphic operation is used.
Drawbacks: The graphics are rescaled and may not appear as the user intends. Also, fonts aren't redrawn and are rescaled with image reduction. You can put graphics outside the source video if the video is scaled back ("pinched"). For example, if a video was originally 640 x 480 and is pinched to 640 x 400 to provide space at the bottom of the TV for graphics.

Destination (encodeSource=false):

Benefits: Individual graphics can be used for each encoder if needed and fonts are drawn at a size to match the destination. Also, the source can be pinched and graphics drawn outside the video.
Drawbacks: Multiple graphic operations are used for each encoder (720p, 360p, etc.)

showTime
Setting the showTime variable in onBeforeScaleFrame will cause the code to send the current time as the text for wowzaText. This variable is used in this example to demonstrate dynamic text by using the current time in an overlay image.

scalingFactor
This double value defines the relationship between the destination video and source video. A value of .5 means that the destination video is half the size of the source video while a value for 2.0 means that the destination video is twice the size of the source video.

After you complete these steps, the project should compile, build, and run. However, you won't see an image on the video yet.

Create an image on the video

Add the following command to the end of the TranscoderVideoDecoderNotifyExample constructor to add the logo image (logo_1.png) to the video:

//create a transparent container for the bottom third of the screen.
mainImage = new OverlayImage(0,srcHeight-lowerThirdHeight,srcWidth,lowerThirdHeight,100);
 
//Create the Wowza logo image
wowzaImage = new OverlayImage(basePath+graphicName,100);
mainImage.addOverlayImage(wowzaImage,srcWidth-wowzaImage.GetWidth(1.0),0);

mainImage is the base container for all animation. It represents a transparent lower 3rd for all other images and text.

wowzaImage is the logo that is drawn on mainImage relative to its x,y position.

When viewing myStream_360p and all other streams, the Wowza logo should be displayed in the lower-right corner.

Create text on the video

Add text to the bottom of the image:

//Add text with a drop shadow
wowzaText = new OverlayImage("Wowza", 12, "SansSerif", Font.BOLD, Color.white, 66,15,100);
wowzaTextShadow = new OverlayImage("Wowza", 12, "SansSerif", Font.BOLD, Color.darkGray, 66,15,100);
mainImage.addOverlayImage(wowzaText, wowzaImage.GetxPos(1.0)+12, 54);
wowzaText.addOverlayImage(wowzaTextShadow, 1, 1);

This will add the text "Wowza" to the bottom of the OverlayImage with a drop shadow.

When viewing myStream_360p and all other streams, the Wowza logo and text should be displayed in the lower-right corner.

Fade the image and text in and out

To fade the image and text in and out, the OverlayImage class has the addFadingStep method for adding an event to fade the overlay.

//Fade the logo and text independently
wowzaImage.addFadingStep(100,0,25);
wowzaImage.addFadingStep(0,100,25);
wowzaText.addFadingStep(0,100,25);
wowzaText.addFadingStep(100,0,25);

The addFadingStep method takes the parameters: (<start value>,<end value>,<number of steps>).

The example code above takes the image opacity value from 0 to 100 in increments of 4 ((end-start)/steps) or ((100-0)/25) and then the reverse (taking the image opacity from 100 to 0). It will keep repeating these events. If the values are addFadingStep(0,100,200), the image would fade in increments of 0.5 ((100-0)/200)) and take longer.

With the above example code, the Wowza logo and text should be displayed in the lower-right corner and fade in-and-out independently of each other when viewing myStream_360p and all other streams.

Note: Perform the following procedure to enable creation of animation sequences described later in this article.

To finish, remove or comment-out the example code above and replace it with the following code.

//do nothing for a bit
mainImage.addFadingStep(50);
wowzaImage.addImageStep(50);
wowzaText.addMovementStep(50);

//Fade the logo and text
mainImage.addFadingStep(0,100,100);

This bit of code overloads the addFadingStep method to include only the step value and perform a 'no operation'.

Rotate the image

To animate the graphic, the OverlayImage class has an addImageStep method for updating the graphic to be used. The following code will rotate the image four times.

//Rotate the image while fading
wowzaImage.addImageStep(1,37,25);
wowzaImage.addImageStep(1,37,25);
wowzaImage.addImageStep(1,37,25);
wowzaImage.addImageStep(1,37,25);

The addImageStep method takes the same start/end/step parameters as the addFadingStep method. The class sets the variable ${com.wowza.wms.plugin.transcoderoverlays.overlayimage.step} to the current step that the image animation is on in order to rotate the image. We use this variable when defining the image (variable graphicName). Each image is rotated by 5 degrees to create a spinning effect.

The above code example spins the image four times (25 steps) during the time that it takes to fade the image (100 steps).

Animate the text

To animate the text, the OverlayImage class has the addMovementStep method for updating the x coordinates of the text that is used.

//Animate the text off screen to original location
wowzaText.addMovementStep(-75, 0, wowzaText.GetxPos(1.0), 54, 100);

The addMovementStep takes a starting x1,y1 coordinate and moves to the x2,y2 coordinate. Moving the text takes the step value and sets the x and y coordinates of the text.

To finish the animation sequence, add the following code.

//hold everything for a bit
mainImage.addFadingStep(50);
wowzaImage.addImageStep(50);
wowzaText.addMovementStep(50);

//Fade out
mainImage.addFadingStep(100,0,50);
wowzaImage.addImageStep(50);
wowzaText.addMovementStep(50);

After the animation is complete, it will hold for 50 steps and then fade out.

Pinch the video

When encoding the destination, we can scale back (pinch) the source to leave room for the graphic sequence:

//Pinch back video
videoBottomPadding.addAnimationStep(0, 60, 50);
videoBottomPadding.addAnimationStep(100);

//unpinch the video
videoBottomPadding.addAnimationStep(60, 0, 50);
mainImage.addFadingStep(50);
wowzaImage.addImageStep(50);
wowzaText.addMovementStep(50);
videoBottomPadding.addAnimationStep(100);

Troubleshooting

Using overlays can put additional load on your server because the complicated image manipulation that occurs when creating image overlays can consume a lot of processing power. This can cause Transcoder to skip frames when transcoding and delay the video. It's important that Wowza Streaming Engine be properly tuned for maximum resources. For more information, see Tune Wowza Streaming Engine for optimal performance.
We recommend that you keep the dimensions of your overlay image small. If your throughput is 30 fps and it takes more than 1/30 second to render the image, then Transcoder may start to skip frames.
This feature supports manipulation of images and text to generate animation sequences. It doesn't support stream manipulation such as picture-in-picture or multi-stream compositing.
Using Transcoder to place overlay images onto video-on-demand streams isn't supported.
Static overlay images that you create by setting Overlays properties in Transcoder templates for decoded and encoded streams will be overridden by the dynamic overlays that you create using this feature.
Closed captions that you create in Wowza Streaming Engine will be displayed on top of dynamic overlays that you create.

Add graphic overlays to transcoded live streams in Wowza Streaming Engine

Prerequisites

Create a new Transcoder module

Create a Transcoder module

Create an image on the video

Create text on the video

Fade the image and text in and out

Rotate the image

Animate the text

Pinch the video

Troubleshooting

More resources

Popular Video Topics

Video Resources

Partners

Company

Stay Connected

Stay Up to Date with the Blog