Generate thumbnail images of live streams using the Wowza Streaming Engine Java API

This article describes code for an HTTP provider that enables generation of a thumbnail image from an H.264 or H.265 live stream without the need for the live stream to be the source of a Transcoder session. The HTTP provider uses an API that enables decoding and scaling of key frames using the decoder portion of the Transcoder.

Notes:

• Wowza Streaming Engine™ 4.5.0 or later is required.
• This HTTP provider only works on Windows and Linux operating systems.

Standard functionality

To use the thumbnail generation functionality that's built in to Wowza Streaming Engine, add the following HTTP provider definition to the [install-dir]/conf/VHost.xml file as the second-to-last entry in the HostPort configuration for port 8086 (admin) port:

<HTTPProvider>
	<BaseClass>com.wowza.wms.transcoder.thumbnailer.HTTPProviderGenerateThumbnail</BaseClass>
	<RequestFilters>thumbnail*</RequestFilters>
	<AuthenticationMethod>none</AuthenticationMethod>
</HTTPProvider>

 

Note: You'll need to save the changes and restart Wowza Streaming Engine for this change to take effect.

The URL format is:

http://[wowza-ip-address]:8086/thumbnail?application=[application-name]&streamname=[stream-name]&size=[width]x[height]&fitmode=[fitmode]&crop=[left],[right],[top],[bottom]&format=[png,jpg]

• application: Application name (required).
• streamname: Stream name (required).
• size: Thumbnail size in pixels (required).
• fitmode: Fit mode (how video frame is fit to frame size). Valid values are letterbox, stretch, fit-height, fit-width, match-source, crop (default is letterbox).
• crop: Number of pixels to crop from source image.
• format: Format of the image. Valid values are jpeg and png (default is jpeg).

Example:

http://[wowza-ip-address]:8086/thumbnail?application=myApplication&streamname=myStream&size=640x360&fitmode=letterbox

Modifying thumbnail image generation

If you want to modify the standard functionality, you can copy this sample HTTP provider code, modify it as you'd like, and build it into a separate HTTP provider. You'll define that module in the [install-dir]/conf/VHost.xml file as described previously.

import java.awt.image.*;
import java.io.*;
import java.util.*;

import javax.imageio.*;

import com.wowza.util.*;
import com.wowza.wms.amf.*;
import com.wowza.wms.application.*;
import com.wowza.wms.http.*;
import com.wowza.wms.logging.*;
import com.wowza.wms.stream.*;
import com.wowza.wms.transcoder.httpprovider.*;
import com.wowza.wms.transcoder.model.*;
import com.wowza.wms.transcoder.thumbnailer.*;
import com.wowza.wms.transcoder.util.*;
import com.wowza.wms.vhost.*;

public class HTTPProviderGenerateThumbnail extends HTTPProvider2Base
{
    private static final Class<HTTPProviderGenerateThumbnail> CLASS = HTTPProviderGenerateThumbnail.class;
    private static final String CLASSNAME = "HTTPProviderGenerateThumbnail";

    public void onHTTPRequest(IVHost vhost, IHTTPRequest req, IHTTPResponse resp)
    {
        if (!doHTTPAuthentication(vhost, req, resp))
            return;

        byte[] outBytes = null;
        String formatStr = "jpeg";
        String fitModeStr = "letterbox";
        String cropStr = null;
        int[] crop = null;

        try
        {
            while(true)
            {
                String queryStr = req.getQueryString();
                if (queryStr == null)
                {
                    WMSLoggerFactory.getLogger(CLASS).warn(CLASSNAME+".onHTTPRequest: Query string missing");
                    break;
                }

                Map<String, String> queryMap = HTTPUtils.splitQueryStr(queryStr);

                String streamName = queryMap.get("streamname");
                if (streamName == null)
                {
                    WMSLoggerFactory.getLogger(CLASS).warn(CLASSNAME+".onHTTPRequest: Streamname not specified");
                    break;
                }

                String comboAppStr = queryMap.get("application");
                if (comboAppStr == null)
                {
                    WMSLoggerFactory.getLogger(CLASS).warn(CLASSNAME+".onHTTPRequest: Application name not specified");
                    break;
                }

                if (queryMap.containsKey("fitmode"))
                    fitModeStr = queryMap.get("fitmode");

                if (queryMap.containsKey("fitMode"))
                    fitModeStr = queryMap.get("fitmode");

                if (queryMap.containsKey("format"))
                    formatStr = queryMap.get("format");

                if (formatStr == null)
                    formatStr = "jpeg";

                formatStr = formatStr.trim().toLowerCase();
                if (formatStr.equals("jpg"))
                    formatStr = "jpeg";

                String sizeStr = queryMap.get("size");
                int width = 0;
                int height = 0;
                if (sizeStr != null)
                {
                    int cindex = sizeStr.indexOf("x");
                    try
                    {
                        width = Integer.parseInt(sizeStr.substring(0, cindex));
                        height = Integer.parseInt(sizeStr.substring(cindex+1));
                    }
                    catch(Exception e)
                    {

                    }
                }

                if (queryMap.containsKey("crop"))
                    cropStr = queryMap.get("crop");

                if (cropStr != null)
                {
                    String[] parts = cropStr.split(",");
                    if (parts.length >= 4)
                    {
                        crop = new int[4];

                        for(int i=0;i<4;i++)
                        {
                            int cropVal = -1;
                            try
                            {
                                cropVal = Integer.parseInt(parts[i].trim());
                            }
                            catch(Exception e)
                            {
                            }
                            if (cropVal > 0)
                            {
                                crop[i] = cropVal;
                            }
                        }
                    }
                }

                String applicationStr = comboAppStr;
                String appInstanceStr = IApplicationInstance.DEFAULT_APPINSTANCE_NAME;

                int cindex = applicationStr.indexOf("/");
                if (cindex > 0)
                {
                    appInstanceStr = applicationStr.substring(cindex+1);
                    applicationStr = applicationStr.substring(0, cindex);
                }

                IApplication applicationLoc = vhost.getApplication(applicationStr);
                if (applicationLoc == null)
                {
                    WMSLoggerFactory.getLogger(CLASS).warn(CLASSNAME+".onHTTPRequest: Application could not be loaded: "+applicationStr);
                    break;
                }

                IApplicationInstance appInstance = applicationLoc.getAppInstance(appInstanceStr);
                if (appInstance == null)
                {
                    WMSLoggerFactory.getLogger(CLASS).warn(CLASSNAME+".onHTTPRequest: Application instance could not be loaded: "+applicationStr+"/"+appInstanceStr);
                    break;
                }

                IMediaStream stream = appInstance.getStreams().getStream(streamName);
                if (stream == null)
                {
                    WMSLoggerFactory.getLogger(CLASS).warn(CLASSNAME+".onHTTPRequest["+applicationStr+"/"+appInstanceStr+"]: Stream not found: "+streamName);
                    break;
                }

                AMFPacket keyFrame = stream.getLastKeyFrame();
                if (keyFrame == null)
                {
                    WMSLoggerFactory.getLogger(CLASS).warn(CLASSNAME+".onHTTPRequest["+applicationStr+"/"+appInstanceStr+"/"+streamName+"]: No key frame");
                    break;
                }

                int fitmode = TranscoderStreamUtils.frameSizeFitModeToId(fitModeStr);
                if (fitmode <= 0)
                {
                    WMSLoggerFactory.getLogger(CLASS).warn(CLASSNAME+".onHTTPRequest["+applicationStr+"/"+appInstanceStr+"/"+streamName+"]: Fit mode not recognized defaulting to letterbox: "+fitModeStr);
                    fitmode = TranscoderStream.FRAMESIZE_FITMODE_LETTERBOX;
                }

                ThumbnailerRequest request = new ThumbnailerRequest(stream, width, height, fitmode);

                request.addPacket(keyFrame);
                if (crop != null)
                    request.setCrop(crop);

                ThumbnailerResponse response = ThumbnailerUtils.generateThumbnail(vhost, request);
                if (response == null)
                {
                    WMSLoggerFactory.getLogger(CLASS).warn(CLASSNAME+".onHTTPRequest["+applicationStr+"/"+appInstanceStr+"/"+streamName+"]: Thumbnail generation failed");
                    break;
                }

                BufferedImage image = ThumbnailerUtils.nativeImageToBufferedImage(response);
                if (image != null)
                {
                    WMSLoggerFactory.getLogger(CLASS).info(CLASSNAME+".onHTTPRequest["+applicationStr+"/"+appInstanceStr+"/"+streamName+"]: Image result: format:"+formatStr+" size:"+image.getWidth()+"x"+image.getHeight());
                    try
                    {
                        ByteArrayOutputStream baos = new ByteArrayOutputStream();
                        ImageIO.write(image, formatStr.equals("jpeg")?"jpg":formatStr, baos);
                        synchronized(this)
                        {
                            outBytes = baos.toByteArray();
                        }
                    }
                    catch(Exception e)
                    {
                        WMSLoggerFactory.getLogger(CLASS).error(CLASSNAME+".onHTTPRequest["+applicationStr+"/"+appInstanceStr+"/"+streamName+"] ", e);
                    }
                }
                break;
            }
        }
        catch (Exception e)
        {
            WMSLoggerFactory.getLogger(HTTPTranscoderThumbnail.class).error("HTTPTranscoderThumbnail ", e);
        }

        try
        {
            if (outBytes != null)
            {
                resp.setHeader("Content-Type", "image/"+formatStr);
                OutputStream out = resp.getOutputStream();
                out.write(outBytes);
            }
            else
                resp.setResponseCode(404);
        }
        catch (Exception e)
        {
            WMSLoggerFactory.getLogger(HTTPTranscoderThumbnail.class).error("HTTPTranscoderThumbnail ", e);
        }
    }
}

Examples

The following example code shows an API for the thumbnail request and response, then later, converting the result to JPEG (as PNG is the default).
 

ThumbnailerRequest request = new ThumbnailerRequest(stream, width, height, fitMode);
request.addPacket(keyFrame);

ThumbnailerResponse response = ThumbnailerUtils.generateThumbnail(vhost, request);

The thumbnail image is returned as an RGBA byte array. This byte array can then be converted to a Java BufferedImage using the ThumbnailerUtils.nativeImageToBufferedImage(response); utility method. The following example code uses this API to convert the result to JPEG data:

BufferedImage image = ThumbnailerUtils.nativeImageToBufferedImage(response);
if (image != null)
{
	WMSLoggerFactory.getLogger(CLASS).info(CLASSNAME+".onHTTPRequest["+applicationStr+"/"+appInstanceStr+"/"+streamName+"]: Image result: format:"+formatStr+" size:"+image.getWidth()+"x"+image.getHeight());
	try
	{
		ByteArrayOutputStream baos = new ByteArrayOutputStream();
		ImageIO.write(image, formatStr.equals("jpeg")?"jpg":formatStr, baos);
		synchronized(this)
		{
			outBytes = baos.toByteArray();
		}
	}
	catch(Exception e)
	{
		WMSLoggerFactory.getLogger(CLASS).error(CLASSNAME+".onHTTPRequest["+applicationStr+"/"+appInstanceStr+"/"+streamName+"] ", e);
	}
}