Gstreamer and OpenMax

What is GStreamer

GStreamer is a library for constructing graphs of media-handling components. The applications it supports range from simple AV playback, audio/video streaming to complex audio (mixing) and video (non-linear editing) processing. The GStreamer core function is to provide a framework for plugins, data flow and media type handling/negotiation. It also provides an API to write applications using the various plugins. GST elements chain via pads (similar to ports in OpenMAX world) and exchange data between source pads of a data generating element and sink pads of the data consumer element. Applications can take advantage of advances in codec and filter technology transparently. Developers can add new codecs and filters by writing a simple plugin with a clean, generic interface.

In omap, both GST-OpenMax and GST-Ducati provide elements that interface with omap hardware.

The code for GStreamer-omap can be found in GStreamer Omap repository in gitorious. Check the GStreamer build Instructions to build GStreamer from source code.

For general GStreamer development info check GStreamer developers Info and GStreamer official website

[edit]GStreamer Stack

GStreamer provides plugins across the entire filter graph. GStreamer plug-ins could be classified into

• protocols handling
• sources: for audio and video (involves protocol plugins)
• formats: parsers, formaters, muxers, demuxers, metadata, subtitles
• codecs: coders and decoders
• filters: converters, mixers, effects, ...
• sinks: for audio and video (involves protocol plugins)

http://gstreamer.freedesktop.org/documentation/
http://gstreamer.freedesktop.org/data/doc/gstreamer/head/manual/html/chapter-elements.html
http://www.omappedia.com/wiki/Gstreamer_overview

http://docs.gstreamer.com/pages/viewpage.action?pageId=327735
http://gstreamer.freedesktop.org/data/doc/gstreamer/head/manual/html/chapter-helloworld.html
http://processors.wiki.ti.com/index.php/Example_GStreamer_Pipelines#Loopback:_Video
http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer-editing-services/html/ges-architecture.html

mirror.linux.org.au/linux.conf.au/2004/papers/49-steve-baker-lcau2004-gstreamer-1.1.sxi.pdf
http://labs.isee.biz/index.php/Example_GStreamer_Pipelines#Loopback:_Video

https://www.google.co.in/search?q=gstreamer+architecture&revid=719076486&sa=X&ei=IdWcUfC3A4es4AT9yICABQ&sqi=2&ved=0CJQBENUCKAE&biw=1600&bih=702

https://www.google.co.in/search?q=gstreamer+overview&revid=484955843&sa=X&ei=pNWcUePLCOHk4QTv5oGABA&ved=0CIwBENUCKAA&biw=1600&bih=702
https://www.google.co.in/search?q=gstreamer+framework&revid=484955843&sa=X&ei=pNWcUePLCOHk4QTv5oGABA&ved=0CI0BENUCKAE&biw=1600&bih=702

https://www.google.co.in/search?q=gstreamer+framework&revid=484955843&sa=X&ei=pNWcUePLCOHk4QTv5oGABA&ved=0CI0BENUCKAE&biw=1600&bih=702

http://www.slideserve.com/trish/gstreamer-whats-new
http://www.docstoc.com/docs/48475449/Celinux-sanfran-gstreamerppt--

Omap GStreamer modifications

Gstreamer for omap adds strided video to GStreamer, correspondent to the video format "video/x-raw-yuv-strided". GStreamer library is patched to work with strided format and GST-Ducati and GST-Openmax support strided video.

StrideTransform is an element that is able to convert between strided and non-strided video.

[edit]Source, sink and filter elements

Video sinks

PVRVideosink is the recommended sink to display video. This element is contained in gst-plugins-bad and depends on libpvr2d library.

XImagesink can be used as a fallback video sink when libpvr2d is not available.

Video sources

v4l2src can be used as a video source.

[edit]Sample pipelines

[edit]Display webcam video

gst-launch v4l2src device=/dev/video3 ! "video/x-raw-yuv, width=640, height=480,framerate=15/1" ! queue ! ffmpegcolorspace ! pvrvideosink

[edit]GST-Ducati

GST-Ducati is a GStreamer plugin that provides elements to decode and encode video. GST-Ducati uses the hardware accellerated codecs on IVA-HD via the codec-engine API. It uses the libdce (distributed codec-engine) library to use the codec on ducati.

From GLP1.5 on, GST-Ducati encoders and decoders should be used instead of GST-OpenMax ones.

[edit]Resources

GST-Ducati and libdce code can be downloaded from GStreamer-omap repository. Source for Debian packages are also available at Ubuntu-omap repository.

[edit]Encoders and decoders elements

The following elements are provided by GST-Ducati plugin:

• ducativc1dec: VC1 decoder
• ducatimpeg2dec: MPEG2 decoder
• ducatimpeg4dec: MPEG4 decoder
• ducatih264dec: H264 decoder
• ducatimpeg4enc: MPEG4 encoder
• ducatih264enc: H264 encoder

[edit]Sample gst-launch pipelines

[edit]Decoding

GST-Ducati decoders are automatically chosen to play video. (Use option -v to see what elements are being used):

gst-launch playbin2 uri=file://home/user/sample.mp4
gst-launch filesrc location=/home/user/sample.mp4 ! decodebin2 ! pvrvideosink
gst-launch filesrc location=/home/user/sample.mp4 ! qtdemux ! h264parse ! ducatih264dec !  pvrvideosink

Note: On Ubuntu 11.04, there are two additional video sinks:

• dri2videosink, based on SGX, for X11 windowed cases. It is more optimized than pvrvideosink
• kmsvideosink, based on DSS, for non-X11 full-screen use cases.

[edit]Encoding

Encode webcam video to file:

gst-launch -e v4l2src ! "video/x-raw-yuv, width=640, height=480,framerate=30/1" ! queue ! ffmpegcolorspace ! ducatih264enc ! h264parse ! qtmux ! filesink location=sample.mp4

[edit]Encode + Decode

The following pipeline shows the encode/decode latency:

gst-launch -v v4l2src ! stridetransform ! tee name=t \
   t. ! queue ! dri2videosink sync=false \
   t. ! queue ! ducatih264enc rate-preset=low-delay profile=100 ! h264parse ! ducatih264dec ! queue ! pvrvideosink sync=false

[edit]Reworking an existing OMX-based pipeline

In the case you have previously working pipelines using GST-OpenMax encoders and decoders, there are some general rules of thumb to make your pipeline work with GST-Ducati and latest GStreamer-Omap:

• Replace omx_h264enc by ducatih264enc ! h264parse.
• Replace omx_mpeg4enc by ducatimpeg4enc ! mpeg4videoparse.
• Replace h264parse access-unit=true output-format=byte by h264parse without options.
• Replace nal2bytestream_h264 element by h264parse.
• Remove any unsupported parameter. For example input-buffers and output-buffers are not needed for DCE encoders anymore.
• Adapt values that may have changed. For example, gst-inspect ducatih264enc says that H264 high-profile is now set with profile=100. Formerly with omx_h264enc it was profile=8.
• Replace video/x-raw-yuv-strided by video/x-raw-yuv.
• Replace v4l2sink by pvrvideosink.

[edit]More samples wanted

Stream Video from File

Stream Video from Webcam

Play Video Stream

[edit]GST-OpenMax

In GST-OpenMax, GStreamer plugin interfaces with OMX codecs, drivers or transforms based on the functionality. The TI GST- OMX SW stack is as follows.

[edit]GST Buffer Flow

This section provides more details on the exact interfaces between a GST plugin (as in GSTOMX_Base_filter) and OpenMAX component at different stages of the life cycle. This description is based on a generic filter behavior (GSTOMX_Base_filter) that is applicable for video and audio decoders/encoders.

[edit]Example TI-OpenMAX filter graph for AV Capture

The following diagram describes a full filter graphs and associations between GST, TI SW and HW components taking AV capture as example

[edit]GST – OMX integration issues and Impact:

• GST does NOT adhere to OpenMAX rule of 1 to 1 mapping of OMX Buffer Header to OMX data buffer (pBuffer).
• GST Src elements (Non-TI element, originator of bitstreams) can potentially provide new data buffer every time; which means OMX Buffer Headers that come to OMX elements via a ETB call could find new pBuffer pointers every time.
• GST Sink elements (TI V4L2 sink, consumer of RAW data – 1080p NV12) maintains a buffer pool through which they re-cycle data buffers.
• The buffer pool could have M buffers (TI V4L2 sink could maintain M tiler buffers) sent to OMX Component using N OMX Buffer headers.
• Value of M could be GREATER or EQUAL to N. This implies that even though we can MAP once the N buffers during OMX_UseBuffer() call. There could potentially be new data buffers (M – N) found in OMX Buffer headers during the FTB call. This mandates a check and map (if required) these new buffers during a FTB call.
• OMX Decoders currently keep a tag of reference buffers w/ OMX Buffer headers (assumes 1 to 1 mapping of OMX Buffer hdr and pBuffer pointer). Hence will break if GST puts arbitrary buffers into OMX Headers.

[edit]Impact:

• OMX Proxy by design does mapping in OMX Usebuffer/OMX Allocate buffer calls. Now need to check for new data buffers and map them during FTB calls.
• Ideally OMX Buffers map only once and caches pre-mapped addresses. Now for bitstream buffers we need to Map/UnMap every time.
• Reference counting in Decoders are w/ OMX Buffer headers, now they need to reference count based on actual data buffer (pBuffer pointer).

Categories: GStreamer | OpenMax | Multimedia | Linux

OPENMAX

http://www.khronos.org/openmax/
http://en.wikipedia.org/wiki/OpenMAX

OpenMAX DL (Development Layer) APIs contains a comprehensive set of audio, video and imaging functions that can be implemented and optimized on new CPUs , hardware engines, and DSPs and then used for a wide range of accelerated codec functionality such as MPEG-4, H.264, MP3, AAC and JPEG.
This version includes clarifications and corrections to the previous version (1.0.1). All implementations should aim to match this version for interoperability.

• OpenMAX DL 1.0.2 Specification (.pdf, 1.5 MB)
• OpenMAX DL Changelog 1.0.1 -> 1.0.2 (.txt)
• OpenMAX DL 1.0.2 redline changes to 1.0.1 -- AVAILABLE SOON -- (.pdf)
• All OpenMAX DL 1.0.2.h header file (.zip, 76 KB)
• A sample OpenMAX DL 1.0.2 implementation written in C is available for download from ARM.
• Optimised OpenMAX DL libraries are available for ARM11 and Cortex-A8 processors from ARM.
• Version 1.0.1 headers All OpenMAX DL 1.0.1.h header file (.zip, 70 KB)
  
  

Background

Development of multimedia hardware platforms is gathering pace as consumer demand grows for improved functionality from applications such as video, audio, voice, and 3D on platforms such as diverse as smartphones, audio and video media players and games consoles. In general, this class of product requires high-performance processing and high data throughput capabilities. Consequently, a variety of solutions has evolved, each designed to accelerate multimedia applications. Examples include:

• General purpose processors with specific multimedia extensions
• Low level hardware accelerators,
• Multiple processor architectures including DSPs
• Dedicated hardware video decoders

One of the key challenges with all of these architectural variants is to develop efficient code. Even though compilers are generally provided, it is rare to be able to exploit the full potential of the entire architecture from a high-level programming language. The outcome is that large portions of the application are often written in assembly language to specifically target the hardware platform. The proliferation of different multimedia hardware solutions means that software must be re-written and optimized for each new platform that it is ported to.
The effect of this inefficiency in implementation is to delay the introduction of new products, increase development costs and reduce product quality, which ultimately slows down innovation in the multimedia domain at a time when market demand is growing.




OpenMAX (Open Media Acceleration) is a royalty-free, cross-platform set of C-language programming interfaces that provides abstractions for routines especially useful for audio, video, and still images processing. It's intended for devices that process large amounts of multimedia data in predictable ways.
OpenMAX provides three layers of interfaces: Application Layer (AL), Integration Layer (IL) and Development Layer (DL). OpenMAX is managed by the non-profit technology consortium

http://www.google.co.in/imgres?um=1&sa=N&biw=1600&bih=702&hl=en&tbm=isch&tbnid=AJnVbKWoEc5gkM:&imgrefurl=http://danielstroe.wordpress.com/2009/12/28/looking-into-the-maemo-multimedia-framework/&docid=dLuZFsoNxkPg4M&imgurl=http://danielstroe.files.wordpress.com/2009/12/800px-omap_architecture.jpg&w=800&h=557&ei=3iSdUeHjPIWJrgel74HYAQ&zoom=1&ved=1t:3588,r:1,s:0,i:84&iact=rc&dur=2179&page=1&tbnh=173&tbnw=249&start=0&ndsp=18&tx=136&ty=82

http://www.google.co.in/imgres?um=1&sa=N&biw=1600&bih=702&hl=en&tbm=isch&tbnid=vMHwhvcwOD6CSM:&imgrefurl=http://www.arm.com/community/partners/display_product/rw/ProductId/4738/&docid=UgsZS8py1sxW_M&imgurl=http://www.arm.com/community/partners/product_images/4724.jpg&w=1295&h=842&ei=3iSdUeHjPIWJrgel74HYAQ&zoom=1&ved=1t:3588,r:2,s:0,i:87&iact=rc&dur=962&page=1&tbnh=173&tbnw=267&start=0&ndsp=18&tx=96&ty=102

http://www.google.co.in/imgres?um=1&sa=N&biw=1600&bih=702&hl=en&tbm=isch&tbnid=A0Xh71PsAqOy5M:&imgrefurl=http://www.arm.com/community/partners/display_product/rw/ProductId/4757/&docid=IhPeIyOJAjLciM&imgurl=http://www.arm.com/community/partners/product_images/4725.jpg&w=1357&h=802&ei=3iSdUeHjPIWJrgel74HYAQ&zoom=1&ved=1t:3588,r:8,s:0,i:105&iact=rc&dur=1012&page=1&tbnh=172&tbnw=292&start=0&ndsp=18&tx=145&ty=39


https://www.google.co.in/search?q=openmax+development+layer&revid=136288028&sa=X&ei=feacUdLxKeuN4gTKpoG4Cw&ved=0CIABENUCKAE&biw=1600&bih=702

https://www.google.co.in/search?q=openmax+architecture&bav=on.2,or.r_cp.r_qf.&bvm=bv.46751780,d.bGE&biw=1600&bih=702&um=1&ie=UTF-8&hl=en&tbm=isch&source=og&sa=N&tab=wi&ei=sySdUcfkEsuWrgf_6YDQDg


https://www.google.co.in/#output=search&sclient=psy-ab&q=what+is+openmax&oq=what+is+openmax&gs_l=hp.1.0.0j0i22i30l4.3715.14302.1.17950.15.15.0.0.0.0.626.3121.0j11j1j0j1j2.15.0...0.0...1c.1.14.hp.-yOBgicsc5Q&psj=1&bav=on.2,or.r_cp.r_qf.&bvm=bv.46865395,d.bmk&fp=feae99c9a81b2431&biw=1454&bih=704

http://processors.wiki.ti.com/index.php/OpenMax_Development_Guide