OpenMAX integration Challenges

  • Thierry, Rob, Kurt as leads
  • Kiko: OpenMAX is meant to provide "comprehensive media codec and application portability [...] across multiple operating systems and silicon platforms." Is it?
  • Rob: OpenMAX is great if it /is/ the application interface
    • However, this isn't the case; we're integrating it into existing multimedia frameworks
    • OMX impositions don't fit well with existing frameworks
  • jim: A very large spec that tries to solve every problem
    • However, the market wants to use it for just one problem
    • As a solution to integrate plugins to talk to codecs
    • All the consumers want to use it in the same way: just plug the codec in
    • Recommendation to provide a note that describes the codec plugin problem
      • Sample code and conformance tests
    • Thierry: likely to provide an OpenMAX Light spec
    • Rob: if you want a video codec API, it should be lower-level
      • Doesn't require pre-allocation, allows refcounting
    • Thierry: historically, it was meant to be a full framework
    • Rob: needs to be a clean break; dropping backwards-compatibility will allow a good solution to work
    • Thierry: value today is in definition of common component format; let's focus on these bits
  • Thierry: should separate issues with implementation from issues with OpenMAX spec
  • Rob: branches of problem
    • On bitstream side, hardware expects data in a certain format; in gstreamer we use caps to describe formats, and parsers to covert between them
      • Jim: sees the same issue on his end
    • Pre-announce buffers and data format in the buffer; might be able to hide format details under an opaque handle, and convert handle to bytes where necessary
      • Jim: Linaro could propose a set of standard components which define parameters supported; exclude color formats that are irrelevant, include relevant ones
  • Jim: are we going to time-bound this work into 3 months?
    • Kiko: we need to understand the problem a bit better first, but once we know enough about the problem and the plan, we will build out a timeline for implementing a change
  • Rob: we really only want a Codec API, not a whole framework
    • Benjamin: agree with Rob suggestion
  • Jim: very few users (only one in their survey) use OpenMAX directly anyway
    • Jim: some ISVs don't /want/ to use StageFright, and will instead provide their own middleware

    • Even these are unlikely to use OpenMAX directly, since Android really discourges it
  • Jim: what would the refcounting model look like
    • Rob: in gstreamer, buffer refcounting is done by IL client (and codec is simpler)
    • Rob: separate "buffer decode done" from "buffer used"
      • Example: TI Ducati codec API: buffer passing is a single call
        • Gets input buffer with a pointer per plane, a buffer ID; returns two arrays, one for buffers ready for display, and the other for buffers which are unlocked
        • Easy to implement gstreamer refcounting on top of that
  • OpenMAX Vendor Survey
  • Implement reference API to discuss with Khronos
    • More specific is better
    • Rob: Prototype implementation is doable in 1/2 cycles

Initial summary of issues we'd like to address in OpenMAX IL

  • I. There is no way to for a codec (an IL component) to tell an IL client that a buffer is free to be reclaimed
    • Could be fixed by supporting explicit reference counting; allowing IL client take care of it
      • TI implemented an extension to cope with this, but it isn't a very good solution because the calls are asynchronous
      • Rob's concern is that even implementing explicit refcounts, the fact the API is asynchronous continues to cause difficulty
    • Could provide callbacks to implement refcounting (Thierry: note this has been rejected before)
    • Thierry points out that ST-Ericsson platforms don't actually need this; Alessandro points out that other vendors apart from TI do, however
  • II. Simplifying state machine [transitions]
    • High complexity to handle a large number of states (TI codec doesn't have that many states)
    • OpenMAX component tries to describe resource state
    • The state machine exarcerbates the buffer management problem; not necessarily a problem in itself
      • Rob: State machine is tied to buffer lifecycle; if a simpler form of managing buffers (refcounting mention above) were available, then the state machine could be simpler
    • Thierry agrees that this has been identified by OMX as a problem as well
    • Gstreamer requires only two states:
      • - A: HW is initialized but idle; available for querying - B: where you can send the hardware to actually run. Stopping can take the HW back to initialized or stop it altogether.
  • III. IL client needs a token to represent native memory shared between rendered/decoder or camera/encoder
    • "Tunneling through IL client; using opaque handles for buffers"
    • Restated from the UMM andle, no straightforward way to use Linux-proposed unified memory approach in OpenMAX
    • Required to allow for UMM work to fit into OpenMAX
    • Likely solution to put the token inside an OMX buffer itself
  • IV. OMX doesn't provide a way to query what bitstream format the OMX component can consume
    • Needed to cope with cases where OMX component can't deal with bitstream format provided
    • Explicitly describe bitstream data format supplied to codec
    • Explicitly state format for each codec, or enumerate possible formats; take input from survey
      • Could look at TI requirements from gst-ducati code
    • For WMA, for instance, OMX didn't tell what the container demux output format was; ideally, it would be better specified, but it's a reasonable assumption that the component can consume most formats.
    • In gstreamer, elements negotiate dynamically to decide whether the component can consume the bitstream directly; in the case it can't, a gstreamer parser is used to convert. So gstreamer does implement a mechanism for the solution, but there isn't a way to discover what the codec supports unless it is documented.
    • Jim points out this sounds like an easy problem to solve; set up conversation between OpenMAX experts (Tom/QC, Rob/TI, Nv) to address quickly
    • Two possible avenues to solve problem:
      • Having a way to query the codec for what bitstream format it can take so gstreamer can pre-parse it if necessary
      • gstreamer could document buffer format expectation that would be provided before actual bitstream; ensure that any component can be configured in "gstreamer mode" so it accepts its demuxer output. Suggestion to create a standard component to encode this requirement.
  • V. Port Settings
    • A: Downstream element (rendering) needs to change resolution (window resize, for instance); something upstream would need to either up or downscale to match.
      • Would like to tell the decoder to change expected resolution, and unclear what mechanism to go about this: can the port be directly modified, or do you actually need to change the component state for it?
        • How do you reconfigure output format in the least destructive way?
        • Unclear if this is a spec limitation, or a limitation with the implementation of the component
    • B: The change in format might also be dictated from the component; for instance, the source resolution changes
      • This is handled well in gstreamer; port settings change event is received and downstream reconfigures
      • Unclear whether event is consistently received across platforms
  • VI. Synchronous API clumsiness
    • General issue with using OMX; causes difficulties across the board in implementing IL clients
      • Benjamin: asynchronous state changes are difficult to manage when you need to wait for acknowlegement (and some functions return errors if you call them in the wrong state)
    • Debugging: difficult to understand causes of error modes; an error is received and you can't identify where the error actually originated.
      • Could be mitigated by allowing a payload to be associated with the error, which could be queried when the error actually occurred.
    • Error codes could also be clarified; work in OMX spec to improve this already underway
    • For alternative models, suggestion to look at TI codec API mentioned above; single call provides buffers and collects status, output
    • Might not be a showstopper if other issues are reasonably addressed
  • VII. Dropping buffer pre-announce
    • OpenMAX requires you to inform buffers when initializing a port, and then ask the buffers to be used/allocated to complete the state transition
    • Unfortunately, you may not know how many buffers you actually need before you parse the SPS
      • In OpenMAX you are expected to reallocate the buffer
      • In gstreamer buffer allocation is done dynamically
      • Historically, in OpenMAX you tend to reuse the same set of buffers, which is why the API looks like this
    • Normally in gstreamer the downstream element will manage a pool of buffers and allocate on-demand
      • In 0.11 upstream manages the pool and you can actually negotiate up-front the size of the pool, then it's an implementation detail whether you pre allocate or not
    • Proposal sent by TI to Khronos
      • Suggested to allow changing the buffer pool dynamically
      • Prototype implemented; but ran into complexity issues
    • Problems exarcerbated by lack of explicit reference counting
    • Thierry: likely that this is something Khronos would consider changing since it seems to no longer be useful
      • Likely to break backwards compatibility
        • Thiery noted that 1.2 will break backwards compat already, so has precedent
        • Would break component API; the IL client would be able to use both types of components (how?)
  • Benjamin: is this towards a kernel-level API, or a userspace API?
    • Rob: feels like the sort of work isn't something we'd want to do in the kernel
    • In particular, handling of software versus hardware codec use shouldn't be kernel-encoded
  • ACTION: collect source code
    • TI gstreamer components (and gst-openmax and gst-ducati)
    • Freescale Android OMX Components
      • Note that Freescale doesn't use Stagefright directly, but their own Graph Manager framework
        • Graph Manager itself is an IL client, and interfaces with OMX components
        • Do implement complete OMX graph (parsers, clock, etc)
    • Look at Android OMX standard glue
  • ACTION (Kiko): identify OpenMAX representatives from members
    • Nikhil (TI)
    • Thierry (ST-Ericsson)
    • Jim (Nvidia)
    • Bryan Murray (Freescale)
    • (ARM; no OMX rep)
    • (Samsung)
  • ACTION (Kiko): set up call with QC/Nvidia to talk about VDPAU/VAAPI as an alternative
  • ACTION (Rob, Kurt): set up wikipage to collect input for API; collect survey data here
  • ACTION (Kurt, Kiko): share initial set with Google, Linaro TSC and collect input
  • ACTION: Jim/Kurt/Thierry call with Neil to define collaboration model
  • ACTION: study how to formally communicate to OMX IL WG Tuesday meeting
  • ACTION: prepare reference for OpenMAX F2F meeting (Phoenix, September)

Day 2 NOTES

  • what should a vendor survey look like?
  • ACTION: obtaining source code from vendors
  • how do you frame the survey to get best feedback?
    • Kurt: lets not make them do an exhaustive search. We can present them some of our findings, and ask for
      • their opinion.
    • Zach - lets focus on the 3 things we are interested in (like ref-counting). <group>is that enough?

    • Do we fix something below OpenMAX?
  • Benjamin: Maybe Khronos has already done this survey ?
    • Jim: sort-of, but not the type questions we are raising here. Thierry - it was done 2 years ago and the market/landscape has changed

      Jim: we are talking about top->down from an integration perspective

  • Kurt: do we need the whole part of OpenMAX, or a subset, or another smaller API?
    • Let's go through list and better describe issues 1st. Focus on idendifing the problems and getting concensus

Here's the OpenMAX state machine: http://www.khronos.org/files/openmax_il_spec_1_1_2.pdf page 25 UNLOADED LOADED INVALID WAIT FOR RESOURCES IDLE EXECUTING PAUSED

Question: EGL_KHR_il_image: an NVIDIA extension to interface with OMX Is this code available?

<CosmicPenguin (QC)> Our troubles have mainly been around buffer sharing, and trying to communicate the format of the buffer, because we use the funky formats sometimes due to GPU limitations

  • my hope is some of this can be solved w/ UMM/handle

<CosmicPenguin (QC)> Thats about all I can add - except that maybe vendors with Khronos access can look at EGL_KHR_il_image - it is the only EGL extension that calles openmax out by name

Events/2011-06-MMWG/OpenMaxIntegration (last modified 2011-07-12 20:35:22)