[MPlayer-dev-eng] Nut and B frames

[D Richard Felker III]

On Wed, Apr 28, 2004 at 12:47:43AM +0200, Michael Niedermayer wrote:
> > Maybe your complaint is that the codec won't know _which_ "pending"
> > frame to output for one of the t=. points. 
> yes
> 
> > But IMO this isn't 
> > possible. The codec always has to know which frame it's going to
> > output next, independent of the container...
> well yes, but theres the codec less streamcopy issue, in which case we need 
> the pts, which are no longer stored in such a way that the demuxer can figure 
> them out without the codec

Ah, streamcopy to a different format that wants decode ordering with
the pts attached... I see. Hmm, this is a problem. How about this for
the nut B frame muxing rules:

Apply your pts-to-dts algorithm to all frames. The monotonicity rule
applies to all frames with pts==dts (which necessarily exist within
every group of <delay>+1 frames by the definition of delay). The
remaining frames can be muxed anywhere as long as they remain in
decode order.

What I was missing before was a clear definition of _which_ frames are
subject to the monotonicity/interleaving rule. But then I realized
delay provides a very simple rule for it. So I'm happy with choice #1
now, as long as you are!

So here's the basic process for a program writing nut files:

while (!eof) {
  if (video.pts < audio.pts || video.out_of_order) {
    write_frame(video)
    video = get_next_video_frame()
  } else {
    write_frame(audio)
    audio = get_next_audio_frame()
  }
}

Naturally it's more complicated with more streams, but it's nice to
see that proper interleaving is easy in the simplest case.

BTW I still think we should have 0-byte packets for (hypothetical)
codecs that store several pictures in one coded frame (think frequency
transformations in time axis). A good codec like this would provide
its own dummy packets to indicate "show the next picture", but if it's
designed for use in its own container or with only a limited set of
fixed framerates, it might not...so it can't hurt to have our own
packets for the same purpose. Unfortunately this would allow people to
abuse nut by storing "DIVX packed" B frames, though...

Rich