[MPlayer-dev-eng] more NUT questions

[D Richard Felker III]

On Sat, Apr 17, 2004 at 08:39:03PM +0300, Ivan Kalvachev wrote:
> >> Just an rough idea.
> >> Make index_packet fixed size. (16k). Write as many entries as possible.
> >> index_packet must have additional (fixed size) pointers to
> >> previous/forward
> >> index_packet. sequence_start packet may point directly to the 1'st
> >> index.
> >
> > Forward pointer (to next index packet) is impossible because it's not
> > known when the index packet is written. Anyway I find this whole
> > design an ugly hack. There has to be a correct way to do it...
> 
> It is kind of hack, but it is nice one. If I have faild to explain.
> We write 16k for index even if it is empty. We use an 16kb memory buffer.
> Every time when we write packet we add it to the index.When the index is
> full (including pointers size) we write the ready index at the empty
> space. Then we write another blank 16k for the next index. And so on.

Buffering/seeking back to write isn't acceptable. It's not possible
for live recording/streaming.

> This will allow to have indexs in right position for streaming and to
> made indexes while capturing.

Only for streaming non-live content, which is useless.

> If you have better idea I would like to hear it.

None of this interleaved index nonsense. Have a viable way to walk
packets without an index and without any nonsequential writing or
reading. Index can optionally be included at end of file for O(1)
seeking rather than O(log n).

> >> > > Solution 4.
> >> > > GOP. All frame headers at one place. Kinda of #2.
> >> > > Even better, it could be some kind of destributed index.
> >> > > small indexes all over connected with forward/backward pointers.
> >> > > Hmm sound familiar, maybe I have seen it before?
> >> I think that this is the right solution:
> >> Headers of frame_type0 frames should be packet together with the last
> >> packed_header-ed packet. This way they would be protected by the crc
> >> of the same packet. Anyway if the packet is broken usually demux of
> >> next frame_type0 it won't be possible.
> >
> > WTF?!?! There is no CRC!!! CRC is totally useless.
> 
> No. We need some system for validation of packet headers data.
> We cannot afford one broken packet to break the chain.

Knowing the packet header is corrupt WILL NOT HELP if you don't have a
way of recovering it. What do you propose the demuxer do if the CRC
doesn't match???

> Just like I gave example with frame_type0.

Without type 0 frames, it's not a problem. You know a packet is
corrupt if the forward/back pointers don't match up. And then if you
want to try recovering, you can brute force search for valid
forward/back pointers. Chance of a false positive is comparable to the
chance of a false positive with a CRC, and you don't waste extra bytes
storing otherwise-useless data (CRC).

> If we have broken part, we may read wrong forward pointer,
> we may jump near the end of file and after that to jump out of the file.

Not unless the place you jump to has a backward-pointer pointing to
where you came from, which is VERY unlikely. Also, if you have an
index, you know any forward pointer that points beyond the next index
entry is invalid.

> The demuxer will find the error and will try to resync (look for startcode).
> Unfortunatly we will think that only this packet is damaged and we
> will sync a long after the problematic position.

No, only a very broken demuxer would do this.

> > What good does this do? Knowing that the data is corrupt is totally
> > useless unless you can repair it. All CRC does is waste space. CRC is
> > absolutely not acceptable!!
> 
> Well, ther are error correction codes available. I have seen
> e.g. rar file to repair and fair big chunk of bytes (512),
> using only 1% overhead (in 1mb file).
> I'm not very skilled in this area but this is the only
> way to learn:)

I don't think you could get recoverability with any reasonable amount
of overhead. 1% of the header sizes would be totally acceptable, but
the only way you could do this (IMO) is with very large block sizes
(like RAR uses) which are not an option. Recovery must be very
localized.

> You cannot have error resistance without some kind of check codes.

Forward/backwards pointers are check codes. So is an index.

> Otherwise I would recomend to remove all crc in all data and to use
> MPEG-TS (or some other format that have error resistance)
> as container above nut.

ROTFL, this is nonsense.

There are two options/possibilities:

1. Error recovery can be implemented with nothing but the reasonable
   data already present in the file. In this case, we use it.

2. Error recovery can't be implemented without wasting extra space on
   useless data. In this case, it DOES NOT BELONG IN NUT. I would
   rather have a high-quality video than waste space for idiots who
   will be corrupting their files. Keep in mind that anyone who is
   serious about collecting videos already has _external_ hash
   checking (crc, md5sum, or builtin to their p2p program, etc.) and
   will _delete_ any file found to be corrupt and restore from a
   "backup". :)

Rich