Use the UTF-16 BOM to detect UTF-16 encoding. Convert the file contents
to UTF-8 on the fly using FFTextReader, which acts as converting wrapper
around AVIOContext. It also can work on a static buffer, needed for
format probing. The FFTextReader wrapper now also takes care of skipping
the UTF-8 BOM.
Fix Ticket #3496.
Here is an extract of fate-samples/sub/vobsub.idx, with an additional
text at the end of each line to better identify each bitmap:
timestamp: 00:04:55:445, filepos: 00001b000 Ace!
timestamp: 00:05:00:049, filepos: 00001b800 Wake up, honey!
timestamp: 00:05:02:018, filepos: 00001c800 I gotta go to work.
timestamp: 00:05:02:035, filepos: 00001d000 <???>
timestamp: 00:05:04:203, filepos: 00001d800 Look after Clayton, okay?
timestamp: 00:05:05:947, filepos: 00001e800 I'll be back tonight.
timestamp: 00:05:07:957, filepos: 00001f800 Bye! Love you.
timestamp: 00:05:21:295, filepos: 000020800 Hey, Ace! What's up?
timestamp: 00:05:23:356, filepos: 000021800 Hey, how's it going?
timestamp: 00:05:24:640, filepos: 000022800 Remember what today is? The 3rd!
timestamp: 00:05:27:193, filepos: 000023800 Look over there!
timestamp: 00:05:28:369, filepos: 000024800 Where are they going?
timestamp: 00:05:28:361, filepos: 000025000 <???>
timestamp: 00:05:29:946, filepos: 000025800 Let's go see.
timestamp: 00:05:31:230, filepos: 000026000 I can't, man. I got Clayton.
Note the two "<???>": they are basically split subtitles (with the
previous one), which the dvdsub decoder is now supposed to reconstruct
with a previous commit. But also note that while the first chunk has
increasing timestamps,
timestamp: 00:05:02:018, filepos: 00001c800
timestamp: 00:05:02:035, filepos: 00001d000
...it's not the case of the second one (and this is not an exception in the
original file):
timestamp: 00:05:28:369, filepos: 000024800
timestamp: 00:05:28:361, filepos: 000025000
For the dvdsub decoder, they need to be "filepos'ed" ordered, but the
FFDemuxSubtitlesQueue is timestamps ordered, which is the reason of the
introduction of a sub sort method in the context, to allow giving
priority to the position, and then the timestamps. With that change, the
dvdsub decoder get fed with ordered packets.
Now the packet size estimation was also broken: the filepos differences
in the vobsub index defines the full data read between two subtitles
chunks, and it is necessary to take into account what is read by the
mpegps_read_pes_header() function since the length returned by that
function doesn't count the size of the data it reads. This is fixed with
the introduction of total_read, and {old,new}_pos. By doing this change,
we can drop the unreliable len16 heuristic and simplify the whole loop.
Note that mpegps_read_pes_header() often read more than one PES packet
(typically in one call it can read 0x1ba and 0x1be chunk along with the
relevant 0x1bd packet), which triggers the "total_read + pkt_size >
psize" check. This is an expected behaviour, which could be avoided by
having a more chunked version of mpegps_read_pes_header().
The latest change is the extraction of each stream into its own
subtitles queue. If we don't do this, the maximum size for a subtitle
chunk is broken, and the previous changes can not work. Having each
stream in a different queue requires some little adjustments in the
seek code of the demuxer.
This commit is only meaningful as a whole change and can not be easily
split. The FATE test changes because it uses the vobsub demuxer.
This fixes a bunch of possible overread in avformat with the idiom p +=
strcspn(p, "\n") + 1 (strcspn() can focus on the trailing '\0' if no
'\n' is found, so the +1 leads to an overread).
Note on lavf/matroskaenc: no extra subtitles.o Makefile dependency is
added because only the header is required for ff_subtitles_next_line().
Note on lavf/mpsubdec: code gets slightly complex to avoid an infinite
loop in the probing since there is no more forced increment.
This function is almost identical to lavf/assdec:read_seek2(). It
performs a generic seek for text subtitles demuxers for the new seeking
API.
The only difference with assdec:read_seek2 is the ts_diff being
unsigned to avoid overflows.
The seek callback in the ASS demuxer will be removed when it is
redesigned to use FFDemuxSubtitlesQueue.
Clément Bœsch