With the introduction of HEVC 444 support, we technically have two
codecs that can handle 444 - HEVC and MJPEG. In the case of MJPEG,
it can decode, but can only output one of the semi-planar formats.
That means we need additional logic to decide whether to use a
444 output format or not.
The 'simple' hwaccels (not h.264 and hevc) all use the same bitstream
management and reference lookup logic so let's refactor all that into
common functions.
I verified that casting a signed int -1 to unsigned char produces 255
according to the C language specification.
I took the reference lookup code from the vp9 hwaccel where the
type is unsigned char, but for vc1, the type is signed int.
This is particularly important because the value used when there's
no reference is different (255 vs -1).
It didn't seem to break anything, but for mpeg1/2/4, this mistake
caused decode errors.
Philip Langdale