Today, lavfi filters are provided a hw_device from the first
hwdec_interop that was loaded, regardless of whether it's the right one
or not. In most situations where a hardware based filter is used, we
need more control over the device.
In this change, a `hwdec_interop` option is added to the lavfi wrapper
filter configuration and this is used to pick the correct hw_device to
inject into the filter or graph (in the case of a graph, all filters
get the same device).
Note that this requires the use of the explicit lavfi syntax to allow
for the extra configuration.
eg:
```
mpv --vf=hwupload
```
becomes
```
mpv --vf=lavfi=[hwupload]:hwdec_interop=cuda-nvdec
```
or
```
mpv --vf=lavfi-bridge=[hwupload]:hwdec_interop=cuda-nvdec
```
It turns out that it's generally more useful to look up hwdecs by image
format, rather than device type. In the situations where we need to
find one, we generally know the image format we're dealing with. Doing
this avoids us having to create mappings from image format to device
type.
The most significant part of this change is filling in the image format
for the various hw interops. There is a hw_imgfmt field today today, but
only a couple of the interops fill it in, and that seems to be because
we've never actually used this piece of metadata before. Well, now we
have a good use for it.
Change all OPT_* macros such that they don't define the entire m_option
initializer, and instead expand only to a part of it, which sets certain
fields. This requires changing almost every option declaration, because
they all use these macros. A declaration now always starts with
{"name", ...
followed by designated initializers only (possibly wrapped in macros).
The OPT_* macros now initialize the .offset and .type fields only,
sometimes also .priv and others.
I think this change makes the option macros less tricky. The old code
had to stuff everything into macro arguments (and attempted to allow
setting arbitrary fields by letting the user pass designated
initializers in the vararg parts). Some of this was made messy due to
C99 and C11 not allowing 0-sized varargs with ',' removal. It's also
possible that this change is pointless, other than cosmetic preferences.
Not too happy about some things. For example, the OPT_CHOICE()
indentation I applied looks a bit ugly.
Much of this change was done with regex search&replace, but some places
required manual editing. In particular, code in "obscure" areas (which I
didn't include in compilation) might be broken now.
In wayland_common.c the author of some option declarations confused the
flags parameter with the default value (though the default value was
also properly set below). I fixed this with this change.
This reverts commit 6385a5fd1b, and in
addition removes the code that automatically inserts the vavpp filter.
The reason is the same as the commit that is being reverted: this
filter seems to trigger driver bugs. It can cause GPU freezes or
just doesn't work.
This variant of disabling the filter is better. There was no way to
add the "force" parameter to the automatically inserted filter, so
the old approach just made manual filter insertion (with the --vf
option or "vf" command) more cumbersome.
This switches the default away from "bob" to the best algorithm reported
as supported by the driver. This is convenient for users, and there is
no reason to use something worse by default.
Untested.
Before this, we made deinterlacing dependent on the video codec metadata
(AVFrame.interlaced_frame for libavcodec). So even if --deinterlace=yes
was set, we skipped deinterlacing if the flag wasn't set. This is very
unreliable and there are many streams with flags incorrectly set.
The potential problem is that this might upset people who alwase enabled
deinterlace and hoped it worked. But it's likely these people were
screwed by this setting anyway. The new behavior is less tricky and
easier to understand, and this preferable. Maybe one day we could
introduce a --deinterlace=auto, which does the right thing, but of
course this would be hard to implement (esecially with hwdec).
Fixes#5219.
Get rid of the old vf.c code. Replace it with a generic filtering
framework, which can potentially handle more than just --vf. At least
reimplementing --af with this code is planned.
This changes some --vf semantics (including runtime behavior and the
"vf" command). The most important ones are listed in interface-changes.
vf_convert.c is renamed to f_swscale.c. It is now an internal filter
that can not be inserted by the user manually.
f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed
once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is
conceptually easy, but a big mess due to the data flow changes).
The existing filters are all changed heavily. The data flow of the new
filter framework is different. Especially EOF handling changes - EOF is
now a "frame" rather than a state, and must be passed through exactly
once.
Another major thing is that all filters must support dynamic format
changes. The filter reconfig() function goes away. (This sounds complex,
but since all filters need to handle EOF draining anyway, they can use
the same code, and it removes the mess with reconfig() having to predict
the output format, which completely breaks with libavfilter anyway.)
In addition, there is no automatic format negotiation or conversion.
libavfilter's primitive and insufficient API simply doesn't allow us to
do this in a reasonable way. Instead, filters can use f_autoconvert as
sub-filter, and tell it which formats they support. This filter will in
turn add actual conversion filters, such as f_swscale, to perform
necessary format changes.
vf_vapoursynth.c uses the same basic principle of operation as before,
but with worryingly different details in data flow. Still appears to
work.
The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are
heavily changed. Fortunately, they all used refqueue.c, which is for
sharing the data flow logic (especially for managing future/past
surfaces and such). It turns out it can be used to factor out most of
the data flow. Some of these filters accepted software input. Instead of
having ad-hoc upload code in each filter, surface upload is now
delegated to f_autoconvert, which can use f_hwupload to perform this.
Exporting VO capabilities is still a big mess (mp_stream_info stuff).
The D3D11 code drops the redundant image formats, and all code uses the
hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a
big mess for now.
f_async_queue is unused.
Finally get rid of all the HWDEC_* things, and instead rely on the
libavutil equivalents. vdpau still uses a shitty hack, but fuck the
vdpau code.
Remove all the now unneeded remains. The vdpau preemption thing was not
unused anymore; if someone cares this could probably be restored.
We allowed any input format that was generally supported by libva, but
this is probably nonsense, as the actual surface format was always fixed
to nv12. We would have to check whether libva can upload a given pixel
format to a nv12 surface. Or we would have to use a separate frame pool
for input surfaces with the exact sw_format - but then we'd also need to
check whether the vaapi VideoProc supports the surface type.
Hardcode nv12 and yuv420p as input formats, which we know can be
uploaded to nv12 surfaces. In theory we could get a list of supported
upload formats from libavutil, but that also require allocating a dummy
hw frames context just for the query.
Add a comment to the upload code why we can allocate an output surface
for input.
In the long run, we'll probably want to use libavfilter's vaapi
deinterlacer, but for now this would break at least user options.
This removes all GPL only code from it, and that's the whole purpose.
Also happens to be much simpler.
The "deinterlace" option still sort of exists, but only as runtime
changeable option. The main change in behavior is that the property will
not report back the actual deint state. Or in other words, if inserting
or initializing the filter fails, the deinterlace property will still
return "yes". This is in line with most recent behavior changes to
properties and options.
I'm not sure what's going on here, but it appears kodi switches forward
and backwards references for advanced VPP deinterlacing modes. This in
turn makes deinterlacing with these modes apparently work. If you don't
switch the directions, you get a stuttering mess.
As far as the libva trace dump is concerned, this makes mpv's libva
deinterlacing API use behave like kodi's, and appears to reproduce
smooth video with advanced libva deinterlacing enabled.
I'm hearing that Mesa actually does it correctly, and I'm not sure what
will happen there. For now, passing "reversal-bug=no" as sub-option to
the vavpp filter will undo this behavior.
Don't give the driver more forward/backward refernces than it requested
in num_forward_references/num_backward_references. This shouldn't
matter, I'm just trying to play it safe.
This makes the difference between passing VA_FRAME_PICTURE or
VA_BOTTOM_FIELD for progressive frames (that should be force-
deinterlaced) to VAProcPipelineParameterBuffer.flags. VA-VPP doesn't
really seem to care, and we can get rid of mp_refqueue_is_interlaced()
entirely. It could be argued it's better to pass field flags instead of
the progressive flag.
This has two reasons:
1. I tend to add new fields to this metadata, and every time I've done
so I've consistently forgotten to update all of the dozens of places in
which this colorimetry metadata might end up getting used. While most
usages don't really care about most of the metadata, sometimes the
intend was simply to “copy” the colorimetry metadata from one struct to
another. With this being inside a substruct, those lines of code can now
simply read a.color = b.color without having to care about added or
removed fields.
2. It makes the type definitions nicer for upcoming refactors.
In going through all of the usages, I also expanded a few where I felt
that omitting the “young” fields was a bug.
This makes vf_vdpaupp use the deinterlacer helper code already used by
vf_vavpp. I nice side-effect is that this also removes some traces of
code originating from vo_vdpau.c, so we can switch it to LGPL.
Extend the refqueue helper with a deint setting. If not set,
mp_refqueue_should_deint() always returns false, which slightly
simplifies vf_vdpaupp. It's of no consequence to vf_vavpp (other than it
has to set it to get expected behavior).
Abstracts the annoying framerate-doubling behavior.
Same deal as with refqueue introduction: the code size blows up, but at
least it can be reused for other filters.
Calling this right at start of filter_ext() also fixes a small
regression from previous commit. The change in reference surfaces due to
the first update_pipeline() with deinterlacing enabled changed behavior
of mp_refqueue_next() and mp_refqueue_has_output(). Since
update_pipeline() was called between those, the frame output logic got
inconsistent, and the first deinterlaced frame was duplicated from the
previous non-deinterlaced frame.
Also reset the number of ref-frames when switching back to non-deint
mode.
Move the handling of the future/past frames and the associated dataflow
rules to a separate source file.
While this on its own seems rather questionable and just inflates the
code, I intend to reuse it for other filters. The logic is annoying
enough that it shouldn't be duplicated a bunch of times.
(I considered other ways of sharing this logic, such as an uber-
deinterlace filter, which would access the hardware deinterlacer via a
different API. Although that sounds like kind of the right approach,
this would have other problems, so let's not, at least for now.)
The main change is with video/hwdec.h. mp_hwdec_info is made opaque (and
renamed to mp_hwdec_devices). Its accessors are mainly thread-safe (or
documented where not), which makes the whole thing saner and cleaner. In
particular, thread-safety rules become less subtle and more obvious.
The new internal API makes it easier to support multiple OpenGL interop
backends. (Although this is not done yet, and it's not clear whether it
ever will.)
This also removes all the API-specific fields from mp_hwdec_ctx and
replaces them with a "ctx" field. For d3d in particular, we drop the
mp_d3d_ctx struct completely, and pass the interfaces directly.
Remove the emulation checks from vaapi.c and vdpau.c; they are
pointless, and the checks that matter are done on the VO layer.
The d3d hardware decoders might slightly change behavior: dxva2-copy
will not use the VO device anymore if the VO supports proper interop.
This pretty much assumes that any in such cases the VO will not use any
form of exclusive mode, which makes using the VO device in copy mode
unnecessary.
This is a big refactor. Some things may be untested and could be broken.
Until now, we have made the assumption that a driver will use only 1
hardware surface format. the format is dictated by the driver (you
don't create surfaces with a specific format - you just pass a
rt_format and get a surface that will be in a specific driver-chosen
format).
In particular, the renderer created a dummy surface to probe the format,
and hoped the decoder would produce the same format. Due to a driver
bug this required a workaround to actually get the same format as the
driver did.
Change this so that the format is determined in the decoder. The format
is then passed down as hw_subfmt, which allows the renderer to configure
itself with the correct format. If the hardware surface changes its
format midstream, the renderer can be reconfigured using the normal
mechanisms.
This calls va_surface_init_subformat() each time after the decoder
returns a surface. Since libavcodec/AVFrame has no concept of sub-
formats, this is unavoidable. It creates and destroys a derived
VAImage, but this shouldn't have any bad performance effects (at
least I didn't notice any measurable effects).
Note that vaDeriveImage() failures are silently ignored as some
drivers (the vdpau wrapper) support neither vaDeriveImage, nor EGL
interop. In addition, we still probe whether we can map an image
in the EGL interop code. This is important as it's the only way
to determine whether EGL interop is supported at all. With respect
to the driver bug mentioned above, it doesn't matter which format
the test surface has.
In vf_vavpp, also remove the rt_format guessing business. I think the
existing logic was a bit meaningless anyway. It's not even a given
that vavpp produces the same rt_format for output.
This can happen if the hw decoder allocates padded surfaces (e.g.
mod16), but the VPP output surface was allocated with the exact size.
Apparently VPP requires matching input and output sizes, or it will add
artifacts. In this case, it added mirrored pixels to the bottom few
pixels.
Note that the previous commit should have fixed this. But it didn't
work, while this commit does.
Fixes#2320.
If not set, VPP will use the whole surface. This is a problem if the
surfaces are padded, and especially if the surfaces are padded by
different amounts.
This is an attempt to fix#2320, but it appears to do nothing at all.
This was missing for extended deinterlacer.
Unfortunately, these deinterlacer still do not work. The provided future
frame (which is all the deinterlacers want) seems to be correct, though.
One minor behavioral change is that this always keeps the previous frame
for PTS computations. This could be avoided (in order to keep exactly
the same behavior as before), but it seems more elegant and should not
do any harm. (Also, if we really cared about reducing hw frame refs,
a more worthy goal is producing the field output incrementally.)
The mapped data (pointed to by the param variable) is not needed before,
so the call can be moved down. Also, this prevents that the buffer
remains mapped forever if the other vaMapBuffer() call above fails (the
cleanup code forgets to unmap the buffer - this commit makes it
unnecessary).
This used a do-while loop, which runs only once, as replacement for a
cleanup goto. While this is ok, doing a goto directly is easier to
follow and is closer to idiomatic C. But mainly remove it so that the
indentation can be reduced.
We do not fill them, so we would pass random IDs to the driver. The code
was originally written to handle bob deinterlacing only, so I guess it
originally passed always 0 anyway, despite having code for reference
surface list allocation.
Also, move down the vaUnmapBuffer() call. This call actually "unmaps"
the param pointer, so accessing it after the unmap call would be
undefined behavior. The "example" in <va/vavpp.h> does this too, but
it's most likely an error.
(Additionally, not even bob deinterlacing worked correctly in my test,
sigh.)
Add filter parameters to VAAPI deinterlacing filter to actually process
bottom fields instead of deinterlacing top field twice.
Signed-off-by: wm4 <wm4@nowhere>
Before this commit, each hw backend had their own specific struct types
for context, and some, like VDA, had none at all. Add a context struct
(mp_hwdec_ctx) that provides a somewhat generic way to pass the hwdec
context around. Some things get slightly better, some slightly more
verbose.
mp_hwdec_info is still around; it's still needed, but is reduced to its
role of handling delayed loading of the hwdec backend.
Or in other words, add support for properly draining remaining frames
from video filters. vf_yadif is buffering at least one frame, and the
buffered frame was not retrieved on EOF.
For most filters, ignore this for now, and just adjust them to the
changed semantics of filter_ext. But for vf_lavfi (used by vf_yadif),
real support is implemented. libavfilter handles this simply by passing
a NULL frame to av_buffersrc_add_frame(), so we just have to make
mp_to_av() handle NULL arguments.
In load_next_vo_frame(), we first try to output a frame buffered in the
VO, then the filter, and then (if EOF is reached and there's still no
new frame) the VO again, with draining enabled. I guess this was
implemented slightly incorrectly before, because the filter chain still
could have had remaining output frames.
It's not really needed to be public. Other code can just use mp_image.
The only disadvantage is that the other code needs to call an accessor
to get the VASurfaceID.
Although I at first thought it would be better to have a separate
implementation for hwaccels because the difference to software images
are too large, it turns out you can actually save some code with it.
Note that the old implementation had a small memory management bug. This
got painted over in commit 269c1e1, but is hereby solved properly.
Also note that I couldn't test vf_vavpp.c (due to lack of hardware), and
I hope I didn't accidentally break it.
Thomas Weißschuh