demux: allow backward cache to use unused forward cache

Until now, the following could happen: if you set a 1GB forward cache,
and a 1GB backward cache, and you opened a 2GB file, it would prune away
the data cached at the start as playback progressed past the 50% mark.

With this commit, nothing gets pruned, because the total memory usage
will still be 2GB, which equals the total allowed memory usage of 1GB +
1GB.

There are no explicit buffers (every packet is malloc'ed and put into a
linked list), so it all comes down to buffer size computations. Both
reader and prune code use these sizes to decide whether a new packet
should be read / an old packet discarded. So just add the remaining free
"space" from the forward buffer to the available backward buffer. Still
respect if the back buffer is set to 0 (e.g. unseekable cache where it
doesn't make sense to keep old packets).

We need to make sure that the forward buffer can always append, as long
as the forward buffer doesn't exceed the set size, even if the back
buffer "borrows" free space from it. For this reason, always keep 1 byte
free, which is enough to allow it to read a new packet. Also, it's now
necessary to call pruning when adding a packet, to get back "borrowed"
space that may need to be free'd up after a packet has been added.

I refrained from doing the same for forward caching (making forward
cache use unused backward cache). This would work, but has a
disadvantage. Assume playback starts paused. Demuxing will stop once the
total allowed low total cache size is reached. When unpausing, the
forward buffer will slowly move to the back buffer. That alone will not
change the total buffer size, so demuxing remains stopped. Playback
would need to pass over data of the size of the back buffer until
demuxing resume; consider this unacceptable. Live playback would break
(or rather, would not resume in unintuitive ways), even normal streaming
may break if the server invalidates the URL due to inactivity. As an
alternative implementation, you could prune the back buffer immediately,
so the forward buffer can grow, but then the back buffer would never
grow. Also makes no sense.

As far as the user interface is concerned, the idea is that the limits
on their own aren't really meaningful, the purpose is merely to vaguely
restrict the cache memory usage. There could be just a single option to
set the total allowed memory usage, but the separate backward cache
controls the default ratio of backward/forward cache sizes. From that
perspective, it doesn't matter if the backward cache uses more of the
total buffer than assigned, if the forward buffer is complete.

DOCS/man/options.rst

@@ -3204,6 +3204,13 @@ Demuxer
     the situation that the forward seek range starts after the current playback
     position (as it removes past packets that are seek points).
 
+    If the end of the file is reached, the remaining unused forward buffer space
+    is "donated" to the backbuffer (unless the backbuffer size is set to 0).
+    This still limits the total cache usage to the sum of the forward and
+    backward cache, and effectively makes better use of the total allowed memory
+    budget. (The opposite does not happen: free backward buffer is never
+    "donated" to the forward buffer.)
+
     Keep in mind that other buffers in the player (like decoders) will cause the
     demuxer to cache "future" frames in the back buffer, which can skew the
     impression about how much data the backbuffer contains.

demux/demux.c

@@ -429,6 +429,7 @@ static struct demux_packet *compute_keyframe_times(struct demux_packet *pkt,
 static void find_backward_restart_pos(struct demux_stream *ds);
 static struct demux_packet *find_seek_target(struct demux_queue *queue,
                                              double pts, int flags);
+static void prune_old_packets(struct demux_internal *in);
 
 static uint64_t get_foward_buffered_bytes(struct demux_stream *ds)
 {
@@ -2030,6 +2031,9 @@ static void add_packet_locked(struct sh_stream *stream, demux_packet_t *dp)
 
     adjust_seek_range_on_packet(ds, dp);
 
+    // May need to reduce backward cache.
+    prune_old_packets(in);
+
     // Possibly update duration based on highest TS demuxed (but ignore subs).
     if (stream->type != STREAM_SUB) {
         if (dp->segmented)
@@ -2192,7 +2196,12 @@ static void prune_old_packets(struct demux_internal *in)
             struct demux_stream *ds = in->streams[n]->ds;
             fw_bytes += get_foward_buffered_bytes(ds);
         }
-        if (in->total_bytes - fw_bytes <= max_bytes)
+        uint64_t max_avail = max_bytes;
+        // Backward cache (if enabled at all) can use unused forward cache.
+        // Still leave 1 byte free, so the read_packet logic doesn't get stuck.
+        if (max_avail && in->max_bytes > (fw_bytes + 1))
+            max_avail += in->max_bytes - (fw_bytes + 1);
+        if (in->total_bytes - fw_bytes <= max_avail)
             break;
 
         // (Start from least recently used range.)