add recently added view

README.md

@@ -3,27 +3,67 @@
 This project will be awesome, only if I invest enough time. This software will replace my
 current cronjob yt-dlp archive service.
 
-Partially inspired by [hobune](https://github.com/rebane2001/hobune). While that project is amazing by it's own, it's just not scaleable.
+Partially inspired by [hobune](https://github.com/rebane2001/hobune). While that project is amazingby it's own, it's just not scaleable.
 
 ## The idea
-The new setup will either be fully running in flask, including the task that checks the
+Having over 250k videos, scaling the current cronjob yt-dlp archive task is just really hard. Filetypes change, things get partially downloaded and such.
-youtube channels every x hours. Or Flask will be used as the gui frontend, and a seperate
+Partially yt-dlp is to blame because it's a package that needs to change all the time. But with this some changes are not accounted for.
-script will do the channel archiving. I have not desided yet.
+yt-dlp will still do the downloads. But a flask frontend will be developed to make all downloaded videos easily indexable.
-
+For it to be quick (unlike hobune) a database has to be implemented. This could get solved by a static site generator type of software, but that is not my choice.
-What currently works is that the gui frontend calls to a seperate database while a cronjob
-handles the downloading of new videos from a list of channels.
 
+The whole software package will use postgresql as a data backend and celery as background tasks.
+Currently development however is using mongodb just because it's easy.
 
 ## How it works currently(legacy)
 In the legacy folder you will find files that are currently in my archive project. How it works is
-that I have a cronjob running every 6 hours what then runs yt-dlp with a config file. In that config
+that I have a cronjob running every 24 hours what then runs yt-dlp with a config file. In that config
 file a channel list contains all the channels that yt-dlp needs to update. If a new video has been
 uploaded, yt-dlp will automatically download a 720p version of the video, all subtitles at that time
 (rip community captions, will not forget you) and a json file with all the rest of the metadata. Oh 
 and also the thumbnail.
 
 This works. But is very slow and uses lots of "API" calls to youtube, which will sometimes will get
-the IP blocked. This needs to be overhauled.
+the IP blocked. This is why full channel upload pages are not downloaded anymore, I have limited to first 50 videos.
+
+## Goals
+Some goals have been set up which will prioritise functionality for the software package.
+The starting status is that info.json files of videos are loaded into the mongodb database on which flask
+will generate a page for channels and videos to load. But this has major limitations which will not be described right now
+but will be reflected in the goals.
+
+### Stage 1
+Tasks which have to be finished before the GUI frontend is usable as a manager and user in no perticular order.
+- [x] Have videos and channels listed on a page
+- [x] Have a secured admin page where the database can be managed
+- [x] Have working video streaming
+- [x] CI/CD pipeline for quicker deployment
+- [x] Add caching to speed up pages
+- [x] Add ratelimiting for expensive pages
+- [x] Ability to show cronjob logs to easily troubleshoot
+
+### Stage 2
+Extra functionality for further development of features.
+- [x] Fix video titles on disk with slugs
+- [x] Working search functionality
+- [x] Video reporting functionality
+- [x] Ability (for external applications) to queue up video ids for download
+- [x] Add websub requesting and receiving ability. (not fully usable yet without celery tasks)
+- [] OIDC or Webauthn logins instead of static argon2 passwords
+
+### Stage 3
+Mainly focused on retiring the cronjob based scripts and moving it to celery based tasks
+- [] manage videos by ID's instead of per channel basis
+- [] download videos from queue
+- [] Manage websub callbacks
+
+### Stage 4
+Mongodb finally has it's limitations.
+- [] Migrate to postgresql
+
+### Stage ...
+Since this is my flagship software which I have developed more features will be added.
+It may take some time since this is just a hobby for me. And I'm not a programmer by title.
+
 
 ## Things learned
 ### Video playlists
@@ -50,26 +90,22 @@ If you swap the channel name to channel id. The folders will never change.
 ### Storage structure
 The following folder structure is pretty nice for using static scripts. The one drawback
 is that you can't search for video id's or titles. Because the search takes too long.
-This is mainly why we need a new system using a database.
+This is mainly why we need a new system using a database mainly for search.
-```
-./videos/{channel_id}/{upload_date}/{video_id}/video_title.mp4
-```
-For the new system using a blob like storage will be key. I had the following in mind. It will be an independant
-random key and not the YouTube video ID because I have notices that multiple real videos exist under the same key by
-uploaders who replace old videos.
 
+The following structure is easily scaleable and usable in a object storage format.
 ```
--| data
+./videos/{channel_id}/{video_id}/video-title-slug-format.info.json
- | - videos
-   | - 128bit_random_id.mp4
- | - subtitles
-   | - same_random_id_EN.srt
-   | - same_random_id_DE.srt
- | - thumbnails
-    | - 128bit_random_id.jpg
 ```
 
 ## API things learned
+### YouTube push notifications in API form exist
+Using the pubsubhubbub service provided by Google we will implement downloading videos based on uploads.
+The API is based on WebSub which is greatly documented.
+
+The hub will give xml+atom notifications when a video is uploaded by a channel and when a video is deleted.
+The goal is to download a video when a notification gets trough, and run a full channel sync when a video is deleted.
+This will be next to periodic full channel polling to download videos which the hub has not notified us about.
+
 ### Etag is useful
 When we will call the api for 50 items in a playlist we also get an etag back. 
 This is a sort of hash of the returned data. 

ayta/__init__.py

@@ -1,15 +1,13 @@
 def create_app(test_config=None):
     import os, secrets
     from flask import Flask
-    from ayta.extensions import limiter, caching, celery_init_app
+    from ayta.extensions import limiter, caching, celery_init_app, oidc
     from werkzeug.middleware.proxy_fix import ProxyFix
 
     from . import filters
 
     config = {'MONGO_CONNECTION': os.environ.get('AYTA_MONGOCONNECTION', 'mongodb://root:example@192.168.66.140:27017'),
-              'S3_CONNECTION': os.environ.get('AYTA_S3CONNECTION', '192.168.66.111:9001'),
+              'OIDC_CLIENT_SECRETS': os.environ.get('AYTA_OIDC_PATH', 'client_secrets.json'),
-              'S3_ACCESSKEY': os.environ.get('AYTA_S3ACCESSKEY', 'lnUiGClFVXVuZbsr'),
-              'S3_SECRETKEY': os.environ.get('AYTA_S3SECRETKEY', 'Qz9NG7rpcOWdK2WL'),
               'CACHE_TYPE': os.environ.get('AYTA_CACHETYPE', 'SimpleCache'),
               'CACHE_DEFAULT_TIMEOUT': int(os.environ.get('AYTA_CACHETIMEOUT', 6)),
               'SECRET_KEY': os.environ.get('AYTA_SECRETKEY', secrets.token_hex(32)),
@@ -19,6 +17,10 @@ def create_app(test_config=None):
                              task_ignore_result=True,)
              }
              
+    # Static configuration settings, do not change
+    
+    config['OIDC_CALLBACK_ROUTE'] = '/api/oidc/callback'  # why is this excension not using it? maybe i should implement oidc by myself?
+    
     app = Flask(__name__)
     app.config.from_mapping(config)
     
@@ -26,6 +28,9 @@ def create_app(test_config=None):
     caching.init_app(app)
     celery_init_app(app)
     
+    if app.config['OIDC_CLIENT_SECRETS']:
+        oidc.init_app(app)
+    
     app.wsgi_app = ProxyFix(app.wsgi_app, x_for=1)
     
     app.jinja_env.filters['pretty_duration'] = filters.pretty_duration
@@ -39,7 +44,7 @@ def create_app(test_config=None):
     from .blueprints import search
     from .blueprints import channel
     from .blueprints import auth
-    from .blueprints import websub
+    from .blueprints import api
 
     app.register_blueprint(watch.bp)
     app.register_blueprint(index.bp)
@@ -47,6 +52,6 @@ def create_app(test_config=None):
     app.register_blueprint(search.bp)
     app.register_blueprint(channel.bp)
     app.register_blueprint(auth.bp)
-    app.register_blueprint(websub.bp)
+    app.register_blueprint(api.bp)
 
     return app

ayta/blueprints/admin.py

@@ -5,7 +5,7 @@ from ..dlp import checkChannelId, getChannelInfo
 from ..decorators import login_required
 from ..tasks import subscribe_websub_callback, unsubscribe_websub_callback
 from datetime import datetime
-import requests
+from secrets import token_urlsafe
 
 bp = Blueprint('admin', __name__, url_prefix='/admin')
 
@@ -150,6 +150,50 @@ def reports():
    
     return render_template('admin/reports.html', reports=reports)
 
+@bp.route('/posters', methods=['GET', 'POST'])
+@login_required
+def posters():
+    if request.method == 'POST':
+        task = request.form.get('task', None)
+        value = request.form.get('value', None)
+        
+        if task == 'add-endpoint':
+            description = request.form.get('description', None)
+            if not description or len(description) <= 7:
+                flash('Description must be at least 8 characters long')
+            
+            if value and len(value) >= 12:
+                get_nosql().poster_newEndpoint(value, description)
+                flash(f'Created endpoint ID: {value}')
+            else:
+                value = token_urlsafe(16)
+                get_nosql().poster_newEndpoint(value, description)
+                flash(f'Created endpoint ID: {value}')
+        elif task == 'retire':
+            get_nosql().poster_retireEndpoint(value)
+            flash(f'Endpoint retired: {value}')
+            
+        elif task == 'clean-retired':
+            get_nosql().poster_cleanRetired()
+            flash(f'Cleaned retired endpoints')
+            
+        elif task == 'manual-queue':
+            get_nosql().poster_insertQueue('manual', value)
+            flash(f'Added to queue: {value}')
+            
+        elif task == 'delete-queue':
+            get_nosql().poster_deleteQueue(value)
+            flash(f'Deleted from queue: {value}')
+        
+        return redirect(url_for('admin.posters'))
+        
+    endpoints = get_nosql().poster_getEndpoints()
+    queue = get_nosql().poster_getQueue()
+   
+    return render_template('admin/posters.html', endpoints=endpoints, queue=queue)
+
+
+
 @bp.route('/files', methods=['GET', 'POST'])
 @login_required
 def files():

ayta/blueprints/api.py

@@ -2,10 +2,12 @@ from flask import Blueprint, render_template, request, redirect, url_for, flash,
 from ..nosql import get_nosql
 from ..extensions import caching, caching_unless
 
-bp = Blueprint('websub', __name__, url_prefix='/websub')
+import re
 
-@bp.route('/c/<cap>', methods=['GET', 'POST'])
+bp = Blueprint('api', __name__, url_prefix='/api')
-def callback(cap):
+
+@bp.route('/websub/<cap>', methods=['GET', 'POST'])
+def websub(cap):
     if request.method == 'GET':
         topic = request.args.get('hub.topic')
         challenge = request.args.get('hub.challenge')
@@ -36,3 +38,29 @@ def callback(cap):
         return '', 202
     
     return abort(404)
+    
+@bp.route('/poster/<cap>', methods=['POST'])
+def poster(cap):
+    # if endpoint does not exist
+    if not get_nosql().poster_isActive(cap):
+        return abort(404)
+    
+    videoId = request.form.get('v')
+    
+    # if request is not valid
+    if not videoId:
+        return abort(400)
+        
+    # if requested string is not correct
+    if not re.match(r"^[a-zA-Z0-9_-]{11}$", videoId):
+        return abort(422)
+        
+    # if given string is already in the archive
+    if get_nosql().check_exists(videoId):
+        return abort(409)
+    
+    # try to insert
+    if get_nosql().poster_insertQueue(cap, videoId):
+        return '', 202
+    else:
+        return abort(409)

ayta/blueprints/channel.py

@@ -8,12 +8,15 @@ bp = Blueprint('channel', __name__, url_prefix='/channel')
 @bp.route('')
 @caching.cached(unless=caching_unless)
 def base():
-    channels = {}
+    channels = []
     channelIds = get_nosql().list_all_channels()
     
     for channelId in channelIds:
-        channels[channelId] = get_nosql().get_channel_info(channelId)
+        channel = get_nosql().get_channel_info(channelId)
-        channels[channelId]['video_count'] = get_nosql().get_channel_videos_count(channelId)
+        channel['video_count'] = get_nosql().get_channel_videos_count(channelId)
+        channels.append(channel)
+        
+    channels = sorted(channels, key=lambda x: x.get('added_date'), reverse=True)
         
     return render_template('channel/index.html', channels=channels)
     
@@ -41,8 +44,23 @@ def channel(channelId):
 def orphaned():
     videoIds = get_nosql().get_orphaned_videos()
     
-    videos = {}
+    videos = []
     for videoId in videoIds:
-        videos[videoId] = get_nosql().get_video_info(videoId, limited=True)
+        videos.append(get_nosql().get_video_info(videoId, limited=True))
+        
+    videos = sorted(videos, key=lambda x: x.get('epoch', 0), reverse=True)
 
     return render_template('channel/orphaned.html', videos=videos)
+    
+@bp.route('/recent')
+@caching.cached(unless=caching_unless)
+def recent():
+    videoIds = get_nosql().get_recent_videos()
+    
+    videos = []
+    for videoId in videoIds:
+        videos.append(get_nosql().get_video_info(videoId, limited=True))
+        
+    videos = sorted(videos, key=lambda x: x.get('epoch', 0), reverse=True)
+
+    return render_template('channel/recent.html', videos=videos)

ayta/extensions.py

@@ -3,10 +3,12 @@ from flask_limiter.util import get_remote_address
 
 from flask_caching import Cache
 
-from flask import Flask, request, session
-
 from celery import Celery, Task
 
+from flask_oidc import OpenIDConnect
+
+from flask import Flask, request, session
+
 def celery_init_app(app: Flask) -> Celery:
     class FlaskTask(Task):
         def __call__(self, *args: object, **kwargs: object) -> object:
@@ -46,3 +48,4 @@ limiter = Limiter(
     
 caching = Cache()
 
+oidc = OpenIDConnect()

ayta/nosql.py

@@ -51,12 +51,13 @@ class Mango:
         self.db = self.client['ayta']
         self.channels = self.db['channels']
         self.info_json = self.db['info_json']
-        self.download_queue = self.db['download_queue']
+        self.posters_queue = self.db['posters_queue']
         self.run_log = self.db['run_log']
         self.channel_log = self.db['channel_log']
         self.websub_callbacks = self.db['websub_callbacks']
         self.websub_data = self.db['websub_data']
         self.reports = self.db['reports']
+        self.posters_endpoints = self.db['posters_endpoints']
             
         self.ensure_indexes()
         #self.clean_info_json()
@@ -112,14 +113,10 @@ class Mango:
         
         stats['videos'] = self.info_json.count_documents({})
         stats['channels'] = self.channels.count_documents({})
-        stats['queue'] = self.download_queue.count_documents({})
+        stats['queue'] = self.posters_queue.count_documents({})
         
         return stats
             
-    def insert_download_queue(self, video):
-        if not self.download_queue.count_documents({'id': video}) >= 1:
-            return self.download_queue.insert_one({'id': video}).inserted_id
-            
     def search_videos(self, query):
         # search the index for the requested query. return limited keys
         results = self.info_json.find({"$text": {"$search": query}}, 
@@ -200,9 +197,20 @@ class Mango:
         
         return tuple(ids)
     
+    def get_recent_videos(self, count=99):
+        """ Returns a SET of YouTube video ID's which have been added last to the info_json collection """
+        result = self.info_json.find({}, {'_id': 0, 'id': 1}, sort=[('_id', pymongo.DESCENDING)]).limit(count)
+
+        ids = []
+        
+        for id in result:
+            ids.append(id['id'])
+
+        return tuple(ids)
+        
     def get_video_info(self, videoId, limited=False):
         if limited:
-            projection = {'_id': 1, 'id': 1, 'title': 1, 'upload_date': 1, 'description': 1, 'channel_id': 1}
+            projection = {'_id': 1, 'id': 1, 'title': 1, 'upload_date': 1, 'description': 1, 'channel_id': 1, 'uploader': 1, 'epoch': 1, 'title_slug': 1}
         else:
             projection = {}
         
@@ -262,7 +270,7 @@ class Mango:
     ##########################################
     
     def websub_newCallback(self, channelId):
-        callbackId = secrets.token_hex(8)
+        callbackId = secrets.token_urlsafe(16)
         
         self.websub_callbacks.insert_one({'id': callbackId, 'channel': channelId, 'status': 'new', 'created_time': current_time(object=True)})
         
@@ -336,10 +344,60 @@ class Mango:
     def websub_cleanRetired(self, days=3):
         days = self.datetime.utcnow() - self.timedelta(days=days)
 
-        self.websub_callbacks.delete_many({'status': 'retired', 'retiring_time': {'$lt': days}})
+        self.websub_callbacks.delete_many({'status': 'retired', 'retired_time': {'$lt': days}})
         
         return True
 
+    ##########################################
+    #           POSTER FUNCTIONS             #
+    ##########################################
+    
+    def poster_newEndpoint(self, endpointId, description=''):
+        self.posters_endpoints.insert_one({'id': endpointId, 'description': description, 'status': 'active', 'created_time': current_time(object=True)})
+        return endpointId
+        
+    def poster_insertQueue(self, endpointId, videoId):
+        # if no document exists
+        if not self.posters_queue.count_documents({'id': videoId}) >= 1:
+            self.posters_queue.insert_one({'id': videoId, 'endpoint': endpointId, 'created_time': current_time(object=True), 'status': 'queued'}).inserted_id
+            return True
+            
+        # key already in queue
+        return False
+        
+    def poster_deleteQueue(self, videoId):
+        if self.posters_queue.delete_one({'id': videoId}):
+            return True
+        return False
+
+    def poster_retireEndpoint(self, endpointId):
+        return self.posters_endpoints.update_one({'id': endpointId}, {'$set': {'status': 'retired', 'retired_time': current_time(object=True)}})
+        
+    def poster_isActive(self, endpointId):
+        status = self.posters_endpoints.find_one({'id': endpointId}, {'status': 1})
+
+        if not status:
+            return False
+            
+        status = status.get('status')
+        
+        if status == 'active':
+            return True
+            
+        return False
+        
+    def poster_getEndpoints(self):
+        return self.posters_endpoints.find({})
+        
+    def poster_getQueue(self):
+        return self.posters_queue.find({})
+        
+    def poster_cleanRetired(self, days=3):
+        days = self.datetime.utcnow() - self.timedelta(days=days)
+
+        self.posters_endpoints.delete_many({'status': 'retired', 'retired_time': {'$lt': days}})
+        
+        return True
 
 ##########################################
 #           HELPER FUNCTIONS             #

ayta/tasks.py

@@ -10,7 +10,7 @@ def subscribe_websub_callback(channelId):
     
     url = 'https://pubsubhubbub.appspot.com/subscribe'
     data = {
-        'hub.callback': f'https://{current_app.config["DOMAIN"]}/websub/c/{callbackId}',
+        'hub.callback': f'https://{current_app.config["DOMAIN"]}/api/websub//{callbackId}',
         'hub.topic': f'https://www.youtube.com/xml/feeds/videos.xml?channel_id={channelId}',
         'hub.verify': 'async',
         'hub.mode': 'subscribe',
@@ -32,7 +32,7 @@ def unsubscribe_websub_callback(callbackId, channelId):
     from .nosql import get_nosql
     
     url = 'https://pubsubhubbub.appspot.com/subscribe'
-    data = {'hub.callback': f'https://{current_app.config["DOMAIN"]}/websub/c/{callbackId}',
+    data = {'hub.callback': f'https://{current_app.config["DOMAIN"]}/api/websub/{callbackId}',
             'hub.topic': f'https://www.youtube.com/xml/feeds/videos.xml?channel_id={channelId}',
             'hub.verify': 'async',
             'hub.mode': 'unsubscribe'

ayta/templates/admin/index.html

@@ -65,5 +65,15 @@
       </div>
 	</a>
   </div>
+  <div class="col s6 l4 m-4">
+	<a href="{{ url_for('admin.posters') }}">
+	  <div class="card black-text">
+        <div class="card-content">
+          <span class="card-title">Posters</span>
+		  <p class="grey-text">User extension posters</p>
+        </div>
+      </div>
+	</a>
+  </div>
 </div>
 {% endblock %}

ayta/templates/admin/posters.html

@@ -0,0 +1,150 @@
+{% extends 'material_base.html' %}
+{% block title %}Posters administration page{% endblock %}
+{% block description %}Posters administration page of the AYTA system{% endblock %}
+
+{% block content %}
+<div class="row">
+  <div class="col s12 l11">
+    <h4>Posters administration page</h4>
+  </div>
+  <div class="col s12 l1 m-5">
+    <form method="POST">
+      <input title="Prunes all deleted endpoints, but keeps last 3 days" type="submit" value="clean-retired" name="task">
+    </form>
+  </div>
+</div>
+<div class="divider"></div>
+<div class="row">
+  <div class="col s12">
+	<h5>Poster options</h5>
+  </div>
+</div>
+<div class="row">
+  <div class="col s12 l4 m-4">
+	<div class="card">
+      <div class="card-content">
+        <span class="card-title">Create new endpoint</span>
+	    <form method="post">
+		  <div class="row">
+            <div class="col s12 input-field">
+              <input placeholder="Custom endpoint" name="value" type="text" class="validate" minlength="12">
+			  <span class="supporting-text">Leaving this empty will create a random secure string</span>
+	        </div>
+			<div class="col s12 input-field">
+              <input placeholder="Description" name="description" type="text" class="validate" minlength="8" maxlength="64" required>
+			  <span class="supporting-text">Description for the endpoint for better administration</span>
+	        </div>
+          <button class="btn mt-4" type="submit" name="task" value="add-endpoint">Create</button>
+		  </div>
+	    </form>
+      </div>
+    </div>
+  </div>
+  <div class="col s12 l4 m-4">
+	<div class="card">
+      <div class="card-content">
+        <span class="card-title">Queue manually</span>
+	    <form method="post">
+		  <div class="row">
+            <div class="col s12 input-field">
+              <input placeholder="Youtube video ID" name="value" type="text" class="validate" minlength="11" maxlength="11" required>
+			  <span class="supporting-text">Must be a valid Youtube video ID</span>
+	        </div>
+			<div class="col s12 mt-5 input-field">
+              <div class="switch">
+		        <label>Queue<input type="checkbox" value="direct" name="value" disabled><span class="lever"></span>Direct</label>
+				<span class="supporting-text">Queue up or start directly</span>
+		      </div>
+	        </div>
+          <button class="btn mt-4" type="submit" name="task" value="manual-queue">Queue</button>
+		  </div>
+	    </form>
+      </div>
+    </div>
+  </div>
+</div>
+<div class="divider"></div>
+<div class="row">
+  <div class="col s6 l9">
+	<h5>Registered endpoints</h5>
+  </div>
+  <div class="col s6 l3 m-4 input-field">
+	<input id="filter_query" type="text">
+    <label for="filter_query">Filter results</label>
+  </div>
+</div>
+<div class="row">
+  <div class="col s12">
+    <table class="striped highlight responsive-table">
+      <thead>
+        <tr>
+		  <th>Actions</th>
+		  <th>id</th>
+		  <th>description</th>
+		  <th>status</th>
+		  <th>created_time</th>
+		  <th>retired_time</th>
+        </tr>
+      </thead>
+      <tbody>
+	    {% for endpoint in endpoints %}
+        <tr class="filterable">
+		  <td>
+		    <form method="post">
+			  <input type="text" value="{{ endpoint.get('id') }}" name="value" hidden>
+		      <button class="btn-small waves-effect waves-light" type="submit" name="task" value="retire" title="Retire endpoint" {% if endpoint.get('status') != 'active' %}disabled{% endif %}>🗑️</button>
+			</form>
+		  </td>
+		  <td>{{ endpoint.get('id') }}</td>
+		  <td>{{ endpoint.get('description') }}</td>
+		  <td>{{ endpoint.get('status') }}</td>
+		  <td>{{ endpoint.get('created_time') }}</td>
+		  <td>{{ endpoint.get('retired_time') }}</td>
+        </tr>
+		{% endfor %}
+      </tbody>
+    </table>
+  </div>
+</div>
+<div class="divider"></div>
+<div class="row">
+  <div class="col s6 l9">
+	<h5>Queued ID's</h5>
+  </div>
+  <div class="col s6 l3 m-4 input-field">
+	<input id="filter_query" type="text">
+    <label for="filter_query">Filter results</label>
+  </div>
+</div>
+<div class="row">
+  <div class="col s12">
+    <table class="striped highlight responsive-table">
+      <thead>
+        <tr>
+		  <th>Actions</th>
+		  <th>id</th>
+		  <th>endpoint</th>
+		  <th>status</th>
+		  <th>created_time</th>
+        </tr>
+      </thead>
+      <tbody>
+	    {% for id in queue %}
+        <tr class="filterable">
+		  <td>
+		    <form method="post">
+			  <input type="text" value="{{ id.get('id') }}" name="value" hidden>
+		      <button class="btn-small waves-effect waves-light" type="submit" name="task" value="delete-queue" title="Delete from queue" {% if id.get('status') != 'queued' %}disabled{% endif %}>🗑️</button>
+			</form>
+		  </td>
+		  <td>{{ id.get('id') }}</td>
+		  <td>{{ id.get('endpoint') }}</td>
+		  <td>{{ id.get('status') }}</td>
+		  <td>{{ id.get('created_time') }}</td>
+        </tr>
+		{% endfor %}
+      </tbody>
+    </table>
+  </div>
+</div>
+{% endblock %}

ayta/templates/channel/channel.html

@@ -25,7 +25,7 @@
 	<div class="card medium black-text">
 	  <a href="{{ url_for('watch.base') }}?v={{ video.get('id') }}">
 	    <div class="card-image">
-		  <img loading="lazy" src="https://archive.ventilaar.net/videos/automatic/{{ video.get('channel_id') }}/{{ video.get('id') }}/{{ video.get('title') }}.jpg">
+		  <img loading="lazy" src="https://archive.ventilaar.net/videos/automatic/{{ video.get('channel_id') }}/{{ video.get('id') }}/{{ video.get('title_slug') }}.jpg">
         </div>
 	  </a>
       <div class="card-content activator">

ayta/templates/channel/index.html

@@ -19,7 +19,17 @@
   </div>
 </div>
 <div class="row">
-  <div class="col s12 m-4 filterable">
+  <div class="col s6 m-4 filterable">
+	<a href="{{ url_for('channel.recent') }}">
+	  <div class="card black-text">
+        <div class="card-content center">
+          <span class="card-title">Recent videos</span>
+		  <p class="grey-text">The last videos to have been added to the archive</p>
+        </div>
+      </div>
+	</a>
+  </div>
+  <div class="col s6 m-4 filterable">
 	<a href="{{ url_for('channel.orphaned') }}">
 	  <div class="card black-text">
         <div class="card-content center">
@@ -31,12 +41,12 @@
   </div>
   {% for channel in channels %}
   <div class="col s6 l4 m-4 filterable">
-	<a href="{{ url_for('channel.channel', channelId=channel) }}">
+	<a href="{{ url_for('channel.channel', channelId=channel.get('id')) }}">
 	  <div class="card black-text">
         <div class="card-content">
-          <span class="card-title">{{ channels[channel].get('original_name') }}</span>
+          <span class="card-title">{{ channel.get('original_name') }}</span>
-		  <p class="grey-text">{{ channels[channel].get('id') }}</p>
+		  <p class="grey-text">{{ channel.get('id') }}</p>
-          <p><b>Added:</b> {{ channels[channel].get('added_date')|pretty_time }} | <b>Active:</b> {{ channels[channel].get('active') }} | <b>Videos:</b> {{ channels[channel].get('video_count') }}</p>
+          <p><b>Added:</b> {{ channel.get('added_date')|pretty_time }} | <b>Active:</b> {{ channel.get('active') }} | <b>Videos:</b> {{ channel.get('video_count') }}</p>
         </div>
       </div>
 	</a>

ayta/templates/channel/orphaned.html

@@ -5,14 +5,14 @@
 {% block content %}
 <div class="row">
   <div class="col s12">
-    <h4>Channels lising page</h4>
+    <h4>Videos lising page</h4>
   </div>
 </div>
 <div class="divider"></div>
 <div class="row">
   <div class="col s6 l9">
 	<h5>Orphaned videos</h5>
-	<p>Videos in the archive which do not have a permanent channel linked. There is a high chance that the videos are manually downloaded.</p>
+	<p>Videos in the archive which do not have a permanent channel linked. There is a high chance that the videos are manually downloaded. Sorted by last added.</p>
   </div>
   <div class="col s6 l3 m-4 input-field">
 	<input id="filter_query" type="text">
@@ -23,18 +23,19 @@
   {% for video in videos %}
   <div class="col s6 l4 m-4 filterable">
 	<div class="card medium black-text">
-	  <a href="{{ url_for('watch.base') }}?v={{ video }}">
+	  <a href="{{ url_for('watch.base') }}?v={{ video.get('id') }}">
 	    <div class="card-image">
-		  <img loading="lazy" src="https://archive.ventilaar.net/videos/automatic/{{ videos[video].get('channel_id') }}/{{ videos[video].get('id') }}/{{ videos[video].get('title') }}.jpg">
+		  <img loading="lazy" src="https://archive.ventilaar.net/videos/automatic/{{ video.get('channel_id') }}/{{ video.get('id') }}/{{ video.get('title_slug') }}.jpg">
         </div>
 	  </a>
       <div class="card-content activator">
-        <span class="card-title">{{ videos[video].get('title') }}</span>
+        <span class="card-title">{{ video.get('title') }}</span>
-		<p class="grey-text">{{ videos[video].get('id') }} | {{ videos[video].get('upload_date')|pretty_time }}</p>
+		<p><b>{{ video.get('uploader') }}</b></p>
+		<p class="grey-text">{{ video.get('id') }} | {{ video.get('upload_date')|pretty_time }}</p>
       </div>
       <div class="card-reveal">
-        <span class="card-title truncate">{{ videos[video].get('title') }}</span>
+        <span class="card-title truncate">{{ video.get('title') }}</span>
-        <p style="white-space: pre-wrap;">{{ videos[video].get('description') }}</p>
+        <p style="white-space: pre-wrap;">{{ video.get('description') }}</p>
       </div>
 	</div>
   </div>

ayta/templates/channel/recent.html

@@ -0,0 +1,44 @@
+{% extends 'material_base.html' %}
+{% block title %}Recent videos{% endblock %}
+{% block description %}The last videos to have been added to the archive{% endblock %}
+
+{% block content %}
+<div class="row">
+  <div class="col s12">
+    <h4>Videos lising page</h4>
+  </div>
+</div>
+<div class="divider"></div>
+<div class="row">
+  <div class="col s6 l9">
+	<h5>Recent videos</h5>
+	<p>The last 99 videos to have been added to the archive.</p>
+  </div>
+  <div class="col s6 l3 m-4 input-field">
+	<input id="filter_query" type="text">
+    <label for="filter_query">Filter results</label>
+  </div>
+</div>
+<div class="row">
+  {% for video in videos %}
+  <div class="col s6 l4 m-4 filterable">
+	<div class="card medium black-text">
+	  <a href="{{ url_for('watch.base') }}?v={{ video.get('id') }}">
+	    <div class="card-image">
+		  <img loading="lazy" src="https://archive.ventilaar.net/videos/automatic/{{ video.get('channel_id') }}/{{ video.get('id') }}/{{ video.get('title_slug') }}.jpg">
+        </div>
+	  </a>
+      <div class="card-content activator">
+        <span class="card-title">{{ video.get('title') }}</span>
+		<p><b>{{ video.get('uploader') }}</b></p>
+		<p class="grey-text">{{ video.get('id') }} | {{ video.get('upload_date')|pretty_time }}</p>
+      </div>
+      <div class="card-reveal">
+        <span class="card-title truncate">{{ video.get('title') }}</span>
+        <p style="white-space: pre-wrap;">{{ video.get('description') }}</p>
+      </div>
+	</div>
+  </div>
+  {% endfor %}
+</div>
+{% endblock %}

ayta/templates/watch/index.html

@@ -31,8 +31,8 @@
 <div class="row">
   <div class="col s12 center-align">
     <video controls class="responsive-video">
-    <source src="https://archive.ventilaar.net/videos/automatic/{{ render.get('info').get('channel_id') }}/{{ render.get('info').get('id') }}/{{ render.get('info').get('title') }}.mp4">
+    <source src="https://archive.ventilaar.net/videos/automatic/{{ render.get('info').get('channel_id') }}/{{ render.get('info').get('id') }}/{{ render.get('info').get('title_slug') }}.mp4">
-	<source src="https://archive.ventilaar.net/videos/automatic/{{ render.get('info').get('channel_id') }}/{{ render.get('info').get('id') }}/{{ render.get('info').get('title') }}.webm">
+	<source src="https://archive.ventilaar.net/videos/automatic/{{ render.get('info').get('channel_id') }}/{{ render.get('info').get('id') }}/{{ render.get('info').get('title_slug') }}.webm">
     Your browser does not support the video tag.
     </video>
   </div>