Compare commits

..

4 Commits

Author SHA1 Message Date
Ventilaar
46bde82d32 Hotfix shared state issue
All checks were successful
Update worker server / build-and-publish (release) Successful in 12s
Generate docker image / build-and-publish (release) Successful in 19s
2024-12-07 14:58:52 +01:00
Ventilaar
6c681d6b07 Uhhh
All checks were successful
Update worker server / build-and-publish (release) Successful in 9s
Generate docker image / build-and-publish (release) Successful in 49s
2024-12-05 22:20:55 +01:00
Ventilaar
0d5d233e90 Cleanup and documentation
All checks were successful
Generate docker image / build-and-publish (release) Successful in 19s
Update worker server / build-and-publish (release) Successful in 20s
2024-12-05 22:15:42 +01:00
Ventilaar
548a4860fc it was google!
All checks were successful
Generate docker image / build-and-publish (release) Successful in 55s
Update worker server / build-and-publish (release) Successful in 10s
2024-10-15 16:23:43 +02:00
5 changed files with 40 additions and 13 deletions

View File

@@ -1,7 +1,7 @@
FROM python:3-alpine FROM python:3.12-alpine
WORKDIR /app WORKDIR /app
COPY requirements.txt /app COPY requirements.txt /app
RUN pip install --no-cache-dir -r requirements.txt RUN pip install --no-cache-dir -r requirements.txt
COPY . /app COPY . /app
EXPOSE 8000 EXPOSE 8000
CMD ["gunicorn", "--bind", "0.0.0.0:8000", "ayta:create_app()"] CMD ["gunicorn", "--bind", "0.0.0.0:8000", "--workers", "1", "ayta:create_app()"]

View File

@@ -6,7 +6,7 @@ current cronjob yt-dlp archive service.
Partially inspired by [hobune](https://github.com/rebane2001/hobune). While that project is amazingby it's own, it's just not scaleable. Partially inspired by [hobune](https://github.com/rebane2001/hobune). While that project is amazingby it's own, it's just not scaleable.
## The idea ## The idea
Having over 250k videos, scaling the current cronjob yt-dlp archive task is just really hard. Filetypes change, things get partially downloaded and such. Having over 350k videos, scaling the current cronjob yt-dlp archive task is just really hard. Filetypes change, things get partially downloaded and such.
Partially yt-dlp is to blame because it's a package that needs to change all the time. But with this some changes are not accounted for. Partially yt-dlp is to blame because it's a package that needs to change all the time. But with this some changes are not accounted for.
yt-dlp will still do the downloads. But a flask frontend will be developed to make all downloaded videos easily indexable. yt-dlp will still do the downloads. But a flask frontend will be developed to make all downloaded videos easily indexable.
For it to be quick (unlike hobune) a database has to be implemented. This could get solved by a static site generator type of software, but that is not my choice. For it to be quick (unlike hobune) a database has to be implemented. This could get solved by a static site generator type of software, but that is not my choice.
@@ -54,7 +54,7 @@ Extra functionality for further development of features.
Mainly focused on retiring the cronjob based scripts and moving it to celery based tasks Mainly focused on retiring the cronjob based scripts and moving it to celery based tasks
- [ ] manage videos by ID's instead of per channel basis - [ ] manage videos by ID's instead of per channel basis
- [ ] download videos from queue - [ ] download videos from queue
- [ ] Manage websub callbacks - [x] Manage websub callbacks
### Stage 4 ### Stage 4
Mongodb finally has it's limitations. Mongodb finally has it's limitations.

View File

@@ -190,14 +190,16 @@ def queue():
flash(f'Cleaned retired endpoints') flash(f'Cleaned retired endpoints')
elif task == 'manual-queue': elif task == 'manual-queue':
if not get_nosql().check_exists(value):
direct = request.form.get('direct', None) direct = request.form.get('direct', None)
if direct: if direct:
task = video_download.delay(value) task = video_download.delay(value)
flash(f"Started task {task.id}") flash(f"Started task {task.id}")
else: else:
get_nosql().queue_insertQueue(value, 'webui') get_nosql().queue_insertQueue(value, 'webui')
flash(f'Added to queue: {value}') flash(f'Added to queue: {value}')
else:
flash(f'This video ID already exists in the archive: {value}')
elif task == 'delete-queue': elif task == 'delete-queue':
get_nosql().queue_deleteQueue(value) get_nosql().queue_deleteQueue(value)

View File

@@ -25,12 +25,24 @@
</li> </li>
<li class="collection-item"> <li class="collection-item">
<a href="{{ url_for('channel.channel', channelId='UCIcgBZ9hEJxHv6r_jDYOMqg') }}"><span class="title">Unus Annus</span></a> <a href="{{ url_for('channel.channel', channelId='UCIcgBZ9hEJxHv6r_jDYOMqg') }}"><span class="title">Unus Annus</span></a>
<p>Reason: This channel does not exist. (Self removed)</p> <p>Reason: This channel does not exist.</p>
</li> </li>
<li class="collection-item"> <li class="collection-item">
<a href="{{ url_for('channel.channel', channelId='UCz1s8aJYSQuaXJCtEi-VWRA') }}"><span class="title">Dutch Legion</span></a> <a href="{{ url_for('channel.channel', channelId='UCz1s8aJYSQuaXJCtEi-VWRA') }}"><span class="title">Dutch Legion</span></a>
<p>Reason: This account has been terminated due to multiple or severe violations of YouTube's policy prohibiting hate speech.</p> <p>Reason: This account has been terminated due to multiple or severe violations of YouTube's policy prohibiting hate speech.</p>
</li> </li>
<li class="collection-item">
<a href="{{ url_for('channel.channel', channelId='UC91-8aNaRbp71UMEb_34ryg') }}"><span class="title">RBMK5000</span></a>
<p>Reason: This channel does not exist.</p>
</li>
<li class="collection-item">
<a href="{{ url_for('channel.channel', channelId='UCoPSAT64vfXlulyWd_dPE3Q') }}"><span class="title">Evilfisher2</span></a>
<p>Reason: This channel was removed because it violated our Community Guidelines.</p>
</li>
<li class="collection-item">
<a href="{{ url_for('channel.channel', channelId='UCZXkvavD2YKnFCzCkZ-bNPw') }}"><span class="title">mrabhy</span></a>
<p>Reason: This channel was removed because it violated our Community Guidelines.</p>
</li>
</ul> </ul>
</div> </div>
<div class="col s12 l6 center-align"> <div class="col s12 l6 center-align">
@@ -45,6 +57,18 @@
</li> </li>
<li class="collection-item"> <li class="collection-item">
<a href="{{ url_for('channel.channel', channelId='UCtfg1tENiu3SgGMZVduFmTg') }}"><span class="title">FiberNinja</span></a> <a href="{{ url_for('channel.channel', channelId='UCtfg1tENiu3SgGMZVduFmTg') }}"><span class="title">FiberNinja</span></a>
<p>Reason: This channel was removed because it violated our Community Guidelines.</p>
</li>
<li class="collection-item">
<a href="{{ url_for('channel.channel', channelId='UCv4VkfbX8YfqodF-4coEEfQ') }}"><span class="title">James Somerton</span></a>
<p>Reason: This channel does not exist.</p>
</li>
<li class="collection-item">
<a href="{{ url_for('channel.channel', channelId='UC8XH9kpilkuss4bVeRZD1kw') }}"><span class="title">Plagued Moth</span></a>
<p>Reason: This channel was removed because it violated our Community Guidelines.</p>
</li>
<li class="collection-item">
<a href="{{ url_for('channel.channel', channelId='UCxZTTWP0QN7-ch2wW1QeFwg') }}"><span class="title">CowOfTheSea</span></a>
<p>Reason: This channel was removed because it violated our Community Guidelines.</p> <p>Reason: This channel was removed because it violated our Community Guidelines.</p>
</li> </li>
</ul> </ul>

View File

@@ -8,4 +8,5 @@ yt-dlp
gunicorn gunicorn
celery celery
sqlalchemy sqlalchemy
requests
pyjwt[crypto] pyjwt[crypto]