bump TA_VERSION

Changed:
- Bump yt-dlp to 2025.01.15
- Fix comment extraction

.github/ISSUE_TEMPLATE/FEATURE-REQUEST.yml

@@ -1,34 +1,12 @@
 name: Feature Request
-description: I have an idea for a great addition to this project
+description: This Project currently doesn't take any new feature requests.
 title: "[Feature Request]: "
 
 body:
-  - type: markdown
-    attributes:
-      value: |
-        Thanks for taking the time to help improve this project! This project is *very* selective with accepting new feature requests. Please read the [how to open an issue](https://github.com/tubearchivist/tubearchivist/blob/master/CONTRIBUTING.md#how-to-open-an-issue) guide carefully before continuing.
-
   - type: checkboxes
-    id: already
+    id: block
     attributes:
-      label: "I've read the documentation"
+      label: "This project doesn't accept any new feature requests for the forseeable future. There is no shortage of ideas and the next development steps are clear for years to come."
       options:
-        - label: I have read the [how to open an issue](https://github.com/tubearchivist/tubearchivist/blob/master/CONTRIBUTING.md#how-to-open-an-issue) guide, particularly the [feature request](https://github.com/tubearchivist/tubearchivist/blob/master/CONTRIBUTING.md#feature-request) section.
+        - label: I understand that this issue will be closed without comment.
           required: true
-
-  - type: textarea
-    id: description
-    attributes:
-      label: Your Feature Request
-      value: "## Is your feature request related to a problem? Please describe.\n\n## Describe the solution you'd like\n\n## Additional context"
-      placeholder: Tell us what you see!
-    validations:
-      required: true
-
-  - type: checkboxes
-    id: help
-    attributes:
-      label: Your help is needed!
-      description: This project is ambitious as it is, please contribute.
-      options:
-        - label: Yes I will work on this in the next few days or weeks.

CONTRIBUTING.md

@@ -62,22 +62,7 @@ Please keep in mind:
 - A bug that can't be reproduced, is difficult or sometimes even impossible to fix. Provide very clear steps *how to reproduce*.
 
 ### Feature Request
-This project needs your help to grow further. There is no shortage of ideas, see the open [issues on GH](https://github.com/tubearchivist/tubearchivist/issues?q=is%3Aopen+is%3Aissue+label%3Aenhancement) and the [roadmap](https://github.com/tubearchivist/tubearchivist#roadmap), what this project lacks is contributors interested in helping with overall improvements of the application. Focus is *not* on adding new features, but improving existing ones.
-
-Existing ideas are easily *multiple years* worth of development effort, at least at current speed. This project is *very* selective with accepting new feature requests at this point.  
-
-Good feature requests usually fall into one or more of these categories:
-- You want to work on your own small scoped idea within the next few days or weeks.
-- Your idea is beneficial for a wide range of users, not just for you.
-- Your idea extends the current project by building on and improving existing functionality.
-- Your idea is quick and easy to implement, for an experienced as well as for a first time contributor.
-
-Your request is likely going to be rejected if:
-- Your idea requires multiple days worth of development time and is unrealistic to be implemented any time soon.
-- There are already other ways to do what you are trying to do.
-- You are trying to do something that only applies to your platform, your specific workflow or your specific setup.
-- Your idea would fundamentally change how the project works or it wouldn't be able to be implemented with backwards compatibility.
-- Your idea is not a good fit for this project.
+This project doesn't take any new feature requests. This project doesn't lack ideas, see the currently open tasks and roadmap. New feature requests aren't helpful at this point in time. Thank you for your understanding.
 
 ### Installation Help
 GitHub is most likely not the best place to ask for installation help. That's inherently individual and one on one.

Dockerfile

@@ -4,7 +4,6 @@
 
 # First stage to build python wheel
 FROM python:3.11.8-slim-bookworm AS builder
-ARG TARGETPLATFORM
 
 RUN apt-get update && apt-get install -y --no-install-recommends \
     build-essential gcc libldap2-dev libsasl2-dev libssl-dev git
@@ -15,13 +14,15 @@ RUN pip install --user -r requirements.txt
 
 # build ffmpeg
 FROM python:3.11.8-slim-bookworm as ffmpeg-builder
+
+ARG TARGETPLATFORM
+
 COPY docker_assets/ffmpeg_download.py ffmpeg_download.py
 RUN python ffmpeg_download.py $TARGETPLATFORM
 
 # build final image
 FROM python:3.11.8-slim-bookworm as tubearchivist
 
-ARG TARGETPLATFORM
 ARG INSTALL_DEBUG
 
 ENV PYTHONUNBUFFERED 1

README.md

@@ -25,7 +25,7 @@
 ## Core functionality
 Once your YouTube video collection grows, it becomes hard to search and find a specific video. That's where Tube Archivist comes in: By indexing your video collection with metadata from YouTube, you can organize, search and enjoy your archived YouTube videos without hassle offline through a convenient web interface. This includes:  
 * Subscribe to your favorite YouTube channels
-* Download Videos using **yt-dlp**
+* Download Videos using **[yt-dlp](https://github.com/yt-dlp/yt-dlp)**
 * Index and make videos searchable
 * Play videos
 * Keep track of viewed and unviewed videos
@@ -41,7 +41,7 @@ Once your YouTube video collection grows, it becomes hard to search and find a s
 For minimal system requirements, the Tube Archivist stack needs around 2GB of available memory for a small testing setup and around 4GB of available memory for a mid to large sized installation. Minimal with dual core with 4 threads, better quad core plus. 
 This project requires docker. Ensure it is installed and running on your system.  
 
-The documentation has additional user provided instructions for [Unraid](https://docs.tubearchivist.com/installation/unraid/), [Synology](https://docs.tubearchivist.com/installation/synology/), [Podman](https://docs.tubearchivist.com/installation/podman/) and [True NAS](https://docs.tubearchivist.com/installation/truenas-scale/).
+The documentation has additional user provided instructions for [Unraid](https://docs.tubearchivist.com/installation/unraid/), [Synology](https://docs.tubearchivist.com/installation/synology/) and [Podman](https://docs.tubearchivist.com/installation/podman/).
 
 The instructions here should get you up and running quickly, for Docker beginners and full explanation about each environment variable, see the [docs](https://docs.tubearchivist.com/installation/docker-compose/).
 
@@ -148,7 +148,6 @@ This can happen if you have nested virtualizations, e.g. LXC running Docker in P
 ## Roadmap
 We have come far, nonetheless we are not short of ideas on how to improve and extend this project. Issues waiting for you to be tackled in no particular order:
 
-- [ ] User roles
 - [ ] Audio download
 - [ ] Podcast mode to serve channel as mp3
 - [ ] Random and repeat controls ([#108](https://github.com/tubearchivist/tubearchivist/issues/108), [#220](https://github.com/tubearchivist/tubearchivist/issues/220))
@@ -159,10 +158,12 @@ We have come far, nonetheless we are not short of ideas on how to improve and ex
 - [ ] Custom searchable notes to videos, channels, playlists ([#144](https://github.com/tubearchivist/tubearchivist/issues/144))
 - [ ] Search comments
 - [ ] Search download queue
-- [ ] Configure shorts, streams and video sizes per channel
+- [ ] Per user videos/channel/playlists
 
 Implemented:
+- [X] Configure shorts, streams and video sizes per channel [2024-07-15]
 - [X] User created playlists [2024-04-10]
+- [X] User roles, aka read only user [2023-11-10]
 - [X] Add statistics of index [2023-09-03]
 - [X] Implement [Apprise](https://github.com/caronc/apprise) for notifications [2023-08-05]
 - [X] Download video comments [2022-11-30]
@@ -200,6 +201,7 @@ This is your time to shine, [read this](https://github.com/tubearchivist/tubearc
 - [lamusmaser/create_info_json](https://gist.github.com/lamusmaser/837fb58f73ea0cad784a33497932e0dd): Script to generate `.info.json` files using `ffmpeg` collecting information from downloaded videos.
 - [lamusmaser/ta_fix_for_video_redirection](https://github.com/lamusmaser/ta_fix_for_video_redirection): Script to fix videos that were incorrectly indexed by YouTube's "Video is Unavailable" response. 
 - [RoninTech/ta-helper](https://github.com/RoninTech/ta-helper): Helper script to provide a symlink association to reference TubeArchivist videos with their original titles.
+- [tangyjoust/Tautulli-Notify-TubeArchivist-of-Plex-Watched-State](https://github.com/tangyjoust/Tautulli-Notify-TubeArchivist-of-Plex-Watched-State) Mark videos watched in Plex (through streaming not manually) through Tautulli back to TubeArchivist
 
 ## Donate
 The best donation to **Tube Archivist** is your time, take a look at the [contribution page](CONTRIBUTING.md) to get started.  
@@ -211,6 +213,8 @@ Second best way to support the development is to provide for caffeinated beverag
 
 ## Notable mentions
 This is a selection of places where this project has been featured on reddit, in the news, blogs or any other online media, newest on top.
+* **xda-developers.com**: 5 obscure self-hosted services worth checking out - Tube Archivist - To save your essential YouTube videos, [2024-10-13][[link](https://www.xda-developers.com/obscure-self-hosted-services/)]
+* **selfhosted.show**: why we're trying Tube Archivist, [2024-06-14][[link](https://selfhosted.show/125)]
 * **ycombinator**: Tube Archivist on Hackernews front page, [2023-07-16][[link](https://news.ycombinator.com/item?id=36744395)]
 * **linux-community.de**: Tube Archivist bringt Ordnung in die Youtube-Sammlung, [German][2023-05-01][[link](https://www.linux-community.de/ausgaben/linuxuser/2023/05/tube-archivist-bringt-ordnung-in-die-youtube-sammlung/)]
 * **noted.lol**: Dev Debrief, An Interview With the Developer of Tube Archivist, [2023-03-30] [[link](https://noted.lol/dev-debrief-tube-archivist/)]
@@ -222,13 +226,3 @@ This is a selection of places where this project has been featured on reddit, in
 * **reddit.com**: Celebrating TubeArchivist v0.1, [2022-01-09] [[link](https://www.reddit.com/r/selfhosted/comments/rzh084/celebrating_tubearchivist_v01/)]
 * **linuxunplugged.com**: Pick: tubearchivist — Your self-hosted YouTube media server, [2021-09-11] [[link](https://linuxunplugged.com/425)] and [2021-10-05] [[link](https://linuxunplugged.com/426)]
 * **reddit.com**: Introducing Tube Archivist, your self hosted Youtube media server, [2021-09-12] [[link](https://www.reddit.com/r/selfhosted/comments/pmj07b/introducing_tube_archivist_your_self_hosted/)]
-
-
-## Sponsor
-Big thank you to [Digitalocean](https://www.digitalocean.com/) for generously donating credit for the tubearchivist.com VPS and buildserver. 
-<p>
-  <a href="https://www.digitalocean.com/">
-    <img src="https://opensource.nyc3.cdn.digitaloceanspaces.com/attribution/assets/PoweredByDO/DO_Powered_by_Badge_blue.svg" width="201px">
-  </a>
-</p>
-

deploy.sh

@@ -157,7 +157,7 @@ function sync_docker {
     fi
 
     echo "latest tags:"
-    git tag | tail -n 5 | sort -r
+    git tag | sort -rV | head -n 5
 
     printf "\ncreate new version:\n"
     read -r VERSION
@@ -189,7 +189,7 @@ function sync_docker_old {
     fi
 
     echo "latest tags:"
-    git tag | tail -n 5 | sort -r
+    git tag | sort -rV | head -n 5
 
     printf "\ncreate new version:\n"
     read -r VERSION

docker-compose.yml

@@ -40,12 +40,12 @@ services:
     depends_on:
       - archivist-es
   archivist-es:
-    image: bbilly1/tubearchivist-es         # only for amd64, or use official es 8.13.2
+    image: bbilly1/tubearchivist-es         # only for amd64, or use official es 8.16.0
     container_name: archivist-es
     restart: unless-stopped
     environment:
       - "ELASTIC_PASSWORD=verysecret"       # matching Elasticsearch password
-      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
+      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
       - "xpack.security.enabled=true"
       - "discovery.type=single-node"
       - "path.repo=/usr/share/elasticsearch/data/snapshot"

tubearchivist/api/views.py

@@ -361,7 +361,7 @@ class ChannelApiListView(ApiBaseView):
         must_list = []
         if query_filter:
             if query_filter not in self.valid_filter:
-                message = f"invalid url query filder: {query_filter}"
+                message = f"invalid url query filter: {query_filter}"
                 print(message)
                 return Response({"message": message}, status=400)
 
@@ -657,7 +657,7 @@ class DownloadApiListView(ApiBaseView):
         must_list = []
         if query_filter:
             if query_filter not in self.valid_filter:
-                message = f"invalid url query filder: {query_filter}"
+                message = f"invalid url query filter: {query_filter}"
                 print(message)
                 return Response({"message": message}, status=400)
 

tubearchivist/config/management/commands/ta_config_backup.py

@@ -0,0 +1,76 @@
+"""backup config for sqlite reset and restore"""
+
+import json
+from pathlib import Path
+
+from django.contrib.auth import get_user_model
+from django.core.management.base import BaseCommand
+from home.models import CustomPeriodicTask
+from home.src.ta.settings import EnvironmentSettings
+from rest_framework.authtoken.models import Token
+
+User = get_user_model()
+
+
+class Command(BaseCommand):
+    """export"""
+
+    help = "Exports all users and their auth tokens to a JSON file"
+    FILE = Path(EnvironmentSettings.CACHE_DIR) / "backup" / "migration.json"
+
+    def handle(self, *args, **kwargs):
+        """entry point"""
+
+        data = {
+            "user_data": self.get_users(),
+            "schedule_data": self.get_schedules(),
+        }
+
+        with open(self.FILE, "w", encoding="utf-8") as json_file:
+            json_file.write(json.dumps(data))
+
+    def get_users(self):
+        """get users"""
+
+        users = User.objects.all()
+
+        user_data = []
+
+        for user in users:
+            user_info = {
+                "username": user.name,
+                "is_staff": user.is_staff,
+                "is_superuser": user.is_superuser,
+                "password": user.password,
+                "tokens": [],
+            }
+
+            try:
+                token = Token.objects.get(user=user)
+                user_info["tokens"] = [token.key]
+            except Token.DoesNotExist:
+                user_info["tokens"] = []
+
+            user_data.append(user_info)
+
+        return user_data
+
+    def get_schedules(self):
+        """get schedules"""
+
+        all_schedules = CustomPeriodicTask.objects.all()
+        schedule_data = []
+
+        for schedule in all_schedules:
+            schedule_info = {
+                "name": schedule.name,
+                "crontab": {
+                    "minute": schedule.crontab.minute,
+                    "hour": schedule.crontab.hour,
+                    "day_of_week": schedule.crontab.day_of_week,
+                },
+            }
+
+            schedule_data.append(schedule_info)
+
+        return schedule_data

tubearchivist/config/management/commands/ta_startup.py

@@ -5,12 +5,14 @@ Functionality:
 """
 
 import os
+from datetime import datetime
 from random import randint
 from time import sleep
 
 from django.conf import settings
 from django.core.management.base import BaseCommand, CommandError
-from django_celery_beat.models import CrontabSchedule
+from django.utils import dateformat
+from django_celery_beat.models import CrontabSchedule, PeriodicTasks
 from home.models import CustomPeriodicTask
 from home.src.es.connect import ElasticWrap
 from home.src.es.index_setup import ElasitIndexWrap
@@ -23,6 +25,7 @@ from home.src.ta.settings import EnvironmentSettings
 from home.src.ta.ta_redis import RedisArchivist
 from home.src.ta.task_config import TASK_CONFIG
 from home.src.ta.task_manager import TaskManager
+from home.tasks import version_check
 
 TOPIC = """
 
@@ -51,7 +54,9 @@ class Command(BaseCommand):
         self._mig_snapshot_check()
         self._mig_schedule_store()
         self._mig_custom_playlist()
+        self._mig_add_missing_timestamp()
         self._create_default_schedules()
+        self._update_schedule_tz()
 
     def _sync_redis_state(self):
         """make sure redis gets new config.json values"""
@@ -147,6 +152,14 @@ class Command(BaseCommand):
         else:
             self.stdout.write(self.style.SUCCESS("    no new update found"))
 
+        version_task = CustomPeriodicTask.objects.filter(name="version_check")
+        if not version_task.exists():
+            return
+
+        if not version_task.first().last_run_at:
+            self.style.SUCCESS("    ✓ send initial version check task")
+            version_check.delay()
+
     def _mig_index_setup(self):
         """migration: validate index mappings"""
         self.stdout.write("[MIGRATION] validate index mappings")
@@ -316,6 +329,22 @@ class Command(BaseCommand):
         sleep(60)
         raise CommandError(message)
 
+    def _mig_add_missing_timestamp(self) -> None:
+        """
+        add missing timestamp for versioncheck
+        migrate from v0.4.8 to v0.4.9
+        """
+        version_tasks = CustomPeriodicTask.objects.filter(name="version_check")
+        if not version_tasks.exists():
+            return
+
+        version_task = version_tasks.first()
+        if not version_task.last_run_at:
+            self.style.SUCCESS("    ✓ send initial version check task")
+            version_check.delay()
+            version_task.last_run_at = dateformat.make_aware(datetime.now())
+            version_task.save()
+
     def _create_default_schedules(self) -> None:
         """
         create default schedules for new installations
@@ -339,6 +368,7 @@ class Command(BaseCommand):
             "check_reindex", schedule=builder.SCHEDULES["check_reindex"]
         )
         check_reindex.task_config.update({"days": 90})
+        check_reindex.last_run_at = dateformat.make_aware(datetime.now())
         check_reindex.save()
         self.stdout.write(
             self.style.SUCCESS(
@@ -349,20 +379,39 @@ class Command(BaseCommand):
         thumbnail_check = builder.get_set_task(
             "thumbnail_check", schedule=builder.SCHEDULES["thumbnail_check"]
         )
+        thumbnail_check.last_run_at = dateformat.make_aware(datetime.now())
+        thumbnail_check.save()
         self.stdout.write(
             self.style.SUCCESS(
                 f"    ✓ created new default schedule: {thumbnail_check}"
             )
         )
         daily_random = f"{randint(0, 59)} {randint(0, 23)} *"
-        version_check = builder.get_set_task(
+        version_check_task = builder.get_set_task(
             "version_check", schedule=daily_random
         )
         self.stdout.write(
             self.style.SUCCESS(
-                f"    ✓ created new default schedule: {version_check}"
+                f"    ✓ created new default schedule: {version_check_task}"
             )
         )
         self.stdout.write(
             self.style.SUCCESS("    ✓ all default schedules created")
         )
+
+    def _update_schedule_tz(self) -> None:
+        """update timezone for Schedule instances"""
+        tz = EnvironmentSettings.TZ
+        to_update = CrontabSchedule.objects.exclude(timezone=tz)
+
+        if not to_update.exists():
+            self.stdout.write(
+                self.style.SUCCESS("    all schedules have correct TZ")
+            )
+            return
+
+        updated = to_update.update(timezone=tz)
+        self.stdout.write(
+            self.style.SUCCESS(f"    ✓ updated {updated} schedules to {tz}.")
+        )
+        PeriodicTasks.update_changed()

tubearchivist/config/settings.py

@@ -276,4 +276,4 @@ CORS_ALLOW_HEADERS = list(default_headers) + [
 
 # TA application settings
 TA_UPSTREAM = "https://github.com/tubearchivist/tubearchivist"
-TA_VERSION = "v0.4.8"
+TA_VERSION = "v0.4.13"

tubearchivist/home/src/download/queue.py

@@ -324,6 +324,10 @@ class PendingList(PendingIndex):
             else:
                 vid_type = VideoTypeEnum.VIDEOS
 
+        if not vid.get("channel"):
+            print(f"{youtube_id}: skip video not part of channel")
+            return False
+
         return self._parse_youtube_details(vid, vid_type)
 
     @staticmethod

tubearchivist/home/src/download/subscriptions.py

@@ -39,11 +39,15 @@ class ChannelSubscription:
         return all_channels
 
     def get_last_youtube_videos(
-        self, channel_id, limit=True, query_filter=VideoTypeEnum.UNKNOWN
+        self,
+        channel_id,
+        limit=True,
+        query_filter=None,
+        channel_overwrites=None,
     ):
         """get a list of last videos from channel"""
-        queries = self._build_queries(query_filter, limit)
-
+        query_handler = VideoQueryBuilder(self.config, channel_overwrites)
+        queries = query_handler.build_queries(query_filter)
         last_videos = []
 
         for vid_type_enum, limit_amount in queries:
@@ -51,55 +55,25 @@ class ChannelSubscription:
                 "skip_download": True,
                 "extract_flat": True,
             }
+            vid_type = vid_type_enum.value
+
             if limit:
                 obs["playlistend"] = limit_amount
 
-            vid_type = vid_type_enum.value
-            channel = YtWrap(obs, self.config).extract(
-                f"https://www.youtube.com/channel/{channel_id}/{vid_type}"
-            )
-            if not channel:
+            url = f"https://www.youtube.com/channel/{channel_id}/{vid_type}"
+            channel_query = YtWrap(obs, self.config).extract(url)
+            if not channel_query:
                 continue
+
             last_videos.extend(
-                [(i["id"], i["title"], vid_type) for i in channel["entries"]]
+                [
+                    (i["id"], i["title"], vid_type)
+                    for i in channel_query["entries"]
+                ]
             )
 
         return last_videos
 
-    def _build_queries(self, query_filter, limit):
-        """build query list for vid_type"""
-        limit_map = {
-            "videos": self.config["subscriptions"]["channel_size"],
-            "streams": self.config["subscriptions"]["live_channel_size"],
-            "shorts": self.config["subscriptions"]["shorts_channel_size"],
-        }
-
-        queries = []
-
-        if query_filter and query_filter.value != "unknown":
-            if limit:
-                query_limit = limit_map.get(query_filter.value)
-            else:
-                query_limit = False
-
-            queries.append((query_filter, query_limit))
-
-            return queries
-
-        for query_item, default_limit in limit_map.items():
-            if not default_limit:
-                # is deactivated in config
-                continue
-
-            if limit:
-                query_limit = default_limit
-            else:
-                query_limit = False
-
-            queries.append((VideoTypeEnum(query_item), query_limit))
-
-        return queries
-
     def find_missing(self):
         """add missing videos from subscribed channels to pending"""
         all_channels = self.get_channels()
@@ -112,7 +86,10 @@ class ChannelSubscription:
         for idx, channel in enumerate(all_channels):
             channel_id = channel["channel_id"]
             print(f"{channel_id}: find missing videos.")
-            last_videos = self.get_last_youtube_videos(channel_id)
+            last_videos = self.get_last_youtube_videos(
+                channel_id,
+                channel_overwrites=channel.get("channel_overwrites"),
+            )
 
             if last_videos:
                 ids_to_add = is_missing([i[0] for i in last_videos])
@@ -144,6 +121,92 @@ class ChannelSubscription:
         channel.sync_to_videos()
 
 
+class VideoQueryBuilder:
+    """Build queries for yt-dlp."""
+
+    def __init__(self, config: dict, channel_overwrites: dict | None = None):
+        self.config = config
+        self.channel_overwrites = channel_overwrites or {}
+
+    def build_queries(
+        self, video_type: VideoTypeEnum | None, limit: bool = True
+    ) -> list[tuple[VideoTypeEnum, int | None]]:
+        """Build queries for all or specific video type."""
+        query_methods = {
+            VideoTypeEnum.VIDEOS: self.videos_query,
+            VideoTypeEnum.STREAMS: self.streams_query,
+            VideoTypeEnum.SHORTS: self.shorts_query,
+        }
+
+        if video_type:
+            # build query for specific type
+            query_method = query_methods.get(video_type)
+            if query_method:
+                query = query_method(limit)
+                if query[1] != 0:
+                    return [query]
+                return []
+
+        # Build and return queries for all video types
+        queries = []
+        for build_query in query_methods.values():
+            query = build_query(limit)
+            if query[1] != 0:
+                queries.append(query)
+
+        return queries
+
+    def videos_query(self, limit: bool) -> tuple[VideoTypeEnum, int | None]:
+        """Build query for videos."""
+        return self._build_generic_query(
+            video_type=VideoTypeEnum.VIDEOS,
+            overwrite_key="subscriptions_channel_size",
+            config_key="channel_size",
+            limit=limit,
+        )
+
+    def streams_query(self, limit: bool) -> tuple[VideoTypeEnum, int | None]:
+        """Build query for streams."""
+        return self._build_generic_query(
+            video_type=VideoTypeEnum.STREAMS,
+            overwrite_key="subscriptions_live_channel_size",
+            config_key="live_channel_size",
+            limit=limit,
+        )
+
+    def shorts_query(self, limit: bool) -> tuple[VideoTypeEnum, int | None]:
+        """Build query for shorts."""
+        return self._build_generic_query(
+            video_type=VideoTypeEnum.SHORTS,
+            overwrite_key="subscriptions_shorts_channel_size",
+            config_key="shorts_channel_size",
+            limit=limit,
+        )
+
+    def _build_generic_query(
+        self,
+        video_type: VideoTypeEnum,
+        overwrite_key: str,
+        config_key: str,
+        limit: bool,
+    ) -> tuple[VideoTypeEnum, int | None]:
+        """Generic query for video page scraping."""
+        if not limit:
+            return (video_type, None)
+
+        if (
+            overwrite_key in self.channel_overwrites
+            and self.channel_overwrites[overwrite_key] is not None
+        ):
+            overwrite = self.channel_overwrites[overwrite_key]
+            return (video_type, overwrite)
+
+        if overwrite := self.config["subscriptions"].get(config_key):
+            return (video_type, overwrite)
+
+        return (video_type, 0)
+
+
 class PlaylistSubscription:
     """manage the playlist download functionality"""
 

tubearchivist/home/src/download/yt_dlp_handler.py

@@ -50,9 +50,10 @@ class VideoDownloader(DownloaderBase):
         self.obs = False
         self._build_obs()
 
-    def run_queue(self, auto_only=False) -> int:
+    def run_queue(self, auto_only=False) -> tuple[int, int]:
         """setup download queue in redis loop until no more items"""
         downloaded = 0
+        failed = 0
         while True:
             video_data = self._get_next(auto_only)
             if self.task.is_stopped() or not video_data:
@@ -66,6 +67,7 @@ class VideoDownloader(DownloaderBase):
 
             success = self._dl_single_vid(youtube_id, channel_id)
             if not success:
+                failed += 1
                 continue
 
             self._notify(video_data, "Add video metadata to index", progress=1)
@@ -82,7 +84,7 @@ class VideoDownloader(DownloaderBase):
         # post processing
         DownloadPostProcess(self.task).run()
 
-        return downloaded
+        return downloaded, failed
 
     def _notify(self, video_data, message, progress=False):
         """send progress notification to task"""
@@ -153,6 +155,7 @@ class VideoDownloader(DownloaderBase):
             "continuedl": True,
             "writethumbnail": False,
             "noplaylist": True,
+            "color": "no_color",
         }
 
     def _build_obs_user(self):
@@ -219,12 +222,6 @@ class VideoDownloader(DownloaderBase):
         self._set_overwrites(obs, channel_id)
         dl_cache = os.path.join(self.CACHE_DIR, "download")
 
-        # check if already in cache to continue from there
-        all_cached = ignore_filelist(os.listdir(dl_cache))
-        for file_name in all_cached:
-            if youtube_id in file_name:
-                obs["outtmpl"] = os.path.join(dl_cache, file_name)
-
         success, message = YtWrap(obs, self.config).download(youtube_id)
         if not success:
             self._handle_error(youtube_id, message)

tubearchivist/home/src/es/index_mapping.json

@@ -75,6 +75,15 @@
                         },
                         "integrate_sponsorblock": {
                             "type": "boolean"
+                        },
+                        "subscriptions_channel_size": {
+                            "type": "long"
+                        },
+                        "subscriptions_live_channel_size": {
+                            "type": "long"
+                        },
+                        "subscriptions_shorts_channel_size": {
+                            "type": "long"
                         }
                     }
                 }
@@ -169,6 +178,15 @@
                                 },
                                 "integrate_sponsorblock": {
                                     "type": "boolean"
+                                },
+                                "subscriptions_channel_size": {
+                                    "type": "long"
+                                },
+                                "subscriptions_live_channel_size": {
+                                    "type": "long"
+                                },
+                                "subscriptions_shorts_channel_size": {
+                                    "type": "long"
                                 }
                             }
                         }

tubearchivist/home/src/frontend/forms.py

@@ -259,3 +259,12 @@ class ChannelOverwriteForm(forms.Form):
     integrate_sponsorblock = forms.ChoiceField(
         widget=forms.Select, choices=SP_CHOICES, required=False
     )
+    subscriptions_channel_size = forms.IntegerField(
+        label=False, required=False
+    )
+    subscriptions_live_channel_size = forms.IntegerField(
+        label=False, required=False
+    )
+    subscriptions_shorts_channel_size = forms.IntegerField(
+        label=False, required=False
+    )

tubearchivist/home/src/index/channel.py

@@ -6,46 +6,16 @@ functionality:
 
 import json
 import os
-import re
 from datetime import datetime
 
-import requests
 from home.src.download.thumbnails import ThumbManager
 from home.src.download.yt_dlp_base import YtWrap
 from home.src.es.connect import ElasticWrap, IndexPaginate
 from home.src.index.generic import YouTubeItem
 from home.src.index.playlist import YoutubePlaylist
-from home.src.ta.helper import requests_headers
 from home.src.ta.settings import EnvironmentSettings
 
 
-def banner_extractor(channel_id: str) -> dict[str, str] | None:
-    """workaround for new channel renderer, upstream #9893"""
-    url = f"https://www.youtube.com/channel/{channel_id}?hl=en"
-    cookies = {"SOCS": "CAI"}
-    response = requests.get(
-        url, cookies=cookies, headers=requests_headers(), timeout=30
-    )
-    if not response.ok:
-        return None
-
-    matched_urls = re.findall(
-        r'"(https://yt3.googleusercontent.com/[^"]+=w(\d{3,4})-fcrop64[^"]*)"',
-        response.text,
-    )
-    if not matched_urls:
-        return None
-
-    sorted_urls = sorted(matched_urls, key=lambda x: int(x[1]), reverse=True)
-    banner = sorted_urls[0][0]
-    channel_art_fallback = {
-        "channel_banner_url": banner,
-        "channel_tvart_url": banner.split("-fcrop64")[0],
-    }
-
-    return channel_art_fallback
-
-
 class YoutubeChannel(YouTubeItem):
     """represents a single youtube channel"""
 
@@ -87,7 +57,7 @@ class YoutubeChannel(YouTubeItem):
             "channel_id": self.youtube_id,
             "channel_last_refresh": int(datetime.now().timestamp()),
             "channel_name": self.youtube_meta["uploader"],
-            "channel_subs": self._extract_follower_count(),
+            "channel_subs": self.youtube_meta.get("channel_follower_count", 0),
             "channel_subscribed": False,
             "channel_tags": self._parse_tags(self.youtube_meta.get("tags")),
             "channel_banner_url": self._get_banner_art(),
@@ -95,34 +65,6 @@ class YoutubeChannel(YouTubeItem):
             "channel_tvart_url": self._get_tv_art(),
             "channel_views": self.youtube_meta.get("view_count") or 0,
         }
-        self._inject_fallback()
-
-    def _inject_fallback(self):
-        """fallback channel art work, workaround for upstream #9893"""
-        if self.json_data["channel_banner_url"]:
-            return
-
-        print(f"{self.youtube_id}: attempt art fallback extraction")
-        fallback = banner_extractor(self.youtube_id)
-        if fallback:
-            print(f"{self.youtube_id}: fallback succeeded: {fallback}")
-            self.json_data.update(fallback)
-
-    def _extract_follower_count(self) -> int:
-        """workaround for upstream #9893, extract subs from first video"""
-        subs = self.youtube_meta.get("channel_follower_count")
-        if subs is not None:
-            return subs
-
-        entries = self.youtube_meta.get("entries", [])
-        if entries:
-            first_entry = entries[0]
-            if isinstance(first_entry, dict):
-                subs_entry = first_entry.get("channel_follower_count")
-                if subs_entry is not None:
-                    return subs_entry
-
-        return 0
 
     def _parse_tags(self, tags):
         """parse channel tags"""
@@ -375,23 +317,30 @@ class YoutubeChannel(YouTubeItem):
             "autodelete_days",
             "index_playlists",
             "integrate_sponsorblock",
+            "subscriptions_channel_size",
+            "subscriptions_live_channel_size",
+            "subscriptions_shorts_channel_size",
         ]
 
         to_write = self.json_data.get("channel_overwrites", {})
         for key, value in overwrites.items():
             if key not in valid_keys:
                 raise ValueError(f"invalid overwrite key: {key}")
-            if value == "disable":
+            elif value == "disable":
                 to_write[key] = False
                 continue
-            if value in [0, "0"]:
+            elif value == "0":
                 if key in to_write:
                     del to_write[key]
                 continue
-            if value == "1":
+            elif value == "1":
                 to_write[key] = True
                 continue
-            if value:
+            elif isinstance(value, int) and int(value) < 0:
+                if key in to_write:
+                    del to_write[key]
+                continue
+            elif value is not None and value != "":
                 to_write.update({key: value})
 
         self.json_data["channel_overwrites"] = to_write

tubearchivist/home/src/index/comments.py

@@ -69,7 +69,6 @@ class Comments:
                 "youtube": {
                     "max_comments": max_comments_list,
                     "comment_sort": [comment_sort],
-                    "player_client": ["ios", "web"],  # workaround yt-dlp #9554
                 }
             },
         }

tubearchivist/home/src/index/playlist.py

@@ -93,13 +93,10 @@ class YoutubePlaylist(YouTubeItem):
         """get all videos in playlist, match downloaded with ids_found"""
         all_members = []
         for idx, entry in enumerate(self.youtube_meta["entries"]):
-            if not entry["channel"]:
-                continue
-
             to_append = {
                 "youtube_id": entry["id"],
                 "title": entry["title"],
-                "uploader": entry["channel"],
+                "uploader": entry.get("channel"),
                 "idx": idx,
                 "downloaded": entry["id"] in ids_found,
             }

tubearchivist/home/src/ta/config_schedule.py

@@ -3,6 +3,9 @@ Functionality:
 - Handle scheduler config update
 """
 
+from datetime import datetime
+
+from django.utils import dateformat
 from django_celery_beat.models import CrontabSchedule
 from home.models import CustomPeriodicTask
 from home.src.ta.config import AppConfig
@@ -64,6 +67,7 @@ class ScheduleBuilder:
         if schedule:
             task_crontab = self.get_set_cron_tab(schedule)
             task.crontab = task_crontab
+            task.last_run_at = dateformat.make_aware(datetime.now())
             task.save()
 
         return task

tubearchivist/home/tasks.py

@@ -7,6 +7,7 @@ Functionality:
 """
 
 from celery import Task, shared_task
+from celery.exceptions import Retry
 from home.src.download.queue import PendingList
 from home.src.download.subscriptions import (
     SubscriptionHandler,
@@ -114,7 +115,13 @@ def update_subscribed(self):
     return None
 
 
-@shared_task(name="download_pending", bind=True, base=BaseTask)
+@shared_task(
+    name="download_pending",
+    bind=True,
+    base=BaseTask,
+    max_retries=3,
+    default_retry_delay=10,
+)
 def download_pending(self, auto_only=False):
     """download latest pending videos"""
     manager = TaskManager()
@@ -124,11 +131,20 @@ def download_pending(self, auto_only=False):
         return None
 
     manager.init(self)
-    downloader = VideoDownloader(task=self)
-    videos_downloaded = downloader.run_queue(auto_only=auto_only)
+    try:
+        downloader = VideoDownloader(task=self)
+        downloaded, failed = downloader.run_queue(auto_only=auto_only)
 
-    if videos_downloaded:
-        return f"downloaded {videos_downloaded} video(s)."
+        if failed:
+            print(f"[task][{self.name}] Videos failed, retry.")
+            self.send_progress("Videos failed, retry.")
+            raise self.retry()
+
+    except Retry as exc:
+        raise exc
+
+    if downloaded:
+        return f"downloaded {downloaded} video(s)."
 
     return None
 

tubearchivist/home/templates/home/channel_id_about.html

@@ -107,16 +107,20 @@
                                 {{ channel_info.channel_overwrites.download_format }}
                             {% else %}
                                 False
-                            {% endif %}</span></p>
+                            {% endif %}</span><br>
+                            Enter "disable" to disable this override.
+                        </p>
                         {{ channel_overwrite_form.download_format }}<br>
                     </div>
                     <div class="overwrite-form-item">
                         <p>Auto delete watched videos after x days: <span class="settings-current">
-                            {% if channel_info.channel_overwrites.autodelete_days %}
+                            {% if channel_info.channel_overwrites.autodelete_days is not None %}
                                 {{ channel_info.channel_overwrites.autodelete_days }}
                             {% else %}
                                 False
-                            {% endif %}</span></p>
+                            {% endif %}</span><br>
+                            Enter a negative number to disable this override.
+                        </p>
                         {{ channel_overwrite_form.autodelete_days }}<br>
                     </div>
                     <div class="overwrite-form-item">
@@ -139,6 +143,39 @@
                             {% endif %}</span></p>
                         {{ channel_overwrite_form.integrate_sponsorblock }}<br>
                     </div>
+                    <h3>Page Size Overrides</h3><br>
+                    <p>Disable standard videos, shorts, or streams for this channel by setting their page size to 0 (zero).</p><br>
+                    <p>Disable page size overwrite for channel by setting to negative value.</p><br>
+                    <div class="overwrite-form-item">
+                        <p>YouTube page size: <span class="settings-current">
+                            {% if channel_info.channel_overwrites.subscriptions_channel_size is not None %}
+                                {{ channel_info.channel_overwrites.subscriptions_channel_size }}
+                            {% else %}
+                                False
+                            {% endif %}</span></p>
+                        <i>Videos to scan to find new items for the <b>Rescan subscriptions</b> task, max recommended 50.</i><br>
+                        {{ channel_overwrite_form.subscriptions_channel_size }}<br>
+                    </div>
+                    <div class="overwrite-form-item">
+                        <p>YouTube Live page size: <span class="settings-current">
+                            {% if channel_info.channel_overwrites.subscriptions_live_channel_size is not None %}
+                                {{ channel_info.channel_overwrites.subscriptions_live_channel_size }}
+                            {% else %}
+                                False
+                            {% endif %}</span></p>
+                        <i>Live Videos to scan to find new items for the <b>Rescan subscriptions</b> task, max recommended 50.</i><br>
+                        {{ channel_overwrite_form.subscriptions_live_channel_size }}<br>
+                    </div>
+                    <div class="overwrite-form-item">
+                        <p>YouTube Shorts page size: <span class="settings-current">
+                            {% if channel_info.channel_overwrites.subscriptions_shorts_channel_size is not None %}
+                                {{ channel_info.channel_overwrites.subscriptions_shorts_channel_size }}
+                            {% else %}
+                                False
+                            {% endif %}</span></p>
+                        <i>Shorts Videos to scan to find new items for the <b>Rescan subscriptions</b> task, max recommended 50.</i><br>
+                        {{ channel_overwrite_form.subscriptions_shorts_channel_size }}<br>
+                    </div><br>
                     <button type="submit">Save Channel Overwrites</button>
                 </form>
             </div>
@@ -146,4 +183,4 @@
     {% endif %}
 </div>
 <script type="text/javascript" src="{% static 'progress.js' %}"></script>
-{% endblock content %}
+{% endblock content %}

tubearchivist/requirements-dev.txt

@@ -1,10 +1,11 @@
 -r requirements.txt
-black
-codespell
-flake8
-isort
-pylint
-pylint-django
-pytest
-pytest-django
-types-requests
+black==24.10.0
+codespell==2.3.0
+flake8==7.1.1
+isort==5.13.2
+pylint-django==2.6.1
+pylint==3.3.3
+pytest-django==4.9.0
+pytest==8.3.4
+requirementscheck==0.0.5
+types-requests==2.32.0.20241016

tubearchivist/requirements.txt

@@ -1,14 +1,14 @@
-apprise==1.8.0
+apprise==1.9.2
 celery==5.4.0
-Django==5.0.6
-django-auth-ldap==4.8.0
-django-celery-beat==2.6.0
-django-cors-headers==4.3.1
-djangorestframework==3.15.1
-Pillow==10.3.0
-redis==5.0.4
-requests==2.32.1
+django-auth-ldap==5.1.0
+django-celery-beat==2.7.0
+django-cors-headers==4.6.0
+Django==5.1.5
+djangorestframework==3.15.2
+Pillow==11.1.0
+redis==5.2.1
+requests==2.32.3
 ryd-client==0.0.6
-uWSGI==2.0.25.1
-whitenoise==6.6.0
-yt-dlp @ git+https://github.com/yt-dlp/yt-dlp@8e15177b4113c355989881e4e030f695a9b59c3a
+uWSGI==2.0.28
+whitenoise==6.8.2
+yt-dlp[default]==2025.1.15