bump TA_VERSION

Changed:
- Bumped yt-dlp
- Added migration to update TZ in schedule when changed
- Fixed issue where version check would never trigger

.github/ISSUE_TEMPLATE/FEATURE-REQUEST.yml

@@ -1,34 +1,12 @@
 name: Feature Request
-description: I have an idea for a great addition to this project
+description: This Project currently doesn't take any new feature requests.
 title: "[Feature Request]: "
 
 body:
-  - type: markdown
-    attributes:
-      value: |
-        Thanks for taking the time to help improve this project! This project is *very* selective with accepting new feature requests. Please read the [how to open an issue](https://github.com/tubearchivist/tubearchivist/blob/master/CONTRIBUTING.md#how-to-open-an-issue) guide carefully before continuing.
-
   - type: checkboxes
-    id: already
+    id: block
     attributes:
-      label: "I've read the documentation"
+      label: "This project doesn't accept any new feature requests for the forseeable future. There is no shortage of ideas and the next development steps are clear for years to come."
       options:
-        - label: I have read the [how to open an issue](https://github.com/tubearchivist/tubearchivist/blob/master/CONTRIBUTING.md#how-to-open-an-issue) guide, particularly the [feature request](https://github.com/tubearchivist/tubearchivist/blob/master/CONTRIBUTING.md#feature-request) section.
+        - label: I understand that this issue will be closed without comment.
           required: true
-
-  - type: textarea
-    id: description
-    attributes:
-      label: Your Feature Request
-      value: "## Is your feature request related to a problem? Please describe.\n\n## Describe the solution you'd like\n\n## Additional context"
-      placeholder: Tell us what you see!
-    validations:
-      required: true
-
-  - type: checkboxes
-    id: help
-    attributes:
-      label: Your help is needed!
-      description: This project is ambitious as it is, please contribute.
-      options:
-        - label: Yes I will work on this in the next few days or weeks.

CONTRIBUTING.md

@@ -62,22 +62,7 @@ Please keep in mind:
 - A bug that can't be reproduced, is difficult or sometimes even impossible to fix. Provide very clear steps *how to reproduce*.
 
 ### Feature Request
-This project needs your help to grow further. There is no shortage of ideas, see the open [issues on GH](https://github.com/tubearchivist/tubearchivist/issues?q=is%3Aopen+is%3Aissue+label%3Aenhancement) and the [roadmap](https://github.com/tubearchivist/tubearchivist#roadmap), what this project lacks is contributors interested in helping with overall improvements of the application. Focus is *not* on adding new features, but improving existing ones.
-
-Existing ideas are easily *multiple years* worth of development effort, at least at current speed. This project is *very* selective with accepting new feature requests at this point.  
-
-Good feature requests usually fall into one or more of these categories:
-- You want to work on your own small scoped idea within the next few days or weeks.
-- Your idea is beneficial for a wide range of users, not just for you.
-- Your idea extends the current project by building on and improving existing functionality.
-- Your idea is quick and easy to implement, for an experienced as well as for a first time contributor.
-
-Your request is likely going to be rejected if:
-- Your idea requires multiple days worth of development time and is unrealistic to be implemented any time soon.
-- There are already other ways to do what you are trying to do.
-- You are trying to do something that only applies to your platform, your specific workflow or your specific setup.
-- Your idea would fundamentally change how the project works or it wouldn't be able to be implemented with backwards compatibility.
-- Your idea is not a good fit for this project.
+This project doesn't take any new feature requests. This project doesn't lack ideas, see the currently open tasks and roadmap. New feature requests aren't helpful at this point in time. Thank you for your understanding.
 
 ### Installation Help
 GitHub is most likely not the best place to ask for installation help. That's inherently individual and one on one.

Dockerfile

@@ -4,7 +4,6 @@
 
 # First stage to build python wheel
 FROM python:3.11.8-slim-bookworm AS builder
-ARG TARGETPLATFORM
 
 RUN apt-get update && apt-get install -y --no-install-recommends \
     build-essential gcc libldap2-dev libsasl2-dev libssl-dev git
@@ -15,13 +14,15 @@ RUN pip install --user -r requirements.txt
 
 # build ffmpeg
 FROM python:3.11.8-slim-bookworm as ffmpeg-builder
+
+ARG TARGETPLATFORM
+
 COPY docker_assets/ffmpeg_download.py ffmpeg_download.py
 RUN python ffmpeg_download.py $TARGETPLATFORM
 
 # build final image
 FROM python:3.11.8-slim-bookworm as tubearchivist
 
-ARG TARGETPLATFORM
 ARG INSTALL_DEBUG
 
 ENV PYTHONUNBUFFERED 1

README.md

@@ -41,7 +41,7 @@ Once your YouTube video collection grows, it becomes hard to search and find a s
 For minimal system requirements, the Tube Archivist stack needs around 2GB of available memory for a small testing setup and around 4GB of available memory for a mid to large sized installation. Minimal with dual core with 4 threads, better quad core plus. 
 This project requires docker. Ensure it is installed and running on your system.  
 
-The documentation has additional user provided instructions for [Unraid](https://docs.tubearchivist.com/installation/unraid/), [Synology](https://docs.tubearchivist.com/installation/synology/), [Podman](https://docs.tubearchivist.com/installation/podman/) and [True NAS](https://docs.tubearchivist.com/installation/truenas-scale/).
+The documentation has additional user provided instructions for [Unraid](https://docs.tubearchivist.com/installation/unraid/), [Synology](https://docs.tubearchivist.com/installation/synology/) and [Podman](https://docs.tubearchivist.com/installation/podman/).
 
 The instructions here should get you up and running quickly, for Docker beginners and full explanation about each environment variable, see the [docs](https://docs.tubearchivist.com/installation/docker-compose/).
 
@@ -148,7 +148,6 @@ This can happen if you have nested virtualizations, e.g. LXC running Docker in P
 ## Roadmap
 We have come far, nonetheless we are not short of ideas on how to improve and extend this project. Issues waiting for you to be tackled in no particular order:
 
-- [ ] User roles
 - [ ] Audio download
 - [ ] Podcast mode to serve channel as mp3
 - [ ] Random and repeat controls ([#108](https://github.com/tubearchivist/tubearchivist/issues/108), [#220](https://github.com/tubearchivist/tubearchivist/issues/220))
@@ -159,10 +158,12 @@ We have come far, nonetheless we are not short of ideas on how to improve and ex
 - [ ] Custom searchable notes to videos, channels, playlists ([#144](https://github.com/tubearchivist/tubearchivist/issues/144))
 - [ ] Search comments
 - [ ] Search download queue
-- [ ] Configure shorts, streams and video sizes per channel
+- [ ] Per user videos/channel/playlists
 
 Implemented:
+- [X] Configure shorts, streams and video sizes per channel [2024-07-15]
 - [X] User created playlists [2024-04-10]
+- [X] User roles, aka read only user [2023-11-10]
 - [X] Add statistics of index [2023-09-03]
 - [X] Implement [Apprise](https://github.com/caronc/apprise) for notifications [2023-08-05]
 - [X] Download video comments [2022-11-30]

docker-compose.yml

@@ -40,12 +40,12 @@ services:
     depends_on:
       - archivist-es
   archivist-es:
-    image: bbilly1/tubearchivist-es         # only for amd64, or use official es 8.13.2
+    image: bbilly1/tubearchivist-es         # only for amd64, or use official es 8.14.3
     container_name: archivist-es
     restart: unless-stopped
     environment:
       - "ELASTIC_PASSWORD=verysecret"       # matching Elasticsearch password
-      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
+      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
       - "xpack.security.enabled=true"
       - "discovery.type=single-node"
       - "path.repo=/usr/share/elasticsearch/data/snapshot"

tubearchivist/api/views.py

@@ -361,7 +361,7 @@ class ChannelApiListView(ApiBaseView):
         must_list = []
         if query_filter:
             if query_filter not in self.valid_filter:
-                message = f"invalid url query filder: {query_filter}"
+                message = f"invalid url query filter: {query_filter}"
                 print(message)
                 return Response({"message": message}, status=400)
 
@@ -657,7 +657,7 @@ class DownloadApiListView(ApiBaseView):
         must_list = []
         if query_filter:
             if query_filter not in self.valid_filter:
-                message = f"invalid url query filder: {query_filter}"
+                message = f"invalid url query filter: {query_filter}"
                 print(message)
                 return Response({"message": message}, status=400)
 

tubearchivist/config/management/commands/ta_startup.py

@@ -5,12 +5,14 @@ Functionality:
 """
 
 import os
+from datetime import datetime
 from random import randint
 from time import sleep
 
 from django.conf import settings
 from django.core.management.base import BaseCommand, CommandError
-from django_celery_beat.models import CrontabSchedule
+from django.utils import dateformat
+from django_celery_beat.models import CrontabSchedule, PeriodicTasks
 from home.models import CustomPeriodicTask
 from home.src.es.connect import ElasticWrap
 from home.src.es.index_setup import ElasitIndexWrap
@@ -23,6 +25,7 @@ from home.src.ta.settings import EnvironmentSettings
 from home.src.ta.ta_redis import RedisArchivist
 from home.src.ta.task_config import TASK_CONFIG
 from home.src.ta.task_manager import TaskManager
+from home.tasks import version_check
 
 TOPIC = """
 
@@ -51,7 +54,9 @@ class Command(BaseCommand):
         self._mig_snapshot_check()
         self._mig_schedule_store()
         self._mig_custom_playlist()
+        self._mig_add_missing_timestamp()
         self._create_default_schedules()
+        self._update_schedule_tz()
 
     def _sync_redis_state(self):
         """make sure redis gets new config.json values"""
@@ -147,6 +152,14 @@ class Command(BaseCommand):
         else:
             self.stdout.write(self.style.SUCCESS("    no new update found"))
 
+        version_task = CustomPeriodicTask.objects.filter(name="version_check")
+        if not version_task.exists():
+            return
+
+        if not version_task.first().last_run_at:
+            self.style.SUCCESS("    ✓ send initial version check task")
+            version_check.delay()
+
     def _mig_index_setup(self):
         """migration: validate index mappings"""
         self.stdout.write("[MIGRATION] validate index mappings")
@@ -316,6 +329,22 @@ class Command(BaseCommand):
         sleep(60)
         raise CommandError(message)
 
+    def _mig_add_missing_timestamp(self) -> None:
+        """
+        add missing timestamp for versioncheck
+        migrate from v0.4.8 to v0.4.9
+        """
+        version_tasks = CustomPeriodicTask.objects.filter(name="version_check")
+        if not version_tasks.exists():
+            return
+
+        version_task = version_tasks.first()
+        if not version_task.last_run_at:
+            self.style.SUCCESS("    ✓ send initial version check task")
+            version_check.delay()
+            version_task.last_run_at = dateformat.make_aware(datetime.now())
+            version_task.save()
+
     def _create_default_schedules(self) -> None:
         """
         create default schedules for new installations
@@ -339,6 +368,7 @@ class Command(BaseCommand):
             "check_reindex", schedule=builder.SCHEDULES["check_reindex"]
         )
         check_reindex.task_config.update({"days": 90})
+        check_reindex.last_run_at = dateformat.make_aware(datetime.now())
         check_reindex.save()
         self.stdout.write(
             self.style.SUCCESS(
@@ -349,20 +379,39 @@ class Command(BaseCommand):
         thumbnail_check = builder.get_set_task(
             "thumbnail_check", schedule=builder.SCHEDULES["thumbnail_check"]
         )
+        thumbnail_check.last_run_at = dateformat.make_aware(datetime.now())
+        thumbnail_check.save()
         self.stdout.write(
             self.style.SUCCESS(
                 f"    ✓ created new default schedule: {thumbnail_check}"
             )
         )
         daily_random = f"{randint(0, 59)} {randint(0, 23)} *"
-        version_check = builder.get_set_task(
+        version_check_task = builder.get_set_task(
             "version_check", schedule=daily_random
         )
         self.stdout.write(
             self.style.SUCCESS(
-                f"    ✓ created new default schedule: {version_check}"
+                f"    ✓ created new default schedule: {version_check_task}"
             )
         )
         self.stdout.write(
             self.style.SUCCESS("    ✓ all default schedules created")
         )
+
+    def _update_schedule_tz(self) -> None:
+        """update timezone for Schedule instances"""
+        tz = EnvironmentSettings.TZ
+        to_update = CrontabSchedule.objects.exclude(timezone=tz)
+
+        if not to_update.exists():
+            self.stdout.write(
+                self.style.SUCCESS("    all schedules have correct TZ")
+            )
+            return
+
+        updated = to_update.update(timezone=tz)
+        self.stdout.write(
+            self.style.SUCCESS(f"    ✓ updated {updated} schedules to {tz}.")
+        )
+        PeriodicTasks.update_changed()

tubearchivist/config/settings.py

@@ -276,4 +276,4 @@ CORS_ALLOW_HEADERS = list(default_headers) + [
 
 # TA application settings
 TA_UPSTREAM = "https://github.com/tubearchivist/tubearchivist"
-TA_VERSION = "v0.4.8"
+TA_VERSION = "v0.4.10"

tubearchivist/home/src/download/queue.py

@@ -324,6 +324,10 @@ class PendingList(PendingIndex):
             else:
                 vid_type = VideoTypeEnum.VIDEOS
 
+        if not vid.get("channel"):
+            print(f"{youtube_id}: skip video not part of channel")
+            return False
+
         return self._parse_youtube_details(vid, vid_type)
 
     @staticmethod

tubearchivist/home/src/download/subscriptions.py

@@ -39,11 +39,15 @@ class ChannelSubscription:
         return all_channels
 
     def get_last_youtube_videos(
-        self, channel_id, limit=True, query_filter=VideoTypeEnum.UNKNOWN
+        self,
+        channel_id,
+        limit=True,
+        query_filter=None,
+        channel_overwrites=None,
     ):
         """get a list of last videos from channel"""
-        queries = self._build_queries(query_filter, limit)
-
+        query_handler = VideoQueryBuilder(self.config, channel_overwrites)
+        queries = query_handler.build_queries(query_filter)
         last_videos = []
 
         for vid_type_enum, limit_amount in queries:
@@ -51,55 +55,25 @@ class ChannelSubscription:
                 "skip_download": True,
                 "extract_flat": True,
             }
+            vid_type = vid_type_enum.value
+
             if limit:
                 obs["playlistend"] = limit_amount
 
-            vid_type = vid_type_enum.value
-            channel = YtWrap(obs, self.config).extract(
-                f"https://www.youtube.com/channel/{channel_id}/{vid_type}"
-            )
-            if not channel:
+            url = f"https://www.youtube.com/channel/{channel_id}/{vid_type}"
+            channel_query = YtWrap(obs, self.config).extract(url)
+            if not channel_query:
                 continue
+
             last_videos.extend(
-                [(i["id"], i["title"], vid_type) for i in channel["entries"]]
+                [
+                    (i["id"], i["title"], vid_type)
+                    for i in channel_query["entries"]
+                ]
             )
 
         return last_videos
 
-    def _build_queries(self, query_filter, limit):
-        """build query list for vid_type"""
-        limit_map = {
-            "videos": self.config["subscriptions"]["channel_size"],
-            "streams": self.config["subscriptions"]["live_channel_size"],
-            "shorts": self.config["subscriptions"]["shorts_channel_size"],
-        }
-
-        queries = []
-
-        if query_filter and query_filter.value != "unknown":
-            if limit:
-                query_limit = limit_map.get(query_filter.value)
-            else:
-                query_limit = False
-
-            queries.append((query_filter, query_limit))
-
-            return queries
-
-        for query_item, default_limit in limit_map.items():
-            if not default_limit:
-                # is deactivated in config
-                continue
-
-            if limit:
-                query_limit = default_limit
-            else:
-                query_limit = False
-
-            queries.append((VideoTypeEnum(query_item), query_limit))
-
-        return queries
-
     def find_missing(self):
         """add missing videos from subscribed channels to pending"""
         all_channels = self.get_channels()
@@ -112,7 +86,10 @@ class ChannelSubscription:
         for idx, channel in enumerate(all_channels):
             channel_id = channel["channel_id"]
             print(f"{channel_id}: find missing videos.")
-            last_videos = self.get_last_youtube_videos(channel_id)
+            last_videos = self.get_last_youtube_videos(
+                channel_id,
+                channel_overwrites=channel.get("channel_overwrites"),
+            )
 
             if last_videos:
                 ids_to_add = is_missing([i[0] for i in last_videos])
@@ -144,6 +121,92 @@ class ChannelSubscription:
         channel.sync_to_videos()
 
 
+class VideoQueryBuilder:
+    """Build queries for yt-dlp."""
+
+    def __init__(self, config: dict, channel_overwrites: dict | None = None):
+        self.config = config
+        self.channel_overwrites = channel_overwrites or {}
+
+    def build_queries(
+        self, video_type: VideoTypeEnum | None, limit: bool = True
+    ) -> list[tuple[VideoTypeEnum, int | None]]:
+        """Build queries for all or specific video type."""
+        query_methods = {
+            VideoTypeEnum.VIDEOS: self.videos_query,
+            VideoTypeEnum.STREAMS: self.streams_query,
+            VideoTypeEnum.SHORTS: self.shorts_query,
+        }
+
+        if video_type:
+            # build query for specific type
+            query_method = query_methods.get(video_type)
+            if query_method:
+                query = query_method(limit)
+                if query[1] != 0:
+                    return [query]
+                return []
+
+        # Build and return queries for all video types
+        queries = []
+        for build_query in query_methods.values():
+            query = build_query(limit)
+            if query[1] != 0:
+                queries.append(query)
+
+        return queries
+
+    def videos_query(self, limit: bool) -> tuple[VideoTypeEnum, int | None]:
+        """Build query for videos."""
+        return self._build_generic_query(
+            video_type=VideoTypeEnum.VIDEOS,
+            overwrite_key="subscriptions_channel_size",
+            config_key="channel_size",
+            limit=limit,
+        )
+
+    def streams_query(self, limit: bool) -> tuple[VideoTypeEnum, int | None]:
+        """Build query for streams."""
+        return self._build_generic_query(
+            video_type=VideoTypeEnum.STREAMS,
+            overwrite_key="subscriptions_live_channel_size",
+            config_key="live_channel_size",
+            limit=limit,
+        )
+
+    def shorts_query(self, limit: bool) -> tuple[VideoTypeEnum, int | None]:
+        """Build query for shorts."""
+        return self._build_generic_query(
+            video_type=VideoTypeEnum.SHORTS,
+            overwrite_key="subscriptions_shorts_channel_size",
+            config_key="shorts_channel_size",
+            limit=limit,
+        )
+
+    def _build_generic_query(
+        self,
+        video_type: VideoTypeEnum,
+        overwrite_key: str,
+        config_key: str,
+        limit: bool,
+    ) -> tuple[VideoTypeEnum, int | None]:
+        """Generic query for video page scraping."""
+        if not limit:
+            return (video_type, None)
+
+        if (
+            overwrite_key in self.channel_overwrites
+            and self.channel_overwrites[overwrite_key] is not None
+        ):
+            overwrite = self.channel_overwrites[overwrite_key]
+            return (video_type, overwrite)
+
+        if overwrite := self.config["subscriptions"].get(config_key):
+            return (video_type, overwrite)
+
+        return (video_type, 0)
+
+
 class PlaylistSubscription:
     """manage the playlist download functionality"""
 

tubearchivist/home/src/download/yt_dlp_handler.py

@@ -50,9 +50,10 @@ class VideoDownloader(DownloaderBase):
         self.obs = False
         self._build_obs()
 
-    def run_queue(self, auto_only=False) -> int:
+    def run_queue(self, auto_only=False) -> tuple[int, int]:
         """setup download queue in redis loop until no more items"""
         downloaded = 0
+        failed = 0
         while True:
             video_data = self._get_next(auto_only)
             if self.task.is_stopped() or not video_data:
@@ -66,6 +67,7 @@ class VideoDownloader(DownloaderBase):
 
             success = self._dl_single_vid(youtube_id, channel_id)
             if not success:
+                failed += 1
                 continue
 
             self._notify(video_data, "Add video metadata to index", progress=1)
@@ -82,7 +84,7 @@ class VideoDownloader(DownloaderBase):
         # post processing
         DownloadPostProcess(self.task).run()
 
-        return downloaded
+        return downloaded, failed
 
     def _notify(self, video_data, message, progress=False):
         """send progress notification to task"""
@@ -153,6 +155,7 @@ class VideoDownloader(DownloaderBase):
             "continuedl": True,
             "writethumbnail": False,
             "noplaylist": True,
+            "color": "no_color",
         }
 
     def _build_obs_user(self):
@@ -219,12 +222,6 @@ class VideoDownloader(DownloaderBase):
         self._set_overwrites(obs, channel_id)
         dl_cache = os.path.join(self.CACHE_DIR, "download")
 
-        # check if already in cache to continue from there
-        all_cached = ignore_filelist(os.listdir(dl_cache))
-        for file_name in all_cached:
-            if youtube_id in file_name:
-                obs["outtmpl"] = os.path.join(dl_cache, file_name)
-
         success, message = YtWrap(obs, self.config).download(youtube_id)
         if not success:
             self._handle_error(youtube_id, message)

tubearchivist/home/src/es/index_mapping.json

@@ -75,6 +75,15 @@
                         },
                         "integrate_sponsorblock": {
                             "type": "boolean"
+                        },
+                        "subscriptions_channel_size": {
+                            "type": "long"
+                        },
+                        "subscriptions_live_channel_size": {
+                            "type": "long"
+                        },
+                        "subscriptions_shorts_channel_size": {
+                            "type": "long"
                         }
                     }
                 }
@@ -169,6 +178,15 @@
                                 },
                                 "integrate_sponsorblock": {
                                     "type": "boolean"
+                                },
+                                "subscriptions_channel_size": {
+                                    "type": "long"
+                                },
+                                "subscriptions_live_channel_size": {
+                                    "type": "long"
+                                },
+                                "subscriptions_shorts_channel_size": {
+                                    "type": "long"
                                 }
                             }
                         }

tubearchivist/home/src/frontend/forms.py

@@ -259,3 +259,12 @@ class ChannelOverwriteForm(forms.Form):
     integrate_sponsorblock = forms.ChoiceField(
         widget=forms.Select, choices=SP_CHOICES, required=False
     )
+    subscriptions_channel_size = forms.IntegerField(
+        label=False, required=False
+    )
+    subscriptions_live_channel_size = forms.IntegerField(
+        label=False, required=False
+    )
+    subscriptions_shorts_channel_size = forms.IntegerField(
+        label=False, required=False
+    )

tubearchivist/home/src/index/channel.py

@@ -6,46 +6,16 @@ functionality:
 
 import json
 import os
-import re
 from datetime import datetime
 
-import requests
 from home.src.download.thumbnails import ThumbManager
 from home.src.download.yt_dlp_base import YtWrap
 from home.src.es.connect import ElasticWrap, IndexPaginate
 from home.src.index.generic import YouTubeItem
 from home.src.index.playlist import YoutubePlaylist
-from home.src.ta.helper import requests_headers
 from home.src.ta.settings import EnvironmentSettings
 
 
-def banner_extractor(channel_id: str) -> dict[str, str] | None:
-    """workaround for new channel renderer, upstream #9893"""
-    url = f"https://www.youtube.com/channel/{channel_id}?hl=en"
-    cookies = {"SOCS": "CAI"}
-    response = requests.get(
-        url, cookies=cookies, headers=requests_headers(), timeout=30
-    )
-    if not response.ok:
-        return None
-
-    matched_urls = re.findall(
-        r'"(https://yt3.googleusercontent.com/[^"]+=w(\d{3,4})-fcrop64[^"]*)"',
-        response.text,
-    )
-    if not matched_urls:
-        return None
-
-    sorted_urls = sorted(matched_urls, key=lambda x: int(x[1]), reverse=True)
-    banner = sorted_urls[0][0]
-    channel_art_fallback = {
-        "channel_banner_url": banner,
-        "channel_tvart_url": banner.split("-fcrop64")[0],
-    }
-
-    return channel_art_fallback
-
-
 class YoutubeChannel(YouTubeItem):
     """represents a single youtube channel"""
 
@@ -87,7 +57,7 @@ class YoutubeChannel(YouTubeItem):
             "channel_id": self.youtube_id,
             "channel_last_refresh": int(datetime.now().timestamp()),
             "channel_name": self.youtube_meta["uploader"],
-            "channel_subs": self._extract_follower_count(),
+            "channel_subs": self.youtube_meta.get("channel_follower_count", 0),
             "channel_subscribed": False,
             "channel_tags": self._parse_tags(self.youtube_meta.get("tags")),
             "channel_banner_url": self._get_banner_art(),
@@ -95,34 +65,6 @@ class YoutubeChannel(YouTubeItem):
             "channel_tvart_url": self._get_tv_art(),
             "channel_views": self.youtube_meta.get("view_count") or 0,
         }
-        self._inject_fallback()
-
-    def _inject_fallback(self):
-        """fallback channel art work, workaround for upstream #9893"""
-        if self.json_data["channel_banner_url"]:
-            return
-
-        print(f"{self.youtube_id}: attempt art fallback extraction")
-        fallback = banner_extractor(self.youtube_id)
-        if fallback:
-            print(f"{self.youtube_id}: fallback succeeded: {fallback}")
-            self.json_data.update(fallback)
-
-    def _extract_follower_count(self) -> int:
-        """workaround for upstream #9893, extract subs from first video"""
-        subs = self.youtube_meta.get("channel_follower_count")
-        if subs is not None:
-            return subs
-
-        entries = self.youtube_meta.get("entries", [])
-        if entries:
-            first_entry = entries[0]
-            if isinstance(first_entry, dict):
-                subs_entry = first_entry.get("channel_follower_count")
-                if subs_entry is not None:
-                    return subs_entry
-
-        return 0
 
     def _parse_tags(self, tags):
         """parse channel tags"""
@@ -375,23 +317,30 @@ class YoutubeChannel(YouTubeItem):
             "autodelete_days",
             "index_playlists",
             "integrate_sponsorblock",
+            "subscriptions_channel_size",
+            "subscriptions_live_channel_size",
+            "subscriptions_shorts_channel_size",
         ]
 
         to_write = self.json_data.get("channel_overwrites", {})
         for key, value in overwrites.items():
             if key not in valid_keys:
                 raise ValueError(f"invalid overwrite key: {key}")
-            if value == "disable":
+            elif value == "disable":
                 to_write[key] = False
                 continue
-            if value in [0, "0"]:
+            elif value == "0":
                 if key in to_write:
                     del to_write[key]
                 continue
-            if value == "1":
+            elif value == "1":
                 to_write[key] = True
                 continue
-            if value:
+            elif isinstance(value, int) and int(value) < 0:
+                if key in to_write:
+                    del to_write[key]
+                continue
+            elif value is not None and value != "":
                 to_write.update({key: value})
 
         self.json_data["channel_overwrites"] = to_write

tubearchivist/home/src/ta/config_schedule.py

@@ -3,6 +3,9 @@ Functionality:
 - Handle scheduler config update
 """
 
+from datetime import datetime
+
+from django.utils import dateformat
 from django_celery_beat.models import CrontabSchedule
 from home.models import CustomPeriodicTask
 from home.src.ta.config import AppConfig
@@ -64,6 +67,7 @@ class ScheduleBuilder:
         if schedule:
             task_crontab = self.get_set_cron_tab(schedule)
             task.crontab = task_crontab
+            task.last_run_at = dateformat.make_aware(datetime.now())
             task.save()
 
         return task

tubearchivist/home/tasks.py

@@ -7,6 +7,7 @@ Functionality:
 """
 
 from celery import Task, shared_task
+from celery.exceptions import Retry
 from home.src.download.queue import PendingList
 from home.src.download.subscriptions import (
     SubscriptionHandler,
@@ -114,7 +115,13 @@ def update_subscribed(self):
     return None
 
 
-@shared_task(name="download_pending", bind=True, base=BaseTask)
+@shared_task(
+    name="download_pending",
+    bind=True,
+    base=BaseTask,
+    max_retries=3,
+    default_retry_delay=10,
+)
 def download_pending(self, auto_only=False):
     """download latest pending videos"""
     manager = TaskManager()
@@ -124,11 +131,20 @@ def download_pending(self, auto_only=False):
         return None
 
     manager.init(self)
-    downloader = VideoDownloader(task=self)
-    videos_downloaded = downloader.run_queue(auto_only=auto_only)
+    try:
+        downloader = VideoDownloader(task=self)
+        downloaded, failed = downloader.run_queue(auto_only=auto_only)
 
-    if videos_downloaded:
-        return f"downloaded {videos_downloaded} video(s)."
+        if failed:
+            print(f"[task][{self.name}] Videos failed, retry.")
+            self.send_progress("Videos failed, retry.")
+            raise self.retry()
+
+    except Retry as exc:
+        raise exc
+
+    if downloaded:
+        return f"downloaded {downloaded} video(s)."
 
     return None
 

tubearchivist/home/templates/home/channel_id_about.html

@@ -107,16 +107,20 @@
                                 {{ channel_info.channel_overwrites.download_format }}
                             {% else %}
                                 False
-                            {% endif %}</span></p>
+                            {% endif %}</span><br>
+                            Enter "disable" to disable this override.
+                        </p>
                         {{ channel_overwrite_form.download_format }}<br>
                     </div>
                     <div class="overwrite-form-item">
                         <p>Auto delete watched videos after x days: <span class="settings-current">
-                            {% if channel_info.channel_overwrites.autodelete_days %}
+                            {% if channel_info.channel_overwrites.autodelete_days is not None %}
                                 {{ channel_info.channel_overwrites.autodelete_days }}
                             {% else %}
                                 False
-                            {% endif %}</span></p>
+                            {% endif %}</span><br>
+                            Enter a negative number to disable this override.
+                        </p>
                         {{ channel_overwrite_form.autodelete_days }}<br>
                     </div>
                     <div class="overwrite-form-item">
@@ -139,6 +143,39 @@
                             {% endif %}</span></p>
                         {{ channel_overwrite_form.integrate_sponsorblock }}<br>
                     </div>
+                    <h3>Page Size Overrides</h3><br>
+                    <p>Disable standard videos, shorts, or streams for this channel by setting their page size to 0 (zero).</p><br>
+                    <p>Disable page size overwrite for channel by setting to negative value.</p><br>
+                    <div class="overwrite-form-item">
+                        <p>YouTube page size: <span class="settings-current">
+                            {% if channel_info.channel_overwrites.subscriptions_channel_size is not None %}
+                                {{ channel_info.channel_overwrites.subscriptions_channel_size }}
+                            {% else %}
+                                False
+                            {% endif %}</span></p>
+                        <i>Videos to scan to find new items for the <b>Rescan subscriptions</b> task, max recommended 50.</i><br>
+                        {{ channel_overwrite_form.subscriptions_channel_size }}<br>
+                    </div>
+                    <div class="overwrite-form-item">
+                        <p>YouTube Live page size: <span class="settings-current">
+                            {% if channel_info.channel_overwrites.subscriptions_live_channel_size is not None %}
+                                {{ channel_info.channel_overwrites.subscriptions_live_channel_size }}
+                            {% else %}
+                                False
+                            {% endif %}</span></p>
+                        <i>Live Videos to scan to find new items for the <b>Rescan subscriptions</b> task, max recommended 50.</i><br>
+                        {{ channel_overwrite_form.subscriptions_live_channel_size }}<br>
+                    </div>
+                    <div class="overwrite-form-item">
+                        <p>YouTube Shorts page size: <span class="settings-current">
+                            {% if channel_info.channel_overwrites.subscriptions_shorts_channel_size is not None %}
+                                {{ channel_info.channel_overwrites.subscriptions_shorts_channel_size }}
+                            {% else %}
+                                False
+                            {% endif %}</span></p>
+                        <i>Shorts Videos to scan to find new items for the <b>Rescan subscriptions</b> task, max recommended 50.</i><br>
+                        {{ channel_overwrite_form.subscriptions_shorts_channel_size }}<br>
+                    </div><br>
                     <button type="submit">Save Channel Overwrites</button>
                 </form>
             </div>
@@ -146,4 +183,4 @@
     {% endif %}
 </div>
 <script type="text/javascript" src="{% static 'progress.js' %}"></script>
-{% endblock content %}
+{% endblock content %}

tubearchivist/requirements.txt

@@ -1,14 +1,14 @@
-apprise==1.8.0
+apprise==1.8.1
 celery==5.4.0
-Django==5.0.6
+Django==5.0.7
 django-auth-ldap==4.8.0
 django-celery-beat==2.6.0
-django-cors-headers==4.3.1
-djangorestframework==3.15.1
-Pillow==10.3.0
-redis==5.0.4
-requests==2.32.1
+django-cors-headers==4.4.0
+djangorestframework==3.15.2
+Pillow==10.4.0
+redis==5.0.8
+requests==2.32.3
 ryd-client==0.0.6
-uWSGI==2.0.25.1
-whitenoise==6.6.0
-yt-dlp @ git+https://github.com/yt-dlp/yt-dlp@8e15177b4113c355989881e4e030f695a9b59c3a
+uWSGI==2.0.26
+whitenoise==6.7.0
+yt-dlp==2024.8.1