Wide range of fixes, #build

Changed:
- Multi auth backends for LDAP
- Auto restart beat scheduler
- Use timestamp for date published
- Fix playlist videos sort
- Add ignore link on channel page

Dockerfile

@@ -70,6 +70,7 @@ RUN sed -i 's/^user www\-data\;$/user root\;/' /etc/nginx/nginx.conf
 COPY ./backend /app
 COPY ./docker_assets/run.sh /app
 COPY ./docker_assets/backend_start.py /app
+COPY ./docker_assets/beat_auto_spawn.sh /app
 
 COPY --from=node-builder ./frontend/dist /app/static
 

README.md

@@ -1,4 +1,4 @@
-![Tube Archivist](assets/tube-archivist-front.jpg?raw=true "Tube Archivist Banner")  
+![Tube Archivist](assets/tube-archivist-front.jpg?raw=true "Tube Archivist Banner")
 [*more screenshots and video*](SHOWCASE.MD)
 
 <div align="center">
@@ -23,13 +23,13 @@
 ------------------------
 
 ## Core functionality
-Once your YouTube video collection grows, it becomes hard to search and find a specific video. That's where Tube Archivist comes in: By indexing your video collection with metadata from YouTube, you can organize, search and enjoy your archived YouTube videos without hassle offline through a convenient web interface. This includes:  
+Once your YouTube video collection grows, it becomes hard to search and find a specific video. That's where Tube Archivist comes in: By indexing your video collection with metadata from YouTube, you can organize, search and enjoy your archived YouTube videos without hassle offline through a convenient web interface. This includes:
 * Subscribe to your favorite YouTube channels
 * Download Videos using **[yt-dlp](https://github.com/yt-dlp/yt-dlp)**
 * Index and make videos searchable
 * Play videos
 * Keep track of viewed and unviewed videos
-  
+
 ## Resources
 - [Discord](https://www.tubearchivist.com/discord): Connect with us on our Discord server.
 - [r/TubeArchivist](https://www.reddit.com/r/TubeArchivist/): Join our Subreddit.
@@ -38,8 +38,8 @@ Once your YouTube video collection grows, it becomes hard to search and find a s
 - [Plex Plugin](https://github.com/tubearchivist/tubearchivist-plex): Add your videos to Plex
 
 ## Installing
-For minimal system requirements, the Tube Archivist stack needs around 2GB of available memory for a small testing setup and around 4GB of available memory for a mid to large sized installation. Minimal with dual core with 4 threads, better quad core plus. 
+For minimal system requirements, the Tube Archivist stack needs around 2GB of available memory for a small testing setup and around 4GB of available memory for a mid to large sized installation. Minimal with dual core with 4 threads, better quad core plus.
-This project requires docker. Ensure it is installed and running on your system.  
+This project requires docker. Ensure it is installed and running on your system.
 
 The documentation has additional user provided instructions for [Unraid](https://docs.tubearchivist.com/installation/unraid/), [Synology](https://docs.tubearchivist.com/installation/synology/) and [Podman](https://docs.tubearchivist.com/installation/podman/).
 
@@ -49,7 +49,7 @@ Take a look at the example [docker-compose.yml](https://github.com/tubearchivist
 
 All environment variables are explained in detail in the docs [here](https://docs.tubearchivist.com/installation/env-vars/).
 
-**TubeArchivist**:  
+**TubeArchivist**:
 | Environment Var | Value |  |
 | ----------- | ----------- | ----------- |
 | TA_HOST | Server IP or hostname `http://tubearchivist.local:8000` | Required |
@@ -72,8 +72,17 @@ All environment variables are explained in detail in the docs [here](https://doc
 | TA_LDAP | Configure TA to use LDAP Authentication | [Read more](https://docs.tubearchivist.com/configuration/ldap/) |
 | DISABLE_STATIC_AUTH | Remove authentication from media files, (Google Cast...) | [Read more](https://docs.tubearchivist.com/installation/env-vars/#disable_static_auth) |
 | DJANGO_DEBUG | Return additional error messages, for debug only | Optional |
+| TA_LOGIN_AUTH_MODE | Configure the order of login authentication backends (Default: single) | Optional |
 
-**ElasticSearch**  
+| TA_LOGIN_AUTH_MODE value | Description |
+| ------------------------ | ----------- |
+| single                   | Only use a single backend (default, or LDAP, or Forward auth, selected by TA_LDAP or TA_ENABLE_AUTH_PROXY) |
+| local                    | Use local password database only |
+| ldap                     | Use LDAP backend only |
+| forwardauth              | Use reverse proxy headers only |
+| ldap_local               | Use LDAP backend in addition to the local password database |
+
+**ElasticSearch**
 | Environment Var | Value | State |
 | ----------- | ----------- | ----------- |
 | ELASTIC_PASSWORD | Matching password `ELASTIC_PASSWORD` from TubeArchivist | Required |
@@ -81,58 +90,58 @@ All environment variables are explained in detail in the docs [here](https://doc
 
 
 ## Update
-Always use the *latest* (the default) or a named semantic version tag for the docker images. The *unstable* tags are only for your testing environment, there might not be an update path for these testing builds. 
+Always use the *latest* (the default) or a named semantic version tag for the docker images. The *unstable* tags are only for your testing environment, there might not be an update path for these testing builds.
 
-You will see the current version number of **Tube Archivist** in the footer of the interface. There is a daily version check task querying tubearchivist.com, notifying you of any new releases in the footer. To update, you need to update the docker images, the method for which will depend on your platform. For example, if you're using `docker-compose`, run `docker-compose pull` and then restart with `docker-compose up -d`. After updating, check the footer to verify you are running the expected version.  
+You will see the current version number of **Tube Archivist** in the footer of the interface. There is a daily version check task querying tubearchivist.com, notifying you of any new releases in the footer. To update, you need to update the docker images, the method for which will depend on your platform. For example, if you're using `docker-compose`, run `docker-compose pull` and then restart with `docker-compose up -d`. After updating, check the footer to verify you are running the expected version.
 
-  - This project is tested for updates between one or two releases maximum. Further updates back may or may not be supported and you might have to reset your index and configurations to update. Ideally apply new updates at least once per month.  
+  - This project is tested for updates between one or two releases maximum. Further updates back may or may not be supported and you might have to reset your index and configurations to update. Ideally apply new updates at least once per month.
-  - There can be breaking changes between updates, particularly as the application grows, new environment variables or settings might be required for you to set in the your docker-compose file. *Always* check the **release notes**: Any breaking changes will be marked there.  
+  - There can be breaking changes between updates, particularly as the application grows, new environment variables or settings might be required for you to set in the your docker-compose file. *Always* check the **release notes**: Any breaking changes will be marked there.
-  - All testing and development is done with the Elasticsearch version number as mentioned in the provided *docker-compose.yml* file. This will be updated when a new release of Elasticsearch is available. Running an older version of Elasticsearch is most likely not going to result in any issues, but it's still recommended to run the same version as mentioned. Use `bbilly1/tubearchivist-es` to automatically get the recommended version.   
+  - All testing and development is done with the Elasticsearch version number as mentioned in the provided *docker-compose.yml* file. This will be updated when a new release of Elasticsearch is available. Running an older version of Elasticsearch is most likely not going to result in any issues, but it's still recommended to run the same version as mentioned. Use `bbilly1/tubearchivist-es` to automatically get the recommended version.
 
 ## Getting Started
 1. Go through the **settings** page and look at the available options. Particularly set *Download Format* to your desired video quality before downloading. **Tube Archivist** downloads the best available quality by default. To support iOS or MacOS and some other browsers a compatible format must be specified. For example:
 ```
 bestvideo[vcodec*=avc1]+bestaudio[acodec*=mp4a]/mp4
 ```
-2. Subscribe to some of your favorite YouTube channels on the **channels** page. 
+2. Subscribe to some of your favorite YouTube channels on the **channels** page.
 3. On the **downloads** page, click on *Rescan subscriptions* to add videos from the subscribed channels to your Download queue or click on *Add to download queue* to manually add Video IDs, links, channels or playlists.
-4. Click on *Start download* and let **Tube Archivist** to it's thing. 
+4. Click on *Start download* and let **Tube Archivist** to it's thing.
 5. Enjoy your archived collection!
 
 
-### Port Collisions  
+### Port Collisions
-If you have a collision on port `8000`, best solution is to use dockers *HOST_PORT* and *CONTAINER_PORT* distinction: To for example change the interface to port 9000 use `9000:8000` in your docker-compose file.  
+If you have a collision on port `8000`, best solution is to use dockers *HOST_PORT* and *CONTAINER_PORT* distinction: To for example change the interface to port 9000 use `9000:8000` in your docker-compose file.
 
 For more information on port collisions, check the docs.
 
-## Common Errors  
+## Common Errors
-Here is a list of common errors and their solutions.  
+Here is a list of common errors and their solutions.
 
 ### `vm.max_map_count`
 **Elastic Search** in Docker requires the kernel setting of the host machine `vm.max_map_count` to be set to at least 262144.
 
-To temporary set the value run:  
+To temporary set the value run:
 ```
 sudo sysctl -w vm.max_map_count=262144
-```  
+```
-To apply the change permanently depends on your host operating system:  
+To apply the change permanently depends on your host operating system:
 
  - For example on Ubuntu Server add `vm.max_map_count = 262144` to the file `/etc/sysctl.conf`.
- - On Arch based systems create a file `/etc/sysctl.d/max_map_count.conf` with the content `vm.max_map_count = 262144`. 
+ - On Arch based systems create a file `/etc/sysctl.d/max_map_count.conf` with the content `vm.max_map_count = 262144`.
- - On any other platform look up in the documentation on how to pass kernel parameters.  
+ - On any other platform look up in the documentation on how to pass kernel parameters.
 
 
 ### Permissions for elasticsearch
-If you see a message similar to `Unable to access 'path.repo' (/usr/share/elasticsearch/data/snapshot)` or `failed to obtain node locks, tried [/usr/share/elasticsearch/data]` and `maybe these locations are not writable` when initially starting elasticsearch, that probably means the container is not allowed to write files to the volume.  
+If you see a message similar to `Unable to access 'path.repo' (/usr/share/elasticsearch/data/snapshot)` or `failed to obtain node locks, tried [/usr/share/elasticsearch/data]` and `maybe these locations are not writable` when initially starting elasticsearch, that probably means the container is not allowed to write files to the volume.
 To fix that issue, shutdown the container and on your host machine run:
 ```
 chown 1000:0 -R /path/to/mount/point
 ```
-This will match the permissions with the **UID** and **GID** of elasticsearch process within the container and should fix the issue.  
+This will match the permissions with the **UID** and **GID** of elasticsearch process within the container and should fix the issue.
 
 
 ### Disk usage
-The Elasticsearch index will turn to ***read only*** if the disk usage of the container goes above 95% until the usage drops below 90% again, you will see error messages like `disk usage exceeded flood-stage watermark`.  
+The Elasticsearch index will turn to ***read only*** if the disk usage of the container goes above 95% until the usage drops below 90% again, you will see error messages like `disk usage exceeded flood-stage watermark`.
 
 Similar to that, TubeArchivist will become all sorts of messed up when running out of disk space. There are some error messages in the logs when that happens, but it's best to make sure to have enough disk space before starting to download.
 
@@ -142,7 +151,7 @@ If you are seeing errors like `failed to create shim: OCI runtime create failed`
 This can happen if you have nested virtualizations, e.g. LXC running Docker in Proxmox.
 
 ## Known limitations
-- Video files created by Tube Archivist need to be playable in your browser of choice. Not every codec is compatible with every browser and might require some testing with format selection. 
+- Video files created by Tube Archivist need to be playable in your browser of choice. Not every codec is compatible with every browser and might require some testing with format selection.
 - Every limitation of **yt-dlp** will also be present in Tube Archivist. If **yt-dlp** can't download or extract a video for any reason, Tube Archivist won't be able to either.
 - There is no flexibility in naming of the media files.
 
@@ -191,7 +200,7 @@ Implemented:
 - [X] Scan your file system to index already downloaded videos [2021-09-14]
 
 ## User Scripts
-This is a list of useful user scripts, generously created from folks like you to extend this project and its functionality. Make sure to check the respective repository links for detailed license information.  
+This is a list of useful user scripts, generously created from folks like you to extend this project and its functionality. Make sure to check the respective repository links for detailed license information.
 
 This is your time to shine, [read this](https://github.com/tubearchivist/tubearchivist/blob/master/CONTRIBUTING.md#user-scripts) then open a PR to add your script here.
 
@@ -200,13 +209,13 @@ This is your time to shine, [read this](https://github.com/tubearchivist/tubearc
 - [DarkFighterLuke/ta_base_url_nginx](https://gist.github.com/DarkFighterLuke/4561b6bfbf83720493dc59171c58ac36): Set base URL with Nginx when you can't use subdomains.
 - [lamusmaser/ta_migration_helper](https://github.com/lamusmaser/ta_migration_helper): Advanced helper script for migration issues to TubeArchivist v0.4.4 or later.
 - [lamusmaser/create_info_json](https://gist.github.com/lamusmaser/837fb58f73ea0cad784a33497932e0dd): Script to generate `.info.json` files using `ffmpeg` collecting information from downloaded videos.
-- [lamusmaser/ta_fix_for_video_redirection](https://github.com/lamusmaser/ta_fix_for_video_redirection): Script to fix videos that were incorrectly indexed by YouTube's "Video is Unavailable" response. 
+- [lamusmaser/ta_fix_for_video_redirection](https://github.com/lamusmaser/ta_fix_for_video_redirection): Script to fix videos that were incorrectly indexed by YouTube's "Video is Unavailable" response.
 - [RoninTech/ta-helper](https://github.com/RoninTech/ta-helper): Helper script to provide a symlink association to reference TubeArchivist videos with their original titles.
 - [tangyjoust/Tautulli-Notify-TubeArchivist-of-Plex-Watched-State](https://github.com/tangyjoust/Tautulli-Notify-TubeArchivist-of-Plex-Watched-State) Mark videos watched in Plex (through streaming not manually) through Tautulli back to TubeArchivist
 - [Dhs92/delete_shorts](https://github.com/Dhs92/delete_shorts): A script to delete ALL YouTube Shorts from TubeArchivist
 
 ## Donate
-The best donation to **Tube Archivist** is your time, take a look at the [contribution page](CONTRIBUTING.md) to get started.  
+The best donation to **Tube Archivist** is your time, take a look at the [contribution page](CONTRIBUTING.md) to get started.
 Second best way to support the development is to provide for caffeinated beverages:
 * [GitHub Sponsor](https://github.com/sponsors/bbilly1) become a sponsor here on GitHub
 * [Paypal.me](https://paypal.me/bbilly1) for a one time coffee

backend/appsettings/src/index_setup.py

@@ -201,7 +201,15 @@ class ElasitIndexWrap:
         if self.backup_run:
             return
 
-        config = AppConfig().config
+        try:
+            config = AppConfig().config
+        except ValueError:
+            # create defaults in ES if config not found
+            print("AppConfig not found, creating defaults...")
+            handler = AppConfig.__new__(AppConfig)
+            handler.sync_defaults()
+            config = AppConfig.CONFIG_DEFAULTS
+
         if config["application"]["enable_snapshot"]:
             # take snapshot if enabled
             ElasticSnapshot().take_snapshot_now(wait=True)

backend/channel/serializers.py

@@ -90,6 +90,7 @@ class ChannelNavSerializer(serializers.Serializer):
     """serialize channel navigation"""
 
     has_pending = serializers.BooleanField()
+    has_ignored = serializers.BooleanField()
     has_playlists = serializers.BooleanField()
     has_videos = serializers.BooleanField()
     has_streams = serializers.BooleanField()

backend/channel/src/index.py

@@ -14,7 +14,6 @@ from common.src.helper import rand_sleep
 from common.src.index_generic import YouTubeItem
 from download.src.thumbnails import ThumbManager
 from download.src.yt_dlp_base import YtWrap
-from playlist.src.index import YoutubePlaylist
 
 
 class YoutubeChannel(YouTubeItem):
@@ -215,6 +214,8 @@ class YoutubeChannel(YouTubeItem):
 
     def delete_playlists(self):
         """delete all indexed playlist from es"""
+        from playlist.src.index import YoutubePlaylist
+
         all_playlists = self.get_indexed_playlists()
         for playlist in all_playlists:
             YoutubePlaylist(playlist["playlist_id"]).delete_metadata()
@@ -277,6 +278,8 @@ class YoutubeChannel(YouTubeItem):
     @staticmethod
     def _index_single_playlist(playlist):
         """add single playlist if needed"""
+        from playlist.src.index import YoutubePlaylist
+
         playlist = YoutubePlaylist(playlist[0])
         playlist.update_playlist(skip_on_empty=True)
 

backend/channel/src/nav.py

@@ -13,6 +13,7 @@ class ChannelNav:
         """build nav items"""
         nav = {
             "has_pending": self._get_has_pending(),
+            "has_ignored": self._get_has_ignored(),
             "has_playlists": self._get_has_playlists(),
         }
         nav.update(self._get_vid_types())
@@ -63,6 +64,24 @@ class ChannelNav:
 
         return bool(response["hits"]["hits"])
 
+    def _get_has_ignored(self):
+        """Check if there are ignored videos in the download queue"""
+        data = {
+            "size": 1,
+            "query": {
+                "bool": {
+                    "must": [
+                        {"term": {"status": {"value": "ignore"}}},
+                        {"term": {"channel_id": {"value": self.channel_id}}},
+                    ]
+                }
+            },
+            "_source": False,
+        }
+        response, _ = ElasticWrap("ta_download/_search").get(data=data)
+
+        return bool(response["hits"]["hits"])
+
     def _get_has_playlists(self):
         """check if channel has playlists"""
         path = "ta_playlist/_search"

backend/common/src/helper.py

@@ -8,7 +8,7 @@ import os
 import random
 import string
 import subprocess
-from datetime import datetime
+from datetime import datetime, timezone
 from time import sleep
 from typing import Any
 from urllib.parse import urlparse
@@ -106,13 +106,14 @@ def requests_headers() -> dict[str, str]:
 def date_parser(timestamp: int | str) -> str:
     """return formatted date string"""
     if isinstance(timestamp, int):
-        date_obj = datetime.fromtimestamp(timestamp)
+        date_obj = datetime.fromtimestamp(timestamp, tz=timezone.utc)
     elif isinstance(timestamp, str):
         date_obj = datetime.strptime(timestamp, "%Y-%m-%d")
+        date_obj = date_obj.replace(tzinfo=timezone.utc)
     else:
         raise TypeError(f"invalid timestamp: {timestamp}")
 
-    return date_obj.date().isoformat()
+    return date_obj.isoformat()
 
 
 def time_parser(timestamp: str) -> float:

backend/common/tests/test_src/test_helper.py

@@ -22,14 +22,14 @@ def test_randomizor_with_positive_length():
 def test_date_parser_with_int():
     """unix timestamp"""
     timestamp = 1621539600
-    expected_date = "2021-05-20"
+    expected_date = "2021-05-20T19:40:00+00:00"
     assert date_parser(timestamp) == expected_date
 
 
 def test_date_parser_with_str():
     """iso timestamp"""
     date_str = "2021-05-21"
-    expected_date = "2021-05-21"
+    expected_date = "2021-05-21T00:00:00+00:00"
     assert date_parser(date_str) == expected_date
 
 

backend/config/management/commands/ta_connection.py

@@ -99,7 +99,12 @@ class Command(BaseCommand):
                 continue
 
             if status_code and status_code == 200:
-                path = "_cluster/health?wait_for_status=yellow&timeout=60s"
+                path = (
+                    "_cluster/health?"
+                    "wait_for_status=yellow&"
+                    "timeout=60s&"
+                    "wait_for_active_shards=1"
+                )
                 _, _ = ElasticWrap(path).get(timeout=60)
                 self.stdout.write(
                     self.style.SUCCESS("    ✓ ES connection established")

backend/config/management/commands/ta_startup.py

@@ -263,7 +263,7 @@ class Command(BaseCommand):
     def _init_app_config(self) -> None:
         """init default app config to ES"""
         self.stdout.write("[10] Check AppConfig")
-        _, status_code = ElasticWrap("ta_config/_doc/appsettings").get()
+        response, status_code = ElasticWrap("ta_config/_doc/appsettings").get()
         if status_code in [200, 201]:
             self.stdout.write(
                 self.style.SUCCESS("    skip completed appsettings init")
@@ -276,6 +276,13 @@ class Command(BaseCommand):
 
             return
 
+        if status_code != 404:
+            message = "    🗙 ta_config index lookup failed"
+            self.stdout.write(self.style.ERROR(message))
+            self.stdout.write(response)
+            sleep(60)
+            raise CommandError(message)
+
         handler = AppConfig.__new__(AppConfig)
         _, status_code = handler.sync_defaults()
         self.stdout.write(

backend/config/settings.py

@@ -195,7 +195,6 @@ if bool(environ.get("TA_LDAP")):
             ldap.OPT_X_TLS_REQUIRE_CERT: ldap.OPT_X_TLS_NEVER,
         }
 
-    AUTHENTICATION_BACKENDS = ("django_auth_ldap.backend.LDAPBackend",)
 
 # Database
 # https://docs.djangoproject.com/en/3.2/ref/settings/#databases
@@ -239,10 +238,31 @@ if bool(environ.get("TA_ENABLE_AUTH_PROXY")):
 
     MIDDLEWARE.append("user.src.remote_user_auth.HttpRemoteUserMiddleware")
 
+
+# Configure Authentication Backend Combinations
+_login_auth_mode = (environ.get("TA_LOGIN_AUTH_MODE") or "single").casefold()
+if _login_auth_mode == "local":
+    AUTHENTICATION_BACKENDS = ("django.contrib.auth.backends.ModelBackend",)
+elif _login_auth_mode == "ldap":
+    AUTHENTICATION_BACKENDS = ("django_auth_ldap.backend.LDAPBackend",)
+elif _login_auth_mode == "forwardauth":
     AUTHENTICATION_BACKENDS = (
         "django.contrib.auth.backends.RemoteUserBackend",
     )
-
+elif _login_auth_mode == "ldap_local":
+    AUTHENTICATION_BACKENDS = (
+        "django_auth_ldap.backend.LDAPBackend",
+        "django.contrib.auth.backends.ModelBackend",
+    )
+else:
+    # If none of these cases match, AUTHENTICATION_BACKENDS is unset, which
+    # means the ModelBackend should be used by default
+    if bool(environ.get("TA_LDAP")):
+        AUTHENTICATION_BACKENDS = ("django_auth_ldap.backend.LDAPBackend",)
+    if bool(environ.get("TA_ENABLE_AUTH_PROXY")):
+        AUTHENTICATION_BACKENDS = (
+            "django.contrib.auth.backends.RemoteUserBackend",
+        )
 
 # Internationalization
 # https://docs.djangoproject.com/en/3.2/topics/i18n/

backend/download/src/queue.py

@@ -167,14 +167,14 @@ class PendingList(PendingIndex):
         self.to_skip = False
         self.missing_videos = False
 
-    def parse_url_list(self):
+    def parse_url_list(self, auto_start=False):
         """extract youtube ids from list"""
         self.missing_videos = []
         self.get_download()
         self.get_indexed()
         total = len(self.youtube_ids)
         for idx, entry in enumerate(self.youtube_ids):
-            self._process_entry(entry)
+            self._process_entry(entry, auto_start=auto_start)
             if not self.task:
                 continue
 
@@ -183,11 +183,11 @@ class PendingList(PendingIndex):
                 progress=(idx + 1) / total,
             )
 
-    def _process_entry(self, entry):
+    def _process_entry(self, entry, auto_start=False):
         """process single entry from url list"""
         vid_type = self._get_vid_type(entry)
         if entry["type"] == "video":
-            self._add_video(entry["url"], vid_type)
+            self._add_video(entry["url"], vid_type, auto_start=auto_start)
         elif entry["type"] == "channel":
             self._parse_channel(entry["url"], vid_type)
         elif entry["type"] == "playlist":
@@ -204,8 +204,14 @@ class PendingList(PendingIndex):
 
         return VideoTypeEnum(vid_type_str)
 
-    def _add_video(self, url, vid_type):
+    def _add_video(self, url, vid_type, auto_start=False):
         """add video to list"""
+        if auto_start and url in set(
+            i["youtube_id"] for i in self.all_pending
+        ):
+            PendingInteract(youtube_id=url, status="priority").update_status()
+            return
+
         if url not in self.missing_videos and url not in self.to_skip:
             self.missing_videos.append((url, vid_type))
         else:
@@ -330,9 +336,6 @@ class PendingList(PendingIndex):
     def _parse_youtube_details(self, vid, vid_type=VideoTypeEnum.VIDEOS):
         """parse response"""
         vid_id = vid.get("id")
-        published = datetime.strptime(vid["upload_date"], "%Y%m%d").strftime(
-            "%Y-%m-%d"
-        )
 
         # build dict
         youtube_details = {
@@ -342,10 +345,23 @@ class PendingList(PendingIndex):
             "title": vid["title"],
             "channel_id": vid["channel_id"],
             "duration": get_duration_str(vid["duration"]),
-            "published": published,
+            "published": self._build_published(vid),
             "timestamp": int(datetime.now().timestamp()),
             "vid_type": vid_type.value,
             "channel_indexed": vid["channel_id"] in self.all_channels,
         }
 
         return youtube_details
+
+    @staticmethod
+    def _build_published(vid):
+        """build published date or timestamp"""
+        timestamp = vid["timestamp"]
+        if timestamp:
+            return timestamp
+
+        upload_date = vid["upload_date"]
+        upload_date_time = datetime.strptime(upload_date, "%Y%m%d")
+        published = upload_date_time.strftime("%Y-%m-%d")
+
+        return published

backend/download/src/yt_dlp_base.py

@@ -20,7 +20,6 @@ class YtWrap:
     OBS_BASE = {
         "default_search": "ytsearch",
         "quiet": True,
-        "check_formats": "selected",
         "socket_timeout": 10,
         "extractor_retries": 3,
         "retries": 10,
@@ -66,6 +65,7 @@ class YtWrap:
 
     def download(self, url):
         """make download request"""
+        self.obs.update({"check_formats": "selected"})
         with yt_dlp.YoutubeDL(self.obs) as ydl:
             try:
                 ydl.download([url])

backend/playlist/src/index.py

@@ -7,7 +7,6 @@ functionality:
 import json
 from datetime import datetime
 
-from channel.src import index as channel
 from common.src.env_settings import EnvironmentSettings
 from common.src.es_connect import ElasticWrap, IndexPaginate
 from common.src.index_generic import YouTubeItem
@@ -74,8 +73,10 @@ class YoutubePlaylist(YouTubeItem):
 
     def _ensure_channel(self):
         """make sure channel is indexed"""
+        from channel.src.index import YoutubeChannel
+
         channel_id = self.json_data["playlist_channel_id"]
-        channel_handler = channel.YoutubeChannel(channel_id)
+        channel_handler = YoutubeChannel(channel_id)
         channel_handler.build_json(upload=True)
 
     def get_local_vids(self) -> list[str]:

backend/task/tasks.py

@@ -156,7 +156,7 @@ def extrac_dl(self, youtube_ids, auto_start=False, status="pending"):
         to_add = youtube_ids
 
     pending_handler = PendingList(youtube_ids=to_add, task=self)
-    pending_handler.parse_url_list()
+    pending_handler.parse_url_list(auto_start=auto_start)
     videos_added = pending_handler.add_to_pending(
         status=status, auto_start=auto_start
     )

backend/video/serializers.py

@@ -12,6 +12,7 @@ class PlayerSerializer(serializers.Serializer):
     """serialize player"""
 
     watched = serializers.BooleanField()
+    watched_date = serializers.IntegerField(required=False)
     duration = serializers.IntegerField()
     duration_str = serializers.CharField()
     progress = serializers.FloatField(required=False)

backend/video/src/index.py

@@ -173,9 +173,6 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
         self._validate_id()
         # extract
         self.channel_id = self.youtube_meta["channel_id"]
-        upload_date = self.youtube_meta["upload_date"]
-        upload_date_time = datetime.strptime(upload_date, "%Y%m%d")
-        published = upload_date_time.strftime("%Y-%m-%d")
         last_refresh = int(datetime.now().timestamp())
         # base64_blur = ThumbManager().get_base64_blur(self.youtube_id)
         base64_blur = False
@@ -187,7 +184,7 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
             "vid_thumb_url": self.youtube_meta["thumbnail"],
             "vid_thumb_base64": base64_blur,
             "tags": self.youtube_meta.get("tags", []),
-            "published": published,
+            "published": self._build_published(),
             "vid_last_refresh": last_refresh,
             "date_downloaded": last_refresh,
             "youtube_id": self.youtube_id,
@@ -196,6 +193,18 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
             "active": True,
         }
 
+    def _build_published(self):
+        """build published date or timestamp"""
+        timestamp = self.youtube_meta["timestamp"]
+        if timestamp:
+            return timestamp
+
+        upload_date = self.youtube_meta["upload_date"]
+        upload_date_time = datetime.strptime(upload_date, "%Y%m%d")
+        published = upload_date_time.strftime("%Y-%m-%d")
+
+        return published
+
     def _validate_id(self):
         """validate expected video ID, raise value error on mismatch"""
         remote_id = self.youtube_meta["id"]

backend/video/src/query_building.py

@@ -1,6 +1,7 @@
 """build query for video fetching"""
 
 from common.src.ta_redis import RedisArchivist
+from playlist.src.index import YoutubePlaylist
 from video.src.constants import OrderEnum, SortEnum, VideoTypeEnum
 
 
@@ -84,6 +85,11 @@ class QueryBuilder:
 
     def parse_sort(self) -> dict | None:
         """build sort key"""
+        playlist = self.request_params.get("playlist")
+        if playlist:
+            # overwrite sort based on idx in playlist
+            return self._get_playlist_sort(playlist_id=playlist)
+
         sort = self.request_params.get("sort")
         if not sort:
             return None
@@ -100,3 +106,39 @@ class QueryBuilder:
         order_by = getattr(OrderEnum, order.upper()).value
 
         return {"sort": [{sort_field: {"order": order_by}}]}
+
+    def _get_playlist_sort(self, playlist_id: str):
+        """get sort for playlist"""
+        playlist = YoutubePlaylist(playlist_id)
+        playlist.get_from_es()
+        if not playlist.json_data:
+            raise ValueError(f"playlist {playlist_id} not found")
+
+        sort_score = {
+            i["youtube_id"]: i["idx"]
+            for i in playlist.json_data["playlist_entries"]
+            if i["downloaded"]
+        }
+        script = (
+            "if(params.scores.containsKey(doc['youtube_id'].value)) "
+            + "{return params.scores[doc['youtube_id'].value];} "
+            + "return 100000;"
+        )
+
+        sort = {
+            "sort": [
+                {
+                    "_script": {
+                        "type": "number",
+                        "script": {
+                            "lang": "painless",
+                            "source": script,
+                            "params": {"scores": sort_score},
+                        },
+                        "order": "asc",
+                    }
+                }
+            ],
+        }
+
+        return sort

backend/video/tests/test_src/test_query_building.py

@@ -16,7 +16,6 @@ def test_build_data():
     qb = QueryBuilder(
         user_id=1,
         channel="test_channel",
-        playlist="test_playlist",
         watch="watched",
         type="videos",
         sort="published",

docker_assets/beat_auto_spawn.sh

@@ -0,0 +1,32 @@
+#!/bin/bash
+# auto restart beat scheduler
+# https://github.com/celery/django-celery-beat/issues/894
+
+if [[ -n "$DJANGO_DEBUG" ]]; then
+  LOGLEVEL="DEBUG"
+else
+  LOGLEVEL="INFO"
+fi
+
+COMMAND="celery -A task beat --loglevel=$LOGLEVEL --scheduler django_celery_beat.schedulers:DatabaseScheduler"
+TIMEOUT=3600
+
+while true; do
+    echo "Starting process beat scheduler"
+
+    $COMMAND &
+    PID=$!
+
+    sleep $TIMEOUT
+
+    # Kill the process if still running
+    if kill -0 $PID 2>/dev/null; then
+        echo "Killing beat process after $TIMEOUT seconds"
+        kill $PID
+        # Wait a bit to allow graceful shutdown, then force kill if needed
+        sleep 10
+        kill -9 $PID 2>/dev/null
+    fi
+
+    echo "Restarting beat..."
+done

docker_assets/run.sh

@@ -3,6 +3,12 @@
 
 set -e
 
+if [[ -n "$DJANGO_DEBUG" ]]; then
+  LOGLEVEL="DEBUG"
+else
+  LOGLEVEL="INFO"
+fi
+
 # stop on pending manual migration
 python manage.py ta_stop_on_error
 
@@ -18,10 +24,11 @@ python manage.py ta_startup
 # start all tasks
 nginx &
 celery -A task.celery worker \
-    --loglevel=INFO \
+    --loglevel=$LOGLEVEL \
     --concurrency 4 \
     --max-tasks-per-child 5 \
     --max-memory-per-child 150000 &
-celery -A task beat --loglevel=INFO \
+
-    --scheduler django_celery_beat.schedulers:DatabaseScheduler &
+./beat_auto_spawn.sh &
+
 python backend_start.py

frontend/package.json

@@ -11,26 +11,26 @@
     "preview": "vite preview"
   },
   "dependencies": {
-    "dompurify": "^3.2.5",
+    "dompurify": "^3.2.6",
     "react": "^19.1.0",
     "react-dom": "^19.1.0",
     "react-router-dom": "^7.6.0",
-    "zustand": "^5.0.4"
+    "zustand": "^5.0.5"
   },
   "devDependencies": {
-    "@types/react": "^19.1.3",
+    "@types/react": "^19.1.5",
-    "@types/react-dom": "^19.1.3",
+    "@types/react-dom": "^19.1.5",
-    "@typescript-eslint/eslint-plugin": "^8.32.0",
+    "@typescript-eslint/eslint-plugin": "^8.32.1",
-    "@typescript-eslint/parser": "^8.32.0",
+    "@typescript-eslint/parser": "^8.32.1",
-    "@vitejs/plugin-react-swc": "^3.9.0",
+    "@vitejs/plugin-react-swc": "^3.10.0",
-    "eslint": "^9.26.0",
+    "eslint": "^9.27.0",
     "eslint-config-prettier": "^10.1.5",
     "eslint-plugin-react-hooks": "^5.2.0",
     "eslint-plugin-react-refresh": "^0.4.20",
     "globals": "^16.1.0",
     "prettier": "3.5.3",
     "typescript": "^5.8.3",
-    "typescript-eslint": "^8.32.0",
+    "typescript-eslint": "^8.32.1",
     "vite": ">=6.3.5",
     "vite-plugin-checker": "^0.9.3"
   }

frontend/src/api/actions/updateUserConfig.ts

@@ -8,6 +8,14 @@ export type ColourVariants =
   | 'midnight.css'
   | 'custom.css';
 
+export const ColourConstant = {
+  Dark: 'dark.css',
+  Light: 'light.css',
+  Matrix: 'matrix.css',
+  Midnight: 'midnight.css',
+  Custom: 'custom.css',
+};
+
 export const FileSizeUnits = {
   Binary: 'binary',
   Metric: 'metric',

frontend/src/api/loader/loadChannelNav.ts

@@ -5,6 +5,7 @@ export type ChannelNavResponseType = {
   has_shorts: boolean;
   has_playlists: boolean;
   has_pending: boolean;
+  has_ignored: boolean;
 };
 
 const loadChannelNav = async (youtubeChannelId: string) => {

frontend/src/components/VideoPlayer.tsx

@@ -150,6 +150,7 @@ const VideoPlayer = ({
   const resetPlaybackSpeedPressed = useKeyPress('=');
   const arrowRightPressed = useKeyPress('ArrowRight');
   const arrowLeftPressed = useKeyPress('ArrowLeft');
+  const pPausedPressed = useKeyPress('p');
 
   const videoId = video.youtube_id;
   const videoUrl = video.media_url;
@@ -214,6 +215,16 @@ const VideoPlayer = ({
     // eslint-disable-next-line react-hooks/exhaustive-deps
   }, [mutePressed]);
 
+  useEffect(() => {
+    if (pPausedPressed) {
+      if (videoRef.current?.paused) {
+        videoRef.current.play();
+      } else {
+        videoRef.current?.pause();
+      }
+    }
+  }, [pPausedPressed]);
+
   useEffect(() => {
     if (increasePlaybackSpeedPressed) {
       const newSpeed = playbackSpeedIndex + 1;
@@ -301,16 +312,25 @@ const VideoPlayer = ({
   }, [subtitlesPressed]);
 
   useEffect(() => {
-    if (arrowLeftPressed) {
+    if (arrowLeftPressed || arrowRightPressed) {
-      infoDialog('- 5 seconds');
+      let timeStep = 5;
-    }
-  }, [arrowLeftPressed]);
 
-  useEffect(() => {
+      if (arrowLeftPressed) {
-    if (arrowRightPressed) {
+        infoDialog('- 5 seconds');
-      infoDialog('+ 5 seconds');
+        timeStep *= -1;
+      }
+
+      if (arrowRightPressed) {
+        infoDialog('+ 5 seconds');
+      }
+
+      const currentCurrentTime = videoRef.current?.currentTime;
+
+      if (currentCurrentTime !== undefined && videoRef.current) {
+        videoRef.current.currentTime = currentCurrentTime + timeStep;
+      }
     }
-  }, [arrowRightPressed]);
+  }, [arrowLeftPressed, arrowRightPressed]);
 
   useEffect(() => {
     if (questionmarkPressed) {
@@ -361,6 +381,11 @@ const VideoPlayer = ({
               });
             }}
             onEnded={handleVideoEnd(videoId, watched)}
+            onKeyDown={e => {
+              if (e.key === 'ArrowLeft' || e.key === 'ArrowRight') {
+                e.preventDefault();
+              }
+            }}
             autoPlay={autoplay}
             controls
             width="100%"
@@ -386,6 +411,10 @@ const VideoPlayer = ({
                 <td>Show help</td>
                 <td>?</td>
               </tr>
+              <tr>
+                <td>Toggle pause play</td>
+                <td>p</td>
+              </tr>
               <tr>
                 <td>Toggle mute</td>
                 <td>m</td>

frontend/src/configuration/colours/Colours.tsx

@@ -1,14 +1,40 @@
+import { ColourConstant, ColourVariants } from '../../api/actions/updateUserConfig';
 import { useUserConfigStore } from '../../stores/UserConfigStore';
-import { ColourConstant } from './colourConstant';
 import CustomStylesheet from './components/Custom';
 import DarkStylesheet from './components/Dark';
 import LightStylesheet from './components/Light';
 import MatrixStylesheet from './components/Matrix';
 import MidnightStylesheet from './components/Midnight';
 
+function getThemeFromLocalStorage(stylesheet: string): ColourVariants {
+  // Check when localStorage when its possibly the default theme. ( e.g. login page )
+  if (stylesheet === ColourConstant.Dark) {
+    const fromLocalStorage = localStorage.getItem('stylesheet');
+
+    if (!fromLocalStorage) {
+      localStorage.setItem('stylesheet', stylesheet);
+    }
+
+    if (fromLocalStorage) {
+      stylesheet = fromLocalStorage;
+    }
+  } else {
+    const fromLocalStorage = localStorage.getItem('stylesheet');
+
+    // Re-sync when localStorage is not the same as in userConfig
+    if (stylesheet !== fromLocalStorage) {
+      localStorage.setItem('stylesheet', stylesheet);
+    }
+  }
+
+  return stylesheet as ColourVariants;
+}
+
 const Colours = () => {
   const { userConfig } = useUserConfigStore();
-  const stylesheet = userConfig?.stylesheet;
+  let stylesheet = userConfig?.stylesheet;
+
+  stylesheet = getThemeFromLocalStorage(stylesheet);
 
   switch (stylesheet) {
     case ColourConstant.Dark:

frontend/src/configuration/colours/colourConstant.ts

@@ -1,7 +0,0 @@
-export const ColourConstant = {
-  Dark: 'dark.css',
-  Light: 'light.css',
-  Matrix: 'matrix.css',
-  Midnight: 'midnight.css',
-  Custom: 'custom.css',
-};

frontend/src/configuration/routes/RouteList.ts

@@ -10,7 +10,8 @@ const Routes = {
   Playlists: '/playlist/',
   Playlist: (id: string) => `/playlist/${id}`,
   Downloads: '/downloads/',
-  DownloadsByChannelId: (channelId: string) => `/downloads/?channel=${channelId}`,
+  DownloadsByChannelId: (channelId: string) => `/downloads/?channel=${channelId}&ignored=false`,
+  IgnoredByChannelId: (channelId: string) => `/downloads/?channel=${channelId}&ignored=true`,
   Search: '/search/',
   SettingsDashboard: '/settings/',
   SettingsUser: '/settings/user/',

frontend/src/pages/ChannelBase.tsx

@@ -27,7 +27,7 @@ const ChannelBase = () => {
   const { data: channelNavData } = channelNav ?? {};
 
   const channel = channelResponseData;
-  const { has_streams, has_shorts, has_playlists, has_pending } = channelNavData || {};
+  const { has_streams, has_shorts, has_playlists, has_pending, has_ignored } = channelNavData || {};
 
   useEffect(() => {
     (async () => {
@@ -84,6 +84,11 @@ const ChannelBase = () => {
               <h3>Downloads</h3>
             </Link>
           )}
+          {has_ignored && isAdmin && (
+            <Link to={Routes.IgnoredByChannelId(channelId)}>
+              <h3>Ignored</h3>
+            </Link>
+          )}
         </div>
 
         <Notifications

frontend/src/pages/Download.tsx

@@ -52,6 +52,7 @@ const Download = () => {
   const { currentPage, setCurrentPage } = useOutletContext() as OutletContextType;
 
   const channelFilterFromUrl = searchParams.get('channel');
+  const ignoredOnlyParam = searchParams.get('ignored');
 
   const [refresh, setRefresh] = useState(false);
   const [showHiddenForm, setShowHiddenForm] = useState(false);
@@ -82,7 +83,8 @@ const Download = () => {
 
   const view = userConfig.view_style_downloads;
   const gridItems = userConfig.grid_items;
-  const showIgnored = userConfig.show_ignored_only;
+  const showIgnored =
+    ignoredOnlyParam !== null ? ignoredOnlyParam === 'true' : userConfig.show_ignored_only;
   const isGridView = view === ViewStyles.grid;
   const gridView = isGridView ? `boxed-${gridItems}` : '';
   const gridViewGrid = isGridView ? `grid-${gridItems}` : '';
@@ -234,6 +236,9 @@ const Download = () => {
                 id="showIgnored"
                 onChange={() => {
                   handleUserConfigUpdate({ show_ignored_only: !showIgnored });
+                  const newParams = new URLSearchParams(searchParams.toString());
+                  newParams.set('ignored', String(!showIgnored));
+                  setSearchParams(newParams);
                   setRefresh(true);
                 }}
                 type="checkbox"

frontend/src/pages/SettingsActions.tsx

@@ -204,8 +204,8 @@ const SettingsActions = () => {
             deleted videos from the filesystem.
           </p>
           <p>
-            Rescan your media folder looking for missing videos and clean up index. More infos on
+            Rescan your media folder looking for missing videos and clean up index. More info on the
-            the Github{' '}
+            Github{' '}
             <a
               href="https://docs.tubearchivist.com/settings/actions/#rescan-filesystem"
               target="_blank"

frontend/src/pages/SettingsUser.tsx

@@ -1,4 +1,5 @@
 import updateUserConfig, {
+  ColourConstant,
   ColourVariants,
   FileSizeUnits,
   UserConfigType,
@@ -10,7 +11,6 @@ import useIsAdmin from '../functions/useIsAdmin';
 import { useUserConfigStore } from '../stores/UserConfigStore';
 import { useEffect, useState } from 'react';
 import ToggleConfig from '../components/ToggleConfig';
-import { ColourConstant } from '../configuration/colours/colourConstant';
 
 const SettingsUser = () => {
   const { userConfig, setUserConfig } = useUserConfigStore();
@@ -38,6 +38,9 @@ const SettingsUser = () => {
   const handleStyleSheetChange = async (selectedStyleSheet: ColourVariants) => {
     handleUserConfigUpdate({ stylesheet: selectedStyleSheet });
     setStyleSheet(selectedStyleSheet);
+
+    // Store in local storage for pages like login, without a userConfig
+    localStorage.setItem('stylesheet', selectedStyleSheet);
   };
 
   const handlePageSizeChange = async () => {