bump TA_VERSION

Changed:
- Fixed not loading new default configs at expected time
- Better startup error handling

.github/ISSUE_TEMPLATE/FEATURE-REQUEST.yml

@@ -1,12 +1,12 @@
 name: Feature Request
-description: Create a new feature request
+description: I want to work on a new feature.
 title: "[Feature Request]: "
 
 body:
   - type: markdown
     attributes:
       value: |
-        Thanks for taking the time to help improve this project!
+        Thanks for taking the time to help improve this project! This project is currently only taking new feature requests from contributors who want to make a PR with their ideas. The backlog of planned features outlined in the roadmap or the open issues here will keep us busy for the foreseeable future.
 
   - type: checkboxes
     id: already
@@ -33,5 +33,5 @@ body:
       label: Your help is needed!
       description: This project is ambitious as it is, please contribute.
       options:
-        - label: Yes I can help with this feature request!
-          required: false
+        - label: Yes I will work on this in the next few days or weeks.
+          required: true

README.md

@@ -66,10 +66,10 @@ There's dedicated user-contributed install steps under [docs/Installation.md](./
 
 For minimal system requirements, the Tube Archivist stack needs around 2GB of available memory for a small testing setup and around 4GB of available memory for a mid to large sized installation. Minimal with dual core with 4 threads, better quad core plus.  
 
-Note for arm64 hosts: The Tube Archivist container is multi arch, so is Elasticsearch. RedisJSON doesn't offer arm builds, but you can use the image `bbilly1/rejson`, an unofficial rebuild for arm64.
-
 This project requires docker. Ensure it is installed and running on your system.
 
+Note for **arm64**: Tube Archivist is a multi arch container, same for redis. For Elasitc Search use the official image for arm64 support. Other architectures are not supported.
+
 Save the [docker-compose.yml](./docker-compose.yml) file from this reposity somewhere permanent on your system, keeping it named `docker-compose.yml`. You'll need to refer to it whenever starting this application.
 
 Edit the following values from that file:
@@ -153,12 +153,20 @@ Wildcards "*" can not be used for the Access-Control-Allow-Origin header. If the
 
 Use `bbilly1/tubearchivist-es` to automatically get the recommended version, or use the official image with the version tag in the docker-compose file.
 
+Use official Elastic Search for **arm64**.  
+
 Stores video meta data and makes everything searchable. Also keeps track of the download queue.
   - Needs to be accessible over the default port `9200`
   - Needs a volume at **/usr/share/elasticsearch/data** to store data
 
 Follow the [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html) for additional installation details.
 
+### Elasticsearch on a custom port
+Should you need to change the port for Elasticsearch to for example *9500*, follow these steps:
+- Set the environment variable `http.port=9500` to the ES container
+- Change the *expose* value for the ES container to match your port number
+- For the Tube Archivist container, change the *ES_URL* environment variable, e.g. `ES_URL=http://archivist-es:9500`
+
 ### Redis JSON
 Functions as a cache and temporary link between the application and the file system. Used to store and display messages and configuration variables.
   - Needs to be accessible over the default port `6379`
@@ -229,7 +237,7 @@ We have come far, nonetheless we are not short of ideas on how to improve and ex
 - [ ] Show total video downloaded vs total videos available in channel
 - [ ] Add statistics of index
 - [ ] Download speed schedule ([#198](https://github.com/tubearchivist/tubearchivist/issues/198))
-- [ ] Auto ignore videos by keyword ([#163](https://github.com/tubearchivist/tubearchivist/issues/163))
+- [ ] Download or Ignore videos by keyword ([#163](https://github.com/tubearchivist/tubearchivist/issues/163))
 - [ ] Custom searchable notes to videos, channels, playlists ([#144](https://github.com/tubearchivist/tubearchivist/issues/144))
 
 Implemented:
@@ -271,6 +279,18 @@ Second best way to support the development is to provide for caffeinated beverag
 * [ko-fi.com](https://ko-fi.com/bbilly1) for an alternative platform
 
 
+## Notable mentions
+This is a selection of places where this project has been featured on reddit, in the news, blogs or any other online media. Sorted by newest on the top.
+* **console.substack.com**: Interview With Simon of Tube Archivist, [2023-01-29] [[link](https://console.substack.com/p/console-142#%C2%A7interview-with-simon-of-tube-archivist)]
+* **reddit.com**: Tube Archivist v0.3.0 - Now Archiving Comments, [2022-12-02] [[link](https://www.reddit.com/r/selfhosted/comments/zaonzp/tube_archivist_v030_now_archiving_comments/)]
+* **reddit.com**: Tube Archivist v0.2 - Now with Full Text Search, [2022-07-24] [[link](https://www.reddit.com/r/selfhosted/comments/w6jfa1/tube_archivist_v02_now_with_full_text_search/)]
+* **noted.lol**: How I Control What Media My Kids Watch Using Tube Archivist, [2022-03-27] [[link](https://noted.lol/how-i-control-what-media-my-kids-watch-using-tube-archivist/)]
+* **thehomelab.wiki**: Tube Archivist - A Youtube-DL Alternative on Steroids, [2022-01-27] [[link](https://thehomelab.wiki/books/news/page/tube-archivist-a-youtube-dl-alternative-on-steroids)]
+* **reddit.com**: Celebrating TubeArchivist v0.1, [2022-01-09] [[link](https://www.reddit.com/r/selfhosted/comments/rzh084/celebrating_tubearchivist_v01/)]
+* **linuxunplugged.com**: Pick: tubearchivist — Your self-hosted YouTube media server, [2021-09-11] [[link](https://linuxunplugged.com/425)] and [2021-10-05] [[link](https://linuxunplugged.com/426)]
+* **reddit.com**: Introducing Tube Archivist, your self hosted Youtube media server, [2021-09-12] [[link](https://www.reddit.com/r/selfhosted/comments/pmj07b/introducing_tube_archivist_your_self_hosted/)]
+
+
 ## Sponsor
 Big thank you to [Digitalocean](https://www.digitalocean.com/) for generously donating credit for the tubearchivist.com VPS and buildserver. 
 <p>

deploy.sh

@@ -85,14 +85,14 @@ function validate {
     # note: this logic is duplicated in the `./github/workflows/lint_python.yml` config
     # if you update this file, you should update that as well
     echo "running black"
-    black --diff --color --check -l 79 "$check_path"
+    black --exclude "migrations/*" --diff --color --check -l 79 "$check_path"
     echo "running codespell"
     codespell --skip="./.git,./package.json,./package-lock.json,./node_modules" "$check_path"
     echo "running flake8"
-    flake8 "$check_path" --count --max-complexity=10 --max-line-length=79 \
-        --show-source --statistics
+    flake8 "$check_path" --exclude "migrations" --count --max-complexity=10 \
+        --max-line-length=79 --show-source --statistics
     echo "running isort"
-    isort --check-only --diff --profile black -l 79 "$check_path"
+    isort --skip "migrations" --check-only --diff --profile black -l 79 "$check_path"
     printf "    \n> all validations passed\n"
 
 }

docker-compose.yml

@@ -24,7 +24,7 @@ services:
       - archivist-es
       - archivist-redis
   archivist-redis:
-    image: redislabs/rejson                 # for arm64 use bbilly1/rejson
+    image: redis/redis-stack-server
     container_name: archivist-redis
     restart: unless-stopped
     expose:

docker_assets/run.sh

@@ -1,71 +1,21 @@
 #!/bin/bash
 # startup script inside the container for tubearchivist
 
-if [[ -z "$ELASTIC_USER" ]]; then
-    export ELASTIC_USER=elastic
-fi
+set -e
 
-cachedir=/cache
-[[ -d $cachedir ]] || cachedir=.
-lockfile=${cachedir}/initsu.lock
-
-required="Missing required environment variable"
-[[ -f $lockfile ]] || : "${TA_USERNAME:?$required}"
-: "${TA_PASSWORD:?$required}"
-: "${ELASTIC_PASSWORD:?$required}"
-: "${TA_HOST:?$required}"
-
-# ugly nginx and uwsgi port overwrite with env vars
-if [[ -n "$TA_PORT" ]]; then
-    sed -i "s/8000/$TA_PORT/g" /etc/nginx/sites-available/default
-fi
-
-if [[ -n "$TA_UWSGI_PORT" ]]; then
-    sed -i "s/8080/$TA_UWSGI_PORT/g" /etc/nginx/sites-available/default
-    sed -i "s/8080/$TA_UWSGI_PORT/g" /app/uwsgi.ini
-fi
-
-# disable auth on static files for cast support
-if [[ -n "$ENABLE_CAST" ]]; then
-    sed -i "/auth_request/d" /etc/nginx/sites-available/default
-fi
-
-# wait for elasticsearch
-counter=0
-until curl -u "$ELASTIC_USER":"$ELASTIC_PASSWORD" "$ES_URL" -fs; do
-    echo "waiting for elastic search to start"
-    counter=$((counter+1))
-    if [[ $counter -eq 12 ]]; then
-        # fail after 2 min
-        echo "failed to connect to elastic search, exiting..."
-        curl -v -u "$ELASTIC_USER":"$ELASTIC_PASSWORD" "$ES_URL"?pretty
-        exit 1
-    fi
-    sleep 10
-done
-
-# start python application
-python manage.py makemigrations
+# django setup
 python manage.py migrate
 
-if [[ -f $lockfile ]]; then
-    echo -e "\e[33;1m[WARNING]\e[0m This is not the first run! Skipping" \
-        "superuser creation.\nTo force it, remove $lockfile"
-else
-    export DJANGO_SUPERUSER_PASSWORD=$TA_PASSWORD
-    output="$(python manage.py createsuperuser --noinput --name "$TA_USERNAME" 2>&1)"
-
-    case "$output" in
-        *"Superuser created successfully"*)
-            echo "$output" && touch $lockfile ;;
-        *"That name is already taken."*)
-            echo "Superuser already exists. Creation will be skipped on next start."
-            touch $lockfile ;;
-        *) echo "$output" && exit 1
-    esac
+if [[ -z "$DJANGO_DEBUG" ]]; then
+    python manage.py collectstatic --noinput -c
 fi
 
-python manage.py collectstatic --noinput -c
+# ta setup
+python manage.py ta_envcheck
+python manage.py ta_connection
+python manage.py ta_startup
+
+# start all tasks
 nginx &
 celery -A home.tasks worker --loglevel=INFO &
 celery -A home beat --loglevel=INFO \

docs/Settings.md

@@ -121,8 +121,8 @@ The field **Refresh older than x days** takes a number where TubeArchivist will
 ## Thumbnail check
 This will check if all expected thumbnails are there and will delete any artwork without matching video.
 
-## Index backup
-Create a zip file of the metadata and select **Max auto backups to keep** to automatically delete old backups created from this task.
+## ZIP file index backup
+Create a zip file of the metadata and select **Max auto backups to keep** to automatically delete old backups created from this task. For data consistency, make sure there aren't any other tasks running that will change the index during the backup process. This is very slow, particularly for large archives. Use snapshots instead. 
 
 
 # Actions
@@ -166,8 +166,8 @@ If the video you are trying to import is not available on YouTube any more, **Tu
 ## Embed thumbnails into media file
 This will write or overwrite all thumbnails in the media file using the downloaded thumbnail. This is only necessary if you didn't download the files with the option *Embed Thumbnail* enabled or want to make sure all media files get the newest thumbnail. Follow the docker-compose logs to monitor progress.
 
-## Backup Database
-This will backup your metadata into a zip file. The file will get stored at *cache/backup* and will contain the necessary files to restore the Elasticsearch index formatted **nd-json** files.  
+## ZIP file index backup
+This will backup your metadata into a zip file. The file will get stored at *cache/backup* and will contain the necessary files to restore the Elasticsearch index formatted **nd-json** files. For data consistency, make sure there aren't any other tasks running that will change the index during the backup process. This is very slow, particularly for large archives.  
 
 BE AWARE: This will **not** backup any media files, just the metadata from the Elasticsearch.
 

docs/Video.md

@@ -1,5 +1,5 @@
 # Video Page
-Every video downloaded gets a dedicated page accessible at `/video/<video-id>/` of your Tube Archivist.
+Every video downloaded gets a dedicated page accessible at `/video/<video-id>/` of your Tube Archivist. Throughout the interface, click on a video title to access the video page.
 
 Clicking on the channel name or the channel icon will bring you to the dedicated channel detail [page](Channels#channel-detail).
 

tubearchivist/api/README.md

@@ -155,6 +155,9 @@ Timestamps either *int* or *float*, end time can't be before start time.
 ## Channel List View
 /api/channel/
 
+Parameter:
+- filter: subscribed
+
 ### Subscribe to a list of channels
 POST /api/channel/
 ```json

tubearchivist/api/views.py

@@ -289,14 +289,27 @@ class ChannelApiListView(ApiBaseView):
     """
 
     search_base = "ta_channel/_search/"
+    valid_filter = ["subscribed"]
 
     def get(self, request):
         """get request"""
-        self.get_document_list(request)
         self.data.update(
             {"sort": [{"channel_name.keyword": {"order": "asc"}}]}
         )
 
+        query_filter = request.GET.get("filter", False)
+        must_list = []
+        if query_filter:
+            if query_filter not in self.valid_filter:
+                message = f"invalid url query filder: {query_filter}"
+                print(message)
+                return Response({"message": message}, status=400)
+
+            must_list.append({"term": {"channel_subscribed": {"value": True}}})
+
+        self.data["query"] = {"bool": {"must": must_list}}
+        self.get_document_list(request)
+
         return Response(self.response)
 
     @staticmethod
@@ -430,7 +443,7 @@ class DownloadApiView(ApiBaseView):
         # pylint: disable=unused-argument
         """delete single video from queue"""
         print(f"{video_id}: delete from queue")
-        PendingInteract(video_id=video_id).delete_item()
+        PendingInteract(video_id).delete_item()
 
         return Response({"success": True})
 

tubearchivist/config/management/commands/ta_connection.py

@@ -0,0 +1,140 @@
+"""
+Functionality:
+- check that all connections are working
+"""
+
+from time import sleep
+
+import requests
+from django.core.management.base import BaseCommand, CommandError
+from home.src.es.connect import ElasticWrap
+from home.src.ta.ta_redis import RedisArchivist
+
+TOPIC = """
+
+#######################
+#  Connection check   #
+#######################
+
+"""
+
+
+class Command(BaseCommand):
+    """command framework"""
+
+    TIMEOUT = 120
+    MIN_MAJOR, MAX_MAJOR = 8, 8
+    MIN_MINOR = 0
+
+    # pylint: disable=no-member
+    help = "Check connections"
+
+    def handle(self, *args, **options):
+        """run all commands"""
+        self.stdout.write(TOPIC)
+        self._redis_connection_check()
+        self._redis_config_set()
+        self._es_connection_check()
+        self._es_version_check()
+        self._es_path_check()
+
+    def _redis_connection_check(self):
+        """check ir redis connection is established"""
+        self.stdout.write("[1] connect to Redis")
+        redis_conn = RedisArchivist().conn
+        for _ in range(5):
+            sleep(2)
+            try:
+                pong = redis_conn.execute_command("PING")
+                if pong:
+                    self.stdout.write(
+                        self.style.SUCCESS("    ✓ Redis connection verified")
+                    )
+                    return
+
+            except Exception:  # pylint: disable=broad-except
+                self.stdout.write("    ... retry Redis connection")
+
+        message = "    🗙 Redis connection failed"
+        self.stdout.write(self.style.ERROR(f"{message}"))
+        RedisArchivist().exec("PING")
+        raise CommandError(message)
+
+    def _redis_config_set(self):
+        """set config for redis if not set already"""
+        self.stdout.write("[2] set Redis config")
+        redis_conn = RedisArchivist().conn
+        timeout_is = int(redis_conn.config_get("timeout").get("timeout"))
+        if not timeout_is:
+            redis_conn.config_set("timeout", 3600)
+
+        self.stdout.write(self.style.SUCCESS("    ✓ Redis config set"))
+
+    def _es_connection_check(self):
+        """wait for elasticsearch connection"""
+        self.stdout.write("[3] connect to Elastic Search")
+        total = self.TIMEOUT // 5
+        for i in range(total):
+            sleep(5)
+            self.stdout.write(f"    ... waiting for ES [{i}/{total}]")
+            try:
+                _, status_code = ElasticWrap("/").get(
+                    timeout=1, print_error=False
+                )
+            except requests.exceptions.ConnectionError:
+                continue
+
+            if status_code and status_code == 200:
+                self.stdout.write(
+                    self.style.SUCCESS("    ✓ ES connection established")
+                )
+                return
+
+        response, status_code = ElasticWrap("/").get(
+            timeout=1, print_error=False
+        )
+
+        message = "    🗙 ES connection failed"
+        self.stdout.write(self.style.ERROR(f"{message}"))
+        self.stdout.write(f"    error message: {response}")
+        self.stdout.write(f"    status code: {status_code}")
+        raise CommandError(message)
+
+    def _es_version_check(self):
+        """check for minimal elasticsearch version"""
+        self.stdout.write("[4] Elastic Search version check")
+        response, _ = ElasticWrap("/").get()
+        version = response["version"]["number"]
+        major = int(version.split(".")[0])
+
+        if self.MIN_MAJOR <= major <= self.MAX_MAJOR:
+            self.stdout.write(
+                self.style.SUCCESS("    ✓ ES version check passed")
+            )
+            return
+
+        message = (
+            "    🗙 ES connection failed. "
+            + f"Expected {self.MIN_MAJOR}.{self.MIN_MINOR} but got {version}"
+        )
+        self.stdout.write(self.style.ERROR(f"{message}"))
+        raise CommandError(message)
+
+    def _es_path_check(self):
+        """check that path.repo var is set"""
+        self.stdout.write("[5] check ES path.repo env var")
+        response, _ = ElasticWrap("_nodes/_all/settings").get()
+        for node in response["nodes"].values():
+            if node["settings"]["path"].get("repo"):
+                self.stdout.write(
+                    self.style.SUCCESS("    ✓ path.repo env var is set")
+                )
+                return
+
+            message = (
+                "    🗙 path.repo env var not found. "
+                + "set the following env var to the ES container:\n"
+                + "    path.repo=/usr/share/elasticsearch/data/snapshot"
+            )
+            self.stdout.write(self.style.ERROR(f"{message}"))
+            raise CommandError(message)

tubearchivist/config/management/commands/ta_envcheck.py

@@ -0,0 +1,195 @@
+"""
+Functionality:
+- Check environment at startup
+- Process config file overwrites from env var
+- Stop startup on error
+- python management.py ta_envcheck
+"""
+
+import os
+import re
+
+from django.core.management.base import BaseCommand, CommandError
+from home.models import Account
+
+LOGO = """
+
+                         ....  .....
+                  ...'',;:cc,. .;::;;,'...
+               ..,;:cccllclc,  .:ccllllcc;,..
+            ..,:cllcc:;,'.',.  ....'',;ccllc:,..
+          ..;cllc:,'..                ...,:cccc:'.
+         .;cccc;..                        ..,:ccc:'.
+       .ckkkOkxollllllllllllc.      .,:::;.  .,cclc;
+      .:0MMMMMMMMMMMMMMMMMMMX:     .cNMMMWx.   .;clc:
+     .;lOXK0000KNMMMMX00000KO;     ;KMMMMMNl.   .;ccl:,.
+     .;:c:'.....kMMMNo........    'OMMMWMMMK:    '::;;'.
+   .......     .xMMMNl           .dWMMXdOMMMO'   ........
+   .:cc:;.     .xMMMNc          .lNMMNo.:XMMWx.    .:cl:.
+   .:llc,.     .:xxxd,          ;KMMMk. .oWMMNl.   .:llc'
+   .cll:.     .;:;;:::,.       'OMMMK:';''kWMMK:   .;llc,
+   .cll:.     .,;;;;;;,.     .,xWMMNl.:l:.;KMMMO'  .;llc'
+   .:llc.      .cOOOk;      .lKNMMWx..:l:..lNMMWx. .:llc'
+   .;lcc,.     .xMMMNc      :KMMMM0, .:lc. .xWMMNl.'ccl:.
+    .cllc.     .xMMMNc     'OMMMMXc...:lc...,0MMMKl:lcc,.
+    .,ccl:.    .xMMMNc    .xWMMMWo.,;;:lc;;;.cXMMMXdcc;.
+     .,clc:.   .xMMMNc   .lNMMMWk. .':clc:,. .dWMMW0o;.
+      .,clcc,. .ckkkx;   .okkkOx,    .';,.    'kKKK0l.
+       .':lcc:'.....      .  ..            ..,;cllc,.
+         .,cclc,....                     ....;clc;..
+          ..,:,..,c:'..              ...';:,..,:,.
+            ....:lcccc:;,'''.....'',;;:clllc,....
+               .'',;:cllllllccccclllllcc:,'..
+                   ...'',,;;;;;;;;;,''...
+                            .....
+
+"""
+
+TOPIC = """
+#######################
+#  Environment Setup  #
+#######################
+
+"""
+
+EXPECTED_ENV_VARS = [
+    "TA_USERNAME",
+    "TA_PASSWORD",
+    "ELASTIC_PASSWORD",
+    "ES_URL",
+    "TA_HOST",
+]
+INST = "https://github.com/tubearchivist/tubearchivist#installing-and-updating"
+NGINX = "/etc/nginx/sites-available/default"
+UWSGI = "/app/uwsgi.ini"
+
+
+class Command(BaseCommand):
+    """command framework"""
+
+    # pylint: disable=no-member
+    help = "Check environment before startup"
+
+    def handle(self, *args, **options):
+        """run all commands"""
+        self.stdout.write(LOGO)
+        self.stdout.write(TOPIC)
+        self._expected_vars()
+        self._elastic_user_overwrite()
+        self._ta_port_overwrite()
+        self._ta_uwsgi_overwrite()
+        self._enable_cast_overwrite()
+        self._create_superuser()
+
+    def _expected_vars(self):
+        """check if expected env vars are set"""
+        self.stdout.write("[1] checking expected env vars")
+        env = os.environ
+        for var in EXPECTED_ENV_VARS:
+            if not env.get(var):
+                message = f"    🗙 expected env var {var} not set\n    {INST}"
+                self.stdout.write(self.style.ERROR(message))
+                raise CommandError(message)
+
+        message = "    ✓ all expected env vars are set"
+        self.stdout.write(self.style.SUCCESS(message))
+
+    def _elastic_user_overwrite(self):
+        """check for ELASTIC_USER overwrite"""
+        self.stdout.write("[2] set default ES user")
+        if not os.environ.get("ELASTIC_USER"):
+            os.environ.setdefault("ELASTIC_USER", "elastic")
+
+        env = os.environ.get("ELASTIC_USER")
+
+        self.stdout.write(self.style.SUCCESS(f"    ✓ ES user is set to {env}"))
+
+    def _ta_port_overwrite(self):
+        """set TA_PORT overwrite for nginx"""
+        self.stdout.write("[3] check TA_PORT overwrite")
+        overwrite = os.environ.get("TA_PORT")
+        if not overwrite:
+            self.stdout.write(self.style.SUCCESS("    TA_PORT is not set"))
+            return
+
+        regex = re.compile(r"listen [0-9]{1,5}")
+        changed = file_overwrite(NGINX, regex, overwrite)
+        if changed:
+            message = f"    ✓ TA_PORT changed to {overwrite}"
+        else:
+            message = f"    ✓ TA_PORT already set to {overwrite}"
+
+        self.stdout.write(self.style.SUCCESS(message))
+
+    def _ta_uwsgi_overwrite(self):
+        """set TA_UWSGI_PORT overwrite"""
+        self.stdout.write("[4] check TA_UWSGI_PORT overwrite")
+        overwrite = os.environ.get("TA_UWSGI_PORT")
+        if not overwrite:
+            message = "    TA_UWSGI_PORT is not set"
+            self.stdout.write(self.style.SUCCESS(message))
+            return
+
+        # nginx
+        regex = re.compile(r"uwsgi_pass localhost:[0-9]{1,5}")
+        to_overwrite = f"uwsgi_pass localhost:{overwrite}"
+        changed = file_overwrite(NGINX, regex, to_overwrite)
+
+        # uwsgi
+        regex = re.compile(r"socket = :[0-9]{1,5}")
+        to_overwrite = f"socket = :{overwrite}"
+        changed = file_overwrite(UWSGI, regex, to_overwrite)
+
+        if changed:
+            message = f"    ✓ TA_UWSGI_PORT changed to {overwrite}"
+        else:
+            message = f"    ✓ TA_UWSGI_PORT already set to {overwrite}"
+
+        self.stdout.write(self.style.SUCCESS(message))
+
+    def _enable_cast_overwrite(self):
+        """cast workaround, remove auth for static files in nginx"""
+        self.stdout.write("[5] check ENABLE_CAST overwrite")
+        overwrite = os.environ.get("ENABLE_CAST")
+        if not overwrite:
+            self.stdout.write(self.style.SUCCESS("    ENABLE_CAST is not set"))
+            return
+
+        regex = re.compile(r"[^\S\r\n]*auth_request /api/ping/;\n")
+        changed = file_overwrite(NGINX, regex, "")
+        if changed:
+            message = "    ✓ process nginx to enable Cast"
+        else:
+            message = "    ✓ Cast is already enabled in nginx"
+
+        self.stdout.write(self.style.SUCCESS(message))
+
+    def _create_superuser(self):
+        """create superuser if not exist"""
+        self.stdout.write("[6] create superuser")
+        is_created = Account.objects.filter(is_superuser=True)
+        if is_created:
+            message = "    superuser already created"
+            self.stdout.write(self.style.SUCCESS(message))
+            return
+
+        name = os.environ.get("TA_USERNAME")
+        password = os.environ.get("TA_PASSWORD")
+        Account.objects.create_superuser(name, password)
+        message = f"    ✓ new superuser with name {name} created"
+        self.stdout.write(self.style.SUCCESS(message))
+
+
+def file_overwrite(file_path, regex, overwrite):
+    """change file content from old to overwrite, return true when changed"""
+    with open(file_path, "r", encoding="utf-8") as f:
+        file_content = f.read()
+
+    changed = re.sub(regex, overwrite, file_content)
+    if changed == file_content:
+        return False
+
+    with open(file_path, "w", encoding="utf-8") as f:
+        f.write(changed)
+
+    return True

tubearchivist/config/management/commands/ta_startup.py

@@ -0,0 +1,171 @@
+"""
+Functionality:
+- Application startup
+- Apply migrations
+"""
+
+import os
+
+from django.core.management.base import BaseCommand, CommandError
+from home.src.es.connect import ElasticWrap
+from home.src.es.index_setup import ElasitIndexWrap
+from home.src.es.snapshot import ElasticSnapshot
+from home.src.ta.config import AppConfig, ReleaseVersion
+from home.src.ta.helper import clear_dl_cache
+from home.src.ta.ta_redis import RedisArchivist
+
+TOPIC = """
+
+#######################
+#  Application Start  #
+#######################
+
+"""
+
+
+class Command(BaseCommand):
+    """command framework"""
+
+    # pylint: disable=no-member
+
+    def handle(self, *args, **options):
+        """run all commands"""
+        self.stdout.write(TOPIC)
+        self._sync_redis_state()
+        self._make_folders()
+        self._release_locks()
+        self._clear_dl_cache()
+        self._version_check()
+        self._mig_index_setup()
+        self._mig_snapshot_check()
+        self._mig_set_vid_type()
+
+    def _sync_redis_state(self):
+        """make sure redis gets new config.json values"""
+        self.stdout.write("[1] set new config.json values")
+        needs_update = AppConfig().load_new_defaults()
+        if needs_update:
+            self.stdout.write(
+                self.style.SUCCESS("    ✓ new config values set")
+            )
+        else:
+            self.stdout.write(self.style.SUCCESS("    no new config values"))
+
+    def _make_folders(self):
+        """make expected cache folders"""
+        self.stdout.write("[2] create expected cache folders")
+        folders = [
+            "backup",
+            "channels",
+            "download",
+            "import",
+            "playlists",
+            "videos",
+        ]
+        cache_dir = AppConfig().config["application"]["cache_dir"]
+        for folder in folders:
+            folder_path = os.path.join(cache_dir, folder)
+            os.makedirs(folder_path, exist_ok=True)
+
+        self.stdout.write(self.style.SUCCESS("    ✓ expected folders created"))
+
+    def _release_locks(self):
+        """make sure there are no leftover locks set in redis"""
+        self.stdout.write("[3] clear leftover locks in redis")
+        all_locks = [
+            "dl_queue_id",
+            "dl_queue",
+            "downloading",
+            "manual_import",
+            "reindex",
+            "rescan",
+            "run_backup",
+            "startup_check",
+        ]
+
+        redis_con = RedisArchivist()
+        has_changed = False
+        for lock in all_locks:
+            if redis_con.del_message(lock):
+                self.stdout.write(
+                    self.style.SUCCESS(f"    ✓ cleared lock {lock}")
+                )
+                has_changed = True
+
+        if not has_changed:
+            self.stdout.write(self.style.SUCCESS("    no locks found"))
+
+    def _clear_dl_cache(self):
+        """clear leftover files from dl cache"""
+        self.stdout.write("[4] clear leftover files from dl cache")
+        config = AppConfig().config
+        leftover_files = clear_dl_cache(config)
+        if leftover_files:
+            self.stdout.write(
+                self.style.SUCCESS(f"    ✓ cleared {leftover_files} files")
+            )
+        else:
+            self.stdout.write(self.style.SUCCESS("    no files found"))
+
+    def _version_check(self):
+        """remove new release key if updated now"""
+        self.stdout.write("[5] check for first run after update")
+        new_version = ReleaseVersion().is_updated()
+        if new_version:
+            self.stdout.write(
+                self.style.SUCCESS(f"    ✓ update to {new_version} completed")
+            )
+        else:
+            self.stdout.write(self.style.SUCCESS("    no new update found"))
+
+    def _mig_index_setup(self):
+        """migration: validate index mappings"""
+        self.stdout.write("[MIGRATION] validate index mappings")
+        ElasitIndexWrap().setup()
+
+    def _mig_snapshot_check(self):
+        """migration setup snapshots"""
+        self.stdout.write("[MIGRATION] setup snapshots")
+        ElasticSnapshot().setup()
+
+    def _mig_set_vid_type(self):
+        """migration: update 0.3.0 to 0.3.1 set vid_type default"""
+        self.stdout.write("[MIGRATION] set default vid_type")
+        index_list = ["ta_video", "ta_download"]
+        data = {
+            "query": {
+                "bool": {
+                    "should": [
+                        {
+                            "bool": {
+                                "must_not": [{"exists": {"field": "vid_type"}}]
+                            }
+                        },
+                        {"term": {"vid_type": {"value": "unknown"}}},
+                    ]
+                }
+            },
+            "script": {"source": "ctx._source['vid_type'] = 'videos'"},
+        }
+
+        for index_name in index_list:
+            path = f"{index_name}/_update_by_query"
+            response, status_code = ElasticWrap(path).post(data=data)
+            if status_code == 200:
+                updated = response.get("updated", 0)
+                if not updated:
+                    self.stdout.write(
+                        f"    no videos needed updating in {index_name}"
+                    )
+                    continue
+
+                self.stdout.write(
+                    self.style.SUCCESS(
+                        f"    ✓ {updated} videos updated in {index_name}"
+                    )
+                )
+            else:
+                message = f"    🗙 {index_name} vid_type update failed"
+                self.stdout.write(self.style.ERROR(message))
+                self.stdout.write(response)
+                raise CommandError(message)

tubearchivist/config/settings.py

@@ -32,7 +32,9 @@ SECRET_KEY = PW_HASH.hexdigest()
 # SECURITY WARNING: don't run with debug turned on in production!
 DEBUG = bool(environ.get("DJANGO_DEBUG"))
 
-ALLOWED_HOSTS = [i.strip() for i in environ.get("TA_HOST").split()]
+ALLOWED_HOSTS = []
+if environ.get("TA_HOST"):
+    ALLOWED_HOSTS = [i.strip() for i in environ.get("TA_HOST").split()]
 
 CSRF_TRUSTED_ORIGINS = []
 for host in ALLOWED_HOSTS:
@@ -58,6 +60,7 @@ INSTALLED_APPS = [
     "rest_framework",
     "rest_framework.authtoken",
     "api",
+    "config",
 ]
 
 MIDDLEWARE = [
@@ -262,4 +265,4 @@ CORS_ALLOW_HEADERS = list(default_headers) + [
 
 # TA application settings
 TA_UPSTREAM = "https://github.com/tubearchivist/tubearchivist"
-TA_VERSION = "v0.3.2"
+TA_VERSION = "v0.3.3"

tubearchivist/home/apps.py

@@ -1,163 +1,6 @@
 """handle custom startup functions"""
 
-import os
-import sys
-
 from django.apps import AppConfig
-from home.src.es.connect import ElasticWrap
-from home.src.es.index_setup import ElasitIndexWrap
-from home.src.es.snapshot import ElasticSnapshot
-from home.src.ta.config import AppConfig as ArchivistConfig
-from home.src.ta.config import ReleaseVersion
-from home.src.ta.helper import clear_dl_cache
-from home.src.ta.ta_redis import RedisArchivist
-
-
-class StartupCheck:
-    """checks to run at application startup"""
-
-    MIN_MAJOR, MAX_MAJOR = 8, 8
-    MIN_MINOR = 0
-
-    def __init__(self):
-        self.config_handler = ArchivistConfig()
-        self.redis_con = RedisArchivist()
-        self.has_run = self.get_has_run()
-
-    def run(self):
-        """run all startup checks"""
-        print("run startup checks")
-        self.set_lock()
-        self.es_version_check()
-        self.release_lock()
-        self.sync_redis_state()
-        self.set_redis_conf()
-        ElasitIndexWrap().setup()
-        self.make_folders()
-        clear_dl_cache(self.config_handler.config)
-        self.snapshot_check()
-        self.ta_version_check()
-        self.es_set_vid_type()
-        self.expire_lock()
-
-    def get_has_run(self):
-        """validate if check has already executed"""
-        return self.redis_con.get_message("startup_check")
-
-    def set_lock(self):
-        """set lock to avoid executing once per thread"""
-        self.redis_con.set_message("startup_check", message={"status": True})
-
-    def expire_lock(self):
-        """startup checks run"""
-        print("startup checks completed")
-        message = {"status": True}
-        self.redis_con.set_message("startup_check", message, expire=120)
-
-    def sync_redis_state(self):
-        """make sure redis gets new config.json values"""
-        print("sync redis")
-        self.config_handler.load_new_defaults()
-
-    def set_redis_conf(self):
-        """set conf values for redis"""
-        self.redis_con.conn.config_set("timeout", 3600)
-
-    def make_folders(self):
-        """make needed cache folders here so docker doesn't mess it up"""
-        folders = [
-            "download",
-            "channels",
-            "videos",
-            "playlists",
-            "import",
-            "backup",
-        ]
-        cache_dir = self.config_handler.config["application"]["cache_dir"]
-        for folder in folders:
-            folder_path = os.path.join(cache_dir, folder)
-            os.makedirs(folder_path, exist_ok=True)
-
-    def release_lock(self):
-        """make sure there are no leftover locks set in redis"""
-        all_locks = [
-            "manual_import",
-            "downloading",
-            "dl_queue",
-            "dl_queue_id",
-            "reindex",
-            "rescan",
-            "run_backup",
-        ]
-        for lock in all_locks:
-            response = self.redis_con.del_message(lock)
-            if response:
-                print("deleted leftover key from redis: " + lock)
-
-    def snapshot_check(self):
-        """setup snapshot config, create if needed"""
-        app = self.config_handler.config["application"]
-        if not app.get("enable_snapshot"):
-            return
-
-        ElasticSnapshot().setup()
-
-    def is_invalid(self, version):
-        """return true if es version is invalid, false if ok"""
-        major, minor = [int(i) for i in version.split(".")[:2]]
-        if not self.MIN_MAJOR <= major <= self.MAX_MAJOR:
-            return True
-
-        if minor >= self.MIN_MINOR:
-            return False
-
-        return True
-
-    def es_version_check(self):
-        """check for minimal elasticsearch version"""
-        response, _ = ElasticWrap("/").get()
-        version = response["version"]["number"]
-        invalid = self.is_invalid(version)
-
-        if invalid:
-            print(
-                "required elasticsearch version: "
-                + f"{self.MIN_MAJOR}.{self.MIN_MINOR}"
-            )
-            sys.exit(1)
-
-        print("elasticsearch version check passed")
-
-    def ta_version_check(self):
-        """remove key if updated now"""
-        ReleaseVersion().is_updated()
-
-    def es_set_vid_type(self):
-        """
-        update path 0.3.0 to 0.3.1, set default vid_type to video
-        fix unidentified vids in unstable
-        """
-        index_list = ["ta_video", "ta_download"]
-        data = {
-            "query": {
-                "bool": {
-                    "should": [
-                        {
-                            "bool": {
-                                "must_not": [{"exists": {"field": "vid_type"}}]
-                            }
-                        },
-                        {"term": {"vid_type": {"value": "unknown"}}},
-                    ]
-                }
-            },
-            "script": {"source": "ctx._source['vid_type'] = 'videos'"},
-        }
-
-        for index_name in index_list:
-            path = f"{index_name}/_update_by_query"
-            response, _ = ElasticWrap(path).post(data=data)
-            print(f"{index_name} vid_type index update ran: {response}")
 
 
 class HomeConfig(AppConfig):
@@ -165,11 +8,3 @@ class HomeConfig(AppConfig):
 
     default_auto_field = "django.db.models.BigAutoField"
     name = "home"
-
-    def ready(self):
-        startup = StartupCheck()
-        if startup.has_run["status"]:
-            print("startup checks run in other thread")
-            return
-
-        startup.run()

tubearchivist/home/config.json

@@ -42,7 +42,7 @@
         "videos": "/youtube",
         "colors": "dark",
         "enable_cast": false,
-        "enable_snapshot": false
+        "enable_snapshot": true
     },
     "scheduler": {
         "update_subscribed": false,
@@ -50,7 +50,7 @@
         "check_reindex":  {"minute": "0", "hour": "12", "day_of_week": "*"},
         "check_reindex_days": 90,
         "thumbnail_check": {"minute": "0", "hour": "17", "day_of_week": "*"},
-        "run_backup": {"minute": "0", "hour": "8", "day_of_week": "0"},
+        "run_backup": false,
         "run_backup_rotate": 5,
         "version_check": "rand-d"
     }

tubearchivist/home/migrations/0001_initial.py

@@ -0,0 +1,35 @@
+# Generated by Django 4.1.5 on 2023-02-02 06:49
+
+from django.db import migrations, models
+import home.models
+
+
+class Migration(migrations.Migration):
+
+    initial = True
+
+    dependencies = [
+        ('auth', '0012_alter_user_first_name_max_length'),
+    ]
+
+    operations = [
+        migrations.CreateModel(
+            name='Account',
+            fields=[
+                ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+                ('password', models.CharField(max_length=128, verbose_name='password')),
+                ('last_login', models.DateTimeField(blank=True, null=True, verbose_name='last login')),
+                ('is_superuser', models.BooleanField(default=False, help_text='Designates that this user has all permissions without explicitly assigning them.', verbose_name='superuser status')),
+                ('name', models.CharField(max_length=150, unique=True)),
+                ('is_staff', models.BooleanField(default=False)),
+                ('groups', models.ManyToManyField(blank=True, help_text='The groups this user belongs to. A user will get all permissions granted to each of their groups.', related_name='user_set', related_query_name='user', to='auth.group', verbose_name='groups')),
+                ('user_permissions', models.ManyToManyField(blank=True, help_text='Specific permissions for this user.', related_name='user_set', related_query_name='user', to='auth.permission', verbose_name='user permissions')),
+            ],
+            options={
+                'abstract': False,
+            },
+            managers=[
+                ('objects', home.models.AccountManager()),
+            ],
+        ),
+    ]

tubearchivist/home/src/download/queue.py

@@ -96,13 +96,13 @@ class PendingIndex:
 class PendingInteract:
     """interact with items in download queue"""
 
-    def __init__(self, video_id=False, status=False):
-        self.video_id = video_id
+    def __init__(self, youtube_id=False, status=False):
+        self.youtube_id = youtube_id
         self.status = status
 
     def delete_item(self):
         """delete single item from pending"""
-        path = f"ta_download/_doc/{self.video_id}"
+        path = f"ta_download/_doc/{self.youtube_id}"
         _, _ = ElasticWrap(path).delete(refresh=True)
 
     def delete_by_status(self):
@@ -114,15 +114,35 @@ class PendingInteract:
     def update_status(self):
         """update status field of pending item"""
         data = {"doc": {"status": self.status}}
-        path = f"ta_download/_update/{self.video_id}"
+        path = f"ta_download/_update/{self.youtube_id}"
         _, _ = ElasticWrap(path).post(data=data)
 
     def get_item(self):
         """return pending item dict"""
-        path = f"ta_download/_doc/{self.video_id}"
+        path = f"ta_download/_doc/{self.youtube_id}"
         response, status_code = ElasticWrap(path).get()
         return response["_source"], status_code
 
+    def get_channel(self):
+        """
+        get channel metadata from queue to not depend on channel to be indexed
+        """
+        data = {
+            "size": 1,
+            "query": {"term": {"channel_id": {"value": self.youtube_id}}},
+        }
+        response, _ = ElasticWrap("ta_download/_search").get(data=data)
+        hits = response["hits"]["hits"]
+        if not hits:
+            channel_name = "NA"
+        else:
+            channel_name = hits[0]["_source"].get("channel_name", "NA")
+
+        return {
+            "channel_id": self.youtube_id,
+            "channel_name": channel_name,
+        }
+
 
 class PendingList(PendingIndex):
     """manage the pending videos list"""
@@ -264,6 +284,7 @@ class PendingList(PendingIndex):
             return False
         # stop if video is streaming live now
         if vid["live_status"] in ["is_upcoming", "is_live"]:
+            print(f"{youtube_id}: skip is_upcoming or is_live")
             return False
 
         if vid["live_status"] == "was_live":

tubearchivist/home/src/download/thumbnails.py

@@ -43,8 +43,12 @@ class ThumbManagerBase:
                 response = requests.get(url, stream=True, timeout=5)
                 if response.ok:
                     try:
-                        return Image.open(response.raw)
-                    except UnidentifiedImageError:
+                        img = Image.open(response.raw)
+                        if isinstance(img, Image.Image):
+                            return img
+                        return self.get_fallback()
+
+                    except (UnidentifiedImageError, OSError):
                         print(f"failed to open thumbnail: {url}")
                         return self.get_fallback()
 
@@ -59,6 +63,7 @@ class ThumbManagerBase:
 
     def get_fallback(self):
         """get fallback thumbnail if not available"""
+        print(f"{self.item_id}: failed to extract thumbnail, use fallback")
         if self.fallback:
             img_raw = Image.open(self.fallback)
             return img_raw

tubearchivist/home/src/es/backup.py

@@ -143,7 +143,6 @@ class ElasticBackup:
         backup_dir = os.path.join(self.cache_dir, "backup")
 
         for json_f in zip_content:
-
             file_name = os.path.join(backup_dir, json_f)
 
             if not json_f.startswith("es_") or not json_f.endswith(".json"):

tubearchivist/home/src/es/connect.py

@@ -32,15 +32,15 @@ class ElasticWrap:
         self.auth = self.config["application"]["es_auth"]
         self.url = f"{es_url}/{self.path}"
 
-    def get(self, data=False):
+    def get(self, data=False, timeout=10, print_error=True):
         """get data from es"""
         if data:
             response = requests.get(
-                self.url, json=data, auth=self.auth, timeout=10
+                self.url, json=data, auth=self.auth, timeout=timeout
             )
         else:
-            response = requests.get(self.url, auth=self.auth, timeout=10)
-        if not response.ok:
+            response = requests.get(self.url, auth=self.auth, timeout=timeout)
+        if print_error and not response.ok:
             print(response.text)
 
         return response.json(), response.status_code

tubearchivist/home/src/index/comments.py

@@ -80,7 +80,6 @@ class Comments:
 
         yt_obs = {
             "skip_download": True,
-            "quiet": False,
             "getcomments": True,
             "extractor_args": {
                 "youtube": {
@@ -110,12 +109,20 @@ class Comments:
         if comments_raw:
             for comment in comments_raw:
                 cleaned_comment = self.clean_comment(comment)
+                if not cleaned_comment:
+                    continue
+
                 comments.append(cleaned_comment)
 
         self.comments_format = comments
 
     def clean_comment(self, comment):
         """parse metadata from comment for indexing"""
+        if not comment.get("text"):
+            # comment text can be empty
+            print(f"{self.youtube_id}: Failed to extract text, {comment}")
+            return False
+
         time_text_datetime = datetime.utcfromtimestamp(comment["timestamp"])
 
         if time_text_datetime.hour == 0 and time_text_datetime.minute == 0:

tubearchivist/home/src/ta/config.py

@@ -137,7 +137,7 @@ class AppConfig:
             config = self.get_config()
             config["scheduler"]["version_check"] = self._build_rand_daily()
             RedisArchivist().set_message("config", config)
-            return
+            return False
 
         needs_update = False
 
@@ -160,6 +160,8 @@ class AppConfig:
         if needs_update:
             RedisArchivist().set_message("config", redis_config)
 
+        return needs_update
+
 
 class ScheduleBuilder:
     """build schedule dicts for beat"""
@@ -263,6 +265,8 @@ class ScheduleBuilder:
 
     def build_schedule(self):
         """build schedule dict as expected by app.conf.beat_schedule"""
+        AppConfig().load_new_defaults()
+        self.config = AppConfig().config
         schedule_dict = {}
 
         for schedule_item in self.SCHEDULES:
@@ -342,11 +346,13 @@ class ReleaseVersion:
         """check if update happened in the mean time"""
         message = self.get_update()
         if not message:
-            return
+            return False
 
         if self._parse_version(message.get("version")) == self.local_version:
-            print(f"[{self.local_version}]: update completed")
             RedisArchivist().del_message(self.NEW_KEY)
+            return settings.TA_VERSION
+
+        return False
 
     def get_update(self):
         """return new version dict if available"""

tubearchivist/home/src/ta/helper.py

@@ -122,10 +122,13 @@ def clear_dl_cache(config):
     """clear leftover files from dl cache"""
     print("clear download cache")
     cache_dir = os.path.join(config["application"]["cache_dir"], "download")
-    for cached in os.listdir(cache_dir):
+    leftover_files = os.listdir(cache_dir)
+    for cached in leftover_files:
         to_delete = os.path.join(cache_dir, cached)
         os.remove(to_delete)
 
+    return len(leftover_files)
+
 
 def get_mapping():
     """read index_mapping.json and get expected mapping and settings"""

tubearchivist/home/tasks.py

@@ -10,7 +10,6 @@ import json
 import os
 
 from celery import Celery, shared_task
-from home.apps import StartupCheck
 from home.src.download.queue import PendingList
 from home.src.download.subscriptions import (
     ChannelSubscription,
@@ -112,9 +111,10 @@ def download_pending():
 def download_single(pending_video):
     """start download single video now"""
     queue = RedisQueue(queue_name="dl_queue")
+
     to_add = {
         "youtube_id": pending_video["youtube_id"],
-        "vid_type": pending_video["vid_type"],
+        "vid_type": pending_video.get("vid_type", VideoTypeEnum.VIDEOS.value),
     }
     queue.add_priority(json.dumps(to_add))
     print(f"Added to queue with priority: {to_add}")
@@ -228,7 +228,7 @@ def kill_dl(task_id):
     _ = RedisArchivist().del_message("dl_queue_id")
     RedisQueue(queue_name="dl_queue").clear()
 
-    clear_dl_cache(CONFIG)
+    _ = clear_dl_cache(CONFIG)
 
     # notify
     mess_dict = {
@@ -313,6 +313,5 @@ def version_check():
     ReleaseVersion().check()
 
 
-# load new defaults then start the schedule here
-StartupCheck().sync_redis_state()
+# start schedule here
 app.conf.beat_schedule = ScheduleBuilder().build_schedule()

tubearchivist/home/templates/home/downloads.html

@@ -90,7 +90,7 @@
                             {% else %}
                                 <span>{{ video.source.channel_name }}</span>
                             {% endif %}
-                            <h3>{{ video.source.title }}</h3>
+                            <a href="https://www.youtube.com/watch?v={{ video.source.youtube_id }}" target="_blank"><h3>{{ video.source.title }}</h3></a>
                         </div>
                         <p>Published: {{ video.source.published }} | Duration: {{ video.source.duration }} | {{ video.source.youtube_id }}</p>
                         <div>

tubearchivist/home/templates/home/settings.html

@@ -286,8 +286,9 @@
             </div>
         </div>
         <div class="settings-group">
-            <h2>Index backup</h2>
+            <h2>ZIP file index backup</h2>
             <div class="settings-item">
+                <p><i>Zip file backups are very slow for large archives and consistency is not guaranteed, use snapshots instead. Make sure no other tasks are running when creating a Zip file backup.</i></p>
                 <p>Current index backup schedule: <span class="settings-current">
                     {% if config.scheduler.run_backup %}
                         {% for key, value in config.scheduler.run_backup.items %}
@@ -332,8 +333,9 @@
         </div>
     </div>
     <div class="settings-group">
-        <h2>Backup database</h2>
+        <h2>ZIP file index backup</h2>
         <p>Export your database to a zip file stored at <span class="settings-current">cache/backup</span>.</p>
+        <p><i>Zip file backups are very slow for large archives and consistency is not guaranteed, use snapshots instead. Make sure no other tasks are running when creating a Zip file backup.</i></p>
         <div id="db-backup">
             <button onclick="dbBackup()">Start backup</button>
         </div>

tubearchivist/home/views.py

@@ -15,6 +15,7 @@ from django.contrib.auth.forms import AuthenticationForm
 from django.http import JsonResponse
 from django.shortcuts import redirect, render
 from django.views import View
+from home.src.download.queue import PendingInteract
 from home.src.download.yt_dlp_base import CookieHandler
 from home.src.es.backup import ElasticBackup
 from home.src.es.connect import ElasticWrap
@@ -32,7 +33,7 @@ from home.src.frontend.forms import (
     UserSettingsForm,
 )
 from home.src.frontend.searching import SearchHandler
-from home.src.index.channel import YoutubeChannel, channel_overwrites
+from home.src.index.channel import channel_overwrites
 from home.src.index.generic import Pagination
 from home.src.index.playlist import YoutubePlaylist
 from home.src.index.reindex import ReindexProgress
@@ -375,13 +376,13 @@ class DownloadView(ArchivistResultsView):
     def get(self, request):
         """handle get request"""
         self.initiate_vars(request)
-        self._update_view_data(request)
+        filter_view = self._update_view_data(request)
         self.find_results()
         self.context.update(
             {
                 "title": "Downloads",
                 "add_form": AddToQueueForm(),
-                "channel_agg_list": self._get_channel_agg(),
+                "channel_agg_list": self._get_channel_agg(filter_view),
             }
         )
         return render(request, "home/downloads.html", self.context)
@@ -401,12 +402,11 @@ class DownloadView(ArchivistResultsView):
                 {"term": {"channel_id": {"value": channel_filter}}}
             )
 
-            channel = YoutubeChannel(channel_filter)
-            channel.get_from_es()
+            channel = PendingInteract(channel_filter).get_channel()
             self.context.update(
                 {
-                    "channel_filter_id": channel_filter,
-                    "channel_filter_name": channel.json_data["channel_name"],
+                    "channel_filter_id": channel.get("channel_id"),
+                    "channel_filter_name": channel.get("channel_name"),
                 }
             )
 
@@ -417,11 +417,13 @@ class DownloadView(ArchivistResultsView):
             }
         )
 
-    def _get_channel_agg(self):
+        return filter_view
+
+    def _get_channel_agg(self, filter_view):
         """get pending channel with count"""
         data = {
             "size": 0,
-            "query": {"term": {"status": {"value": "pending"}}},
+            "query": {"term": {"status": {"value": filter_view}}},
             "aggs": {
                 "channel_downloads": {
                     "multi_terms": {

tubearchivist/requirements.txt

@@ -1,13 +1,13 @@
-beautifulsoup4==4.11.1
+beautifulsoup4==4.11.2
 celery==5.2.7
-Django==4.1.5
+Django==4.1.7
 django-auth-ldap==4.1.0
 django-cors-headers==3.13.0
 djangorestframework==3.14.0
 Pillow==9.4.0
-redis==4.4.2
-requests==2.28.1
+redis==4.5.1
+requests==2.28.2
 ryd-client==0.0.6
 uWSGI==2.0.21
 whitenoise==6.3.0
-yt_dlp==2023.1.6
+yt_dlp==2023.2.17

tubearchivist/static/progress.js

@@ -28,16 +28,12 @@ function checkMessages() {
 // get messages for page on timer
 function getMessages(dataOrigin) {
   fetch('/progress/')
-    .then(response => {
-      return response.json();
-    })
+    .then(response => response.json())
     .then(responseData => {
-      let messages = buildMessage(responseData, dataOrigin);
+      const messages = buildMessage(responseData, dataOrigin);
       if (messages.length > 0) {
         // restart itself
-        setTimeout(function () {
-          getMessages(dataOrigin);
-        }, 3000);
+        setTimeout(() => getMessages(dataOrigin), 500);
       }
     });
 }

tubearchivist/static/script.js

@@ -142,7 +142,7 @@ function toggleCheckbox(checkbox) {
   let payload = JSON.stringify(payloadDict);
   sendPost(payload);
   setTimeout(function () {
-    let currPage = window.location.pathname + window.location.search;
+    let currPage = window.location.pathname;
     window.location.replace(currPage);
   }, 500);
 }