bump TA_VERSION

fix linter
fix startup migration fail with unavailable shards
2026-08-08 20:29:27 +00:00 · 2023-03-04 22:33:50 +07:00 · 2023-03-04 22:08:48 +07:00 · 2023-03-04 18:10:18 +07:00 · 2023-03-04 17:37:58 +07:00 · 2023-03-04 17:03:46 +07:00
39 changed files with 874 additions and 489 deletions
--- a/.github/ISSUE_TEMPLATE/FEATURE-REQUEST.yml
+++ b/.github/ISSUE_TEMPLATE/FEATURE-REQUEST.yml
@@ -1,12 +1,12 @@
 name: Feature Request
-description: Create a new feature request
+description: I want to work on a new feature.
 title: "[Feature Request]: "

 body:
  - type: markdown
    attributes:
      value: |
-        Thanks for taking the time to help improve this project!
+        Thanks for taking the time to help improve this project! This project is currently only taking new feature requests from contributors who want to make a PR with their ideas. The backlog of planned features outlined in the roadmap or the open issues here will keep us busy for the foreseeable future.

  - type: checkboxes
    id: already
@@ -33,5 +33,5 @@ body:
      label: Your help is needed!
      description: This project is ambitious as it is, please contribute.
      options:
-        - label: Yes I can help with this feature request!
-          required: false
+        - label: Yes I will work on this in the next few days or weeks.
+          required: true
--- a/README.md
+++ b/README.md
@@ -66,10 +66,10 @@ There's dedicated user-contributed install steps under [docs/Installation.md](./

 For minimal system requirements, the Tube Archivist stack needs around 2GB of available memory for a small testing setup and around 4GB of available memory for a mid to large sized installation. Minimal with dual core with 4 threads, better quad core plus.  

-Note for arm64 hosts: The Tube Archivist container is multi arch, so is Elasticsearch. RedisJSON doesn't offer arm builds, but you can use the image `bbilly1/rejson`, an unofficial rebuild for arm64.
-
 This project requires docker. Ensure it is installed and running on your system.

+Note for **arm64**: Tube Archivist is a multi arch container, same for redis. For Elasitc Search use the official image for arm64 support. Other architectures are not supported.
+
 Save the [docker-compose.yml](./docker-compose.yml) file from this reposity somewhere permanent on your system, keeping it named `docker-compose.yml`. You'll need to refer to it whenever starting this application.

 Edit the following values from that file:
@@ -153,12 +153,20 @@ Wildcards "*" can not be used for the Access-Control-Allow-Origin header. If the

 Use `bbilly1/tubearchivist-es` to automatically get the recommended version, or use the official image with the version tag in the docker-compose file.

+Use official Elastic Search for **arm64**.  
+
 Stores video meta data and makes everything searchable. Also keeps track of the download queue.
  - Needs to be accessible over the default port `9200`
  - Needs a volume at **/usr/share/elasticsearch/data** to store data

 Follow the [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html) for additional installation details.

+### Elasticsearch on a custom port
+Should you need to change the port for Elasticsearch to for example *9500*, follow these steps:
+- Set the environment variable `http.port=9500` to the ES container
+- Change the *expose* value for the ES container to match your port number
+- For the Tube Archivist container, change the *ES_URL* environment variable, e.g. `ES_URL=http://archivist-es:9500`
+
 ### Redis JSON
 Functions as a cache and temporary link between the application and the file system. Used to store and display messages and configuration variables.
  - Needs to be accessible over the default port `6379`
@@ -172,6 +180,7 @@ For some architectures it might be required to run Redis JSON on a nonstandard p

 ### Updating Tube Archivist
 You will see the current version number of **Tube Archivist** in the footer of the interface. There is a daily version check task querying tubearchivist.com, notifying you of any new releases in the footer. To take advantage of the latest fixes and improvements, make sure you are running the *latest and greatest*.  
+* This project is tested for updates between one or two releases maximum. Further updates back may or may not be supported and you might have to reset your index and configurations to update. Ideally apply new updates at least once per month.
 * There can be breaking changes between updates, particularly as the application grows, new environment variables or settings might be required for you to set in the your docker-compose file. *Always* check the **release notes**: Any breaking changes will be marked there.  
 * All testing and development is done with the Elasticsearch version number as mentioned in the provided *docker-compose.yml* file. This will be updated when a new release of Elasticsearch is available. Running an older version of Elasticsearch is most likely not going to result in any issues, but it's still recommended to run the same version as mentioned. Use `bbilly1/tubearchivist-es` to automatically get the recommended version.

@@ -229,7 +238,7 @@ We have come far, nonetheless we are not short of ideas on how to improve and ex
 - [ ] Show total video downloaded vs total videos available in channel
 - [ ] Add statistics of index
 - [ ] Download speed schedule ([#198](https://github.com/tubearchivist/tubearchivist/issues/198))
- [ ] Auto ignore videos by keyword ([#163](https://github.com/tubearchivist/tubearchivist/issues/163))
+- [ ] Download or Ignore videos by keyword ([#163](https://github.com/tubearchivist/tubearchivist/issues/163))
 - [ ] Custom searchable notes to videos, channels, playlists ([#144](https://github.com/tubearchivist/tubearchivist/issues/144))

 Implemented:
@@ -271,6 +280,18 @@ Second best way to support the development is to provide for caffeinated beverag
 * [ko-fi.com](https://ko-fi.com/bbilly1) for an alternative platform


+## Notable mentions
+This is a selection of places where this project has been featured on reddit, in the news, blogs or any other online media. Sorted by newest on the top.
+* **console.substack.com**: Interview With Simon of Tube Archivist, [2023-01-29] [[link](https://console.substack.com/p/console-142#%C2%A7interview-with-simon-of-tube-archivist)]
+* **reddit.com**: Tube Archivist v0.3.0 - Now Archiving Comments, [2022-12-02] [[link](https://www.reddit.com/r/selfhosted/comments/zaonzp/tube_archivist_v030_now_archiving_comments/)]
+* **reddit.com**: Tube Archivist v0.2 - Now with Full Text Search, [2022-07-24] [[link](https://www.reddit.com/r/selfhosted/comments/w6jfa1/tube_archivist_v02_now_with_full_text_search/)]
+* **noted.lol**: How I Control What Media My Kids Watch Using Tube Archivist, [2022-03-27] [[link](https://noted.lol/how-i-control-what-media-my-kids-watch-using-tube-archivist/)]
+* **thehomelab.wiki**: Tube Archivist - A Youtube-DL Alternative on Steroids, [2022-01-27] [[link](https://thehomelab.wiki/books/news/page/tube-archivist-a-youtube-dl-alternative-on-steroids)]
+* **reddit.com**: Celebrating TubeArchivist v0.1, [2022-01-09] [[link](https://www.reddit.com/r/selfhosted/comments/rzh084/celebrating_tubearchivist_v01/)]
+* **linuxunplugged.com**: Pick: tubearchivist — Your self-hosted YouTube media server, [2021-09-11] [[link](https://linuxunplugged.com/425)] and [2021-10-05] [[link](https://linuxunplugged.com/426)]
+* **reddit.com**: Introducing Tube Archivist, your self hosted Youtube media server, [2021-09-12] [[link](https://www.reddit.com/r/selfhosted/comments/pmj07b/introducing_tube_archivist_your_self_hosted/)]
+
+
 ## Sponsor
 Big thank you to [Digitalocean](https://www.digitalocean.com/) for generously donating credit for the tubearchivist.com VPS and buildserver. 
 <p>
--- a/deploy.sh
+++ b/deploy.sh
@@ -85,14 +85,14 @@ function validate {
    # note: this logic is duplicated in the `./github/workflows/lint_python.yml` config
    # if you update this file, you should update that as well
    echo "running black"
-    black --diff --color --check -l 79 "$check_path"
+    black --exclude "migrations/*" --diff --color --check -l 79 "$check_path"
    echo "running codespell"
    codespell --skip="./.git,./package.json,./package-lock.json,./node_modules" "$check_path"
    echo "running flake8"
-    flake8 "$check_path" --count --max-complexity=10 --max-line-length=79 \
-        --show-source --statistics
+    flake8 "$check_path" --exclude "migrations" --count --max-complexity=10 \
+        --max-line-length=79 --show-source --statistics
    echo "running isort"
-    isort --check-only --diff --profile black -l 79 "$check_path"
+    isort --skip "migrations" --check-only --diff --profile black -l 79 "$check_path"
    printf "    \n> all validations passed\n"

 }
@@ -214,8 +214,6 @@ elif [[ $1 == "test" ]]; then
    sync_test "$2"
 elif [[ $1 == "validate" ]]; then
    validate "$2"
-elif [[ $1 == "versioncheck" ]]; then
-    python version_check.py
 elif [[ $1 == "docker" ]]; then
    sync_docker
 elif [[ $1 == "unstable" ]]; then
@@ -223,7 +221,7 @@ elif [[ $1 == "unstable" ]]; then
 elif [[ $1 == "es" ]]; then
    sync_latest_es
 else
-    echo "valid options are: blackhole | test | validate | versioncheck | docker | unstable | es"
+    echo "valid options are: blackhole | test | validate | docker | unstable | es"
 fi


--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -24,7 +24,7 @@ services:
      - archivist-es
      - archivist-redis
  archivist-redis:
-    image: redislabs/rejson                 # for arm64 use bbilly1/rejson
+    image: redis/redis-stack-server
    container_name: archivist-redis
    restart: unless-stopped
    expose:
@@ -34,7 +34,7 @@ services:
    depends_on:
      - archivist-es
  archivist-es:
-    image: bbilly1/tubearchivist-es         # only for amd64, or use official es 8.6.0
+    image: bbilly1/tubearchivist-es         # only for amd64, or use official es 8.6.2
    container_name: archivist-es
    restart: unless-stopped
    environment:
--- a/docker_assets/run.sh
+++ b/docker_assets/run.sh
@@ -1,71 +1,21 @@
 #!/bin/bash
 # startup script inside the container for tubearchivist

-if [[ -z "$ELASTIC_USER" ]]; then
-    export ELASTIC_USER=elastic
-fi
+set -e

-cachedir=/cache
-[[ -d $cachedir ]] || cachedir=.
-lockfile=${cachedir}/initsu.lock
-
-required="Missing required environment variable"
-[[ -f $lockfile ]] || : "${TA_USERNAME:?$required}"
-: "${TA_PASSWORD:?$required}"
-: "${ELASTIC_PASSWORD:?$required}"
-: "${TA_HOST:?$required}"
-
-# ugly nginx and uwsgi port overwrite with env vars
-if [[ -n "$TA_PORT" ]]; then
-    sed -i "s/8000/$TA_PORT/g" /etc/nginx/sites-available/default
-fi
-
-if [[ -n "$TA_UWSGI_PORT" ]]; then
-    sed -i "s/8080/$TA_UWSGI_PORT/g" /etc/nginx/sites-available/default
-    sed -i "s/8080/$TA_UWSGI_PORT/g" /app/uwsgi.ini
-fi
-
-# disable auth on static files for cast support
-if [[ -n "$ENABLE_CAST" ]]; then
-    sed -i "/auth_request/d" /etc/nginx/sites-available/default
-fi
-
-# wait for elasticsearch
-counter=0
-until curl -u "$ELASTIC_USER":"$ELASTIC_PASSWORD" "$ES_URL" -fs; do
-    echo "waiting for elastic search to start"
-    counter=$((counter+1))
-    if [[ $counter -eq 12 ]]; then
-        # fail after 2 min
-        echo "failed to connect to elastic search, exiting..."
-        curl -v -u "$ELASTIC_USER":"$ELASTIC_PASSWORD" "$ES_URL"?pretty
-        exit 1
-    fi
-    sleep 10
-done
-
-# start python application
-python manage.py makemigrations
+# django setup
 python manage.py migrate

-if [[ -f $lockfile ]]; then
-    echo -e "\e[33;1m[WARNING]\e[0m This is not the first run! Skipping" \
-        "superuser creation.\nTo force it, remove $lockfile"
-else
-    export DJANGO_SUPERUSER_PASSWORD=$TA_PASSWORD
-    output="$(python manage.py createsuperuser --noinput --name "$TA_USERNAME" 2>&1)"
-
-    case "$output" in
-        *"Superuser created successfully"*)
-            echo "$output" && touch $lockfile ;;
-        *"That name is already taken."*)
-            echo "Superuser already exists. Creation will be skipped on next start."
-            touch $lockfile ;;
-        *) echo "$output" && exit 1
-    esac
+if [[ -z "$DJANGO_DEBUG" ]]; then
+    python manage.py collectstatic --noinput -c
 fi

-python manage.py collectstatic --noinput -c
+# ta setup
+python manage.py ta_envcheck
+python manage.py ta_connection
+python manage.py ta_startup
+
+# start all tasks
 nginx &
 celery -A home.tasks worker --loglevel=INFO &
 celery -A home beat --loglevel=INFO \
--- a/docs/Search.md
+++ b/docs/Search.md
@@ -1,31 +1,32 @@
 # Search Page
 Accessible at `/search/` of your **Tube Archivist**, search your archive for Videos, Channels and Playlists - or even full text search throughout your indexed subtitles.

+Just start typing to start a **simple** search *or* **start your query with a primary keyword** to search for a specific type and narrow down the result with secondary keywords. Secondary keywords can be in any order. Use *yes* or *no* for boolean values.
+
+- This will return 30 results per query, pagination is not implemented yet.
 - All your queries are case insensitive and are normalized to lowercase.
 - All your queries are analyzed for the english language, this means *singular*, *plural* and word variations like *-ing*, *-ed*, *-able* etc are treated as synonyms.
+- Keyword value parsing begins with the `keyword:` name all the way until the end of query or the next keyword, e.g. in `video:learn python channel:corey`, the keyword `video` has value `learn python`.
 - Fuzzy search is activated for all your searches by default. This can catch typos in your queries or in the matching documents with one to two letters difference, depending on the query length. You can configure fuzziness with the secondary keyword `fuzzy:`, e.g:
  - `fuzzy:0` or `fuzzy:no`: Deactivate fuzzy matching.
  - `fuzzy:1`: Set fuzziness to one letter difference.
  - `fuzzy:2`: Set fuzziness to two letters difference.
 - All text searches are ranked, meaning the better a match the higher ranked the result. Unless otherwise stated, queries with multiple words are processed with the `and` operator, meaning all words need to match so each word will narrow down the result.
- This will return 30 results per query, pagination is not implemented yet.
-
-Just start typing to start a *simple* search or start your query with a primary keyword to search for a specific type and narrow down the result with secondary keywords. Secondary keywords can be in any order. Use *yes* or *no* for boolean values.

 ## Simple
-Start your query without a keyword to make a simple query. This will search in *video titles*, *channel names* and *playlist titles* and will return matching videos, channels and playlists. Keyword searches will return more results in a particular category due to the fact that more fields are searched for matches.
+Start your query without a keyword to make a simple query (primary keyword `simple:` is implied). This will search in *video titles*, *channel names* and *playlist titles* and will return matching videos, channels and playlists. Keyword searches will return more results in a particular category due to the fact that more fields are searched for matches. Simple queries do not have any secondary keywords.

 ## Video
-Start your query with the primary keyword `video:` to search for videos only. This will search through the *video titles*, *tags* and *category* fields. Narrow your search down with secondary keywords:
+Start your query with the **primary keyword** `video:` to search for videos only. This will search through the *video titles*, *tags* and *category* fields. Narrow your search down with secondary keywords:
 - `channel:` search for videos matching the channel name.
 - `active:` is a boolean value, to search for videos that are still active on youtube or that are not active any more.

 **Example**:
- `video:learn python channel:corey shafer active:yes`: This will return all videos with the term *Learn Python* from the channel *Corey Shafer* that are still *Active* on YouTube.
+- `video:learn python channel:corey schafer active:yes`: This will return all videos with the term *Learn Python* from the channel *Corey Schafer* that are still *Active* on YouTube.
 - `video: channel:tom scott active:no`: Note the omitted term after the primary key, this will show all videos from the channel *Tom Scott* that are no longer active on YouTube.

 ## Channel
-Start with the `channel:` primary keyword to search for channels matching your query. This will search through the *channel name* and *channel description* fields. Narrow your search down with secondary keywords:
+Start with the `channel:` **primary keyword** to search for channels matching your query. This will search through the *channel name* and *channel description* fields. Narrow your search down with secondary keywords:
 - `subscribed:` is a boolean value, search for channels that you are subscribed to or not.
 - `active:` is a boolean value, to search for channels that are still active on YouTube or that are no longer active.

@@ -34,7 +35,7 @@ Start with the `channel:` primary keyword to search for channels matching your q
 - `channel: active:no`: Note the omitted term after the primary key, this will return all channels that are no longer active on YouTube.

 ## Playlist
-Start your query with the primary keyword `playlist:` to search for playlists only. This will search through the *playlist title* and *playlist description* fields. Narrow down your search with these secondary keywords:
+Start your query with the **primary keyword** `playlist:` to search for playlists only. This will search through the *playlist title* and *playlist description* fields. Narrow down your search with these secondary keywords:
 - `subscribed`: is a boolean value, search for playlists that you are subscribed to or not.
 - `active:` is a boolean value, to search for playlists that are still active on YouTube or that are no longer active.

@@ -44,7 +45,7 @@ Start your query with the primary keyword `playlist:` to search for playlists on
 - `playlist:html css active:yes`: Search for playlists containing *HTML CSS* that are still active on YouTube.

 ## Full
-Start a full text search by beginning your query with the primary keyword `full:`. This will search through your indexed Subtitles showing segments with possible matches. This will only show any results if you have activated *subtitle download and index* on the settings page. The operator for full text searches is `or` meaning when searching for multiple words not all words need to match, but additional words will change the ranking of the result, the more words match and the better they match, the higher ranked the result. The matching words will get highlighted in the text preview.
+Start a full text search by beginning your query with the **primary keyword** `full:`. This will search through your indexed Subtitles showing segments with possible matches. This will only show any results if you have activated *subtitle download and index* on the settings page. The operator for full text searches is `or` meaning when searching for multiple words not all words need to match, but additional words will change the ranking of the result, the more words match and the better they match, the higher ranked the result. The matching words will get highlighted in the text preview.

 Clicking the play button on the thumbnail will open the inplace player at the timestamp from where the segment starts. Same when clicking the video title, this will open the video page and put the player at the segment timestamp. This will overwrite any previous playback position.

--- a/docs/Settings.md
+++ b/docs/Settings.md
@@ -121,8 +121,8 @@ The field **Refresh older than x days** takes a number where TubeArchivist will
 ## Thumbnail check
 This will check if all expected thumbnails are there and will delete any artwork without matching video.

-## Index backup
-Create a zip file of the metadata and select **Max auto backups to keep** to automatically delete old backups created from this task.
+## ZIP file index backup
+Create a zip file of the metadata and select **Max auto backups to keep** to automatically delete old backups created from this task. For data consistency, make sure there aren't any other tasks running that will change the index during the backup process. This is very slow, particularly for large archives. Use snapshots instead. 


 # Actions
@@ -166,8 +166,8 @@ If the video you are trying to import is not available on YouTube any more, **Tu
 ## Embed thumbnails into media file
 This will write or overwrite all thumbnails in the media file using the downloaded thumbnail. This is only necessary if you didn't download the files with the option *Embed Thumbnail* enabled or want to make sure all media files get the newest thumbnail. Follow the docker-compose logs to monitor progress.

-## Backup Database
-This will backup your metadata into a zip file. The file will get stored at *cache/backup* and will contain the necessary files to restore the Elasticsearch index formatted **nd-json** files.  
+## ZIP file index backup
+This will backup your metadata into a zip file. The file will get stored at *cache/backup* and will contain the necessary files to restore the Elasticsearch index formatted **nd-json** files. For data consistency, make sure there aren't any other tasks running that will change the index during the backup process. This is very slow, particularly for large archives.  

 BE AWARE: This will **not** backup any media files, just the metadata from the Elasticsearch.

@@ -185,6 +185,7 @@ This function will go through all your media files and looks at the whole index
 - Should the filename not match with the indexed media url, this will rename the video files correctly and update the index with the new link.
 - When you delete media files from the filesystem outside of the Tube Archivist interface, this will delete leftover metadata from the index.
 - When you have media files that are not indexed yet, this will grab the metadata from YouTube like it was a newly downloaded video. This can be useful when restoring from an older backup file with missing metadata but already downloaded mediafiles. NOTE: This only works if the media files are named in the same convention as Tube Archivist does, particularly the YouTube ID needs to be at the same index in the filename, alternatively see above for *Manual Media Files Import*.
-This will also check all of your thumbnails and download any that are missing.
+- The task will stop, when adding a video fails, for example if the video is no longer available on YouTube.
+- This will also check all of your thumbnails and download any that are missing.

 BE AWARE: There is no undo.
--- a/docs/Video.md
+++ b/docs/Video.md
@@ -1,5 +1,5 @@
 # Video Page
-Every video downloaded gets a dedicated page accessible at `/video/<video-id>/` of your Tube Archivist.
+Every video downloaded gets a dedicated page accessible at `/video/<video-id>/` of your Tube Archivist. Throughout the interface, click on a video title to access the video page.

 Clicking on the channel name or the channel icon will bring you to the dedicated channel detail [page](Channels#channel-detail).

--- a/tubearchivist/api/README.md
+++ b/tubearchivist/api/README.md
@@ -155,6 +155,9 @@ Timestamps either *int* or *float*, end time can't be before start time.
 ## Channel List View
 /api/channel/

+Parameter:
+- filter: subscribed
+
 ### Subscribe to a list of channels
 POST /api/channel/
 ```json
--- a/tubearchivist/api/views.py
+++ b/tubearchivist/api/views.py
@@ -289,14 +289,27 @@ class ChannelApiListView(ApiBaseView):
    """

    search_base = "ta_channel/_search/"
+    valid_filter = ["subscribed"]

    def get(self, request):
        """get request"""
-        self.get_document_list(request)
        self.data.update(
            {"sort": [{"channel_name.keyword": {"order": "asc"}}]}
        )

+        query_filter = request.GET.get("filter", False)
+        must_list = []
+        if query_filter:
+            if query_filter not in self.valid_filter:
+                message = f"invalid url query filder: {query_filter}"
+                print(message)
+                return Response({"message": message}, status=400)
+
+            must_list.append({"term": {"channel_subscribed": {"value": True}}})
+
+        self.data["query"] = {"bool": {"must": must_list}}
+        self.get_document_list(request)
+
        return Response(self.response)

    @staticmethod
@@ -430,7 +443,7 @@ class DownloadApiView(ApiBaseView):
        # pylint: disable=unused-argument
        """delete single video from queue"""
        print(f"{video_id}: delete from queue")
-        PendingInteract(video_id=video_id).delete_item()
+        PendingInteract(video_id).delete_item()

        return Response({"success": True})

--- a/tubearchivist/config/management/init.py
+++ b/tubearchivist/config/management/init.py
--- a/tubearchivist/config/management/commands/init.py
+++ b/tubearchivist/config/management/commands/init.py
--- a/tubearchivist/config/management/commands/ta_connection.py
+++ b/tubearchivist/config/management/commands/ta_connection.py
@@ -0,0 +1,140 @@
+"""
+Functionality:
+- check that all connections are working
+"""
+
+from time import sleep
+
+import requests
+from django.core.management.base import BaseCommand, CommandError
+from home.src.es.connect import ElasticWrap
+from home.src.ta.ta_redis import RedisArchivist
+
+TOPIC = """
+
+#######################
+#  Connection check   #
+#######################
+
+"""
+
+
+class Command(BaseCommand):
+    """command framework"""
+
+    TIMEOUT = 120
+    MIN_MAJOR, MAX_MAJOR = 8, 8
+    MIN_MINOR = 0
+
+    # pylint: disable=no-member
+    help = "Check connections"
+
+    def handle(self, *args, **options):
+        """run all commands"""
+        self.stdout.write(TOPIC)
+        self._redis_connection_check()
+        self._redis_config_set()
+        self._es_connection_check()
+        self._es_version_check()
+        self._es_path_check()
+
+    def _redis_connection_check(self):
+        """check ir redis connection is established"""
+        self.stdout.write("[1] connect to Redis")
+        redis_conn = RedisArchivist().conn
+        for _ in range(5):
+            sleep(2)
+            try:
+                pong = redis_conn.execute_command("PING")
+                if pong:
+                    self.stdout.write(
+                        self.style.SUCCESS("    ✓ Redis connection verified")
+                    )
+                    return
+
+            except Exception:  # pylint: disable=broad-except
+                self.stdout.write("    ... retry Redis connection")
+
+        message = "    🗙 Redis connection failed"
+        self.stdout.write(self.style.ERROR(f"{message}"))
+        RedisArchivist().exec("PING")
+        raise CommandError(message)
+
+    def _redis_config_set(self):
+        """set config for redis if not set already"""
+        self.stdout.write("[2] set Redis config")
+        redis_conn = RedisArchivist().conn
+        timeout_is = int(redis_conn.config_get("timeout").get("timeout"))
+        if not timeout_is:
+            redis_conn.config_set("timeout", 3600)
+
+        self.stdout.write(self.style.SUCCESS("    ✓ Redis config set"))
+
+    def _es_connection_check(self):
+        """wait for elasticsearch connection"""
+        self.stdout.write("[3] connect to Elastic Search")
+        total = self.TIMEOUT // 5
+        for i in range(total):
+            sleep(5)
+            self.stdout.write(f"    ... waiting for ES [{i}/{total}]")
+            try:
+                _, status_code = ElasticWrap("/").get(
+                    timeout=1, print_error=False
+                )
+            except requests.exceptions.ConnectionError:
+                continue
+
+            if status_code and status_code == 200:
+                self.stdout.write(
+                    self.style.SUCCESS("    ✓ ES connection established")
+                )
+                return
+
+        response, status_code = ElasticWrap("/").get(
+            timeout=1, print_error=False
+        )
+
+        message = "    🗙 ES connection failed"
+        self.stdout.write(self.style.ERROR(f"{message}"))
+        self.stdout.write(f"    error message: {response}")
+        self.stdout.write(f"    status code: {status_code}")
+        raise CommandError(message)
+
+    def _es_version_check(self):
+        """check for minimal elasticsearch version"""
+        self.stdout.write("[4] Elastic Search version check")
+        response, _ = ElasticWrap("/").get()
+        version = response["version"]["number"]
+        major = int(version.split(".")[0])
+
+        if self.MIN_MAJOR <= major <= self.MAX_MAJOR:
+            self.stdout.write(
+                self.style.SUCCESS("    ✓ ES version check passed")
+            )
+            return
+
+        message = (
+            "    🗙 ES connection failed. "
+            + f"Expected {self.MIN_MAJOR}.{self.MIN_MINOR} but got {version}"
+        )
+        self.stdout.write(self.style.ERROR(f"{message}"))
+        raise CommandError(message)
+
+    def _es_path_check(self):
+        """check that path.repo var is set"""
+        self.stdout.write("[5] check ES path.repo env var")
+        response, _ = ElasticWrap("_nodes/_all/settings").get()
+        for node in response["nodes"].values():
+            if node["settings"]["path"].get("repo"):
+                self.stdout.write(
+                    self.style.SUCCESS("    ✓ path.repo env var is set")
+                )
+                return
+
+            message = (
+                "    🗙 path.repo env var not found. "
+                + "set the following env var to the ES container:\n"
+                + "    path.repo=/usr/share/elasticsearch/data/snapshot"
+            )
+            self.stdout.write(self.style.ERROR(f"{message}"))
+            raise CommandError(message)
--- a/tubearchivist/config/management/commands/ta_envcheck.py
+++ b/tubearchivist/config/management/commands/ta_envcheck.py
@@ -0,0 +1,196 @@
+"""
+Functionality:
+- Check environment at startup
+- Process config file overwrites from env var
+- Stop startup on error
+- python management.py ta_envcheck
+"""
+
+import os
+import re
+
+from django.core.management.base import BaseCommand, CommandError
+from home.models import Account
+
+LOGO = """
+
+                         ....  .....
+                  ...'',;:cc,. .;::;;,'...
+               ..,;:cccllclc,  .:ccllllcc;,..
+            ..,:cllcc:;,'.',.  ....'',;ccllc:,..
+          ..;cllc:,'..                ...,:cccc:'.
+         .;cccc;..                        ..,:ccc:'.
+       .ckkkOkxollllllllllllc.      .,:::;.  .,cclc;
+      .:0MMMMMMMMMMMMMMMMMMMX:     .cNMMMWx.   .;clc:
+     .;lOXK0000KNMMMMX00000KO;     ;KMMMMMNl.   .;ccl:,.
+     .;:c:'.....kMMMNo........    'OMMMWMMMK:    '::;;'.
+   .......     .xMMMNl           .dWMMXdOMMMO'   ........
+   .:cc:;.     .xMMMNc          .lNMMNo.:XMMWx.    .:cl:.
+   .:llc,.     .:xxxd,          ;KMMMk. .oWMMNl.   .:llc'
+   .cll:.     .;:;;:::,.       'OMMMK:';''kWMMK:   .;llc,
+   .cll:.     .,;;;;;;,.     .,xWMMNl.:l:.;KMMMO'  .;llc'
+   .:llc.      .cOOOk;      .lKNMMWx..:l:..lNMMWx. .:llc'
+   .;lcc,.     .xMMMNc      :KMMMM0, .:lc. .xWMMNl.'ccl:.
+    .cllc.     .xMMMNc     'OMMMMXc...:lc...,0MMMKl:lcc,.
+    .,ccl:.    .xMMMNc    .xWMMMWo.,;;:lc;;;.cXMMMXdcc;.
+     .,clc:.   .xMMMNc   .lNMMMWk. .':clc:,. .dWMMW0o;.
+      .,clcc,. .ckkkx;   .okkkOx,    .';,.    'kKKK0l.
+       .':lcc:'.....      .  ..            ..,;cllc,.
+         .,cclc,....                     ....;clc;..
+          ..,:,..,c:'..              ...';:,..,:,.
+            ....:lcccc:;,'''.....'',;;:clllc,....
+               .'',;:cllllllccccclllllcc:,'..
+                   ...'',,;;;;;;;;;,''...
+                            .....
+
+"""
+
+TOPIC = """
+#######################
+#  Environment Setup  #
+#######################
+
+"""
+
+EXPECTED_ENV_VARS = [
+    "TA_USERNAME",
+    "TA_PASSWORD",
+    "ELASTIC_PASSWORD",
+    "ES_URL",
+    "TA_HOST",
+]
+INST = "https://github.com/tubearchivist/tubearchivist#installing-and-updating"
+NGINX = "/etc/nginx/sites-available/default"
+UWSGI = "/app/uwsgi.ini"
+
+
+class Command(BaseCommand):
+    """command framework"""
+
+    # pylint: disable=no-member
+    help = "Check environment before startup"
+
+    def handle(self, *args, **options):
+        """run all commands"""
+        self.stdout.write(LOGO)
+        self.stdout.write(TOPIC)
+        self._expected_vars()
+        self._elastic_user_overwrite()
+        self._ta_port_overwrite()
+        self._ta_uwsgi_overwrite()
+        self._enable_cast_overwrite()
+        self._create_superuser()
+
+    def _expected_vars(self):
+        """check if expected env vars are set"""
+        self.stdout.write("[1] checking expected env vars")
+        env = os.environ
+        for var in EXPECTED_ENV_VARS:
+            if not env.get(var):
+                message = f"    🗙 expected env var {var} not set\n    {INST}"
+                self.stdout.write(self.style.ERROR(message))
+                raise CommandError(message)
+
+        message = "    ✓ all expected env vars are set"
+        self.stdout.write(self.style.SUCCESS(message))
+
+    def _elastic_user_overwrite(self):
+        """check for ELASTIC_USER overwrite"""
+        self.stdout.write("[2] set default ES user")
+        if not os.environ.get("ELASTIC_USER"):
+            os.environ.setdefault("ELASTIC_USER", "elastic")
+
+        env = os.environ.get("ELASTIC_USER")
+
+        self.stdout.write(self.style.SUCCESS(f"    ✓ ES user is set to {env}"))
+
+    def _ta_port_overwrite(self):
+        """set TA_PORT overwrite for nginx"""
+        self.stdout.write("[3] check TA_PORT overwrite")
+        overwrite = os.environ.get("TA_PORT")
+        if not overwrite:
+            self.stdout.write(self.style.SUCCESS("    TA_PORT is not set"))
+            return
+
+        regex = re.compile(r"listen [0-9]{1,5}")
+        to_overwrite = f"listen {overwrite}"
+        changed = file_overwrite(NGINX, regex, to_overwrite)
+        if changed:
+            message = f"    ✓ TA_PORT changed to {overwrite}"
+        else:
+            message = f"    ✓ TA_PORT already set to {overwrite}"
+
+        self.stdout.write(self.style.SUCCESS(message))
+
+    def _ta_uwsgi_overwrite(self):
+        """set TA_UWSGI_PORT overwrite"""
+        self.stdout.write("[4] check TA_UWSGI_PORT overwrite")
+        overwrite = os.environ.get("TA_UWSGI_PORT")
+        if not overwrite:
+            message = "    TA_UWSGI_PORT is not set"
+            self.stdout.write(self.style.SUCCESS(message))
+            return
+
+        # nginx
+        regex = re.compile(r"uwsgi_pass localhost:[0-9]{1,5}")
+        to_overwrite = f"uwsgi_pass localhost:{overwrite}"
+        changed = file_overwrite(NGINX, regex, to_overwrite)
+
+        # uwsgi
+        regex = re.compile(r"socket = :[0-9]{1,5}")
+        to_overwrite = f"socket = :{overwrite}"
+        changed = file_overwrite(UWSGI, regex, to_overwrite)
+
+        if changed:
+            message = f"    ✓ TA_UWSGI_PORT changed to {overwrite}"
+        else:
+            message = f"    ✓ TA_UWSGI_PORT already set to {overwrite}"
+
+        self.stdout.write(self.style.SUCCESS(message))
+
+    def _enable_cast_overwrite(self):
+        """cast workaround, remove auth for static files in nginx"""
+        self.stdout.write("[5] check ENABLE_CAST overwrite")
+        overwrite = os.environ.get("ENABLE_CAST")
+        if not overwrite:
+            self.stdout.write(self.style.SUCCESS("    ENABLE_CAST is not set"))
+            return
+
+        regex = re.compile(r"[^\S\r\n]*auth_request /api/ping/;\n")
+        changed = file_overwrite(NGINX, regex, "")
+        if changed:
+            message = "    ✓ process nginx to enable Cast"
+        else:
+            message = "    ✓ Cast is already enabled in nginx"
+
+        self.stdout.write(self.style.SUCCESS(message))
+
+    def _create_superuser(self):
+        """create superuser if not exist"""
+        self.stdout.write("[6] create superuser")
+        is_created = Account.objects.filter(is_superuser=True)
+        if is_created:
+            message = "    superuser already created"
+            self.stdout.write(self.style.SUCCESS(message))
+            return
+
+        name = os.environ.get("TA_USERNAME")
+        password = os.environ.get("TA_PASSWORD")
+        Account.objects.create_superuser(name, password)
+        message = f"    ✓ new superuser with name {name} created"
+        self.stdout.write(self.style.SUCCESS(message))
+
+
+def file_overwrite(file_path, regex, overwrite):
+    """change file content from old to overwrite, return true when changed"""
+    with open(file_path, "r", encoding="utf-8") as f:
+        file_content = f.read()
+
+    changed = re.sub(regex, overwrite, file_content)
+    if changed == file_content:
+        return False
+
+    with open(file_path, "w", encoding="utf-8") as f:
+        f.write(changed)
+
+    return True
--- a/tubearchivist/config/management/commands/ta_startup.py
+++ b/tubearchivist/config/management/commands/ta_startup.py
@@ -0,0 +1,178 @@
+"""
+Functionality:
+- Application startup
+- Apply migrations
+"""
+
+import os
+from time import sleep
+
+from django.core.management.base import BaseCommand, CommandError
+from home.src.es.connect import ElasticWrap
+from home.src.es.index_setup import ElasitIndexWrap
+from home.src.es.snapshot import ElasticSnapshot
+from home.src.ta.config import AppConfig, ReleaseVersion
+from home.src.ta.helper import clear_dl_cache
+from home.src.ta.ta_redis import RedisArchivist
+
+TOPIC = """
+
+#######################
+#  Application Start  #
+#######################
+
+"""
+
+
+class Command(BaseCommand):
+    """command framework"""
+
+    # pylint: disable=no-member
+
+    def handle(self, *args, **options):
+        """run all commands"""
+        self.stdout.write(TOPIC)
+        self._sync_redis_state()
+        self._make_folders()
+        self._release_locks()
+        self._clear_dl_cache()
+        self._version_check()
+        self._mig_index_setup()
+        self._mig_snapshot_check()
+        self._mig_set_vid_type()
+
+    def _sync_redis_state(self):
+        """make sure redis gets new config.json values"""
+        self.stdout.write("[1] set new config.json values")
+        needs_update = AppConfig().load_new_defaults()
+        if needs_update:
+            self.stdout.write(
+                self.style.SUCCESS("    ✓ new config values set")
+            )
+        else:
+            self.stdout.write(self.style.SUCCESS("    no new config values"))
+
+    def _make_folders(self):
+        """make expected cache folders"""
+        self.stdout.write("[2] create expected cache folders")
+        folders = [
+            "backup",
+            "channels",
+            "download",
+            "import",
+            "playlists",
+            "videos",
+        ]
+        cache_dir = AppConfig().config["application"]["cache_dir"]
+        for folder in folders:
+            folder_path = os.path.join(cache_dir, folder)
+            os.makedirs(folder_path, exist_ok=True)
+
+        self.stdout.write(self.style.SUCCESS("    ✓ expected folders created"))
+
+    def _release_locks(self):
+        """make sure there are no leftover locks set in redis"""
+        self.stdout.write("[3] clear leftover locks in redis")
+        all_locks = [
+            "dl_queue_id",
+            "dl_queue",
+            "downloading",
+            "manual_import",
+            "reindex",
+            "rescan",
+            "run_backup",
+            "startup_check",
+        ]
+
+        redis_con = RedisArchivist()
+        has_changed = False
+        for lock in all_locks:
+            if redis_con.del_message(lock):
+                self.stdout.write(
+                    self.style.SUCCESS(f"    ✓ cleared lock {lock}")
+                )
+                has_changed = True
+
+        if not has_changed:
+            self.stdout.write(self.style.SUCCESS("    no locks found"))
+
+    def _clear_dl_cache(self):
+        """clear leftover files from dl cache"""
+        self.stdout.write("[4] clear leftover files from dl cache")
+        config = AppConfig().config
+        leftover_files = clear_dl_cache(config)
+        if leftover_files:
+            self.stdout.write(
+                self.style.SUCCESS(f"    ✓ cleared {leftover_files} files")
+            )
+        else:
+            self.stdout.write(self.style.SUCCESS("    no files found"))
+
+    def _version_check(self):
+        """remove new release key if updated now"""
+        self.stdout.write("[5] check for first run after update")
+        new_version = ReleaseVersion().is_updated()
+        if new_version:
+            self.stdout.write(
+                self.style.SUCCESS(f"    ✓ update to {new_version} completed")
+            )
+        else:
+            self.stdout.write(self.style.SUCCESS("    no new update found"))
+
+    def _mig_index_setup(self):
+        """migration: validate index mappings"""
+        self.stdout.write("[MIGRATION] validate index mappings")
+        ElasitIndexWrap().setup()
+
+    def _mig_snapshot_check(self):
+        """migration setup snapshots"""
+        self.stdout.write("[MIGRATION] setup snapshots")
+        ElasticSnapshot().setup()
+
+    def _mig_set_vid_type(self):
+        """migration: update 0.3.0 to 0.3.1 set vid_type default"""
+        self.stdout.write("[MIGRATION] set default vid_type")
+        index_list = ["ta_video", "ta_download"]
+        data = {
+            "query": {
+                "bool": {
+                    "should": [
+                        {
+                            "bool": {
+                                "must_not": [{"exists": {"field": "vid_type"}}]
+                            }
+                        },
+                        {"term": {"vid_type": {"value": "unknown"}}},
+                    ]
+                }
+            },
+            "script": {"source": "ctx._source['vid_type'] = 'videos'"},
+        }
+
+        for index_name in index_list:
+            path = f"{index_name}/_update_by_query"
+            response, status_code = ElasticWrap(path).post(data=data)
+            if status_code == 503:
+                message = f"    🗙 {index_name} retry failed migration."
+                self.stdout.write(self.style.ERROR(message))
+                sleep(10)
+                response, status_code = ElasticWrap(path).post(data=data)
+
+            if status_code == 200:
+                updated = response.get("updated", 0)
+                if not updated:
+                    self.stdout.write(
+                        f"    no videos needed updating in {index_name}"
+                    )
+                    continue
+
+                self.stdout.write(
+                    self.style.SUCCESS(
+                        f"    ✓ {updated} videos updated in {index_name}"
+                    )
+                )
+            else:
+                message = f"    🗙 {index_name} vid_type update failed"
+                self.stdout.write(self.style.ERROR(message))
+                self.stdout.write(response)
+                raise CommandError(message)
--- a/tubearchivist/config/settings.py
+++ b/tubearchivist/config/settings.py
@@ -32,7 +32,9 @@ SECRET_KEY = PW_HASH.hexdigest()
 # SECURITY WARNING: don't run with debug turned on in production!
 DEBUG = bool(environ.get("DJANGO_DEBUG"))

-ALLOWED_HOSTS = [i.strip() for i in environ.get("TA_HOST").split()]
+ALLOWED_HOSTS = []
+if environ.get("TA_HOST"):
+    ALLOWED_HOSTS = [i.strip() for i in environ.get("TA_HOST").split()]

 CSRF_TRUSTED_ORIGINS = []
 for host in ALLOWED_HOSTS:
@@ -58,6 +60,7 @@ INSTALLED_APPS = [
    "rest_framework",
    "rest_framework.authtoken",
    "api",
+    "config",
 ]

 MIDDLEWARE = [
@@ -262,4 +265,4 @@ CORS_ALLOW_HEADERS = list(default_headers) + [

 # TA application settings
 TA_UPSTREAM = "https://github.com/tubearchivist/tubearchivist"
-TA_VERSION = "v0.3.2"
+TA_VERSION = "v0.3.4"
--- a/tubearchivist/home/apps.py
+++ b/tubearchivist/home/apps.py
@@ -1,163 +1,6 @@
 """handle custom startup functions"""

-import os
-import sys
-
 from django.apps import AppConfig
-from home.src.es.connect import ElasticWrap
-from home.src.es.index_setup import ElasitIndexWrap
-from home.src.es.snapshot import ElasticSnapshot
-from home.src.ta.config import AppConfig as ArchivistConfig
-from home.src.ta.config import ReleaseVersion
-from home.src.ta.helper import clear_dl_cache
-from home.src.ta.ta_redis import RedisArchivist
-
-
-class StartupCheck:
-    """checks to run at application startup"""
-
-    MIN_MAJOR, MAX_MAJOR = 8, 8
-    MIN_MINOR = 0
-
-    def __init__(self):
-        self.config_handler = ArchivistConfig()
-        self.redis_con = RedisArchivist()
-        self.has_run = self.get_has_run()
-
-    def run(self):
-        """run all startup checks"""
-        print("run startup checks")
-        self.set_lock()
-        self.es_version_check()
-        self.release_lock()
-        self.sync_redis_state()
-        self.set_redis_conf()
-        ElasitIndexWrap().setup()
-        self.make_folders()
-        clear_dl_cache(self.config_handler.config)
-        self.snapshot_check()
-        self.ta_version_check()
-        self.es_set_vid_type()
-        self.expire_lock()
-
-    def get_has_run(self):
-        """validate if check has already executed"""
-        return self.redis_con.get_message("startup_check")
-
-    def set_lock(self):
-        """set lock to avoid executing once per thread"""
-        self.redis_con.set_message("startup_check", message={"status": True})
-
-    def expire_lock(self):
-        """startup checks run"""
-        print("startup checks completed")
-        message = {"status": True}
-        self.redis_con.set_message("startup_check", message, expire=120)
-
-    def sync_redis_state(self):
-        """make sure redis gets new config.json values"""
-        print("sync redis")
-        self.config_handler.load_new_defaults()
-
-    def set_redis_conf(self):
-        """set conf values for redis"""
-        self.redis_con.conn.config_set("timeout", 3600)
-
-    def make_folders(self):
-        """make needed cache folders here so docker doesn't mess it up"""
-        folders = [
-            "download",
-            "channels",
-            "videos",
-            "playlists",
-            "import",
-            "backup",
-        ]
-        cache_dir = self.config_handler.config["application"]["cache_dir"]
-        for folder in folders:
-            folder_path = os.path.join(cache_dir, folder)
-            os.makedirs(folder_path, exist_ok=True)
-
-    def release_lock(self):
-        """make sure there are no leftover locks set in redis"""
-        all_locks = [
-            "manual_import",
-            "downloading",
-            "dl_queue",
-            "dl_queue_id",
-            "reindex",
-            "rescan",
-            "run_backup",
-        ]
-        for lock in all_locks:
-            response = self.redis_con.del_message(lock)
-            if response:
-                print("deleted leftover key from redis: " + lock)
-
-    def snapshot_check(self):
-        """setup snapshot config, create if needed"""
-        app = self.config_handler.config["application"]
-        if not app.get("enable_snapshot"):
-            return
-
-        ElasticSnapshot().setup()
-
-    def is_invalid(self, version):
-        """return true if es version is invalid, false if ok"""
-        major, minor = [int(i) for i in version.split(".")[:2]]
-        if not self.MIN_MAJOR <= major <= self.MAX_MAJOR:
-            return True
-
-        if minor >= self.MIN_MINOR:
-            return False
-
-        return True
-
-    def es_version_check(self):
-        """check for minimal elasticsearch version"""
-        response, _ = ElasticWrap("/").get()
-        version = response["version"]["number"]
-        invalid = self.is_invalid(version)
-
-        if invalid:
-            print(
-                "required elasticsearch version: "
-                + f"{self.MIN_MAJOR}.{self.MIN_MINOR}"
-            )
-            sys.exit(1)
-
-        print("elasticsearch version check passed")
-
-    def ta_version_check(self):
-        """remove key if updated now"""
-        ReleaseVersion().is_updated()
-
-    def es_set_vid_type(self):
-        """
-        update path 0.3.0 to 0.3.1, set default vid_type to video
-        fix unidentified vids in unstable
-        """
-        index_list = ["ta_video", "ta_download"]
-        data = {
-            "query": {
-                "bool": {
-                    "should": [
-                        {
-                            "bool": {
-                                "must_not": [{"exists": {"field": "vid_type"}}]
-                            }
-                        },
-                        {"term": {"vid_type": {"value": "unknown"}}},
-                    ]
-                }
-            },
-            "script": {"source": "ctx._source['vid_type'] = 'videos'"},
-        }
-
-        for index_name in index_list:
-            path = f"{index_name}/_update_by_query"
-            response, _ = ElasticWrap(path).post(data=data)
-            print(f"{index_name} vid_type index update ran: {response}")


 class HomeConfig(AppConfig):
@@ -165,11 +8,3 @@ class HomeConfig(AppConfig):

    default_auto_field = "django.db.models.BigAutoField"
    name = "home"
-
-    def ready(self):
-        startup = StartupCheck()
-        if startup.has_run["status"]:
-            print("startup checks run in other thread")
-            return
-
-        startup.run()
--- a/tubearchivist/home/config.json
+++ b/tubearchivist/home/config.json
@@ -42,7 +42,7 @@
        "videos": "/youtube",
        "colors": "dark",
        "enable_cast": false,
-        "enable_snapshot": false
+        "enable_snapshot": true
    },
    "scheduler": {
        "update_subscribed": false,
@@ -50,7 +50,7 @@
        "check_reindex":  {"minute": "0", "hour": "12", "day_of_week": "*"},
        "check_reindex_days": 90,
        "thumbnail_check": {"minute": "0", "hour": "17", "day_of_week": "*"},
-        "run_backup": {"minute": "0", "hour": "8", "day_of_week": "0"},
+        "run_backup": false,
        "run_backup_rotate": 5,
        "version_check": "rand-d"
    }
--- a/tubearchivist/home/migrations/0001_initial.py
+++ b/tubearchivist/home/migrations/0001_initial.py
@@ -0,0 +1,35 @@
+# Generated by Django 4.1.5 on 2023-02-02 06:49
+
+from django.db import migrations, models
+import home.models
+
+
+class Migration(migrations.Migration):
+
+    initial = True
+
+    dependencies = [
+        ('auth', '0012_alter_user_first_name_max_length'),
+    ]
+
+    operations = [
+        migrations.CreateModel(
+            name='Account',
+            fields=[
+                ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+                ('password', models.CharField(max_length=128, verbose_name='password')),
+                ('last_login', models.DateTimeField(blank=True, null=True, verbose_name='last login')),
+                ('is_superuser', models.BooleanField(default=False, help_text='Designates that this user has all permissions without explicitly assigning them.', verbose_name='superuser status')),
+                ('name', models.CharField(max_length=150, unique=True)),
+                ('is_staff', models.BooleanField(default=False)),
+                ('groups', models.ManyToManyField(blank=True, help_text='The groups this user belongs to. A user will get all permissions granted to each of their groups.', related_name='user_set', related_query_name='user', to='auth.group', verbose_name='groups')),
+                ('user_permissions', models.ManyToManyField(blank=True, help_text='Specific permissions for this user.', related_name='user_set', related_query_name='user', to='auth.permission', verbose_name='user permissions')),
+            ],
+            options={
+                'abstract': False,
+            },
+            managers=[
+                ('objects', home.models.AccountManager()),
+            ],
+        ),
+    ]
--- a/tubearchivist/home/src/download/queue.py
+++ b/tubearchivist/home/src/download/queue.py
@@ -96,13 +96,13 @@ class PendingIndex:
 class PendingInteract:
    """interact with items in download queue"""

-    def __init__(self, video_id=False, status=False):
-        self.video_id = video_id
+    def __init__(self, youtube_id=False, status=False):
+        self.youtube_id = youtube_id
        self.status = status

    def delete_item(self):
        """delete single item from pending"""
-        path = f"ta_download/_doc/{self.video_id}"
+        path = f"ta_download/_doc/{self.youtube_id}"
        _, _ = ElasticWrap(path).delete(refresh=True)

    def delete_by_status(self):
@@ -114,27 +114,43 @@ class PendingInteract:
    def update_status(self):
        """update status field of pending item"""
        data = {"doc": {"status": self.status}}
-        path = f"ta_download/_update/{self.video_id}"
+        path = f"ta_download/_update/{self.youtube_id}"
        _, _ = ElasticWrap(path).post(data=data)

    def get_item(self):
        """return pending item dict"""
-        path = f"ta_download/_doc/{self.video_id}"
+        path = f"ta_download/_doc/{self.youtube_id}"
        response, status_code = ElasticWrap(path).get()
        return response["_source"], status_code

+    def get_channel(self):
+        """
+        get channel metadata from queue to not depend on channel to be indexed
+        """
+        data = {
+            "size": 1,
+            "query": {"term": {"channel_id": {"value": self.youtube_id}}},
+        }
+        response, _ = ElasticWrap("ta_download/_search").get(data=data)
+        hits = response["hits"]["hits"]
+        if not hits:
+            channel_name = "NA"
+        else:
+            channel_name = hits[0]["_source"].get("channel_name", "NA")
+
+        return {
+            "channel_id": self.youtube_id,
+            "channel_name": channel_name,
+        }
+

 class PendingList(PendingIndex):
    """manage the pending videos list"""

    yt_obs = {
-        "default_search": "ytsearch",
-        "quiet": True,
-        "check_formats": "selected",
        "noplaylist": True,
        "writethumbnail": True,
        "simulate": True,
-        "socket_timeout": 3,
    }

    def __init__(self, youtube_ids=False):
@@ -214,6 +230,7 @@ class PendingList(PendingIndex):

        for idx, (youtube_id, vid_type) in enumerate(self.missing_videos):
            print(f"{youtube_id} ({vid_type}): add to download queue")
+            self._notify_add(idx)
            video_details = self.get_youtube_details(youtube_id, vid_type)
            if not video_details:
                continue
@@ -226,8 +243,6 @@ class PendingList(PendingIndex):
            url = video_details["vid_thumb_url"]
            ThumbManager(youtube_id).download_video_thumb(url)

-            self._notify_add(idx)
-
        if bulk_list:
            # add last newline
            bulk_list.append("\n")
@@ -264,6 +279,7 @@ class PendingList(PendingIndex):
            return False
        # stop if video is streaming live now
        if vid["live_status"] in ["is_upcoming", "is_live"]:
+            print(f"{youtube_id}: skip is_upcoming or is_live")
            return False

        if vid["live_status"] == "was_live":
--- a/tubearchivist/home/src/download/thumbnails.py
+++ b/tubearchivist/home/src/download/thumbnails.py
@@ -43,8 +43,12 @@ class ThumbManagerBase:
                response = requests.get(url, stream=True, timeout=5)
                if response.ok:
                    try:
-                        return Image.open(response.raw)
-                    except UnidentifiedImageError:
+                        img = Image.open(response.raw)
+                        if isinstance(img, Image.Image):
+                            return img
+                        return self.get_fallback()
+
+                    except (UnidentifiedImageError, OSError):
                        print(f"failed to open thumbnail: {url}")
                        return self.get_fallback()

@@ -59,6 +63,7 @@ class ThumbManagerBase:

    def get_fallback(self):
        """get fallback thumbnail if not available"""
+        print(f"{self.item_id}: failed to extract thumbnail, use fallback")
        if self.fallback:
            img_raw = Image.open(self.fallback)
            return img_raw
--- a/tubearchivist/home/src/download/yt_dlp_base.py
+++ b/tubearchivist/home/src/download/yt_dlp_base.py
@@ -20,8 +20,9 @@ class YtWrap:
        "default_search": "ytsearch",
        "quiet": True,
        "check_formats": "selected",
-        "socket_timeout": 3,
+        "socket_timeout": 10,
        "extractor_retries": 3,
+        "retries": 10,
    }

    def __init__(self, obs_request, config=False):
--- a/tubearchivist/home/src/download/yt_dlp_handler.py
+++ b/tubearchivist/home/src/download/yt_dlp_handler.py
@@ -192,7 +192,7 @@ class VideoDownloader:
                "vid_type", VideoTypeEnum.VIDEOS.value
            )
            video_type = VideoTypeEnum(tmp_vid_type)
-            print(f"Downloading type: {video_type}")
+            print(f"{youtube_id}: Downloading type: {video_type}")

            success = self._dl_single_vid(youtube_id)
            if not success:
@@ -204,7 +204,7 @@ class VideoDownloader:
                "title": "Indexing....",
                "message": "Add video metadata to index.",
            }
-            RedisArchivist().set_message(self.MSG, mess_dict, expire=60)
+            RedisArchivist().set_message(self.MSG, mess_dict, expire=120)

            vid_dict = index_new_video(
                youtube_id,
@@ -223,8 +223,10 @@ class VideoDownloader:

            if queue.has_item():
                message = "Continue with next video."
+                expire = False
            else:
                message = "Download queue is finished."
+                expire = 10

            self.move_to_archive(vid_dict)
            mess_dict = {
@@ -233,7 +235,7 @@ class VideoDownloader:
                "title": "Completed",
                "message": message,
            }
-            RedisArchivist().set_message(self.MSG, mess_dict, expire=10)
+            RedisArchivist().set_message(self.MSG, mess_dict, expire=expire)
            self._delete_from_pending(youtube_id)

        # post processing
@@ -260,7 +262,7 @@ class VideoDownloader:
            "title": "Looking for videos to download",
            "message": "Scanning your download queue.",
        }
-        RedisArchivist().set_message(self.MSG, mess_dict, expire=True)
+        RedisArchivist().set_message(self.MSG, mess_dict)
        pending = PendingList()
        pending.get_download()
        to_add = [
@@ -293,8 +295,11 @@ class VideoDownloader:
        title = "Downloading: " + response["info_dict"]["title"]

        try:
+            size = response.get("_total_bytes_str")
+            if size.strip() == "N/A":
+                size = response.get("_total_bytes_estimate_str", "N/A")
+
            percent = response["_percent_str"]
-            size = response["_total_bytes_str"]
            speed = response["_speed_str"]
            eta = response["_eta_str"]
            message = f"{percent} of {size} at {speed} - time left: {eta}"
@@ -318,7 +323,6 @@ class VideoDownloader:
    def _build_obs_basic(self):
        """initial obs"""
        self.obs = {
-            "default_search": "ytsearch",
            "merge_output_format": "mp4",
            "outtmpl": (
                self.config["application"]["cache_dir"]
@@ -326,13 +330,9 @@ class VideoDownloader:
            ),
            "progress_hooks": [self._progress_hook],
            "noprogress": True,
-            "quiet": True,
            "continuedl": True,
-            "retries": 3,
            "writethumbnail": False,
            "noplaylist": True,
-            "check_formats": "selected",
-            "socket_timeout": 3,
        }

    def _build_obs_user(self):
--- a/tubearchivist/home/src/es/backup.py
+++ b/tubearchivist/home/src/es/backup.py
@@ -143,7 +143,6 @@ class ElasticBackup:
        backup_dir = os.path.join(self.cache_dir, "backup")

        for json_f in zip_content:
-
            file_name = os.path.join(backup_dir, json_f)

            if not json_f.startswith("es_") or not json_f.endswith(".json"):
--- a/tubearchivist/home/src/es/connect.py
+++ b/tubearchivist/home/src/es/connect.py
@@ -32,15 +32,15 @@ class ElasticWrap:
        self.auth = self.config["application"]["es_auth"]
        self.url = f"{es_url}/{self.path}"

-    def get(self, data=False):
+    def get(self, data=False, timeout=10, print_error=True):
        """get data from es"""
        if data:
            response = requests.get(
-                self.url, json=data, auth=self.auth, timeout=10
+                self.url, json=data, auth=self.auth, timeout=timeout
            )
        else:
-            response = requests.get(self.url, auth=self.auth, timeout=10)
-        if not response.ok:
+            response = requests.get(self.url, auth=self.auth, timeout=timeout)
+        if print_error and not response.ok:
            print(response.text)

        return response.json(), response.status_code
--- a/tubearchivist/home/src/index/comments.py
+++ b/tubearchivist/home/src/index/comments.py
@@ -80,7 +80,6 @@ class Comments:

        yt_obs = {
            "skip_download": True,
-            "quiet": False,
            "getcomments": True,
            "extractor_args": {
                "youtube": {
@@ -110,12 +109,20 @@ class Comments:
        if comments_raw:
            for comment in comments_raw:
                cleaned_comment = self.clean_comment(comment)
+                if not cleaned_comment:
+                    continue
+
                comments.append(cleaned_comment)

        self.comments_format = comments

    def clean_comment(self, comment):
        """parse metadata from comment for indexing"""
+        if not comment.get("text"):
+            # comment text can be empty
+            print(f"{self.youtube_id}: Failed to extract text, {comment}")
+            return False
+
        time_text_datetime = datetime.utcfromtimestamp(comment["timestamp"])

        if time_text_datetime.hour == 0 and time_text_datetime.minute == 0:
@@ -203,6 +210,9 @@ class CommentList:
            return

        total_videos = len(self.video_ids)
+        if notify:
+            self._notify(f"add comments for {total_videos} videos", False)
+
        for idx, video_id in enumerate(self.video_ids):
            comment = Comments(video_id, config=self.config)
            if notify:
@@ -212,16 +222,16 @@ class CommentList:
                comment.upload_comments()

        if notify:
-            self.notify_final(total_videos)
+            self._notify(f"added comments for {total_videos} videos", 5)

    @staticmethod
-    def notify_final(total_videos):
-        """send final notification"""
+    def _notify(message, expire):
+        """send notification"""
        key = "message:download"
        message = {
            "status": key,
            "level": "info",
            "title": "Download and index comments finished",
-            "message": f"added comments for {total_videos} videos",
+            "message": message,
        }
-        RedisArchivist().set_message(key, message, expire=4)
+        RedisArchivist().set_message(key, message, expire=expire)
--- a/tubearchivist/home/src/index/reindex.py
+++ b/tubearchivist/home/src/index/reindex.py
@@ -257,11 +257,7 @@ class Reindex(ReindexBase):

        # read current state
        video.get_from_es()
-        player = video.json_data["player"]
-        date_downloaded = video.json_data["date_downloaded"]
-        channel_dict = video.json_data["channel"]
-        playlist = video.json_data.get("playlist")
-        subtitles = video.json_data.get("subtitles")
+        es_meta = video.json_data.copy()

        # get new
        video.build_json()
@@ -269,17 +265,21 @@ class Reindex(ReindexBase):
            video.deactivate()
            return

-        video.delete_subtitles(subtitles=subtitles)
+        video.delete_subtitles(subtitles=es_meta.get("subtitles"))
        video.check_subtitles()

        # add back
-        video.json_data["player"] = player
-        video.json_data["date_downloaded"] = date_downloaded
-        video.json_data["channel"] = channel_dict
-        if playlist:
-            video.json_data["playlist"] = playlist
+        video.json_data["player"] = es_meta.get("player")
+        video.json_data["date_downloaded"] = es_meta.get("date_downloaded")
+        video.json_data["channel"] = es_meta.get("channel")
+        if es_meta.get("playlist"):
+            video.json_data["playlist"] = es_meta.get("playlist")

        video.upload_to_es()
+        if es_meta.get("media_url") != video.json_data["media_url"]:
+            self._rename_media_file(
+                es_meta.get("media_url"), video.json_data["media_url"]
+            )

        thumb_handler = ThumbManager(youtube_id)
        thumb_handler.delete_video_thumb()
@@ -289,6 +289,14 @@ class Reindex(ReindexBase):

        return

+    def _rename_media_file(self, media_url_is, media_url_should):
+        """handle title change"""
+        print(f"[reindex] fix media_url {media_url_is} to {media_url_should}")
+        videos = self.config["application"]["videos"]
+        old_path = os.path.join(videos, media_url_is)
+        new_path = os.path.join(videos, media_url_should)
+        os.rename(old_path, new_path)
+
    @staticmethod
    def _reindex_single_channel(channel_id):
        """refresh channel data and sync to videos"""
--- a/tubearchivist/home/src/ta/config.py
+++ b/tubearchivist/home/src/ta/config.py
@@ -137,7 +137,7 @@ class AppConfig:
            config = self.get_config()
            config["scheduler"]["version_check"] = self._build_rand_daily()
            RedisArchivist().set_message("config", config)
-            return
+            return False

        needs_update = False

@@ -160,6 +160,8 @@ class AppConfig:
        if needs_update:
            RedisArchivist().set_message("config", redis_config)

+        return needs_update
+

 class ScheduleBuilder:
    """build schedule dicts for beat"""
@@ -263,6 +265,8 @@ class ScheduleBuilder:

    def build_schedule(self):
        """build schedule dict as expected by app.conf.beat_schedule"""
+        AppConfig().load_new_defaults()
+        self.config = AppConfig().config
        schedule_dict = {}

        for schedule_item in self.SCHEDULES:
@@ -342,11 +346,13 @@ class ReleaseVersion:
        """check if update happened in the mean time"""
        message = self.get_update()
        if not message:
-            return
+            return False

        if self._parse_version(message.get("version")) == self.local_version:
-            print(f"[{self.local_version}]: update completed")
            RedisArchivist().del_message(self.NEW_KEY)
+            return settings.TA_VERSION
+
+        return False

    def get_update(self):
        """return new version dict if available"""
--- a/tubearchivist/home/src/ta/helper.py
+++ b/tubearchivist/home/src/ta/helper.py
@@ -122,10 +122,13 @@ def clear_dl_cache(config):
    """clear leftover files from dl cache"""
    print("clear download cache")
    cache_dir = os.path.join(config["application"]["cache_dir"], "download")
-    for cached in os.listdir(cache_dir):
+    leftover_files = os.listdir(cache_dir)
+    for cached in leftover_files:
        to_delete = os.path.join(cache_dir, cached)
        os.remove(to_delete)

+    return len(leftover_files)
+

 def get_mapping():
    """read index_mapping.json and get expected mapping and settings"""
--- a/tubearchivist/home/tasks.py
+++ b/tubearchivist/home/tasks.py
@@ -10,7 +10,6 @@ import json
 import os

 from celery import Celery, shared_task
-from home.apps import StartupCheck
 from home.src.download.queue import PendingList
 from home.src.download.subscriptions import (
    ChannelSubscription,
@@ -112,9 +111,10 @@ def download_pending():
 def download_single(pending_video):
    """start download single video now"""
    queue = RedisQueue(queue_name="dl_queue")
+
    to_add = {
        "youtube_id": pending_video["youtube_id"],
-        "vid_type": pending_video["vid_type"],
+        "vid_type": pending_video.get("vid_type", VideoTypeEnum.VIDEOS.value),
    }
    queue.add_priority(json.dumps(to_add))
    print(f"Added to queue with priority: {to_add}")
@@ -228,7 +228,7 @@ def kill_dl(task_id):
    _ = RedisArchivist().del_message("dl_queue_id")
    RedisQueue(queue_name="dl_queue").clear()

-    clear_dl_cache(CONFIG)
+    _ = clear_dl_cache(CONFIG)

    # notify
    mess_dict = {
@@ -313,6 +313,5 @@ def version_check():
    ReleaseVersion().check()


-# load new defaults then start the schedule here
-StartupCheck().sync_redis_state()
+# start schedule here
 app.conf.beat_schedule = ScheduleBuilder().build_schedule()
--- a/tubearchivist/home/templates/home/downloads.html
+++ b/tubearchivist/home/templates/home/downloads.html
@@ -90,7 +90,7 @@
                            {% else %}
                                <span>{{ video.source.channel_name }}</span>
                            {% endif %}
-                            <h3>{{ video.source.title }}</h3>
+                            <a href="https://www.youtube.com/watch?v={{ video.source.youtube_id }}" target="_blank"><h3>{{ video.source.title }}</h3></a>
                        </div>
                        <p>Published: {{ video.source.published }} | Duration: {{ video.source.duration }} | {{ video.source.youtube_id }}</p>
                        <div>
--- a/tubearchivist/home/templates/home/search.html
+++ b/tubearchivist/home/templates/home/search.html
@@ -7,30 +7,80 @@
    </div>
    <div class="multi-search-box">
        {{ search_form }}
-        <p>Start typing or use <span class="settings-current">video:</span>, <span class="settings-current">channel:</span>, <span class="settings-current">playlist:</span> or <span class="settings-current">full:</span> keywords for advanced queries. <a href="https://github.com/tubearchivist/tubearchivist/wiki/Search" target="_blank">Learn more</a>.</p>
    </div>
-    <div class="multi-search-result">
-        <h2>Video Results</h2>
-        <div id="video-results" class="video-list {{ all_styles.home }} {% if all_styles.home == "grid" %}grid-{{ grid_items }}{% endif %}">
-            <p>No videos found.</p>
+    <div id="multi-search-results" style="display: none;">
+        <div class="multi-search-result">
+            <h2>Video Results</h2>
+            <div id="video-results" class="video-list {{ all_styles.home }} {% if all_styles.home == "grid" %}grid-{{ grid_items }}{% endif %}">
+                <p>No videos found.</p>
+            </div>
+        </div>
+        <div class="multi-search-result">
+            <h2>Channel Results</h2>
+            <div id="channel-results" class="channel-list {{ all_styles.channel }}">
+                <p>No channels found.</p>
+            </div>
+        </div>
+        <div class="multi-search-result">
+            <h2>Playlist Results</h2>
+            <div id="playlist-results" class="playlist-list {{ all_styles.playlist }}">
+                <p>No playlists found.</p>
+            </div>
+        </div>
+        <div class="multi-search-result">
+            <h2>Fulltext Results</h2>
+            <div id="fulltext-results" class="video-list list">
+                <p>No fulltext results found.</p>
+            </div>
        </div>
    </div>
-    <div class="multi-search-result">
-        <h2>Channel Results</h2>
-        <div id="channel-results" class="channel-list {{ all_styles.channel }}">
-            <p>No channels found.</p>
+    <div id="multi-search-results-placeholder" style="display: block;">
+        <div>
+            <h2>Example queries</h2>
+            <ul>
+                <li><span class="value">music video</span> — basic search</li>
+                <li><span>video: active:</span><span class="value">no</span> — all videos deleted from YouTube</li>
+                <li><span>video:</span><span class="value">learn javascript</span><span> channel:</span><span class="value">corey schafer</span><span> active:</span><span class="value">yes</span></li>
+                <li><span>channel:</span><span class="value">linux</span><span> subscribed:</span><span class="value">yes</span></li>
+                <li><span>playlist:</span><span class="value">backend engineering</span><span> active:</span><span class="value">yes</span><span> subscribed:</span><span class="value">yes</span></li>
+            </ul>
        </div>
-    </div>
-    <div class="multi-search-result">
-        <h2>Playlist Results</h2>
-        <div id="playlist-results" class="playlist-list {{ all_styles.playlist }}">
-            <p>No playlists found.</p>
-        </div>
-    </div>
-    <div class="multi-search-result">
-        <h2>Fulltext Results</h2>
-        <div id="fulltext-results" class="video-list list">
-            <p>No fulltext results found.</p>
+        <div>
+            <h2>Keywords cheatsheet</h2>
+            <p>For detailed usage check <a href="https://github.com/tubearchivist/tubearchivist/wiki/Search" target="_blank">wiki</a>.</p>
+            <div>
+                <ul>
+                    <li><span>simple:</span> (implied) — search in video titles, channel names and playlist titles</li>
+                    <li>
+                        <span>video:</span> — search in video titles, tags and category field
+                        <ul>
+                            <li><span>channel:</span> — channel name</li>
+                            <li><span>active:</span><span class="value">yes/no</span> — whether the video is still active on YouTube</li>
+                        </ul>
+                    </li>
+                    <li>
+                        <span>channel:</span> — search in channel name and channel description
+                        <ul>
+                            <li><span>subscribed:</span><span class="value">yes/no</span> — whether you are subscribed to the channel</li>
+                            <li><span>active:</span><span class="value">yes/no</span> — whether the video is still active on YouTube</li>
+                        </ul>
+                    </li>
+                    <li>
+                        <span>playlist:</span> — search in channel name and channel description
+                        <ul>
+                            <li><span>subscribed:</span><span class="value">yes/no</span> — whether you are subscribed to the channel</li>
+                            <li><span>active:</span><span class="value">yes/no</span> — whether the video is still active on YouTube</li>
+                        </ul>
+                    </li>
+                    <li>
+                        <span>full:</span> — search in video subtitles
+                        <ul>
+                            <li><span>lang:</span> — subtitles language (use two-letter ISO country code, same as the one from settings page)</li>
+                            <li><span>source:</span><span class="value">auto/user</span> — <i>auto</i> to search though auto-generated subtitles only, or <i>user</i> to search through user-uploaded subtitles only</li>
+                        </ul>
+                    </li>
+                </ul>
+            </div>
        </div>
    </div>
 </div>
--- a/tubearchivist/home/templates/home/settings.html
+++ b/tubearchivist/home/templates/home/settings.html
@@ -286,8 +286,9 @@
            </div>
        </div>
        <div class="settings-group">
-            <h2>Index backup</h2>
+            <h2>ZIP file index backup</h2>
            <div class="settings-item">
+                <p><i>Zip file backups are very slow for large archives and consistency is not guaranteed, use snapshots instead. Make sure no other tasks are running when creating a Zip file backup.</i></p>
                <p>Current index backup schedule: <span class="settings-current">
                    {% if config.scheduler.run_backup %}
                        {% for key, value in config.scheduler.run_backup.items %}
@@ -332,8 +333,9 @@
        </div>
    </div>
    <div class="settings-group">
-        <h2>Backup database</h2>
+        <h2>ZIP file index backup</h2>
        <p>Export your database to a zip file stored at <span class="settings-current">cache/backup</span>.</p>
+        <p><i>Zip file backups are very slow for large archives and consistency is not guaranteed, use snapshots instead. Make sure no other tasks are running when creating a Zip file backup.</i></p>
        <div id="db-backup">
            <button onclick="dbBackup()">Start backup</button>
        </div>
--- a/tubearchivist/home/views.py
+++ b/tubearchivist/home/views.py
@@ -15,6 +15,7 @@ from django.contrib.auth.forms import AuthenticationForm
 from django.http import JsonResponse
 from django.shortcuts import redirect, render
 from django.views import View
+from home.src.download.queue import PendingInteract
 from home.src.download.yt_dlp_base import CookieHandler
 from home.src.es.backup import ElasticBackup
 from home.src.es.connect import ElasticWrap
@@ -32,7 +33,7 @@ from home.src.frontend.forms import (
    UserSettingsForm,
 )
 from home.src.frontend.searching import SearchHandler
-from home.src.index.channel import YoutubeChannel, channel_overwrites
+from home.src.index.channel import channel_overwrites
 from home.src.index.generic import Pagination
 from home.src.index.playlist import YoutubePlaylist
 from home.src.index.reindex import ReindexProgress
@@ -375,13 +376,13 @@ class DownloadView(ArchivistResultsView):
    def get(self, request):
        """handle get request"""
        self.initiate_vars(request)
-        self._update_view_data(request)
+        filter_view = self._update_view_data(request)
        self.find_results()
        self.context.update(
            {
                "title": "Downloads",
                "add_form": AddToQueueForm(),
-                "channel_agg_list": self._get_channel_agg(),
+                "channel_agg_list": self._get_channel_agg(filter_view),
            }
        )
        return render(request, "home/downloads.html", self.context)
@@ -401,12 +402,11 @@ class DownloadView(ArchivistResultsView):
                {"term": {"channel_id": {"value": channel_filter}}}
            )

-            channel = YoutubeChannel(channel_filter)
-            channel.get_from_es()
+            channel = PendingInteract(channel_filter).get_channel()
            self.context.update(
                {
-                    "channel_filter_id": channel_filter,
-                    "channel_filter_name": channel.json_data["channel_name"],
+                    "channel_filter_id": channel.get("channel_id"),
+                    "channel_filter_name": channel.get("channel_name"),
                }
            )

@@ -417,11 +417,13 @@ class DownloadView(ArchivistResultsView):
            }
        )

-    def _get_channel_agg(self):
+        return filter_view
+
+    def _get_channel_agg(self, filter_view):
        """get pending channel with count"""
        data = {
            "size": 0,
-            "query": {"term": {"status": {"value": "pending"}}},
+            "query": {"term": {"status": {"value": filter_view}}},
            "aggs": {
                "channel_downloads": {
                    "multi_terms": {
--- a/tubearchivist/requirements.txt
+++ b/tubearchivist/requirements.txt
@@ -1,13 +1,13 @@
-beautifulsoup4==4.11.1
+beautifulsoup4==4.11.2
 celery==5.2.7
-Django==4.1.5
+Django==4.1.7
 django-auth-ldap==4.1.0
-django-cors-headers==3.13.0
+django-cors-headers==3.14.0
 djangorestframework==3.14.0
 Pillow==9.4.0
-redis==4.4.2
-requests==2.28.1
+redis==4.5.1
+requests==2.28.2
 ryd-client==0.0.6
 uWSGI==2.0.21
-whitenoise==6.3.0
-yt_dlp==2023.1.6
+whitenoise==6.4.0
+yt_dlp==2023.3.3
--- a/tubearchivist/static/css/style.css
+++ b/tubearchivist/static/css/style.css
@@ -892,10 +892,24 @@ video:-webkit-full-screen {
    width: 100%;
 }

-.multi-search-result {
+.multi-search-result, #multi-search-results-placeholder {
    padding: 1rem 0;
 }

+#multi-search-results-placeholder span {
+    font-family: monospace;
+    color: var(--accent-font-dark);
+    background-color: var(--highlight-bg);
+}
+
+#multi-search-results-placeholder span.value {
+    color: var(--accent-font-light);
+}
+
+#multi-search-results-placeholder ul {
+    margin-top: 10px;
+}
+
 /* channel overview page */
 .channel-list.list {
    display: block;
--- a/tubearchivist/static/progress.js
+++ b/tubearchivist/static/progress.js
@@ -28,16 +28,12 @@ function checkMessages() {
 // get messages for page on timer
 function getMessages(dataOrigin) {
  fetch('/progress/')
-    .then(response => {
-      return response.json();
-    })
+    .then(response => response.json())
    .then(responseData => {
-      let messages = buildMessage(responseData, dataOrigin);
+      const messages = buildMessage(responseData, dataOrigin);
      if (messages.length > 0) {
        // restart itself
-        setTimeout(function () {
-          getMessages(dataOrigin);
-        }, 3000);
+        setTimeout(() => getMessages(dataOrigin), 500);
      }
    });
 }
--- a/tubearchivist/static/script.js
+++ b/tubearchivist/static/script.js
@@ -142,7 +142,7 @@ function toggleCheckbox(checkbox) {
  let payload = JSON.stringify(payloadDict);
  sendPost(payload);
  setTimeout(function () {
-    let currPage = window.location.pathname + window.location.search;
+    let currPage = window.location.pathname;
    window.location.replace(currPage);
  }, 500);
 }
@@ -865,21 +865,33 @@ function setProgressBar(videoId, currentTime, duration) {

 // multi search form
 let searchTimeout = null;
+let searchHttpRequest = null;
 function searchMulti(query) {
  clearTimeout(searchTimeout);
  searchTimeout = setTimeout(function () {
-    if (query.length > 1) {
-      let http = new XMLHttpRequest();
-      http.onreadystatechange = function () {
-        if (http.readyState === 4) {
-          let response = JSON.parse(http.response);
+    if (query.length > 0) {
+      if (searchHttpRequest) {
+        searchHttpRequest.abort();
+      }
+      searchHttpRequest = new XMLHttpRequest();
+      searchHttpRequest.onreadystatechange = function () {
+        if (searchHttpRequest.readyState === 4) {
+          const response = JSON.parse(searchHttpRequest.response);
          populateMultiSearchResults(response.results, response.queryType);
        }
      };
-      http.open('GET', `/api/search/?query=${query}`, true);
-      http.setRequestHeader('X-CSRFToken', getCookie('csrftoken'));
-      http.setRequestHeader('Content-type', 'application/json');
-      http.send();
+      searchHttpRequest.open('GET', `/api/search/?query=${query}`, true);
+      searchHttpRequest.setRequestHeader('X-CSRFToken', getCookie('csrftoken'));
+      searchHttpRequest.setRequestHeader('Content-type', 'application/json');
+      searchHttpRequest.send();
+    } else {
+      if (searchHttpRequest) {
+        searchHttpRequest.abort();
+        searchHttpRequest = null;
+      }
+      // show the placeholder container and hide the results container
+      document.getElementById('multi-search-results').style.display = 'none';
+      document.getElementById('multi-search-results-placeholder').style.display = 'block';
    }
  }, 500);
 }
@@ -890,6 +902,9 @@ function getViewDefaults(view) {
 }

 function populateMultiSearchResults(allResults, queryType) {
+  // show the results container and hide the placeholder container
+  document.getElementById('multi-search-results').style.display = 'block';
+  document.getElementById('multi-search-results-placeholder').style.display = 'none';
  // videos
  let defaultVideo = getViewDefaults('home');
  let allVideos = allResults.video_results;
--- a/version_check.py
+++ b/version_check.py
@@ -1,115 +0,0 @@
-#!/usr/bin/env python
-""" check requirements.txt for outdated packages """
-
-import pathlib
-
-import requests
-
-
-class Requirements:
-    """handle requirements.txt"""
-
-    FILE_PATH = "tubearchivist/requirements.txt"
-    LOCK = "/tmp/tubearchivist-requirements.lock"
-
-    def __init__(self):
-        self.exists = self.checked_today()
-        self.all_requirements = False
-        self.all_updates = False
-
-    def checked_today(self):
-        """skip requirements check when lock file exists"""
-        exists = pathlib.Path(self.LOCK).exists()
-        return exists
-
-    def look_for_updates(self):
-        """look through requirements and check for updates"""
-        self.all_requirements = self.get_dependencies()
-        self.all_updates = self.check_packages()
-
-    def get_dependencies(self):
-        """read out requirements.txt"""
-
-        all_requirements = []
-        with open(self.FILE_PATH, "r", encoding="utf-8") as f:
-            dependencies = f.readlines()
-
-        for dependency in dependencies:
-            package, version = dependency.split("==")
-            all_requirements.append((package, version.strip()))
-
-        all_requirements.sort(key=lambda x: x[0].lower())
-
-        return all_requirements
-
-    def check_packages(self):
-        """compare installed with remote version"""
-
-        total = len(self.all_requirements)
-        print(f"checking versions for {total} packages...")
-
-        all_updates = {}
-
-        for dependency in self.all_requirements:
-            package, version_installed = dependency
-            url = f"https://pypi.org/pypi/{package}/json"
-            response = requests.get(url).json()
-            version_remote = response["info"]["version"]
-            homepage = response["info"]["home_page"]
-            if version_remote != version_installed:
-                to_update = {
-                    package: {"from": version_installed, "to": version_remote}
-                }
-                all_updates.update(to_update)
-                message = (
-                    f"update {package} {version_installed}"
-                    + f"==> {version_remote}\n    {homepage}"
-                )
-                print(message)
-
-        if not all_updates:
-            print("no updates found")
-
-        # remember that
-        pathlib.Path(self.LOCK).touch()
-
-        return all_updates
-
-    def apply_updates(self):
-        """update requirements.txt file with new versions"""
-
-        to_write = []
-
-        for requirement in self.all_requirements:
-            package, old_version = requirement
-
-            if package in self.all_updates.keys():
-                package_version = self.all_updates[package]["to"]
-            else:
-                package_version = old_version
-
-            to_write.append(f"{package}=={package_version}\n")
-
-        with open(self.FILE_PATH, "w", encoding="utf-8") as f:
-            f.writelines(to_write)
-
-        print("requirements.txt updates")
-
-
-def main():
-    """main to check for updates"""
-    handler = Requirements()
-    if handler.exists:
-        return
-
-    handler.look_for_updates()
-    if handler.all_updates:
-        input_response = input("\nupdate requirements.txt? [y/n] ")
-        if input_response == "y":
-            handler.apply_updates()
-        else:
-            print("skip update...")
-
-
-if __name__ == "__main__":
-    main()
Author	SHA1	Message	Date
simon	9280159531	bump TA_VERSION	2023-03-04 22:33:50 +07:00
simon	8d13b947e2	fix linter	2023-03-04 22:08:48 +07:00
simon	22ef33c6ca	fix startup migration fail with unavailable shards	2023-03-04 18:10:18 +07:00
simon	ee568534f6	improve comment notify timeouts	2023-03-04 17:37:58 +07:00
simon	c204c6ca7f	finetune timeouts and notifications	2023-03-04 17:03:46 +07:00
simon	87cfd9e154	move up notification for faster feedback	2023-03-04 16:45:14 +07:00
simon	72e5d7ccf8	disable expire for queue setup message	2023-03-04 11:51:48 +07:00
simon	a459661fa2	fallback to estimate size for progress hook	2023-03-04 11:51:12 +07:00
simon	a56de2818d	consolidate yt-dlp obs	2023-03-04 11:33:22 +07:00
simon	9ea41bd2a9	clarify update compatibility	2023-03-04 11:12:27 +07:00
simon	14b48e676c	bump es version	2023-03-04 10:57:13 +07:00
simon	3397418cab	Merge branch 'master' into testing	2023-03-04 10:52:18 +07:00
p0358	7e9fed3d26	Add search usage summary on the search page and improve the search docs (#426 ) * add search usage summary on the search page * fix JS linter and switch multi-search-results elements from class to id * changes according to feedback * finetune colors --------- Co-authored-by: simon <simobilleter@gmail.com>	2023-03-04 10:51:38 +07:00
simon	7dc67d6b6f	bump yt-dlp	2023-03-04 10:21:10 +07:00
simon	c4b175b2db	reindex simplify video meta unpack	2023-03-01 13:51:43 +07:00
simon	b52aa4daba	fix media_url change in reindex, #424	2023-03-01 13:45:01 +07:00
simon	8d76961e77	bump deps	2023-03-01 13:44:05 +07:00
simon	4cf7eab950	remove versioncheck helper script to separate lib	2023-03-01 11:30:55 +07:00
simon	5f2febf09c	note about impossible filsystem import, #440	2023-03-01 06:59:10 +07:00
Clark	e276700b6d	Fix #436 Include listen directive in replacement (#437 )	2023-02-20 09:37:05 +07:00
simon	4a64748da8	bump TA_VERSION	2023-02-18 10:53:07 +07:00
simon	5d8dc76e7a	New install startup fix, #build Changed: - Fixed not loading new default configs at expected time - Better startup error handling	2023-02-18 09:57:11 +07:00
simon	4dfc9cef52	better TA_HOST error handling	2023-02-18 09:56:50 +07:00
simon	571c2ce3a4	ensure load_new_defaults for all manage.py commands, #434	2023-02-18 09:20:09 +07:00
simon	229555d97d	fix raise commanderror when migration failed	2023-02-18 09:14:49 +07:00
simon	73da59ffbf	switch to redis-stack-server	2023-02-17 22:49:07 +07:00
simon	c7a05a9ba3	a warning about zip file backups, use snapshots instead	2023-02-17 22:08:11 +07:00
simon	9e1316b543	yt-dlp uploader id fix, #build Changed: - Fix for yt-dlp parsing error [2023.02.17] - Fix for various thumbnail extraction errors - Fix download queue filter - [API] Add filter subscribed channels	2023-02-17 20:58:19 +07:00
simon	72924fab66	bump yt-dlp, #433	2023-02-17 20:25:21 +07:00
simon	72f091b85e	fix download filter for ignored videos, #401	2023-02-17 14:04:52 +07:00
simon	fdf5b2d802	clarify video page access from title, #432	2023-02-17 13:48:00 +07:00
simon	a34c9479f2	use channel from queue for download query filtering, #431	2023-02-17 12:21:00 +07:00
simon	162c05628c	rename PendingInteract youtube_id kwargs	2023-02-17 10:26:10 +07:00
simon	1125c0c4bf	handle malformed thumbnail download, #430	2023-02-17 09:17:04 +07:00
simon	0e5421a558	bump django	2023-02-17 08:37:38 +07:00
simon	5ec0636807	[API] add filter subscribed to channel list	2023-02-14 11:12:58 +07:00
simon	6641db3e7e	skip empty comment, #429	2023-02-13 12:42:55 +07:00
simon	77c26134d1	add print for thumbnail fallback, #425	2023-02-13 12:15:12 +07:00
simon	ef685ecb42	handle PIL error empty thumbnail image, #425	2023-02-13 11:55:47 +07:00
simon	df1018e70c	bump redis	2023-02-11 18:24:41 +07:00
simon	0658424b94	Merge branch 'master' into testing	2023-02-11 17:29:09 +07:00
p0358	d1b5644c63	refresh /progress more often (3000 ms => 500 ms) (#427 )	2023-02-11 17:28:46 +07:00
lamusmaser	4936f2fdf2	412 fix `vid_type` with downloads (single and bulk) (#413 ) * update TA_VERSION v0.3.2 * Update `single_download` task to request proper `vid_type`. * Modify `single_download` to match bulk download. * pr 413: undo TA_VERSION update --------- Co-authored-by: simon <simobilleter@gmail.com>	2023-02-11 17:07:36 +07:00
simon	0fcd6c086b	add kip log output for is_upcoming or is_live	2023-02-10 22:16:04 +07:00
simon	34c5b2613c	add list of notable mentions	2023-02-05 15:49:43 +07:00
simon	6b246e7808	instructions for ES on custom port	2023-02-04 09:07:45 +07:00
simon	eda7528f7d	Startup Rewrite, #build Changed: - Changed startup functions to manage.py commands - Changed disable json backup, enable snapshots by default - Changed use commited django migrations - Added YT link to download queue title	2023-02-03 15:35:40 +07:00
simon	fa46b30e61	add unstable footer	2023-02-03 15:26:14 +07:00
Krzysztof Olszewski	2acaeb34db	#409:Ability to visit source URL for preview from download queue (#422 )	2023-02-03 15:17:16 +07:00
simon	f8b0977a7f	fix linter	2023-02-03 14:57:57 +07:00
simon	4bab1a9815	bump django	2023-02-03 14:50:59 +07:00
simon	ce7adfee3a	disable backup, enable snapshot	2023-02-03 14:50:41 +07:00
simon	a350d735be	commit migrations	2023-02-02 14:19:06 +07:00
simon	3f86786ee8	reduce comments log output	2023-02-02 14:14:32 +07:00
simon	d6c124aae8	collectstatic if not in debug	2023-02-02 13:12:12 +07:00
simon	2d942fe9b5	load defaults for schedule builder at startup	2023-02-02 13:06:00 +07:00
simon	f18e2ce599	remove old startup class	2023-02-02 12:46:54 +07:00
simon	08350181f3	create ta_startup manage.py command	2023-02-02 12:43:02 +07:00
simon	114c7c2811	return new version in is_updated	2023-02-02 12:06:24 +07:00
simon	83057d9e54	return files clearded from clear_dl_cache	2023-02-02 10:46:59 +07:00
simon	d73a131616	extend timeout	2023-02-01 16:06:40 +07:00
simon	1b5a810520	finalize connection check manage.py command	2023-02-01 16:05:29 +07:00
simon	2f685859cf	bump bs4 lib	2023-02-01 15:27:28 +07:00
simon	cacf6e43b8	add envcheck and basic connection check startup command	2023-01-28 08:37:58 +07:00
simon	991ab6a2bd	bump requests lib	2023-01-27 20:13:33 +07:00
simon	188eb9fc41	update feature request desc	2023-01-27 09:57:10 +07:00
simon	4db3775eba	manage new feature expectations	2023-01-27 09:55:11 +07:00
simon	57841ea1c0	clarify keyword filter roadmap item	2023-01-27 09:41:57 +07:00