bump TA_VERSION

improve manual import docs
fix startup migration error missing enable_snapshot var
2026-08-09 12:59:33 +00:00 · 2022-11-05 11:58:41 +07:00 · 2022-11-05 11:36:53 +07:00 · 2022-11-05 11:17:12 +07:00 · 2022-11-05 09:35:28 +07:00 · 2022-11-03 18:18:51 +07:00
50 changed files with 3855 additions and 1077 deletions
--- a/.eslintrc.js
+++ b/.eslintrc.js
@@ -0,0 +1,17 @@
+'use strict';
+module.exports = {
+  extends: ['eslint:recommended', 'eslint-config-prettier'],
+  parserOptions: {
+    ecmaVersion: 2020,
+  },
+  env: {
+    browser: true,
+  },
+  rules: {
+    strict: ['error', 'global'],
+    'no-unused-vars': ['error', { vars: 'local' }],
+    eqeqeq: ['error', 'always', { null: 'ignore' }],
+    curly: ['error', 'multi-line'],
+    'no-var': 'error',
+  },
+};
--- a/.github/workflows/lint_js.yml
+++ b/.github/workflows/lint_js.yml
@@ -0,0 +1,16 @@
+name: lint_js
+
+on: [pull_request, push]
+
+jobs:
+  check:
+    name: lint_js
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/setup-node@v3
+        with:
+          node-version: '16'
+      - run: npm ci
+      - run: npm run lint
+      - run: npm run format -- --check
--- a/.github/workflows/lint_python.yml
+++ b/.github/workflows/lint_python.yml
@@ -4,16 +4,21 @@ jobs:
  lint_python:
    runs-on: ubuntu-latest
    steps:
-      - uses: actions/checkout@v2
-      - uses: actions/setup-python@v2
+      - uses: actions/checkout@v3
+      - uses: actions/setup-python@v4
+        with:
+          python-version: '3.10'
+      # note: this logic is duplicated in the `validate` function in ./deploy.sh
+      # if you update this file, you should update that as well
      - run: pip install --upgrade pip wheel
      - run: pip install bandit black codespell flake8 flake8-bugbear
-                         flake8-comprehensions isort
-      - run: black --check --diff --line-length 79 .
-      - run: codespell
-      - run: flake8 . --count --max-complexity=10 --max-line-length=79
-                      --show-source --statistics
-      - run: isort --check-only --line-length 79 --profile black .
+                         flake8-comprehensions isort requests
+      - run: ./deploy.sh validate
+      # - run: black --check --diff --line-length 79 .
+      # - run: codespell --skip="./.git,./package.json,./package-lock.json,./node_modules"
+      # - run: flake8 . --count --max-complexity=10 --max-line-length=79
+      #                 --show-source --statistics
+      # - run: isort --check-only --line-length 79 --profile black .
      # - run: pip install -r tubearchivist/requirements.txt
      # - run: mkdir --parents --verbose .mypy_cache
      # - run: mypy --ignore-missing-imports --install-types --non-interactive .
--- a/.gitignore
+++ b/.gitignore
@@ -5,4 +5,7 @@ __pycache__
 db.sqlite3

 # vscode custom conf
-.vscode
+.vscode
+
+# JavaScript stuff
+node_modules
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -15,13 +15,29 @@ The wiki is where all user functions are documented in detail. These pages are m

 ## Development Environment

-I have learned the hard way, that working on a dockerized application outside of docker is very error prone and in general not a good idea. So if you want to test your changes, it's best to run them in a docker testing environment.  
+I have learned the hard way, that working on a dockerized application outside of docker is very error prone and in general not a good idea. So if you want to test your changes, it's best to run them in a docker testing environment. You might be able to run the application directly, but this document assumes you're using docker.
+
+### Instructions
+
+Set up docker on your development machine.
+
+Clone this repository.
+
+Functional changes should be made against the unstable `testing` branch, so check that branch out, then make a new branch for your work.
+
+Edit the `docker-compose.yml` file and replace the [`image: bbilly1/tubearchivist` line](https://github.com/tubearchivist/tubearchivist/blob/4af12aee15620e330adf3624c984c3acf6d0ac8b/docker-compose.yml#L7) with `build: .`. Also make any other changes to the environment variables and so on necessary to run the application, just like you're launching the application as normal.
+
+Run `docker compose up --build`. This will bring up the application. Kill it with `ctrl-c` or by running `docker compose down` from a new terminal window in the same directory.
+
+Make your changes locally and re-run `docker compose up --build`. The `Dockerfile` is structured in a way that the actual application code is in the last layer so rebuilding the image with only code changes utilizes the build cache for everything else and will just take a few seconds.
+
+### Develop environment inside a VM
+
+You may find it nice to run everything inside of a VM, though this is not necessary. There's a `deploy.sh` script which has some helpers for this use case. YMMV, this is what one of the developers does:

-This is my setup I have landed on, YMMV:
 - Clone the repo, work on it with your favorite code editor in your local filesystem. *testing* branch is the where all the changes are happening, might be unstable and is WIP.
 - Then I have a VM running standard Ubuntu Server LTS with docker installed. The VM keeps my projects separate and offers convenient snapshot functionality. The VM also offers ways to simulate lowend environments by limiting CPU cores and memory. You can use this [Ansible Docker Ubuntu](https://github.com/bbilly1/ansible-playbooks) playbook to get started quickly. But you could also just run docker on your host system.
- The `Dockerfile` is structured in a way that the actual application code is in the last layer so rebuilding the image with only code changes utilizes the build cache for everything else and will just take a few seconds.
- Take a look at the `deploy.sh` file. I have my local DNS resolve `tubearchivist.local` to the IP of the VM for convenience. To deploy the latest changes and rebuild the application to the testing VM run:
+- I have my local DNS resolve `tubearchivist.local` to the IP of the VM for convenience. To deploy the latest changes and rebuild the application to the testing VM run:
 ```bash
 ./deploy.sh test
 ```
@@ -60,6 +76,10 @@ Do you see anything on the roadmap that you would like to take a closer look at

 To fix a bug or implement a feature, fork the repository and make all changes to the testing branch. When ready, create a pull request.

+## Making changes to the JavaScript
+
+The JavaScript does not require any build step; you just edit the files directly. However, there is config for eslint and prettier (a linter and formatter respectively); their use is recommended but not required. To use them, install `node`, run `npm i` from the root directory of this repository to install dependencies, then run `npm run lint` and `npm run format` to run eslint and prettier respectively.
+
 ## Releases

 There are three different docker tags:
--- a/30
+++ b/30
@@ -3,18 +3,18 @@


 # First stage to build python wheel
-FROM python:3.10.7-slim-bullseye AS builder
+FROM python:3.10.8-slim-bullseye AS builder
 ARG TARGETPLATFORM

-RUN apt-get update
-RUN apt-get install -y --no-install-recommends build-essential gcc libldap2-dev libsasl2-dev libssl-dev
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    build-essential gcc libldap2-dev libsasl2-dev libssl-dev

 # install requirements
 COPY ./tubearchivist/requirements.txt /requirements.txt
 RUN pip install --user -r requirements.txt

 # build final image
-FROM python:3.10.7-slim-bullseye as tubearchivist
+FROM python:3.10.8-slim-bullseye as tubearchivist

 ARG TARGETPLATFORM
 ARG INSTALL_DEBUG
@@ -32,19 +32,23 @@ RUN apt-get clean && apt-get -y update && apt-get -y install --no-install-recomm
    curl \
    xz-utils && rm -rf /var/lib/apt/lists/*

-# get newest patched ffmpeg and ffprobe builds for amd64 fall back to repo ffmpeg for arm64
-RUN if [ "$TARGETPLATFORM" = "linux/amd64" ] ; then \
+# install patched ffmpeg build, default to linux64
+RUN if [ "$TARGETPLATFORM" = "linux/arm64" ] ; then \
+    curl -s https://api.github.com/repos/yt-dlp/FFmpeg-Builds/releases/latest \
+        | grep browser_download_url \
+        | grep ".*master.*linuxarm64.*tar.xz" \
+        | cut -d '"' -f 4 \
+        | xargs curl -L --output ffmpeg.tar.xz ; \
+    else \
    curl -s https://api.github.com/repos/yt-dlp/FFmpeg-Builds/releases/latest \
        | grep browser_download_url \
        | grep ".*master.*linux64.*tar.xz" \
        | cut -d '"' -f 4 \
-        | xargs curl -L --output ffmpeg.tar.xz && \
-        tar -xf ffmpeg.tar.xz --strip-components=2 --no-anchored -C /usr/bin/ "ffmpeg" && \
-        tar -xf ffmpeg.tar.xz --strip-components=2 --no-anchored -C /usr/bin/ "ffprobe" && \
-        rm ffmpeg.tar.xz \
-    ; elif [ "$TARGETPLATFORM" = "linux/arm64" ] ; then \
-        apt-get -y update && apt-get -y install --no-install-recommends ffmpeg && rm -rf /var/lib/apt/lists/* \
-    ; fi
+        | xargs curl -L --output ffmpeg.tar.xz ; \
+    fi && \
+    tar -xf ffmpeg.tar.xz --strip-components=2 --no-anchored -C /usr/bin/ "ffmpeg" && \
+    tar -xf ffmpeg.tar.xz --strip-components=2 --no-anchored -C /usr/bin/ "ffprobe" && \
+    rm ffmpeg.tar.xz

 # install debug tools for testing environment
 RUN if [ "$INSTALL_DEBUG" ] ; then \
--- a/README.md
+++ b/README.md
@@ -61,7 +61,38 @@ Once your YouTube video collection grows, it becomes hard to search and find a s
 - [Tube Archivist Metrics](https://github.com/tubearchivist/tubearchivist-metrics) to create statistics in Prometheus/OpenMetrics format.

 ## Installing and updating
-Take a look at the example `docker-compose.yml` file provided. Use the *latest* or the named semantic version tag. The *unstable* tag is for intermediate testing and as the name implies, is **unstable** and not be used on your main installation but in a [testing environment](CONTRIBUTING.md).  
+There's dedicated user-contributed install steps under [docs/Installation.md](./docs/Installation.md) for Unraid, Truenas and Synology which you can use instead of this section if you happen to be using one of those. Otherwise, continue on.
+
+For minimal system requirements, the Tube Archivist stack needs around 2GB of available memory for a small testing setup and around 4GB of available memory for a mid to large sized installation.  
+
+Note for arm64 hosts: The Tube Archivist container is multi arch, so is Elasticsearch. RedisJSON doesn't offer arm builds, but you can use the image `bbilly1/rejson` (an unofficial rebuild for arm64) instead of [the official one](https://github.com/tubearchivist/tubearchivist/blob/4af12aee15620e330adf3624c984c3acf6d0ac8b/docker-compose.yml#L27).
+
+This project requires docker. Ensure it is installed and running on your system.
+
+Save the [docker-compose.yml](./docker-compose.yml) file from this reposity somewhere permanent on your system, keeping it named `docker-compose.yml`. You'll need to refer to it whenever starting this application.
+
+Edit the following values from that file:
+- under `tubearchivist`->`environment`:
+  - `HOST_UID`: your UID, if you want TubeArchivist to create files with your UID. Remove if you are OK with files being owned by the the container user.
+  - `HOST_GID`: as above but GID.
+  - `TA_HOST`: change it to the address of the machine you're running this on. This can be an IP address or a domain name.
+  - `TA_PASSWORD`: pick a password to use when logging in.
+  - `ELASTIC_PASSWORD`: pick a password for the elastic service. You won't need to type this yourself.
+  - `TZ`: your time zone. If you don't know yours, you can look it up [here](https://www.timezoneconverter.com/cgi-bin/findzone/findzone).
+- under `archivist-es`->`environment`:
+  - `"ELASTIC_PASSWORD=verysecret"`: change `verysecret` to match the ELASTIC_PASSWORD you picked above.
+
+By default Docker will store all data, including downloaded data, in its own data-root directory (which you can find by running `docker info` and looking for the "Docker Root Dir"). If you want to use other locations, you can replace the `media:`, `cache:`, `redis:`, and `es:` volume names with absolute paths; if you do, remove them from the `volumes:` list at the bottom of the file.
+
+From a terminal, `cd` into the directory you saved the `docker-compose.yml` file in and run `docker compose up --detach`. The first time you do this it will download the appropriate images, which can take a minute.
+
+You can follow the logs with `docker compose logs -f`. Once it's ready it will print something like `celery@1234567890ab ready`. At this point you should be able to go to `http://your-host:8000` and log in with the `TA_USER`/`TA_PASSWORD` credentials.
+
+You can bring the application down by running `docker compose down` in the same directory.
+
+Use the *latest* (the default) or a named semantic version tag for the docker images. The *unstable* tag is for intermediate testing and as the name implies, is **unstable** and not be used on your main installation but in a [testing environment](CONTRIBUTING.md).  
+
+## Installation Details

 Tube Archivist depends on three main components split up into separate docker containers:  

@@ -94,8 +125,12 @@ You can configure LDAP with the following environment variables:
 - `TA_LDAP_DISABLE_CERT_CHECK` (ex: `true`) Set to anything besides empty string to disable certificate checking when connecting over LDAPS.
 - `TA_LDAP_BIND_DN` (ex: `uid=search-user,ou=users,dc=your-server`) DN of the user that is able to perform searches on your LDAP account.
 - `TA_LDAP_BIND_PASSWORD` (ex: `yoursecretpassword`) Password for the search user.
+ - `TA_LDAP_USER_ATTR_MAP_USERNAME` (default: `uid`) Bind attribute used to map LDAP user's username
+ - `TA_LDAP_USER_ATTR_MAP_PERSONALNAME` (default: `givenName`) Bind attribute used to match LDAP user's First Name/Personal Name.
+ - `TA_LDAP_USER_ATTR_MAP_SURNAME` (default: `sn`) Bind attribute used to match LDAP user's Last Name/Surname.
+ - `TA_LDAP_USER_ATTR_MAP_EMAIL` (default: `mail`) Bind attribute used to match LDAP user's EMail address
 - `TA_LDAP_USER_BASE` (ex: `ou=users,dc=your-server`) Search base for user filter.
- - `TA_LDAP_USER_FILTER` (ex: `(objectClass=user)`) Filter for valid users. Login usernames are automatically matched using `uid` and does not need to be specified in this filter.
+ - `TA_LDAP_USER_FILTER` (ex: `(objectClass=user)`) Filter for valid users. Login usernames are matched using the attribute specified in `TA_LDAP_USER_ATTR_MAP_USERNAME` and should not be specified in this filter.

 When LDAP authentication is enabled, django passwords (e.g. the password defined in TA_PASSWORD), will not allow you to login, only the LDAP server is used.

@@ -126,11 +161,9 @@ You will see the current version number of **Tube Archivist** in the footer of t
 * There can be breaking changes between updates, particularly as the application grows, new environment variables or settings might be required for you to set in the your docker-compose file. *Always* check the **release notes**: Any breaking changes will be marked there.  
 * All testing and development is done with the Elasticsearch version number as mentioned in the provided *docker-compose.yml* file. This will be updated when a new release of Elasticsearch is available. Running an older version of Elasticsearch is most likely not going to result in any issues, but it's still recommended to run the same version as mentioned. Use `bbilly1/tubearchivist-es` to automatically get the recommended version.

-### Alternative installation instructions:
- **arm64**: The Tube Archivist container is multi arch, so is Elasticsearch. RedisJSON doesn't offer arm builds, you can use `bbilly1/rejson`, an unofficial rebuild for arm64.
- **Helm Chart**: There is a Helm Chart available at https://github.com/insuusvenerati/helm-charts. Mostly self-explanatory but feel free to ask questions in the discord / subreddit.
- **Wiki**: There are additional helpful installation instructions in the [wiki](https://github.com/tubearchivist/tubearchivist/wiki/Installation) for Unraid, Truenas and Synology.
+### Helm charts

+There is a Helm Chart available at https://github.com/insuusvenerati/helm-charts. Mostly self-explanatory but feel free to ask questions in the discord / subreddit.

 ## Potential pitfalls
 ### vm.max_map_count
@@ -162,7 +195,7 @@ Similar to that, TubeArchivist will become all sorts of messed up when running o
 ## Getting Started
 1. Go through the **settings** page and look at the available options. Particularly set *Download Format* to your desired video quality before downloading. **Tube Archivist** downloads the best available quality by default. To support iOS or MacOS and some other browsers a compatible format must be specified. For example:
 ```
-bestvideo[VCODEC=avc1]+bestaudio[ACODEC=mp4a]/mp4
+bestvideo[vcodec*=avc1]+bestaudio[acodec*=mp4a]/mp4
 ```
 2. Subscribe to some of your favorite YouTube channels on the **channels** page. 
 3. On the **downloads** page, click on *Rescan subscriptions* to add videos from the subscribed channels to your Download queue or click on *Add to download queue* to manually add Video IDs, links, channels or playlists.
--- a/deploy.sh
+++ b/deploy.sh
@@ -18,11 +18,7 @@ set -e

 function sync_blackhole {

-    # docker commands need sudo, only build amd64
    host="blackhole.local"
-
-    read -sp 'Password: ' remote_pw
-    export PASS=$remote_pw
    
    rsync -a --progress --delete-after \
        --exclude ".git" \
@@ -32,8 +28,8 @@ function sync_blackhole {
        --exclude "db.sqlite3" \
        . -e ssh "$host":tubearchivist

-    echo "$PASS" | ssh "$host" 'sudo -S docker buildx build --platform linux/amd64 -t bbilly1/tubearchivist:latest tubearchivist --load 2>/dev/null'
-    echo "$PASS" | ssh "$host" 'sudo -S docker compose up -d 2>/dev/null'
+    ssh "$host" 'docker build -t bbilly1/tubearchivist --build-arg TARGETPLATFORM="linux/amd64" tubearchivist'
+    ssh "$host" 'docker compose up -d'

 }

@@ -86,10 +82,12 @@ function validate {

    echo "run validate on $check_path"

+    # note: this logic is duplicated in the `./github/workflows/lint_python.yml` config
+    # if you update this file, you should update that as well
    echo "running black"
    black --diff --color --check -l 79 "$check_path"
    echo "running codespell"
-    codespell --skip="./.git" "$check_path"
+    codespell --skip="./.git,./package.json,./package-lock.json,./node_modules" "$check_path"
    echo "running flake8"
    flake8 "$check_path" --count --max-complexity=10 --max-line-length=79 \
        --show-source --statistics
@@ -215,9 +213,9 @@ if [[ $1 == "blackhole" ]]; then
 elif [[ $1 == "test" ]]; then
    sync_test "$2"
 elif [[ $1 == "validate" ]]; then
-    # check package versions in requirements.txt for updates
-    python version_check.py
    validate "$2"
+elif [[ $1 == "versioncheck" ]]; then
+    python version_check.py
 elif [[ $1 == "docker" ]]; then
    sync_docker
 elif [[ $1 == "unstable" ]]; then
@@ -225,7 +223,7 @@ elif [[ $1 == "unstable" ]]; then
 elif [[ $1 == "es" ]]; then
    sync_latest_es
 else
-    echo "valid options are: blackhole | test | validate | docker | unstable | es"
+    echo "valid options are: blackhole | test | validate | versioncheck | docker | unstable | es"
 fi


--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -34,14 +34,15 @@ services:
    depends_on:
      - archivist-es
  archivist-es:
-    image: bbilly1/tubearchivist-es         # only for amd64, or use official es 8.3.3
+    image: bbilly1/tubearchivist-es         # only for amd64, or use official es 8.4.3
    container_name: archivist-es
    restart: unless-stopped
    environment:
-      - "xpack.security.enabled=true"
      - "ELASTIC_PASSWORD=verysecret"       # matching Elasticsearch password
-      - "discovery.type=single-node"
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
+      - "xpack.security.enabled=true"
+      - "discovery.type=single-node"
+      - "path.repo=/usr/share/elasticsearch/data/snapshot"
    ulimits:
      memlock:
        soft: -1
--- a/docs/Channels.md
+++ b/docs/Channels.md
@@ -24,7 +24,7 @@ Each channel will get a dedicated channel detail page accessible at `/channel/<c

 Additionally there is a *Channel Playlist* page, accessible at `/channel/<channel-id>/playlist/` to show all indexed playlists from this channel. 

-On the *Channel About* page you can see additional metadata.
+On the *Channel About* page, accessible at `/channel/<channel-id>/about/`, you can see additional metadata.
 - The button **Delete Channel** will delete the channel plus all videos of this channel, both media files and metadata additionally this will also delete playlists metadata belonging to that channel.

 The channel customize form gives options to change settings on a per channel basis. Any configurations here will overwrite your configurations from the [settings](Settings) page.
@@ -32,3 +32,5 @@ The channel customize form gives options to change settings on a per channel bas
 - **Auto Delete**: Automatically delete watched videos from this channel after selected days.
 - **Index Playlists**: Automatically add all Playlists with at least a video downloaded to your index. Only do this for channels where you care about playlists as this will slow down indexing new videos for having to check which playlist this belongs to.
 - **SponsorBlock**: Using [SponsorBlock](https://sponsor.ajay.app/) to get and skip sponsored content. Customize per channel: You can *disable* or *enable* SponsorBlock for certain channels only to overwrite the behavior set on the [Settings](settings) page. Selecting *unset* will remove the overwrite and your setting will fall back to the default on the settings page.
+
+If you have any videos pending in the download queue, a *Downloads* link will show, bringing you directly to the [downloads](Downloads) page, filtering the list by the selected channel.
--- a/docs/Downloads.md
+++ b/docs/Downloads.md
@@ -29,6 +29,8 @@ The **Add to Download Queue** icon <img src="assets/icon-add.png?raw=true" alt="
 ## The Download Queue
 Below the three buttons you find the download queue. New items will get added at the bottom of the queue, the next video to download once you click on **Start Download** will be the first in the list.

+You can filter the download queue with the **filter** dropdown box, the filter will show once you have more than one channel in the download queue. Select the channel to filter by name, the number in parentheses indicates how many videos you have pending from this channel. Reset the filter by selecting *all* from the dropdown. This will generate links for the top 30 channels with pending videos.
+
 Every video in the download queue has two buttons:
 - **Ignore**: This will remove that video from the download queue and this video will not get added again, even when you **Rescan Subscriptions**.
 - **Download now**: This will give priority to this video. If the download process is already running, the prioritized video will get downloaded as soon as the current video is finished. If there is no download process running, this will start downloading this single video and stop after that.  
--- a/docs/FAQ.md
+++ b/docs/FAQ.md
@@ -29,3 +29,8 @@ So Docker is the only supported installation method. If you don't have any exper

 ## 4. Finetuning Elasticsearch
 A minimal configuration of Elasticsearch (ES) is provided in the example docker-compose.yml file. ES is highly configurable and very interesting to learn more about. Refer to the [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html) if you want to get into it.
+
+## 5. When I subscribe to a channel it only downloads the most recent 50 videos
+Subscribing to a channel is a different operation from downloading it. You can [add the channel to the download queue](https://github.com/tubearchivist/tubearchivist/wiki/Downloads#add-to-download-queue) to download all past videos.
+
+If you want to download the existing videos and also automatically download new videos, then you should both download the channel and subscribe to it.
--- a/docs/Home.md
+++ b/docs/Home.md
@@ -30,3 +30,12 @@ Table of contents:
 An empty checkbox icon <img src="assets/icon-unseen.png?raw=true" alt="unseen icon" width="20px" style="margin:0 5px;"> will show for videos you haven't marked as watched. Click on it and the icon will change to a filled checkbox <img src="assets/icon-seen.png?raw=true" alt="seen icon" width="20px" style="margin:0 5px;"> indicating it as watched - click again to revert.

 When available the <img src="assets/icon-gridview.png?raw=true" alt="gridview icon" width="20px" style="margin:0 5px;"> gridview icon will display the list in a grid. A grid row holds 3 items by default, use the <img src="assets/icon-add.png?raw=true" alt="listview icon" width="20px" style="margin:0 5px;"> icon to add more or the <img src="assets/icon-substract.png?raw=true" alt="listview icon" width="20px" style="margin:0 5px;"> icon to remove items per row, depending on your screen size. The <img src="assets/icon-listview.png?raw=true" alt="listview icon" width="20px" style="margin:0 5px;"> listview icon will arrange the items in a list. The sort icon <img src="assets/icon-sort.png?raw=true" alt="listview icon" width="20px" style="margin:0 5px;"> will open additional sort options.
+
+You can control the video player with the following keyboard shortcuts:
+- `?`: Show help
+- `m`: toggle mute
+- `c`: toggle subtitles if available
+- `>`: increase playback speed
+- `<`: decrease playback speed
+- `←` (left arrow): jump back 5 seconds
+- `→` (right arrow): jump forward 5 seconds
--- a/docs/Search.md
+++ b/docs/Search.md
@@ -3,7 +3,10 @@ Accessible at `/search/` of your **Tube Archivist**, search your archive for Vid

 - All your queries are case insensitive and are normalized to lowercase.
 - All your queries are analyzed for the english language, this means *singular*, *plural* and word variations like *-ing*, *-ed*, *-able* etc are treated as synonyms.
- Fuzzy search is activated for all your searches. This can catch typos in your queries or in the matching documents with one to two letters difference, depending on the query length.
+- Fuzzy search is activated for all your searches by default. This can catch typos in your queries or in the matching documents with one to two letters difference, depending on the query length. You can configure fuzziness with the secondary keyword `fuzzy:`, e.g:
+  - `fuzzy:0` or `fuzzy:no`: Deactivate fuzzy matching.
+  - `fuzzy:1`: Set fuzziness to one letter difference.
+  - `fuzzy:2`: Set fuzziness to two letters difference.
 - All text searches are ranked, meaning the better a match the higher ranked the result. Unless otherwise stated, queries with multiple words are processed with the `and` operator, meaning all words need to match so each word will narrow down the result.
 - This will return 30 results per query, pagination is not implemented yet.

--- a/docs/Settings.md
+++ b/docs/Settings.md
@@ -63,17 +63,30 @@ All third party integrations of TubeArchivist will **always** be *opt in*.
 - **API**: Your access token for the Tube Archivist API.
 - **returnyoutubedislike.com**: This will get return dislikes and average ratings for each video by integrating with the API from [returnyoutubedislike.com](https://www.returnyoutubedislike.com/).
 - **SponsorBlock**: Using [SponsorBlock](https://sponsor.ajay.app/) to get and skip sponsored content. If a video doesn't have timestamps, or has unlocked timestamps, use the browser addon to contribute to this excellent project. Can also be activated and deactivated as a per [channel overwrite](Settings#channel-customize).
- **Cast**: Enabling the cast integration in the settings page will load an additional JS library from **Google**.
-    * Requirements
-        - HTTPS
-            * To use the cast integration HTTPS needs to be enabled, which can be done using a reverse proxy. This is a requirement by Google as communication to the cast device is required to be encrypted, but the content itself is not.
-        - Supported Browser
-            * A supported browser is required for this integration such as Google Chrome. Other browsers, especially Chromium-based browsers, may support casting by enabling it in the settings.
-        - Subtitles
-            * Subtitles are supported however they do not work out of the box and require additional configuration. Due to requirements by Google, to use subtitles you need additional headers which will need to be configured in your reverse proxy. See this [page](https://developers.google.com/cast/docs/web_sender/advanced#cors_requirements) for the specific requirements.
-               > You need the following headers: Content-Type, Accept-Encoding, and Range. Note that the last two headers, Accept-Encoding and Range, are additional headers that you may not have needed previously.
-               > Wildcards "*" cannot be used for the Access-Control-Allow-Origin header. If the page has protected media content, it must use a domain instead of a wildcard.
+- **Cast**: Enabling the cast integration in the settings page will load an additional JS library from **Google**.  
+*NOTE*: This feature is currently broken due to an authentication issue, see #331.
+Requirements:
+    - HTTPS: To use the cast integration HTTPS needs to be enabled, which can be done using a reverse proxy. This is a requirement by Google as communication to the cast device is required to be encrypted, but the content itself is not.
+    - Supported Browser:A supported browser is required for this integration such as Google Chrome. Other browsers, especially Chromium-based browsers, may support casting by enabling it in the settings.
+    - Subtitles: Subtitles are supported however they do not work out of the box and require additional configuration. Due to requirements by Google, to use subtitles you need additional headers which will need to be configured in your reverse proxy. See this [page](https://developers.google.com/cast/docs/web_sender/advanced#cors_requirements) for the specific requirements.  
+    You need the following headers: Content-Type, Accept-Encoding, and Range. Note that the last two headers, Accept-Encoding and Range, are additional headers that you may not have needed previously.  
+    Wildcards "*" cannot be used for the Access-Control-Allow-Origin header. If the page has protected media content, it must use a domain instead of a wildcard.

+## Snapshots
+System snapshots will automatically make daily snapshots of the Elasticsearch index. The task will start at 12pm your local time. Snapshots are deduplicated, meaning that each snapshot will only have to backup changes since the last snapshot. The initial snapshot may be slow, but subsequent runs will be much faster. There is also a cleanup function implemented, that will remove snapshots older than 30 days.
+
+This will make a snapshot of your metadata index only, no media files or additional configuration variables you have set on the settings page will be backed up.
+
+Due to these improvements compared to the previous backup solution, system snapshots will replace the current backup system in a future version.
+
+Before activating system snapshots, you'll have to add one additional environment variables to the *archivist-es* container:
+```
+path.repo=/usr/share/elasticsearch/data/snapshot
+```
+The variable `path.repo` will set the folder where the snapshots will go inside the Elasticsearch container, you can't change it, but the variable needs to be set. Rebuild the container for changes to take effect, e.g `docker compose up -d`.
+
+- **Create snapshot now**: Will start the snapshot process now, outside of the regular daily schedule.
+- **Restore**: Restore your index to that point in time.

 # Scheduler Setup
 Schedule settings expect a cron like format, where the first value is minute, second is hour and third is day of the week. Day 0 is Sunday, day 1 is Monday etc.
@@ -116,6 +129,8 @@ Additional database functionality.
 The button **Delete all queued** will delete all pending videos from the download queue. The button **Delete all ignored** will delete all videos you have previously ignored.

 ## Manual Media Files Import
+NOTE: This is inherently error prone, as there are many variables, some outside of the control of this project. Read this carefully and use at your own risk. 
+
 Add the files you'd like to import to the */cache/import* folder. Only add files, don't add subdirectories. All files you are adding, need to have the same *base name* as the media file. Then start the process from the settings page *Manual Media Files Import*.

 Valid media extensions are *.mp4*, *.mkv* or *.webm*. If you have other file extensions or incompatible codecs, convert them first to mp4. **Tube Archivist** can identify the videos with one of the following methods.
@@ -133,11 +148,9 @@ Detect the YouTube ID from filename, this accepts the default yt-dlp naming conv
 - The YouTube ID in square brackets at the end of the filename is the crucial part.

 ### Offline import:
-**NOTE**: This is untested. Please provide feedback.  
-
 If the video you are trying to import is not available on YouTube any more, **Tube Archivist** can import the required metadata:
 - The file `<base-name>.info.json` is required to extract the required information.
- Add the thumbnail as `<base-name>.<ext>`, where valid file extensions are *.jpg*, *.png* or *.webp*. If there is no thumbnail file, **Tube Archivist** will try to extract it from the media file or will fallback to a default thumbnail.
+- Add the thumbnail as `<base-name>.<ext>`, where valid file extensions are *.jpg*, *.png* or *.webp*. If there is no thumbnail file, **Tube Archivist** will try to extract the embedded cover from the media file or will fallback to a default thumbnail.
 - Add subtitles as `<base-name>.<lang>.vtt` where *lang* is the two letter ISO country code. This will archive all subtitle files you add to the import folder, independent from your configurations. Subtitles can be archived and used in the player, but they can't be indexed or made searchable due to the fact, that they have a very different structure than the subtitles as **Tube Archivist** needs them.
 - For videos, where the whole channel is not available any more, you can add the `<channel-id>.info.json` file as generated by *youtube-dl/yt-dlp* to get the full metadata. Alternatively **Tube Archivist** will extract as much info as possible from the video info.json file. 

--- a/package-lock.json
+++ b/package-lock.json
--- a/package.json
+++ b/package.json
@@ -0,0 +1,17 @@
+{
+  "private": true,
+  "scripts": {
+    "lint": "eslint 'tubearchivist/static/**/*.js'",
+    "format": "prettier --write 'tubearchivist/static/**/*.js'"
+  },
+  "devDependencies": {
+    "eslint": "^8.26.0",
+    "prettier": "^2.7.1",
+    "eslint-config-prettier": "^8.5.0"
+  },
+  "prettier": {
+    "singleQuote": true,
+    "arrowParens": "avoid",
+    "printWidth": 100
+  }
+}
--- a/tubearchivist/api/README.md
+++ b/tubearchivist/api/README.md
@@ -29,6 +29,10 @@ Note:
 - [Download Queue List](#download-queue-list-view)
 - [Download Queue Single](#download-queue-item-view)

+**Snapshot management**
+- [Snapshot List](#snapshot-list-view)
+- [Snapshot Single](#snapshot-item-view)
+
 **Additional**
 - [Login](#login-view)
 - [Task](#task-view) WIP
@@ -61,6 +65,7 @@ The list views return a paginate object with the following keys:
 - prev_pages: *array of ints* of previous pages, if available
 - current_page: *int* current page from query
 - max_hits: *bool* if max of 10k results is reached
+- params: *str* additional url encoded query parameters
 - last_page: *int* of last page link
 - next_pages: *array of ints* of next pages
 - total_hits: *int* total results
@@ -169,6 +174,7 @@ GET /api/download/

 Parameter:
 - filter: pending, ignore
+- channel: channel-id

 ### Add list of videos to download queue
 POST /api/download/
@@ -205,6 +211,48 @@ Add to queue previously ignored video:
 DELETE /api/download/\<video_id>/  
 Forget or delete from download queue

+## Snapshot List View
+GET /api/snapshot/  
+Return snapshot config and a list of available snapshots.
+
+```json
+{
+    "next_exec": epoch,
+    "next_exec_str": "date_str",
+    "expire_after": "30d",
+    "snapshots": []
+}
+```
+
+POST /api/snapshot/  
+Create new snapshot now, will return immediately, task will run async in the background, will return snapshot name: 
+```json
+{
+    "snapshot_name": "ta_daily_<random-id>
+}
+```
+
+## Snapshot Item View
+GET /api/snapshot/\<snapshot-id>/  
+Return metadata of a single snapshot
+```json
+{
+    "id": "ta_daily_<random-id>,
+    "state": "SUCCESS",
+    "es_version": "0.0.0",
+    "start_date": "date_str",
+    "end_date": "date_str",
+    "end_stamp": epoch,
+    "duration_s": 0
+}
+```
+
+GET /api/snapshot/\<snapshot-id>/  
+Restore this snapshot
+
+DELETE /api/snapshot/\<snapshot-id>/  
+Remove this snapshot from index
+
 ## Login View
 Return token and user ID for username and password:  
 POST /api/login
--- a/tubearchivist/api/urls.py
+++ b/tubearchivist/api/urls.py
@@ -13,6 +13,8 @@ from api.views import (
    PlaylistApiVideoView,
    PlaylistApiView,
    SearchView,
+    SnapshotApiListView,
+    SnapshotApiView,
    TaskApiView,
    VideoApiListView,
    VideoApiView,
@@ -89,6 +91,16 @@ urlpatterns = [
        TaskApiView.as_view(),
        name="api-task",
    ),
+    path(
+        "snapshot/",
+        SnapshotApiListView.as_view(),
+        name="api-snapshot-list",
+    ),
+    path(
+        "snapshot/<slug:snapshot_id>/",
+        SnapshotApiView.as_view(),
+        name="api-snapshot",
+    ),
    path(
        "cookie/",
        CookieView.as_view(),
--- a/tubearchivist/api/views.py
+++ b/tubearchivist/api/views.py
@@ -5,6 +5,7 @@ from api.src.task_processor import TaskHandler
 from home.src.download.queue import PendingInteract
 from home.src.download.yt_dlp_base import CookieHandler
 from home.src.es.connect import ElasticWrap
+from home.src.es.snapshot import ElasticSnapshot
 from home.src.frontend.searching import SearchForm
 from home.src.index.generic import Pagination
 from home.src.index.video import SponsorBlock
@@ -53,9 +54,7 @@ class ApiBaseView(APIView):

    def initiate_pagination(self, request):
        """set initial pagination values"""
-        user_id = request.user.id
-        page_get = int(request.GET.get("page", 0))
-        self.pagination_handler = Pagination(page_get, user_id)
+        self.pagination_handler = Pagination(request)
        self.data.update(
            {
                "size": self.pagination_handler.pagination["page_size"],
@@ -368,13 +367,23 @@ class DownloadApiListView(ApiBaseView):
        """get request"""
        query_filter = request.GET.get("filter", False)
        self.data.update({"sort": [{"timestamp": {"order": "asc"}}]})
+
+        must_list = []
        if query_filter:
            if query_filter not in self.valid_filter:
                message = f"invalid url query filder: {query_filter}"
                print(message)
                return Response({"message": message}, status=400)

-            self.data["query"] = {"term": {"status": {"value": query_filter}}}
+            must_list.append({"term": {"status": {"value": query_filter}}})
+
+        filter_channel = request.GET.get("channel", False)
+        if filter_channel:
+            must_list.append(
+                {"term": {"channel_id": {"value": filter_channel}}}
+            )
+
+        self.data["query"] = {"bool": {"must": must_list}}

        self.get_document_list(request)
        return Response(self.response)
@@ -477,6 +486,70 @@ class TaskApiView(ApiBaseView):
        return Response(response)


+class SnapshotApiListView(ApiBaseView):
+    """resolves to /api/snapshot/
+    GET: returns snashot config plus list of existing snapshots
+    POST: take snapshot now
+    """
+
+    @staticmethod
+    def get(request):
+        """handle get request"""
+        # pylint: disable=unused-argument
+        snapshots = ElasticSnapshot().get_snapshot_stats()
+
+        return Response(snapshots)
+
+    @staticmethod
+    def post(request):
+        """take snapshot now with post request"""
+        # pylint: disable=unused-argument
+        response = ElasticSnapshot().take_snapshot_now()
+
+        return Response(response)
+
+
+class SnapshotApiView(ApiBaseView):
+    """resolves to /api/snapshot/<snapshot-id>/
+    GET: return a single snapshot
+    POST: restore snapshot
+    DELETE: delete a snapshot
+    """
+
+    @staticmethod
+    def get(request, snapshot_id):
+        """handle get request"""
+        # pylint: disable=unused-argument
+        snapshot = ElasticSnapshot().get_single_snapshot(snapshot_id)
+
+        if not snapshot:
+            return Response({"message": "snapshot not found"}, status=404)
+
+        return Response(snapshot)
+
+    @staticmethod
+    def post(request, snapshot_id):
+        """restore snapshot with post request"""
+        # pylint: disable=unused-argument
+        response = ElasticSnapshot().restore_all(snapshot_id)
+        if not response:
+            message = {"message": "failed to restore snapshot"}
+            return Response(message, status=400)
+
+        return Response(response)
+
+    @staticmethod
+    def delete(request, snapshot_id):
+        """delete snapshot from index"""
+        # pylint: disable=unused-argument
+        response = ElasticSnapshot().delete_single_snapshot(snapshot_id)
+        if not response:
+            message = {"message": "failed to delete snapshot"}
+            return Response(message, status=400)
+
+        return Response(response)
+
+
 class CookieView(ApiBaseView):
    """resolves to /api/cookie/
    GET: check if cookie is enabled
--- a/tubearchivist/config/settings.py
+++ b/tubearchivist/config/settings.py
@@ -103,20 +103,75 @@ if bool(environ.get("TA_LDAP")):
    global AUTH_LDAP_BIND_PASSWORD
    AUTH_LDAP_BIND_PASSWORD = environ.get("TA_LDAP_BIND_PASSWORD")

+    """
+    Since these are new environment variables, taking the opporunity to use
+    more accurate env names.
+    Given Names are *_technically_* different from Personal names, as people
+    who change their names have different given names and personal names,
+    and they go by personal names. Additionally, "LastName" is actually
+    incorrect for many cultures, such as Korea, where the
+    family name comes first, and the personal name comes last.
+
+    But we all know people are going to try to guess at these, so still want
+    to include names that people will guess, hence using first/last as well.
+    """
+
+    # Attribute mapping options
+
+    global AUTH_LDAP_USER_ATTR_MAP_USERNAME
+    AUTH_LDAP_USER_ATTR_MAP_USERNAME = (
+        environ.get("TA_LDAP_USER_ATTR_MAP_USERNAME")
+        or environ.get("TA_LDAP_USER_ATTR_MAP_UID")
+        or "uid"
+    )
+
+    global AUTH_LDAP_USER_ATTR_MAP_PERSONALNAME
+    AUTH_LDAP_USER_ATTR_MAP_PERSONALNAME = (
+        environ.get("TA_LDAP_USER_ATTR_MAP_PERSONALNAME")
+        or environ.get("TA_LDAP_USER_ATTR_MAP_FIRSTNAME")
+        or environ.get("TA_LDAP_USER_ATTR_MAP_GIVENNAME")
+        or "givenName"
+    )
+
+    global AUTH_LDAP_USER_ATTR_MAP_SURNAME
+    AUTH_LDAP_USER_ATTR_MAP_SURNAME = (
+        environ.get("TA_LDAP_USER_ATTR_MAP_SURNAME")
+        or environ.get("TA_LDAP_USER_ATTR_MAP_LASTNAME")
+        or environ.get("TA_LDAP_USER_ATTR_MAP_FAMILYNAME")
+        or "sn"
+    )
+
+    global AUTH_LDAP_USER_ATTR_MAP_EMAIL
+    AUTH_LDAP_USER_ATTR_MAP_EMAIL = (
+        environ.get("TA_LDAP_USER_ATTR_MAP_EMAIL")
+        or environ.get("TA_LDAP_USER_ATTR_MAP_MAIL")
+        or "mail"
+    )
+
+    global AUTH_LDAP_USER_BASE
+    AUTH_LDAP_USER_BASE = environ.get("TA_LDAP_USER_BASE")
+
+    global AUTH_LDAP_USER_FILTER
+    AUTH_LDAP_USER_FILTER = environ.get("TA_LDAP_USER_FILTER")
+
    global AUTH_LDAP_USER_SEARCH
    # pylint: disable=no-member
    AUTH_LDAP_USER_SEARCH = LDAPSearch(
-        environ.get("TA_LDAP_USER_BASE"),
+        AUTH_LDAP_USER_BASE,
        ldap.SCOPE_SUBTREE,
-        "(&(uid=%(user)s)" + environ.get("TA_LDAP_USER_FILTER") + ")",
+        "(&("
+        + AUTH_LDAP_USER_ATTR_MAP_USERNAME
+        + "=%(user)s)"
+        + AUTH_LDAP_USER_FILTER
+        + ")",
    )

    global AUTH_LDAP_USER_ATTR_MAP
    AUTH_LDAP_USER_ATTR_MAP = {
-        "username": "uid",
-        "first_name": "givenName",
-        "last_name": "sn",
-        "email": "mail",
+        "username": AUTH_LDAP_USER_ATTR_MAP_USERNAME,
+        "first_name": AUTH_LDAP_USER_ATTR_MAP_PERSONALNAME,
+        "last_name": AUTH_LDAP_USER_ATTR_MAP_SURNAME,
+        "email": AUTH_LDAP_USER_ATTR_MAP_EMAIL,
    }

    if bool(environ.get("TA_LDAP_DISABLE_CERT_CHECK")):
@@ -207,4 +262,4 @@ CORS_ALLOW_HEADERS = list(default_headers) + [

 # TA application settings
 TA_UPSTREAM = "https://github.com/tubearchivist/tubearchivist"
-TA_VERSION = "v0.2.2"
+TA_VERSION = "v0.2.4"
--- a/tubearchivist/home/apps.py
+++ b/tubearchivist/home/apps.py
@@ -6,6 +6,7 @@ import sys
 from django.apps import AppConfig
 from home.src.es.connect import ElasticWrap
 from home.src.es.index_setup import index_check
+from home.src.es.snapshot import ElasticSnapshot
 from home.src.ta.config import AppConfig as ArchivistConfig
 from home.src.ta.ta_redis import RedisArchivist

@@ -30,6 +31,7 @@ class StartupCheck:
        self.sync_redis_state()
        self.set_redis_conf()
        self.make_folders()
+        self.snapshot_check()
        self.set_has_run()

    def get_has_run(self):
@@ -63,10 +65,7 @@ class StartupCheck:
        cache_dir = self.config_handler.config["application"]["cache_dir"]
        for folder in folders:
            folder_path = os.path.join(cache_dir, folder)
-            try:
-                os.makedirs(folder_path)
-            except FileExistsError:
-                continue
+            os.makedirs(folder_path, exist_ok=True)

    def release_lock(self):
        """make sure there are no leftover locks set in redis"""
@@ -84,6 +83,14 @@ class StartupCheck:
            if response:
                print("deleted leftover key from redis: " + lock)

+    def snapshot_check(self):
+        """setup snapshot config, create if needed"""
+        app = self.config_handler.config["application"]
+        if not app.get("enable_snapshot"):
+            return
+
+        ElasticSnapshot().setup()
+
    def is_invalid(self, version):
        """return true if es version is invalid, false if ok"""
        major, minor = [int(i) for i in version.split(".")[:2]]
--- a/tubearchivist/home/config.json
+++ b/tubearchivist/home/config.json
@@ -37,7 +37,8 @@
        "cache_dir": "/cache",
        "videos": "/youtube",
        "colors": "dark",
-        "enable_cast": false
+        "enable_cast": false,
+        "enable_snapshot": false
    },
    "scheduler": {
        "update_subscribed": false,
--- a/tubearchivist/home/src/download/queue.py
+++ b/tubearchivist/home/src/download/queue.py
@@ -246,7 +246,7 @@ class PendingList(PendingIndex):
            print(f"{youtube_id}: skipping premium video, id not matching")
            return False
        # stop if video is streaming live now
-        if vid["is_live"]:
+        if vid["live_status"] in ["is_upcoming", "is_live"]:
            return False

        return self._parse_youtube_details(vid)
--- a/tubearchivist/home/src/download/subscriptions.py
+++ b/tubearchivist/home/src/download/subscriptions.py
@@ -63,6 +63,7 @@ class ChannelSubscription:

        for idx, channel in enumerate(all_channels):
            channel_id = channel["channel_id"]
+            print(f"{channel_id}: find missing videos.")
            last_videos = self.get_last_youtube_videos(channel_id)

            if last_videos:
--- a/tubearchivist/home/src/download/thumbnails.py
+++ b/tubearchivist/home/src/download/thumbnails.py
@@ -14,7 +14,7 @@ from home.src.download import queue  # partial import
 from home.src.es.connect import IndexPaginate
 from home.src.ta.config import AppConfig
 from mutagen.mp4 import MP4, MP4Cover
-from PIL import Image, ImageFile, ImageFilter
+from PIL import Image, ImageFile, ImageFilter, UnidentifiedImageError

 ImageFile.LOAD_TRUNCATED_IMAGES = True

@@ -42,7 +42,12 @@ class ThumbManagerBase:
            try:
                response = requests.get(url, stream=True, timeout=5)
                if response.ok:
-                    return Image.open(response.raw)
+                    try:
+                        return Image.open(response.raw)
+                    except UnidentifiedImageError:
+                        print(f"failed to open thumbnail: {url}")
+                        return self.get_fallback()
+
                if response.status_code == 404:
                    return self.get_fallback()

--- a/tubearchivist/home/src/download/yt_dlp_handler.py
+++ b/tubearchivist/home/src/download/yt_dlp_handler.py
@@ -175,6 +175,14 @@ class VideoDownloader:
            if not success:
                continue

+            mess_dict = {
+                "status": self.MSG,
+                "level": "info",
+                "title": "Indexing....",
+                "message": "Add video metadata to index.",
+            }
+            RedisArchivist().set_message(self.MSG, mess_dict, expire=60)
+
            vid_dict = index_new_video(
                youtube_id, video_overwrites=self.video_overwrites
            )
@@ -187,12 +195,17 @@ class VideoDownloader:
            }
            RedisArchivist().set_message(self.MSG, mess_dict)

+            if queue.has_item():
+                message = "Continue with next video."
+            else:
+                message = "Download queue is finished."
+
            self.move_to_archive(vid_dict)
            mess_dict = {
                "status": self.MSG,
                "level": "info",
                "title": "Completed",
-                "message": "",
+                "message": message,
            }
            RedisArchivist().set_message(self.MSG, mess_dict, expire=10)
            self._delete_from_pending(youtube_id)
--- a/tubearchivist/home/src/es/snapshot.py
+++ b/tubearchivist/home/src/es/snapshot.py
@@ -0,0 +1,257 @@
+"""
+functionality:
+- handle snapshots in ES
+"""
+
+from datetime import datetime
+from os import environ
+from zoneinfo import ZoneInfo
+
+from home.src.es.connect import ElasticWrap
+from home.src.es.index_setup import get_mapping
+
+
+class ElasticSnapshot:
+    """interact with snapshots on ES"""
+
+    REPO = "ta_snapshot"
+    REPO_SETTINGS = {
+        "compress": "true",
+        "chunk_size": "1g",
+        "location": "/usr/share/elasticsearch/data/snapshot",
+    }
+    POLICY = "ta_daily"
+
+    def __init__(self):
+        self.all_indices = self._get_all_indices()
+
+    def _get_all_indices(self):
+        """return all indices names managed by TA"""
+        mapping = get_mapping()
+        all_indices = [f"ta_{i['index_name']}" for i in mapping]
+
+        return all_indices
+
+    def setup(self):
+        """setup the snapshot in ES, create or update if needed"""
+        print("snapshot: run setup")
+        repo_exists = self._check_repo_exists()
+        if not repo_exists:
+            self.create_repo()
+
+        policy_exists = self._check_policy_exists()
+        if not policy_exists:
+            self.create_policy()
+
+        is_outdated = self._needs_startup_snapshot()
+        if is_outdated:
+            _ = self.take_snapshot_now()
+
+    def _check_repo_exists(self):
+        """check if expected repo already exists"""
+        path = f"_snapshot/{self.REPO}"
+        response, statuscode = ElasticWrap(path).get()
+        if statuscode == 200:
+            print(f"snapshot: repo {self.REPO} already created")
+            matching = response[self.REPO]["settings"] == self.REPO_SETTINGS
+            if not matching:
+                print(f"snapshot: update repo settings {self.REPO_SETTINGS}")
+
+            return matching
+
+        print(f"snapshot: setup repo {self.REPO} config {self.REPO_SETTINGS}")
+        return False
+
+    def create_repo(self):
+        """create filesystem repo"""
+        path = f"_snapshot/{self.REPO}"
+        data = {
+            "type": "fs",
+            "settings": self.REPO_SETTINGS,
+        }
+        response, statuscode = ElasticWrap(path).post(data=data)
+        if statuscode == 200:
+            print(f"snapshot: repo setup correctly: {response}")
+
+    def _check_policy_exists(self):
+        """check if snapshot policy is set correctly"""
+        policy = self._get_policy()
+        expected_policy = self._build_policy_data()
+        if not policy:
+            print(f"snapshot: create policy {self.POLICY} {expected_policy}")
+            return False
+
+        if policy["policy"] != expected_policy:
+            print(f"snapshot: update policy settings {expected_policy}")
+            return False
+
+        print("snapshot: policy is set.")
+        return True
+
+    def _get_policy(self):
+        """get policy from es"""
+        path = f"_slm/policy/{self.POLICY}"
+        response, statuscode = ElasticWrap(path).get()
+        if statuscode != 200:
+            return False
+
+        return response[self.POLICY]
+
+    def create_policy(self):
+        """create snapshot lifetime policy"""
+        path = f"_slm/policy/{self.POLICY}"
+        data = self._build_policy_data()
+        response, statuscode = ElasticWrap(path).put(data)
+        if statuscode == 200:
+            print(f"snapshot: policy setup correctly: {response}")
+
+    def _build_policy_data(self):
+        """build policy dict from config"""
+        at_12 = datetime.now().replace(hour=12, minute=0, second=0)
+        hour = at_12.astimezone(ZoneInfo("UTC")).hour
+
+        return {
+            "schedule": f"0 0 {hour} * * ?",
+            "name": f"<{self.POLICY}_>",
+            "repository": self.REPO,
+            "config": {
+                "indices": self.all_indices,
+                "include_global_state": True,
+            },
+            "retention": {
+                "expire_after": "30d",
+                "min_count": 5,
+                "max_count": 50,
+            },
+        }
+
+    def _needs_startup_snapshot(self):
+        """check if last snapshot is expired"""
+        snap_dicts = self._get_all_snapshots()
+        if not snap_dicts:
+            print("snapshot: create initial snapshot")
+            return True
+
+        last_stamp = snap_dicts[0]["end_stamp"]
+        now = int(datetime.now().strftime("%s"))
+        outdated = (now - last_stamp) / 60 / 60 > 24
+        if outdated:
+            print("snapshot: is outdated, create new now")
+
+        print("snapshot: last snapshot is up-to-date")
+        return outdated
+
+    def take_snapshot_now(self):
+        """execute daily snapshot now"""
+        path = f"_slm/policy/{self.POLICY}/_execute"
+        response, statuscode = ElasticWrap(path).post()
+        if statuscode == 200:
+            print(f"snapshot: executing now: {response}")
+
+        return response
+
+    def get_snapshot_stats(self):
+        """get snapshot info for frontend"""
+        snapshot_info = self._build_policy_details()
+        if snapshot_info:
+            snapshot_info.update({"snapshots": self._get_all_snapshots()})
+
+        return snapshot_info
+
+    def get_single_snapshot(self, snapshot_id):
+        """get single snapshot metadata"""
+        path = f"_snapshot/{self.REPO}/{snapshot_id}"
+        response, statuscode = ElasticWrap(path).get()
+        if statuscode == 404:
+            print(f"snapshots: not found: {snapshot_id}")
+            return False
+
+        snapshot = response["snapshots"][0]
+        return self._parse_single_snapshot(snapshot)
+
+    def _get_all_snapshots(self):
+        """get a list of all registered snapshots"""
+        path = f"_snapshot/{self.REPO}/*?sort=start_time&order=desc"
+        response, statuscode = ElasticWrap(path).get()
+        if statuscode == 404:
+            print("snapshots: not configured")
+            return False
+
+        all_snapshots = response["snapshots"]
+        if not all_snapshots:
+            print("snapshots: no snapshots found")
+            return False
+
+        snap_dicts = []
+        for snapshot in all_snapshots:
+            snap_dict = self._parse_single_snapshot(snapshot)
+            snap_dicts.append(snap_dict)
+
+        return snap_dicts
+
+    def _parse_single_snapshot(self, snapshot):
+        """extract relevant metadata from single snapshot"""
+        snap_dict = {
+            "id": snapshot["snapshot"],
+            "state": snapshot["state"],
+            "es_version": snapshot["version"],
+            "start_date": self._date_converter(snapshot["start_time"]),
+            "end_date": self._date_converter(snapshot["end_time"]),
+            "end_stamp": snapshot["end_time_in_millis"] // 1000,
+            "duration_s": snapshot["duration_in_millis"] // 1000,
+        }
+        return snap_dict
+
+    def _build_policy_details(self):
+        """get additional policy details"""
+        policy = self._get_policy()
+        if not policy:
+            return False
+
+        next_exec = policy["next_execution_millis"] // 1000
+        next_exec_date = datetime.fromtimestamp(next_exec)
+        next_exec_str = next_exec_date.strftime("%Y-%m-%d %H:%M")
+        expire_after = policy["policy"]["retention"]["expire_after"]
+        policy_metadata = {
+            "next_exec": next_exec,
+            "next_exec_str": next_exec_str,
+            "expire_after": expire_after,
+        }
+        return policy_metadata
+
+    @staticmethod
+    def _date_converter(date_utc):
+        """convert datetime string"""
+        expected_format = "%Y-%m-%dT%H:%M:%S.%fZ"
+        date = datetime.strptime(date_utc, expected_format)
+        local_datetime = date.replace(tzinfo=ZoneInfo("localtime"))
+        converted = local_datetime.astimezone(ZoneInfo(environ.get("TZ")))
+        converted_str = converted.strftime("%Y-%m-%d %H:%M")
+
+        return converted_str
+
+    def restore_all(self, snapshot_name):
+        """restore snapshot by name"""
+        for index in self.all_indices:
+            _, _ = ElasticWrap(index).delete()
+
+        path = f"_snapshot/{self.REPO}/{snapshot_name}/_restore"
+        data = {"indices": "*"}
+        response, statuscode = ElasticWrap(path).post(data=data)
+        if statuscode == 200:
+            print(f"snapshot: executing now: {response}")
+            return response
+
+        print(f"snapshot: failed to restore, {statuscode} {response}")
+        return False
+
+    def delete_single_snapshot(self, snapshot_id):
+        """delete single snapshot from index"""
+        path = f"_snapshot/{self.REPO}/{snapshot_id}"
+        response, statuscode = ElasticWrap(path).delete()
+        if statuscode == 200:
+            print(f"snapshot: deleting {snapshot_id} {response}")
+            return response
+
+        print(f"snapshot: failed to delete, {statuscode} {response}")
+        return False
--- a/tubearchivist/home/src/frontend/forms.py
+++ b/tubearchivist/home/src/frontend/forms.py
@@ -74,6 +74,12 @@ class ApplicationSettingsForm(forms.Form):
        ("1", "enable Cast"),
    ]

+    SNAPSHOT_CHOICES = [
+        ("", "-- change snapshot settings --"),
+        ("0", "disable system snapshots"),
+        ("1", "enable system snapshots"),
+    ]
+
    SUBTITLE_SOURCE_CHOICES = [
        ("", "-- change subtitle source settings"),
        ("user", "only download user created"),
@@ -92,7 +98,9 @@ class ApplicationSettingsForm(forms.Form):
        ("1", "enable cookie"),
    ]

-    subscriptions_channel_size = forms.IntegerField(required=False)
+    subscriptions_channel_size = forms.IntegerField(
+        required=False, min_value=1
+    )
    downloads_limit_count = forms.IntegerField(required=False)
    downloads_limit_speed = forms.IntegerField(required=False)
    downloads_throttledratelimit = forms.IntegerField(required=False)
@@ -124,6 +132,9 @@ class ApplicationSettingsForm(forms.Form):
    application_enable_cast = forms.ChoiceField(
        widget=forms.Select, choices=CAST_CHOICES, required=False
    )
+    application_enable_snapshot = forms.ChoiceField(
+        widget=forms.Select, choices=SNAPSHOT_CHOICES, required=False
+    )


 class SchedulerSettingsForm(forms.Form):
--- a/tubearchivist/home/src/frontend/searching.py
+++ b/tubearchivist/home/src/frontend/searching.py
@@ -191,7 +191,7 @@ class SearchParser:

    def __init__(self, search_query):
        self.query_words = search_query.lower().split()
-        self.query_map = False
+        self.query_map = {"term": [], "fuzzy": []}
        self.append_to = "term"

    def run(self):
@@ -214,11 +214,11 @@ class SearchParser:
        if ":" in first_word:
            index_match, query_string = first_word.split(":")
            if index_match in key_word_map:
-                self.query_map = key_word_map.get(index_match)
+                self.query_map.update(key_word_map.get(index_match))
                self.query_words[0] = query_string
                return index_match

-        self.query_map = key_word_map.get("simple")
+        self.query_map.update(key_word_map.get("simple"))
        print(f"query_map: {self.query_map}")

        return "simple"
@@ -229,29 +229,24 @@ class SearchParser:
        return {
            "simple": {
                "index": "ta_video,ta_channel,ta_playlist",
-                "term": [],
            },
            "video": {
                "index": "ta_video",
-                "term": [],
                "channel": [],
                "active": [],
            },
            "channel": {
                "index": "ta_channel",
-                "term": [],
                "active": [],
                "subscribed": [],
            },
            "playlist": {
                "index": "ta_playlist",
-                "term": [],
                "active": [],
                "subscribed": [],
            },
            "full": {
                "index": "ta_subtitle",
-                "term": [],
                "lang": [],
                "source": [],
            },
@@ -329,6 +324,20 @@ class QueryBuilder:

        return query

+    def _get_fuzzy(self):
+        """return fuziness valuee"""
+        fuzzy_value = self.query_map.get("fuzzy", ["auto"])[0]
+        if fuzzy_value == "no":
+            return 0
+
+        if not fuzzy_value.isdigit():
+            return "auto"
+
+        if int(fuzzy_value) > 2:
+            return "2"
+
+        return fuzzy_value
+
    def _build_simple(self):
        """build simple cross index query"""
        must_list = []
@@ -339,7 +348,7 @@ class QueryBuilder:
                    "multi_match": {
                        "query": term,
                        "type": "bool_prefix",
-                        "fuzziness": "auto",
+                        "fuzziness": self._get_fuzzy(),
                        "operator": "and",
                        "fields": [
                            "channel_name._2gram",
@@ -368,7 +377,7 @@ class QueryBuilder:
                    "multi_match": {
                        "query": term,
                        "type": "bool_prefix",
-                        "fuzziness": "auto",
+                        "fuzziness": self._get_fuzzy(),
                        "operator": "and",
                        "fields": [
                            "title._2gram^2",
@@ -390,7 +399,7 @@ class QueryBuilder:
                    "multi_match": {
                        "query": channel,
                        "type": "bool_prefix",
-                        "fuzziness": "auto",
+                        "fuzziness": self._get_fuzzy(),
                        "operator": "and",
                        "fields": [
                            "channel.channel_name._2gram",
@@ -413,7 +422,7 @@ class QueryBuilder:
                    "multi_match": {
                        "query": term,
                        "type": "bool_prefix",
-                        "fuzziness": "auto",
+                        "fuzziness": self._get_fuzzy(),
                        "operator": "and",
                        "fields": [
                            "channel_description",
@@ -445,7 +454,7 @@ class QueryBuilder:
                    "multi_match": {
                        "query": term,
                        "type": "bool_prefix",
-                        "fuzziness": "auto",
+                        "fuzziness": self._get_fuzzy(),
                        "operator": "and",
                        "fields": [
                            "playlist_description",
@@ -477,7 +486,7 @@ class QueryBuilder:
                    "match": {
                        "subtitle_line": {
                            "query": term,
-                            "fuzziness": "auto",
+                            "fuzziness": self._get_fuzzy(),
                        }
                    }
                }
--- a/tubearchivist/home/src/frontend/watched.py
+++ b/tubearchivist/home/src/frontend/watched.py
@@ -57,6 +57,14 @@ class WatchState:
            print(response)
            raise ValueError("failed to mark video as watched")

+    def _get_source(self):
+        """build source line for update_by_query script"""
+        source = [
+            "ctx._source.player['watched'] = true",
+            f"ctx._source.player['watched_date'] = {self.stamp}",
+        ]
+        return "; ".join(source)
+
    def mark_channel_watched(self):
        """change watched status of every video in channel"""
        path = "ta_video/_update_by_query"
@@ -67,7 +75,7 @@ class WatchState:
        data = {
            "query": {"bool": {"must": must_list}},
            "script": {
-                "source": "ctx._source.player['watched'] = true",
+                "source": self._get_source(),
                "lang": "painless",
            },
        }
@@ -87,7 +95,7 @@ class WatchState:
        data = {
            "query": {"bool": {"must": must_list}},
            "script": {
-                "source": "ctx._source.player['watched'] = true",
+                "source": self._get_source(),
                "lang": "painless",
            },
        }
--- a/tubearchivist/home/src/index/generic.py
+++ b/tubearchivist/home/src/index/generic.py
@@ -73,16 +73,25 @@ class Pagination:
    figure out the pagination based on page size and total_hits
    """

-    def __init__(self, page_get, user_id, search_get=False):
-        self.user_id = user_id
+    def __init__(self, request):
+        self.request = request
+        self.page_get = False
+        self.params = False
+        self.get_params()
        self.page_size = self.get_page_size()
-        self.page_get = page_get
-        self.search_get = search_get
        self.pagination = self.first_guess()

+    def get_params(self):
+        """process url query parameters"""
+        query_dict = self.request.GET.copy()
+        self.page_get = int(query_dict.get("page", 0))
+
+        _ = query_dict.pop("page", False)
+        self.params = query_dict.urlencode()
+
    def get_page_size(self):
        """get default or user modified page_size"""
-        key = f"{self.user_id}:page_size"
+        key = f"{self.request.user.id}:page_size"
        page_size = RedisArchivist().get_message(key)["status"]
        if not page_size:
            config = AppConfig().config
@@ -108,9 +117,9 @@ class Pagination:
            "prev_pages": prev_pages,
            "current_page": page_get,
            "max_hits": False,
+            "params": self.params,
        }
-        if self.search_get:
-            pagination.update({"search_get": self.search_get})
+
        return pagination

    def validate(self, total_hits):
--- a/tubearchivist/home/src/index/playlist.py
+++ b/tubearchivist/home/src/index/playlist.py
@@ -18,7 +18,10 @@ class YoutubePlaylist(YouTubeItem):

    es_path = False
    index_name = "ta_playlist"
-    yt_obs = {"extract_flat": True}
+    yt_obs = {
+        "extract_flat": True,
+        "allow_playlist_files": True,
+    }
    yt_base = "https://www.youtube.com/playlist?list="

    def __init__(self, youtube_id):
@@ -45,13 +48,19 @@ class YoutubePlaylist(YouTubeItem):

    def process_youtube_meta(self):
        """extract relevant fields from youtube"""
+        try:
+            playlist_thumbnail = self.youtube_meta["thumbnails"][-1]["url"]
+        except IndexError:
+            print(f"{self.youtube_id}: thumbnail extraction failed")
+            playlist_thumbnail = False
+
        self.json_data = {
            "playlist_id": self.youtube_id,
            "playlist_active": True,
            "playlist_name": self.youtube_meta["title"],
            "playlist_channel": self.youtube_meta["channel"],
            "playlist_channel_id": self.youtube_meta["channel_id"],
-            "playlist_thumbnail": self.youtube_meta["thumbnails"][-1]["url"],
+            "playlist_thumbnail": playlist_thumbnail,
            "playlist_description": self.youtube_meta["description"] or False,
            "playlist_last_refresh": int(datetime.now().strftime("%s")),
        }
--- a/tubearchivist/home/src/index/video.py
+++ b/tubearchivist/home/src/index/video.py
@@ -48,9 +48,17 @@ class SponsorBlock:
        url = f"{self.API}/skipSegments?videoID={youtube_id}"
        headers = {"User-Agent": self.user_agent}
        print(f"{youtube_id}: get sponsorblock timestamps")
-        response = requests.get(url, headers=headers)
+        try:
+            response = requests.get(url, headers=headers, timeout=10)
+        except requests.ReadTimeout:
+            print(f"{youtube_id}: sponsorblock API timeout")
+            return False
+
        if not response.ok:
-            print(f"{youtube_id}: sponsorblock failed: {response.text}")
+            print(f"{youtube_id}: sponsorblock failed: {response.status_code}")
+            if response.status_code == 503:
+                return False
+
            sponsor_dict = {
                "last_refresh": self.last_refresh,
                "is_enabled": True,
--- a/tubearchivist/home/src/ta/ta_redis.py
+++ b/tubearchivist/home/src/ta/ta_redis.py
@@ -143,3 +143,8 @@ class RedisQueue(RedisBase):
    def trim(self, size):
        """trim the queue based on settings amount"""
        self.conn.execute_command("LTRIM", self.key, 0, size)
+
+    def has_item(self):
+        """check if queue as at least one pending item"""
+        result = self.conn.execute_command("LRANGE", self.key, 0, 0)
+        return bool(result)
--- a/tubearchivist/home/templates/home/base.html
+++ b/tubearchivist/home/templates/home/base.html
@@ -79,16 +79,16 @@
            <div class="pagination">
                {% if pagination %}
                    {% if pagination.current_page > 1 %}
-                        {% if pagination.search_get %}
-                        <a class="pagination-item" href="{{ request.path }}?search={{ pagination.search_get }}">First</a>
+                        {% if pagination.params %}
+                        <a class="pagination-item" href="{{ request.path }}?{{ pagination.params }}">First</a>
                        {% else %}
                        <a class="pagination-item" href="{{ request.path }}">First</a>
                        {% endif %}
                    {% endif %}
                    {% if pagination.prev_pages %}
                        {% for page in pagination.prev_pages %}
-                            {% if pagination.search_get %}
-                            <a class="pagination-item" href="?page={{ page }}&search={{ pagination.search_get }}">{{ page }}</a>
+                            {% if pagination.params %}
+                            <a class="pagination-item" href="?page={{ page }}&{{ pagination.params }}">{{ page }}</a>
                            {% else %}
                            <a class="pagination-item" href="?page={{ page }}">{{ page }}</a>
                            {% endif %}
@@ -100,16 +100,16 @@
                    {% if pagination.next_pages %}
                        <span> ></span>
                        {% for page in pagination.next_pages %}
-                            {% if pagination.search_get %}
-                            <a class="pagination-item" href="?page={{ page }}&search={{ pagination.search_get }}">{{ page }}</a>
+                            {% if pagination.params %}
+                            <a class="pagination-item" href="?page={{ page }}&{{ pagination.params }}">{{ page }}</a>
                            {% else %}
                            <a class="pagination-item" href="?page={{ page }}">{{ page }}</a>
                            {% endif %}
                        {% endfor %}
                    {% endif %}
                    {% if pagination.last_page > 0 %}
-                        {% if pagination.search_get %}
-                        <a class="pagination-item" href="?page={{ pagination.last_page }}&search={{ pagination.search_get }}">
+                        {% if pagination.params %}
+                        <a class="pagination-item" href="?page={{ pagination.last_page }}&{{ pagination.params }}">
                            {% if pagination.max_hits %}
                                Max ({{ pagination.last_page }})
                            {% else %}
--- a/tubearchivist/home/templates/home/channel.html
+++ b/tubearchivist/home/templates/home/channel.html
@@ -36,7 +36,7 @@
            <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="channel" data-value="list" alt="list view">
        </div>
    </div>
-    <h2>Total matching channels: {{ max_hits }}</h2>
+    <h2>Total channels: {{ max_hits }}</h2>
    <div class="channel-list {{ view_style }}">
        {% if results %}
            {% for channel in results %}
@@ -81,4 +81,4 @@
    </div>
 </div>
 <script type="text/javascript" src="{% static 'progress.js' %}"></script>
-{% endblock content %}
+{% endblock content %}
--- a/tubearchivist/home/templates/home/channel_id.html
+++ b/tubearchivist/home/templates/home/channel_id.html
@@ -10,6 +10,9 @@
        <a href="{% url 'channel_id' channel_info.channel_id %}"><h3>Videos</h3></a>
        <a href="{% url 'channel_id_playlist' channel_info.channel_id %}"><h3>Playlists</h3></a>
        <a href="{% url 'channel_id_about' channel_info.channel_id %}"><h3>About</h3></a>
+        {% if has_pending %}
+            <a href="{% url 'downloads' %}?channel={{ channel_info.channel_id }}"><h3>Downloads</h3></a>
+        {% endif %}
    </div>
    <div id="notifications" data="channel_id"></div>
    <div class="info-box info-box-2">
@@ -44,7 +47,7 @@
    </div>
 </div>
 <div class="boxed-content {% if view_style == "grid" %}boxed-{{ grid_items }}{% endif %}">
-    <div class="view-controls">
+    <div class="view-controls three">
        <div class="toggle">
            <span>Hide watched videos:</span>
            <div class="toggleBox">
--- a/tubearchivist/home/templates/home/channel_id_about.html
+++ b/tubearchivist/home/templates/home/channel_id_about.html
@@ -10,6 +10,9 @@
        <a href="{% url 'channel_id' channel_info.channel_id %}"><h3>Videos</h3></a>
        <a href="{% url 'channel_id_playlist' channel_info.channel_id %}"><h3>Playlists</h3></a>
        <a href="{% url 'channel_id_about' channel_info.channel_id %}"><h3>About</h3></a>
+        {% if has_pending %}
+            <a href="{% url 'downloads' %}?channel={{ channel_info.channel_id }}"><h3>Downloads</h3></a>
+        {% endif %}
    </div>
    <div class="info-box info-box-3">
        <div class="info-box-item">
--- a/tubearchivist/home/templates/home/channel_id_playlist.html
+++ b/tubearchivist/home/templates/home/channel_id_playlist.html
@@ -10,6 +10,9 @@
        <a href="{% url 'channel_id' channel_info.channel_id %}"><h3>Videos</h3></a>
        <a href="{% url 'channel_id_playlist' channel_info.channel_id %}"><h3>Playlists</h3></a>
        <a href="{% url 'channel_id_about' channel_info.channel_id %}"><h3>About</h3></a>
+        {% if has_pending %}
+            <a href="{% url 'downloads' %}?channel={{ channel_info.channel_id }}"><h3>Downloads</h3></a>
+        {% endif %}
    </div>
    <div class="view-controls">
        <div class="toggle">
--- a/tubearchivist/home/templates/home/downloads.html
+++ b/tubearchivist/home/templates/home/downloads.html
@@ -3,7 +3,7 @@
 {% block content %}
 <div class="boxed-content">
    <div class="title-bar">
-        <h1>Downloads</h1>
+        <h1>Downloads {% if channel_filter_id %} for {{ channel_filter_name }}{% endif %}</h1>
    </div>
    <div id="notifications" data="download"></div>
    <div id="downloadControl"></div>
@@ -28,7 +28,7 @@
            </div>
        </div>
    </div>
-    <div class="view-controls">
+    <div class="view-controls three">
        <div class="toggle">
            <span>Show only ignored videos:</span>
            <div class="toggleBox">
@@ -41,6 +41,14 @@
            </div>
        </div>
        <div class="view-icons">
+            {% if channel_agg_list|length > 1 %}
+                <select name="channel_filter" id="channel_filter" onchange="channelFilterDownload(this.value)">
+                    <option value="all" {% if not channel_filter_id %}selected{% endif %}>all</option>
+                    {% for channel in channel_agg_list %}
+                        <option {% if channel_filter_id == channel.id %}selected{% endif %} value="{{ channel.id }}">{{ channel.name }} ({{channel.count}})</option>
+                    {% endfor %}
+                </select>
+            {% endif %}
            {% if view_style == "grid" %}
                <div class="grid-count">
                    {% if grid_items < 7 %}
@@ -55,7 +63,7 @@
            <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="downloads" data-value="list" alt="list view">
        </div>
    </div>
-    <h3>Total videos: {{ max_hits }}{% if max_hits == 10000 %}+{% endif %}</h3>
+    <h3>Total videos in queue: {{ max_hits }}{% if max_hits == 10000 %}+{% endif %} {% if channel_filter_id %} - from channel <i>{{ channel_filter_name }}</i>{% endif %}</h3>
 </div>
 <div class="boxed-content {% if view_style == "grid" %}boxed-{{ grid_items }}{% endif %}">
    <div class="video-list {{ view_style }} {% if view_style == "grid" %}grid-{{ grid_items }}{% endif %}">
@@ -98,4 +106,4 @@
    </div>
 </div>
 <script type="text/javascript" src="{% static 'progress.js' %}"></script>
-{% endblock content %}
+{% endblock content %}
--- a/tubearchivist/home/templates/home/home.html
+++ b/tubearchivist/home/templates/home/home.html
@@ -45,7 +45,7 @@
    <div class="title-bar">
        <h1>Recent Videos</h1>
    </div>
-    <div class="view-controls">
+    <div class="view-controls three">
        <div class="toggle">
            <span>Hide watched:</span>
            <div class="toggleBox">
--- a/tubearchivist/home/templates/home/settings.html
+++ b/tubearchivist/home/templates/home/settings.html
@@ -77,7 +77,7 @@
                <ul>
                    <li><span class="settings-current">bestvideo[height<=720]+bestaudio/best[height<=720]</span>: best audio and max video height of 720p.</li>
                    <li><span class="settings-current">bestvideo[height<=1080]+bestaudio/best[height<=1080]</span>: best audio and max video height of 1080p.</li>
-                    <li><span class="settings-current">bestvideo[height<=1080][VCODEC=avc1]+bestaudio[ACODEC=mp4a]/mp4</span>: Max 1080p video height with iOS compatible video and audio codecs.</li>
+                    <li><span class="settings-current">bestvideo[height<=1080][VCODEC*=avc1]+bestaudio[ACODEC*=mp4a]/mp4</span>: Max 1080p video height with iOS compatible video and audio codecs.</li>
                    <li><span class="settings-current">0</span>: deactivate and download the best quality possible as decided by yt-dlp.</li>
                </ul>
                <i>Make sure your custom format gets merged into a single file. Check out the <a href="https://github.com/yt-dlp/yt-dlp#format-selection" target="_blank">documentation</a> for valid configurations.</i><br>
@@ -153,6 +153,23 @@
                {{ app_form.application_enable_cast }}
            </div>
        </div>
+        <div class="settings-group">
+            <h2 id="snapshots">Snapshots</h2>
+            <div class="settings-item">
+                <p>Current system snapshot: <span class="settings-current">{{ config.application.enable_snapshot }}</span></p>
+                <i>Automatically create daily deduplicated snapshots of the index, stored in Elasticsearch. Read first before activating: <a target="_blank" href="https://github.com/tubearchivist/tubearchivist/wiki/Settings#snapshots">Wiki</a>.</i><br>
+                {{ app_form.application_enable_snapshot }}
+            </div>
+            <div>
+                {% if snapshots %}
+                    <p>Create next snapshot: <span class="settings-current">{{ snapshots.next_exec_str }}</span>, snapshots expire after <span class="settings-current">{{ snapshots.expire_after }}</span>. <button onclick="createSnapshot()" id="createButton">Create snapshot now</button></p>
+                    <br>
+                    {% for snapshot in snapshots.snapshots %}
+                        <p><button id="{{ snapshot.id }}" onclick="restoreSnapshot(id)">Restore</button> Snapshot created on: <span class="settings-current">{{ snapshot.start_date }}</span>, took <span class="settings-current">{{ snapshot.duration_s }}s</span> to create. State: <i>{{ snapshot.state }}</i></p>
+                    {% endfor %}
+                {% endif %}
+            </div>
+        </div>
        <button type="submit" name="application-settings">Update Application Configurations</button>
    </form>
    <div class="title-bar">
--- a/tubearchivist/home/templates/home/video.html
+++ b/tubearchivist/home/templates/home/video.html
@@ -2,7 +2,9 @@
 {% block content %}
 {% load static %}
 {% load humanize %}
-<div class="video-main"></div>
+<div class="video-main">
+    <div class="video-modal"><span class="video-modal-text"></span></div>
+</div>
 <div class="notifications" id="notifications"></div>
 <div class="sponsorblock" id="sponsorblock">
    {% if video.sponsorblock.is_enabled %}
@@ -67,7 +69,7 @@
                    <p class="thumb-icon"><img class="dislike" src="{% static 'img/icon-thumb.svg' %}" alt="thumbs-down">: {{ video.stats.dislike_count|intcomma }}</p>
                {% endif %}
                {% if video.stats.average_rating %}
-                    <p class="rating-stars">Rating: 
+                    <p class="rating-stars">Rating:
                        {% for star in video.stats.average_rating %}
                            <img src="/static/img/icon-star-{{ star }}.svg" alt="{{ star }}">
                        {% endfor %}
@@ -90,7 +92,7 @@
                <a href="{% url 'playlist_id' playlist_item.playlist_meta.playlist_id %}">
                    <h3>Playlist [{{ playlist_item.playlist_meta.current_idx|add:"1" }}]: {{ playlist_item.playlist_meta.playlist_name }}</h3>
                </a>
-                <div class="playlist-nav">    
+                <div class="playlist-nav">
                    <div class="playlist-nav-item">
                        {% if playlist_item.playlist_previous %}
                            <a href="{% url 'video' playlist_item.playlist_previous.youtube_id %}">
--- a/tubearchivist/home/views.py
+++ b/tubearchivist/home/views.py
@@ -18,6 +18,7 @@ from django.views import View
 from home.src.download.yt_dlp_base import CookieHandler
 from home.src.es.connect import ElasticWrap
 from home.src.es.index_setup import get_available_backups
+from home.src.es.snapshot import ElasticSnapshot
 from home.src.frontend.api_calls import PostData
 from home.src.frontend.forms import (
    AddToQueueForm,
@@ -31,7 +32,7 @@ from home.src.frontend.forms import (
    UserSettingsForm,
 )
 from home.src.frontend.searching import SearchHandler
-from home.src.index.channel import channel_overwrites
+from home.src.index.channel import YoutubeChannel, channel_overwrites
 from home.src.index.generic import Pagination
 from home.src.index.playlist import YoutubePlaylist
 from home.src.ta.config import AppConfig, ScheduleBuilder
@@ -237,14 +238,10 @@ class ArchivistResultsView(ArchivistViewConfig):

    def initiate_vars(self, request):
        """search in es for vidoe hits"""
-        page_get = int(request.GET.get("page", 0))
        self.user_id = request.user.id
        self.config_builder(self.user_id)
        self.search_get = request.GET.get("search", False)
-        search_encoded = self._url_encode(self.search_get)
-        self.pagination_handler = Pagination(
-            page_get=page_get, user_id=self.user_id, search_get=search_encoded
-        )
+        self.pagination_handler = Pagination(request)
        self.sort_by = self._sort_by_overwrite()
        self._initial_data()

@@ -362,29 +359,80 @@ class DownloadView(ArchivistResultsView):
    def get(self, request):
        """handle get request"""
        self.initiate_vars(request)
-        self._update_view_data()
+        self._update_view_data(request)
        self.find_results()
        self.context.update(
            {
                "title": "Downloads",
                "add_form": AddToQueueForm(),
+                "channel_agg_list": self._get_channel_agg(),
            }
        )
        return render(request, "home/downloads.html", self.context)

-    def _update_view_data(self):
+    def _update_view_data(self, request):
        """update downloads view specific data dict"""
        if self.context["show_ignored_only"]:
            filter_view = "ignore"
        else:
            filter_view = "pending"
+
+        must_list = [{"term": {"status": {"value": filter_view}}}]
+
+        channel_filter = request.GET.get("channel", False)
+        if channel_filter:
+            must_list.append(
+                {"term": {"channel_id": {"value": channel_filter}}}
+            )
+
+            channel = YoutubeChannel(channel_filter)
+            channel.get_from_es()
+            self.context.update(
+                {
+                    "channel_filter_id": channel_filter,
+                    "channel_filter_name": channel.json_data["channel_name"],
+                }
+            )
+
        self.data.update(
            {
-                "query": {"term": {"status": {"value": filter_view}}},
+                "query": {"bool": {"must": must_list}},
                "sort": [{"timestamp": {"order": "asc"}}],
            }
        )

+    def _get_channel_agg(self):
+        """get pending channel with count"""
+        data = {
+            "size": 0,
+            "query": {"term": {"status": {"value": "pending"}}},
+            "aggs": {
+                "channel_downloads": {
+                    "multi_terms": {
+                        "size": 30,
+                        "terms": [
+                            {"field": "channel_name.keyword"},
+                            {"field": "channel_id"},
+                        ],
+                        "order": {"_count": "desc"},
+                    }
+                }
+            },
+        }
+        response, _ = ElasticWrap(self.es_search).get(data=data)
+        buckets = response["aggregations"]["channel_downloads"]["buckets"]
+
+        buckets_sorted = []
+        for i in buckets:
+            bucket = {
+                "name": i["key"][0],
+                "id": i["key"][1],
+                "count": i["doc_count"],
+            }
+            buckets_sorted.append(bucket)
+
+        return buckets_sorted
+
    @staticmethod
    def post(request):
        """handle post requests"""
@@ -414,7 +462,37 @@ class DownloadView(ArchivistResultsView):
        return redirect("downloads", permanent=True)


-class ChannelIdView(ArchivistResultsView):
+class ChannelIdBaseView(ArchivistResultsView):
+    """base class for all channel-id views"""
+
+    def get_channel_meta(self, channel_id):
+        """get metadata for channel"""
+        path = f"ta_channel/_doc/{channel_id}"
+        response, _ = ElasticWrap(path).get()
+        channel_info = SearchProcess(response).process()
+
+        return channel_info
+
+    def channel_has_pending(self, channel_id):
+        """check if channel has pending videos in queue"""
+        path = "ta_download/_search"
+        data = {
+            "size": 1,
+            "query": {
+                "bool": {
+                    "must": [
+                        {"term": {"status": {"value": "pending"}}},
+                        {"term": {"channel_id": {"value": channel_id}}},
+                    ]
+                }
+            },
+        }
+        response, _ = ElasticWrap(path).get(data=data)
+
+        self.context.update({"has_pending": bool(response["hits"]["hits"])})
+
+
+class ChannelIdView(ChannelIdBaseView):
    """resolves to /channel/<channel-id>/
    display single channel page from channel_id
    """
@@ -428,6 +506,7 @@ class ChannelIdView(ArchivistResultsView):
        self._update_view_data(channel_id)
        self.find_results()
        self.match_progress()
+        self.channel_has_pending(channel_id)

        if self.context["results"]:
            channel_info = self.context["results"][0]["source"]["channel"]
@@ -478,7 +557,7 @@ class ChannelIdView(ArchivistResultsView):
        return redirect("channel_id", channel_id, permanent=True)


-class ChannelIdAboutView(ArchivistResultsView):
+class ChannelIdAboutView(ChannelIdBaseView):
    """resolves to /channel/<channel-id>/about/
    show metadata, handle per channel conf
    """
@@ -488,6 +567,7 @@ class ChannelIdAboutView(ArchivistResultsView):
    def get(self, request, channel_id):
        """handle get request"""
        self.initiate_vars(request)
+        self.channel_has_pending(channel_id)

        path = f"ta_channel/_doc/{channel_id}"
        response, _ = ElasticWrap(path).get()
@@ -521,7 +601,7 @@ class ChannelIdAboutView(ArchivistResultsView):
        return redirect("channel_id_about", channel_id, permanent=True)


-class ChannelIdPlaylistView(ArchivistResultsView):
+class ChannelIdPlaylistView(ChannelIdBaseView):
    """resolves to /channel/<channel-id>/playlist/
    show all playlists of channel
    """
@@ -534,8 +614,9 @@ class ChannelIdPlaylistView(ArchivistResultsView):
        self.initiate_vars(request)
        self._update_view_data(channel_id)
        self.find_results()
+        self.channel_has_pending(channel_id)

-        channel_info = self._get_channel_meta(channel_id)
+        channel_info = self.get_channel_meta(channel_id)
        channel_name = channel_info["channel_name"]
        self.context.update(
            {
@@ -556,14 +637,6 @@ class ChannelIdPlaylistView(ArchivistResultsView):

        self.data["query"] = {"bool": {"must": must_list}}

-    def _get_channel_meta(self, channel_id):
-        """get metadata for channel"""
-        path = f"ta_channel/_doc/{channel_id}"
-        response, _ = ElasticWrap(path).get()
-        channel_info = SearchProcess(response).process()
-
-        return channel_info
-

 class ChannelView(ArchivistResultsView):
    """resolves to /channel/
@@ -870,6 +943,7 @@ class SettingsView(View):
        user_form = UserSettingsForm()
        app_form = ApplicationSettingsForm()
        scheduler_form = SchedulerSettingsForm()
+        snapshots = ElasticSnapshot().get_snapshot_stats()
        token = self.get_token(request)

        context = {
@@ -881,6 +955,7 @@ class SettingsView(View):
            "user_form": user_form,
            "app_form": app_form,
            "scheduler_form": scheduler_form,
+            "snapshots": snapshots,
            "version": settings.TA_VERSION,
        }

@@ -928,6 +1003,8 @@ class SettingsView(View):
        for config_value, updated_value in updated:
            if config_value == "cookie_import":
                self.process_cookie(config, updated_value)
+            if config_value == "enable_snapshot":
+                ElasticSnapshot().setup()

    def process_cookie(self, config, updated_value):
        """import and validate cookie"""
--- a/tubearchivist/requirements.txt
+++ b/tubearchivist/requirements.txt
@@ -1,13 +1,13 @@
 beautifulsoup4==4.11.1
 celery==5.2.7
-Django==4.0.6
+Django==4.1.3
 django-auth-ldap==4.1.0
 django-cors-headers==3.13.0
-djangorestframework==3.13.1
-Pillow==9.2.0
+djangorestframework==3.14.0
+Pillow==9.3.0
 redis==4.3.4
 requests==2.28.1
 ryd-client==0.0.6
-uWSGI==2.0.20
+uWSGI==2.0.21
 whitenoise==6.2.0
-yt_dlp==2022.9.1
+yt_dlp==2022.10.4
--- a/tubearchivist/static/cast-videos.js
+++ b/tubearchivist/static/cast-videos.js
@@ -1,148 +1,157 @@
+'use strict';
+
+/* global cast chrome getVideoPlayerVideoId postVideoProgress setProgressBar getVideoPlayer getVideoPlayerWatchStatus watchedThreshold isWatched getVideoData getURL getVideoPlayerCurrentTime */
+
 function initializeCastApi() {
-    cast.framework.CastContext.getInstance().setOptions({
-        receiverApplicationId: chrome.cast.media.DEFAULT_MEDIA_RECEIVER_APP_ID, // Use built in receiver app on cast device, see https://developers.google.com/cast/docs/styled_receiver if you want to be able to add a theme, splash screen or watermark. Has a $5 one time fee.
-        autoJoinPolicy: chrome.cast.AutoJoinPolicy.ORIGIN_SCOPED
-    });
+  cast.framework.CastContext.getInstance().setOptions({
+    receiverApplicationId: chrome.cast.media.DEFAULT_MEDIA_RECEIVER_APP_ID, // Use built in receiver app on cast device, see https://developers.google.com/cast/docs/styled_receiver if you want to be able to add a theme, splash screen or watermark. Has a $5 one time fee.
+    autoJoinPolicy: chrome.cast.AutoJoinPolicy.ORIGIN_SCOPED,
+  });

-    var player = new cast.framework.RemotePlayer();
-    var playerController = new cast.framework.RemotePlayerController(player);
+  let player = new cast.framework.RemotePlayer();
+  let playerController = new cast.framework.RemotePlayerController(player);

-    // Add event listerner to check if a connection to a cast device is initiated
-    playerController.addEventListener(
-        cast.framework.RemotePlayerEventType.IS_CONNECTED_CHANGED, function() { 
-            castConnectionChange(player) 
-        }
-    );
-    playerController.addEventListener(
-        cast.framework.RemotePlayerEventType.CURRENT_TIME_CHANGED, function() { 
-            castVideoProgress(player) 
-        }
-    );
-    playerController.addEventListener(
-        cast.framework.RemotePlayerEventType.IS_PAUSED_CHANGED, function() { 
-            castVideoPaused(player) 
-        }
-    );
+  // Add event listerner to check if a connection to a cast device is initiated
+  playerController.addEventListener(
+    cast.framework.RemotePlayerEventType.IS_CONNECTED_CHANGED,
+    function () {
+      castConnectionChange(player);
+    }
+  );
+  playerController.addEventListener(
+    cast.framework.RemotePlayerEventType.CURRENT_TIME_CHANGED,
+    function () {
+      castVideoProgress(player);
+    }
+  );
+  playerController.addEventListener(
+    cast.framework.RemotePlayerEventType.IS_PAUSED_CHANGED,
+    function () {
+      castVideoPaused(player);
+    }
+  );
 }

-
 function castConnectionChange(player) {
-    // If cast connection is initialized start cast
-    if (player.isConnected) {
-        // console.log("Cast Connected.");
-        castStart();
-    } else if (!player.isConnected) {
-        // console.log("Cast Disconnected.");
-    }
+  // If cast connection is initialized start cast
+  if (player.isConnected) {
+    // console.log("Cast Connected.");
+    castStart();
+  } else if (!player.isConnected) {
+    // console.log("Cast Disconnected.");
+  }
 }

 function castVideoProgress(player) {
-    var videoId = getVideoPlayerVideoId();
-    if (player.mediaInfo.contentId.includes(videoId)) {
-        var currentTime = player.currentTime;
-        var duration = player.duration;
-        if ((currentTime % 10) <= 1.0 && currentTime != 0 && duration != 0) { // Check progress every 10 seconds or else progress is checked a few times a second
-            postVideoProgress(videoId, currentTime);
-            setProgressBar(videoId, currentTime, duration);
-            if (!getVideoPlayerWatchStatus()) { // Check if video is already marked as watched
-                if (watchedThreshold(currentTime, duration)) {
-                    isWatched(videoId);
-                }
-            }
+  let videoId = getVideoPlayerVideoId();
+  if (player.mediaInfo.contentId.includes(videoId)) {
+    let currentTime = player.currentTime;
+    let duration = player.duration;
+    if (currentTime % 10 <= 1.0 && currentTime !== 0 && duration !== 0) {
+      // Check progress every 10 seconds or else progress is checked a few times a second
+      postVideoProgress(videoId, currentTime);
+      setProgressBar(videoId, currentTime, duration);
+      if (!getVideoPlayerWatchStatus()) {
+        // Check if video is already marked as watched
+        if (watchedThreshold(currentTime, duration)) {
+          isWatched(videoId);
        }
+      }
    }
+  }
 }

 function castVideoPaused(player) {
-    var videoId = getVideoPlayerVideoId();
-    var currentTime = player.currentTime;
-    var duration = player.duration;
-    if (player.mediaInfo != null) {
-        if (player.mediaInfo.contentId.includes(videoId)) {
-            if (currentTime != 0 && duration != 0) {
-                postVideoProgress(videoId, currentTime);
-            }
-        }
+  let videoId = getVideoPlayerVideoId();
+  let currentTime = player.currentTime;
+  let duration = player.duration;
+  if (player.mediaInfo != null) {
+    if (player.mediaInfo.contentId.includes(videoId)) {
+      if (currentTime !== 0 && duration !== 0) {
+        postVideoProgress(videoId, currentTime);
+      }
    }
+  }
 }

 function castStart() {
-    var castSession = cast.framework.CastContext.getInstance().getCurrentSession();
-    // Check if there is already media playing on the cast target to prevent recasting on page reload or switching to another video page
-    if (!castSession.getMediaSession()) { 
-        var videoId = getVideoPlayerVideoId();
-        var videoData = getVideoData(videoId);
-        var contentId = getURL() + videoData.data.media_url;
-        var contentTitle = videoData.data.title;
-        var contentImage = getURL() + videoData.data.vid_thumb_url;
+  let castSession = cast.framework.CastContext.getInstance().getCurrentSession();
+  // Check if there is already media playing on the cast target to prevent recasting on page reload or switching to another video page
+  if (!castSession.getMediaSession()) {
+    let videoId = getVideoPlayerVideoId();
+    let videoData = getVideoData(videoId);
+    let contentId = getURL() + videoData.data.media_url;
+    let contentTitle = videoData.data.title;
+    let contentImage = getURL() + videoData.data.vid_thumb_url;

-        contentType = 'video/mp4'; // Set content type, only videos right now so it is hard coded
-        contentCurrentTime = getVideoPlayerCurrentTime(); // Get video's current position
-        contentActiveSubtitle = [];
-        // Check if a subtitle is turned on.
-        for (var i = 0; i < getVideoPlayer().textTracks.length; i++) {
-            if (getVideoPlayer().textTracks[i].mode == "showing") {
-                contentActiveSubtitle =[i + 1];
-            }
-        }
-        contentSubtitles = [];
-        var videoSubtitles = videoData.data.subtitles; // Array of subtitles
-        if (typeof(videoSubtitles) != 'undefined' && videoData.config.downloads.subtitle) {
-            for (var i = 0; i < videoSubtitles.length; i++) {
-                subtitle = new chrome.cast.media.Track(i, chrome.cast.media.TrackType.TEXT);
-                subtitle.trackContentId = videoSubtitles[i].media_url;
-                subtitle.trackContentType = 'text/vtt';
-                subtitle.subtype = chrome.cast.media.TextTrackType.SUBTITLES;
-                subtitle.name = videoSubtitles[i].name;
-                subtitle.language = videoSubtitles[i].lang;
-                subtitle.customData = null;
-                contentSubtitles.push(subtitle);
-            }
-        }
+    let contentType = 'video/mp4'; // Set content type, only videos right now so it is hard coded
+    let contentCurrentTime = getVideoPlayerCurrentTime(); // Get video's current position
+    let contentActiveSubtitle = [];
+    // Check if a subtitle is turned on.
+    for (let i = 0; i < getVideoPlayer().textTracks.length; i++) {
+      if (getVideoPlayer().textTracks[i].mode === 'showing') {
+        contentActiveSubtitle = [i + 1];
+      }
+    }
+    let contentSubtitles = [];
+    let videoSubtitles = videoData.data.subtitles; // Array of subtitles
+    if (typeof videoSubtitles !== 'undefined' && videoData.config.downloads.subtitle) {
+      for (let i = 0; i < videoSubtitles.length; i++) {
+        let subtitle = new chrome.cast.media.Track(i, chrome.cast.media.TrackType.TEXT);
+        subtitle.trackContentId = videoSubtitles[i].media_url;
+        subtitle.trackContentType = 'text/vtt';
+        subtitle.subtype = chrome.cast.media.TextTrackType.SUBTITLES;
+        subtitle.name = videoSubtitles[i].name;
+        subtitle.language = videoSubtitles[i].lang;
+        subtitle.customData = null;
+        contentSubtitles.push(subtitle);
+      }
+    }

-        mediaInfo = new chrome.cast.media.MediaInfo(contentId, contentType); // Create MediaInfo var that contains url and content type
-        // mediaInfo.streamType = chrome.cast.media.StreamType.BUFFERED; // Set type of stream, BUFFERED, LIVE, OTHER
-        mediaInfo.metadata = new chrome.cast.media.GenericMediaMetadata(); // Create metadata var and add it to MediaInfo
-        mediaInfo.metadata.title = contentTitle.replace("&amp;", "&"); // Set the video title
-        mediaInfo.metadata.images = [new chrome.cast.Image(contentImage)]; // Set the video thumbnail
-        // mediaInfo.textTrackStyle = new chrome.cast.media.TextTrackStyle();
-        mediaInfo.tracks = contentSubtitles;
+    let mediaInfo = new chrome.cast.media.MediaInfo(contentId, contentType); // Create MediaInfo var that contains url and content type
+    // mediaInfo.streamType = chrome.cast.media.StreamType.BUFFERED; // Set type of stream, BUFFERED, LIVE, OTHER
+    mediaInfo.metadata = new chrome.cast.media.GenericMediaMetadata(); // Create metadata var and add it to MediaInfo
+    mediaInfo.metadata.title = contentTitle.replace('&amp;', '&'); // Set the video title
+    mediaInfo.metadata.images = [new chrome.cast.Image(contentImage)]; // Set the video thumbnail
+    // mediaInfo.textTrackStyle = new chrome.cast.media.TextTrackStyle();
+    mediaInfo.tracks = contentSubtitles;

-        var request = new chrome.cast.media.LoadRequest(mediaInfo); // Create request with the previously set MediaInfo. 
-        // request.queueData = new chrome.cast.media.QueueData(); // See https://developers.google.com/cast/docs/reference/web_sender/chrome.cast.media.QueueData for playlist support.
-        request.currentTime = shiftCurrentTime(contentCurrentTime); // Set video start position based on the browser video position
-        request.activeTrackIds = contentActiveSubtitle; // Set active subtitle based on video player
-        // request.autoplay = false; // Set content to auto play, true by default
-        castSession.loadMedia(request).then(
-            function() { 
-                castSuccessful();
-            }, 
-            function() { 
-                castFailed(errorCode); 
-            }
-        ); // Send request to cast device
-    }   
+    let request = new chrome.cast.media.LoadRequest(mediaInfo); // Create request with the previously set MediaInfo.
+    // request.queueData = new chrome.cast.media.QueueData(); // See https://developers.google.com/cast/docs/reference/web_sender/chrome.cast.media.QueueData for playlist support.
+    request.currentTime = shiftCurrentTime(contentCurrentTime); // Set video start position based on the browser video position
+    request.activeTrackIds = contentActiveSubtitle; // Set active subtitle based on video player
+    // request.autoplay = false; // Set content to auto play, true by default
+    castSession.loadMedia(request).then(
+      function () {
+        castSuccessful();
+      },
+      function (error) {
+        castFailed(error.code);
+      }
+    ); // Send request to cast device
+  }
 }

-function shiftCurrentTime(contentCurrentTime) { // Shift media back 3 seconds to prevent missing some of the content
-    if (contentCurrentTime > 5) {
-        return(contentCurrentTime - 3);
-    } else {
-        return(0);
-    }
+function shiftCurrentTime(contentCurrentTime) {
+  // Shift media back 3 seconds to prevent missing some of the content
+  if (contentCurrentTime > 5) {
+    return contentCurrentTime - 3;
+  } else {
+    return 0;
+  }
 }

 function castSuccessful() {
-    // console.log('Cast Successful.');
-    getVideoPlayer().pause(); // Pause browser video on successful cast
+  // console.log('Cast Successful.');
+  getVideoPlayer().pause(); // Pause browser video on successful cast
 }

 function castFailed(errorCode) {
-    console.log('Error code: ' + errorCode);
+  console.log('Error code: ' + errorCode);
 }

-window['__onGCastApiAvailable'] = function(isAvailable) {
-    if (isAvailable) {
-        initializeCastApi();
-    }
-}
+window['__onGCastApiAvailable'] = function (isAvailable) {
+  if (isAvailable) {
+    initializeCastApi();
+  }
+};
--- a/tubearchivist/static/css/style.css
+++ b/tubearchivist/static/css/style.css
@@ -344,6 +344,7 @@ button:hover {
 .grid-count {
    display: flex;
    justify-content: end;
+    align-items: center;
 }

 .view-icons img {
@@ -387,6 +388,7 @@ button:hover {
    display: grid;
    align-content: space-evenly;
    height: 100vh;
+    position: relative; /* needed for modal */
 }

 .notifications {
@@ -743,6 +745,22 @@ video:-webkit-full-screen {
 /* video page */
 .video-main {
    margin: 1rem 0;
+    position: relative; /* needed for modal */
+}
+
+.video-modal {
+    position: absolute;
+    z-index: 1;
+    top: 20%;
+    width: 100%;
+    text-align: center;
+}
+
+.video-modal-text {
+    background: rgba(0,0,0,.5);
+    color: #eeeeee;
+    font-size: 1.3em;
+    display: none;
 }

 .video-main video {
@@ -1164,6 +1182,10 @@ video:-webkit-full-screen {
        margin: 15px;
        text-align: center;
    }
+    .view-controls.three {
+        grid-template-columns: unset;
+        justify-content: center;
+    }
    .sort {
        display: block;
    }
--- a/tubearchivist/static/progress.js
+++ b/tubearchivist/static/progress.js
@@ -1,106 +1,110 @@
 /**
 * Handle multi channel notifications
- * 
+ *
 */

-checkMessages()
+'use strict';
+
+checkMessages();

 // page map to notification status
 const messageTypes = {
-    "download": ["message:download", "message:add", "message:rescan", "message:playlistscan"],
-    "channel": ["message:subchannel"],
-    "channel_id": ["message:playlistscan"],
-    "playlist": ["message:subplaylist"],
-    "setting": ["message:setting"]
-}
+  download: ['message:download', 'message:add', 'message:rescan', 'message:playlistscan'],
+  channel: ['message:subchannel'],
+  channel_id: ['message:playlistscan'],
+  playlist: ['message:subplaylist'],
+  setting: ['message:setting'],
+};

 // start to look for messages
 function checkMessages() {
-    var notifications = document.getElementById("notifications");
-    if (notifications) {
-        var dataOrigin = notifications.getAttribute("data");
-        getMessages(dataOrigin);
-    }
+  let notifications = document.getElementById('notifications');
+  if (notifications) {
+    let dataOrigin = notifications.getAttribute('data');
+    getMessages(dataOrigin);
+  }
 }

 // get messages for page on timer
 function getMessages(dataOrigin) {
-    fetch('/progress/').then(response => {
-        return response.json();
-    }).then(responseData => {
-        var messages = buildMessage(responseData, dataOrigin);
-        if (messages.length > 0) {
-            // restart itself
-            setTimeout(function() {
-                getMessages(dataOrigin);
-            }, 3000);
-        };
+  fetch('/progress/')
+    .then(response => {
+      return response.json();
+    })
+    .then(responseData => {
+      let messages = buildMessage(responseData, dataOrigin);
+      if (messages.length > 0) {
+        // restart itself
+        setTimeout(function () {
+          getMessages(dataOrigin);
+        }, 3000);
+      }
    });
 }

 // make div for all messages, return relevant
 function buildMessage(responseData, dataOrigin) {
-    // filter relevan messages
-    var allMessages = responseData["messages"];
-    var messages = allMessages.filter(function(value) {
-        return messageTypes[dataOrigin].includes(value["status"])
-    }, dataOrigin);
-    // build divs
-    var notificationDiv = document.getElementById("notifications");
-    var nots = notificationDiv.childElementCount;
-    notificationDiv.innerHTML = "";
-    for (let i = 0; i < messages.length; i++) {
-        var messageData = messages[i];
-        var messageStatus = messageData["status"];
-        var messageBox = document.createElement("div");
-        var title = document.createElement("h3");
-        title.innerHTML = messageData["title"];
-        var message = document.createElement("p");
-        message.innerHTML = messageData["message"];
-        messageBox.appendChild(title);
-        messageBox.appendChild(message);
-        messageBox.classList.add(messageData["level"], "notification");
-        notificationDiv.appendChild(messageBox);
-        if (messageStatus === "message:download") {
-            checkDownloadIcons();
-        };
-    };
-    // reload page when no more notifications
-    if (nots > 0 && messages.length === 0) {
-        location.reload();
-    };
-    return messages
+  // filter relevan messages
+  let allMessages = responseData['messages'];
+  let messages = allMessages.filter(function (value) {
+    return messageTypes[dataOrigin].includes(value['status']);
+  }, dataOrigin);
+  // build divs
+  let notificationDiv = document.getElementById('notifications');
+  let nots = notificationDiv.childElementCount;
+  notificationDiv.innerHTML = '';
+  for (let i = 0; i < messages.length; i++) {
+    let messageData = messages[i];
+    let messageStatus = messageData['status'];
+    let messageBox = document.createElement('div');
+    let title = document.createElement('h3');
+    title.innerHTML = messageData['title'];
+    let message = document.createElement('p');
+    message.innerHTML = messageData['message'];
+    messageBox.appendChild(title);
+    messageBox.appendChild(message);
+    messageBox.classList.add(messageData['level'], 'notification');
+    notificationDiv.appendChild(messageBox);
+    if (messageStatus === 'message:download') {
+      checkDownloadIcons();
+    }
+  }
+  // reload page when no more notifications
+  if (nots > 0 && messages.length === 0) {
+    location.reload();
+  }
+  return messages;
 }

 // check if download icons are needed
 function checkDownloadIcons() {
-    var iconBox = document.getElementById("downloadControl");
-    if (iconBox.childElementCount === 0) {
-        var downloadIcons = buildDownloadIcons();
-        iconBox.appendChild(downloadIcons);
-    };
+  let iconBox = document.getElementById('downloadControl');
+  if (iconBox.childElementCount === 0) {
+    let downloadIcons = buildDownloadIcons();
+    iconBox.appendChild(downloadIcons);
+  }
 }

 // add dl control icons
 function buildDownloadIcons() {
-    var downloadIcons = document.createElement('div');
-    downloadIcons.classList = 'dl-control-icons';
-    // stop icon
-    var stopIcon = document.createElement('img');
-    stopIcon.setAttribute('id', "stop-icon");
-    stopIcon.setAttribute('title', "Stop Download Queue");
-    stopIcon.setAttribute('src', "/static/img/icon-stop.svg");
-    stopIcon.setAttribute('alt', "stop icon");
-    stopIcon.setAttribute('onclick', 'stopQueue()');
-    // kill icon
-    var killIcon = document.createElement('img');
-    killIcon.setAttribute('id', "kill-icon");
-    killIcon.setAttribute('title', "Kill Download Queue");
-    killIcon.setAttribute('src', "/static/img/icon-close.svg");
-    killIcon.setAttribute('alt', "kill icon");
-    killIcon.setAttribute('onclick', 'killQueue()');
-    // stich together
-    downloadIcons.appendChild(stopIcon);
-    downloadIcons.appendChild(killIcon);
-    return downloadIcons
+  let downloadIcons = document.createElement('div');
+  downloadIcons.classList = 'dl-control-icons';
+  // stop icon
+  let stopIcon = document.createElement('img');
+  stopIcon.setAttribute('id', 'stop-icon');
+  stopIcon.setAttribute('title', 'Stop Download Queue');
+  stopIcon.setAttribute('src', '/static/img/icon-stop.svg');
+  stopIcon.setAttribute('alt', 'stop icon');
+  stopIcon.setAttribute('onclick', 'stopQueue()');
+  // kill icon
+  let killIcon = document.createElement('img');
+  killIcon.setAttribute('id', 'kill-icon');
+  killIcon.setAttribute('title', 'Kill Download Queue');
+  killIcon.setAttribute('src', '/static/img/icon-close.svg');
+  killIcon.setAttribute('alt', 'kill icon');
+  killIcon.setAttribute('onclick', 'killQueue()');
+  // stich together
+  downloadIcons.appendChild(stopIcon);
+  downloadIcons.appendChild(killIcon);
+  return downloadIcons;
 }
--- a/tubearchivist/static/script.js
+++ b/tubearchivist/static/script.js
Author	SHA1	Message	Date
simon	eda56236ce	bump TA_VERSION	2022-11-05 11:58:41 +07:00
simon	b4620504b8	improve manual import docs	2022-11-05 11:36:53 +07:00
simon	b751ae183a	fix startup migration error missing enable_snapshot var	2022-11-05 11:17:12 +07:00
simon	0b2b8aacef	add snapshot wiki link	2022-11-05 09:35:28 +07:00
simon	e21d66ce85	add video player shortcuts docs	2022-11-03 18:18:51 +07:00
simon	66b72da02a	update fuzzy search docs	2022-11-03 18:08:50 +07:00
simon	e0d6ce6c7f	update snapshot docs	2022-11-03 17:58:17 +07:00
simon	97e9d6f8ce	add path.repo env var to es container	2022-11-03 17:34:03 +07:00
crocs	0fb2f3fcc6	Changed the wording in a few places (#357 ) * Update channel.html Updated wording, removed "matching" as it's not necessary. * Update downloads.html	2022-11-03 17:14:55 +07:00
simon	32fc89f4e7	configure fuzzines and snapshot, #build Changed: - Added fuzzy: secondary keyword for search - Added take snapshot now button - Changed snapshot policy to 12pm TZ aware - Fixed channel page size validator > 0	2022-11-02 12:29:00 +07:00
simon	10fbfadf51	bump TA_VERSION unstable	2022-11-02 12:23:34 +07:00
simon	e9c457cac2	bump django	2022-11-02 12:18:17 +07:00
simon	0f191d8a3e	configure fuzziness in search	2022-11-02 11:59:31 +07:00
simon	dcf317e471	form validate channel pagesize greater than 0, #334	2022-11-02 10:30:21 +07:00
simon	749261c146	take snapshot at 12 localtime	2022-11-02 10:17:27 +07:00
simon	238c6bc080	Merge branch 'master' into testing	2022-11-02 08:42:24 +07:00
simon	476d4d4909	add create snapshot now button	2022-10-31 21:21:28 +07:00
simon	651a642e71	bump pillow	2022-10-29 21:33:16 +07:00
simon	01a8c65df9	move version_check.py to separate arg	2022-10-29 21:21:22 +07:00
simon	d69460bf98	System snapshots, #build Changed: - Added: Dedublicated snapshots, read docs/Settings#snapshots first - Added: Actions for better Python and JS linting - Changed: Clean up JS - Changed: Use patched ffmpeg builds for arm64 - API: Added endpoints to interact with snapshots - Fixed: mobile layout for channel filter dopdown on downloads	2022-10-29 21:10:04 +07:00
lamusmaser	f0848283bd	README updated with format update (#353 ) * Add update to README to include format adjustments under Getting Started. * Include updates to the Settings page of the Wiki.	2022-10-29 21:04:49 +07:00
simon	fdc1df313b	add partial snapshots docs	2022-10-29 21:03:27 +07:00
simon	b425e7d75c	frontend API button to restore snapshot	2022-10-29 20:12:54 +07:00
simon	fb25b76950	fix spelling	2022-10-29 17:45:20 +07:00
simon	a18723991a	implement snapshot API views	2022-10-29 17:43:38 +07:00
simon	74aa134971	fix next_exec epoch time builder	2022-10-29 17:43:14 +07:00
simon	24b02150a7	handle delete, restore snapshot return values	2022-10-29 17:27:04 +07:00
simon	f717c71c66	implement _parse_single_snapshot, slug compatible naming	2022-10-29 16:54:16 +07:00
simon	814de87768	add snapshot check at startup	2022-10-29 15:57:59 +07:00
simon	c392563ef8	use exist_ok for os.makedirs at startup	2022-10-29 15:52:33 +07:00
simon	1fbd603374	frontend implementation for snapshot integration	2022-10-29 15:49:47 +07:00
simon	749f61aba6	fix empty policy building	2022-10-29 15:32:08 +07:00
simon	59f347c135	implement snapshot startup check	2022-10-28 16:47:50 +07:00
simon	904f449d37	combine snapshot_stats for frontend	2022-10-27 18:40:27 +07:00
simon	9b69a8dc91	fix mobile view-controls break	2022-10-27 16:15:32 +07:00
Kevin Gibbons	50ea7db66c	Add FAQ item about downloading vs subscribing (#352 ) * Add FAQ item about downloading vs subscribing * note download+subscribe	2022-10-27 13:44:16 +07:00
lamusmaser	e4c5529412	Update Github Actions to use `deploy.sh` and remove individual calls for testing functions. (#351 ) * Remove individual calls for testing functions. Call `deploy.sh validate` for linting checks. * Increment actions versions to remove deprecation errors. Add `requests` to `pip install` line. * Specify Python version with `actions/setup-python` call.	2022-10-27 13:30:24 +07:00
Kevin Gibbons	b4fe9a784f	add github workflow to enforce linting of JS files (#348 )	2022-10-27 13:27:30 +07:00
BrianCArnold	6b2fe125d4	Add LDAP attribute mapping env variables. (#344 ) * Added ability to specify LDAP attribute mapping. Specifically made sure that LDAP auth works the same if the new environment variables aren't set, in order to maintain behavior for users who are already using LDAP if they don't set the new envvars. * Updated env var name to match the name of the parent global. * Updated README.md to include information on new user attribute mapping environment variables. * Added additional environment var options, and updated comment explaining why there are multiple. * I'm not a python programmer, so these were stupid mistakes. Works now. * Addressing lint error. * Finished updating formatting according to black linter. * Shortened comments to fit within line length. * fix whitespace linting Co-authored-by: simon <simobilleter@gmail.com>	2022-10-27 13:20:41 +07:00
simon	e7f960bf46	implement basics snapshot management	2022-10-26 23:17:55 +07:00
simon	02ed521f21	bump uwsgi	2022-10-26 23:17:03 +07:00
Kevin Gibbons	39902cb1c6	Format/slightly modernize the JS (#345 ) * add basic JS tooling * fix accidental uses of global variables * auto-format * add and fix a couple more standard lint rules * remove useless return false from settimeout callbacks * document JS contributing * fix whitespace in package.json * add JS stuff to codespell skiplist * codespell take two * update github action and add comments about duplicated logic	2022-10-26 09:43:58 +07:00
simon	700a8cb54a	remove leftover ffmpeg binary	2022-10-25 12:13:11 +07:00
simon	3eacc06383	Player shortcuts, #build Changed: - Added player shortcuts, '?' - Fixed playlist missing channel metadata - Changed arm64 build to use patched ffmpeg	2022-10-25 11:25:12 +07:00
simon	f8dccfcd15	use linuxarm64 ffmpeg builds for arm64	2022-10-25 11:22:32 +07:00
simon	725f17bcd2	fix playlist missing channel metadata	2022-10-25 11:13:14 +07:00
Kevin Gibbons	ff82690d3c	add keyboard shortcuts to video player (#342 ) * add keyboard shortcuts to video player * fix modal on the inline player	2022-10-24 20:11:00 +07:00
Kevin Gibbons	51f7210195	More detailed contributing steps (#343 ) * More detailed contributing steps * restore whitespace	2022-10-24 19:59:29 +07:00
Kevin Gibbons	25d2ddd08a	More detailed install instructions (#340 )	2022-10-24 10:38:52 +07:00
simon	4af12aee15	better indentation formatting	2022-10-23 14:14:30 +07:00
simon	ed341fbc51	add cast auth comment to wiki	2022-10-23 14:11:40 +07:00
simon	f0874b2d02	add timeout for sponsorblock api requests, handle 503	2022-10-23 12:46:10 +07:00
simon	baacd3ee39	better message for running and indexing queue	2022-10-23 12:21:27 +07:00
simon	9cd23c3666	error handeling for playlist_thumbnail extraction	2022-10-23 11:50:26 +07:00
simon	0e17e2a6cf	bump TA_VERSION to v0.2.3	2022-10-23 10:55:57 +07:00
simon	2dea0aa57b	bump archivist-es	2022-10-23 10:55:14 +07:00
simon	ba1c8c15c4	fix missing build-arg for local deployment	2022-10-23 10:50:16 +07:00
simon	f16915be11	clarify filter only shows when more than 1	2022-10-23 10:49:26 +07:00
simon	858d437f3f	simplify local production deployment	2022-10-23 00:11:03 +07:00
simon	de30ac302a	add documentation for download channel filter	2022-10-23 00:10:38 +07:00
simon	dd8597307c	fix wrong python version in final image	2022-10-22 23:30:44 +07:00
simon	608403c113	Download filter select, #build Changed: - added download filter dropdown - fix for UnidentifiedImageError thumbnail error - fix for _update_by_query channel missing watched_date - fix for chrome compatibility text reveal	2022-10-22 22:55:38 +07:00
simon	40eff8e30e	fix chrome compatibility issue for description text reveal, #327	2022-10-22 22:32:57 +07:00
simon	0bba36cbc3	add watched_date for _update_by_query mark as watched, #309	2022-10-22 22:00:35 +07:00
simon	a5788117de	add dropdown channel agg for download page	2022-10-22 21:23:57 +07:00
simon	3f1075d0b2	bump python version	2022-10-22 20:24:58 +07:00
simon	dea2688b49	handle UnidentifiedImageError in ThumbManager, #325	2022-10-17 19:26:01 +07:00
simon	4f1daeb18c	Downloads channel filter, #build Changed: - Added downloads channel filter to channel pages - API: Filter download list view by channel - Fixed: is_live status check	2022-10-17 19:01:59 +07:00
simon	927e6fa909	create channel parameter for downloads api view	2022-10-17 18:58:21 +07:00
simon	bd7cdb3942	append query parameters to pagination	2022-10-17 18:40:20 +07:00
simon	9dfd967a32	implement downloads filter per channel	2022-10-17 13:29:21 +07:00
simon	6eee762d3a	add status for sub refresh	2022-10-14 11:27:27 +07:00
simon	a8c5773f81	fix is_live status check before adding to queue	2022-10-05 16:12:58 +02:00
simon	fbb52dc93f	implement basic channel query string for downloads page	2022-10-05 15:47:17 +02:00
simon	c9e936da21	bump libraries	2022-10-05 15:46:44 +02:00
simon	c825e67f69	bump django and restframework	2022-09-30 18:02:44 +02:00
simon	bcb7b9443b	bump tubearchivist-es	2022-09-19 15:52:22 +07:00
simon	896d166dcf	add minimal system requirements documentation	2022-09-19 15:40:51 +07:00