merge for v0.1.5 build

Changed:
- Added cookie validation button
- Fixed all the open bug reports! :-)

.github/workflows/lint_python.yml

@@ -9,10 +9,9 @@ jobs:
       - run: pip install --upgrade pip wheel
       - run: pip install bandit black codespell flake8 flake8-bugbear
                          flake8-comprehensions isort
-      - run: bandit --recursive  --skip B105,B108,B404,B603,B607 .
       - run: black --check --diff --line-length 79 .
       - run: codespell
-      - run: flake8 . --count --max-complexity=12 --max-line-length=79
+      - run: flake8 . --count --max-complexity=10 --max-line-length=79
                       --show-source --statistics
       - run: isort --check-only --line-length 79 --profile black .
       # - run: pip install -r tubearchivist/requirements.txt

.gitignore

@@ -3,3 +3,6 @@ __pycache__
 
 # django testing db
 db.sqlite3
+
+# vscode custom conf
+.vscode

.vscode/settings.json

@@ -1,5 +0,0 @@
-{
-    "python.linting.pylintEnabled": true,
-    "python.linting.pycodestyleEnabled": false,
-    "python.linting.enabled": true
-}

CONTRIBUTING.md

@@ -5,7 +5,7 @@ If you haven't already, the best place to start is the README. This will give yo
 
 ## Report a bug
 
-If you notice something is not working as expected, check to see if it has been previously reported in the [open issues](https://github.com/bbilly1/tubearchivist/issues).
+If you notice something is not working as expected, check to see if it has been previously reported in the [open issues](https://github.com/tubearchivist/tubearchivist/issues).
 If it has not yet been disclosed, go ahead and create an issue.  
 If the issue doesn't move forward due to a lack of response, I assume it's solved and will close it after some time to keep the list fresh. 
 
@@ -19,24 +19,24 @@ I have learned the hard way, that working on a dockerized application outside of
 
 This is my setup I have landed on, YMMV:
 - Clone the repo, work on it with your favorite code editor in your local filesystem. *testing* branch is the where all the changes are happening, might be unstable and is WIP.
-- Then I have a VM on KVM hypervisor running standard Ubuntu Server LTS with docker installed. The VM keeps my projects separate and offers convenient snapshot functionality. The VM also offers ways to simulate lowend environments by limiting CPU cores and memory. But you could also just run docker on your host system.
-- The `Dockerfile` is structured in a way that the actual application code is in the last layer so rebuilding the image with only code changes utilizes the build cache for everything else and will take just 2-3 secs.
+- Then I have a VM on KVM hypervisor running standard Ubuntu Server LTS with docker installed. The VM keeps my projects separate and offers convenient snapshot functionality. The VM also offers ways to simulate lowend environments by limiting CPU cores and memory. You can use this [Ansible Docker Ubuntu](https://github.com/bbilly1/ansible-playbooks) playbook to get started quickly. But you could also just run docker on your host system.
+- The `Dockerfile` is structured in a way that the actual application code is in the last layer so rebuilding the image with only code changes utilizes the build cache for everything else and will just take a few seconds.
 - Take a look at the `deploy.sh` file. I have my local DNS resolve `tubearchivist.local` to the IP of the VM for convenience. To deploy the latest changes and rebuild the application to the testing VM run:
 ```bash
 ./deploy.sh test
 ```
-- The command above will also copy the file `tubarchivist/testing.sh` into the working folder of the container. Running this script will install additional debugging tools I regularly use in testing. 
+- The command above will call the docker build command with `--build-arg INSTALL_DEBUG=1` to install additional useful debug tools.
 - The `test` argument takes another optional argument to build for a specific architecture valid options are: `amd64`, `arm64` and `multi`, default is `amd64`.
 - This `deploy.sh` file is not meant to be universally usable for every possible environment but could serve as an idea on how to automatically rebuild containers to test changes - customize to your liking. 
 
 ## Working with Elasticsearch
 Additionally to the required services as listed in the example docker-compose file, the **Dev Tools** of [Kibana](https://www.elastic.co/guide/en/kibana/current/docker.html) are invaluable for running and testing Elasticsearch queries.  
 
-If you want to run queries in on the Elasticsearch container directly from your host with for example `curl` or something like *postman*, you might want to **publish** the port 9200 instead of just **exposing** it.
+If you want to run queries on the Elasticsearch container directly from your host with for example `curl` or something like *postman*, you might want to **publish** the port 9200 instead of just **exposing** it.
 
 ## Implementing a new feature
 
-Do you see anything on the roadmap that you would like to take a closer look at but you are not sure, what's the best way to tackle that? Or anything not on there yet you'd like to implement but are not sure how? Open up an issue and we try to find a solution together.
+Do you see anything on the roadmap that you would like to take a closer look at but you are not sure, what's the best way to tackle that? Or anything not on there yet you'd like to implement but are not sure how? Reach out on Discord and we'll look into it together.
 
 ## Making changes
 
@@ -44,14 +44,19 @@ To fix a bug or implement a feature, fork the repository and make all changes to
 
 ## Releases
 
-Everything on the master branch is what's in the latest release and is what you get in your container when you `pull` either the *:latest* tag or the newest named version. If you want to test the newest changes and improvements, clone the repository and build the docker container with the Dockerfile from the testing branch.
+There are three different docker tags:
+- **latest**: As the name implies is the latest multiarch release for regular usage.
+- **unstable**: Intermediate amd64 builds for quick testing and improved collaboration. Don't mix with a *latest* installation, for your testing environment only. This is untested and WIP and will have breaking changes between commits that might require a reset to resolve. 
+- **semantic versioning**: There will be a handful named version tags that will also have a matching release and tag on github.
+
+If you want to see what's in your container, checkout the matching release tag. A merge to **master** usually means a *latest* or *unstable* release. If you want to preview changes in your testing environment, pull the *unstable* tag or clone the repository and build the docker container with the Dockerfile from the **testing** branch.
 
 ## Code formatting and linting
 
-To keep things clean and consistent for everybody, there is a github action setup to lint and check the changes. You can test your code locally first if you want. For example if you made changes in the **download** module, run
+To keep things clean and consistent for everybody, there is a github action setup to lint and check the changes. You can test your code locally first if you want. For example if you made changes in the **video** module, run
 
 ```shell
-./deploy.sh validate tubearchivist/home/src/download.py
+./deploy.sh validate tubearchivist/home/src/index/video.py
 ```
 
-to validate your changes. If you omit the path, all the project files will get checked. This is subject to change as the codebase improves. 
+to validate your changes. If you omit the path, all the project files will get checked. This is subject to change as the codebase improves. 

Dockerfile

@@ -1,21 +1,42 @@
-# build the tube archivist image from default python slim image
+# multi stage to build tube archivist
+# first stage to build python wheel, copy into final image
 
-FROM python:3.9.7-slim-bullseye
+
+# First stage to build python wheel
+FROM python:3.10.4-slim-bullseye AS builder
 ARG TARGETPLATFORM
 
+RUN apt-get update
+RUN apt-get install -y --no-install-recommends build-essential gcc
+
+# install requirements
+COPY ./tubearchivist/requirements.txt /requirements.txt
+RUN pip install --user -r requirements.txt
+
+# build final image
+FROM python:3.10.4-slim-bullseye as tubearchivist
+
+ARG TARGETPLATFORM
+ARG INSTALL_DEBUG
+
 ENV PYTHONUNBUFFERED 1
 
+# copy build requirements
+COPY --from=builder /root/.local /root/.local
+ENV PATH=/root/.local/bin:$PATH
+
 # install distro packages needed
 RUN apt-get clean && apt-get -y update && apt-get -y install --no-install-recommends \
-    build-essential \
     nginx \
-    curl && rm -rf /var/lib/apt/lists/*
+    atomicparsley \
+    curl \
+    xz-utils && rm -rf /var/lib/apt/lists/*
 
 # get newest patched ffmpeg and ffprobe builds for amd64 fall back to repo ffmpeg for arm64
 RUN if [ "$TARGETPLATFORM" = "linux/amd64" ] ; then \
     curl -s https://api.github.com/repos/yt-dlp/FFmpeg-Builds/releases/latest \
         | grep browser_download_url \
-        | grep linux64-gpl-4.4.tar.xz \
+        | grep ".*master.*linux64.*tar.xz" \
         | cut -d '"' -f 4 \
         | xargs curl -L --output ffmpeg.tar.xz && \
         tar -xf ffmpeg.tar.xz --strip-components=2 --no-anchored -C /usr/bin/ "ffmpeg" && \
@@ -25,22 +46,25 @@ RUN if [ "$TARGETPLATFORM" = "linux/amd64" ] ; then \
         apt-get -y update && apt-get -y install --no-install-recommends ffmpeg && rm -rf /var/lib/apt/lists/* \
     ; fi
 
-# copy config files
-COPY nginx.conf /etc/nginx/conf.d/
+# install debug tools for testing environment
+RUN if [ "$INSTALL_DEBUG" ] ; then \
+        apt-get -y update && apt-get -y install --no-install-recommends \
+        vim htop bmon net-tools iputils-ping procps \
+        && pip install --user ipython \
+    ; fi
 
 # make folders
 RUN mkdir /cache
 RUN mkdir /youtube
 RUN mkdir /app
 
-# install python dependencies
-COPY ./tubearchivist/requirements.txt /requirements.txt
-RUN pip install --no-cache-dir -r requirements.txt --src /usr/local/src
+# copy config files
+COPY docker_assets/nginx.conf /etc/nginx/sites-available/default
 
 # copy application into container
 COPY ./tubearchivist /app
-COPY ./run.sh /app
-COPY ./uwsgi.ini /app
+COPY ./docker_assets/run.sh /app
+COPY ./docker_assets/uwsgi.ini /app
 
 # volumes
 VOLUME /cache

README.md

@@ -2,11 +2,15 @@
 
 <center><h1>Your self hosted YouTube media server</h1></center>
 
+Tube Archivist has a new home: https://github.com/tubearchivist/tubearchivist
+
 ## Table of contents:
-* [Wiki](https://github.com/bbilly1/tubearchivist/wiki) for a detailed documentation
+* [Wiki](https://github.com/tubearchivist/tubearchivist/wiki) with [FAQ](https://github.com/tubearchivist/tubearchivist/wiki/FAQ)
 * [Core functionality](#core-functionality)
 * [Screenshots](#screenshots)
 * [Problem Tube Archivist tries to solve](#problem-tube-archivist-tries-to-solve)
+* [Connect](#connect)
+* [Extended Universe](#extended-universe)
 * [Installing and updating](#installing-and-updating)
 * [Getting Started](#getting-started)
 * [Potential pitfalls](#potential-pitfalls)
@@ -23,6 +27,9 @@
 * Play videos
 * Keep track of viewed and unviewed videos
 
+## Tube Archivist on YouTube
+[![ibracorp-youtube-video-thumb](assets/tube-archivist-ibracorp-O8H8Z01c0Ys.jpg)](https://www.youtube.com/watch?v=O8H8Z01c0Ys)
+
 ## Screenshots
 ![home screenshot](assets/tube-archivist-screenshot-home.png?raw=true "Tube Archivist Home")  
 *Home Page*
@@ -42,8 +49,18 @@
 ## Problem Tube Archivist tries to solve
 Once your YouTube video collection grows, it becomes hard to search and find a specific video. That's where Tube Archivist comes in: By indexing your video collection with metadata from YouTube, you can organize, search and enjoy your archived YouTube videos without hassle offline through a convenient web interface.
 
+## Connect
+- [Discord](https://discord.gg/AFwz8nE7BK): Connect with us on our Discord server.
+- [r/TubeArchivist](https://www.reddit.com/r/TubeArchivist/): Join our Subreddit.
+
+## Extended Universe
+- [Browser Extension](https://github.com/tubearchivist/browser-extension) Tube Archivist Companion, for [Firefox](https://addons.mozilla.org/addon/tubearchivist-companion/) and [Chrome](https://chrome.google.com/webstore/detail/tubearchivist-companion/jjnkmicfnfojkkgobdfeieblocadmcie)
+- [Tube Archivist Metrics](https://github.com/tubearchivist/tubearchivist-metrics) to create statistics in Prometheus/OpenMetrics format.
+
 ## Installing and updating
-Take a look at the example `docker-compose.yml` file provided. Tube Archivist depends on three main components split up into separate docker containers:  
+Take a look at the example `docker-compose.yml` file provided. Use the *latest* or the named semantic version tag. The *unstable* tag is for intermediate testing and as the name implies, is **unstable** and not be used on your main installation but in a [testing environment](CONTRIBUTING.md).  
+
+Tube Archivist depends on three main components split up into separate docker containers:  
 
 ### Tube Archivist
 The main Python application that displays and serves your video collection, built with Django.
@@ -54,8 +71,22 @@ The main Python application that displays and serves your video collection, buil
   - The environment variables `HOST_UID` and `HOST_GID` allows Tube Archivist to `chown` the video files to the main host system user instead of the container user. Those two variables are optional, not setting them will disable that functionality. That might be needed if the underlying filesystem doesn't support `chown` like *NFS*. 
   - Change the environment variables `TA_USERNAME` and `TA_PASSWORD` to create the initial credentials. 
   - `ELASTIC_PASSWORD` is for the password for Elasticsearch. The environment variable `ELASTIC_USER` is optional, should you want to change the username from the default *elastic*.
+  - For the scheduler to know what time it is, set your timezone with the `TZ` environment variable, defaults to *UTC*.
+
+### Port collisions
+If you have a collision on port `8000`, best solution is to use dockers *HOST_PORT* and *CONTAINER_PORT* distinction: To for example change the interface to port 9000 use `9000:8000` in your docker-compose file.  
+
+Should that not be an option, the Tube Archivist container takes these two additional environment variables:
+- **TA_PORT**: To actually change the port where nginx listens, make sure to also change the ports value in your docker-compose file.
+- **TA_UWSGI_PORT**: To change the default uwsgi port 8080 used for container internal networking between uwsgi serving the django application and nginx.  
+
+Changing any of these two environment variables will change the files *nginx.conf* and *uwsgi.ini* at startup using `sed` in your container.
 
 ### Elasticsearch
+**Note**: Newest Tube Archivist depends on Elasticsearch version 7.17 to provide an automatic updatepath in the future. 
+
+Use `bbilly1/tubearchivist-es` to automatically get the recommended version, or use the official image with the version tag in the docker-compose file.
+
 Stores video meta data and makes everything searchable. Also keeps track of the download queue.
   - Needs to be accessible over the default port `9200`
   - Needs a volume at **/usr/share/elasticsearch/data** to store data
@@ -76,13 +107,13 @@ For some architectures it might be required to run Redis JSON on a nonstandard p
 ### Updating Tube Archivist
 You will see the current version number of **Tube Archivist** in the footer of the interface so you can compare it with the latest release to make sure you are running the *latest and greatest*.  
 * There can be breaking changes between updates, particularly as the application grows, new environment variables or settings might be required for you to set in the your docker-compose file. *Always* check the **release notes**: Any breaking changes will be marked there.  
-* All testing and development is done with the Elasticsearch version number as mentioned in the provided *docker-compose.yml* file. This will be updated when a new release of Elasticsearch is available. Running an older version of Elasticsearch is most likely not going to result in any issues, but it's still recommended to run the same version as mentioned.
+* All testing and development is done with the Elasticsearch version number as mentioned in the provided *docker-compose.yml* file. This will be updated when a new release of Elasticsearch is available. Running an older version of Elasticsearch is most likely not going to result in any issues, but it's still recommended to run the same version as mentioned. Use `bbilly1/tubearchivist-es` to automatically get the recommended version.
 
 ### Alternative installation instructions:
-- **arm64**: Newest Tube Archivist container is multi arch, so is Elasticsearch. RedisJSON doesn't offer arm builds, you can use `bbilly1/rejson`, an unofficial rebuild for arm64.
-  - NOTE: This is untested, looking for feedback.
-- **Synology**: There is a [discussion thread](https://github.com/bbilly1/tubearchivist/discussions/48) with Synology installation instructions.
+- **arm64**: The Tube Archivist container is multi arch, so is Elasticsearch. RedisJSON doesn't offer arm builds, you can use `bbilly1/rejson`, an unofficial rebuild for arm64.
+- **Synology**: There is a [discussion thread](https://github.com/tubearchivist/tubearchivist/discussions/48) with Synology installation instructions.
 - **Unraid**: The three containers needed are all in the Community Applications. First install `TubeArchivist RedisJSON` followed by `TubeArchivist ES`, and finally you can install `TubeArchivist`. If you have unraid specific issues, report those to the [support thread](https://forums.unraid.net/topic/114073-support-crocs-tube-archivist/ "support thread").
+- **Helm Chart**: There is a Helm Chart available at https://github.com/insuusvenerati/helm-charts. Mostly self-explanatory but feel free to ask questions in the discord / subreddit.
 
 
 ## Potential pitfalls
@@ -107,8 +138,13 @@ chown 1000:0 /path/to/mount/point
 ```
 This will match the permissions with the **UID** and **GID** of elasticsearch within the container and should fix the issue.
 
+### Disk usage
+The Elasticsearch index will turn to *read only* if the disk usage of the container goes above 95% until the usage drops below 90% again, you will see error messages like `disk usage exceeded flood-stage watermark`, [link](https://github.com/tubearchivist/tubearchivist#disk-usage).  
+
+Similar to that, TubeArchivist will become all sorts of messed up when running out of disk space. There are some error messages in the logs when that happens, but it's best to make sure to have enough disk space before starting to download.
+
 ## Getting Started
-1. Go through the **settings** page and look at the available options. Particularly set *Download Format* to your desired video quality before downloading. **Tube Archivist** downloads the best available quality by default. To support iOS or MacOS a compatible format must be specified. For example:
+1. Go through the **settings** page and look at the available options. Particularly set *Download Format* to your desired video quality before downloading. **Tube Archivist** downloads the best available quality by default. To support iOS or MacOS and some other browsers a compatible format must be specified. For example:
 ```
 bestvideo[VCODEC=avc1]+bestaudio[ACODEC=mp4a]/mp4
 ```
@@ -118,33 +154,46 @@ bestvideo[VCODEC=avc1]+bestaudio[ACODEC=mp4a]/mp4
 5. Enjoy your archived collection!
   
 ## Roadmap
-This should be considered as a **minimal viable product**, there is an extensive list of future functions and improvements planned.
+We have come far, nonetheless we are not short of ideas on how to improve and extend this project. Issues waiting for you to be tackled in no particular order:
 
-### Functionality
 - [ ] User roles
-- [ ] Create playlists
 - [ ] Podcast mode to serve channel as mp3
 - [ ] Implement [PyFilesystem](https://github.com/PyFilesystem/pyfilesystem2) for flexible video storage
-- [ ] Optional automatic deletion of watched items after a specified time
-- [ ] Subtitle download & indexing
+- [ ] Implement [Apprise](https://github.com/caronc/apprise) for notifications ([#97](https://github.com/tubearchivist/tubearchivist/issues/97))
+- [ ] User created playlists, random and repeat controls ([#108](https://github.com/tubearchivist/tubearchivist/issues/108), [#220](https://github.com/tubearchivist/tubearchivist/issues/220))
+- [ ] Auto play or play next link ([#226](https://github.com/tubearchivist/tubearchivist/issues/226))
+- [ ] Show similar videos on video page
+- [ ] Multi language support
+- [ ] Show total video downloaded vs total videos available in channel
+- [ ] Make items in grid row configurable to use more of the screen
+- [ ] Add statistics of index
+- [ ] Implement complete offline media file import from json file ([#138](https://github.com/tubearchivist/tubearchivist/issues/138))
+- [ ] Filter and query in search form, search by url query ([#134](https://github.com/tubearchivist/tubearchivist/issues/134), [#139](https://github.com/tubearchivist/tubearchivist/issues/139))
+- [ ] Auto ignore videos by keyword ([#163](https://github.com/tubearchivist/tubearchivist/issues/163))
+- [ ] Custom searchable notes to videos, channels, playlists ([#144](https://github.com/tubearchivist/tubearchivist/issues/144))
+- [ ] Download video comments
+
+Implemented:
+- [X] Add passing browser cookies to yt-dlp [2022-05-08]
+- [X] Add [SponsorBlock](https://sponsor.ajay.app/) integration [2022-04-16]
+- [X] Implement per channel settings [2022-03-26]
+- [X] Subtitle download & indexing [2022-02-13]
+- [X] Fancy advanced unified search interface [2022-01-08]
+- [X] Auto rescan and auto download on a schedule [2021-12-17]
+- [X] Optional automatic deletion of watched items after a specified time [2021-12-17]
+- [X] Create playlists [2021-11-27]
 - [X] Access control [2021-11-01]
 - [X] Delete videos and channel [2021-10-16]
 - [X] Add thumbnail embed option [2021-10-16]
+- [X] Create a github wiki for user documentation [2021-10-03]
+- [X] Grid and list view for both channel and video list pages [2021-10-03]
 - [X] Un-ignore videos [2021-10-03]
 - [X] Dynamic download queue [2021-09-26]
 - [X] Backup and restore [2021-09-22]
 - [X] Scan your file system to index already downloaded videos [2021-09-14]
 
-### UI
-- [ ] Show similar videos on video page
-- [ ] Multi language support
-- [ ] Show total video downloaded vs total videos available in channel
-- [X] Grid and list view for both channel and video list pages [2021-10-03]
-- [X] Create a github wiki for user documentation [2021-10-03]
-
-
 ## Known limitations
-- Video files created by Tube Archivist need to be **mp4** video files for best browser compatibility.
+- Video files created by Tube Archivist need to be playable in your browser of choice. Not every codec is compatible with every browser and might require some testing with format selection. 
 - Every limitation of **yt-dlp** will also be present in Tube Archivist. If **yt-dlp** can't download or extract a video for any reason, Tube Archivist won't be able to either.
 - For now this is meant to be run in a trusted network environment. Not everything is properly authenticated.
 - There is currently no flexibility in naming of the media files.
@@ -155,4 +204,4 @@ The best donation to **Tube Archivist** is your time, take a look at the [contri
 Second best way to support the development is to provide for caffeinated beverages:
 * [Paypal.me](https://paypal.me/bbilly1) for a one time coffee
 * [Paypal Subscription](https://www.paypal.com/webapps/billing/plans/subscribe?plan_id=P-03770005GR991451KMFGVPMQ) for a monthly coffee
-* [co-fi.com](https://ko-fi.com/bbilly1) for an alternative platform
+* [ko-fi.com](https://ko-fi.com/bbilly1) for an alternative platform

deploy.sh

@@ -3,7 +3,8 @@
 # deploy all needed project files to different servers:
 # test for local vm for testing
 # blackhole for local production
-# docker to publish
+# unstable to publish intermediate releases
+# docker to publish regular release
 
 # create builder:
 # docker buildx create --name tubearchivist
@@ -11,7 +12,7 @@
 # docker buildx inspect --bootstrap
 
 # more details:
-# https://github.com/bbilly1/tubearchivist/issues/6
+# https://github.com/tubearchivist/tubearchivist/issues/6
 
 set -e
 
@@ -32,7 +33,7 @@ function sync_blackhole {
         . -e ssh "$host":tubearchivist
 
     echo "$PASS" | ssh "$host" 'sudo -S docker buildx build --platform linux/amd64 -t bbilly1/tubearchivist:latest tubearchivist --load 2>/dev/null'
-    echo "$PASS" | ssh "$host" 'sudo -S docker-compose up -d 2>/dev/null'
+    echo "$PASS" | ssh "$host" 'sudo -S docker compose up -d 2>/dev/null'
 
 }
 
@@ -42,7 +43,10 @@ function sync_test {
     # pass argument to build for specific platform
 
     host="tubearchivist.local"
+    # make base folder
+    ssh "$host" "mkdir -p docker"
 
+    # copy project files to build image
     rsync -a --progress --delete-after \
         --exclude ".git" \
         --exclude ".gitignore" \
@@ -51,7 +55,8 @@ function sync_test {
         --exclude "db.sqlite3" \
         . -e ssh "$host":tubearchivist
 
-    rsync -r --progress --delete docker-compose.yml -e ssh "$host":docker
+    # copy default docker-compose file if not exist
+    rsync --progress --ignore-existing docker-compose.yml -e ssh "$host":docker
 
     if [[ $1 = "amd64" ]]; then
         platform="linux/amd64"
@@ -63,11 +68,8 @@ function sync_test {
         platform="linux/amd64"
     fi
 
-    ssh "$host" "docker buildx build --platform $platform -t bbilly1/tubearchivist:latest tubearchivist --load"
-    ssh "$host" 'docker-compose -f docker/docker-compose.yml up -d'
-
-    ssh "$host" 'docker cp tubearchivist/tubearchivist/testing.sh tubearchivist:/app/testing.sh'
-    ssh "$host" 'docker exec tubearchivist chmod +x /app/testing.sh'
+    ssh "$host" "docker buildx build --build-arg INSTALL_DEBUG=1 --platform $platform -t bbilly1/tubearchivist:latest tubearchivist --load"
+    ssh "$host" 'docker compose -f docker/docker-compose.yml up -d'
 
 }
 
@@ -83,15 +85,13 @@ function validate {
     fi
 
     echo "run validate on $check_path"
-    
-    echo "running bandit"
-    bandit --recursive --skip B105,B108,B404,B603,B607 "$check_path"
+
     echo "running black"
     black --diff --color --check -l 79 "$check_path"
     echo "running codespell"
     codespell --skip="./.git" "$check_path"
     echo "running flake8"
-    flake8 "$check_path" --count --max-complexity=12 --max-line-length=79 \
+    flake8 "$check_path" --count --max-complexity=10 --max-line-length=79 \
         --show-source --statistics
     echo "running isort"
     isort --check-only --diff --profile black -l 79 "$check_path"
@@ -100,6 +100,50 @@ function validate {
 }
 
 
+# update latest tag compatible es for set and forget
+function sync_latest_es {
+
+    VERSION=$(grep "bbilly1/tubearchivist-es" docker-compose.yml | awk '{print $NF}')
+    printf "\nsync new ES version %s\nContinue?\n" "$VERSION"
+    read -rn 1
+
+    if [[ $(systemctl is-active docker) != 'active' ]]; then
+        echo "starting docker"
+        sudo systemctl start docker
+    fi
+
+    sudo docker image pull docker.elastic.co/elasticsearch/elasticsearch:"$VERSION"
+
+    sudo docker tag \
+        docker.elastic.co/elasticsearch/elasticsearch:"$VERSION" \
+        bbilly1/tubearchivist-es
+
+    sudo docker tag \
+        docker.elastic.co/elasticsearch/elasticsearch:"$VERSION" \
+        bbilly1/tubearchivist-es:"$VERSION"
+
+    sudo docker push bbilly1/tubearchivist-es
+    sudo docker push bbilly1/tubearchivist-es:"$VERSION"
+
+}
+
+
+# publish unstable tag to docker
+function sync_unstable {
+
+    if [[ $(systemctl is-active docker) != 'active' ]]; then
+        echo "starting docker"
+        sudo systemctl start docker
+    fi
+
+    # start amd64 build
+    sudo docker buildx build \
+        --platform linux/amd64 \
+        -t bbilly1/tubearchivist:unstable --push .
+
+}
+
+
 function sync_docker {
 
     # check things
@@ -114,10 +158,7 @@ function sync_docker {
     fi
 
     echo "latest tags:"
-    git tag
-
-    echo "latest docker images:"
-    sudo docker image ls bbilly1/tubearchivist
+    git tag | tail -n 10
 
     printf "\ncreate new version:\n"
     read -r VERSION
@@ -140,20 +181,23 @@ function sync_docker {
 }
 
 
-# check package versions in requirements.txt for updates
-python version_check.py
-
-
 if [[ $1 == "blackhole" ]]; then
     sync_blackhole
 elif [[ $1 == "test" ]]; then
     sync_test "$2"
 elif [[ $1 == "validate" ]]; then
+    # check package versions in requirements.txt for updates
+    python version_check.py
     validate "$2"
 elif [[ $1 == "docker" ]]; then
     sync_docker
+    sync_unstable
+elif [[ $1 == "unstable" ]]; then
+    sync_unstable
+elif [[ $1 == "es" ]]; then
+    sync_latest_es
 else
-    echo "valid options are: blackhole | test | validate | docker"
+    echo "valid options are: blackhole | test | validate | docker | unstable | es"
 fi
 
 

docker-compose.yml

@@ -8,36 +8,37 @@ services:
     ports:
       - 8000:8000
     volumes:
-      - ./volumes/tubearchivist/media:/youtube
-      - ./volumes/tubearchivist/cache:/cache
+      - media:/youtube
+      - cache:/cache
     environment:
-      - ES_URL=http://archivist-es:9200
-      - REDIS_HOST=archivist-redis
+      - ES_URL=http://archivist-es:9200     # needs protocol e.g. http and port
+      - REDIS_HOST=archivist-redis          # don't add protocol
       - HOST_UID=1000
       - HOST_GID=1000
-      - TA_USERNAME=tubearchivist
-      - TA_PASSWORD=verysecret
-      - ELASTIC_PASSWORD=verysecret
+      - TA_USERNAME=tubearchivist           # your initial TA credentials
+      - TA_PASSWORD=verysecret              # your initial TA credentials
+      - ELASTIC_PASSWORD=verysecret         # set password for Elasticsearch
+      - TZ=America/New_York                 # set your time zone
     depends_on:
       - archivist-es
       - archivist-redis
   archivist-redis:
-    image: redislabs/rejson:latest
+    image: redislabs/rejson:latest          # for arm64 use bbilly1/rejson
     container_name: archivist-redis
     restart: always
     expose:
       - "6379"
     volumes:
-      - ./volumes/tubearchivist/redis:/data
+      - redis:/data
     depends_on:
       - archivist-es
   archivist-es:
-    image: docker.elastic.co/elasticsearch/elasticsearch:7.15.1
+    image: bbilly1/tubearchivist-es         # only for amd64, or use official es 7.17.3
     container_name: archivist-es
     restart: always
     environment:
       - "xpack.security.enabled=true"
-      - "ELASTIC_PASSWORD=verysecret"
+      - "ELASTIC_PASSWORD=verysecret"       # matching Elasticsearch password
       - "discovery.type=single-node"
       - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
     ulimits:
@@ -45,6 +46,12 @@ services:
         soft: -1
         hard: -1
     volumes:
-      - ./volumes/tubearchivist/es:/usr/share/elasticsearch/data
+      - es:/usr/share/elasticsearch/data    # check for permission error when using bind mount, see readme
     expose:
       - "9200"
+
+volumes:
+  media:
+  cache:
+  redis:
+  es:

docker_assets/nginx.conf

@@ -0,0 +1,29 @@
+server {
+
+    listen 8000;
+    
+    location /cache/videos/ {
+        alias /cache/videos/;
+    }
+    
+    location /cache/channels/ {
+        alias /cache/channels/;
+    }
+    
+    location /cache/playlists/ {
+        alias /cache/playlists/;
+    }
+    
+    location /media/ {
+        alias /youtube/;
+        types {
+            text/vtt vtt;
+        }
+    }
+    
+    location / {
+        include uwsgi_params;
+        uwsgi_pass localhost:8080;
+    }
+    
+}

docker_assets/run.sh

@@ -1,7 +1,6 @@
 #!/bin/bash
 # startup script inside the container for tubearchivist
 
-# check environment
 if [[ -z "$ELASTIC_USER" ]]; then
     export ELASTIC_USER=elastic
 fi
@@ -14,6 +13,16 @@ for each in "${ENV_VARS[@]}"; do
     fi
 done
 
+# ugly nginx and uwsgi port overwrite with env vars
+if [[ -n "$TA_PORT" ]]; then
+    sed -i "s/8000/$TA_PORT/g" /etc/nginx/sites-available/default
+fi
+
+if [[ -n "$TA_UWSGI_PORT" ]]; then
+    sed -i "s/8080/$TA_UWSGI_PORT/g" /etc/nginx/sites-available/default
+    sed -i "s/8080/$TA_UWSGI_PORT/g" /app/uwsgi.ini
+fi
+
 # wait for elasticsearch
 counter=0
 until curl -u "$ELASTIC_USER":"$ELASTIC_PASSWORD" "$ES_URL" -fs; do
@@ -36,4 +45,6 @@ export DJANGO_SUPERUSER_PASSWORD=$TA_PASSWORD && \
 python manage.py collectstatic --noinput -c
 nginx &
 celery -A home.tasks worker --loglevel=INFO &
+celery -A home beat --loglevel=INFO \
+    -s "${BEAT_SCHEDULE_PATH:-/cache/celerybeat-schedule}" &
 uwsgi --ini uwsgi.ini

docs/Channels.md

@@ -1,11 +1,11 @@
 # Channels Overview and Channel Detail Page
 
-The channels are organized on two different levels:
+The channels are organized on two different levels, similar as the [playlists](Playlists):
 
 ## Channels Overview
 Accessible at `/channel/` of your Tube Archivist, the **Overview Page** shows a list of all channels you have indexed. 
-- You can filter that list to show or hide subscribed channels from the drop down menu. Clicking on the channel banner or the channel name will direct you to the *Channel Detail Page*.
-- If you are subscribed to a channel a *Unsubscribe* button will show.
+- You can filter that list to show or hide subscribed channels with the toggle. Clicking on the channel banner or the channel name will direct you to the *Channel Detail Page*.
+- If you are subscribed to a channel a *Unsubscribe* button will show, if you aren't subscribed, a *Subscribe* button will show instead. 
 
 The **Subscribe to Channels** button <img src="assets/icon-add.png?raw=true" alt="add icon" width="20px" style="margin:0 5px;"> opens a text field to subscribe to a channel. You have a few options:
 - Enter the YouTube channel ID, a 25 character alphanumeric string. For example *UCBa659QWEk1AI4Tg--mrJ2A*
@@ -14,11 +14,19 @@ The **Subscribe to Channels** button <img src="assets/icon-add.png?raw=true" alt
 - Enter the video URL for any video and let Tube Archivist extract the channel ID for you. For example *https://www.youtube.com/watch?v=2tdiKTSdE9Y*
 - Add one per line.
 
-The search icon <img src="assets/icon-search.png?raw=true" alt="search icon" width="20px" style="margin:0 5px;"> opens a text box to search for indexed channel names. Possible matches will show as you type. 
+You can search your indexed channels by clicking on the search icon <img src="assets/icon-search.png?raw=true" alt="search icon" width="20px" style="margin:0 5px;">. This will open a dedicated page.
 
 ## Channel Detail
 Each channel will get a dedicated channel detail page accessible at `/channel/<channel-id>/` of your Tube Archivist. This page shows all the videos you have downloaded from this channel plus additional metadata. 
-- If you are subscribed to the channel, an *Unsubscribe* button will show.
+- If you are subscribed to the channel, an *Unsubscribe* button will show, else the *Subscribe* button will show.
 - You can *Show* the channel description, that matches with the *About* tab on YouTube.
 - The **Mark as Watched** button will mark all videos of this channel as watched.
-- The button **Delete Channel** will delete the channel plus all videos of this channel, both media files and metadata.
+- The button **Delete Channel** will delete the channel plus all videos of this channel, both media files and metadata additionally this will also delete playlists metadata belonging to that channel.
+- The button **Show Playlists** will go to the [playlists](Playlists) page and filter the list to only show playlists from this channel.
+
+### Channel Customize
+Clicking on the *Configure* button will open a form with options to configure settings on a per channel basis. Any configurations here will overwrite your settings from the [settings](Settings) page.
+- **Download Format**: Overwrite the download qualities for videos from this channel.
+- **Auto Delete**: Automatically delete watched videos from this channel after selected days.
+- **Index Playlists**: Automatically add all Playlists with at least a video downloaded to your index. Only do this for channels where you care about playlists as this will slow down indexing new videos for having to check which playlist this belongs to.
+- **SponsorBlock**: Using [SponsorBlock](https://sponsor.ajay.app/) to get and skip sponsored content. Customize per channel: You can *disable* or *enable* SponsorBlock for certain channels only to overwrite the behavior set on the [Settings](settings) page. Selecting *unset* will remove the overwrite and your setting will fall back to the default on the settings page.

docs/Downloads.md

@@ -3,15 +3,17 @@ Accessible at `/downloads/` of your Tube Archivist, this page handles all the do
 
 
 ## Rescan Subscriptions
-The **Rescan Subscriptions** icon <img src="assets/icon-rescan.png?raw=true" alt="rescan icon" width="20px" style="margin:0 5px;"> will start a background task to look for new videos from the channels you are subscribed to. You can define the channel page size on the [settings page](Settings#subscriptions). With the default channel page size, expect this process to take around 2-3 seconds for each channel you are subscribed to. A status message will show the progress.
+The **Rescan Subscriptions** icon <img src="assets/icon-rescan.png?raw=true" alt="rescan icon" width="20px" style="margin:0 5px;"> will start a background task to look for new videos from the channels and playlists you are subscribed to. You can define the channel and playlist page size on the [settings page](Settings#subscriptions). With the default page size, expect this process to take around 2-3 seconds for each channel or playlist you are subscribed to. A status message will show the progress.
 
-Then for every video found, **Tube Archivist** will skip the video if it has already been downloaded or if you added it to the *ignored* list before. All the other videos will get added to the download queue. Expect this to take around 1 second for each video as **Tube Archivist** needs to grab some additional metadata. New videos will get added at the bottom of the download queue.
+Then for every video found, **Tube Archivist** will skip the video if it has already been downloaded or if you added it to the *ignored* list before. All the other videos will get added to the download queue. Expect this to take around 2 seconds for each video as **Tube Archivist** needs to grab some additional metadata. New videos will get added at the bottom of the download queue.
 
 ## Download Queue
 The **Start Download** icon <img src="assets/icon-download.png?raw=true" alt="download icon" width="20px" style="margin:0 5px;"> will start the download process starting from the top of the queue. Take a look at the relevant settings on the [Settings Page](Settings#downloads). Once the process started, a progress message will show with additional details and controls: 
 - The stop icon <img src="assets/icon-stop.png?raw=true" alt="stop icon" width="20px" style="margin:0 5px;"> will gracefully stop the download process, once the current video has been finished successfully.
 - The cancel icon <img src="assets/icon-close-red.png?raw=true" alt="close icon" width="20px" style="margin:0 5px;"> is equivalent to killing the process and will stop the download immediately. Any leftover files will get deleted, the canceled video will still be available in the download queue.
 
+After downloading, Tube Archivist tries to add new videos to already indexed playlists.
+
 ## Add to Download Queue
 The **Add to Download Queue** icon <img src="assets/icon-add.png?raw=true" alt="add icon" width="20px" style="margin:0 5px;"> opens a text field to manually add videos to the download queue. You have a few options:
 - Add a link to a YouTube video. For example *https://www.youtube.com/watch?v=2tdiKTSdE9Y*.
@@ -19,7 +21,9 @@ The **Add to Download Queue** icon <img src="assets/icon-add.png?raw=true" alt="
 - Add a link to a YouTube video by providing the shortened URL, for example *https://youtu.be/2tdiKTSdE9Y*.
 - Add a Channel ID or Channel URL to add every available video to the download queue. This will ignore the channel page size as described before and is meant for an initial download of the whole channel. You can still ignore selected videos before starting the download.
 - Add a channel name like for example *https://www.youtube.com/c/TomScottGo*.
-- Add a playlist ID or URL to add every available video in the list to the download queue, for example *https://www.youtube.com/playlist?list=PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha* or *PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha*. Note that when you add a link to a video in a playlist, Tube Archivist assumes you want to download only the specific video and not the whole playlist, for example *https://www.youtube.com/watch?v=CINVwWHlzTY&list=PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha* will only add one video *CINVwWHlzTY* to the queue.
+- Add a playlist ID or URL to add every available video in the list to the download queue, for example *https://www.youtube.com/playlist?list=PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha* or *PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha*. 
+  - Note: When adding a playlist to the queue, this playlist will automatically get [indexed](Playlists#playlist-detail).
+  - Note: When you add a link to a video in a playlist, Tube Archivist assumes you want to download only the specific video and not the whole playlist, for example *https://www.youtube.com/watch?v=CINVwWHlzTY&list=PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha* will only add one video *CINVwWHlzTY* to the queue.
 - Add one link per line.
 
 ## The Download Queue
@@ -27,9 +31,13 @@ Below the three buttons you find the download queue. New items will get added at
 
 Every video in the download queue has two buttons:
 - **Ignore**: This will remove that video from the download queue and this video will not get added again, even when you **Rescan Subscriptions**.
-- **Download now**: This will give priority to this video. If the download process is already running, the prioritized video will get downloaded as soon as the current video is finished. If there is no download process running, this will start downloading this single video and stop after that.
+- **Download now**: This will give priority to this video. If the download process is already running, the prioritized video will get downloaded as soon as the current video is finished. If there is no download process running, this will start downloading this single video and stop after that.  
+
+The button **Delete all queued** will delete all pending videos from the download queue.
 
 You can flip the view by activating **Show Only Ignored Videos**. This will show all videos you have previously *ignored*.  
 Every video in the ignored list has two buttons:
 - **Forget**: This will delete the item form the ignored list.
-- **Add to Queue**: This will add the ignored video back to the download queue.
+- **Add to Queue**: This will add the ignored video back to the download queue.  
+
+The button **Delete all ignored** will delete all videos you have previously ignored.

docs/FAQ.md

@@ -0,0 +1,34 @@
+# Frequently Asked Questions
+
+## 1. Scope of this project
+Tube Archivist is *Your self hosted YouTube media server*, which also defines the primary scope of what this project tries to do:
+- **Self hosted**: This assumes you have full control over the underlying operating system and hardware and can configure things to work properly with Docker, it's volumes and networks as well as whatever disk storage and filesystem you choose to use.
+- **YouTube**: Downloading, indexing and playing videos from YouTube, there are currently no plans to expand this to any additional platforms.
+- **Media server**: This project tries to be a stand alone media server in it's own web interface.
+
+Additionally to that, progress is also happening on:
+- **API**: Endpoints for additional integrations.
+- **Browser Extension**: To integrate between youtube.com and Tube Archivist.
+
+Defining the scope is important for the success of any project:
+- A scope too broad will result in development effort spreading too thin and will run into danger that his project tries to do too many things and none of them well.
+- A too narrow scope will make this project uninteresting and will exclude audiences that could also benefit from this project.
+- Not defining a scope will easily lead to misunderstandings and false hopes of where this project tries to go.
+
+Of course this is subject to change, as this project continues to grow and more people contribute.
+
+## 2. Emby/Plex/Jellyfin/Kodi integrations
+Although there are similarities between these excellent projects and Tube Archivist, they have a very different use case. Trying to fit the metadata relations and database structure of a YouTube archival project into these media servers that specialize in Movies and TV shows is always going to be limiting.
+
+Part of the scope is to be its own media server, so that's where the focus and effort of this project is. That being said, the nature of self hosted and open source software gives you all the possible freedom to use your media as you wish.
+
+## 3. To Docker or not to Docker
+This project is a classical docker application: There are multiple moving parts that need to be able to interact with each other and need to be compatible with multiple architectures and operating systems. Additionally Docker also drastically reduces development complexity which is highly appreciated.  
+
+So Docker is the only supported installation method. If you don't have any experience with Docker, consider investing the time to learn this very useful technology.
+
+## 4. Finetuning Elasticsearch
+A minimal configuration of Elasticsearch (ES) is provided in the example docker-compose.yml file. ES is highly configurable and very interesting to learn more about. Refer to the [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html) if you want to get into it.
+
+## 5. Advanced Authentication
+If you like to use things like SSO, LDAP or 2FA to login, consider using something like Authelia as a reverse proxy so this project can focus on the core task. Tube Archivist has a *remember me* checkbox at login to extend your sessions lifetime in your browser. 

docs/Home.md

@@ -2,11 +2,14 @@
 Welcome to the official Tube Archivist Wiki. This is an up-to-date documentation of user functionality.
 
 Table of contents:
-* [Main](Main): Tube Archivist landing page
-* [Channels](Channels): Browse your channels, handle subscriptions
+* [FAQ](FAQ): Frequently asked questions what this project is and tries to do
+* [Channels](Channels): Browse your channels, handle channel subscriptions
+* [Playlists](Playlists): Browse your indexed playlists, handle playlist subscriptions
 * [Downloads](Downloads): Scanning subscriptions, handle download queue
 * [Settings](Settings): All the configuration options
+* [Video](Video): All details of a single video and playlist navigation.
 * [Users](Users): User management admin interface
+* [Installation](Installation): WIP - detailed installation instructions for various platforms.
 
 ## Getting Started
 1. [Subscribe](Channels#channels-overview) to some of your favourite YouTube channels.
@@ -19,9 +22,10 @@ Table of contents:
 * Clicking on the channel name or the channel icon brings you to the dedicated channel page to show videos from that channel.
 * Clicking on a video title brings you to the dedicated video page and shows additional details.
 * Clicking on a video thumbnail opens the video player and starts streaming the selected video.
-* Hover over the playing video to show additional control options.
+* Clicking on the search icon <img src="assets/icon-search.png?raw=true" alt="gridview icon" width="20px" style="margin:0 5px;"> will open a dedicated search page to search over your complete index.
+* The pagination - if available - builds links for up to 10'000 results, use the search, sort or filter functionality to find what you are looking for.
 
 
 An empty checkbox icon <img src="assets/icon-unseen.png?raw=true" alt="unseen icon" width="20px" style="margin:0 5px;"> will show for videos you haven't marked as watched. Click on it and the icon will change to a filled checkbox <img src="assets/icon-seen.png?raw=true" alt="seen icon" width="20px" style="margin:0 5px;"> indicating it as watched - click again to revert.
 
-When available the <img src="assets/icon-gridview.png?raw=true" alt="gridview icon" width="20px" style="margin:0 5px;"> gridview icon will display the list in a grid, the <img src="assets/icon-listview.png?raw=true" alt="listview icon" width="20px" style="margin:0 5px;"> listview icon will arrange the items in a list.
+When available the <img src="assets/icon-gridview.png?raw=true" alt="gridview icon" width="20px" style="margin:0 5px;"> gridview icon will display the list in a grid, the <img src="assets/icon-listview.png?raw=true" alt="listview icon" width="20px" style="margin:0 5px;"> listview icon will arrange the items in a list. The sort icon <img src="assets/icon-sort.png?raw=true" alt="listview icon" width="20px" style="margin:0 5px;"> will open additional sort options.

docs/Installation.md

@@ -0,0 +1,56 @@
+# Detailed Installation Instructions for Various Platforms
+
+## Unraid
+
+Tube Archivist, and all if it's dependencies are located in the [community applications](https://forums.unraid.net/topic/38582-plug-in-community-applications/) store. The three containers you will need are as follows:
+
+-   **TubeArchivist-RedisJSON**: This container acts as a cache and temporary link between the application and the file system. Used to store and display messages and configuration variables.
+-   **TubeArchivist-ES**: ElasticSearch stores video meta data and makes everything searchable. Also keeps track of the download queue.
+-   **TubeArchivist**: Once your YouTube video collection grows, it becomes hard to search and find a specific video. That's where Tube Archivist comes in: By indexing your video collection with metadata from YouTube, you can organize, search and enjoy your archived YouTube videos without hassle offline through a convenient web interface.
+
+### Step 1: Install  `TubeArchivist-RedisJSON`
+
+![enter image description here](https://i.imgur.com/ycAqFRU.png)
+This is the easiest container to setup of the thee, just make sure that you do not have any port conflicts, and that your `/data` is mounted to the correct path. The other containers will map to the same directory.
+
+If you need to install `TubeArchivist-RedisJSON`on a different port, you'll have to follow [these steps](https://github.com/tubearchivist/tubearchivist#redis-on-a-custom-port) later on when installing the `TubeArchivist` container
+
+
+### Step 2: Install  `TubeArchivist-ES`
+![enter image description here](https://i.imgur.com/o6tsTdt.png)
+ElasticSeach is also pretty easy to setup. Again, make sure you have no port conflicts, make sure that you mapped `/usr/share/elasticsearch/data` to the same directory as `RedisJSON`, and make sure to change the default password to something more secure. 
+
+There is three additional settings in the "show more settings" area, but leave those as they are.
+
+
+### Step 3: Install  `TubeArchivist`
+
+![enter image description here](https://i.imgur.com/dwSCfgO.png)
+It's finally time to set up TubeArchivist!
+
+ - `Port:`Again, make sure that you have no port conflicts on 8000.
+   
+ - `Youtube Media Path:` is where you'll download all of your videos to.
+   Make sure that this is an empty directory to not cause confusion when
+   starting the application. If you have existing videos that you'd like
+   to import into Tube Archivist, please checkout the [settings
+   wiki.](https://github.com/tubearchivist/tubearchivist/wiki/Settings#manual-media-files-import)
+   
+   
+- `Appdata:` This should be the same base path as the other two containers.
+   
+ - `TA Username:`This will be your username for TubeArchivist.
+   
+ - `TA Password:`This will be your password for TubeArchivist.
+   
+ - `Redis` This will be JUST the ip address of your redis container
+
+ - `ElasticSearch Password:`This is the password you defined in the `TubeArchivist-ES` container.
+ - `ElasticSearch:` This seems to cause some confusion, but it's a pretty simple step, just replace the IP and Port to match you `TubeArchivist-ES` container.
+
+ (example: if your IP is 192.168.1.15, the value should be http://192.168.1.15:9200)
+
+ - `Time Zone:` This is an important step for your scheduler, to find your timezone, use a site like [TimeZoneConverter](http://www.timezoneconverter.com/cgi-bin/findzone.tzc) 
+
+### From there, you should be able to start up your containers and you're good to go!
+If you're still having trouble, join us on [discord](https://discord.gg/AFwz8nE7BK) and come to the #unraid channel.

docs/Main.md

@@ -1,10 +0,0 @@
-# Tube Archivist Home Page Functionality
-
-This is the landing page, when you first open **Tube Archivist**. You have a few options to sort and filter that view:
-- With the **Sort Order** you can select how the "Recent Videos" are sorted: 
-  - **Date Published**: Sorts the list by date when the video was published on YouTube, newest on top.
-  - **Date Downloaded**: Sorts the list based on when you have downloaded the video to your archive, newest on top.
-- With **Hide Watched** you can filter out videos you have already marked as watched to only show unwatched videos. 
-- You can use those two options together to for example filter the list to *Hide Watched* videos **and** sort by date downloaded.
-
-Additionally the search icon <img src="assets/icon-search.png?raw=true" alt="search icon" width="20px" style="margin:0 5px;"> opens a text field to search your collection.

docs/Playlists.md

@@ -0,0 +1,23 @@
+# Playlist Overview and Playlist Detail Page
+The playlists are organized in two different levels, similar as the [channels](Channels):
+
+## Playlist Overview
+Accessible at `/playlist/` of your Tube Archivist, this **Overview Page** shows a list of all playlists you have indexed over all your channels.
+- You can filter that list to show only subscribed to playlists with the toggle.
+
+You can index playlists of a channel from the channel detail page as described [here](Channels#channel-detail).
+
+The **Subscribe to Playlist** button <img src="assets/icon-add.png?raw=true" alt="add icon" width="20px" style="margin:0 5px;"> opens a text field to subscribe to playlists. You have a few options:
+- Enter the YouTube playlist id, for example: *PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha*
+- Enter the Youtube dedicated playlist url, for example: *https://www.youtube.com/playlist?list=PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha*
+- Add one per line.
+- NOTE: It doesn't make sense to subscribe to a playlist if you are already subscribed the corresponding channel as this will slow down the **Rescan Subscriptions** [task](Downloads#rescan-subscriptions).
+
+You can search your indexed playlists by clicking on the search icon <img src="assets/icon-search.png?raw=true" alt="search icon" width="20px" style="margin:0 5px;">. This will open a dedicated page.
+
+## Playlist Detail
+Each playlist will get a dedicated playlist detail page accessible at `/playlist/<playlist-id>/` of your Tube Archivist. This page shows all the videos you have downloaded from this playlist.
+
+- If you are subscribed to the playlist, an Unsubscribe button will show, else the Subscribe button will show.
+- The Mark as Watched button will mark all videos of this playlist as watched.
+- The **Delete Playlist** button will give you the option to delete just the *metadata* which won't delete any media files or *delete all* which will delete metadata plus all videos belonging to this playlist.

docs/Settings.md

@@ -19,19 +19,98 @@ Settings related to the download process.
 - **Download Speed Limit**: Set your download speed limit in KB/s. This will pass the option `--limit-rate` to yt-dlp.
 - **Throttled Rate Limit**: Restart download if the download speed drops below this value in KB/s. This will pass the option `--throttled-rate` to yt-dlp. Using this option might have a negative effect if you have an unstable or slow internet connection.
 - **Sleep Interval**: Time in seconds to sleep between requests to YouTube. It's a good idea to set this to **3** seconds. Might be necessary to avoid throttling.
+- **Auto Delete Watched Videos**: Automatically delete videos marked as watched after selected days. If activated, checks your videos after download task is finished.
 
 ## Download Format
 Additional settings passed to yt-dlp.
-- **Format**: This controls which streams get downloaded and is equivalent to passing `--format` to yt-dlp. Use one of the recommended one or look at the documentation of [yt-dlp](https://github.com/yt-dlp/yt-dlp#format-selection). Please note: The option `--merge-output-format mp4` is automatically passed to yt-dlp to guarantee browser compatibility.
+- **Format**: This controls which streams get downloaded and is equivalent to passing `--format` to yt-dlp. Use one of the recommended one or look at the documentation of [yt-dlp](https://github.com/yt-dlp/yt-dlp#format-selection). Please note: The option `--merge-output-format mp4` is automatically passed to yt-dlp to guarantee browser compatibility. Similar to that, `--check-formats` is passed as well to check that the selected formats are actually downloadable.
 - **Embed Metadata**: This saves the available tags directly into the media file by passing `--embed-metadata` to yt-dlp.
 - **Embed Thumbnail**: This will save the thumbnail into the media file by passing `--embed-thumbnail` to yt-dlp.
 
+## Subtitles
+- **Download Setting**: Select the subtitle language you like to download. Add a comma separated list for multiple languages.
+- **Source Settings**: User created subtitles are provided from the uploader and are usually the video script. Auto generated is from YouTube, quality varies, particularly for auto translated tracks.
+- **Index Settings**: Enabling subtitle indexing will add the lines to Elasticsearch and will make subtitles searchable. This will increase the index size and is not recommended on low-end hardware.
+
+## Cookie
+Importing your YouTube Cookie into Tube Archivist allows yt-dlp to bypass age restrictions, gives access to private videos and your *watch later* or *liked videos*.
+
+### Security concerns
+Cookies are used to store your session and contain your access token to your google account, this file can be used to take over your account. Treat that file with utmost care as you would any other password or credential. *Tube Archivist* stores your cookie in Redis and will export the file container internally if it's needed.
+
+### Export your cookie
+- Install **Cookies.txt** addon for [chrome](https://chrome.google.com/webstore/detail/get-cookiestxt/bgaddhkoddajcdgocldbbfleckgcbcid) or [firefox](https://addons.mozilla.org/firefox/addon/cookies-txt).
+- Visit YouTube and login with whichever YouTube account you wish to use to generate the cookies.
+- Click on the extension icon in the toolbar - it will drop down showing the active cookies for YT.
+- Click Export to export the cookies, filename is by default *cookies.google.txt*.
+
+### Import your cookie
+Place the file *cookies.google.txt* into the *cache/import* folder of Tube Archivist and enable the cookie import. Once you click on *Update Application Configurations* to save your changes, your cookie will get imported and stored internally.
+
+Once imported, a **Validate Cookie File** button will show, where you can confirm if your cookie is working or not.
+
+### Use your cookie
+Once imported, additionally to the advantages above, your [Watch Later](https://www.youtube.com/playlist?list=WL) and [Liked Videos](https://www.youtube.com/playlist?list=LL) become a regular playlist you can download and subscribe to as any other [playlist](Playlists).
+
+### Limitation
+There is only one cookie per Tube Archivist instance, this will be shared between all users.
+
+## Integrations
+All third party integrations of TubeArchivist will **always** be *opt in*.
+- **API**: Your access token for the Tube Archivist API.
+- **returnyoutubedislike.com**: This will get return dislikes and average ratings for each video by integrating with the API from [returnyoutubedislike.com](https://www.returnyoutubedislike.com/).
+- **SponsorBlock**: Using [SponsorBlock](https://sponsor.ajay.app/) to get and skip sponsored content. If a video doesn't have timestamps, or has unlocked timestamps, use the browser addon to contribute to this excellent project. Can also be activated and deactivated as a per [channel overwrite](Settings#channel-customize).
+- **Cast**: Enabling the cast integration in the settings page will load an additional JS library from **Google**.
+    * Requirements
+        - HTTPS
+            * To use the cast integration HTTPS needs to be enabled, which can be done using a reverse proxy. This is a requirement by Google as communication to the cast device is required to be encrypted, but the content itself is not.
+        - Supported Browser
+            * A supported browser is required for this integration such as Google Chrome. Other browsers, especially Chromium-based browsers, may support casting by enabling it in the settings.
+        - Subtitles
+            * Subtitles are supported however they do not work out of the box and require additional configuration. Due to requirements by Google, to use subtitles you need additional headers which will need to be configured in your reverse proxy. See this [page](https://developers.google.com/cast/docs/web_sender/advanced#cors_requirements) for the specific requirements.
+               > You need the following headers: Content-Type, Accept-Encoding, and Range. Note that the last two headers, Accept-Encoding and Range, are additional headers that you may not have needed previously.
+               > Wildcards "*" cannot be used for the Access-Control-Allow-Origin header. If the page has protected media content, it must use a domain instead of a wildcard.
+
+
+# Scheduler Setup
+Schedule settings expect a cron like format, where the first value is minute, second is hour and third is day of the week. Day 0 is Sunday, day 1 is Monday etc.
+
+Examples:
+- **0 15 \***: Run task every day at 15:00 in the afternoon.
+- **30 8 \*/2**: Run task every second day of the week (Sun, Tue, Thu, Sat) at 08:30 in the morning.
+- **0 \*/3,8-17 \***: Execute every hour divisible by 3, and every hour during office hours (8 in the morning - 5 in the afternoon).
+- **0 8,16 \***: Execute every day at 8 in the morning and at 4 in the afternoon.
+- **auto**: Sensible default.
+- **0**: (zero), deactivate that task.
+
+NOTE:
+- Changes in the scheduler settings require a container restart to take effect.
+- Cron format as *number*/*number* are none standard cron and are not supported by the scheduler, for example **0 0/12 \*** is invalid, use **0 \*/12 \*** instead.
+- Avoid an unnecessary frequent schedule to not get blocked by YouTube. For that reason * or wildcards for minutes are not supported.
+
+## Rescan Subscriptions
+That's the equivalent task as run from the downloads page looking through your channel and playlist and add missing videos to the download queue.
+
+## Start download
+Start downloading all videos currently in the download queue.
+
+## Refresh Metadata
+Rescan videos, channels and playlists on youtube and update metadata periodically. This will also deactivate an item and exclude it from future refreshes if the link on YouTube is no longer available. This task is meant to be run once per day, set your schedule accordingly.
+
+The field **Refresh older than x days** takes a number where TubeArchivist will consider an item as *outdated*. This value is used to calculate how many items need to be refreshed today based on the total indexed. This will spread out the requests to YouTube. Sensible value here is **90** days.
+
+## Thumbnail check
+This will check if all expected thumbnails are there and will delete any artwork without matching video.
+
+## Index backup
+Create a zip file of the metadata and select **Max auto backups to keep** to automatically delete old backups created from this task.
+
 
 # Actions
 Additional database functionality.
 
 ## Manual Media Files Import
-So far this depends on the video you are trying to import to be still available on YouTube to get the metadata. Add the files you like to import to the */cache/import* folder. Then start the process from the settings page *Manual Media Files Import*. Make sure to follow one of the two methods below.
+So far this depends on the video you are trying to import to be still available on YouTube to get the metadata. Add the files you'd like to import to the */cache/import* folder. Then start the process from the settings page *Manual Media Files Import*. Make sure to follow one of the two methods below.
 
 ### Method 1:
 Add a matching *.json* file with the media file. Both files need to have the same base name, for example:
@@ -48,6 +127,8 @@ Detect the YouTube ID from filename, this accepts the default yt-dlp naming conv
 
 ### Some notes:
 - This will **consume** the files you put into the import folder: Files will get converted to mp4 if needed (this might take a long time...) and moved to the archive, *.json* files will get deleted upon completion to avoid having duplicates on the next run.
+- For best file transcoding quality, convert your media files with desired settings first before importing (#138).
+- There should be no subdirectories added to */cache/import*, only video files. If your existing video library has video files inside subdirectories, you can get all the files into one directory by running `find ./ -mindepth 2 -type f -exec mv '{}' . \;` from the top-level directory of your existing video library. You can also delete any remaining empty subdirectories with `find ./ -mindepth 1 -type d -delete`.
 - Maybe start with a subset of your files to import to make sure everything goes well...
 - Follow the logs to monitor progress and errors: `docker-compose logs -f tubearchivist`.
 
@@ -60,7 +141,11 @@ This will backup your metadata into a zip file. The file will get stored at *cac
 BE AWARE: This will **not** backup any media files, just the metadata from the Elasticsearch.
 
 ## Restore From Backup
-The restore functionality will expect the same zip file in *cache/backup* as created from the **Backup database** function. This will recreate the index from the snapshot. If there are multiple backup files in the folder, the newest one will take priority. 
+The restore functionality will expect the same zip file in *cache/backup* as created from the **Backup database** function. This will recreate the index from the snapshot. There will be a list of all available backup to choose from. The *source* tag can have these different values:
+- **manual**: For backups manually created from here on the settings page.
+- **auto**: For backups automatically created via a sceduled task.
+- **update**: For backups created after a Tube Archivist update due to changes in the index.
+- **False**: Undefined.
 
 BE AWARE: This will **replace** your current index with the one from the backup file. This won't restore any media files.
 
@@ -69,5 +154,6 @@ This function will go through all your media files and looks at the whole index
 - Should the filename not match with the indexed media url, this will rename the video files correctly and update the index with the new link.
 - When you delete media files from the filesystem outside of the Tube Archivist interface, this will delete leftover metadata from the index.
 - When you have media files that are not indexed yet, this will grab the metadata from YouTube like it was a newly downloaded video. This can be useful when restoring from an older backup file with missing metadata but already downloaded mediafiles. NOTE: This only works if the media files are named in the same convention as Tube Archivist does, particularly the YouTube ID needs to be at the same index in the filename, alternatively see above for *Manual Media Files Import*.
+-This will also check all of your thumbnails and download any that are missing.
 
 BE AWARE: There is no undo.

docs/Video.md

@@ -0,0 +1,11 @@
+# Video Page
+
+Every video downloaded gets a dedicated page accessible at `/video/<video-id>` of your Tube Archivist.
+
+Clicking on the channel name or the channel icon will bring you to the dedicated channel detail [page](Channels#channel-detail).
+
+The button **Delete Video** will delete that video including the media file.
+
+When available, a playlist navigation will show at the bottom. Clicking on the playlist name will bring you to the dedicated [Playlist Detail](Playlists#playlist-detail) page showing all videos downloaded from that playlist. The number in square brackets indicates the position of the current video in that playlist.
+
+Clicking on the next or previous video name or thumbnail will bring you to that dedicated video page.

nginx.conf

@@ -1,18 +0,0 @@
-server {
-
-    listen 8000;
-
-    location /cache/ {
-        alias /cache/;
-    }
-
-    location /media/ {
-        alias /youtube/;
-    }
-
-    location / {
-        include uwsgi_params;
-        uwsgi_pass localhost:8080;
-    }
-
-}

tubearchivist/api/README.md

@@ -0,0 +1,233 @@
+# TubeArchivist API
+Documentation of available API endpoints.  
+**Note: This is very early alpha and will change!**
+
+## Authentication
+API token will get automatically created, accessible on the settings page. Token needs to be passed as an authorization header with every request. Additionally session based authentication is enabled too: When you are logged into your TubeArchivist instance, you'll have access to the api in the browser for testing.
+
+Curl example:
+```shell
+curl -v /api/video/<video-id>/ \
+    -H "Authorization: Token xxxxxxxxxx"
+```
+
+Python requests example:
+```python
+import requests
+
+url = "/api/video/<video-id>/"
+headers = {"Authorization": "Token xxxxxxxxxx"}
+response = requests.get(url, headers=headers)
+```
+
+## Pagination
+The list views return a paginate object with the following keys:
+- page_size: int current page size set in config
+- page_from: int first result idx
+- prev_pages: array of ints of previous pages, if available
+- current_page: int current page from query
+- max_hits: reached: bool if max of 10k results is reached
+- last_page: int of last page link
+- next_pages: array of ints of next pages
+- total_hits: int total results
+
+Pass page number as a query parameter: `page=2`. Defaults to *0*, `page=1` is redundant and falls back to *0*. If a page query doesn't return any results, you'll get `HTTP 404 Not Found`.
+
+## Login View
+Return token and user ID for username and password:  
+POST /api/login
+```json
+{
+    "username": "tubearchivist",
+    "password": "verysecret"
+}
+```
+
+after successful login returns 
+```json
+{
+    "token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
+    "user_id": 1
+}
+```
+
+## Video List View
+/api/video/
+
+## Video Item View
+/api/video/\<video_id>/
+
+## Video Progress View
+/api/video/\<video_id>/progress  
+
+Progress is stored for each user.
+
+### Get last player position of a video
+GET /api/video/\<video_id>/progress
+```json
+{
+    "youtube_id": "<video_id>",
+    "user_id": 1,
+    "position": 100
+}
+```
+
+### Post player position of video
+POST /api/video/\<video_id>/progress
+```json
+{
+    "position": 100
+}
+```
+
+### Delete player position of video
+DELETE /api/video/\<video_id>/progress  
+
+
+## Sponsor Block View
+/api/video/\<video_id>/sponsor/
+
+Integrate with sponsorblock
+
+### Get list of segments
+GET /api/video/\<video_id>/sponsor/
+
+
+### Vote on existing segment
+**This only simulates the request**  
+POST /api/video/\<video_id>/sponsor/
+```json
+{
+    "vote": {
+        "uuid": "<uuid>",
+        "yourVote": 1
+    }
+}
+```
+yourVote needs to be *int*: 0 for downvote, 1 for upvote, 20 to undo vote
+
+### Create new segment
+**This only simulates the request**  
+POST /api/video/\<video_id>/sponsor/
+```json
+{
+    "segment": {
+        "startTime": 5,
+        "endTime": 10
+    }
+}
+```
+Timestamps either *int* or *float*, end time can't be before start time.
+
+
+## Channel List View
+/api/channel/
+
+### Subscribe to a list of channels
+POST /api/channel/
+```json
+{
+    "data": [
+        {"channel_id": "UC9-y-6csu5WGm29I7JiwpnA", "channel_subscribed": true}
+    ]
+}
+```
+
+## Channel Item View
+/api/channel/\<channel_id>/
+
+## Channel Videos View
+/api/channel/\<channel_id>/video/
+
+## Playlist List View
+/api/playlist/
+
+## Playlists Item View
+/api/playlist/\<playlist_id>/
+
+## Playlist Videos View
+/api/playlist/\<playlist_id>/video/
+
+## Download Queue List View
+GET /api/download/
+
+Parameter:
+- filter: pending, ignore
+
+### Add list of videos to download queue
+POST /api/download/
+```json
+{
+    "data": [
+        {"youtube_id": "NYj3DnI81AQ", "status": "pending"}
+    ]
+}
+```
+
+### Delete download queue items by filter
+DELETE /api/download/?filter=ignore  
+DELETE /api/download/?filter=pending
+
+## Download Queue Item View
+GET /api/download/\<video_id>/  
+POST /api/download/\<video_id>/
+
+Ignore video in download queue:
+```json
+{
+    "status": "ignore"
+}
+```
+
+Add to queue previously ignored video:
+```json
+{
+    "status": "pending"
+}
+```
+
+DELETE /api/download/\<video_id>/  
+Forget or delete from download queue
+
+## Ping View
+Validate your connection with the API  
+GET /api/ping
+
+When valid returns message with user id: 
+```json
+{
+    "response": "pong",
+    "user": 1
+}
+```
+
+## Task View
+Start a background task
+POST /api/task/
+```json
+{
+    "run": "task_name"
+}
+```
+
+List of valid task names:
+- **download_pending**: Start the download queue
+- **rescan_pending**: Rescan your subscriptions
+
+
+## Cookie View
+Check your youtube cookie settings  
+GET /api/cookie/
+```json
+{
+    "cookie_enabled": true
+}
+```
+
+POST /api/cookie/  
+Send empty post request to validate cookie.
+```json
+{
+    "cookie_validated": true
+}
+```

tubearchivist/api/admin.py

@@ -0,0 +1,3 @@
+from django.contrib import admin  # noqa: F401
+
+# Register your models here.

tubearchivist/api/apps.py

@@ -0,0 +1,10 @@
+"""apps file for api package"""
+
+from django.apps import AppConfig
+
+
+class ApiConfig(AppConfig):
+    """app config"""
+
+    default_auto_field = "django.db.models.BigAutoField"
+    name = "api"

tubearchivist/api/models.py

@@ -0,0 +1,3 @@
+"""api models"""
+
+# from django.db import models

tubearchivist/api/src/search_processor.py

@@ -0,0 +1,125 @@
+"""
+Functionality:
+- processing search results for frontend
+- this is duplicated code from home.src.frontend.searching.SearchHandler
+"""
+
+import urllib.parse
+
+from home.src.download.thumbnails import ThumbManager
+from home.src.ta.config import AppConfig
+from home.src.ta.helper import date_praser
+
+
+class SearchProcess:
+    """process search results"""
+
+    CONFIG = AppConfig().config
+    CACHE_DIR = CONFIG["application"]["cache_dir"]
+
+    def __init__(self, response):
+        self.response = response
+        self.processed = False
+
+    def process(self):
+        """dedect type and process"""
+        if "_source" in self.response.keys():
+            # single
+            self.processed = self._process_result(self.response)
+
+        elif "hits" in self.response.keys():
+            # multiple
+            self.processed = []
+            all_sources = self.response["hits"]["hits"]
+            for result in all_sources:
+                self.processed.append(self._process_result(result))
+
+        return self.processed
+
+    def _process_result(self, result):
+        """dedect which type of data to process"""
+        index = result["_index"]
+        processed = False
+        if index == "ta_video":
+            processed = self._process_video(result["_source"])
+        if index == "ta_channel":
+            processed = self._process_channel(result["_source"])
+        if index == "ta_playlist":
+            processed = self._process_playlist(result["_source"])
+        if index == "ta_download":
+            processed = self._process_download(result["_source"])
+
+        return processed
+
+    @staticmethod
+    def _process_channel(channel_dict):
+        """run on single channel"""
+        channel_id = channel_dict["channel_id"]
+        art_base = f"/cache/channels/{channel_id}"
+        date_str = date_praser(channel_dict["channel_last_refresh"])
+        channel_dict.update(
+            {
+                "channel_last_refresh": date_str,
+                "channel_banner_url": f"{art_base}_banner.jpg",
+                "channel_thumb_url": f"{art_base}_thumb.jpg",
+                "channel_tvart_url": False,
+            }
+        )
+
+        return dict(sorted(channel_dict.items()))
+
+    def _process_video(self, video_dict):
+        """run on single video dict"""
+        video_id = video_dict["youtube_id"]
+        media_url = urllib.parse.quote(video_dict["media_url"])
+        vid_last_refresh = date_praser(video_dict["vid_last_refresh"])
+        published = date_praser(video_dict["published"])
+        vid_thumb_url = ThumbManager().vid_thumb_path(video_id)
+        channel = self._process_channel(video_dict["channel"])
+
+        if "subtitles" in video_dict:
+            for idx, _ in enumerate(video_dict["subtitles"]):
+                url = video_dict["subtitles"][idx]["media_url"]
+                video_dict["subtitles"][idx]["media_url"] = f"/media/{url}"
+
+        video_dict.update(
+            {
+                "channel": channel,
+                "media_url": f"/media/{media_url}",
+                "vid_last_refresh": vid_last_refresh,
+                "published": published,
+                "vid_thumb_url": f"{self.CACHE_DIR}/{vid_thumb_url}",
+            }
+        )
+
+        return dict(sorted(video_dict.items()))
+
+    @staticmethod
+    def _process_playlist(playlist_dict):
+        """run on single playlist dict"""
+        playlist_id = playlist_dict["playlist_id"]
+        playlist_last_refresh = date_praser(
+            playlist_dict["playlist_last_refresh"]
+        )
+        playlist_dict.update(
+            {
+                "playlist_thumbnail": f"/cache/playlists/{playlist_id}.jpg",
+                "playlist_last_refresh": playlist_last_refresh,
+            }
+        )
+
+        return dict(sorted(playlist_dict.items()))
+
+    def _process_download(self, download_dict):
+        """run on single download item"""
+        video_id = download_dict["youtube_id"]
+        vid_thumb_url = ThumbManager().vid_thumb_path(video_id)
+        published = date_praser(download_dict["published"])
+
+        download_dict.update(
+            {
+                "vid_thumb_url": f"{self.CACHE_DIR}/{vid_thumb_url}",
+                "published": published,
+            }
+        )
+        return dict(sorted(download_dict.items()))

tubearchivist/api/src/task_processor.py

@@ -0,0 +1,54 @@
+"""
+Functionality:
+- process tasks from API
+- validate
+- handover to celery
+"""
+
+from home.src.ta.ta_redis import RedisArchivist
+from home.tasks import download_pending, update_subscribed
+
+
+class TaskHandler:
+    """handle tasks from api"""
+
+    def __init__(self, data):
+        self.data = data
+
+    def run_task(self):
+        """map data and run"""
+        task_name = self.data["run"]
+        try:
+            to_run = self.exec_map(task_name)
+        except KeyError as err:
+            print(f"invalid task name {task_name}")
+            raise ValueError from err
+
+        response = to_run()
+        response.update({"task": task_name})
+        return response
+
+    def exec_map(self, task_name):
+        """map dict key and return function to execute"""
+        exec_map = {
+            "download_pending": self._download_pending,
+            "rescan_pending": self._rescan_pending,
+        }
+
+        return exec_map[task_name]
+
+    @staticmethod
+    def _rescan_pending():
+        """look for new items in subscribed channels"""
+        print("rescan subscribed channels")
+        update_subscribed.delay()
+        return {"success": True}
+
+    @staticmethod
+    def _download_pending():
+        """start the download queue"""
+        print("download pending")
+        running = download_pending.delay()
+        print("set task id: " + running.id)
+        RedisArchivist().set_message("dl_queue_id", running.id, expire=False)
+        return {"success": True}

tubearchivist/api/tests.py

@@ -0,0 +1,3 @@
+from django.test import TestCase  # noqa: F401
+
+# Create your tests here.

tubearchivist/api/urls.py

@@ -0,0 +1,96 @@
+"""all api urls"""
+
+from api.views import (
+    ChannelApiListView,
+    ChannelApiVideoView,
+    ChannelApiView,
+    CookieView,
+    DownloadApiListView,
+    DownloadApiView,
+    LoginApiView,
+    PingView,
+    PlaylistApiListView,
+    PlaylistApiVideoView,
+    PlaylistApiView,
+    TaskApiView,
+    VideoApiListView,
+    VideoApiView,
+    VideoProgressView,
+    VideoSponsorView,
+)
+from django.urls import path
+
+urlpatterns = [
+    path("ping/", PingView.as_view(), name="ping"),
+    path("login/", LoginApiView.as_view(), name="api-login"),
+    path(
+        "video/",
+        VideoApiListView.as_view(),
+        name="api-video-list",
+    ),
+    path(
+        "video/<slug:video_id>/",
+        VideoApiView.as_view(),
+        name="api-video",
+    ),
+    path(
+        "video/<slug:video_id>/progress/",
+        VideoProgressView.as_view(),
+        name="api-video-progress",
+    ),
+    path(
+        "video/<slug:video_id>/sponsor/",
+        VideoSponsorView.as_view(),
+        name="api-video-sponsor",
+    ),
+    path(
+        "channel/",
+        ChannelApiListView.as_view(),
+        name="api-channel-list",
+    ),
+    path(
+        "channel/<slug:channel_id>/",
+        ChannelApiView.as_view(),
+        name="api-channel",
+    ),
+    path(
+        "channel/<slug:channel_id>/video/",
+        ChannelApiVideoView.as_view(),
+        name="api-channel-video",
+    ),
+    path(
+        "playlist/",
+        PlaylistApiListView.as_view(),
+        name="api-playlist-list",
+    ),
+    path(
+        "playlist/<slug:playlist_id>/",
+        PlaylistApiView.as_view(),
+        name="api-playlist",
+    ),
+    path(
+        "playlist/<slug:playlist_id>/video/",
+        PlaylistApiVideoView.as_view(),
+        name="api-playlist-video",
+    ),
+    path(
+        "download/",
+        DownloadApiListView.as_view(),
+        name="api-download-list",
+    ),
+    path(
+        "download/<slug:video_id>/",
+        DownloadApiView.as_view(),
+        name="api-download",
+    ),
+    path(
+        "task/",
+        TaskApiView.as_view(),
+        name="api-task",
+    ),
+    path(
+        "cookie/",
+        CookieView.as_view(),
+        name="api-cookie",
+    ),
+]

tubearchivist/api/views.py

@@ -0,0 +1,489 @@
+"""all API views"""
+
+from api.src.search_processor import SearchProcess
+from api.src.task_processor import TaskHandler
+from home.src.download.queue import PendingInteract
+from home.src.download.yt_cookie import CookieHandler
+from home.src.es.connect import ElasticWrap
+from home.src.index.generic import Pagination
+from home.src.index.video import SponsorBlock
+from home.src.ta.config import AppConfig
+from home.src.ta.helper import UrlListParser
+from home.src.ta.ta_redis import RedisArchivist, RedisQueue
+from home.tasks import extrac_dl, subscribe_to
+from rest_framework.authentication import (
+    SessionAuthentication,
+    TokenAuthentication,
+)
+from rest_framework.authtoken.models import Token
+from rest_framework.authtoken.views import ObtainAuthToken
+from rest_framework.permissions import IsAuthenticated
+from rest_framework.response import Response
+from rest_framework.views import APIView
+
+
+class ApiBaseView(APIView):
+    """base view to inherit from"""
+
+    authentication_classes = [SessionAuthentication, TokenAuthentication]
+    permission_classes = [IsAuthenticated]
+    search_base = False
+    data = False
+
+    def __init__(self):
+        super().__init__()
+        self.response = {"data": False, "config": AppConfig().config}
+        self.data = {"query": {"match_all": {}}}
+        self.status_code = False
+        self.context = False
+        self.pagination_handler = False
+
+    def get_document(self, document_id):
+        """get single document from es"""
+        path = f"{self.search_base}{document_id}"
+        print(path)
+        response, status_code = ElasticWrap(path).get()
+        try:
+            self.response["data"] = SearchProcess(response).process()
+        except KeyError:
+            print(f"item not found: {document_id}")
+            self.response["data"] = False
+        self.status_code = status_code
+
+    def initiate_pagination(self, request):
+        """set initial pagination values"""
+        user_id = request.user.id
+        page_get = int(request.GET.get("page", 0))
+        self.pagination_handler = Pagination(page_get, user_id)
+        self.data.update(
+            {
+                "size": self.pagination_handler.pagination["page_size"],
+                "from": self.pagination_handler.pagination["page_from"],
+            }
+        )
+
+    def get_document_list(self, request):
+        """get a list of results"""
+        print(self.search_base)
+        self.initiate_pagination(request)
+        es_handler = ElasticWrap(self.search_base)
+        response, status_code = es_handler.get(data=self.data)
+        self.response["data"] = SearchProcess(response).process()
+        if self.response["data"]:
+            self.status_code = status_code
+        else:
+            self.status_code = 404
+
+        self.pagination_handler.validate(response["hits"]["total"]["value"])
+        self.response["paginate"] = self.pagination_handler.pagination
+
+
+class VideoApiView(ApiBaseView):
+    """resolves to /api/video/<video_id>/
+    GET: returns metadata dict of video
+    """
+
+    search_base = "ta_video/_doc/"
+
+    def get(self, request, video_id):
+        # pylint: disable=unused-argument
+        """get request"""
+        self.get_document(video_id)
+        return Response(self.response, status=self.status_code)
+
+
+class VideoApiListView(ApiBaseView):
+    """resolves to /api/video/
+    GET: returns list of videos
+    """
+
+    search_base = "ta_video/_search/"
+
+    def get(self, request):
+        """get request"""
+        self.data.update({"sort": [{"published": {"order": "desc"}}]})
+        self.get_document_list(request)
+
+        return Response(self.response)
+
+
+class VideoProgressView(ApiBaseView):
+    """resolves to /api/video/<video_id>/
+    handle progress status for video
+    """
+
+    def get(self, request, video_id):
+        """get progress for a single video"""
+        user_id = request.user.id
+        key = f"{user_id}:progress:{video_id}"
+        video_progress = RedisArchivist().get_message(key)
+        position = video_progress.get("position", 0)
+
+        self.response = {
+            "youtube_id": video_id,
+            "user_id": user_id,
+            "position": position,
+        }
+        return Response(self.response)
+
+    def post(self, request, video_id):
+        """set progress position in redis"""
+        position = request.data.get("position", 0)
+        key = f"{request.user.id}:progress:{video_id}"
+        message = {"position": position, "youtube_id": video_id}
+        RedisArchivist().set_message(key, message, expire=False)
+        self.response = request.data
+
+        return Response(self.response)
+
+    def delete(self, request, video_id):
+        """delete progress position"""
+        key = f"{request.user.id}:progress:{video_id}"
+        RedisArchivist().del_message(key)
+        self.response = {"progress-reset": video_id}
+
+        return Response(self.response)
+
+
+class VideoSponsorView(ApiBaseView):
+    """resolves to /api/video/<video_id>/sponsor/
+    handle sponsor block integration
+    """
+
+    search_base = "ta_video/_doc/"
+
+    def get(self, request, video_id):
+        """get sponsor info"""
+        # pylint: disable=unused-argument
+
+        self.get_document(video_id)
+        sponsorblock = self.response["data"].get("sponsorblock")
+
+        return Response(sponsorblock)
+
+    def post(self, request, video_id):
+        """post verification and timestamps"""
+        if "segment" in request.data:
+            response, status_code = self._create_segment(request, video_id)
+        elif "vote" in request.data:
+            response, status_code = self._vote_on_segment(request)
+
+        return Response(response, status=status_code)
+
+    @staticmethod
+    def _create_segment(request, video_id):
+        """create segment in API"""
+        start_time = request.data["segment"]["startTime"]
+        end_time = request.data["segment"]["endTime"]
+        response, status_code = SponsorBlock(request.user.id).post_timestamps(
+            video_id, start_time, end_time
+        )
+
+        return response, status_code
+
+    @staticmethod
+    def _vote_on_segment(request):
+        """validate on existing segment"""
+        user_id = request.user.id
+        uuid = request.data["vote"]["uuid"]
+        vote = request.data["vote"]["yourVote"]
+        response, status_code = SponsorBlock(user_id).vote_on_segment(
+            uuid, vote
+        )
+
+        return response, status_code
+
+
+class ChannelApiView(ApiBaseView):
+    """resolves to /api/channel/<channel_id>/
+    GET: returns metadata dict of channel
+    """
+
+    search_base = "ta_channel/_doc/"
+
+    def get(self, request, channel_id):
+        # pylint: disable=unused-argument
+        """get request"""
+        self.get_document(channel_id)
+        return Response(self.response, status=self.status_code)
+
+
+class ChannelApiListView(ApiBaseView):
+    """resolves to /api/channel/
+    GET: returns list of channels
+    POST: edit a list of channels
+    """
+
+    search_base = "ta_channel/_search/"
+
+    def get(self, request):
+        """get request"""
+        self.get_document_list(request)
+        self.data.update(
+            {"sort": [{"channel_name.keyword": {"order": "asc"}}]}
+        )
+
+        return Response(self.response)
+
+    @staticmethod
+    def post(request):
+        """subscribe to list of channels"""
+        data = request.data
+        try:
+            to_add = data["data"]
+        except KeyError:
+            message = "missing expected data key"
+            print(message)
+            return Response({"message": message}, status=400)
+
+        pending = [i["channel_id"] for i in to_add if i["channel_subscribed"]]
+        url_str = " ".join(pending)
+        subscribe_to.delay(url_str)
+
+        return Response(data)
+
+
+class ChannelApiVideoView(ApiBaseView):
+    """resolves to /api/channel/<channel-id>/video
+    GET: returns a list of videos of channel
+    """
+
+    search_base = "ta_video/_search/"
+
+    def get(self, request, channel_id):
+        """handle get request"""
+        self.data.update(
+            {
+                "query": {
+                    "term": {"channel.channel_id": {"value": channel_id}}
+                },
+                "sort": [{"published": {"order": "desc"}}],
+            }
+        )
+        self.get_document_list(request)
+
+        return Response(self.response, status=self.status_code)
+
+
+class PlaylistApiListView(ApiBaseView):
+    """resolves to /api/playlist/
+    GET: returns list of indexed playlists
+    """
+
+    search_base = "ta_playlist/_search/"
+
+    def get(self, request):
+        """handle get request"""
+        self.data.update(
+            {"sort": [{"playlist_name.keyword": {"order": "asc"}}]}
+        )
+        self.get_document_list(request)
+        return Response(self.response)
+
+
+class PlaylistApiView(ApiBaseView):
+    """resolves to /api/playlist/<playlist_id>/
+    GET: returns metadata dict of playlist
+    """
+
+    search_base = "ta_playlist/_doc/"
+
+    def get(self, request, playlist_id):
+        # pylint: disable=unused-argument
+        """get request"""
+        self.get_document(playlist_id)
+        return Response(self.response, status=self.status_code)
+
+
+class PlaylistApiVideoView(ApiBaseView):
+    """resolves to /api/playlist/<playlist_id>/video
+    GET: returns list of videos in playlist
+    """
+
+    search_base = "ta_video/_search/"
+
+    def get(self, request, playlist_id):
+        """handle get request"""
+        self.data["query"] = {
+            "term": {"playlist.keyword": {"value": playlist_id}}
+        }
+        self.data.update({"sort": [{"published": {"order": "desc"}}]})
+
+        self.get_document_list(request)
+        return Response(self.response, status=self.status_code)
+
+
+class DownloadApiView(ApiBaseView):
+    """resolves to /api/download/<video_id>/
+    GET: returns metadata dict of an item in the download queue
+    POST: update status of item to pending or ignore
+    DELETE: forget from download queue
+    """
+
+    search_base = "ta_download/_doc/"
+    valid_status = ["pending", "ignore"]
+
+    def get(self, request, video_id):
+        # pylint: disable=unused-argument
+        """get request"""
+        self.get_document(video_id)
+        return Response(self.response, status=self.status_code)
+
+    def post(self, request, video_id):
+        """post to video to change status"""
+        item_status = request.data["status"]
+        if item_status not in self.valid_status:
+            message = f"{video_id}: invalid status {item_status}"
+            print(message)
+            return Response({"message": message}, status=400)
+
+        print(f"{video_id}: change status to {item_status}")
+        PendingInteract(video_id=video_id, status=item_status).update_status()
+        RedisQueue().clear_item(video_id)
+
+        return Response(request.data)
+
+    @staticmethod
+    def delete(request, video_id):
+        # pylint: disable=unused-argument
+        """delete single video from queue"""
+        print(f"{video_id}: delete from queue")
+        PendingInteract(video_id=video_id).delete_item()
+
+        return Response({"success": True})
+
+
+class DownloadApiListView(ApiBaseView):
+    """resolves to /api/download/
+    GET: returns latest videos in the download queue
+    POST: add a list of videos to download queue
+    DELETE: remove items based on query filter
+    """
+
+    search_base = "ta_download/_search/"
+    valid_filter = ["pending", "ignore"]
+
+    def get(self, request):
+        """get request"""
+        query_filter = request.GET.get("filter", False)
+        self.data.update({"sort": [{"timestamp": {"order": "asc"}}]})
+        if query_filter:
+            if query_filter not in self.valid_filter:
+                message = f"invalid url query filder: {query_filter}"
+                print(message)
+                return Response({"message": message}, status=400)
+
+            self.data["query"] = {"term": {"status": {"value": query_filter}}}
+
+        self.get_document_list(request)
+        return Response(self.response)
+
+    @staticmethod
+    def post(request):
+        """add list of videos to download queue"""
+        print(f"request meta data: {request.META}")
+        data = request.data
+        try:
+            to_add = data["data"]
+        except KeyError:
+            message = "missing expected data key"
+            print(message)
+            return Response({"message": message}, status=400)
+
+        pending = [i["youtube_id"] for i in to_add if i["status"] == "pending"]
+        url_str = " ".join(pending)
+        try:
+            youtube_ids = UrlListParser(url_str).process_list()
+        except ValueError:
+            message = f"failed to parse: {url_str}"
+            print(message)
+            return Response({"message": message}, status=400)
+
+        extrac_dl.delay(youtube_ids)
+
+        return Response(data)
+
+    def delete(self, request):
+        """delete download queue"""
+        query_filter = request.GET.get("filter", False)
+        if query_filter not in self.valid_filter:
+            message = f"invalid url query filter: {query_filter}"
+            print(message)
+            return Response({"message": message}, status=400)
+
+        message = f"delete queue by status: {query_filter}"
+        print(message)
+        PendingInteract(status=query_filter).delete_by_status()
+
+        return Response({"message": message})
+
+
+class PingView(ApiBaseView):
+    """resolves to /api/ping/
+    GET: test your connection
+    """
+
+    @staticmethod
+    def get(request):
+        """get pong"""
+        data = {"response": "pong", "user": request.user.id}
+        return Response(data)
+
+
+class LoginApiView(ObtainAuthToken):
+    """resolves to /api/login/
+    POST: return token and username after successful login
+    """
+
+    def post(self, request, *args, **kwargs):
+        """post data"""
+        # pylint: disable=no-member
+        serializer = self.serializer_class(
+            data=request.data, context={"request": request}
+        )
+        serializer.is_valid(raise_exception=True)
+        user = serializer.validated_data["user"]
+        token, _ = Token.objects.get_or_create(user=user)
+
+        print(f"returning token for user with id {user.pk}")
+
+        return Response({"token": token.key, "user_id": user.pk})
+
+
+class TaskApiView(ApiBaseView):
+    """resolves to /api/task/
+    POST: start a new background task
+    """
+
+    def post(self, request):
+        """handle post request"""
+
+        data = request.data
+        print(data)
+        response = TaskHandler(data).run_task()
+
+        return Response(response)
+
+
+class CookieView(ApiBaseView):
+    """resolves to /api/cookie/
+    GET: check if cookie is enabled
+    POST: verify validity of cookie
+    """
+
+    @staticmethod
+    def get(request):
+        """handle get request"""
+        # pylint: disable=unused-argument
+        config = AppConfig().config
+        cookie_enabled = config["downloads"]["cookie_import"]
+
+        return Response({"cookie_enabled": cookie_enabled})
+
+    @staticmethod
+    def post(request):
+        """handle post request"""
+        # pylint: disable=unused-argument
+        validated = CookieHandler().validate()
+
+        return Response({"cookie_validated": validated})

tubearchivist/config/settings.py

@@ -14,7 +14,8 @@ import hashlib
 from os import environ, path
 from pathlib import Path
 
-from home.src.config import AppConfig
+from corsheaders.defaults import default_headers
+from home.src.ta.config import AppConfig
 
 # Build paths inside the project like this: BASE_DIR / 'subdir'.
 BASE_DIR = Path(__file__).resolve().parent.parent
@@ -41,14 +42,19 @@ INSTALLED_APPS = [
     "django.contrib.contenttypes",
     "django.contrib.sessions",
     "django.contrib.messages",
+    "corsheaders",
     "whitenoise.runserver_nostatic",
     "django.contrib.staticfiles",
     "django.contrib.humanize",
+    "rest_framework",
+    "rest_framework.authtoken",
+    "api",
 ]
 
 MIDDLEWARE = [
     "django.middleware.security.SecurityMiddleware",
     "django.contrib.sessions.middleware.SessionMiddleware",
+    "corsheaders.middleware.CorsMiddleware",
     "whitenoise.middleware.WhiteNoiseMiddleware",
     "django.middleware.common.CommonMiddleware",
     "django.middleware.csrf.CsrfViewMiddleware",
@@ -116,7 +122,7 @@ AUTH_USER_MODEL = "home.Account"
 # https://docs.djangoproject.com/en/3.2/topics/i18n/
 
 LANGUAGE_CODE = "en-us"
-TIME_ZONE = "UTC"
+TIME_ZONE = environ.get("TZ") or "UTC"
 USE_I18N = True
 USE_L10N = True
 USE_TZ = True
@@ -137,3 +143,24 @@ DEFAULT_AUTO_FIELD = "django.db.models.BigAutoField"
 
 LOGIN_URL = "/login/"
 LOGOUT_REDIRECT_URL = "/login/"
+
+# Cors needed for browser extension
+# background.js makes the request so HTTP_ORIGIN will be from extension
+if environ.get("DISABLE_CORS"):
+    # disable cors
+    CORS_ORIGIN_ALLOW_ALL = True
+else:
+    CORS_ALLOWED_ORIGIN_REGEXES = [
+        r"moz-extension://*",
+        r"chrome-extension://*",
+    ]
+    CORS_ALLOWED_ORIGINS = ["http://localhost:3000"]
+
+
+CORS_ALLOW_HEADERS = list(default_headers) + [
+    "mode",
+]
+
+# TA application settings
+TA_UPSTREAM = "https://github.com/tubearchivist/tubearchivist"
+TA_VERSION = "v0.1.5"

tubearchivist/config/urls.py

@@ -18,5 +18,6 @@ from django.urls import include, path
 
 urlpatterns = [
     path("", include("home.urls")),
+    path("api/", include("api.urls")),
     path("admin/", admin.site.urls),
 ]

tubearchivist/home/apps.py

@@ -1,41 +1,108 @@
 """handle custom startup functions"""
 
 import os
+import sys
 
 from django.apps import AppConfig
-from home.src.config import AppConfig as ArchivistConfig
-from home.src.helper import RedisArchivist
-from home.src.index_management import index_check
-from home.src.thumbnails import validate_thumbnails
+from home.src.es.connect import ElasticWrap
+from home.src.es.index_setup import index_check
+from home.src.ta.config import AppConfig as ArchivistConfig
+from home.src.ta.ta_redis import RedisArchivist
 
 
-def sync_redis_state():
-    """make sure redis gets new config.json values"""
-    print("sync redis")
-    config_handler = ArchivistConfig()
-    config_handler.load_new_defaults()
+class StartupCheck:
+    """checks to run at application startup"""
 
+    MIN_MAJOR, MAX_MAJOR = 7, 7
+    MIN_MINOR = 17
 
-def make_folders():
-    """make needed cache folders here so docker doesn't mess it up"""
-    folders = ["download", "channels", "videos", "import", "backup"]
-    config = ArchivistConfig().config
-    cache_dir = config["application"]["cache_dir"]
-    for folder in folders:
-        folder_path = os.path.join(cache_dir, folder)
-        try:
-            os.makedirs(folder_path)
-        except FileExistsError:
-            continue
+    def __init__(self):
+        self.config_handler = ArchivistConfig()
+        self.redis_con = RedisArchivist()
+        self.has_run = self.get_has_run()
 
+    def run(self):
+        """run all startup checks"""
+        print("run startup checks")
+        self.es_version_check()
+        self.release_lock()
+        index_check()
+        self.sync_redis_state()
+        self.make_folders()
+        self.set_has_run()
 
-def release_lock():
-    """make sure there are no leftover locks set in redis on container start"""
-    all_locks = ["manual_import", "downloading", "dl_queue", "dl_queue_id"]
-    for lock in all_locks:
-        response = RedisArchivist().del_message(lock)
-        if response:
-            print("deleted leftover key from redis: " + lock)
+    def get_has_run(self):
+        """validate if check has already executed"""
+        return self.redis_con.get_message("startup_check")
+
+    def set_has_run(self):
+        """startup checks run"""
+        message = {"status": True}
+        self.redis_con.set_message("startup_check", message, expire=120)
+
+    def sync_redis_state(self):
+        """make sure redis gets new config.json values"""
+        print("sync redis")
+        self.config_handler.load_new_defaults()
+
+    def make_folders(self):
+        """make needed cache folders here so docker doesn't mess it up"""
+        folders = [
+            "download",
+            "channels",
+            "videos",
+            "playlists",
+            "import",
+            "backup",
+        ]
+        cache_dir = self.config_handler.config["application"]["cache_dir"]
+        for folder in folders:
+            folder_path = os.path.join(cache_dir, folder)
+            try:
+                os.makedirs(folder_path)
+            except FileExistsError:
+                continue
+
+    def release_lock(self):
+        """make sure there are no leftover locks set in redis"""
+        all_locks = [
+            "startup_check",
+            "manual_import",
+            "downloading",
+            "dl_queue",
+            "dl_queue_id",
+            "rescan",
+        ]
+        for lock in all_locks:
+            response = self.redis_con.del_message(lock)
+            if response:
+                print("deleted leftover key from redis: " + lock)
+
+    def is_invalid(self, version):
+        """return true if es version is invalid, false if ok"""
+        major, minor = [int(i) for i in version.split(".")[:2]]
+        if not self.MIN_MAJOR <= major <= self.MAX_MAJOR:
+            return True
+
+        if minor >= self.MIN_MINOR:
+            return False
+
+        return True
+
+    def es_version_check(self):
+        """check for minimal elasticsearch version"""
+        response, _ = ElasticWrap("/").get()
+        version = response["version"]["number"]
+        invalid = self.is_invalid(version)
+
+        if invalid:
+            print(
+                "required elasticsearch version: "
+                + f"{self.MIN_MAJOR}.{self.MIN_MINOR}"
+            )
+            sys.exit(1)
+
+        print("elasticsearch version check passed")
 
 
 class HomeConfig(AppConfig):
@@ -45,8 +112,9 @@ class HomeConfig(AppConfig):
     name = "home"
 
     def ready(self):
-        release_lock()
-        index_check()
-        sync_redis_state()
-        make_folders()
-        validate_thumbnails()
+        startup = StartupCheck()
+        if startup.has_run["status"]:
+            print("startup checks run in other thread")
+            return
+
+        startup.run()

tubearchivist/home/config.json

@@ -7,7 +7,8 @@
     "default_view": {
         "home": "grid",
         "channel": "list",
-        "downloads": "list"
+        "downloads": "list",
+        "playlist": "grid"
     },
     "subscriptions": {
         "auto_search": false,
@@ -18,16 +19,33 @@
         "limit_count": false,
         "limit_speed": false,
         "sleep_interval": 3,
+        "autodelete_days": false,
         "format": false,
         "add_metadata": false,
         "add_thumbnail": false,
-        "throttledratelimit": false
+        "subtitle": false,
+        "subtitle_source": false,
+        "subtitle_index": false,
+        "cookie_import": false,
+        "throttledratelimit": false,
+        "integrate_ryd": false,
+        "integrate_sponsorblock": false
     },
     "application": {
         "app_root": "/app",
         "cache_dir": "/cache",
         "videos": "/youtube",
         "file_template": "%(id)s_%(title)s.mp4",
-        "colors": "dark"
+        "colors": "dark",
+        "enable_cast": false
+    },
+    "scheduler": {
+        "update_subscribed": false,
+        "download_pending": false,
+        "check_reindex":  {"minute": "0", "hour": "12", "day_of_week": "*"},
+        "check_reindex_days": 90,
+        "thumbnail_check": {"minute": "0", "hour": "17", "day_of_week": "*"},
+        "run_backup": {"minute": "0", "hour": "8", "day_of_week": "0"},
+        "run_backup_rotate": 5
     }
 }

tubearchivist/home/forms.py

@@ -1,114 +0,0 @@
-"""functionality:
-- hold all form classes used in the views
-"""
-
-from django import forms
-from django.contrib.auth.forms import AuthenticationForm
-from django.forms.widgets import PasswordInput, TextInput
-
-
-class CustomAuthForm(AuthenticationForm):
-    """better styled login form"""
-
-    username = forms.CharField(
-        widget=TextInput(attrs={"placeholder": "Username"}), label=False
-    )
-    password = forms.CharField(
-        widget=PasswordInput(attrs={"placeholder": "Password"}), label=False
-    )
-
-
-class UserSettingsForm(forms.Form):
-    """user configurations values"""
-
-    CHOICES = [
-        ("", "-- change color scheme --"),
-        ("dark", "Dark"),
-        ("light", "Light"),
-    ]
-
-    colors = forms.ChoiceField(
-        widget=forms.Select, choices=CHOICES, required=False
-    )
-    page_size = forms.IntegerField(required=False)
-
-
-class ApplicationSettingsForm(forms.Form):
-    """handle all application settings"""
-
-    METADATA_CHOICES = [
-        ("", "-- change metadata embed --"),
-        ("0", "don't embed metadata"),
-        ("1", "embed metadata"),
-    ]
-
-    THUMBNAIL_CHOICES = [
-        ("", "-- change thumbnail embed --"),
-        ("0", "don't embed thumbnail"),
-        ("1", "embed thumbnail"),
-    ]
-
-    subscriptions_channel_size = forms.IntegerField(required=False)
-    downloads_limit_count = forms.IntegerField(required=False)
-    downloads_limit_speed = forms.IntegerField(required=False)
-    downloads_throttledratelimit = forms.IntegerField(required=False)
-    downloads_sleep_interval = forms.IntegerField(required=False)
-    downloads_format = forms.CharField(required=False)
-    downloads_add_metadata = forms.ChoiceField(
-        widget=forms.Select, choices=METADATA_CHOICES, required=False
-    )
-    downloads_add_thumbnail = forms.ChoiceField(
-        widget=forms.Select, choices=THUMBNAIL_CHOICES, required=False
-    )
-
-
-class VideoSearchForm(forms.Form):
-    """search videos form"""
-
-    searchInput = forms.CharField(
-        label="Search your videos",
-        widget=forms.TextInput(attrs={"autocomplete": "off"}),
-    )
-
-
-class ChannelSearchForm(forms.Form):
-    """search for channels"""
-
-    searchInput = forms.CharField(
-        label="",
-        widget=forms.TextInput(
-            attrs={
-                "oninput": "searchChannels(this.value)",
-                "autocomplete": "off",
-                "list": "resultBox",
-            }
-        ),
-    )
-
-
-class AddToQueueForm(forms.Form):
-    """text area form to add to downloads"""
-
-    vid_url = forms.CharField(
-        label=False,
-        widget=forms.Textarea(
-            attrs={
-                "rows": 4,
-                "placeholder": "Enter Video Urls or IDs here...",
-            }
-        ),
-    )
-
-
-class SubscribeToChannelForm(forms.Form):
-    """text area form to subscribe to multiple channels"""
-
-    subscribe = forms.CharField(
-        label=False,
-        widget=forms.Textarea(
-            attrs={
-                "rows": 3,
-                "placeholder": "Input channel ID, URL or Video of a channel",
-            }
-        ),
-    )

tubearchivist/home/src/config.py

@@ -1,152 +0,0 @@
-"""
-Functionality:
-- read and write config
-- load config variables into redis
-- needs to be a separate module to avoid circular import
-"""
-
-import json
-import os
-
-from home.src.helper import RedisArchivist
-
-
-class AppConfig:
-    """handle user settings and application variables"""
-
-    def __init__(self, user_id=False):
-        self.user_id = user_id
-        self.config = self.get_config()
-        self.colors = self.get_colors()
-
-    def get_config(self):
-        """get config from default file or redis if changed"""
-        config = self.get_config_redis()
-        if not config:
-            config = self.get_config_file()
-
-        if self.user_id:
-            key = f"{self.user_id}:page_size"
-            page_size = RedisArchivist().get_message(key)["status"]
-            if page_size:
-                config["archive"]["page_size"] = page_size
-
-        config["application"].update(self.get_config_env())
-        return config
-
-    def get_config_file(self):
-        """read the defaults from config.json"""
-        with open("home/config.json", "r", encoding="utf-8") as f:
-            config_str = f.read()
-            config_file = json.loads(config_str)
-
-        config_file["application"].update(self.get_config_env())
-
-        return config_file
-
-    @staticmethod
-    def get_config_env():
-        """read environment application variables"""
-        host_uid_env = os.environ.get("HOST_UID")
-        if host_uid_env:
-            host_uid = int(host_uid_env)
-        else:
-            host_uid = False
-
-        host_gid_env = os.environ.get("HOST_GID")
-        if host_gid_env:
-            host_gid = int(host_gid_env)
-        else:
-            host_gid = False
-
-        es_pass = os.environ.get("ELASTIC_PASSWORD")
-        es_user = os.environ.get("ELASTIC_USER", default="elastic")
-
-        application = {
-            "REDIS_HOST": os.environ.get("REDIS_HOST"),
-            "es_url": os.environ.get("ES_URL"),
-            "es_auth": (es_user, es_pass),
-            "HOST_UID": host_uid,
-            "HOST_GID": host_gid,
-        }
-
-        return application
-
-    @staticmethod
-    def get_config_redis():
-        """read config json set from redis to overwrite defaults"""
-        config = RedisArchivist().get_message("config")
-        if not list(config.values())[0]:
-            return False
-
-        return config
-
-    def update_config(self, form_post):
-        """update config values from settings form"""
-        config = self.config
-        for key, value in form_post.items():
-            to_write = value[0]
-            if len(to_write):
-                if to_write == "0":
-                    to_write = False
-                elif to_write == "1":
-                    to_write = True
-                elif to_write.isdigit():
-                    to_write = int(to_write)
-
-                config_dict, config_value = key.split("_", maxsplit=1)
-                config[config_dict][config_value] = to_write
-
-        RedisArchivist().set_message("config", config, expire=False)
-
-    @staticmethod
-    def set_user_config(form_post, user_id):
-        """set values in redis for user settings"""
-        for key, value in form_post.items():
-            to_write = value[0]
-            if len(to_write):
-                if to_write.isdigit():
-                    to_write = int(to_write)
-                message = {"status": to_write}
-                redis_key = f"{user_id}:{key}"
-                RedisArchivist().set_message(redis_key, message, expire=False)
-
-    def get_colors(self):
-        """overwrite config if user has set custom values"""
-        colors = False
-        if self.user_id:
-            col_dict = RedisArchivist().get_message(f"{self.user_id}:colors")
-            colors = col_dict["status"]
-
-        if not colors:
-            colors = self.config["application"]["colors"]
-
-        self.config["application"]["colors"] = colors
-        return colors
-
-    def load_new_defaults(self):
-        """check config.json for missing defaults"""
-        default_config = self.get_config_file()
-        redis_config = self.get_config_redis()
-
-        # check for customizations
-        if not redis_config:
-            return
-
-        needs_update = False
-
-        for key, value in default_config.items():
-            # missing whole main key
-            if key not in redis_config:
-                redis_config.update({key: value})
-                needs_update = True
-                continue
-
-            # missing nested values
-            for sub_key, sub_value in value.items():
-                if sub_key not in redis_config[key].keys():
-                    redis_config[key].update({sub_key: sub_value})
-                    needs_update = True
-
-        if needs_update:
-            RedisArchivist().set_message("config", redis_config, expire=False)

tubearchivist/home/src/download.py

@@ -1,650 +0,0 @@
-"""
-Functionality:
-- handele the download queue
-- manage subscriptions to channels
-- downloading videos
-"""
-
-import json
-import os
-import shutil
-from datetime import datetime
-from time import sleep
-
-import requests
-import yt_dlp as youtube_dl
-from home.src.config import AppConfig
-from home.src.helper import (
-    DurationConverter,
-    RedisArchivist,
-    RedisQueue,
-    clean_string,
-    ignore_filelist,
-)
-from home.src.index import YoutubeChannel, index_new_video
-
-
-class PendingList:
-    """manage the pending videos list"""
-
-    CONFIG = AppConfig().config
-    ES_URL = CONFIG["application"]["es_url"]
-    ES_AUTH = CONFIG["application"]["es_auth"]
-    VIDEOS = CONFIG["application"]["videos"]
-
-    def __init__(self):
-        self.all_channel_ids = False
-        self.all_downloaded = False
-
-    @staticmethod
-    def parse_url_list(youtube_ids):
-        """extract youtube ids from list"""
-        missing_videos = []
-        for entry in youtube_ids:
-            # notify
-            mess_dict = {
-                "status": "pending",
-                "level": "info",
-                "title": "Adding to download queue.",
-                "message": "Extracting lists",
-            }
-            RedisArchivist().set_message("progress:download", mess_dict)
-            # extract
-            url = entry["url"]
-            url_type = entry["type"]
-            if url_type == "video":
-                missing_videos.append(url)
-            elif url_type == "channel":
-                video_results = ChannelSubscription().get_last_youtube_videos(
-                    url, limit=False
-                )
-                youtube_ids = [i[0] for i in video_results]
-                missing_videos = missing_videos + youtube_ids
-            elif url_type == "playlist":
-                video_results = playlist_extractor(url)
-                youtube_ids = [i[0] for i in video_results]
-                missing_videos = missing_videos + youtube_ids
-
-        return missing_videos
-
-    def add_to_pending(self, missing_videos):
-        """build the bulk json data from pending"""
-        # check if channel is indexed
-        channel_handler = ChannelSubscription()
-        all_indexed = channel_handler.get_channels(subscribed_only=False)
-        self.all_channel_ids = [i["channel_id"] for i in all_indexed]
-        # check if already there
-        self.all_downloaded = self.get_all_downloaded()
-
-        bulk_list, all_videos_added = self.build_bulk(missing_videos)
-        # add last newline
-        bulk_list.append("\n")
-        query_str = "\n".join(bulk_list)
-        headers = {"Content-type": "application/x-ndjson"}
-        url = self.ES_URL + "/_bulk"
-        request = requests.post(
-            url, data=query_str, headers=headers, auth=self.ES_AUTH
-        )
-        if not request.ok:
-            print(request)
-
-        return all_videos_added
-
-    def build_bulk(self, missing_videos):
-        """build the bulk lists"""
-        bulk_list = []
-        all_videos_added = []
-        counter = 1
-        for youtube_id in missing_videos:
-            # check if already downloaded
-            if youtube_id in self.all_downloaded:
-                continue
-
-            video = self.get_youtube_details(youtube_id)
-            # skip on download error
-            if not video:
-                continue
-
-            channel_indexed = video["channel_id"] in self.all_channel_ids
-            video["channel_indexed"] = channel_indexed
-            thumb_url = video["vid_thumb_url"]
-            video["status"] = "pending"
-            action = {"create": {"_id": youtube_id, "_index": "ta_download"}}
-            bulk_list.append(json.dumps(action))
-            bulk_list.append(json.dumps(video))
-            all_videos_added.append((youtube_id, thumb_url))
-            # notify
-            progress = f"{counter}/{len(missing_videos)}"
-            mess_dict = {
-                "status": "pending",
-                "level": "info",
-                "title": "Adding new videos to download queue.",
-                "message": "Progress: " + progress,
-            }
-            RedisArchivist().set_message("progress:download", mess_dict)
-            if counter % 25 == 0:
-                print("adding to queue progress: " + progress)
-            counter = counter + 1
-
-        return bulk_list, all_videos_added
-
-    @staticmethod
-    def get_youtube_details(youtube_id):
-        """get details from youtubedl for single pending video"""
-        obs = {
-            "default_search": "ytsearch",
-            "quiet": True,
-            "skip_download": True,
-        }
-        try:
-            vid = youtube_dl.YoutubeDL(obs).extract_info(youtube_id)
-        except youtube_dl.utils.DownloadError:
-            print("failed to extract info for: " + youtube_id)
-            return False
-        # stop if video is streaming live now
-        if vid["is_live"]:
-            return False
-        # parse response
-        seconds = vid["duration"]
-        duration_str = DurationConverter.get_str(seconds)
-        if duration_str == "NA":
-            print(f"skip extracting duration for: {youtube_id}")
-        upload_date = vid["upload_date"]
-        upload_dt = datetime.strptime(upload_date, "%Y%m%d")
-        published = upload_dt.strftime("%Y-%m-%d")
-        # build dict
-        youtube_details = {
-            "youtube_id": youtube_id,
-            "channel_name": vid["channel"],
-            "vid_thumb_url": vid["thumbnail"],
-            "title": vid["title"],
-            "channel_id": vid["channel_id"],
-            "duration": duration_str,
-            "published": published,
-            "timestamp": int(datetime.now().strftime("%s")),
-        }
-        return youtube_details
-
-    def get_all_pending(self):
-        """get a list of all pending videos in ta_download"""
-        headers = {"Content-type": "application/json"}
-        # get PIT ID
-        url = self.ES_URL + "/ta_download/_pit?keep_alive=1m"
-        response = requests.post(url, auth=self.ES_AUTH)
-        json_data = json.loads(response.text)
-        pit_id = json_data["id"]
-        # query
-        data = {
-            "size": 50,
-            "query": {"match_all": {}},
-            "pit": {"id": pit_id, "keep_alive": "1m"},
-            "sort": [{"timestamp": {"order": "asc"}}],
-        }
-        query_str = json.dumps(data)
-        url = self.ES_URL + "/_search"
-        all_pending = []
-        all_ignore = []
-        while True:
-            response = requests.get(
-                url, data=query_str, headers=headers, auth=self.ES_AUTH
-            )
-            json_data = json.loads(response.text)
-            all_hits = json_data["hits"]["hits"]
-            if all_hits:
-                for hit in all_hits:
-                    status = hit["_source"]["status"]
-                    if status == "pending":
-                        all_pending.append(hit["_source"])
-                    elif status == "ignore":
-                        all_ignore.append(hit["_source"])
-                    search_after = hit["sort"]
-                # update search_after with last hit data
-                data["search_after"] = search_after
-                query_str = json.dumps(data)
-            else:
-                break
-        # clean up PIT
-        query_str = json.dumps({"id": pit_id})
-        requests.delete(
-            self.ES_URL + "/_pit",
-            data=query_str,
-            headers=headers,
-            auth=self.ES_AUTH,
-        )
-        return all_pending, all_ignore
-
-    def get_all_indexed(self):
-        """get a list of all videos indexed"""
-        headers = {"Content-type": "application/json"}
-        # get PIT ID
-        url = self.ES_URL + "/ta_video/_pit?keep_alive=1m"
-        response = requests.post(url, auth=self.ES_AUTH)
-        json_data = json.loads(response.text)
-        pit_id = json_data["id"]
-        # query
-        data = {
-            "size": 500,
-            "query": {"match_all": {}},
-            "pit": {"id": pit_id, "keep_alive": "1m"},
-            "sort": [{"published": {"order": "desc"}}],
-        }
-        query_str = json.dumps(data)
-        url = self.ES_URL + "/_search"
-        all_indexed = []
-        while True:
-            response = requests.get(
-                url, data=query_str, headers=headers, auth=self.ES_AUTH
-            )
-            json_data = json.loads(response.text)
-            all_hits = json_data["hits"]["hits"]
-            if all_hits:
-                for hit in all_hits:
-                    all_indexed.append(hit)
-                    search_after = hit["sort"]
-                # update search_after with last hit data
-                data["search_after"] = search_after
-                query_str = json.dumps(data)
-            else:
-                break
-        # clean up PIT
-        query_str = json.dumps({"id": pit_id})
-        requests.delete(
-            self.ES_URL + "/_pit",
-            data=query_str,
-            headers=headers,
-            auth=self.ES_AUTH,
-        )
-        return all_indexed
-
-    def get_all_downloaded(self):
-        """get a list of all videos in archive"""
-        channel_folders = os.listdir(self.VIDEOS)
-        all_channel_folders = ignore_filelist(channel_folders)
-        all_downloaded = []
-        for channel_folder in all_channel_folders:
-            channel_path = os.path.join(self.VIDEOS, channel_folder)
-            videos = os.listdir(channel_path)
-            all_videos = ignore_filelist(videos)
-            youtube_vids = [i[9:20] for i in all_videos]
-            for youtube_id in youtube_vids:
-                all_downloaded.append(youtube_id)
-        return all_downloaded
-
-    def delete_from_pending(self, youtube_id):
-        """delete the youtube_id from ta_download"""
-        url = f"{self.ES_URL}/ta_download/_doc/{youtube_id}"
-        response = requests.delete(url, auth=self.ES_AUTH)
-        if not response.ok:
-            print(response.text)
-
-    def ignore_from_pending(self, ignore_list):
-        """build the bulk query string"""
-
-        stamp = int(datetime.now().strftime("%s"))
-        bulk_list = []
-
-        for youtube_id in ignore_list:
-            action = {"update": {"_id": youtube_id, "_index": "ta_download"}}
-            source = {"doc": {"status": "ignore", "timestamp": stamp}}
-            bulk_list.append(json.dumps(action))
-            bulk_list.append(json.dumps(source))
-
-        # add last newline
-        bulk_list.append("\n")
-        query_str = "\n".join(bulk_list)
-
-        headers = {"Content-type": "application/x-ndjson"}
-        url = self.ES_URL + "/_bulk"
-        request = requests.post(
-            url, data=query_str, headers=headers, auth=self.ES_AUTH
-        )
-        mess_dict = {
-            "status": "ignore",
-            "level": "info",
-            "title": "Added to ignore list",
-            "message": "",
-        }
-        RedisArchivist().set_message("progress:download", mess_dict)
-        if not request.ok:
-            print(request)
-
-
-class ChannelSubscription:
-    """manage the list of channels subscribed"""
-
-    def __init__(self):
-        config = AppConfig().config
-        self.es_url = config["application"]["es_url"]
-        self.es_auth = config["application"]["es_auth"]
-        self.channel_size = config["subscriptions"]["channel_size"]
-
-    def get_channels(self, subscribed_only=True):
-        """get a list of all channels subscribed to"""
-        headers = {"Content-type": "application/json"}
-        # get PIT ID
-        url = self.es_url + "/ta_channel/_pit?keep_alive=1m"
-        response = requests.post(url, auth=self.es_auth)
-        json_data = json.loads(response.text)
-        pit_id = json_data["id"]
-        # query
-        if subscribed_only:
-            data = {
-                "query": {"term": {"channel_subscribed": {"value": True}}},
-                "size": 50,
-                "pit": {"id": pit_id, "keep_alive": "1m"},
-                "sort": [{"channel_name.keyword": {"order": "asc"}}],
-            }
-        else:
-            data = {
-                "query": {"match_all": {}},
-                "size": 50,
-                "pit": {"id": pit_id, "keep_alive": "1m"},
-                "sort": [{"channel_name.keyword": {"order": "asc"}}],
-            }
-        query_str = json.dumps(data)
-        url = self.es_url + "/_search"
-        all_channels = []
-        while True:
-            response = requests.get(
-                url, data=query_str, headers=headers, auth=self.es_auth
-            )
-            json_data = json.loads(response.text)
-            all_hits = json_data["hits"]["hits"]
-            if all_hits:
-                for hit in all_hits:
-                    source = hit["_source"]
-                    search_after = hit["sort"]
-                    all_channels.append(source)
-                # update search_after with last hit data
-                data["search_after"] = search_after
-                query_str = json.dumps(data)
-            else:
-                break
-        # clean up PIT
-        query_str = json.dumps({"id": pit_id})
-        requests.delete(
-            self.es_url + "/_pit",
-            data=query_str,
-            headers=headers,
-            auth=self.es_auth,
-        )
-        return all_channels
-
-    def get_last_youtube_videos(self, channel_id, limit=True):
-        """get a list of last videos from channel"""
-        url = f"https://www.youtube.com/channel/{channel_id}/videos"
-        obs = {
-            "default_search": "ytsearch",
-            "quiet": True,
-            "skip_download": True,
-            "extract_flat": True,
-        }
-        if limit:
-            obs["playlistend"] = self.channel_size
-        chan = youtube_dl.YoutubeDL(obs).extract_info(url, download=False)
-        last_videos = [(i["id"], i["title"]) for i in chan["entries"]]
-        return last_videos
-
-    def find_missing(self):
-        """add missing videos from subscribed channels to pending"""
-        all_channels = self.get_channels()
-        pending_handler = PendingList()
-        all_pending, all_ignore = pending_handler.get_all_pending()
-        all_ids = [i["youtube_id"] for i in all_ignore + all_pending]
-        all_downloaded = pending_handler.get_all_downloaded()
-        to_ignore = all_ids + all_downloaded
-        missing_videos = []
-        counter = 1
-        for channel in all_channels:
-            channel_id = channel["channel_id"]
-            last_videos = self.get_last_youtube_videos(channel_id)
-            RedisArchivist().set_message(
-                "progress:download",
-                {
-                    "status": "rescan",
-                    "level": "info",
-                    "title": "Scanning channels: Looking for new videos.",
-                    "message": f"Progress: {counter}/{len(all_channels)}",
-                },
-            )
-            for video in last_videos:
-                youtube_id = video[0]
-                if youtube_id not in to_ignore:
-                    missing_videos.append(youtube_id)
-            counter = counter + 1
-
-        return missing_videos
-
-    def change_subscribe(self, channel_id, channel_subscribed):
-        """subscribe or unsubscribe from channel and update"""
-        if not isinstance(channel_subscribed, bool):
-            print("invalid status, should be bool")
-            return
-        headers = {"Content-type": "application/json"}
-        channel_handler = YoutubeChannel(channel_id)
-        channel_dict = channel_handler.channel_dict
-        channel_dict["channel_subscribed"] = channel_subscribed
-        if channel_subscribed:
-            # handle subscribe
-            url = self.es_url + "/ta_channel/_doc/" + channel_id
-            payload = json.dumps(channel_dict)
-            print(channel_dict)
-        else:
-            url = self.es_url + "/ta_channel/_update/" + channel_id
-            payload = json.dumps({"doc": channel_dict})
-        # update channel
-        request = requests.post(
-            url, data=payload, headers=headers, auth=self.es_auth
-        )
-        if not request.ok:
-            print(request.text)
-        # sync to videos
-        channel_handler.sync_to_videos()
-        if channel_handler.source == "scraped":
-            channel_handler.get_channel_art()
-
-
-def playlist_extractor(playlist_id):
-    """return youtube_ids from a playlist_id"""
-    url = "https://www.youtube.com/playlist?list=" + playlist_id
-    obs = {
-        "default_search": "ytsearch",
-        "quiet": True,
-        "ignoreerrors": True,
-        "skip_download": True,
-        "extract_flat": True,
-    }
-    playlist = youtube_dl.YoutubeDL(obs).extract_info(url, download=False)
-    playlist_vids = [(i["id"], i["title"]) for i in playlist["entries"]]
-    return playlist_vids
-
-
-class VideoDownloader:
-    """
-    handle the video download functionality
-    if not initiated with list, take from queue
-    """
-
-    def __init__(self, youtube_id_list=False):
-        self.youtube_id_list = youtube_id_list
-        self.config = AppConfig().config
-
-    def run_queue(self):
-        """setup download queue in redis loop until no more items"""
-        queue = RedisQueue("dl_queue")
-
-        limit_queue = self.config["downloads"]["limit_count"]
-        if limit_queue:
-            queue.trim(limit_queue - 1)
-
-        while True:
-            youtube_id = queue.get_next()
-            if not youtube_id:
-                break
-
-            try:
-                self.dl_single_vid(youtube_id)
-            except youtube_dl.utils.DownloadError:
-                print("failed to download " + youtube_id)
-                continue
-            vid_dict = index_new_video(youtube_id)
-            self.move_to_archive(vid_dict)
-            self.delete_from_pending(youtube_id)
-
-    @staticmethod
-    def add_pending():
-        """add pending videos to download queue"""
-        all_pending, _ = PendingList().get_all_pending()
-        to_add = [i["youtube_id"] for i in all_pending]
-        if not to_add:
-            # there is nothing pending
-            print("download queue is empty")
-            mess_dict = {
-                "status": "downloading",
-                "level": "error",
-                "title": "Download queue is empty",
-                "message": "",
-            }
-            RedisArchivist().set_message("progress:download", mess_dict)
-            return
-
-        queue = RedisQueue("dl_queue")
-        queue.add_list(to_add)
-
-    @staticmethod
-    def progress_hook(response):
-        """process the progress_hooks from youtube_dl"""
-        # title
-        filename = response["filename"][12:].replace("_", " ")
-        title = "Downloading: " + os.path.split(filename)[-1]
-        # message
-        try:
-            percent = response["_percent_str"]
-            size = response["_total_bytes_str"]
-            speed = response["_speed_str"]
-            eta = response["_eta_str"]
-            message = f"{percent} of {size} at {speed} - time left: {eta}"
-        except KeyError:
-            message = ""
-        mess_dict = {
-            "status": "downloading",
-            "level": "info",
-            "title": title,
-            "message": message,
-        }
-        RedisArchivist().set_message("progress:download", mess_dict)
-
-    def build_obs(self):
-        """build obs dictionary for yt-dlp"""
-        obs = {
-            "default_search": "ytsearch",
-            "merge_output_format": "mp4",
-            "restrictfilenames": True,
-            "outtmpl": (
-                self.config["application"]["cache_dir"]
-                + "/download/"
-                + self.config["application"]["file_template"]
-            ),
-            "progress_hooks": [self.progress_hook],
-            "noprogress": True,
-            "quiet": True,
-            "continuedl": True,
-            "retries": 3,
-            "writethumbnail": False,
-        }
-        if self.config["downloads"]["format"]:
-            obs["format"] = self.config["downloads"]["format"]
-        if self.config["downloads"]["limit_speed"]:
-            obs["ratelimit"] = self.config["downloads"]["limit_speed"] * 1024
-
-        throttle = self.config["downloads"]["throttledratelimit"]
-        if throttle:
-            obs["throttledratelimit"] = throttle * 1024
-
-        external = False
-        if external:
-            obs["external_downloader"] = "aria2c"
-
-        postprocessors = []
-
-        if self.config["downloads"]["add_metadata"]:
-            postprocessors.append(
-                {
-                    "key": "FFmpegMetadata",
-                    "add_chapters": True,
-                    "add_metadata": True,
-                }
-            )
-
-        if self.config["downloads"]["add_thumbnail"]:
-            postprocessors.append(
-                {
-                    "key": "EmbedThumbnail",
-                    "already_have_thumbnail": True,
-                }
-            )
-            obs["writethumbnail"] = True
-
-        obs["postprocessors"] = postprocessors
-
-        return obs
-
-    def dl_single_vid(self, youtube_id):
-        """download single video"""
-        dl_cache = self.config["application"]["cache_dir"] + "/download/"
-        obs = self.build_obs()
-
-        # check if already in cache to continue from there
-        all_cached = ignore_filelist(os.listdir(dl_cache))
-        for file_name in all_cached:
-            if youtube_id in file_name:
-                obs["outtmpl"] = os.path.join(dl_cache, file_name)
-        with youtube_dl.YoutubeDL(obs) as ydl:
-            try:
-                ydl.download([youtube_id])
-            except youtube_dl.utils.DownloadError:
-                print("retry failed download: " + youtube_id)
-                sleep(10)
-                ydl.download([youtube_id])
-
-        if obs["writethumbnail"]:
-            # webp files don't get cleaned up automatically
-            all_cached = ignore_filelist(os.listdir(dl_cache))
-            to_clean = [i for i in all_cached if not i.endswith(".mp4")]
-            for file_name in to_clean:
-                file_path = os.path.join(dl_cache, file_name)
-                os.remove(file_path)
-
-    def move_to_archive(self, vid_dict):
-        """move downloaded video from cache to archive"""
-        videos = self.config["application"]["videos"]
-        host_uid = self.config["application"]["HOST_UID"]
-        host_gid = self.config["application"]["HOST_GID"]
-        channel_name = clean_string(vid_dict["channel"]["channel_name"])
-        # make archive folder with correct permissions
-        new_folder = os.path.join(videos, channel_name)
-        if not os.path.exists(new_folder):
-            os.makedirs(new_folder)
-            if host_uid and host_gid:
-                os.chown(new_folder, host_uid, host_gid)
-        # find real filename
-        cache_dir = self.config["application"]["cache_dir"]
-        all_cached = ignore_filelist(os.listdir(cache_dir + "/download/"))
-        for file_str in all_cached:
-            if vid_dict["youtube_id"] in file_str:
-                old_file = file_str
-        old_file_path = os.path.join(cache_dir, "download", old_file)
-        new_file_path = os.path.join(videos, vid_dict["media_url"])
-        # move media file and fix permission
-        shutil.move(old_file_path, new_file_path)
-        if host_uid and host_gid:
-            os.chown(new_file_path, host_uid, host_gid)
-
-    def delete_from_pending(self, youtube_id):
-        """delete downloaded video from pending index if its there"""
-        es_url = self.config["application"]["es_url"]
-        es_auth = self.config["application"]["es_auth"]
-        url = f"{es_url}/ta_download/_doc/{youtube_id}"
-        response = requests.delete(url, auth=es_auth)
-        if not response.ok and not response.status_code == 404:
-            print(response.text)

tubearchivist/home/src/download/queue.py

@@ -0,0 +1,296 @@
+"""
+Functionality:
+- handle download queue
+- linked with ta_dowload index
+"""
+
+import json
+from datetime import datetime
+
+import yt_dlp
+from home.src.download.subscriptions import (
+    ChannelSubscription,
+    PlaylistSubscription,
+)
+from home.src.download.thumbnails import ThumbManager
+from home.src.download.yt_cookie import CookieHandler
+from home.src.es.connect import ElasticWrap, IndexPaginate
+from home.src.index.playlist import YoutubePlaylist
+from home.src.ta.config import AppConfig
+from home.src.ta.helper import DurationConverter
+from home.src.ta.ta_redis import RedisArchivist
+
+
+class PendingIndex:
+    """base class holding all export methods"""
+
+    def __init__(self):
+        self.all_pending = False
+        self.all_ignored = False
+        self.all_videos = False
+        self.all_channels = False
+        self.channel_overwrites = False
+        self.video_overwrites = False
+        self.to_skip = False
+
+    def get_download(self):
+        """get a list of all pending videos in ta_download"""
+        data = {
+            "query": {"match_all": {}},
+            "sort": [{"timestamp": {"order": "asc"}}],
+        }
+        all_results = IndexPaginate("ta_download", data).get_results()
+
+        self.all_pending = []
+        self.all_ignored = []
+        self.to_skip = []
+
+        for result in all_results:
+            self.to_skip.append(result["youtube_id"])
+            if result["status"] == "pending":
+                self.all_pending.append(result)
+            elif result["status"] == "ignore":
+                self.all_ignored.append(result)
+
+    def get_indexed(self):
+        """get a list of all videos indexed"""
+        data = {
+            "query": {"match_all": {}},
+            "sort": [{"published": {"order": "desc"}}],
+        }
+        self.all_videos = IndexPaginate("ta_video", data).get_results()
+        for video in self.all_videos:
+            self.to_skip.append(video["youtube_id"])
+
+    def get_channels(self):
+        """get a list of all channels indexed"""
+        self.all_channels = []
+        self.channel_overwrites = {}
+        data = {
+            "query": {"match_all": {}},
+            "sort": [{"channel_id": {"order": "asc"}}],
+        }
+        channels = IndexPaginate("ta_channel", data).get_results()
+
+        for channel in channels:
+            channel_id = channel["channel_id"]
+            self.all_channels.append(channel_id)
+            if channel.get("channel_overwrites"):
+                self.channel_overwrites.update(
+                    {channel_id: channel.get("channel_overwrites")}
+                )
+
+        self._map_overwrites()
+
+    def _map_overwrites(self):
+        """map video ids to channel ids overwrites"""
+        self.video_overwrites = {}
+        for video in self.all_pending:
+            video_id = video["youtube_id"]
+            channel_id = video["channel_id"]
+            overwrites = self.channel_overwrites.get(channel_id, False)
+            if overwrites:
+                self.video_overwrites.update({video_id: overwrites})
+
+
+class PendingInteract:
+    """interact with items in download queue"""
+
+    def __init__(self, video_id=False, status=False):
+        self.video_id = video_id
+        self.status = status
+
+    def delete_item(self):
+        """delete single item from pending"""
+        path = f"ta_download/_doc/{self.video_id}"
+        _, _ = ElasticWrap(path).delete(refresh=True)
+
+    def delete_by_status(self):
+        """delete all matching item by status"""
+        data = {"query": {"term": {"status": {"value": self.status}}}}
+        path = "ta_download/_delete_by_query"
+        _, _ = ElasticWrap(path).post(data=data)
+
+    def update_status(self):
+        """update status field of pending item"""
+        data = {"doc": {"status": self.status}}
+        path = f"ta_download/_update/{self.video_id}"
+        _, _ = ElasticWrap(path).post(data=data)
+
+
+class PendingList(PendingIndex):
+    """manage the pending videos list"""
+
+    yt_obs = {
+        "default_search": "ytsearch",
+        "quiet": True,
+        "check_formats": "selected",
+        "noplaylist": True,
+        "writethumbnail": True,
+        "simulate": True,
+        "socket_timeout": 3,
+    }
+
+    def __init__(self, youtube_ids=False):
+        super().__init__()
+        self.process_config()
+        self.youtube_ids = youtube_ids
+        self.to_skip = False
+        self.missing_videos = False
+
+    def process_config(self):
+        """add user config to yt_obs"""
+        config = AppConfig().config
+        if config["downloads"]["cookie_import"]:
+            cookie_path = CookieHandler().use()
+            self.yt_obs.update({"cookiefile": cookie_path})
+
+    def close_config(self):
+        """remove config after task finished"""
+        config = AppConfig().config
+        if config["downloads"]["cookie_import"]:
+            CookieHandler().hide()
+            try:
+                del self.yt_obs["cookiefile"]
+            except KeyError:
+                pass
+
+    def parse_url_list(self):
+        """extract youtube ids from list"""
+        self.missing_videos = []
+        self.get_download()
+        self.get_indexed()
+        for entry in self.youtube_ids:
+            # notify
+            mess_dict = {
+                "status": "message:add",
+                "level": "info",
+                "title": "Adding to download queue.",
+                "message": "Extracting lists",
+            }
+            RedisArchivist().set_message("message:add", mess_dict)
+            self._process_entry(entry)
+
+    def _process_entry(self, entry):
+        """process single entry from url list"""
+        if entry["type"] == "video":
+            self._add_video(entry["url"])
+        elif entry["type"] == "channel":
+            self._parse_channel(entry["url"])
+        elif entry["type"] == "playlist":
+            self._parse_playlist(entry["url"])
+            new_thumbs = PlaylistSubscription().process_url_str(
+                [entry], subscribed=False
+            )
+            ThumbManager().download_playlist(new_thumbs)
+        else:
+            raise ValueError(f"invalid url_type: {entry}")
+
+    def _add_video(self, url):
+        """add video to list"""
+        if url not in self.missing_videos and url not in self.to_skip:
+            self.missing_videos.append(url)
+
+    def _parse_channel(self, url):
+        """add all videos of channel to list"""
+        video_results = ChannelSubscription().get_last_youtube_videos(
+            url, limit=False
+        )
+        youtube_ids = [i[0] for i in video_results]
+        for video_id in youtube_ids:
+            self._add_video(video_id)
+
+    def _parse_playlist(self, url):
+        """add all videos of playlist to list"""
+        playlist = YoutubePlaylist(url)
+        playlist.build_json()
+        video_results = playlist.json_data.get("playlist_entries")
+        youtube_ids = [i["youtube_id"] for i in video_results]
+        for video_id in youtube_ids:
+            self._add_video(video_id)
+
+    def add_to_pending(self, status="pending"):
+        """add missing videos to pending list"""
+        self.get_channels()
+        bulk_list = []
+
+        thumb_handler = ThumbManager()
+        for idx, youtube_id in enumerate(self.missing_videos):
+            video_details = self.get_youtube_details(youtube_id)
+            if not video_details:
+                continue
+
+            video_details["status"] = status
+            action = {"create": {"_id": youtube_id, "_index": "ta_download"}}
+            bulk_list.append(json.dumps(action))
+            bulk_list.append(json.dumps(video_details))
+
+            thumb_needed = [(youtube_id, video_details["vid_thumb_url"])]
+            thumb_handler.download_vid(thumb_needed)
+            self._notify_add(idx)
+
+        if bulk_list:
+            # add last newline
+            bulk_list.append("\n")
+            query_str = "\n".join(bulk_list)
+            _, _ = ElasticWrap("_bulk").post(query_str, ndjson=True)
+
+        self.close_config()
+
+    def _notify_add(self, idx):
+        """send notification for adding videos to download queue"""
+        progress = f"{idx + 1}/{len(self.missing_videos)}"
+        mess_dict = {
+            "status": "message:add",
+            "level": "info",
+            "title": "Adding new videos to download queue.",
+            "message": "Progress: " + progress,
+        }
+        if idx + 1 == len(self.missing_videos):
+            RedisArchivist().set_message("message:add", mess_dict, expire=4)
+        else:
+            RedisArchivist().set_message("message:add", mess_dict)
+
+        if idx + 1 % 25 == 0:
+            print("adding to queue progress: " + progress)
+
+    def get_youtube_details(self, youtube_id):
+        """get details from youtubedl for single pending video"""
+        try:
+            vid = yt_dlp.YoutubeDL(self.yt_obs).extract_info(youtube_id)
+        except yt_dlp.utils.DownloadError:
+            print(f"{youtube_id}: failed to extract info")
+            return False
+        if vid.get("id") != youtube_id:
+            # skip premium videos with different id
+            print(f"{youtube_id}: skipping premium video, id not matching")
+            return False
+        # stop if video is streaming live now
+        if vid["is_live"]:
+            return False
+
+        return self._parse_youtube_details(vid)
+
+    def _parse_youtube_details(self, vid):
+        """parse response"""
+        vid_id = vid.get("id")
+        duration_str = DurationConverter.get_str(vid["duration"])
+        if duration_str == "NA":
+            print(f"skip extracting duration for: {vid_id}")
+        published = datetime.strptime(vid["upload_date"], "%Y%m%d").strftime(
+            "%Y-%m-%d"
+        )
+
+        # build dict
+        youtube_details = {
+            "youtube_id": vid_id,
+            "channel_name": vid["channel"],
+            "vid_thumb_url": vid["thumbnail"],
+            "title": vid["title"],
+            "channel_id": vid["channel_id"],
+            "channel_indexed": vid["channel_id"] in self.all_channels,
+            "duration": duration_str,
+            "published": published,
+            "timestamp": int(datetime.now().strftime("%s")),
+        }
+        return youtube_details

tubearchivist/home/src/download/subscriptions.py

@@ -0,0 +1,223 @@
+"""
+Functionality:
+- handle channel subscriptions
+- handle playlist subscriptions
+"""
+
+import yt_dlp
+from home.src.download import queue  # partial import
+from home.src.es.connect import IndexPaginate
+from home.src.index.channel import YoutubeChannel
+from home.src.index.playlist import YoutubePlaylist
+from home.src.ta.config import AppConfig
+from home.src.ta.ta_redis import RedisArchivist
+
+
+class ChannelSubscription:
+    """manage the list of channels subscribed"""
+
+    def __init__(self):
+        config = AppConfig().config
+        self.es_url = config["application"]["es_url"]
+        self.es_auth = config["application"]["es_auth"]
+        self.channel_size = config["subscriptions"]["channel_size"]
+
+    @staticmethod
+    def get_channels(subscribed_only=True):
+        """get a list of all channels subscribed to"""
+        data = {
+            "sort": [{"channel_name.keyword": {"order": "asc"}}],
+        }
+        if subscribed_only:
+            data["query"] = {"term": {"channel_subscribed": {"value": True}}}
+        else:
+            data["query"] = {"match_all": {}}
+
+        all_channels = IndexPaginate("ta_channel", data).get_results()
+
+        return all_channels
+
+    def get_last_youtube_videos(self, channel_id, limit=True):
+        """get a list of last videos from channel"""
+        url = f"https://www.youtube.com/channel/{channel_id}/videos"
+        obs = {
+            "default_search": "ytsearch",
+            "quiet": True,
+            "skip_download": True,
+            "extract_flat": True,
+        }
+        if limit:
+            obs["playlistend"] = self.channel_size
+
+        try:
+            chan = yt_dlp.YoutubeDL(obs).extract_info(url, download=False)
+        except yt_dlp.utils.DownloadError:
+            print(f"{channel_id}: failed to extract videos, skipping.")
+            return False
+
+        last_videos = [(i["id"], i["title"]) for i in chan["entries"]]
+        return last_videos
+
+    def find_missing(self):
+        """add missing videos from subscribed channels to pending"""
+        all_channels = self.get_channels()
+        pending = queue.PendingList()
+        pending.get_download()
+        pending.get_indexed()
+
+        missing_videos = []
+
+        for idx, channel in enumerate(all_channels):
+            channel_id = channel["channel_id"]
+            last_videos = self.get_last_youtube_videos(channel_id)
+
+            if last_videos:
+                for video in last_videos:
+                    if video[0] not in pending.to_skip:
+                        missing_videos.append(video[0])
+            # notify
+            message = {
+                "status": "message:rescan",
+                "level": "info",
+                "title": "Scanning channels: Looking for new videos.",
+                "message": f"Progress: {idx + 1}/{len(all_channels)}",
+            }
+            if idx + 1 == len(all_channels):
+                RedisArchivist().set_message(
+                    "message:rescan", message=message, expire=4
+                )
+            else:
+                RedisArchivist().set_message("message:rescan", message=message)
+
+        return missing_videos
+
+    @staticmethod
+    def change_subscribe(channel_id, channel_subscribed):
+        """subscribe or unsubscribe from channel and update"""
+        channel = YoutubeChannel(channel_id)
+        channel.build_json()
+        channel.json_data["channel_subscribed"] = channel_subscribed
+        channel.upload_to_es()
+        channel.sync_to_videos()
+
+
+class PlaylistSubscription:
+    """manage the playlist download functionality"""
+
+    def __init__(self):
+        self.config = AppConfig().config
+
+    @staticmethod
+    def get_playlists(subscribed_only=True):
+        """get a list of all active playlists"""
+        data = {
+            "sort": [{"playlist_channel.keyword": {"order": "desc"}}],
+        }
+        data["query"] = {
+            "bool": {"must": [{"term": {"playlist_active": {"value": True}}}]}
+        }
+        if subscribed_only:
+            data["query"]["bool"]["must"].append(
+                {"term": {"playlist_subscribed": {"value": True}}}
+            )
+
+        all_playlists = IndexPaginate("ta_playlist", data).get_results()
+
+        return all_playlists
+
+    def process_url_str(self, new_playlists, subscribed=True):
+        """process playlist subscribe form url_str"""
+        data = {
+            "query": {"match_all": {}},
+            "sort": [{"published": {"order": "desc"}}],
+        }
+        all_indexed = IndexPaginate("ta_video", data).get_results()
+        all_youtube_ids = [i["youtube_id"] for i in all_indexed]
+
+        new_thumbs = []
+        for idx, playlist in enumerate(new_playlists):
+            url_type = playlist["type"]
+            playlist_id = playlist["url"]
+            if not url_type == "playlist":
+                print(f"{playlist_id} not a playlist, skipping...")
+                continue
+
+            playlist_h = YoutubePlaylist(playlist_id)
+            playlist_h.all_youtube_ids = all_youtube_ids
+            playlist_h.build_json()
+            playlist_h.json_data["playlist_subscribed"] = subscribed
+            playlist_h.upload_to_es()
+            playlist_h.add_vids_to_playlist()
+            self.channel_validate(playlist_h.json_data["playlist_channel_id"])
+            thumb = playlist_h.json_data["playlist_thumbnail"]
+            new_thumbs.append((playlist_id, thumb))
+            # notify
+            message = {
+                "status": "message:subplaylist",
+                "level": "info",
+                "title": "Subscribing to Playlists",
+                "message": f"Processing {idx + 1} of {len(new_playlists)}",
+            }
+            RedisArchivist().set_message(
+                "message:subplaylist", message=message
+            )
+
+        return new_thumbs
+
+    @staticmethod
+    def channel_validate(channel_id):
+        """make sure channel of playlist is there"""
+        channel = YoutubeChannel(channel_id)
+        channel.build_json(upload=True)
+
+    @staticmethod
+    def change_subscribe(playlist_id, subscribe_status):
+        """change the subscribe status of a playlist"""
+        playlist = YoutubePlaylist(playlist_id)
+        playlist.build_json()
+        playlist.json_data["playlist_subscribed"] = subscribe_status
+        playlist.upload_to_es()
+
+    @staticmethod
+    def get_to_ignore():
+        """get all youtube_ids already downloaded or ignored"""
+        pending = queue.PendingList()
+        pending.get_download()
+        pending.get_indexed()
+
+        return pending.to_skip
+
+    def find_missing(self):
+        """find videos in subscribed playlists not downloaded yet"""
+        all_playlists = [i["playlist_id"] for i in self.get_playlists()]
+        to_ignore = self.get_to_ignore()
+
+        missing_videos = []
+        for idx, playlist_id in enumerate(all_playlists):
+            size_limit = self.config["subscriptions"]["channel_size"]
+            playlist = YoutubePlaylist(playlist_id)
+            playlist.update_playlist()
+            if not playlist:
+                playlist.deactivate()
+                continue
+
+            playlist_entries = playlist.json_data["playlist_entries"]
+            if size_limit:
+                del playlist_entries[size_limit:]
+
+            all_missing = [i for i in playlist_entries if not i["downloaded"]]
+
+            message = {
+                "status": "message:rescan",
+                "level": "info",
+                "title": "Scanning playlists: Looking for new videos.",
+                "message": f"Progress: {idx + 1}/{len(all_playlists)}",
+            }
+            RedisArchivist().set_message("message:rescan", message=message)
+
+            for video in all_missing:
+                youtube_id = video["youtube_id"]
+                if youtube_id not in to_ignore:
+                    missing_videos.append(youtube_id)
+
+        return missing_videos

tubearchivist/home/src/download/thumbnails.py

@@ -1,18 +1,23 @@
 """
 functionality:
 - handle download and caching for thumbnails
+- check for missing thumbnails
 """
 
+import base64
 import os
 from collections import Counter
+from io import BytesIO
 from time import sleep
 
-import home.src.download as download
 import requests
-from home.src.config import AppConfig
-from home.src.helper import RedisArchivist, ignore_filelist
+from home.src.download import queue  # partial import
+from home.src.download import subscriptions  # partial import
+from home.src.ta.config import AppConfig
+from home.src.ta.helper import ignore_filelist
+from home.src.ta.ta_redis import RedisArchivist
 from mutagen.mp4 import MP4, MP4Cover
-from PIL import Image
+from PIL import Image, ImageFilter
 
 
 class ThumbManager:
@@ -23,6 +28,7 @@ class ThumbManager:
     CACHE_DIR = CONFIG["application"]["cache_dir"]
     VIDEO_DIR = os.path.join(CACHE_DIR, "videos")
     CHANNEL_DIR = os.path.join(CACHE_DIR, "channels")
+    PLAYLIST_DIR = os.path.join(CACHE_DIR, "playlists")
 
     def get_all_thumbs(self):
         """get all video artwork already downloaded"""
@@ -54,20 +60,22 @@ class ThumbManager:
     def get_needed_thumbs(self, missing_only=False):
         """get a list of all missing thumbnails"""
         all_thumbs = self.get_all_thumbs()
-        all_indexed = download.PendingList().get_all_indexed()
-        all_in_queue, all_ignored = download.PendingList().get_all_pending()
+
+        pending = queue.PendingList()
+        pending.get_download()
+        pending.get_indexed()
 
         needed_thumbs = []
-        for video in all_indexed:
-            youtube_id = video["_source"]["youtube_id"]
-            thumb_url = video["_source"]["vid_thumb_url"]
+        for video in pending.all_videos:
+            youtube_id = video["youtube_id"]
+            thumb_url = video["vid_thumb_url"]
             if missing_only:
                 if youtube_id + ".jpg" not in all_thumbs:
                     needed_thumbs.append((youtube_id, thumb_url))
             else:
                 needed_thumbs.append((youtube_id, thumb_url))
 
-        for video in all_in_queue + all_ignored:
+        for video in pending.all_pending + pending.all_ignored:
             youtube_id = video["youtube_id"]
             thumb_url = video["vid_thumb_url"]
             if missing_only:
@@ -83,9 +91,8 @@ class ThumbManager:
         all_channel_art = os.listdir(self.CHANNEL_DIR)
         files = [i[0:24] for i in all_channel_art]
         cached_channel_ids = [k for (k, v) in Counter(files).items() if v > 1]
-        channels = download.ChannelSubscription().get_channels(
-            subscribed_only=False
-        )
+        channel_sub = subscriptions.ChannelSubscription()
+        channels = channel_sub.get_channels(subscribed_only=False)
 
         missing_channels = []
         for channel in channels:
@@ -99,6 +106,22 @@ class ThumbManager:
 
         return missing_channels
 
+    def get_missing_playlists(self):
+        """get all missing playlist artwork"""
+        all_downloaded = ignore_filelist(os.listdir(self.PLAYLIST_DIR))
+        all_ids_downloaded = [i.replace(".jpg", "") for i in all_downloaded]
+        playlist_sub = subscriptions.PlaylistSubscription()
+        playlists = playlist_sub.get_playlists(subscribed_only=False)
+
+        missing_playlists = []
+        for playlist in playlists:
+            playlist_id = playlist["playlist_id"]
+            if playlist_id not in all_ids_downloaded:
+                playlist_thumb = playlist["playlist_thumbnail"]
+                missing_playlists.append((playlist_id, playlist_thumb))
+
+        return missing_playlists
+
     def get_raw_img(self, img_url, thumb_type):
         """get raw image from youtube and handle 404"""
         try:
@@ -143,8 +166,7 @@ class ThumbManager:
     def download_vid(self, missing_thumbs, notify=True):
         """download all missing thumbnails from list"""
         print(f"downloading {len(missing_thumbs)} thumbnails")
-        counter = 1
-        for youtube_id, thumb_url in missing_thumbs:
+        for idx, (youtube_id, thumb_url) in enumerate(missing_thumbs):
             folder_path = os.path.join(self.VIDEO_DIR, youtube_id[0].lower())
             thumb_path = os.path.join(
                 self.CACHE_DIR, self.vid_thumb_path(youtube_id)
@@ -158,22 +180,25 @@ class ThumbManager:
                 new_height = width / 16 * 9
                 offset = (height - new_height) / 2
                 img_raw = img_raw.crop((0, offset, width, height - offset))
-
             img_raw.convert("RGB").save(thumb_path)
 
-            progress = f"{counter}/{len(missing_thumbs)}"
+            progress = f"{idx + 1}/{len(missing_thumbs)}"
             if notify:
                 mess_dict = {
-                    "status": "pending",
+                    "status": "message:add",
                     "level": "info",
-                    "title": "Downloading Thumbnails",
-                    "message": "Progress: " + progress,
+                    "title": "Processing Videos",
+                    "message": "Downloading Thumbnails, Progress: " + progress,
                 }
-                RedisArchivist().set_message("progress:download", mess_dict)
+                if idx + 1 == len(missing_thumbs):
+                    RedisArchivist().set_message(
+                        "message:add", mess_dict, expire=4
+                    )
+                else:
+                    RedisArchivist().set_message("message:add", mess_dict)
 
-            if counter % 25 == 0:
+            if idx + 1 % 25 == 0:
                 print("thumbnail progress: " + progress)
-            counter = counter + 1
 
     def download_chan(self, missing_channels):
         """download needed artwork for channels"""
@@ -194,12 +219,44 @@ class ThumbManager:
             img_raw.convert("RGB").save(banner_path)
 
             mess_dict = {
-                "status": "pending",
+                "status": "message:download",
                 "level": "info",
-                "title": "Adding to download queue.",
-                "message": "Downloading Channel Art...",
+                "title": "Processing Channels",
+                "message": "Downloading Channel Art.",
             }
-            RedisArchivist().set_message("progress:download", mess_dict)
+            RedisArchivist().set_message("message:download", mess_dict)
+
+    def download_playlist(self, missing_playlists):
+        """download needed artwork for playlists"""
+        print(f"downloading {len(missing_playlists)} playlist artwork")
+        for playlist in missing_playlists:
+            playlist_id, playlist_thumb_url = playlist
+            thumb_path = os.path.join(self.PLAYLIST_DIR, playlist_id + ".jpg")
+            img_raw = self.get_raw_img(playlist_thumb_url, "video")
+            img_raw.convert("RGB").save(thumb_path)
+
+            mess_dict = {
+                "status": "message:download",
+                "level": "info",
+                "title": "Processing Playlists",
+                "message": "Downloading Playlist Art.",
+            }
+            RedisArchivist().set_message("message:download", mess_dict)
+
+    def get_base64_blur(self, youtube_id):
+        """return base64 encoded placeholder"""
+        img_path = self.vid_thumb_path(youtube_id)
+        file_path = os.path.join(self.CACHE_DIR, img_path)
+        img_raw = Image.open(file_path)
+        img_raw.thumbnail((img_raw.width // 20, img_raw.height // 20))
+        img_blur = img_raw.filter(ImageFilter.BLUR)
+        buffer = BytesIO()
+        img_blur.save(buffer, format="JPEG")
+        img_data = buffer.getvalue()
+        img_base64 = base64.b64encode(img_data).decode()
+        data_url = f"data:image/jpg;base64,{img_base64}"
+
+        return data_url
 
     @staticmethod
     def vid_thumb_path(youtube_id):
@@ -238,13 +295,14 @@ class ThumbManager:
 
     def get_thumb_list(self):
         """get list of mediafiles and matching thumbnails"""
-        all_indexed = download.PendingList().get_all_indexed()
+        pending = queue.PendingList()
+        pending.get_download()
+        pending.get_indexed()
+
         video_list = []
-        for video in all_indexed:
-            youtube_id = video["_source"]["youtube_id"]
-            media_url = os.path.join(
-                self.MEDIA_DIR, video["_source"]["media_url"]
-            )
+        for video in pending.all_videos:
+            youtube_id = video["youtube_id"]
+            media_url = os.path.join(self.MEDIA_DIR, video["media_url"])
             thumb_path = os.path.join(
                 self.CACHE_DIR, self.vid_thumb_path(youtube_id)
             )
@@ -285,4 +343,6 @@ def validate_thumbnails():
     handler.download_vid(thumbs_to_download)
     missing_channels = handler.get_missing_channels()
     handler.download_chan(missing_channels)
+    missing_playlists = handler.get_missing_playlists()
+    handler.download_playlist(missing_playlists)
     handler.cleanup_downloaded()

tubearchivist/home/src/download/yt_cookie.py

@@ -0,0 +1,88 @@
+"""
+functionality:
+- import yt cookie from filesystem
+- make cookie available for yt-dlp
+"""
+
+import os
+
+import yt_dlp
+from home.src.ta.config import AppConfig
+from home.src.ta.ta_redis import RedisArchivist
+
+
+class CookieHandler:
+    """handle youtube cookie for yt-dlp"""
+
+    CONFIG = AppConfig().config
+    CACHE_PATH = CONFIG["application"]["cache_dir"]
+    COOKIE_FILE_NAME = "cookies.google.txt"
+    COOKIE_KEY = "cookie"
+    COOKIE_PATH = "cookie.txt"
+
+    def import_cookie(self):
+        """import cookie from file"""
+        import_path = os.path.join(
+            self.CACHE_PATH, "import", self.COOKIE_FILE_NAME
+        )
+        with open(import_path, encoding="utf-8") as cookie_file:
+            cookie = cookie_file.read()
+
+        RedisArchivist().set_message(self.COOKIE_KEY, cookie, expire=False)
+
+        os.remove(import_path)
+        print("cookie: import successfully")
+
+    def use(self):
+        """make cookie available in FS"""
+        cookie = RedisArchivist().get_message(self.COOKIE_KEY)
+        if isinstance(cookie, dict):
+            print("no cookie imported")
+            raise FileNotFoundError
+
+        if os.path.exists(self.COOKIE_PATH):
+            return self.COOKIE_PATH
+
+        with open(self.COOKIE_PATH, "w", encoding="utf-8") as cookie_file:
+            cookie_file.write(cookie)
+
+        print("cookie: made available")
+        return self.COOKIE_PATH
+
+    def hide(self):
+        """hide cookie file if not in use"""
+        try:
+            os.remove(self.COOKIE_PATH)
+        except FileNotFoundError:
+            print("cookie: not available")
+            return
+
+        print("cookie: hidden")
+
+    def revoke(self):
+        """revoke cookie"""
+        self.hide()
+        RedisArchivist().del_message(self.COOKIE_KEY)
+        print("cookie: revoked")
+
+    def validate(self):
+        """validate cookie using the liked videos playlist"""
+        try:
+            _ = self.use()
+        except FileNotFoundError:
+            return False
+
+        url = "https://www.youtube.com/playlist?list=LL"
+        yt_obs = {
+            "quiet": True,
+            "skip_download": True,
+            "extract_flat": True,
+            "cookiefile": self.COOKIE_PATH,
+        }
+        try:
+            response = yt_dlp.YoutubeDL(yt_obs).extract_info(url)
+        except yt_dlp.utils.DownloadError:
+            print("failed to validate cookie")
+            response = False
+
+        return bool(response)

tubearchivist/home/src/download/yt_dlp_handler.py

@@ -0,0 +1,411 @@
+"""
+functionality:
+- handle yt_dlp
+- build options and post processor
+- download video files
+- move to archive
+"""
+
+import os
+import shutil
+from datetime import datetime
+from time import sleep
+
+import yt_dlp
+from home.src.download.queue import PendingList
+from home.src.download.subscriptions import PlaylistSubscription
+from home.src.download.yt_cookie import CookieHandler
+from home.src.es.connect import ElasticWrap, IndexPaginate
+from home.src.index.channel import YoutubeChannel
+from home.src.index.playlist import YoutubePlaylist
+from home.src.index.video import YoutubeVideo, index_new_video
+from home.src.ta.config import AppConfig
+from home.src.ta.helper import clean_string, ignore_filelist
+from home.src.ta.ta_redis import RedisArchivist, RedisQueue
+
+
+class DownloadPostProcess:
+    """handle task to run after download queue finishes"""
+
+    def __init__(self, download):
+        self.download = download
+        self.now = int(datetime.now().strftime("%s"))
+        self.pending = False
+
+    def run(self):
+        """run all functions"""
+        self.pending = PendingList()
+        self.pending.get_download()
+        self.pending.get_channels()
+        self.pending.get_indexed()
+        self.auto_delete_all()
+        self.auto_delete_overwrites()
+        self.validate_playlists()
+        self.pending.close_config()
+
+    def auto_delete_all(self):
+        """handle auto delete"""
+        autodelete_days = self.download.config["downloads"]["autodelete_days"]
+        if not autodelete_days:
+            return
+
+        print(f"auto delete older than {autodelete_days} days")
+        now_lte = self.now - autodelete_days * 24 * 60 * 60
+        data = {
+            "query": {"range": {"player.watched_date": {"lte": now_lte}}},
+            "sort": [{"player.watched_date": {"order": "asc"}}],
+        }
+        self._auto_delete_watched(data)
+
+    def auto_delete_overwrites(self):
+        """handle per channel auto delete from overwrites"""
+        for channel_id, value in self.pending.channel_overwrites.items():
+            if "autodelete_days" in value:
+                autodelete_days = value.get("autodelete_days")
+                print(f"{channel_id}: delete older than {autodelete_days}d")
+                now_lte = self.now - autodelete_days * 24 * 60 * 60
+                must_list = [
+                    {"range": {"player.watched_date": {"lte": now_lte}}},
+                    {"term": {"channel.channel_id": {"value": channel_id}}},
+                ]
+                data = {
+                    "query": {"bool": {"must": must_list}},
+                    "sort": [{"player.watched_date": {"order": "desc"}}],
+                }
+                self._auto_delete_watched(data)
+
+    @staticmethod
+    def _auto_delete_watched(data):
+        """delete watched videos after x days"""
+        to_delete = IndexPaginate("ta_video", data).get_results()
+        if not to_delete:
+            return
+
+        for video in to_delete:
+            youtube_id = video["youtube_id"]
+            print(f"{youtube_id}: auto delete video")
+            YoutubeVideo(youtube_id).delete_media_file()
+
+        print("add deleted to ignore list")
+        vids = [{"type": "video", "url": i["youtube_id"]} for i in to_delete]
+        pending = PendingList(youtube_ids=vids)
+        pending.parse_url_list()
+        pending.add_to_pending(status="ignore")
+
+    def validate_playlists(self):
+        """look for playlist needing to update"""
+        for id_c, channel_id in enumerate(self.download.channels):
+            channel = YoutubeChannel(channel_id)
+            overwrites = self.pending.channel_overwrites.get(channel_id, False)
+            if overwrites and overwrites.get("index_playlists"):
+                # validate from remote
+                channel.index_channel_playlists()
+                continue
+
+            # validate from local
+            playlists = channel.get_indexed_playlists()
+            all_channel_playlist = [i["playlist_id"] for i in playlists]
+            self._validate_channel_playlist(all_channel_playlist, id_c)
+
+    def _validate_channel_playlist(self, all_channel_playlist, id_c):
+        """scan channel for playlist needing update"""
+        all_youtube_ids = [i["youtube_id"] for i in self.pending.all_videos]
+        for id_p, playlist_id in enumerate(all_channel_playlist):
+            playlist = YoutubePlaylist(playlist_id)
+            playlist.all_youtube_ids = all_youtube_ids
+            playlist.build_json(scrape=True)
+            if not playlist.json_data:
+                playlist.deactivate()
+
+            playlist.add_vids_to_playlist()
+            playlist.upload_to_es()
+            self._notify_playlist_progress(all_channel_playlist, id_c, id_p)
+
+    def _notify_playlist_progress(self, all_channel_playlist, id_c, id_p):
+        """notify to UI"""
+        title = (
+            "Processing playlists for channels: "
+            + f"{id_c + 1}/{len(self.download.channels)}"
+        )
+        message = f"Progress: {id_p + 1}/{len(all_channel_playlist)}"
+        mess_dict = {
+            "status": "message:download",
+            "level": "info",
+            "title": title,
+            "message": message,
+        }
+        if id_p + 1 == len(all_channel_playlist):
+            RedisArchivist().set_message(
+                "message:download", mess_dict, expire=4
+            )
+        else:
+            RedisArchivist().set_message("message:download", mess_dict)
+
+
+class VideoDownloader:
+    """
+    handle the video download functionality
+    if not initiated with list, take from queue
+    """
+
+    def __init__(self, youtube_id_list=False):
+        self.obs = False
+        self.video_overwrites = False
+        self.youtube_id_list = youtube_id_list
+        self.config = AppConfig().config
+        self._build_obs()
+        self.channels = set()
+
+    def run_queue(self):
+        """setup download queue in redis loop until no more items"""
+        pending = PendingList()
+        pending.get_download()
+        pending.get_channels()
+        self.video_overwrites = pending.video_overwrites
+
+        queue = RedisQueue()
+
+        limit_queue = self.config["downloads"]["limit_count"]
+        if limit_queue:
+            queue.trim(limit_queue - 1)
+
+        while True:
+            youtube_id = queue.get_next()
+            if not youtube_id:
+                break
+
+            try:
+                self._dl_single_vid(youtube_id)
+            except yt_dlp.utils.DownloadError:
+                print("failed to download " + youtube_id)
+                continue
+            vid_dict = index_new_video(
+                youtube_id, video_overwrites=self.video_overwrites
+            )
+            self.channels.add(vid_dict["channel"]["channel_id"])
+            mess_dict = {
+                "status": "message:download",
+                "level": "info",
+                "title": "Moving....",
+                "message": "Moving downloaded file to storage folder",
+            }
+            RedisArchivist().set_message("message:download", mess_dict, False)
+
+            self.move_to_archive(vid_dict)
+            mess_dict = {
+                "status": "message:download",
+                "level": "info",
+                "title": "Completed",
+                "message": "",
+            }
+            RedisArchivist().set_message("message:download", mess_dict, 10)
+            self._delete_from_pending(youtube_id)
+
+        # post processing
+        self._add_subscribed_channels()
+        DownloadPostProcess(self).run()
+
+    @staticmethod
+    def add_pending():
+        """add pending videos to download queue"""
+        mess_dict = {
+            "status": "message:download",
+            "level": "info",
+            "title": "Looking for videos to download",
+            "message": "Scanning your download queue.",
+        }
+        RedisArchivist().set_message("message:download", mess_dict)
+        pending = PendingList()
+        pending.get_download()
+        to_add = [i["youtube_id"] for i in pending.all_pending]
+        if not to_add:
+            # there is nothing pending
+            print("download queue is empty")
+            mess_dict = {
+                "status": "message:download",
+                "level": "error",
+                "title": "Download queue is empty",
+                "message": "Add some videos to the queue first.",
+            }
+            RedisArchivist().set_message("message:download", mess_dict)
+            return
+
+        RedisQueue().add_list(to_add)
+
+    @staticmethod
+    def _progress_hook(response):
+        """process the progress_hooks from yt_dlp"""
+        # title
+        path = os.path.split(response["filename"])[-1][12:]
+        filename = os.path.splitext(os.path.splitext(path)[0])[0]
+        filename_clean = filename.replace("_", " ")
+        title = "Downloading: " + filename_clean
+        # message
+        try:
+            percent = response["_percent_str"]
+            size = response["_total_bytes_str"]
+            speed = response["_speed_str"]
+            eta = response["_eta_str"]
+            message = f"{percent} of {size} at {speed} - time left: {eta}"
+        except KeyError:
+            message = "processing"
+        mess_dict = {
+            "status": "message:download",
+            "level": "info",
+            "title": title,
+            "message": message,
+        }
+        RedisArchivist().set_message("message:download", mess_dict)
+
+    def _build_obs(self):
+        """collection to build all obs passed to yt-dlp"""
+        self._build_obs_basic()
+        self._build_obs_user()
+        self._build_obs_postprocessors()
+
+    def _build_obs_basic(self):
+        """initial obs"""
+        self.obs = {
+            "default_search": "ytsearch",
+            "merge_output_format": "mp4",
+            "restrictfilenames": True,
+            "outtmpl": (
+                self.config["application"]["cache_dir"]
+                + "/download/"
+                + self.config["application"]["file_template"]
+            ),
+            "progress_hooks": [self._progress_hook],
+            "noprogress": True,
+            "quiet": True,
+            "continuedl": True,
+            "retries": 3,
+            "writethumbnail": False,
+            "noplaylist": True,
+            "check_formats": "selected",
+            "socket_timeout": 3,
+        }
+
+    def _build_obs_user(self):
+        """build user customized options"""
+        if self.config["downloads"]["format"]:
+            self.obs["format"] = self.config["downloads"]["format"]
+        if self.config["downloads"]["limit_speed"]:
+            self.obs["ratelimit"] = (
+                self.config["downloads"]["limit_speed"] * 1024
+            )
+        if self.config["downloads"]["cookie_import"]:
+            cookie_path = CookieHandler().use()
+            self.obs["cookiefile"] = cookie_path
+
+        throttle = self.config["downloads"]["throttledratelimit"]
+        if throttle:
+            self.obs["throttledratelimit"] = throttle * 1024
+
+    def _build_obs_postprocessors(self):
+        """add postprocessor to obs"""
+        postprocessors = []
+
+        if self.config["downloads"]["add_metadata"]:
+            postprocessors.append(
+                {
+                    "key": "FFmpegMetadata",
+                    "add_chapters": True,
+                    "add_metadata": True,
+                }
+            )
+
+        if self.config["downloads"]["add_thumbnail"]:
+            postprocessors.append(
+                {
+                    "key": "EmbedThumbnail",
+                    "already_have_thumbnail": True,
+                }
+            )
+            self.obs["writethumbnail"] = True
+
+        self.obs["postprocessors"] = postprocessors
+
+    def get_format_overwrites(self, youtube_id):
+        """get overwrites from single video"""
+        overwrites = self.video_overwrites.get(youtube_id, False)
+        if overwrites:
+            return overwrites.get("download_format", False)
+
+        return False
+
+    def _dl_single_vid(self, youtube_id):
+        """download single video"""
+        obs = self.obs.copy()
+        format_overwrite = self.get_format_overwrites(youtube_id)
+        if format_overwrite:
+            obs["format"] = format_overwrite
+
+        dl_cache = self.config["application"]["cache_dir"] + "/download/"
+
+        # check if already in cache to continue from there
+        all_cached = ignore_filelist(os.listdir(dl_cache))
+        for file_name in all_cached:
+            if youtube_id in file_name:
+                obs["outtmpl"] = os.path.join(dl_cache, file_name)
+
+        with yt_dlp.YoutubeDL(obs) as ydl:
+            try:
+                ydl.download([youtube_id])
+            except yt_dlp.utils.DownloadError:
+                print("retry failed download: " + youtube_id)
+                sleep(10)
+                ydl.download([youtube_id])
+
+        if self.obs["writethumbnail"]:
+            # webp files don't get cleaned up automatically
+            all_cached = ignore_filelist(os.listdir(dl_cache))
+            to_clean = [i for i in all_cached if not i.endswith(".mp4")]
+            for file_name in to_clean:
+                file_path = os.path.join(dl_cache, file_name)
+                os.remove(file_path)
+
+    def move_to_archive(self, vid_dict):
+        """move downloaded video from cache to archive"""
+        videos = self.config["application"]["videos"]
+        host_uid = self.config["application"]["HOST_UID"]
+        host_gid = self.config["application"]["HOST_GID"]
+        channel_name = clean_string(vid_dict["channel"]["channel_name"])
+        if len(channel_name) <= 3:
+            # fall back to channel id
+            channel_name = vid_dict["channel"]["channel_id"]
+        # make archive folder with correct permissions
+        new_folder = os.path.join(videos, channel_name)
+        if not os.path.exists(new_folder):
+            os.makedirs(new_folder)
+            if host_uid and host_gid:
+                os.chown(new_folder, host_uid, host_gid)
+        # find real filename
+        cache_dir = self.config["application"]["cache_dir"]
+        all_cached = ignore_filelist(os.listdir(cache_dir + "/download/"))
+        for file_str in all_cached:
+            if vid_dict["youtube_id"] in file_str:
+                old_file = file_str
+        old_file_path = os.path.join(cache_dir, "download", old_file)
+        new_file_path = os.path.join(videos, vid_dict["media_url"])
+        # move media file and fix permission
+        shutil.move(old_file_path, new_file_path)
+        if host_uid and host_gid:
+            os.chown(new_file_path, host_uid, host_gid)
+
+    @staticmethod
+    def _delete_from_pending(youtube_id):
+        """delete downloaded video from pending index if its there"""
+        path = f"ta_download/_doc/{youtube_id}"
+        _, _ = ElasticWrap(path).delete()
+
+    def _add_subscribed_channels(self):
+        """add all channels subscribed to refresh"""
+        all_subscribed = PlaylistSubscription().get_playlists()
+        if not all_subscribed:
+            return
+
+        channel_ids = [i["playlist_channel_id"] for i in all_subscribed]
+        for channel_id in channel_ids:
+            self.channels.add(channel_id)
+
+        return

tubearchivist/home/src/es/connect.py

@@ -0,0 +1,154 @@
+"""
+functionality:
+- wrapper around requests to call elastic search
+- reusable search_after to extract total index
+"""
+
+import json
+
+import requests
+from home.src.ta.config import AppConfig
+
+
+class ElasticWrap:
+    """makes all calls to elastic search
+    returns response json and status code tuple
+    """
+
+    def __init__(self, path, config=False):
+        self.url = False
+        self.auth = False
+        self.path = path
+        self.config = config
+        self._get_config()
+
+    def _get_config(self):
+        """add config if not passed"""
+        if not self.config:
+            self.config = AppConfig().config
+
+        es_url = self.config["application"]["es_url"]
+        self.auth = self.config["application"]["es_auth"]
+        self.url = f"{es_url}/{self.path}"
+
+    def get(self, data=False):
+        """get data from es"""
+        if data:
+            response = requests.get(self.url, json=data, auth=self.auth)
+        else:
+            response = requests.get(self.url, auth=self.auth)
+        if not response.ok:
+            print(response.text)
+
+        return response.json(), response.status_code
+
+    def post(self, data=False, ndjson=False):
+        """post data to es"""
+        if ndjson:
+            headers = {"Content-type": "application/x-ndjson"}
+            payload = data
+        else:
+            headers = {"Content-type": "application/json"}
+            payload = json.dumps(data)
+
+        if data:
+            response = requests.post(
+                self.url, data=payload, headers=headers, auth=self.auth
+            )
+        else:
+            response = requests.post(self.url, headers=headers, auth=self.auth)
+
+        if not response.ok:
+            print(response.text)
+
+        return response.json(), response.status_code
+
+    def put(self, data, refresh=False):
+        """put data to es"""
+        if refresh:
+            self.url = f"{self.url}/?refresh=true"
+        response = requests.put(f"{self.url}", json=data, auth=self.auth)
+        if not response.ok:
+            print(response.text)
+            print(data)
+            raise ValueError("failed to add item to index")
+
+        return response.json(), response.status_code
+
+    def delete(self, data=False, refresh=False):
+        """delete document from es"""
+        if refresh:
+            self.url = f"{self.url}/?refresh=true"
+        if data:
+            response = requests.delete(self.url, json=data, auth=self.auth)
+        else:
+            response = requests.delete(self.url, auth=self.auth)
+
+        if not response.ok:
+            print(response.text)
+
+        return response.json(), response.status_code
+
+
+class IndexPaginate:
+    """use search_after to go through whole index"""
+
+    DEFAULT_SIZE = 500
+
+    def __init__(self, index_name, data, size=False, keep_source=False):
+        self.index_name = index_name
+        self.data = data
+        self.pit_id = False
+        self.size = size
+        self.keep_source = keep_source
+
+    def get_results(self):
+        """get all results"""
+        self.get_pit()
+        self.validate_data()
+        all_results = self.run_loop()
+        self.clean_pit()
+        return all_results
+
+    def get_pit(self):
+        """get pit for index"""
+        path = f"{self.index_name}/_pit?keep_alive=10m"
+        response, _ = ElasticWrap(path).post()
+        self.pit_id = response["id"]
+
+    def validate_data(self):
+        """add pit and size to data"""
+        if "sort" not in self.data.keys():
+            print(self.data)
+            raise ValueError("missing sort key in data")
+
+        size = self.size or self.DEFAULT_SIZE
+
+        self.data["size"] = size
+        self.data["pit"] = {"id": self.pit_id, "keep_alive": "10m"}
+
+    def run_loop(self):
+        """loop through results until last hit"""
+        all_results = []
+        while True:
+            response, _ = ElasticWrap("_search").get(data=self.data)
+            all_hits = response["hits"]["hits"]
+            if all_hits:
+                for hit in all_hits:
+                    if self.keep_source:
+                        source = hit
+                    else:
+                        source = hit["_source"]
+                    search_after = hit["sort"]
+                    all_results.append(source)
+                # update search_after with last hit data
+                self.data["search_after"] = search_after
+            else:
+                break
+
+        return all_results
+
+    def clean_pit(self):
+        """delete pit from elastic search"""
+        data = {"id": self.pit_id}
+        ElasticWrap("_pit").delete(data=data)

tubearchivist/home/src/es/index_mapping.json

@@ -0,0 +1,465 @@
+{
+    "index_config": [{
+            "index_name": "channel",
+            "expected_map": {
+                "channel_id": {
+                    "type": "keyword"
+                },
+                "channel_name": {
+                    "type": "text",
+                    "analyzer": "english",
+                    "fields": {
+                        "keyword": {
+                            "type": "keyword",
+                            "ignore_above": 256,
+                            "normalizer": "to_lower"
+                        },
+                        "search_as_you_type": {
+                            "type": "search_as_you_type",
+                            "doc_values": false,
+                            "max_shingle_size": 3
+                        }
+                    }
+                },
+                "channel_banner_url": {
+                    "type": "keyword",
+                    "index": false
+                },
+                "channel_tvart_url": {
+                    "type": "keyword",
+                    "index": false
+                },
+                "channel_thumb_url": {
+                    "type": "keyword",
+                    "index": false
+                },
+                "channel_description": {
+                    "type": "text"
+                },
+                "channel_last_refresh": {
+                    "type": "date",
+                    "format": "epoch_second"
+                },
+                "channel_overwrites": {
+                    "properties": {
+                        "download_format": {
+                            "type": "text"
+                        },
+                        "autodelete_days": {
+                            "type": "long"
+                        },
+                        "index_playlists": {
+                            "type": "boolean"
+                        },
+                        "integrate_sponsorblock": {
+                            "type" : "boolean"
+                        }
+                    }
+                }
+            },
+            "expected_set": {
+                "analysis": {
+                    "normalizer": {
+                        "to_lower": {
+                            "type": "custom",
+                            "filter": ["lowercase"]
+                        }
+                    }
+                },
+                "number_of_replicas": "0"
+            }
+        },
+        {
+            "index_name": "video",
+            "expected_map": {
+                "vid_thumb_url": {
+                    "type": "text",
+                    "index": false
+                },
+                "vid_thumb_base64": {
+                    "type": "text",
+                    "index": false
+                },
+                "date_downloaded": {
+                    "type": "date"
+                },
+                "channel": {
+                    "properties": {
+                        "channel_id": {
+                            "type": "keyword"
+                        },
+                        "channel_name": {
+                            "type": "text",
+                            "analyzer": "english",
+                            "fields": {
+                                "keyword": {
+                                    "type": "keyword",
+                                    "ignore_above": 256,
+                                    "normalizer": "to_lower"
+                                },
+                                "search_as_you_type": {
+                                    "type": "search_as_you_type",
+                                    "doc_values": false,
+                                    "max_shingle_size": 3
+                                }
+                            }
+                        },
+                        "channel_banner_url": {
+                            "type": "keyword",
+                            "index": false
+                        },
+                        "channel_tvart_url": {
+                            "type": "keyword",
+                            "index": false
+                        },
+                        "channel_thumb_url": {
+                            "type": "keyword",
+                            "index": false
+                        },
+                        "channel_description": {
+                            "type": "text"
+                        },
+                        "channel_last_refresh": {
+                            "type": "date",
+                            "format": "epoch_second"
+                        },
+                        "channel_overwrites": {
+                            "properties": {
+                                "download_format": {
+                                    "type": "text"
+                                },
+                                "autodelete_days": {
+                                    "type": "long"
+                                },
+                                "index_playlists": {
+                                    "type": "boolean"
+                                },
+                                "integrate_sponsorblock": {
+                                    "type" : "boolean"
+                                }
+                            }
+                        }
+                    }
+                },
+                "description": {
+                    "type": "text"
+                },
+                "media_url": {
+                    "type": "keyword",
+                    "index": false
+                },
+                "tags": {
+                    "type": "text",
+                    "analyzer": "english",
+                    "fields": {
+                        "keyword": {
+                            "type": "keyword",
+                            "ignore_above": 256
+                        }
+                    }
+                },
+                "title": {
+                    "type": "text",
+                    "analyzer": "english",
+                    "fields": {
+                        "keyword": {
+                            "type": "keyword",
+                            "ignore_above": 256,
+                            "normalizer": "to_lower"
+                        },
+                        "search_as_you_type": {
+                            "type": "search_as_you_type",
+                            "doc_values": false,
+                            "max_shingle_size": 3
+                        }
+                    }
+                },
+                "vid_last_refresh": {
+                    "type": "date"
+                },
+                "youtube_id": {
+                    "type": "keyword"
+                },
+                "published": {
+                    "type": "date"
+                },
+                "playlist": {
+                    "type": "text",
+                    "fields": {
+                        "keyword": {
+                            "type": "keyword",
+                            "ignore_above": 256,
+                            "normalizer": "to_lower"
+                        }
+                    }
+                },
+                "stats" : {
+                    "properties" : {
+                        "average_rating" : {
+                            "type" : "float"
+                        },
+                        "dislike_count" : {
+                            "type" : "long"
+                        },
+                        "like_count" : {
+                            "type" : "long"
+                        },
+                        "view_count" : {
+                            "type" : "long"
+                        }
+                    }
+                },
+                "subtitles": {
+                    "properties": {
+                        "ext": {
+                            "type": "keyword",
+                            "index": false
+                        },
+                        "lang": {
+                            "type": "keyword",
+                            "index": false
+                        },
+                        "media_url": {
+                            "type": "keyword",
+                            "index": false
+                        },
+                        "name": {
+                            "type": "keyword"
+                        },
+                        "source": {
+                            "type": "keyword"
+                        },
+                        "url": {
+                            "type": "keyword",
+                            "index": false
+                        }
+                    }
+                },
+                "sponsorblock": {
+                    "properties": {
+                        "last_refresh": {
+                            "type": "date"
+                        },
+                        "has_unlocked": {
+                            "type": "boolean"
+                        },
+                        "is_enabled": {
+                            "type": "boolean"
+                        },
+                        "segments" : {
+                            "properties" : {
+                                "UUID" : {
+                                    "type": "keyword"
+                                },
+                                "actionType" : {
+                                    "type": "keyword"
+                                },
+                                "category" : {
+                                    "type": "keyword"
+                                },
+                                "locked" : {
+                                    "type" : "short"
+                                },
+                                "segment" : {
+                                    "type" : "float"
+                                },
+                                "videoDuration" : {
+                                    "type" : "float"
+                                },
+                                "votes" : {
+                                    "type" : "long"
+                                }
+                            }
+                        }
+                    }
+                }
+            },
+            "expected_set": {
+                "analysis": {
+                    "normalizer": {
+                        "to_lower": {
+                            "type": "custom",
+                            "filter": ["lowercase"]
+                        }
+                    }
+                },
+                "number_of_replicas": "0"
+            }
+        },
+        {
+            "index_name": "download",
+            "expected_map": {
+                "timestamp": {
+                    "type": "date"
+                },
+                "channel_id": {
+                    "type": "keyword"
+                },
+                "channel_name": {
+                    "type": "text",
+                    "fields": {
+                        "keyword": {
+                            "type": "keyword",
+                            "ignore_above": 256,
+                            "normalizer": "to_lower"
+                        }
+                    }
+                },
+                "status": {
+                    "type": "keyword"
+                },
+                "title": {
+                    "type": "text",
+                    "fields": {
+                        "keyword": {
+                            "type": "keyword",
+                            "ignore_above": 256,
+                            "normalizer": "to_lower"
+                        }
+                    }
+                },
+                "vid_thumb_url": {
+                    "type": "keyword"
+                },
+                "youtube_id": {
+                    "type": "keyword"
+                }
+            },
+            "expected_set": {
+                "analysis": {
+                    "normalizer": {
+                        "to_lower": {
+                            "type": "custom",
+                            "filter": ["lowercase"]
+                        }
+                    }
+                },
+                "number_of_replicas": "0"
+            }
+        },
+        {
+            "index_name": "playlist",
+            "expected_map": {
+                "playlist_id": {
+                    "type": "keyword"
+                },
+                "playlist_description": {
+                    "type": "text"
+                },
+                "playlist_name": {
+                    "type": "text",
+                    "analyzer": "english",
+                    "fields": {
+                        "keyword": {
+                            "type": "keyword",
+                            "ignore_above": 256,
+                            "normalizer": "to_lower"
+                        },
+                        "search_as_you_type": {
+                            "type": "search_as_you_type",
+                            "doc_values": false,
+                            "max_shingle_size": 3
+                        }
+                    }
+                },
+                "playlist_channel": {
+                    "type": "text",
+                    "fields": {
+                        "keyword": {
+                            "type": "keyword",
+                            "ignore_above": 256,
+                            "normalizer": "to_lower"
+                        }
+                    }
+                },
+                "playlist_channel_id": {
+                    "type": "keyword"
+                },
+                "playlist_thumbnail": {
+                    "type": "keyword"
+                },
+                "playlist_last_refresh": {
+                    "type": "date"
+                }
+            },
+            "expected_set": {
+                "analysis": {
+                    "normalizer": {
+                        "to_lower": {
+                            "type": "custom",
+                            "filter": ["lowercase"]
+                        }
+                    }
+                },
+                "number_of_replicas": "0"
+            }
+        },
+        {
+            "index_name": "subtitle",
+            "expected_map": {
+                "youtube_id": {
+                    "type": "keyword"
+                },
+                "title": {
+                    "type": "text",
+                    "fields": {
+                        "keyword": {
+                            "type": "keyword",
+                            "ignore_above": 256,
+                            "normalizer": "to_lower"
+                        }
+                    }
+                },
+                "subtitle_fragment_id": {
+                    "type": "keyword"
+                },
+                "subtitle_channel": {
+                    "type": "text",
+                    "fields": {
+                        "keyword": {
+                            "type": "keyword",
+                            "ignore_above": 256,
+                            "normalizer": "to_lower"
+                        }
+                    }
+                },
+                "subtitle_channel_id": {
+                    "type": "keyword"
+                },
+                "subtitle_start": {
+                    "type": "text"
+                },
+                "subtitle_end": {
+                    "type": "text"
+                },
+                "subtitle_last_refresh": {
+                    "type": "date"
+                },
+                "subtitle_index": {
+                    "type" : "long"
+                },
+                "subtitle_lang": {
+                    "type": "keyword"
+                },
+                "subtitle_source": {
+                    "type": "keyword"
+                },
+                "subtitle_line": {
+                    "type" : "text",
+                    "analyzer": "english"
+                }
+            },
+            "expected_set": {
+                "analysis": {
+                    "normalizer": {
+                        "to_lower": {
+                            "type": "custom",
+                            "filter": ["lowercase"]
+                        }
+                    }
+                },
+                "number_of_replicas": "0"
+            }
+        }
+    ]
+}

tubearchivist/home/src/es/index_setup.py

@@ -0,0 +1,402 @@
+"""
+functionality:
+- setup elastic index at first start
+- verify and update index mapping and settings if needed
+- backup and restore metadata
+"""
+
+import json
+import os
+import zipfile
+from datetime import datetime
+
+from home.src.es.connect import ElasticWrap, IndexPaginate
+from home.src.ta.config import AppConfig
+from home.src.ta.helper import ignore_filelist
+
+
+class ElasticIndex:
+    """
+    handle mapping and settings on elastic search for a given index
+    """
+
+    def __init__(self, index_name, expected_map, expected_set):
+        self.index_name = index_name
+        self.expected_map = expected_map
+        self.expected_set = expected_set
+        self.exists, self.details = self.index_exists()
+
+    def index_exists(self):
+        """check if index already exists and return mapping if it does"""
+        response, status_code = ElasticWrap(f"ta_{self.index_name}").get()
+        exists = status_code == 200
+        details = response.get(f"ta_{self.index_name}", False)
+
+        return exists, details
+
+    def validate(self):
+        """
+        check if all expected mappings and settings match
+        returns True when rebuild is needed
+        """
+
+        if self.expected_map:
+            rebuild = self.validate_mappings()
+            if rebuild:
+                return rebuild
+
+        if self.expected_set:
+            rebuild = self.validate_settings()
+            if rebuild:
+                return rebuild
+
+        return False
+
+    def validate_mappings(self):
+        """check if all mappings are as expected"""
+        now_map = self.details["mappings"]["properties"]
+
+        for key, value in self.expected_map.items():
+            # nested
+            if list(value.keys()) == ["properties"]:
+                for key_n, value_n in value["properties"].items():
+                    if key not in now_map:
+                        print(key_n, value_n)
+                        return True
+                    if key_n not in now_map[key]["properties"].keys():
+                        print(key_n, value_n)
+                        return True
+                    if not value_n == now_map[key]["properties"][key_n]:
+                        print(key_n, value_n)
+                        return True
+
+                continue
+
+            # not nested
+            if key not in now_map.keys():
+                print(key, value)
+                return True
+            if not value == now_map[key]:
+                print(key, value)
+                return True
+
+        return False
+
+    def validate_settings(self):
+        """check if all settings are as expected"""
+
+        now_set = self.details["settings"]["index"]
+
+        for key, value in self.expected_set.items():
+            if key not in now_set.keys():
+                print(key, value)
+                return True
+
+            if not value == now_set[key]:
+                print(key, value)
+                return True
+
+        return False
+
+    def rebuild_index(self):
+        """rebuild with new mapping"""
+        self.reindex("backup")
+        self.delete_index(backup=False)
+        self.create_blank()
+        self.reindex("restore")
+        self.delete_index()
+
+    def reindex(self, method):
+        """create on elastic search"""
+        if method == "backup":
+            source = f"ta_{self.index_name}"
+            destination = f"ta_{self.index_name}_backup"
+        elif method == "restore":
+            source = f"ta_{self.index_name}_backup"
+            destination = f"ta_{self.index_name}"
+
+        data = {"source": {"index": source}, "dest": {"index": destination}}
+        _, _ = ElasticWrap("_reindex?refresh=true").post(data=data)
+
+    def delete_index(self, backup=True):
+        """delete index passed as argument"""
+        path = f"ta_{self.index_name}"
+        if backup:
+            path = path + "_backup"
+
+        _, _ = ElasticWrap(path).delete()
+
+    def create_blank(self):
+        """apply new mapping and settings for blank new index"""
+        data = {}
+        if self.expected_set:
+            data.update({"settings": self.expected_set})
+        if self.expected_map:
+            data.update({"mappings": {"properties": self.expected_map}})
+
+        _, _ = ElasticWrap(f"ta_{self.index_name}").put(data)
+
+
+class ElasticBackup:
+    """dump index to nd-json files for later bulk import"""
+
+    def __init__(self, index_config, reason):
+        self.config = AppConfig().config
+        self.cache_dir = self.config["application"]["cache_dir"]
+        self.index_config = index_config
+        self.reason = reason
+        self.timestamp = datetime.now().strftime("%Y%m%d")
+        self.backup_files = []
+
+    @staticmethod
+    def get_all_documents(index_name):
+        """export all documents of a single index"""
+        data = {
+            "query": {"match_all": {}},
+            "sort": [{"_doc": {"order": "desc"}}],
+        }
+        paginate = IndexPaginate(f"ta_{index_name}", data, keep_source=True)
+        all_results = paginate.get_results()
+
+        return all_results
+
+    @staticmethod
+    def build_bulk(all_results):
+        """build bulk query data from all_results"""
+        bulk_list = []
+
+        for document in all_results:
+            document_id = document["_id"]
+            es_index = document["_index"]
+            action = {"index": {"_index": es_index, "_id": document_id}}
+            source = document["_source"]
+            bulk_list.append(json.dumps(action))
+            bulk_list.append(json.dumps(source))
+
+        # add last newline
+        bulk_list.append("\n")
+        file_content = "\n".join(bulk_list)
+
+        return file_content
+
+    def write_es_json(self, file_content, index_name):
+        """write nd-json file for es _bulk API to disk"""
+        file_name = f"es_{index_name}-{self.timestamp}.json"
+        file_path = os.path.join(self.cache_dir, "backup", file_name)
+        with open(file_path, "w", encoding="utf-8") as f:
+            f.write(file_content)
+
+        self.backup_files.append(file_path)
+
+    def write_ta_json(self, all_results, index_name):
+        """write generic json file to disk"""
+        file_name = f"ta_{index_name}-{self.timestamp}.json"
+        file_path = os.path.join(self.cache_dir, "backup", file_name)
+        to_write = [i["_source"] for i in all_results]
+        file_content = json.dumps(to_write)
+        with open(file_path, "w", encoding="utf-8") as f:
+            f.write(file_content)
+
+        self.backup_files.append(file_path)
+
+    def zip_it(self):
+        """pack it up into single zip file"""
+        file_name = f"ta_backup-{self.timestamp}-{self.reason}.zip"
+        backup_folder = os.path.join(self.cache_dir, "backup")
+        backup_file = os.path.join(backup_folder, file_name)
+
+        with zipfile.ZipFile(
+            backup_file, "w", compression=zipfile.ZIP_DEFLATED
+        ) as zip_f:
+            for backup_file in self.backup_files:
+                zip_f.write(backup_file, os.path.basename(backup_file))
+
+        # cleanup
+        for backup_file in self.backup_files:
+            os.remove(backup_file)
+
+    def post_bulk_restore(self, file_name):
+        """send bulk to es"""
+        file_path = os.path.join(self.cache_dir, file_name)
+        with open(file_path, "r", encoding="utf-8") as f:
+            data = f.read()
+
+        if not data.strip():
+            return
+
+        _, _ = ElasticWrap("_bulk").post(data=data, ndjson=True)
+
+    def get_all_backup_files(self):
+        """build all available backup files for view"""
+        backup_dir = os.path.join(self.cache_dir, "backup")
+        backup_files = os.listdir(backup_dir)
+        all_backup_files = ignore_filelist(backup_files)
+        all_available_backups = [
+            i
+            for i in all_backup_files
+            if i.startswith("ta_") and i.endswith(".zip")
+        ]
+        all_available_backups.sort(reverse=True)
+
+        backup_dicts = []
+        for backup_file in all_available_backups:
+            file_split = backup_file.split("-")
+            if len(file_split) == 2:
+                timestamp = file_split[1].strip(".zip")
+                reason = False
+            elif len(file_split) == 3:
+                timestamp = file_split[1]
+                reason = file_split[2].strip(".zip")
+
+            to_add = {
+                "filename": backup_file,
+                "timestamp": timestamp,
+                "reason": reason,
+            }
+            backup_dicts.append(to_add)
+
+        return backup_dicts
+
+    def unpack_zip_backup(self, filename):
+        """extract backup zip and return filelist"""
+        backup_dir = os.path.join(self.cache_dir, "backup")
+        file_path = os.path.join(backup_dir, filename)
+
+        with zipfile.ZipFile(file_path, "r") as z:
+            zip_content = z.namelist()
+            z.extractall(backup_dir)
+
+        return zip_content
+
+    def restore_json_files(self, zip_content):
+        """go through the unpacked files and restore"""
+        backup_dir = os.path.join(self.cache_dir, "backup")
+
+        for json_f in zip_content:
+
+            file_name = os.path.join(backup_dir, json_f)
+
+            if not json_f.startswith("es_") or not json_f.endswith(".json"):
+                os.remove(file_name)
+                continue
+
+            print("restoring: " + json_f)
+            self.post_bulk_restore(file_name)
+            os.remove(file_name)
+
+    @staticmethod
+    def index_exists(index_name):
+        """check if index already exists to skip"""
+        _, status_code = ElasticWrap(f"ta_{index_name}").get()
+        exists = status_code == 200
+
+        return exists
+
+    def rotate_backup(self):
+        """delete old backups if needed"""
+        rotate = self.config["scheduler"]["run_backup_rotate"]
+        if not rotate:
+            return
+
+        all_backup_files = self.get_all_backup_files()
+        auto = [i for i in all_backup_files if i["reason"] == "auto"]
+
+        if len(auto) <= rotate:
+            print("no backup files to rotate")
+            return
+
+        backup_dir = os.path.join(self.cache_dir, "backup")
+
+        all_to_delete = auto[rotate:]
+        for to_delete in all_to_delete:
+            file_path = os.path.join(backup_dir, to_delete["filename"])
+            print(f"remove old backup file: {file_path}")
+            os.remove(file_path)
+
+
+def get_mapping():
+    """read index_mapping.json and get expected mapping and settings"""
+    with open("home/src/es/index_mapping.json", "r", encoding="utf-8") as f:
+        index_config = json.load(f).get("index_config")
+
+    return index_config
+
+
+def index_check(force_restore=False):
+    """check if all indexes are created and have correct mapping"""
+
+    backed_up = False
+    index_config = get_mapping()
+
+    for index in index_config:
+        index_name = index["index_name"]
+        expected_map = index["expected_map"]
+        expected_set = index["expected_set"]
+        handler = ElasticIndex(index_name, expected_map, expected_set)
+        # force restore
+        if force_restore:
+            handler.delete_index(backup=False)
+            handler.create_blank()
+            continue
+
+        # create new
+        if not handler.exists:
+            print(f"create new blank index with name ta_{index_name}...")
+            handler.create_blank()
+            continue
+
+        # validate index
+        rebuild = handler.validate()
+        if rebuild:
+            # make backup before rebuild
+            if not backed_up:
+                print("running backup first")
+                backup_all_indexes(reason="update")
+                backed_up = True
+
+            print(f"applying new mappings to index ta_{index_name}...")
+            handler.rebuild_index()
+            continue
+
+        # else all good
+        print(f"ta_{index_name} index is created and up to date...")
+
+
+def get_available_backups():
+    """return dict of available backups for settings view"""
+    index_config = get_mapping()
+    backup_handler = ElasticBackup(index_config, reason=False)
+    all_backup_files = backup_handler.get_all_backup_files()
+    return all_backup_files
+
+
+def backup_all_indexes(reason):
+    """backup all es indexes to disk"""
+    index_config = get_mapping()
+    backup_handler = ElasticBackup(index_config, reason)
+
+    for index in backup_handler.index_config:
+        index_name = index["index_name"]
+        print(f"backup: export in progress for {index_name}")
+        if not backup_handler.index_exists(index_name):
+            continue
+        all_results = backup_handler.get_all_documents(index_name)
+        file_content = backup_handler.build_bulk(all_results)
+        backup_handler.write_es_json(file_content, index_name)
+        backup_handler.write_ta_json(all_results, index_name)
+
+    backup_handler.zip_it()
+
+    if reason == "auto":
+        backup_handler.rotate_backup()
+
+
+def restore_from_backup(filename):
+    """restore indexes from backup file"""
+    # delete
+    index_check(force_restore=True)
+    # recreate
+    index_config = get_mapping()
+    backup_handler = ElasticBackup(index_config, reason=False)
+    zip_content = backup_handler.unpack_zip_backup(filename)
+    backup_handler.restore_json_files(zip_content)

tubearchivist/home/src/frontend/api_calls.py

@@ -0,0 +1,313 @@
+"""
+Functionality:
+- collection of functions and tasks from frontend
+- called via user input
+"""
+
+from home.src.download.queue import PendingInteract
+from home.src.download.subscriptions import (
+    ChannelSubscription,
+    PlaylistSubscription,
+)
+from home.src.frontend.searching import SearchForm
+from home.src.frontend.watched import WatchState
+from home.src.index.channel import YoutubeChannel
+from home.src.index.playlist import YoutubePlaylist
+from home.src.index.video import YoutubeVideo
+from home.src.ta.helper import UrlListParser
+from home.src.ta.ta_redis import RedisArchivist, RedisQueue
+from home.tasks import (
+    download_pending,
+    download_single,
+    index_channel_playlists,
+    kill_dl,
+    re_sync_thumbs,
+    rescan_filesystem,
+    run_backup,
+    run_manual_import,
+    run_restore_backup,
+    subscribe_to,
+    update_subscribed,
+)
+
+
+class PostData:
+    """
+    map frontend http post values to backend funcs
+    handover long running tasks to celery
+    """
+
+    def __init__(self, post_dict, current_user):
+        self.post_dict = post_dict
+        self.to_exec, self.exec_val = list(post_dict.items())[0]
+        self.current_user = current_user
+
+    def run_task(self):
+        """execute and return task result"""
+        to_exec = self.exec_map()
+        task_result = to_exec()
+        return task_result
+
+    def exec_map(self):
+        """map dict key and return function to execute"""
+        exec_map = {
+            "watched": self._watched,
+            "un_watched": self._un_watched,
+            "change_view": self._change_view,
+            "rescan_pending": self._rescan_pending,
+            "ignore": self._ignore,
+            "dl_pending": self._dl_pending,
+            "queue": self._queue_handler,
+            "unsubscribe": self._unsubscribe,
+            "subscribe": self._subscribe,
+            "sort_order": self._sort_order,
+            "hide_watched": self._hide_watched,
+            "show_subed_only": self._show_subed_only,
+            "dlnow": self._dlnow,
+            "show_ignored_only": self._show_ignored_only,
+            "forgetIgnore": self._forget_ignore,
+            "addSingle": self._add_single,
+            "deleteQueue": self._delete_queue,
+            "manual-import": self._manual_import,
+            "re-embed": self._re_embed,
+            "db-backup": self._db_backup,
+            "db-restore": self._db_restore,
+            "fs-rescan": self._fs_rescan,
+            "multi_search": self._multi_search,
+            "delete-video": self._delete_video,
+            "delete-channel": self._delete_channel,
+            "delete-playlist": self._delete_playlist,
+            "find-playlists": self._find_playlists,
+        }
+
+        return exec_map[self.to_exec]
+
+    def _watched(self):
+        """mark as watched"""
+        WatchState(self.exec_val).mark_as_watched()
+        return {"success": True}
+
+    def _un_watched(self):
+        """mark as unwatched"""
+        WatchState(self.exec_val).mark_as_unwatched()
+        return {"success": True}
+
+    def _change_view(self):
+        """process view changes in home, channel, and downloads"""
+        origin, new_view = self.exec_val.split(":")
+        key = f"{self.current_user}:view:{origin}"
+        print(f"change view: {key} to {new_view}")
+        RedisArchivist().set_message(key, {"status": new_view}, expire=False)
+        return {"success": True}
+
+    @staticmethod
+    def _rescan_pending():
+        """look for new items in subscribed channels"""
+        print("rescan subscribed channels")
+        update_subscribed.delay()
+        return {"success": True}
+
+    def _ignore(self):
+        """ignore from download queue"""
+        video_id = self.exec_val
+        print(f"{video_id}: ignore video from download queue")
+        PendingInteract(video_id=video_id, status="ignore").update_status()
+        # also clear from redis queue
+        RedisQueue().clear_item(video_id)
+        return {"success": True}
+
+    @staticmethod
+    def _dl_pending():
+        """start the download queue"""
+        print("download pending")
+        running = download_pending.delay()
+        task_id = running.id
+        print(f"{task_id}: set task id")
+        RedisArchivist().set_message("dl_queue_id", task_id, expire=False)
+        return {"success": True}
+
+    def _queue_handler(self):
+        """queue controls from frontend"""
+        to_execute = self.exec_val
+        if to_execute == "stop":
+            print("stopping download queue")
+            RedisQueue().clear()
+        elif to_execute == "kill":
+            task_id = RedisArchivist().get_message("dl_queue_id")
+            if not isinstance(task_id, str):
+                task_id = False
+            else:
+                print("brutally killing " + task_id)
+            kill_dl(task_id)
+
+        return {"success": True}
+
+    def _unsubscribe(self):
+        """unsubscribe from channels or playlists"""
+        id_unsub = self.exec_val
+        print(f"{id_unsub}: unsubscribe")
+        to_unsub_list = UrlListParser(id_unsub).process_list()
+        for to_unsub in to_unsub_list:
+            unsub_type = to_unsub["type"]
+            unsub_id = to_unsub["url"]
+            if unsub_type == "playlist":
+                PlaylistSubscription().change_subscribe(
+                    unsub_id, subscribe_status=False
+                )
+            elif unsub_type == "channel":
+                ChannelSubscription().change_subscribe(
+                    unsub_id, channel_subscribed=False
+                )
+            else:
+                raise ValueError("failed to process " + id_unsub)
+
+        return {"success": True}
+
+    def _subscribe(self):
+        """subscribe to channel or playlist, called from js buttons"""
+        id_sub = self.exec_val
+        print(f"{id_sub}: subscribe")
+        subscribe_to.delay(id_sub)
+        return {"success": True}
+
+    def _sort_order(self):
+        """change the sort between published to downloaded"""
+        sort_order = {"status": self.exec_val}
+        if self.exec_val in ["asc", "desc"]:
+            RedisArchivist().set_message(
+                f"{self.current_user}:sort_order", sort_order, expire=False
+            )
+        else:
+            RedisArchivist().set_message(
+                f"{self.current_user}:sort_by", sort_order, expire=False
+            )
+        return {"success": True}
+
+    def _hide_watched(self):
+        """toggle if to show watched vids or not"""
+        key = f"{self.current_user}:hide_watched"
+        message = {"status": bool(int(self.exec_val))}
+        print(f"toggle {key}: {message}")
+        RedisArchivist().set_message(key, message, expire=False)
+        return {"success": True}
+
+    def _show_subed_only(self):
+        """show or hide subscribed channels only on channels page"""
+        key = f"{self.current_user}:show_subed_only"
+        message = {"status": bool(int(self.exec_val))}
+        print(f"toggle {key}: {message}")
+        RedisArchivist().set_message(key, message, expire=False)
+        return {"success": True}
+
+    def _dlnow(self):
+        """start downloading single vid now"""
+        youtube_id = self.exec_val
+        print(f"{youtube_id}: downloading now")
+        running = download_single.delay(youtube_id=youtube_id)
+        task_id = running.id
+        print("set task id: " + task_id)
+        RedisArchivist().set_message("dl_queue_id", task_id, expire=False)
+        return {"success": True}
+
+    def _show_ignored_only(self):
+        """switch view on /downloads/ to show ignored only"""
+        show_value = self.exec_val
+        key = f"{self.current_user}:show_ignored_only"
+        value = {"status": show_value}
+        print(f"Filter download view ignored only: {show_value}")
+        RedisArchivist().set_message(key, value, expire=False)
+        return {"success": True}
+
+    def _forget_ignore(self):
+        """delete from ta_download index"""
+        video_id = self.exec_val
+        print(f"{video_id}: forget from download")
+        PendingInteract(video_id=video_id).delete_item()
+        return {"success": True}
+
+    def _add_single(self):
+        """add single youtube_id to download queue"""
+        video_id = self.exec_val
+        print(f"{video_id}: add single vid to download queue")
+        PendingInteract(video_id=video_id, status="pending").update_status()
+        return {"success": True}
+
+    def _delete_queue(self):
+        """delete download queue"""
+        status = self.exec_val
+        print("deleting from download queue: " + status)
+        PendingInteract(status=status).delete_by_status()
+        return {"success": True}
+
+    @staticmethod
+    def _manual_import():
+        """run manual import from settings page"""
+        print("starting manual import")
+        run_manual_import.delay()
+        return {"success": True}
+
+    @staticmethod
+    def _re_embed():
+        """rewrite thumbnails into media files"""
+        print("start video thumbnail embed process")
+        re_sync_thumbs.delay()
+        return {"success": True}
+
+    @staticmethod
+    def _db_backup():
+        """backup es to zip from settings page"""
+        print("backing up database")
+        run_backup.delay("manual")
+        return {"success": True}
+
+    def _db_restore(self):
+        """restore es zip from settings page"""
+        print("restoring index from backup zip")
+        filename = self.exec_val
+        run_restore_backup.delay(filename)
+        return {"success": True}
+
+    @staticmethod
+    def _fs_rescan():
+        """start file system rescan task"""
+        print("start filesystem scan")
+        rescan_filesystem.delay()
+        return {"success": True}
+
+    def _multi_search(self):
+        """search through all indexes"""
+        search_query = self.exec_val
+        print("searching for: " + search_query)
+        search_results = SearchForm().multi_search(search_query)
+        return search_results
+
+    def _delete_video(self):
+        """delete media file, metadata and thumb"""
+        youtube_id = self.exec_val
+        YoutubeVideo(youtube_id).delete_media_file()
+        return {"success": True}
+
+    def _delete_channel(self):
+        """delete channel and all matching videos"""
+        channel_id = self.exec_val
+        YoutubeChannel(channel_id).delete_channel()
+        return {"success": True}
+
+    def _delete_playlist(self):
+        """delete playlist, only metadata or incl all videos"""
+        playlist_dict = self.exec_val
+        playlist_id = playlist_dict["playlist-id"]
+        playlist_action = playlist_dict["playlist-action"]
+        print(f"{playlist_id}: delete playlist {playlist_action}")
+        if playlist_action == "metadata":
+            YoutubePlaylist(playlist_id).delete_metadata()
+        elif playlist_action == "all":
+            YoutubePlaylist(playlist_id).delete_videos_playlist()
+
+        return {"success": True}
+
+    def _find_playlists(self):
+        """add all playlists of a channel"""
+        channel_id = self.exec_val
+        index_channel_playlists.delay(channel_id)
+        return {"success": True}

tubearchivist/home/src/frontend/forms.py

@@ -0,0 +1,224 @@
+"""functionality:
+- hold all form classes used in the views
+"""
+
+from django import forms
+from django.contrib.auth.forms import AuthenticationForm
+from django.forms.widgets import PasswordInput, TextInput
+
+
+class CustomAuthForm(AuthenticationForm):
+    """better styled login form"""
+
+    username = forms.CharField(
+        widget=TextInput(
+            attrs={
+                "placeholder": "Username",
+                "autofocus": True,
+                "autocomplete": True,
+            }
+        ),
+        label=False,
+    )
+    password = forms.CharField(
+        widget=PasswordInput(attrs={"placeholder": "Password"}), label=False
+    )
+    remember_me = forms.BooleanField(required=False)
+
+
+class UserSettingsForm(forms.Form):
+    """user configurations values"""
+
+    CHOICES = [
+        ("", "-- change color scheme --"),
+        ("dark", "Dark"),
+        ("light", "Light"),
+    ]
+
+    colors = forms.ChoiceField(
+        widget=forms.Select, choices=CHOICES, required=False
+    )
+    page_size = forms.IntegerField(required=False)
+
+
+class ApplicationSettingsForm(forms.Form):
+    """handle all application settings"""
+
+    METADATA_CHOICES = [
+        ("", "-- change metadata embed --"),
+        ("0", "don't embed metadata"),
+        ("1", "embed metadata"),
+    ]
+
+    THUMBNAIL_CHOICES = [
+        ("", "-- change thumbnail embed --"),
+        ("0", "don't embed thumbnail"),
+        ("1", "embed thumbnail"),
+    ]
+
+    RYD_CHOICES = [
+        ("", "-- change ryd integrations"),
+        ("0", "disable ryd integration"),
+        ("1", "enable ryd integration"),
+    ]
+
+    SP_CHOICES = [
+        ("", "-- change sponsorblock integrations"),
+        ("0", "disable sponsorblock integration"),
+        ("1", "enable sponsorblock integration"),
+    ]
+
+    CAST_CHOICES = [
+        ("", "-- change Cast integration --"),
+        ("0", "disable Cast"),
+        ("1", "enable Cast"),
+    ]
+
+    SUBTITLE_SOURCE_CHOICES = [
+        ("", "-- change subtitle source settings"),
+        ("user", "only download user created"),
+        ("auto", "also download auto generated"),
+    ]
+
+    SUBTITLE_INDEX_CHOICES = [
+        ("", "-- change subtitle index settings --"),
+        ("0", "disable subtitle index"),
+        ("1", "enable subtitle index"),
+    ]
+
+    COOKIE_IMPORT_CHOICES = [
+        ("", "-- change cookie settings"),
+        ("0", "disable cookie"),
+        ("1", "enable cookie"),
+    ]
+
+    subscriptions_channel_size = forms.IntegerField(required=False)
+    downloads_limit_count = forms.IntegerField(required=False)
+    downloads_limit_speed = forms.IntegerField(required=False)
+    downloads_throttledratelimit = forms.IntegerField(required=False)
+    downloads_sleep_interval = forms.IntegerField(required=False)
+    downloads_autodelete_days = forms.IntegerField(required=False)
+    downloads_format = forms.CharField(required=False)
+    downloads_add_metadata = forms.ChoiceField(
+        widget=forms.Select, choices=METADATA_CHOICES, required=False
+    )
+    downloads_add_thumbnail = forms.ChoiceField(
+        widget=forms.Select, choices=THUMBNAIL_CHOICES, required=False
+    )
+    downloads_subtitle = forms.CharField(required=False)
+    downloads_subtitle_source = forms.ChoiceField(
+        widget=forms.Select, choices=SUBTITLE_SOURCE_CHOICES, required=False
+    )
+    downloads_subtitle_index = forms.ChoiceField(
+        widget=forms.Select, choices=SUBTITLE_INDEX_CHOICES, required=False
+    )
+    downloads_cookie_import = forms.ChoiceField(
+        widget=forms.Select, choices=COOKIE_IMPORT_CHOICES, required=False
+    )
+    downloads_integrate_ryd = forms.ChoiceField(
+        widget=forms.Select, choices=RYD_CHOICES, required=False
+    )
+    downloads_integrate_sponsorblock = forms.ChoiceField(
+        widget=forms.Select, choices=SP_CHOICES, required=False
+    )
+    application_enable_cast = forms.ChoiceField(
+        widget=forms.Select, choices=CAST_CHOICES, required=False
+    )
+
+
+class SchedulerSettingsForm(forms.Form):
+    """handle scheduler settings"""
+
+    update_subscribed = forms.CharField(required=False)
+    download_pending = forms.CharField(required=False)
+    check_reindex = forms.CharField(required=False)
+    check_reindex_days = forms.IntegerField(required=False)
+    thumbnail_check = forms.CharField(required=False)
+    run_backup = forms.CharField(required=False)
+    run_backup_rotate = forms.IntegerField(required=False)
+
+
+class MultiSearchForm(forms.Form):
+    """multi search form for /search/"""
+
+    searchInput = forms.CharField(
+        label="",
+        widget=forms.TextInput(
+            attrs={
+                "autocomplete": "off",
+                "oninput": "searchMulti(this.value)",
+                "autofocus": True,
+            }
+        ),
+    )
+    home = forms.CharField(widget=forms.HiddenInput())
+    channel = forms.CharField(widget=forms.HiddenInput())
+    playlist = forms.CharField(widget=forms.HiddenInput())
+
+
+class AddToQueueForm(forms.Form):
+    """text area form to add to downloads"""
+
+    vid_url = forms.CharField(
+        label=False,
+        widget=forms.Textarea(
+            attrs={
+                "rows": 4,
+                "placeholder": "Enter Video Urls or IDs here...",
+            }
+        ),
+    )
+
+
+class SubscribeToChannelForm(forms.Form):
+    """text area form to subscribe to multiple channels"""
+
+    subscribe = forms.CharField(
+        label="Subscribe to channels",
+        widget=forms.Textarea(
+            attrs={
+                "rows": 3,
+                "placeholder": "Input channel ID, URL or Video of a channel",
+            }
+        ),
+    )
+
+
+class SubscribeToPlaylistForm(forms.Form):
+    """text area form to subscribe to multiple playlists"""
+
+    subscribe = forms.CharField(
+        label="Subscribe to playlists",
+        widget=forms.Textarea(
+            attrs={
+                "rows": 3,
+                "placeholder": "Input playlist IDs or URLs",
+            }
+        ),
+    )
+
+
+class ChannelOverwriteForm(forms.Form):
+    """custom overwrites for channel settings"""
+
+    PLAYLIST_INDEX = [
+        ("", "-- change playlist index --"),
+        ("0", "Disable playlist index"),
+        ("1", "Enable playlist index"),
+    ]
+
+    SP_CHOICES = [
+        ("", "-- change sponsorblock integrations"),
+        ("disable", "disable sponsorblock integration"),
+        ("1", "enable sponsorblock integration"),
+        ("0", "unset sponsorblock integration"),
+    ]
+
+    download_format = forms.CharField(label=False, required=False)
+    autodelete_days = forms.IntegerField(label=False, required=False)
+    index_playlists = forms.ChoiceField(
+        widget=forms.Select, choices=PLAYLIST_INDEX, required=False
+    )
+    integrate_sponsorblock = forms.ChoiceField(
+        widget=forms.Select, choices=SP_CHOICES, required=False
+    )

tubearchivist/home/src/frontend/searching.py

@@ -6,36 +6,26 @@ Functionality:
 - calculate pagination values
 """
 
-import math
 import urllib.parse
 from datetime import datetime
 
-import requests
-from home.src.config import AppConfig
-from home.src.helper import RedisArchivist
-from home.src.thumbnails import ThumbManager
+from home.src.download.thumbnails import ThumbManager
+from home.src.es.connect import ElasticWrap
+from home.src.ta.config import AppConfig
 
 
 class SearchHandler:
     """search elastic search"""
 
-    CONFIG = AppConfig().config
-    CACHE_DIR = CONFIG["application"]["cache_dir"]
-    ES_AUTH = CONFIG["application"]["es_auth"]
-
-    def __init__(self, url, data):
+    def __init__(self, path, config, data=False):
         self.max_hits = None
-        self.url = url
+        self.path = path
+        self.config = config
         self.data = data
 
     def get_data(self):
         """get the data"""
-        if self.data:
-            response = requests.get(
-                self.url, json=self.data, auth=self.ES_AUTH
-            ).json()
-        else:
-            response = requests.get(self.url, auth=self.ES_AUTH).json()
+        response, _ = ElasticWrap(self.path, config=self.config).get(self.data)
 
         if "hits" in response.keys():
             self.max_hits = response["hits"]["total"]["value"]
@@ -121,6 +111,12 @@ class SearchHandler:
             date_str = datetime.strftime(date_refresh, "%d %b, %Y")
             hit["source"]["vid_last_refresh"] = date_str
 
+        if "playlist_last_refresh" in hit_keys:
+            playlist_last_refresh = hit["source"]["playlist_last_refresh"]
+            date_refresh = datetime.fromtimestamp(playlist_last_refresh)
+            date_str = datetime.strftime(date_refresh, "%d %b, %Y")
+            hit["source"]["playlist_last_refresh"] = date_str
+
         if "vid_thumb_url" in hit_keys:
             youtube_id = hit["source"]["youtube_id"]
             thumb_path = ThumbManager().vid_thumb_path(youtube_id)
@@ -147,90 +143,61 @@ class SearchForm:
     """build query from search form data"""
 
     CONFIG = AppConfig().config
-    ES_URL = CONFIG["application"]["es_url"]
 
-    def search_channels(self, search_query):
-        """fancy searching channels as you type"""
-        url = self.ES_URL + "/ta_channel/_search"
+    def multi_search(self, search_query):
+        """searching through index"""
+        path = "ta_video,ta_channel,ta_playlist/_search"
         data = {
-            "size": 10,
+            "size": 30,
             "query": {
                 "multi_match": {
                     "query": search_query,
                     "type": "bool_prefix",
+                    "operator": "and",
+                    "fuzziness": "auto",
                     "fields": [
-                        "channel_name.search_as_you_type",
+                        "category",
+                        "channel_description",
                         "channel_name._2gram",
                         "channel_name._3gram",
+                        "channel_name.search_as_you_type",
+                        "playlist_description",
+                        "playlist_name._2gram",
+                        "playlist_name._3gram",
+                        "playlist_name.search_as_you_type",
+                        "tags",
+                        "title._2gram",
+                        "title._3gram",
+                        "title.search_as_you_type",
                     ],
                 }
             },
         }
-        look_up = SearchHandler(url, data)
+        look_up = SearchHandler(path, config=self.CONFIG, data=data)
         search_results = look_up.get_data()
-        return {"results": search_results}
+        all_results = self.build_results(search_results)
+
+        return {"results": all_results}
 
     @staticmethod
-    def search_videos():
-        """searching for videos"""
-        # TBD palceholder for now
-        return False
+    def build_results(search_results):
+        """build the all_results dict"""
+        video_results = []
+        channel_results = []
+        playlist_results = []
+        if search_results:
+            for result in search_results:
+                if result["_index"] == "ta_video":
+                    video_results.append(result)
+                elif result["_index"] == "ta_channel":
+                    channel_results.append(result)
+                elif result["_index"] == "ta_playlist":
+                    playlist_results.append(result)
 
-
-class Pagination:
-    """
-    figure out the pagination based on page size and total_hits
-    """
-
-    def __init__(self, page_get, user_id, search_get=False):
-        self.user_id = user_id
-        self.page_size = self.get_page_size()
-        self.page_get = page_get
-        self.search_get = search_get
-        self.pagination = self.first_guess()
-
-    def get_page_size(self):
-        """get default or user modified page_size"""
-        key = f"{self.user_id}:page_size"
-        page_size = RedisArchivist().get_message(key)["status"]
-        if not page_size:
-            config = AppConfig().config
-            page_size = config["archive"]["page_size"]
-
-        return page_size
-
-    def first_guess(self):
-        """build first guess before api call"""
-        page_get = self.page_get
-        if page_get in [0, 1]:
-            page_from = 0
-            prev_pages = False
-        elif page_get > 1:
-            page_from = (page_get - 1) * self.page_size
-            prev_pages = [
-                i for i in range(page_get - 1, page_get - 6, -1) if i > 1
-            ]
-            prev_pages.reverse()
-        pagination = {
-            "page_size": self.page_size,
-            "page_from": page_from,
-            "prev_pages": prev_pages,
-            "current_page": page_get,
+        all_results = {
+            "video_results": video_results,
+            "channel_results": channel_results,
+            "playlist_results": playlist_results,
         }
-        if self.search_get:
-            pagination.update({"search_get": self.search_get})
-        return pagination
 
-    def validate(self, total_hits):
-        """validate pagination with total_hits after making api call"""
-        page_get = self.page_get
-        max_pages = math.ceil(total_hits / self.page_size)
-        if page_get < max_pages and max_pages > 1:
-            self.pagination["last_page"] = max_pages
-        else:
-            self.pagination["last_page"] = False
-        next_pages = [
-            i for i in range(page_get + 1, page_get + 6) if 1 < i < max_pages
-        ]
-
-        self.pagination["next_pages"] = next_pages
+        return all_results

tubearchivist/home/src/frontend/watched.py

@@ -0,0 +1,98 @@
+"""
+functionality:
+- handle watched state for videos, channels and playlists
+"""
+
+from datetime import datetime
+
+from home.src.es.connect import ElasticWrap
+from home.src.ta.helper import UrlListParser
+
+
+class WatchState:
+    """handle watched checkbox for videos and channels"""
+
+    def __init__(self, youtube_id):
+        self.youtube_id = youtube_id
+        self.stamp = int(datetime.now().strftime("%s"))
+
+    def mark_as_watched(self):
+        """update es with new watched value"""
+        url_type = self.dedect_type()
+        if url_type == "video":
+            self.mark_vid_watched()
+        elif url_type == "channel":
+            self.mark_channel_watched()
+        elif url_type == "playlist":
+            self.mark_playlist_watched()
+
+        print(f"{self.youtube_id}: marked as watched")
+
+    def mark_as_unwatched(self):
+        """revert watched state to false"""
+        url_type = self.dedect_type()
+        if url_type == "video":
+            self.mark_vid_watched(revert=True)
+
+        print(f"{self.youtube_id}: revert as unwatched")
+
+    def dedect_type(self):
+        """find youtube id type"""
+        print(self.youtube_id)
+        url_process = UrlListParser(self.youtube_id).process_list()
+        url_type = url_process[0]["type"]
+        return url_type
+
+    def mark_vid_watched(self, revert=False):
+        """change watched status of single video"""
+        path = f"ta_video/_update/{self.youtube_id}"
+        data = {
+            "doc": {"player": {"watched": True, "watched_date": self.stamp}}
+        }
+        if revert:
+            data["doc"]["player"]["watched"] = False
+
+        response, status_code = ElasticWrap(path).post(data=data)
+        if status_code != 200:
+            print(response)
+            raise ValueError("failed to mark video as watched")
+
+    def mark_channel_watched(self):
+        """change watched status of every video in channel"""
+        path = "ta_video/_update_by_query"
+        must_list = [
+            {"term": {"channel.channel_id": {"value": self.youtube_id}}},
+            {"term": {"player.watched": {"value": False}}},
+        ]
+        data = {
+            "query": {"bool": {"must": must_list}},
+            "script": {
+                "source": "ctx._source.player['watched'] = true",
+                "lang": "painless",
+            },
+        }
+
+        response, status_code = ElasticWrap(path).post(data=data)
+        if status_code != 200:
+            print(response)
+            raise ValueError("failed mark channel as watched")
+
+    def mark_playlist_watched(self):
+        """change watched state of all videos in playlist"""
+        path = "ta_video/_update_by_query"
+        must_list = [
+            {"term": {"playlist.keyword": {"value": self.youtube_id}}},
+            {"term": {"player.watched": {"value": False}}},
+        ]
+        data = {
+            "query": {"bool": {"must": must_list}},
+            "script": {
+                "source": "ctx._source.player['watched'] = true",
+                "lang": "painless",
+            },
+        }
+
+        response, status_code = ElasticWrap(path).post(data=data)
+        if status_code != 200:
+            print(response)
+            raise ValueError("failed mark playlist as watched")

tubearchivist/home/src/index.py

@@ -1,535 +0,0 @@
-"""
-Functionality:
-- index new videos into elastisearch
-- extract video info with yt_dlp
-- scrape youtube channel page if needed
-"""
-
-import json
-import os
-import re
-from datetime import datetime
-from time import sleep
-
-import requests
-import yt_dlp as youtube_dl
-from bs4 import BeautifulSoup
-from home.src.config import AppConfig
-from home.src.helper import DurationConverter, UrlListParser, clean_string
-from home.src.thumbnails import ThumbManager
-
-
-class YoutubeChannel:
-    """represents a single youtube channel"""
-
-    CONFIG = AppConfig().config
-    ES_URL = CONFIG["application"]["es_url"]
-    ES_AUTH = CONFIG["application"]["es_auth"]
-    CACHE_DIR = CONFIG["application"]["cache_dir"]
-    VIDEOS = CONFIG["application"]["videos"]
-
-    def __init__(self, channel_id):
-        self.channel_id = channel_id
-        self.json_data = None
-        self.source = None
-        self.channel_dict = self.build_channel_dict()
-
-    def build_channel_dict(self, scrape=False):
-        """combine the dicts build from extracted json payload"""
-        if scrape:
-            channel_dict = False
-        else:
-            channel_dict = self.get_es_channel()
-        if not channel_dict:
-            print("scrape data from youtube")
-            self.scrape_channel()
-            channel_dict = self.parse_channel_main()
-            channel_dict.update(self.parse_channel_meta())
-            self.source = "scraped"
-        return channel_dict
-
-    def get_es_channel(self):
-        """get from elastic search first if possible"""
-        channel_id = self.channel_id
-        url = f"{self.ES_URL}/ta_channel/_doc/{channel_id}"
-        response = requests.get(url, auth=self.ES_AUTH)
-        if response.ok:
-            channel_source = response.json()["_source"]
-            self.source = "elastic"
-            return channel_source
-        return False
-
-    def scrape_channel(self):
-        """scrape channel page for additional infos"""
-        channel_id = self.channel_id
-        url = f"https://www.youtube.com/channel/{channel_id}/about?hl=en"
-        cookies = {"CONSENT": "YES+xxxxxxxxxxxxxxxxxxxxxxxxxxx"}
-        response = requests.get(url, cookies=cookies, auth=self.ES_AUTH)
-        if response.ok:
-            channel_page = response.text
-        else:
-            print(f"failed to extract channel info for: {channel_id}")
-            raise ConnectionError
-        soup = BeautifulSoup(channel_page, "html.parser")
-        # load script into json
-        all_scripts = soup.find("body").find_all("script")
-        for script in all_scripts:
-            if "var ytInitialData = " in str(script):
-                script_content = str(script)
-                break
-        # extract payload
-        script_content = script_content.split("var ytInitialData = ")[1]
-        json_raw = script_content.rstrip(";</script>")
-        json_data = json.loads(json_raw)
-        # add to self
-        self.json_data = json_data
-
-    def parse_channel_main(self):
-        """extract maintab values from scraped channel json data"""
-        main_tab = self.json_data["header"]["c4TabbedHeaderRenderer"]
-        channel_name = main_tab["title"]
-        last_refresh = int(datetime.now().strftime("%s"))
-        # channel_subs
-        try:
-            sub_text_simple = main_tab["subscriberCountText"]["simpleText"]
-            sub_text = sub_text_simple.split(" ")[0]
-            if sub_text[-1] == "K":
-                channel_subs = int(float(sub_text.replace("K", "")) * 1000)
-            elif sub_text[-1] == "M":
-                channel_subs = int(float(sub_text.replace("M", "")) * 1000000)
-            elif int(sub_text) >= 0:
-                channel_subs = int(sub_text)
-            else:
-                message = f"{sub_text} not dealt with"
-                print(message)
-        except KeyError:
-            channel_subs = 0
-        # banner
-        try:
-            all_banners = main_tab["banner"]["thumbnails"]
-            banner = sorted(all_banners, key=lambda k: k["width"])[-1]["url"]
-        except KeyError:
-            banner = False
-        # build and return dict
-        main_channel_dict = {
-            "channel_active": True,
-            "channel_last_refresh": last_refresh,
-            "channel_subs": channel_subs,
-            "channel_banner_url": banner,
-            "channel_name": channel_name,
-            "channel_id": self.channel_id,
-        }
-        return main_channel_dict
-
-    def parse_channel_meta(self):
-        """extract meta tab values from channel payload"""
-        # meta tab
-        json_data = self.json_data
-        meta_tab = json_data["metadata"]["channelMetadataRenderer"]
-        description = meta_tab["description"]
-        all_thumbs = meta_tab["avatar"]["thumbnails"]
-        thumb_url = sorted(all_thumbs, key=lambda k: k["width"])[-1]["url"]
-        # stats tab
-        renderer = "twoColumnBrowseResultsRenderer"
-        all_tabs = json_data["contents"][renderer]["tabs"]
-        for tab in all_tabs:
-            if "tabRenderer" in tab.keys():
-                if tab["tabRenderer"]["title"] == "About":
-                    about_tab = tab["tabRenderer"]["content"][
-                        "sectionListRenderer"
-                    ]["contents"][0]["itemSectionRenderer"]["contents"][0][
-                        "channelAboutFullMetadataRenderer"
-                    ]
-                    break
-        try:
-            channel_views_text = about_tab["viewCountText"]["simpleText"]
-            channel_views = int(re.sub(r"\D", "", channel_views_text))
-        except KeyError:
-            channel_views = 0
-
-        meta_channel_dict = {
-            "channel_description": description,
-            "channel_thumb_url": thumb_url,
-            "channel_views": channel_views,
-        }
-
-        return meta_channel_dict
-
-    def get_channel_art(self):
-        """download channel art for new channels"""
-        channel_id = self.channel_id
-        channel_thumb = self.channel_dict["channel_thumb_url"]
-        channel_banner = self.channel_dict["channel_banner_url"]
-        ThumbManager().download_chan(
-            [(channel_id, channel_thumb, channel_banner)]
-        )
-
-    def upload_to_es(self):
-        """upload channel data to elastic search"""
-        url = f"{self.ES_URL}/ta_channel/_doc/{self.channel_id}"
-        response = requests.put(url, json=self.channel_dict, auth=self.ES_AUTH)
-        print(f"added {self.channel_id} to es")
-        if not response.ok:
-            print(response.text)
-
-    def sync_to_videos(self):
-        """sync new channel_dict to all videos of channel"""
-        headers = {"Content-type": "application/json"}
-        channel_id = self.channel_id
-        # add ingest pipeline
-        processors = []
-        for field, value in self.channel_dict.items():
-            line = {"set": {"field": "channel." + field, "value": value}}
-            processors.append(line)
-        data = {"description": channel_id, "processors": processors}
-        payload = json.dumps(data)
-        url = self.ES_URL + "/_ingest/pipeline/" + channel_id
-        request = requests.put(
-            url, data=payload, headers=headers, auth=self.ES_AUTH
-        )
-        if not request.ok:
-            print(request.text)
-        # apply pipeline
-        data = {"query": {"match": {"channel.channel_id": channel_id}}}
-        payload = json.dumps(data)
-        url = self.ES_URL + "/ta_video/_update_by_query?pipeline=" + channel_id
-        request = requests.post(
-            url, data=payload, headers=headers, auth=self.ES_AUTH
-        )
-        if not request.ok:
-            print(request.text)
-
-    def get_folder_path(self):
-        """get folder where media files get stored"""
-        channel_name = self.channel_dict["channel_name"]
-        folder_name = clean_string(channel_name)
-        folder_path = os.path.join(self.VIDEOS, folder_name)
-        return folder_path
-
-    def delete_es_videos(self):
-        """delete all channel documents from elasticsearch"""
-        headers = {"Content-type": "application/json"}
-        data = {
-            "query": {
-                "term": {"channel.channel_id": {"value": self.channel_id}}
-            }
-        }
-        payload = json.dumps(data)
-        url = self.ES_URL + "/ta_video/_delete_by_query"
-        response = requests.post(
-            url, data=payload, headers=headers, auth=self.ES_AUTH
-        )
-        if not response.ok:
-            print(response.text)
-
-    def delete_channel(self):
-        """delete channel and all videos"""
-        print(f"deleting {self.channel_id} and all matching media files")
-        folder_path = self.get_folder_path()
-        print("delete all media files")
-        try:
-            all_videos = os.listdir(folder_path)
-            for video in all_videos:
-                video_path = os.path.join(folder_path, video)
-                os.remove(video_path)
-            os.rmdir(folder_path)
-        except FileNotFoundError:
-            print(f"no videos found for {folder_path}")
-
-        ThumbManager().delete_chan_thumb(self.channel_id)
-
-        print("delete indexed videos")
-        self.delete_es_videos()
-        url = self.ES_URL + "/ta_channel/_doc/" + self.channel_id
-        response = requests.delete(url, auth=self.ES_AUTH)
-        if not response.ok:
-            print(response.text)
-
-
-class YoutubeVideo:
-    """represents a single youtube video"""
-
-    CONFIG = AppConfig().config
-    ES_URL = CONFIG["application"]["es_url"]
-    ES_AUTH = CONFIG["application"]["es_auth"]
-    CACHE_DIR = CONFIG["application"]["cache_dir"]
-    VIDEOS = CONFIG["application"]["videos"]
-
-    def __init__(self, youtube_id):
-        self.youtube_id = youtube_id
-        self.channel_id = None
-        self.vid_dict = None
-
-    def get_vid_dict(self):
-        """wrapper to loop around youtube_dl to retry on failure"""
-        print(f"get video data for {self.youtube_id}")
-        vid_dict = False
-        for i in range(3):
-            try:
-                vid_dict = self.get_youtubedl_vid_data()
-            except KeyError as e:
-                print(e)
-                sleep((i + 1) ** 2)
-                continue
-            else:
-                break
-
-        self.vid_dict = vid_dict
-
-    def get_youtubedl_vid_data(self):
-        """parse youtubedl extract info"""
-        youtube_id = self.youtube_id
-        obs = {
-            "quiet": True,
-            "default_search": "ytsearch",
-            "skip_download": True,
-        }
-        try:
-            vid = youtube_dl.YoutubeDL(obs).extract_info(youtube_id)
-        except (
-            youtube_dl.utils.ExtractorError,
-            youtube_dl.utils.DownloadError,
-        ):
-            print("failed to get info for " + youtube_id)
-            return False
-        # extract
-        self.channel_id = vid["channel_id"]
-        upload_date = vid["upload_date"]
-        upload_date_time = datetime.strptime(upload_date, "%Y%m%d")
-        published = upload_date_time.strftime("%Y-%m-%d")
-        last_refresh = int(datetime.now().strftime("%s"))
-        # likes
-        try:
-            like_count = vid["like_count"]
-        except KeyError:
-            like_count = 0
-        try:
-            dislike_count = vid["dislike_count"]
-        except KeyError:
-            dislike_count = 0
-        # build dicts
-        stats = {
-            "view_count": vid["view_count"],
-            "like_count": like_count,
-            "dislike_count": dislike_count,
-            "average_rating": vid["average_rating"],
-        }
-        vid_basic = {
-            "title": vid["title"],
-            "description": vid["description"],
-            "category": vid["categories"],
-            "vid_thumb_url": vid["thumbnail"],
-            "tags": vid["tags"],
-            "published": published,
-            "stats": stats,
-            "vid_last_refresh": last_refresh,
-            "date_downloaded": last_refresh,
-            "youtube_id": youtube_id,
-            "active": True,
-            "channel": False,
-        }
-
-        return vid_basic
-
-    def add_player(self, missing_vid):
-        """add player information for new videos"""
-        cache_path = self.CACHE_DIR + "/download/"
-        videos = self.VIDEOS
-
-        if missing_vid:
-            # coming from scan_filesystem
-            channel_name, file_name, _ = missing_vid
-            vid_path = os.path.join(videos, channel_name, file_name)
-        else:
-            # coming from VideoDownload
-            all_cached = os.listdir(cache_path)
-            for file_cached in all_cached:
-                if self.youtube_id in file_cached:
-                    vid_path = os.path.join(cache_path, file_cached)
-                    break
-
-        duration_handler = DurationConverter()
-        duration = duration_handler.get_sec(vid_path)
-        duration_str = duration_handler.get_str(duration)
-        player = {
-            "watched": False,
-            "duration": duration,
-            "duration_str": duration_str,
-        }
-        self.vid_dict["player"] = player
-
-    def build_file_path(self, channel_name):
-        """build media_url from where file will be located"""
-        clean_channel_name = clean_string(channel_name)
-        timestamp = self.vid_dict["published"].replace("-", "")
-        youtube_id = self.vid_dict["youtube_id"]
-        title = self.vid_dict["title"]
-        clean_title = clean_string(title)
-        filename = f"{timestamp}_{youtube_id}_{clean_title}.mp4"
-        media_url = os.path.join(clean_channel_name, filename)
-        self.vid_dict["media_url"] = media_url
-
-    def get_es_data(self):
-        """get current data from elastic search"""
-        url = self.ES_URL + "/ta_video/_doc/" + self.youtube_id
-        response = requests.get(url, auth=self.ES_AUTH)
-        if not response.ok:
-            print(response.text)
-        es_vid_dict = json.loads(response.text)
-        return es_vid_dict
-
-    def upload_to_es(self):
-        """upload channel data to elastic search"""
-        url = f"{self.ES_URL}/ta_video/_doc/{self.youtube_id}"
-        response = requests.put(url, json=self.vid_dict, auth=self.ES_AUTH)
-        if not response.ok:
-            print(response.text)
-
-    def deactivate(self):
-        """deactivate document on extractor error"""
-        youtube_id = self.youtube_id
-        headers = {"Content-type": "application/json"}
-        url = f"{self.ES_URL}/ta_video/_update/{youtube_id}"
-        data = {"script": "ctx._source.active = false"}
-        json_str = json.dumps(data)
-        response = requests.post(
-            url, data=json_str, headers=headers, auth=self.ES_AUTH
-        )
-        print(f"deactivated {youtube_id}")
-        if not response.ok:
-            print(response.text)
-
-    def delete_media_file(self):
-        """delete video file, meta data, thumbnails"""
-        # delete media file
-        es_vid_dict = self.get_es_data()
-        media_url = es_vid_dict["_source"]["media_url"]
-        print(f"delete {media_url} from file system")
-        to_delete = os.path.join(self.VIDEOS, media_url)
-        os.remove(to_delete)
-        # delete from index
-        url = f"{self.ES_URL}/ta_video/_doc/{self.youtube_id}"
-        response = requests.delete(url, auth=self.ES_AUTH)
-        if not response.ok:
-            print(response.text)
-        # delete thumbs from cache
-        ThumbManager().delete_vid_thumb(self.youtube_id)
-
-
-class WatchState:
-    """handle watched checkbox for videos and channels"""
-
-    CONFIG = AppConfig().config
-    ES_URL = CONFIG["application"]["es_url"]
-    ES_AUTH = CONFIG["application"]["es_auth"]
-    HEADERS = {"Content-type": "application/json"}
-
-    def __init__(self, youtube_id):
-        self.youtube_id = youtube_id
-        self.stamp = int(datetime.now().strftime("%s"))
-
-    def mark_as_watched(self):
-        """update es with new watched value"""
-        url_type = self.dedect_type()
-        if url_type == "video":
-            self.mark_vid_watched()
-        elif url_type == "channel":
-            self.mark_channel_watched()
-
-        print(f"marked {self.youtube_id} as watched")
-
-    def mark_as_unwatched(self):
-        """revert watched state to false"""
-        url_type = self.dedect_type()
-        if url_type == "video":
-            self.mark_vid_watched(revert=True)
-        elif url_type == "channel":
-            self.mark_channel_watched(revert=True)
-
-        print(f"revert {self.youtube_id} as unwatched")
-
-    def dedect_type(self):
-        """find youtube id type"""
-        url_process = UrlListParser(self.youtube_id).process_list()
-        url_type = url_process[0]["type"]
-        return url_type
-
-    def mark_vid_watched(self, revert=False):
-        """change watched status of single video"""
-        url = self.ES_URL + "/ta_video/_update/" + self.youtube_id
-        data = {
-            "doc": {"player": {"watched": True, "watched_date": self.stamp}}
-        }
-        if revert:
-            data["doc"]["player"]["watched"] = False
-
-        payload = json.dumps(data)
-        request = requests.post(
-            url, data=payload, headers=self.HEADERS, auth=self.ES_AUTH
-        )
-        if not request.ok:
-            print(request.text)
-
-    def mark_channel_watched(self, revert=False):
-        """change watched status of every video in channel"""
-        es_url = self.ES_URL
-        headers = self.HEADERS
-        youtube_id = self.youtube_id
-        # create pipeline
-        data = {
-            "description": youtube_id,
-            "processors": [
-                {"set": {"field": "player.watched", "value": True}},
-                {"set": {"field": "player.watched_date", "value": self.stamp}},
-            ],
-        }
-        if revert:
-            data["processors"][0]["set"]["value"] = False
-
-        payload = json.dumps(data)
-        url = f"{es_url}/_ingest/pipeline/{youtube_id}"
-        request = requests.put(
-            url, data=payload, headers=headers, auth=self.ES_AUTH
-        )
-        if not request.ok:
-            print(request.text)
-            raise ValueError("failed to post ingest pipeline")
-
-        # apply pipeline
-        must_list = [
-            {"term": {"channel.channel_id": {"value": youtube_id}}},
-            {"term": {"player.watched": {"value": False}}},
-        ]
-        data = {"query": {"bool": {"must": must_list}}}
-        payload = json.dumps(data)
-        url = f"{es_url}/ta_video/_update_by_query?pipeline={youtube_id}"
-        request = requests.post(
-            url, data=payload, headers=headers, auth=self.ES_AUTH
-        )
-        if not request.ok:
-            print(request.text)
-
-
-def index_new_video(youtube_id, missing_vid=False):
-    """combine video and channel classes for new video index"""
-    vid_handler = YoutubeVideo(youtube_id)
-    vid_handler.get_vid_dict()
-    if not vid_handler.vid_dict:
-        raise ValueError("failed to get metadata for " + youtube_id)
-
-    channel_handler = YoutubeChannel(vid_handler.channel_id)
-    # add filepath to vid_dict
-    channel_name = channel_handler.channel_dict["channel_name"]
-    vid_handler.build_file_path(channel_name)
-    # add channel and player to video
-    vid_handler.add_player(missing_vid)
-    vid_handler.vid_dict["channel"] = channel_handler.channel_dict
-    # add new channel to es
-    if channel_handler.source == "scraped":
-        channel_handler.channel_dict["channel_subscribed"] = False
-        channel_handler.upload_to_es()
-        channel_handler.get_channel_art()
-    # upload video to es
-    vid_handler.upload_to_es()
-    # return vid_dict for further processing
-    return vid_handler.vid_dict

tubearchivist/home/src/index/channel.py

@@ -0,0 +1,375 @@
+"""
+functionality:
+- get metadata from youtube for a channel
+- index and update in es
+"""
+
+import json
+import os
+import re
+from datetime import datetime
+
+import requests
+import yt_dlp
+from bs4 import BeautifulSoup
+from home.src.download import queue  # partial import
+from home.src.download.thumbnails import ThumbManager
+from home.src.es.connect import ElasticWrap, IndexPaginate
+from home.src.index.generic import YouTubeItem
+from home.src.index.playlist import YoutubePlaylist
+from home.src.ta.helper import clean_string, requests_headers
+from home.src.ta.ta_redis import RedisArchivist
+
+
+class ChannelScraper:
+    """custom scraper using bs4 to scrape channel about page
+    will be able to be integrated into yt-dlp
+    once #2237 and #2350 are merged upstream
+    """
+
+    def __init__(self, channel_id):
+        self.channel_id = channel_id
+        self.soup = False
+        self.yt_json = False
+        self.json_data = False
+
+    def get_json(self):
+        """main method to return channel dict"""
+        self.get_soup()
+        self._extract_yt_json()
+        self._parse_channel_main()
+        self._parse_channel_meta()
+        return self.json_data
+
+    def get_soup(self):
+        """return soup from youtube"""
+        print(f"{self.channel_id}: scrape channel data from youtube")
+        url = f"https://www.youtube.com/channel/{self.channel_id}/about?hl=en"
+        cookies = {"CONSENT": "YES+xxxxxxxxxxxxxxxxxxxxxxxxxxx"}
+        response = requests.get(
+            url, cookies=cookies, headers=requests_headers()
+        )
+        if response.ok:
+            channel_page = response.text
+        else:
+            print(f"{self.channel_id}: failed to extract channel info")
+            raise ConnectionError
+        self.soup = BeautifulSoup(channel_page, "html.parser")
+
+    def _extract_yt_json(self):
+        """parse soup and get ytInitialData json"""
+        all_scripts = self.soup.find("body").find_all("script")
+        for script in all_scripts:
+            if "var ytInitialData = " in str(script):
+                script_content = str(script)
+                break
+        # extract payload
+        script_content = script_content.split("var ytInitialData = ")[1]
+        json_raw = script_content.rstrip(";</script>")
+        self.yt_json = json.loads(json_raw)
+
+    def _parse_channel_main(self):
+        """extract maintab values from scraped channel json data"""
+        main_tab = self.yt_json["header"]["c4TabbedHeaderRenderer"]
+        # build and return dict
+        self.json_data = {
+            "channel_active": True,
+            "channel_last_refresh": int(datetime.now().strftime("%s")),
+            "channel_subs": self._get_channel_subs(main_tab),
+            "channel_name": main_tab["title"],
+            "channel_banner_url": self._get_thumbnails(main_tab, "banner"),
+            "channel_tvart_url": self._get_thumbnails(main_tab, "tvBanner"),
+            "channel_id": self.channel_id,
+            "channel_subscribed": False,
+        }
+
+    @staticmethod
+    def _get_thumbnails(main_tab, thumb_name):
+        """extract banner url from main_tab"""
+        try:
+            all_banners = main_tab[thumb_name]["thumbnails"]
+            banner = sorted(all_banners, key=lambda k: k["width"])[-1]["url"]
+        except KeyError:
+            banner = False
+
+        return banner
+
+    @staticmethod
+    def _get_channel_subs(main_tab):
+        """process main_tab to get channel subs as int"""
+        try:
+            sub_text_simple = main_tab["subscriberCountText"]["simpleText"]
+            sub_text = sub_text_simple.split(" ")[0]
+            if sub_text[-1] == "K":
+                channel_subs = int(float(sub_text.replace("K", "")) * 1000)
+            elif sub_text[-1] == "M":
+                channel_subs = int(float(sub_text.replace("M", "")) * 1000000)
+            elif int(sub_text) >= 0:
+                channel_subs = int(sub_text)
+            else:
+                message = f"{sub_text} not dealt with"
+                print(message)
+        except KeyError:
+            channel_subs = 0
+
+        return channel_subs
+
+    def _parse_channel_meta(self):
+        """extract meta tab values from channel payload"""
+        # meta tab
+        meta_tab = self.yt_json["metadata"]["channelMetadataRenderer"]
+        all_thumbs = meta_tab["avatar"]["thumbnails"]
+        thumb_url = sorted(all_thumbs, key=lambda k: k["width"])[-1]["url"]
+        # stats tab
+        renderer = "twoColumnBrowseResultsRenderer"
+        all_tabs = self.yt_json["contents"][renderer]["tabs"]
+        for tab in all_tabs:
+            if "tabRenderer" in tab.keys():
+                if tab["tabRenderer"]["title"] == "About":
+                    about_tab = tab["tabRenderer"]["content"][
+                        "sectionListRenderer"
+                    ]["contents"][0]["itemSectionRenderer"]["contents"][0][
+                        "channelAboutFullMetadataRenderer"
+                    ]
+                    break
+        try:
+            channel_views_text = about_tab["viewCountText"]["simpleText"]
+            channel_views = int(re.sub(r"\D", "", channel_views_text))
+        except KeyError:
+            channel_views = 0
+
+        self.json_data.update(
+            {
+                "channel_description": meta_tab["description"],
+                "channel_thumb_url": thumb_url,
+                "channel_views": channel_views,
+            }
+        )
+
+
+class YoutubeChannel(YouTubeItem):
+    """represents a single youtube channel"""
+
+    es_path = False
+    index_name = "ta_channel"
+    yt_base = "https://www.youtube.com/channel/"
+
+    def __init__(self, youtube_id):
+        super().__init__(youtube_id)
+        self.es_path = f"{self.index_name}/_doc/{youtube_id}"
+        self.all_playlists = False
+
+    def build_json(self, upload=False):
+        """get from es or from youtube"""
+        self.get_from_es()
+        if self.json_data:
+            return
+
+        self.get_from_youtube()
+        if upload:
+            self.upload_to_es()
+        return
+
+    def get_from_youtube(self):
+        """use bs4 to scrape channel about page"""
+        self.json_data = ChannelScraper(self.youtube_id).get_json()
+        self.get_channel_art()
+
+    def get_channel_art(self):
+        """download channel art for new channels"""
+        channel_id = self.youtube_id
+        channel_thumb = self.json_data["channel_thumb_url"]
+        channel_banner = self.json_data["channel_banner_url"]
+        ThumbManager().download_chan(
+            [(channel_id, channel_thumb, channel_banner)]
+        )
+
+    def sync_to_videos(self):
+        """sync new channel_dict to all videos of channel"""
+        # add ingest pipeline
+        processors = []
+        for field, value in self.json_data.items():
+            line = {"set": {"field": "channel." + field, "value": value}}
+            processors.append(line)
+        data = {"description": self.youtube_id, "processors": processors}
+        ingest_path = f"_ingest/pipeline/{self.youtube_id}"
+        _, _ = ElasticWrap(ingest_path).put(data)
+        # apply pipeline
+        data = {"query": {"match": {"channel.channel_id": self.youtube_id}}}
+        update_path = f"ta_video/_update_by_query?pipeline={self.youtube_id}"
+        _, _ = ElasticWrap(update_path).post(data)
+
+    def get_folder_path(self):
+        """get folder where media files get stored"""
+        channel_name = self.json_data["channel_name"]
+        folder_name = clean_string(channel_name)
+        if len(folder_name) <= 3:
+            # fall back to channel id
+            folder_name = self.json_data["channel_id"]
+        folder_path = os.path.join(self.app_conf["videos"], folder_name)
+        return folder_path
+
+    def delete_es_videos(self):
+        """delete all channel documents from elasticsearch"""
+        data = {
+            "query": {
+                "term": {"channel.channel_id": {"value": self.youtube_id}}
+            }
+        }
+        _, _ = ElasticWrap("ta_video/_delete_by_query").post(data)
+
+    def delete_playlists(self):
+        """delete all indexed playlist from es"""
+        all_playlists = self.get_indexed_playlists()
+        for playlist in all_playlists:
+            playlist_id = playlist["playlist_id"]
+            YoutubePlaylist(playlist_id).delete_metadata()
+
+    def delete_channel(self):
+        """delete channel and all videos"""
+        print(f"{self.youtube_id}: delete channel")
+        self.get_from_es()
+        folder_path = self.get_folder_path()
+        print(f"{self.youtube_id}: delete all media files")
+        try:
+            all_videos = os.listdir(folder_path)
+            for video in all_videos:
+                video_path = os.path.join(folder_path, video)
+                os.remove(video_path)
+            os.rmdir(folder_path)
+        except FileNotFoundError:
+            print(f"no videos found for {folder_path}")
+
+        print(f"{self.youtube_id}: delete indexed playlists")
+        self.delete_playlists()
+        print(f"{self.youtube_id}: delete indexed videos")
+        self.delete_es_videos()
+        self.del_in_es()
+
+    def index_channel_playlists(self):
+        """add all playlists of channel to index"""
+        print(f"{self.youtube_id}: index all playlists")
+        self.get_from_es()
+        channel_name = self.json_data["channel_name"]
+        mess_dict = {
+            "status": "message:playlistscan",
+            "level": "info",
+            "title": "Looking for playlists",
+            "message": f"{channel_name}: Scanning channel in progress",
+        }
+        RedisArchivist().set_message("message:playlistscan", mess_dict)
+        self.get_all_playlists()
+        if not self.all_playlists:
+            print(f"{self.youtube_id}: no playlists found.")
+            return
+
+        all_youtube_ids = self.get_all_video_ids()
+        for idx, playlist in enumerate(self.all_playlists):
+            self._notify_single_playlist(idx, playlist)
+            self._index_single_playlist(playlist, all_youtube_ids)
+
+    def _notify_single_playlist(self, idx, playlist):
+        """send notification"""
+        channel_name = self.json_data["channel_name"]
+        mess_dict = {
+            "status": "message:playlistscan",
+            "level": "info",
+            "title": f"{channel_name}: Scanning channel for playlists",
+            "message": f"Progress: {idx + 1}/{len(self.all_playlists)}",
+        }
+        RedisArchivist().set_message("message:playlistscan", mess_dict)
+        print("add playlist: " + playlist[1])
+
+    @staticmethod
+    def _index_single_playlist(playlist, all_youtube_ids):
+        """add single playlist if needed"""
+        playlist = YoutubePlaylist(playlist[0])
+        playlist.all_youtube_ids = all_youtube_ids
+        playlist.build_json()
+        if not playlist.json_data:
+            return
+
+        entries = playlist.json_data["playlist_entries"]
+        downloaded = [i for i in entries if i["downloaded"]]
+        if not downloaded:
+            return
+
+        playlist.upload_to_es()
+        playlist.add_vids_to_playlist()
+        playlist.get_playlist_art()
+
+    @staticmethod
+    def get_all_video_ids():
+        """match all playlists with videos"""
+        handler = queue.PendingList()
+        handler.get_download()
+        handler.get_indexed()
+        all_youtube_ids = [i["youtube_id"] for i in handler.all_videos]
+
+        return all_youtube_ids
+
+    def get_all_playlists(self):
+        """get all playlists owned by this channel"""
+        url = (
+            f"https://www.youtube.com/channel/{self.youtube_id}"
+            + "/playlists?view=1&sort=dd&shelf_id=0"
+        )
+        obs = {
+            "quiet": True,
+            "skip_download": True,
+            "extract_flat": True,
+        }
+        playlists = yt_dlp.YoutubeDL(obs).extract_info(url)
+        all_entries = [(i["id"], i["title"]) for i in playlists["entries"]]
+        self.all_playlists = all_entries
+
+    def get_indexed_playlists(self):
+        """get all indexed playlists from channel"""
+        data = {
+            "query": {
+                "term": {"playlist_channel_id": {"value": self.youtube_id}}
+            },
+            "sort": [{"playlist_channel.keyword": {"order": "desc"}}],
+        }
+        all_playlists = IndexPaginate("ta_playlist", data).get_results()
+        return all_playlists
+
+    def get_overwrites(self):
+        """get all per channel overwrites"""
+        return self.json_data.get("channel_overwrites", False)
+
+    def set_overwrites(self, overwrites):
+        """set per channel overwrites"""
+        valid_keys = [
+            "download_format",
+            "autodelete_days",
+            "index_playlists",
+            "integrate_sponsorblock",
+        ]
+
+        to_write = self.json_data.get("channel_overwrites", {})
+        for key, value in overwrites.items():
+            if key not in valid_keys:
+                raise ValueError(f"invalid overwrite key: {key}")
+            if value == "disable":
+                to_write[key] = False
+                continue
+            if value in [0, "0"]:
+                del to_write[key]
+                continue
+            if value == "1":
+                to_write[key] = True
+                continue
+            if value:
+                to_write.update({key: value})
+
+        self.json_data["channel_overwrites"] = to_write
+
+
+def channel_overwrites(channel_id, overwrites):
+    """collection to overwrite settings per channel"""
+    channel = YoutubeChannel(channel_id)
+    channel.build_json()
+    channel.set_overwrites(overwrites)
+    channel.upload_to_es()
+    channel.sync_to_videos()

tubearchivist/home/src/index/filesystem.py

@@ -10,212 +10,22 @@ import os
 import re
 import shutil
 import subprocess
-from datetime import datetime
-from math import ceil
-from time import sleep
 
-import requests
-from home.src.config import AppConfig
-from home.src.download import ChannelSubscription, PendingList, VideoDownloader
-from home.src.helper import (
-    RedisArchivist,
-    clean_string,
-    get_total_hits,
-    ignore_filelist,
-)
-from home.src.index import YoutubeChannel, YoutubeVideo, index_new_video
-from home.src.thumbnails import ThumbManager
-
-
-class Reindex:
-    """check for outdated documents and refresh data from youtube"""
-
-    def __init__(self):
-        # config
-        config = AppConfig().config
-        self.sleep_interval = config["downloads"]["sleep_interval"]
-        self.es_url = config["application"]["es_url"]
-        self.es_auth = config["application"]["es_auth"]
-        self.refresh_interval = 90
-        # scan
-        self.video_daily, self.channel_daily = self.get_daily()
-        self.all_youtube_ids = False
-        self.all_channel_ids = False
-
-    def get_daily(self):
-        """get daily refresh values"""
-        total_videos = get_total_hits(
-            "ta_video", self.es_url, self.es_auth, "active"
-        )
-        video_daily = ceil(total_videos / self.refresh_interval * 1.2)
-        total_channels = get_total_hits(
-            "ta_channel", self.es_url, self.es_auth, "channel_active"
-        )
-        channel_daily = ceil(total_channels / self.refresh_interval * 1.2)
-        return (video_daily, channel_daily)
-
-    def get_outdated_vids(self):
-        """get daily videos to refresh"""
-        headers = {"Content-type": "application/json"}
-        now = int(datetime.now().strftime("%s"))
-        now_3m = now - 3 * 30 * 24 * 60 * 60
-        size = self.video_daily
-        data = {
-            "size": size,
-            "query": {
-                "bool": {
-                    "must": [
-                        {"match": {"active": True}},
-                        {"range": {"vid_last_refresh": {"lte": now_3m}}},
-                    ]
-                }
-            },
-            "sort": [{"vid_last_refresh": {"order": "asc"}}],
-            "_source": False,
-        }
-        query_str = json.dumps(data)
-        url = self.es_url + "/ta_video/_search"
-        response = requests.get(
-            url, data=query_str, headers=headers, auth=self.es_auth
-        )
-        if not response.ok:
-            print(response.text)
-        response_dict = json.loads(response.text)
-        all_youtube_ids = [i["_id"] for i in response_dict["hits"]["hits"]]
-        return all_youtube_ids
-
-    def get_outdated_channels(self):
-        """get daily channels to refresh"""
-        headers = {"Content-type": "application/json"}
-        now = int(datetime.now().strftime("%s"))
-        now_3m = now - 3 * 30 * 24 * 60 * 60
-        size = self.channel_daily
-        data = {
-            "size": size,
-            "query": {
-                "bool": {
-                    "must": [
-                        {"match": {"channel_active": True}},
-                        {"range": {"channel_last_refresh": {"lte": now_3m}}},
-                    ]
-                }
-            },
-            "sort": [{"channel_last_refresh": {"order": "asc"}}],
-            "_source": False,
-        }
-        query_str = json.dumps(data)
-        url = self.es_url + "/ta_channel/_search"
-        response = requests.get(
-            url, data=query_str, headers=headers, auth=self.es_auth
-        )
-        if not response.ok:
-            print(response.text)
-        response_dict = json.loads(response.text)
-        all_channel_ids = [i["_id"] for i in response_dict["hits"]["hits"]]
-        return all_channel_ids
-
-    def check_outdated(self):
-        """add missing vids and channels"""
-        self.all_youtube_ids = self.get_outdated_vids()
-        self.all_channel_ids = self.get_outdated_channels()
-
-    def rescrape_all_channels(self):
-        """sync new data from channel to all matching videos"""
-        sleep_interval = self.sleep_interval
-        channel_sub_handler = ChannelSubscription()
-        all_channels = channel_sub_handler.get_channels(subscribed_only=False)
-        all_channel_ids = [i["channel_id"] for i in all_channels]
-
-        counter = 1
-        for channel_id in all_channel_ids:
-            message = f"Progress: {counter}/{len(all_channels)}"
-            mess_dict = {
-                "status": "scraping",
-                "level": "info",
-                "title": "Scraping all youtube channels",
-                "message": message,
-            }
-            RedisArchivist().set_message("progress:download", mess_dict)
-            channel_index = YoutubeChannel(channel_id)
-            subscribed = channel_index.channel_dict["channel_subscribed"]
-            channel_index.channel_dict = channel_index.build_channel_dict(
-                scrape=True
-            )
-            channel_index.channel_dict["channel_subscribed"] = subscribed
-            channel_index.upload_to_es()
-            channel_index.sync_to_videos()
-            counter = counter + 1
-            if sleep_interval:
-                sleep(sleep_interval)
-
-    @staticmethod
-    def reindex_single_video(youtube_id):
-        """refresh data for single video"""
-        vid_handler = YoutubeVideo(youtube_id)
-        vid_handler.get_vid_dict()
-        if not vid_handler.vid_dict:
-            # stop if deactivated
-            vid_handler.deactivate()
-            return
-
-        es_vid_dict = vid_handler.get_es_data()
-        player = es_vid_dict["_source"]["player"]
-        date_downloaded = es_vid_dict["_source"]["date_downloaded"]
-        channel_dict = es_vid_dict["_source"]["channel"]
-        channel_name = channel_dict["channel_name"]
-        vid_handler.build_file_path(channel_name)
-        # add to vid_dict
-        vid_handler.vid_dict["player"] = player
-        vid_handler.vid_dict["date_downloaded"] = date_downloaded
-        vid_handler.vid_dict["channel"] = channel_dict
-        # update
-        vid_handler.upload_to_es()
-        thumb_handler = ThumbManager()
-        thumb_handler.delete_vid_thumb(youtube_id)
-        to_download = (youtube_id, vid_handler.vid_dict["vid_thumb_url"])
-        thumb_handler.download_vid([to_download], notify=False)
-
-    @staticmethod
-    def reindex_single_channel(channel_id):
-        """refresh channel data and sync to videos"""
-        channel_handler = YoutubeChannel(channel_id)
-        subscribed = channel_handler.channel_dict["channel_subscribed"]
-        channel_handler.channel_dict = channel_handler.build_channel_dict(
-            scrape=True
-        )
-        channel_handler.channel_dict["channel_subscribed"] = subscribed
-        # update
-        channel_handler.upload_to_es()
-        channel_handler.sync_to_videos()
-        thumb_handler = ThumbManager()
-        thumb_handler.delete_chan_thumb(channel_id)
-        channel_thumb = channel_handler.channel_dict["channel_thumb_url"]
-        channel_banner = channel_handler.channel_dict["channel_banner_url"]
-        to_download = (channel_id, channel_thumb, channel_banner)
-        thumb_handler.download_chan([to_download])
-
-    def reindex(self):
-        """reindex what's needed"""
-        # videos
-        print(f"reindexing {len(self.all_youtube_ids)} videos")
-        for youtube_id in self.all_youtube_ids:
-            self.reindex_single_video(youtube_id)
-            if self.sleep_interval:
-                sleep(self.sleep_interval)
-        # channels
-        print(f"reindexing {len(self.all_channel_ids)} channels")
-        for channel_id in self.all_channel_ids:
-            self.reindex_single_channel(channel_id)
-            if self.sleep_interval:
-                sleep(self.sleep_interval)
+from home.src.download.queue import PendingList
+from home.src.download.yt_cookie import CookieHandler
+from home.src.download.yt_dlp_handler import VideoDownloader
+from home.src.es.connect import ElasticWrap
+from home.src.index.reindex import Reindex
+from home.src.index.video import index_new_video
+from home.src.ta.config import AppConfig
+from home.src.ta.helper import clean_string, ignore_filelist
+from home.src.ta.ta_redis import RedisArchivist
 
 
 class FilesystemScanner:
     """handle scanning and fixing from filesystem"""
 
     CONFIG = AppConfig().config
-    ES_URL = CONFIG["application"]["es_url"]
-    ES_AUTH = CONFIG["application"]["es_auth"]
     VIDEOS = CONFIG["application"]["videos"]
 
     def __init__(self):
@@ -234,8 +44,9 @@ class FilesystemScanner:
         all_downloaded = []
         for channel_name in all_channels:
             channel_path = os.path.join(self.VIDEOS, channel_name)
-            videos = os.listdir(channel_path)
-            all_videos = ignore_filelist(videos)
+            channel_files = os.listdir(channel_path)
+            channel_files_clean = ignore_filelist(channel_files)
+            all_videos = [i for i in channel_files_clean if i.endswith(".mp4")]
             for video in all_videos:
                 youtube_id = video[9:20]
                 all_downloaded.append((channel_name, video, youtube_id))
@@ -246,13 +57,15 @@ class FilesystemScanner:
     def get_all_indexed():
         """get a list of all indexed videos"""
         index_handler = PendingList()
-        all_indexed_raw = index_handler.get_all_indexed()
+        index_handler.get_download()
+        index_handler.get_indexed()
+
         all_indexed = []
-        for video in all_indexed_raw:
-            youtube_id = video["_id"]
-            media_url = video["_source"]["media_url"]
-            published = video["_source"]["published"]
-            title = video["_source"]["title"]
+        for video in index_handler.all_videos:
+            youtube_id = video["youtube_id"]
+            media_url = video["media_url"]
+            published = video["published"]
+            title = video["title"]
             all_indexed.append((youtube_id, media_url, published, title))
         return all_indexed
 
@@ -333,25 +146,16 @@ class FilesystemScanner:
             bulk_list.append(json.dumps(source))
         # add last newline
         bulk_list.append("\n")
-        query_str = "\n".join(bulk_list)
-        # make the call
-        headers = {"Content-type": "application/x-ndjson"}
-        url = self.ES_URL + "/_bulk"
-        request = requests.post(
-            url, data=query_str, headers=headers, auth=self.ES_AUTH
-        )
-        if not request.ok:
-            print(request.text)
+        data = "\n".join(bulk_list)
+        _, _ = ElasticWrap("_bulk").post(data=data, ndjson=True)
 
     def delete_from_index(self):
         """find indexed but deleted mediafile"""
         for indexed in self.to_delete:
             youtube_id = indexed[0]
             print(f"deleting {youtube_id} from index")
-            url = self.ES_URL + "/ta_video/_doc/" + youtube_id
-            request = requests.delete(url, auth=self.ES_AUTH)
-            if not request.ok:
-                print(request.text)
+            path = f"ta_video/_doc/{youtube_id}"
+            _, _ = ElasticWrap(path).delete()
 
 
 class ManualImport:
@@ -499,19 +303,18 @@ def scan_filesystem():
         print("index new videos")
         for missing_vid in filesystem_handler.to_index:
             youtube_id = missing_vid[2]
-            index_new_video(youtube_id, missing_vid=missing_vid)
+            index_new_video(youtube_id)
 
 
 def reindex_old_documents():
     """daily refresh of old documents"""
-    # check needed last run
-    now = int(datetime.now().strftime("%s"))
-    last_reindex = RedisArchivist().get_message("last_reindex")
-    if isinstance(last_reindex, int) and now - last_reindex < 60 * 60 * 24:
-        return
-    # continue if needed
-    reindex_handler = Reindex()
-    reindex_handler.check_outdated()
-    reindex_handler.reindex()
-    # set timestamp
-    RedisArchivist().set_message("last_reindex", now, expire=False)
+    handler = Reindex()
+    if handler.config["downloads"]["cookie_import"]:
+        CookieHandler().use()
+    try:
+        handler.check_outdated()
+        handler.reindex()
+        RedisArchivist().set_message("last_reindex", handler.now, expire=False)
+    finally:
+        if handler.config["downloads"]["cookie_import"]:
+            CookieHandler().hide()

tubearchivist/home/src/index/generic.py

@@ -0,0 +1,155 @@
+"""
+functionality:
+- generic base class to inherit from for video, channel and playlist
+"""
+
+import math
+
+import yt_dlp
+from home.src.download.yt_cookie import CookieHandler
+from home.src.es.connect import ElasticWrap
+from home.src.ta.config import AppConfig
+from home.src.ta.ta_redis import RedisArchivist
+
+
+class YouTubeItem:
+    """base class for youtube"""
+
+    es_path = False
+    index_name = False
+    yt_base = False
+    yt_obs = {
+        "quiet": True,
+        "default_search": "ytsearch",
+        "skip_download": True,
+        "check_formats": "selected",
+        "socket_timeout": 3,
+        "noplaylist": True,
+    }
+
+    def __init__(self, youtube_id):
+        self.youtube_id = youtube_id
+        self.config = False
+        self.app_conf = False
+        self.youtube_meta = False
+        self.json_data = False
+        self._get_conf()
+
+    def _get_conf(self):
+        """read user conf"""
+        self.config = AppConfig().config
+        self.app_conf = self.config["application"]
+        if self.config["downloads"]["cookie_import"]:
+            cookie_path = CookieHandler().use()
+            self.yt_obs.update({"cookiefile": cookie_path})
+
+    def get_from_youtube(self):
+        """use yt-dlp to get meta data from youtube"""
+        print(f"{self.youtube_id}: get metadata from youtube")
+        try:
+            yt_item = yt_dlp.YoutubeDL(self.yt_obs)
+            response = yt_item.extract_info(self.yt_base + self.youtube_id)
+        except (
+            yt_dlp.utils.ExtractorError,
+            yt_dlp.utils.DownloadError,
+        ):
+            print(f"{self.youtube_id}: failed to get info from youtube")
+            response = False
+
+        self.youtube_meta = response
+
+    def get_from_es(self):
+        """get indexed data from elastic search"""
+        print(f"{self.youtube_id}: get metadata from es")
+        response, _ = ElasticWrap(f"{self.es_path}").get()
+        source = response.get("_source")
+        self.json_data = source
+
+    def upload_to_es(self):
+        """add json_data to elastic"""
+        _, _ = ElasticWrap(self.es_path).put(self.json_data, refresh=True)
+
+    def deactivate(self):
+        """deactivate document in es"""
+        print(f"{self.youtube_id}: deactivate document")
+        key_match = {
+            "ta_video": "active",
+            "ta_channel": "channel_active",
+            "ta_playlist": "playlist_active",
+        }
+        update_path = f"{self.index_name}/_update/{self.youtube_id}"
+        data = {
+            "script": f"ctx._source.{key_match.get(self.index_name)} = false"
+        }
+        _, _ = ElasticWrap(update_path).post(data)
+
+    def del_in_es(self):
+        """delete item from elastic search"""
+        print(f"{self.youtube_id}: delete from es")
+        _, _ = ElasticWrap(self.es_path).delete(refresh=True)
+
+
+class Pagination:
+    """
+    figure out the pagination based on page size and total_hits
+    """
+
+    def __init__(self, page_get, user_id, search_get=False):
+        self.user_id = user_id
+        self.page_size = self.get_page_size()
+        self.page_get = page_get
+        self.search_get = search_get
+        self.pagination = self.first_guess()
+
+    def get_page_size(self):
+        """get default or user modified page_size"""
+        key = f"{self.user_id}:page_size"
+        page_size = RedisArchivist().get_message(key)["status"]
+        if not page_size:
+            config = AppConfig().config
+            page_size = config["archive"]["page_size"]
+
+        return page_size
+
+    def first_guess(self):
+        """build first guess before api call"""
+        page_get = self.page_get
+        if page_get in [0, 1]:
+            page_from = 0
+            prev_pages = False
+        elif page_get > 1:
+            page_from = (page_get - 1) * self.page_size
+            prev_pages = [
+                i for i in range(page_get - 1, page_get - 6, -1) if i > 1
+            ]
+            prev_pages.reverse()
+        pagination = {
+            "page_size": self.page_size,
+            "page_from": page_from,
+            "prev_pages": prev_pages,
+            "current_page": page_get,
+            "max_hits": False,
+        }
+        if self.search_get:
+            pagination.update({"search_get": self.search_get})
+        return pagination
+
+    def validate(self, total_hits):
+        """validate pagination with total_hits after making api call"""
+        page_get = self.page_get
+        max_pages = math.ceil(total_hits / self.page_size)
+        if total_hits >= 10000:
+            # es returns maximal 10000 results
+            self.pagination["max_hits"] = True
+            max_pages = max_pages - 1
+
+        if page_get < max_pages and max_pages > 1:
+            self.pagination["last_page"] = max_pages
+        else:
+            self.pagination["last_page"] = False
+        next_pages = [
+            i for i in range(page_get + 1, page_get + 6) if 1 < i < max_pages
+        ]
+
+        self.pagination["next_pages"] = next_pages
+        self.pagination["total_hits"] = total_hits

tubearchivist/home/src/index/playlist.py

@@ -0,0 +1,208 @@
+"""
+functionality:
+- get metadata from youtube for a playlist
+- index and update in es
+"""
+
+import json
+from datetime import datetime
+
+from home.src.download.thumbnails import ThumbManager
+from home.src.es.connect import ElasticWrap
+from home.src.index.generic import YouTubeItem
+from home.src.index.video import YoutubeVideo
+
+
+class YoutubePlaylist(YouTubeItem):
+    """represents a single youtube playlist"""
+
+    es_path = False
+    index_name = "ta_playlist"
+    yt_obs = {
+        "default_search": "ytsearch",
+        "quiet": True,
+        "skip_download": True,
+        "extract_flat": True,
+    }
+    yt_base = "https://www.youtube.com/playlist?list="
+
+    def __init__(self, youtube_id):
+        super().__init__(youtube_id)
+        self.es_path = f"{self.index_name}/_doc/{youtube_id}"
+        self.all_members = False
+        self.nav = False
+        self.all_youtube_ids = []
+
+    def build_json(self, scrape=False):
+        """collection to create json_data"""
+        self.get_from_es()
+        if self.json_data:
+            subscribed = self.json_data.get("playlist_subscribed")
+        else:
+            subscribed = False
+
+        if scrape or not self.json_data:
+            self.get_from_youtube()
+            self.process_youtube_meta()
+            self.get_entries()
+            self.json_data["playlist_entries"] = self.all_members
+            self.get_playlist_art()
+            self.json_data["playlist_subscribed"] = subscribed
+
+    def process_youtube_meta(self):
+        """extract relevant fields from youtube"""
+        self.json_data = {
+            "playlist_id": self.youtube_id,
+            "playlist_active": True,
+            "playlist_name": self.youtube_meta["title"],
+            "playlist_channel": self.youtube_meta["channel"],
+            "playlist_channel_id": self.youtube_meta["channel_id"],
+            "playlist_thumbnail": self.youtube_meta["thumbnails"][-1]["url"],
+            "playlist_description": self.youtube_meta["description"] or False,
+            "playlist_last_refresh": int(datetime.now().strftime("%s")),
+        }
+
+    def get_entries(self, playlistend=False):
+        """get all videos in playlist"""
+        if playlistend:
+            # implement playlist end
+            print(playlistend)
+        all_members = []
+        for idx, entry in enumerate(self.youtube_meta["entries"]):
+            if self.all_youtube_ids:
+                downloaded = entry["id"] in self.all_youtube_ids
+            else:
+                downloaded = False
+            if not entry["uploader"]:
+                continue
+            to_append = {
+                "youtube_id": entry["id"],
+                "title": entry["title"],
+                "uploader": entry["uploader"],
+                "idx": idx,
+                "downloaded": downloaded,
+            }
+            all_members.append(to_append)
+
+        self.all_members = all_members
+
+    @staticmethod
+    def get_playlist_art():
+        """download artwork of playlist"""
+        thumbnails = ThumbManager()
+        missing_playlists = thumbnails.get_missing_playlists()
+        thumbnails.download_playlist(missing_playlists)
+
+    def add_vids_to_playlist(self):
+        """sync the playlist id to videos"""
+        script = (
+            'if (!ctx._source.containsKey("playlist")) '
+            + "{ctx._source.playlist = [params.playlist]} "
+            + "else if (!ctx._source.playlist.contains(params.playlist)) "
+            + "{ctx._source.playlist.add(params.playlist)} "
+            + "else {ctx.op = 'none'}"
+        )
+
+        bulk_list = []
+        for entry in self.json_data["playlist_entries"]:
+            video_id = entry["youtube_id"]
+            action = {"update": {"_id": video_id, "_index": "ta_video"}}
+            source = {
+                "script": {
+                    "source": script,
+                    "lang": "painless",
+                    "params": {"playlist": self.youtube_id},
+                }
+            }
+            bulk_list.append(json.dumps(action))
+            bulk_list.append(json.dumps(source))
+
+        # add last newline
+        bulk_list.append("\n")
+        query_str = "\n".join(bulk_list)
+
+        ElasticWrap("_bulk").post(query_str, ndjson=True)
+
+    def update_playlist(self):
+        """update metadata for playlist with data from YouTube"""
+        self.get_from_es()
+        subscribed = self.json_data["playlist_subscribed"]
+        self.get_from_youtube()
+        if not self.json_data:
+            # return false to deactivate
+            return False
+
+        self.json_data["playlist_subscribed"] = subscribed
+        self.upload_to_es()
+        return True
+
+    def build_nav(self, youtube_id):
+        """find next and previous in playlist of a given youtube_id"""
+        all_entries_available = self.json_data["playlist_entries"]
+        all_entries = [i for i in all_entries_available if i["downloaded"]]
+        current = [i for i in all_entries if i["youtube_id"] == youtube_id]
+        # stop if not found or playlist of 1
+        if not current or not len(all_entries) > 1:
+            return
+
+        current_idx = all_entries.index(current[0])
+        if current_idx == 0:
+            previous_item = False
+        else:
+            previous_item = all_entries[current_idx - 1]
+            prev_thumb = ThumbManager().vid_thumb_path(
+                previous_item["youtube_id"]
+            )
+            previous_item["vid_thumb"] = prev_thumb
+
+        if current_idx == len(all_entries) - 1:
+            next_item = False
+        else:
+            next_item = all_entries[current_idx + 1]
+            next_thumb = ThumbManager().vid_thumb_path(next_item["youtube_id"])
+            next_item["vid_thumb"] = next_thumb
+
+        self.nav = {
+            "playlist_meta": {
+                "current_idx": current[0]["idx"],
+                "playlist_id": self.youtube_id,
+                "playlist_name": self.json_data["playlist_name"],
+                "playlist_channel": self.json_data["playlist_channel"],
+            },
+            "playlist_previous": previous_item,
+            "playlist_next": next_item,
+        }
+        return
+
+    def delete_metadata(self):
+        """delete metadata for playlist"""
+        script = (
+            "ctx._source.playlist.removeAll("
+            + "Collections.singleton(params.playlist)) "
+        )
+        data = {
+            "query": {
+                "term": {"playlist.keyword": {"value": self.youtube_id}}
+            },
+            "script": {
+                "source": script,
+                "lang": "painless",
+                "params": {"playlist": self.youtube_id},
+            },
+        }
+        _, _ = ElasticWrap("ta_video/_update_by_query").post(data)
+        self.del_in_es()
+
+    def delete_videos_playlist(self):
+        """delete playlist with all videos"""
+        print(f"{self.youtube_id}: delete playlist")
+        self.get_from_es()
+        all_youtube_id = [
+            i["youtube_id"]
+            for i in self.json_data["playlist_entries"]
+            if i["downloaded"]
+        ]
+        for youtube_id in all_youtube_id:
+            YoutubeVideo(youtube_id).delete_media_file()
+
+        self.delete_metadata()

tubearchivist/home/src/index/reindex.py

@@ -0,0 +1,300 @@
+"""
+functionality:
+- periodically refresh documents
+- index and update in es
+"""
+
+import os
+import shutil
+from datetime import datetime
+from math import ceil
+from time import sleep
+
+from home.src.download.queue import PendingList
+from home.src.download.thumbnails import ThumbManager
+from home.src.download.yt_dlp_handler import VideoDownloader
+from home.src.es.connect import ElasticWrap
+from home.src.index.channel import YoutubeChannel
+from home.src.index.playlist import YoutubePlaylist
+from home.src.index.video import YoutubeVideo
+from home.src.ta.config import AppConfig
+
+
+class Reindex:
+    """check for outdated documents and refresh data from youtube"""
+
+    MATCH_FIELD = {
+        "ta_video": "active",
+        "ta_channel": "channel_active",
+        "ta_playlist": "playlist_active",
+    }
+    MULTIPLY = 1.2
+
+    def __init__(self):
+        # config
+        self.now = int(datetime.now().strftime("%s"))
+        self.config = AppConfig().config
+        self.interval = self.config["scheduler"]["check_reindex_days"]
+        # scan
+        self.all_youtube_ids = False
+        self.all_channel_ids = False
+        self.all_playlist_ids = False
+
+    def _get_daily(self):
+        """get daily refresh values"""
+        total_videos = self._get_total_hits("ta_video")
+        video_daily = ceil(total_videos / self.interval * self.MULTIPLY)
+        if video_daily >= 10000:
+            video_daily = 9999
+
+        total_channels = self._get_total_hits("ta_channel")
+        channel_daily = ceil(total_channels / self.interval * self.MULTIPLY)
+        total_playlists = self._get_total_hits("ta_playlist")
+        playlist_daily = ceil(total_playlists / self.interval * self.MULTIPLY)
+        return (video_daily, channel_daily, playlist_daily)
+
+    def _get_total_hits(self, index):
+        """get total hits from index"""
+        match_field = self.MATCH_FIELD[index]
+        path = f"{index}/_search?filter_path=hits.total"
+        data = {"query": {"match": {match_field: True}}}
+        response, _ = ElasticWrap(path).post(data=data)
+        total_hits = response["hits"]["total"]["value"]
+        return total_hits
+
+    def _get_unrated_vids(self):
+        """get max 200 videos without rating if ryd integration is enabled"""
+        data = {
+            "size": 200,
+            "query": {
+                "bool": {
+                    "must_not": [{"exists": {"field": "stats.average_rating"}}]
+                }
+            },
+        }
+        response, _ = ElasticWrap("ta_video/_search").get(data=data)
+
+        missing_rating = [i["_id"] for i in response["hits"]["hits"]]
+        self.all_youtube_ids = self.all_youtube_ids + missing_rating
+
+    def _get_outdated_vids(self, size):
+        """get daily videos to refresh"""
+        now_lte = self.now - self.interval * 24 * 60 * 60
+        must_list = [
+            {"match": {"active": True}},
+            {"range": {"vid_last_refresh": {"lte": now_lte}}},
+        ]
+        data = {
+            "size": size,
+            "query": {"bool": {"must": must_list}},
+            "sort": [{"vid_last_refresh": {"order": "asc"}}],
+            "_source": False,
+        }
+        response, _ = ElasticWrap("ta_video/_search").get(data=data)
+
+        all_youtube_ids = [i["_id"] for i in response["hits"]["hits"]]
+        return all_youtube_ids
+
+    def _get_outdated_channels(self, size):
+        """get daily channels to refresh"""
+        now_lte = self.now - self.interval * 24 * 60 * 60
+        must_list = [
+            {"match": {"channel_active": True}},
+            {"range": {"channel_last_refresh": {"lte": now_lte}}},
+        ]
+        data = {
+            "size": size,
+            "query": {"bool": {"must": must_list}},
+            "sort": [{"channel_last_refresh": {"order": "asc"}}],
+            "_source": False,
+        }
+        response, _ = ElasticWrap("ta_channel/_search").get(data=data)
+
+        all_channel_ids = [i["_id"] for i in response["hits"]["hits"]]
+        return all_channel_ids
+
+    def _get_outdated_playlists(self, size):
+        """get daily outdated playlists to refresh"""
+        now_lte = self.now - self.interval * 24 * 60 * 60
+        must_list = [
+            {"match": {"playlist_active": True}},
+            {"range": {"playlist_last_refresh": {"lte": now_lte}}},
+        ]
+        data = {
+            "size": size,
+            "query": {"bool": {"must": must_list}},
+            "sort": [{"playlist_last_refresh": {"order": "asc"}}],
+            "_source": False,
+        }
+        response, _ = ElasticWrap("ta_playlist/_search").get(data=data)
+
+        all_playlist_ids = [i["_id"] for i in response["hits"]["hits"]]
+        return all_playlist_ids
+
+    def check_outdated(self):
+        """add missing vids and channels"""
+        video_daily, channel_daily, playlist_daily = self._get_daily()
+        self.all_youtube_ids = self._get_outdated_vids(video_daily)
+        self.all_channel_ids = self._get_outdated_channels(channel_daily)
+        self.all_playlist_ids = self._get_outdated_playlists(playlist_daily)
+
+        integrate_ryd = self.config["downloads"]["integrate_ryd"]
+        if integrate_ryd:
+            self._get_unrated_vids()
+
+    @staticmethod
+    def _reindex_single_video(youtube_id):
+        """refresh data for single video"""
+        video = YoutubeVideo(youtube_id)
+
+        # read current state
+        video.get_from_es()
+        player = video.json_data["player"]
+        date_downloaded = video.json_data["date_downloaded"]
+        channel_dict = video.json_data["channel"]
+        playlist = video.json_data.get("playlist")
+
+        # get new
+        video.build_json()
+        if not video.youtube_meta:
+            video.deactivate()
+            return
+
+        video.delete_subtitles()
+        video.check_subtitles()
+
+        # add back
+        video.json_data["player"] = player
+        video.json_data["date_downloaded"] = date_downloaded
+        video.json_data["channel"] = channel_dict
+        if playlist:
+            video.json_data["playlist"] = playlist
+
+        video.upload_to_es()
+
+        thumb_handler = ThumbManager()
+        thumb_handler.delete_vid_thumb(youtube_id)
+        to_download = (youtube_id, video.json_data["vid_thumb_url"])
+        thumb_handler.download_vid([to_download], notify=False)
+        return
+
+    @staticmethod
+    def _reindex_single_channel(channel_id):
+        """refresh channel data and sync to videos"""
+        channel = YoutubeChannel(channel_id)
+        channel.get_from_es()
+        subscribed = channel.json_data["channel_subscribed"]
+        overwrites = channel.json_data.get("channel_overwrites", False)
+        channel.get_from_youtube()
+        channel.json_data["channel_subscribed"] = subscribed
+        if overwrites:
+            channel.json_data["channel_overwrites"] = overwrites
+        channel.upload_to_es()
+        channel.sync_to_videos()
+
+    @staticmethod
+    def _reindex_single_playlist(playlist_id, all_indexed_ids):
+        """refresh playlist data"""
+        playlist = YoutubePlaylist(playlist_id)
+        playlist.get_from_es()
+        subscribed = playlist.json_data["playlist_subscribed"]
+        playlist.all_youtube_ids = all_indexed_ids
+        playlist.build_json(scrape=True)
+        if not playlist.json_data:
+            playlist.deactivate()
+            return
+
+        playlist.json_data["playlist_subscribed"] = subscribed
+        playlist.upload_to_es()
+        return
+
+    def reindex(self):
+        """reindex what's needed"""
+        sleep_interval = self.config["downloads"]["sleep_interval"]
+        # videos
+        print(f"reindexing {len(self.all_youtube_ids)} videos")
+        for youtube_id in self.all_youtube_ids:
+            try:
+                self._reindex_single_video(youtube_id)
+            except FileNotFoundError:
+                # handle channel name change here
+                ChannelUrlFixer(youtube_id, self.config).run()
+                self._reindex_single_video(youtube_id)
+            if sleep_interval:
+                sleep(sleep_interval)
+        # channels
+        print(f"reindexing {len(self.all_channel_ids)} channels")
+        for channel_id in self.all_channel_ids:
+            self._reindex_single_channel(channel_id)
+            if sleep_interval:
+                sleep(sleep_interval)
+        # playlist
+        print(f"reindexing {len(self.all_playlist_ids)} playlists")
+        if self.all_playlist_ids:
+            handler = PendingList()
+            handler.get_download()
+            handler.get_indexed()
+            all_indexed_ids = [i["youtube_id"] for i in handler.all_videos]
+            for playlist_id in self.all_playlist_ids:
+                self._reindex_single_playlist(playlist_id, all_indexed_ids)
+                if sleep_interval:
+                    sleep(sleep_interval)
+
+
+class ChannelUrlFixer:
+    """fix not matching channel names in reindex"""
+
+    def __init__(self, youtube_id, config):
+        self.youtube_id = youtube_id
+        self.config = config
+        self.video = False
+
+    def run(self):
+        """check and run if needed"""
+        print(f"{self.youtube_id}: failed to build channel path, try to fix.")
+        video_path_is, video_folder_is = self.get_as_is()
+        if not os.path.exists(video_path_is):
+            print(f"giving up reindex, video in video: {self.video.json_data}")
+            raise ValueError
+
+        _, video_folder_should = self.get_as_should()
+
+        if video_folder_is != video_folder_should:
+            self.process(video_path_is)
+        else:
+            print(f"{self.youtube_id}: skip channel url fixer")
+
+    def get_as_is(self):
+        """get video object as is"""
+        self.video = YoutubeVideo(self.youtube_id)
+        self.video.get_from_es()
+        video_path_is = os.path.join(
+            self.config["application"]["videos"],
+            self.video.json_data["media_url"],
+        )
+        video_folder_is = os.path.split(video_path_is)[0]
+
+        return video_path_is, video_folder_is
+
+    def get_as_should(self):
+        """add fresh metadata from remote"""
+        self.video.get_from_youtube()
+        self.video.add_file_path()
+
+        video_path_should = os.path.join(
+            self.config["application"]["videos"],
+            self.video.json_data["media_url"],
+        )
+        video_folder_should = os.path.split(video_path_should)[0]
+        return video_path_should, video_folder_should
+
+    def process(self, video_path_is):
+        """fix filepath"""
+        print(f"{self.youtube_id}: fixing channel rename.")
+        cache_dir = self.config["application"]["cache_dir"]
+        new_file_path = os.path.join(
+            cache_dir, "download", self.youtube_id + ".mp4"
+        )
+        shutil.move(video_path_is, new_file_path)
+        VideoDownloader().move_to_archive(self.video.json_data)
+        self.video.update_media_url()

tubearchivist/home/src/index/video.py

@@ -0,0 +1,630 @@
+"""
+functionality:
+- get metadata from youtube for a video
+- index and update in es
+"""
+
+import json
+import os
+from datetime import datetime
+
+import requests
+from django.conf import settings
+from home.src.es.connect import ElasticWrap
+from home.src.index import channel as ta_channel
+from home.src.index.generic import YouTubeItem
+from home.src.ta.helper import (
+    DurationConverter,
+    clean_string,
+    randomizor,
+    requests_headers,
+)
+from home.src.ta.ta_redis import RedisArchivist
+from ryd_client import ryd_client
+
+
+class YoutubeSubtitle:
+    """handle video subtitle functionality"""
+
+    def __init__(self, video):
+        self.video = video
+        self.languages = False
+
+    def _sub_conf_parse(self):
+        """add additional conf values to self"""
+        languages_raw = self.video.config["downloads"]["subtitle"]
+        if languages_raw:
+            self.languages = [i.strip() for i in languages_raw.split(",")]
+
+    def get_subtitles(self):
+        """check what to do"""
+        self._sub_conf_parse()
+        if not self.languages:
+            # no subtitles
+            return False
+
+        relevant_subtitles = []
+        for lang in self.languages:
+            user_sub = self._get_user_subtitles(lang)
+            if user_sub:
+                relevant_subtitles.append(user_sub)
+                continue
+
+            if self.video.config["downloads"]["subtitle_source"] == "auto":
+                auto_cap = self._get_auto_caption(lang)
+                if auto_cap:
+                    relevant_subtitles.append(auto_cap)
+
+        return relevant_subtitles
+
+    def _get_auto_caption(self, lang):
+        """get auto_caption subtitles"""
+        print(f"{self.video.youtube_id}-{lang}: get auto generated subtitles")
+        all_subtitles = self.video.youtube_meta.get("automatic_captions")
+
+        if not all_subtitles:
+            return False
+
+        video_media_url = self.video.json_data["media_url"]
+        media_url = video_media_url.replace(".mp4", f"-{lang}.vtt")
+        all_formats = all_subtitles.get(lang)
+        if not all_formats:
+            return False
+
+        subtitle = [i for i in all_formats if i["ext"] == "json3"][0]
+        subtitle.update(
+            {"lang": lang, "source": "auto", "media_url": media_url}
+        )
+
+        return subtitle
+
+    def _normalize_lang(self):
+        """normalize country specific language keys"""
+        all_subtitles = self.video.youtube_meta.get("subtitles")
+        if not all_subtitles:
+            return False
+
+        all_keys = list(all_subtitles.keys())
+        for key in all_keys:
+            lang = key.split("-")[0]
+            old = all_subtitles.pop(key)
+            if lang == "live_chat":
+                continue
+            all_subtitles[lang] = old
+
+        return all_subtitles
+
+    def _get_user_subtitles(self, lang):
+        """get subtitles uploaded from channel owner"""
+        print(f"{self.video.youtube_id}-{lang}: get user uploaded subtitles")
+        all_subtitles = self._normalize_lang()
+        if not all_subtitles:
+            return False
+
+        video_media_url = self.video.json_data["media_url"]
+        media_url = video_media_url.replace(".mp4", f"-{lang}.vtt")
+        all_formats = all_subtitles.get(lang)
+        if not all_formats:
+            # no user subtitles found
+            return False
+
+        subtitle = [i for i in all_formats if i["ext"] == "json3"][0]
+        subtitle.update(
+            {"lang": lang, "source": "user", "media_url": media_url}
+        )
+
+        return subtitle
+
+    def download_subtitles(self, relevant_subtitles):
+        """download subtitle files to archive"""
+        videos_base = self.video.config["application"]["videos"]
+        for subtitle in relevant_subtitles:
+            dest_path = os.path.join(videos_base, subtitle["media_url"])
+            source = subtitle["source"]
+            lang = subtitle.get("lang")
+            response = requests.get(
+                subtitle["url"], headers=requests_headers()
+            )
+            if not response.ok:
+                print(f"{self.video.youtube_id}: failed to download subtitle")
+                print(response.text)
+                continue
+
+            parser = SubtitleParser(response.text, lang, source)
+            parser.process()
+            subtitle_str = parser.get_subtitle_str()
+            self._write_subtitle_file(dest_path, subtitle_str)
+            if self.video.config["downloads"]["subtitle_index"]:
+                query_str = parser.create_bulk_import(self.video, source)
+                self._index_subtitle(query_str)
+
+    @staticmethod
+    def _write_subtitle_file(dest_path, subtitle_str):
+        """write subtitle file to disk"""
+        # create folder here for first video of channel
+        os.makedirs(os.path.split(dest_path)[0], exist_ok=True)
+        with open(dest_path, "w", encoding="utf-8") as subfile:
+            subfile.write(subtitle_str)
+
+    @staticmethod
+    def _index_subtitle(query_str):
+        """send subtitle to es for indexing"""
+        _, _ = ElasticWrap("_bulk").post(data=query_str, ndjson=True)
+
+    def delete(self):
+        """delete subtitles from index and filesystem"""
+        youtube_id = self.video.youtube_id
+        # delete files
+        videos_base = self.video.config["application"]["videos"]
+        if not self.video.json_data.get("subtitles"):
+            return
+
+        files = [i["media_url"] for i in self.video.json_data["subtitles"]]
+        for file_name in files:
+            file_path = os.path.join(videos_base, file_name)
+            try:
+                os.remove(file_path)
+            except FileNotFoundError:
+                print(f"{youtube_id}: {file_path} failed to delete")
+        # delete from index
+        path = "ta_subtitle/_delete_by_query?refresh=true"
+        data = {"query": {"term": {"youtube_id": {"value": youtube_id}}}}
+        _, _ = ElasticWrap(path).post(data=data)
+
+
+class SubtitleParser:
+    """parse subtitle str from youtube"""
+
+    def __init__(self, subtitle_str, lang, source):
+        self.subtitle_raw = json.loads(subtitle_str)
+        self.lang = lang
+        self.source = source
+        self.all_cues = False
+
+    def process(self):
+        """extract relevant que data"""
+        all_events = self.subtitle_raw.get("events")
+        if self.source == "auto":
+            all_events = self._flat_auto_caption(all_events)
+
+        self.all_cues = []
+        for idx, event in enumerate(all_events):
+            if "dDurationMs" not in event:
+                # some events won't have a duration
+                print(f"failed to parse event without duration: {event}")
+                continue
+
+            cue = {
+                "start": self._ms_conv(event["tStartMs"]),
+                "end": self._ms_conv(event["tStartMs"] + event["dDurationMs"]),
+                "text": "".join([i.get("utf8") for i in event["segs"]]),
+                "idx": idx + 1,
+            }
+            self.all_cues.append(cue)
+
+    @staticmethod
+    def _flat_auto_caption(all_events):
+        """flatten autocaption segments"""
+        flatten = []
+        for event in all_events:
+            if "segs" not in event.keys():
+                continue
+            text = "".join([i.get("utf8") for i in event.get("segs")])
+            if not text.strip():
+                continue
+
+            if flatten:
+                # fix overlapping retiming issue
+                if "dDurationMs" not in flatten[-1]:
+                    # some events won't have a duration
+                    print(f"failed to parse event without duration: {event}")
+                    continue
+
+                last_end = flatten[-1]["tStartMs"] + flatten[-1]["dDurationMs"]
+                if event["tStartMs"] < last_end:
+                    joined = flatten[-1]["segs"][0]["utf8"] + "\n" + text
+                    flatten[-1]["segs"][0]["utf8"] = joined
+                    continue
+
+            event.update({"segs": [{"utf8": text}]})
+            flatten.append(event)
+
+        return flatten
+
+    @staticmethod
+    def _ms_conv(ms):
+        """convert ms to timestamp"""
+        hours = str((ms // (1000 * 60 * 60)) % 24).zfill(2)
+        minutes = str((ms // (1000 * 60)) % 60).zfill(2)
+        secs = str((ms // 1000) % 60).zfill(2)
+        millis = str(ms % 1000).zfill(3)
+
+        return f"{hours}:{minutes}:{secs}.{millis}"
+
+    def get_subtitle_str(self):
+        """create vtt text str from cues"""
+        subtitle_str = f"WEBVTT\nKind: captions\nLanguage: {self.lang}"
+
+        for cue in self.all_cues:
+            stamp = f"{cue.get('start')} --> {cue.get('end')}"
+            cue_text = f"\n\n{cue.get('idx')}\n{stamp}\n{cue.get('text')}"
+            subtitle_str = subtitle_str + cue_text
+
+        return subtitle_str
+
+    def create_bulk_import(self, video, source):
+        """subtitle lines for es import"""
+        documents = self._create_documents(video, source)
+        bulk_list = []
+
+        for document in documents:
+            document_id = document.get("subtitle_fragment_id")
+            action = {"index": {"_index": "ta_subtitle", "_id": document_id}}
+            bulk_list.append(json.dumps(action))
+            bulk_list.append(json.dumps(document))
+
+        bulk_list.append("\n")
+        query_str = "\n".join(bulk_list)
+
+        return query_str
+
+    def _create_documents(self, video, source):
+        """process documents"""
+        documents = self._chunk_list(video.youtube_id)
+        channel = video.json_data.get("channel")
+        meta_dict = {
+            "youtube_id": video.youtube_id,
+            "title": video.json_data.get("title"),
+            "subtitle_channel": channel.get("channel_name"),
+            "subtitle_channel_id": channel.get("channel_id"),
+            "subtitle_last_refresh": int(datetime.now().strftime("%s")),
+            "subtitle_lang": self.lang,
+            "subtitle_source": source,
+        }
+
+        _ = [i.update(meta_dict) for i in documents]
+
+        return documents
+
+    def _chunk_list(self, youtube_id):
+        """join cues for bulk import"""
+        chunk_list = []
+
+        chunk = {}
+        for cue in self.all_cues:
+            if chunk:
+                text = f"{chunk.get('subtitle_line')} {cue.get('text')}\n"
+                chunk["subtitle_line"] = text
+            else:
+                idx = len(chunk_list) + 1
+                chunk = {
+                    "subtitle_index": idx,
+                    "subtitle_line": cue.get("text"),
+                    "subtitle_start": cue.get("start"),
+                }
+
+            chunk["subtitle_fragment_id"] = f"{youtube_id}-{self.lang}-{idx}"
+
+            if cue["idx"] % 5 == 0:
+                chunk["subtitle_end"] = cue.get("end")
+                chunk_list.append(chunk)
+                chunk = {}
+
+        return chunk_list
+
+
+class SponsorBlock:
+    """handle sponsor block integration"""
+
+    API = "https://sponsor.ajay.app/api"
+
+    def __init__(self, user_id=False):
+        self.user_id = user_id
+        self.user_agent = f"{settings.TA_UPSTREAM} {settings.TA_VERSION}"
+        self.last_refresh = int(datetime.now().strftime("%s"))
+
+    def get_sb_id(self):
+        """get sponsorblock userid or generate if needed"""
+        if not self.user_id:
+            print("missing request user id")
+            raise ValueError
+
+        key = f"{self.user_id}:id_sponsorblock"
+        sb_id = RedisArchivist().get_message(key)
+        if not sb_id["status"]:
+            sb_id = {"status": randomizor(32)}
+            RedisArchivist().set_message(key, sb_id, expire=False)
+
+        return sb_id
+
+    def get_timestamps(self, youtube_id):
+        """get timestamps from the API"""
+        url = f"{self.API}/skipSegments?videoID={youtube_id}"
+        headers = {"User-Agent": self.user_agent}
+        print(f"{youtube_id}: get sponsorblock timestamps")
+        response = requests.get(url, headers=headers)
+        if not response.ok:
+            print(f"{youtube_id}: sponsorblock failed: {response.text}")
+            sponsor_dict = {
+                "last_refresh": self.last_refresh,
+                "is_enabled": True,
+                "segments": [],
+            }
+        else:
+            all_segments = response.json()
+            sponsor_dict = self._get_sponsor_dict(all_segments)
+
+        return sponsor_dict
+
+    def _get_sponsor_dict(self, all_segments):
+        """format and process response"""
+        has_unlocked = False
+        cleaned_segments = []
+        for segment in all_segments:
+            if not segment["locked"]:
+                has_unlocked = True
+            del segment["userID"]
+            del segment["description"]
+            cleaned_segments.append(segment)
+
+        sponsor_dict = {
+            "last_refresh": self.last_refresh,
+            "has_unlocked": has_unlocked,
+            "is_enabled": True,
+            "segments": cleaned_segments,
+        }
+        return sponsor_dict
+
+    def post_timestamps(self, youtube_id, start_time, end_time):
+        """post timestamps to api"""
+        user_id = self.get_sb_id().get("status")
+        data = {
+            "videoID": youtube_id,
+            "startTime": start_time,
+            "endTime": end_time,
+            "category": "sponsor",
+            "userID": user_id,
+            "userAgent": self.user_agent,
+        }
+        url = f"{self.API}/skipSegments?videoID={youtube_id}"
+        print(f"post: {data}")
+        print(f"to: {url}")
+
+        return {"success": True}, 200
+
+    def vote_on_segment(self, uuid, vote):
+        """send vote on existing segment"""
+        user_id = self.get_sb_id().get("status")
+        data = {
+            "UUID": uuid,
+            "userID": user_id,
+            "type": vote,
+        }
+        url = f"{self.API}/api/voteOnSponsorTime"
+        print(f"post: {data}")
+        print(f"to: {url}")
+
+        return {"success": True}, 200
+
+
+class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
+    """represents a single youtube video"""
+
+    es_path = False
+    index_name = "ta_video"
+    yt_base = "https://www.youtube.com/watch?v="
+
+    def __init__(self, youtube_id, video_overwrites=False):
+        super().__init__(youtube_id)
+        self.channel_id = False
+        self.video_overwrites = video_overwrites
+        self.es_path = f"{self.index_name}/_doc/{youtube_id}"
+
+    def build_json(self):
+        """build json dict of video"""
+        self.get_from_youtube()
+        if not self.youtube_meta:
+            return
+
+        self._process_youtube_meta()
+        self._add_channel()
+        self._add_stats()
+        self.add_file_path()
+        self.add_player()
+        if self.config["downloads"]["integrate_ryd"]:
+            self._get_ryd_stats()
+
+        if self._check_get_sb():
+            self._get_sponsorblock()
+
+        return
+
+    def _check_get_sb(self):
+        """check if need to run sponsor block"""
+        integrate = self.config["downloads"]["integrate_sponsorblock"]
+
+        if self.video_overwrites:
+            single_overwrite = self.video_overwrites.get(self.youtube_id)
+            if not single_overwrite:
+                return integrate
+
+            if "integrate_sponsorblock" in single_overwrite:
+                return single_overwrite.get("integrate_sponsorblock")
+
+        return integrate
+
+    def _process_youtube_meta(self):
+        """extract relevant fields from youtube"""
+        # extract
+        self.channel_id = self.youtube_meta["channel_id"]
+        upload_date = self.youtube_meta["upload_date"]
+        upload_date_time = datetime.strptime(upload_date, "%Y%m%d")
+        published = upload_date_time.strftime("%Y-%m-%d")
+        last_refresh = int(datetime.now().strftime("%s"))
+        # base64_blur = ThumbManager().get_base64_blur(self.youtube_id)
+        base64_blur = False
+        # build json_data basics
+        self.json_data = {
+            "title": self.youtube_meta["title"],
+            "description": self.youtube_meta["description"],
+            "category": self.youtube_meta["categories"],
+            "vid_thumb_url": self.youtube_meta["thumbnail"],
+            "vid_thumb_base64": base64_blur,
+            "tags": self.youtube_meta["tags"],
+            "published": published,
+            "vid_last_refresh": last_refresh,
+            "date_downloaded": last_refresh,
+            "youtube_id": self.youtube_id,
+            "active": True,
+        }
+
+    def _add_channel(self):
+        """add channel dict to video json_data"""
+        channel = ta_channel.YoutubeChannel(self.channel_id)
+        channel.build_json(upload=True)
+        self.json_data.update({"channel": channel.json_data})
+
+    def _add_stats(self):
+        """add stats dicst to json_data"""
+        # likes
+        like_count = self.youtube_meta.get("like_count", 0)
+        dislike_count = self.youtube_meta.get("dislike_count", 0)
+        self.json_data.update(
+            {
+                "stats": {
+                    "view_count": self.youtube_meta["view_count"],
+                    "like_count": like_count,
+                    "dislike_count": dislike_count,
+                    "average_rating": self.youtube_meta["average_rating"],
+                }
+            }
+        )
+
+    def build_dl_cache_path(self):
+        """find video path in dl cache"""
+        cache_dir = self.app_conf["cache_dir"]
+        cache_path = f"{cache_dir}/download/"
+        all_cached = os.listdir(cache_path)
+        for file_cached in all_cached:
+            if self.youtube_id in file_cached:
+                vid_path = os.path.join(cache_path, file_cached)
+                return vid_path
+
+        raise FileNotFoundError
+
+    def add_player(self):
+        """add player information for new videos"""
+        try:
+            # when indexing from download task
+            vid_path = self.build_dl_cache_path()
+        except FileNotFoundError as err:
+            # when reindexing needs to handle title rename
+            channel = os.path.split(self.json_data["media_url"])[0]
+            channel_dir = os.path.join(self.app_conf["videos"], channel)
+            all_files = os.listdir(channel_dir)
+            for file in all_files:
+                if self.youtube_id in file and file.endswith(".mp4"):
+                    vid_path = os.path.join(channel_dir, file)
+                    break
+            else:
+                raise FileNotFoundError("could not find video file") from err
+
+        duration_handler = DurationConverter()
+        duration = duration_handler.get_sec(vid_path)
+        duration_str = duration_handler.get_str(duration)
+        self.json_data.update(
+            {
+                "player": {
+                    "watched": False,
+                    "duration": duration,
+                    "duration_str": duration_str,
+                }
+            }
+        )
+
+    def add_file_path(self):
+        """build media_url for where file will be located"""
+        channel_name = self.json_data["channel"]["channel_name"]
+        clean_channel_name = clean_string(channel_name)
+        if len(clean_channel_name) <= 3:
+            # fall back to channel id
+            clean_channel_name = self.json_data["channel"]["channel_id"]
+
+        timestamp = self.json_data["published"].replace("-", "")
+        youtube_id = self.json_data["youtube_id"]
+        title = self.json_data["title"]
+        clean_title = clean_string(title)
+        filename = f"{timestamp}_{youtube_id}_{clean_title}.mp4"
+        media_url = os.path.join(clean_channel_name, filename)
+        self.json_data["media_url"] = media_url
+
+    def delete_media_file(self):
+        """delete video file, meta data"""
+        self.get_from_es()
+        video_base = self.app_conf["videos"]
+        media_url = self.json_data.get("media_url")
+        file_path = os.path.join(video_base, media_url)
+        try:
+            os.remove(file_path)
+        except FileNotFoundError:
+            print(f"{self.youtube_id}: failed {media_url}, continue.")
+
+        self.del_in_es()
+        self.delete_subtitles()
+
+    def delete_subtitles(self):
+        """delete indexed subtitles"""
+        print(f"{self.youtube_id}: delete subtitles")
+        YoutubeSubtitle(self).delete()
+
+    def _get_ryd_stats(self):
+        """get optional stats from returnyoutubedislikeapi.com"""
+        try:
+            print(f"{self.youtube_id}: get ryd stats")
+            result = ryd_client.get(self.youtube_id)
+        except requests.exceptions.ConnectionError:
+            print(f"{self.youtube_id}: failed to query ryd api, skipping")
+            return False
+
+        if result["status"] == 404:
+            return False
+
+        dislikes = {
+            "dislike_count": result["dislikes"],
+            "average_rating": result["rating"],
+        }
+        self.json_data["stats"].update(dislikes)
+
+        return True
+
+    def _get_sponsorblock(self):
+        """get optional sponsorblock timestamps from sponsor.ajay.app"""
+        sponsorblock = SponsorBlock().get_timestamps(self.youtube_id)
+        if sponsorblock:
+            self.json_data["sponsorblock"] = sponsorblock
+
+    def check_subtitles(self):
+        """optionally add subtitles"""
+        handler = YoutubeSubtitle(self)
+        subtitles = handler.get_subtitles()
+        if subtitles:
+            self.json_data["subtitles"] = subtitles
+            handler.download_subtitles(relevant_subtitles=subtitles)
+
+    def update_media_url(self):
+        """update only media_url in es for reindex channel rename"""
+        data = {"doc": {"media_url": self.json_data["media_url"]}}
+        path = f"{self.index_name}/_update/{self.youtube_id}"
+        _, _ = ElasticWrap(path).post(data=data)
+
+
+def index_new_video(youtube_id, video_overwrites=False):
+    """combined classes to create new video in index"""
+    video = YoutubeVideo(youtube_id, video_overwrites=video_overwrites)
+    video.build_json()
+    if not video.json_data:
+        raise ValueError("failed to get metadata for " + youtube_id)
+
+    video.check_subtitles()
+    video.upload_to_es()
+    return video.json_data

tubearchivist/home/src/index_management.py

@@ -1,546 +0,0 @@
-"""
-Functionality:
-- initial elastic search setup
-- index configuration is represented in INDEX_CONFIG
-- index mapping and settings validation
-- backup and restore
-"""
-
-import json
-import os
-import zipfile
-from datetime import datetime
-
-import requests
-from home.src.config import AppConfig
-from home.src.helper import ignore_filelist
-
-# expected mapping and settings
-INDEX_CONFIG = [
-    {
-        "index_name": "channel",
-        "expected_map": {
-            "channel_id": {
-                "type": "keyword",
-            },
-            "channel_name": {
-                "type": "text",
-                "fields": {
-                    "keyword": {
-                        "type": "keyword",
-                        "ignore_above": 256,
-                        "normalizer": "to_lower",
-                    },
-                    "search_as_you_type": {
-                        "type": "search_as_you_type",
-                        "doc_values": False,
-                        "max_shingle_size": 3,
-                    },
-                },
-            },
-            "channel_banner_url": {"type": "keyword", "index": False},
-            "channel_thumb_url": {"type": "keyword", "index": False},
-            "channel_description": {"type": "text"},
-            "channel_last_refresh": {"type": "date", "format": "epoch_second"},
-        },
-        "expected_set": {
-            "analysis": {
-                "normalizer": {
-                    "to_lower": {"type": "custom", "filter": ["lowercase"]}
-                }
-            },
-            "number_of_replicas": "0",
-        },
-    },
-    {
-        "index_name": "video",
-        "expected_map": {
-            "vid_thumb_url": {"type": "text", "index": False},
-            "date_downloaded": {"type": "date"},
-            "channel": {
-                "properties": {
-                    "channel_id": {
-                        "type": "keyword",
-                    },
-                    "channel_name": {
-                        "type": "text",
-                        "fields": {
-                            "keyword": {
-                                "type": "keyword",
-                                "ignore_above": 256,
-                                "normalizer": "to_lower",
-                            },
-                            "search_as_you_type": {
-                                "type": "search_as_you_type",
-                                "doc_values": False,
-                                "max_shingle_size": 3,
-                            },
-                        },
-                    },
-                    "channel_banner_url": {"type": "keyword", "index": False},
-                    "channel_thumb_url": {"type": "keyword", "index": False},
-                    "channel_description": {"type": "text"},
-                    "channel_last_refresh": {
-                        "type": "date",
-                        "format": "epoch_second",
-                    },
-                }
-            },
-            "description": {"type": "text"},
-            "media_url": {"type": "keyword", "index": False},
-            "title": {
-                "type": "text",
-                "fields": {
-                    "keyword": {
-                        "type": "keyword",
-                        "ignore_above": 256,
-                        "normalizer": "to_lower",
-                    },
-                    "search_as_you_type": {
-                        "type": "search_as_you_type",
-                        "doc_values": False,
-                        "max_shingle_size": 3,
-                    },
-                },
-            },
-            "vid_last_refresh": {"type": "date"},
-            "youtube_id": {"type": "keyword"},
-            "published": {"type": "date"},
-        },
-        "expected_set": {
-            "analysis": {
-                "normalizer": {
-                    "to_lower": {"type": "custom", "filter": ["lowercase"]}
-                }
-            },
-            "number_of_replicas": "0",
-        },
-    },
-    {
-        "index_name": "download",
-        "expected_map": {
-            "timestamp": {"type": "date"},
-            "channel_id": {"type": "keyword"},
-            "channel_name": {
-                "type": "text",
-                "fields": {
-                    "keyword": {
-                        "type": "keyword",
-                        "ignore_above": 256,
-                        "normalizer": "to_lower",
-                    }
-                },
-            },
-            "status": {"type": "keyword"},
-            "title": {
-                "type": "text",
-                "fields": {
-                    "keyword": {
-                        "type": "keyword",
-                        "ignore_above": 256,
-                        "normalizer": "to_lower",
-                    }
-                },
-            },
-            "vid_thumb_url": {"type": "keyword"},
-            "youtube_id": {"type": "keyword"},
-        },
-        "expected_set": {
-            "analysis": {
-                "normalizer": {
-                    "to_lower": {"type": "custom", "filter": ["lowercase"]}
-                }
-            },
-            "number_of_replicas": "0",
-        },
-    },
-]
-
-
-class ElasticIndex:
-    """
-    handle mapping and settings on elastic search for a given index
-    """
-
-    CONFIG = AppConfig().config
-    ES_URL = CONFIG["application"]["es_url"]
-    ES_AUTH = CONFIG["application"]["es_auth"]
-    HEADERS = {"Content-type": "application/json"}
-
-    def __init__(self, index_name, expected_map, expected_set):
-        self.index_name = index_name
-        self.expected_map = expected_map
-        self.expected_set = expected_set
-        self.exists, self.details = self.index_exists()
-
-    def index_exists(self):
-        """check if index already exists and return mapping if it does"""
-        index_name = self.index_name
-        url = f"{self.ES_URL}/ta_{index_name}"
-        response = requests.get(url, auth=self.ES_AUTH)
-        exists = response.ok
-
-        if exists:
-            details = response.json()[f"ta_{index_name}"]
-        else:
-            details = False
-
-        return exists, details
-
-    def validate(self):
-        """
-        check if all expected mappings and settings match
-        returns True when rebuild is needed
-        """
-
-        if self.expected_map:
-            rebuild = self.validate_mappings()
-            if rebuild:
-                return rebuild
-
-        if self.expected_set:
-            rebuild = self.validate_settings()
-            if rebuild:
-                return rebuild
-
-        return False
-
-    def validate_mappings(self):
-        """check if all mappings are as expected"""
-
-        expected_map = self.expected_map
-        now_map = self.details["mappings"]["properties"]
-
-        for key, value in expected_map.items():
-            # nested
-            if list(value.keys()) == ["properties"]:
-                for key_n, value_n in value["properties"].items():
-                    if key_n not in now_map[key]["properties"].keys():
-                        print(key_n, value_n)
-                        return True
-                    if not value_n == now_map[key]["properties"][key_n]:
-                        print(key_n, value_n)
-                        return True
-
-                continue
-
-            # not nested
-            if key not in now_map.keys():
-                print(key, value)
-                return True
-            if not value == now_map[key]:
-                print(key, value)
-                return True
-
-        return False
-
-    def validate_settings(self):
-        """check if all settings are as expected"""
-
-        now_set = self.details["settings"]["index"]
-
-        for key, value in self.expected_set.items():
-            if key not in now_set.keys():
-                print(key, value)
-                return True
-
-            if not value == now_set[key]:
-                print(key, value)
-                return True
-
-        return False
-
-    def rebuild_index(self):
-        """rebuild with new mapping"""
-        # backup
-        self.reindex("backup")
-        # delete original
-        self.delete_index(backup=False)
-        # create new
-        self.create_blank()
-        self.reindex("restore")
-        # delete backup
-        self.delete_index()
-
-    def reindex(self, method):
-        """create on elastic search"""
-        index_name = self.index_name
-        if method == "backup":
-            source = f"ta_{index_name}"
-            destination = f"ta_{index_name}_backup"
-        elif method == "restore":
-            source = f"ta_{index_name}_backup"
-            destination = f"ta_{index_name}"
-
-        query = {"source": {"index": source}, "dest": {"index": destination}}
-        data = json.dumps(query)
-        url = self.ES_URL + "/_reindex?refresh=true"
-        response = requests.post(
-            url=url, data=data, headers=self.HEADERS, auth=self.ES_AUTH
-        )
-        if not response.ok:
-            print(response.text)
-
-    def delete_index(self, backup=True):
-        """delete index passed as argument"""
-        if backup:
-            url = f"{self.ES_URL}/ta_{self.index_name}_backup"
-        else:
-            url = f"{self.ES_URL}/ta_{self.index_name}"
-        response = requests.delete(url, auth=self.ES_AUTH)
-        if not response.ok:
-            print(response.text)
-
-    def create_blank(self):
-        """apply new mapping and settings for blank new index"""
-        expected_map = self.expected_map
-        expected_set = self.expected_set
-        # stich payload
-        payload = {}
-        if expected_set:
-            payload.update({"settings": expected_set})
-        if expected_map:
-            payload.update({"mappings": {"properties": expected_map}})
-        # create
-        url = f"{self.ES_URL}/ta_{self.index_name}"
-        data = json.dumps(payload)
-        response = requests.put(
-            url=url, data=data, headers=self.HEADERS, auth=self.ES_AUTH
-        )
-        if not response.ok:
-            print(response.text)
-
-
-class ElasticBackup:
-    """dump index to nd-json files for later bulk import"""
-
-    def __init__(self, index_config):
-        self.config = AppConfig().config
-        self.index_config = index_config
-        self.timestamp = datetime.now().strftime("%Y%m%d")
-        self.backup_files = []
-
-    def get_all_documents(self, index_name):
-        """export all documents of a single index"""
-        headers = {"Content-type": "application/json"}
-        es_url = self.config["application"]["es_url"]
-        es_auth = self.config["application"]["es_auth"]
-        # get PIT ID
-        url = f"{es_url}/ta_{index_name}/_pit?keep_alive=1m"
-        response = requests.post(url, auth=es_auth)
-        json_data = json.loads(response.text)
-        pit_id = json_data["id"]
-        # build query
-        data = {
-            "query": {"match_all": {}},
-            "size": 100,
-            "pit": {"id": pit_id, "keep_alive": "1m"},
-            "sort": [{"_id": {"order": "asc"}}],
-        }
-        query_str = json.dumps(data)
-        url = es_url + "/_search"
-        # loop until nothing left
-        all_results = []
-        while True:
-            response = requests.get(
-                url, data=query_str, headers=headers, auth=es_auth
-            )
-            json_data = json.loads(response.text)
-            all_hits = json_data["hits"]["hits"]
-            if all_hits:
-                for hit in all_hits:
-                    search_after = hit["sort"]
-                    all_results.append(hit)
-                # update search_after with last hit data
-                data["search_after"] = search_after
-                query_str = json.dumps(data)
-            else:
-                break
-        # clean up PIT
-        query_str = json.dumps({"id": pit_id})
-        requests.delete(
-            es_url + "/_pit", data=query_str, headers=headers, auth=es_auth
-        )
-
-        return all_results
-
-    @staticmethod
-    def build_bulk(all_results):
-        """build bulk query data from all_results"""
-        bulk_list = []
-
-        for document in all_results:
-            document_id = document["_id"]
-            es_index = document["_index"]
-            action = {"index": {"_index": es_index, "_id": document_id}}
-            source = document["_source"]
-            bulk_list.append(json.dumps(action))
-            bulk_list.append(json.dumps(source))
-
-        # add last newline
-        bulk_list.append("\n")
-        file_content = "\n".join(bulk_list)
-
-        return file_content
-
-    def write_es_json(self, file_content, index_name):
-        """write nd-json file for es _bulk API to disk"""
-        cache_dir = self.config["application"]["cache_dir"]
-        file_name = f"es_{index_name}-{self.timestamp}.json"
-        file_path = os.path.join(cache_dir, "backup", file_name)
-        with open(file_path, "w", encoding="utf-8") as f:
-            f.write(file_content)
-
-        self.backup_files.append(file_path)
-
-    def write_ta_json(self, all_results, index_name):
-        """write generic json file to disk"""
-        cache_dir = self.config["application"]["cache_dir"]
-        file_name = f"ta_{index_name}-{self.timestamp}.json"
-        file_path = os.path.join(cache_dir, "backup", file_name)
-        to_write = [i["_source"] for i in all_results]
-        file_content = json.dumps(to_write)
-        with open(file_path, "w", encoding="utf-8") as f:
-            f.write(file_content)
-
-        self.backup_files.append(file_path)
-
-    def zip_it(self):
-        """pack it up into single zip file"""
-        cache_dir = self.config["application"]["cache_dir"]
-        file_name = f"ta_backup-{self.timestamp}.zip"
-        backup_folder = os.path.join(cache_dir, "backup")
-        backup_file = os.path.join(backup_folder, file_name)
-
-        with zipfile.ZipFile(
-            backup_file, "w", compression=zipfile.ZIP_DEFLATED
-        ) as zip_f:
-            for backup_file in self.backup_files:
-                zip_f.write(backup_file, os.path.basename(backup_file))
-
-        # cleanup
-        for backup_file in self.backup_files:
-            os.remove(backup_file)
-
-    def post_bulk_restore(self, file_name):
-        """send bulk to es"""
-        cache_dir = self.config["application"]["cache_dir"]
-        es_url = self.config["application"]["es_url"]
-        es_auth = self.config["application"]["es_auth"]
-        headers = {"Content-type": "application/x-ndjson"}
-        file_path = os.path.join(cache_dir, file_name)
-
-        with open(file_path, "r", encoding="utf-8") as f:
-            query_str = f.read()
-
-        if not query_str.strip():
-            return
-
-        url = es_url + "/_bulk"
-        request = requests.post(
-            url, data=query_str, headers=headers, auth=es_auth
-        )
-        if not request.ok:
-            print(request.text)
-
-    def unpack_zip_backup(self):
-        """extract backup zip and return filelist"""
-        cache_dir = self.config["application"]["cache_dir"]
-        backup_dir = os.path.join(cache_dir, "backup")
-        backup_files = os.listdir(backup_dir)
-        all_backup_files = ignore_filelist(backup_files)
-        all_available_backups = [
-            i
-            for i in all_backup_files
-            if i.startswith("ta_") and i.endswith(".zip")
-        ]
-        all_available_backups.sort()
-        newest_backup = all_available_backups[-1]
-        file_path = os.path.join(backup_dir, newest_backup)
-
-        with zipfile.ZipFile(file_path, "r") as z:
-            zip_content = z.namelist()
-            z.extractall(backup_dir)
-
-        return zip_content
-
-    def restore_json_files(self, zip_content):
-        """go through the unpacked files and restore"""
-
-        cache_dir = self.config["application"]["cache_dir"]
-        backup_dir = os.path.join(cache_dir, "backup")
-
-        for json_f in zip_content:
-
-            file_name = os.path.join(backup_dir, json_f)
-
-            if not json_f.startswith("es_") or not json_f.endswith(".json"):
-                os.remove(file_name)
-                continue
-
-            print("restoring: " + json_f)
-            self.post_bulk_restore(file_name)
-            os.remove(file_name)
-
-
-def backup_all_indexes():
-    """backup all es indexes to disk"""
-    backup_handler = ElasticBackup(INDEX_CONFIG)
-
-    for index in backup_handler.index_config:
-        index_name = index["index_name"]
-        all_results = backup_handler.get_all_documents(index_name)
-        file_content = backup_handler.build_bulk(all_results)
-        backup_handler.write_es_json(file_content, index_name)
-        backup_handler.write_ta_json(all_results, index_name)
-
-    backup_handler.zip_it()
-
-
-def restore_from_backup():
-    """restore indexes from backup file"""
-    # delete
-    index_check(force_restore=True)
-    # recreate
-    backup_handler = ElasticBackup(INDEX_CONFIG)
-    zip_content = backup_handler.unpack_zip_backup()
-    backup_handler.restore_json_files(zip_content)
-
-
-def index_check(force_restore=False):
-    """check if all indexes are created and have correct mapping"""
-
-    backed_up = False
-
-    for index in INDEX_CONFIG:
-        index_name = index["index_name"]
-        expected_map = index["expected_map"]
-        expected_set = index["expected_set"]
-        handler = ElasticIndex(index_name, expected_map, expected_set)
-        # force restore
-        if force_restore:
-            handler.delete_index(backup=False)
-            handler.create_blank()
-            continue
-
-        # create new
-        if not handler.exists:
-            print(f"create new blank index with name ta_{index_name}...")
-            handler.create_blank()
-            continue
-
-        # validate index
-        rebuild = handler.validate()
-        if rebuild:
-            # make backup before rebuild
-            if not backed_up:
-                print("running backup first")
-                backup_all_indexes()
-                backed_up = True
-
-            print(f"applying new mappings to index ta_{index_name}...")
-            handler.rebuild_index()
-            continue
-
-        # else all good
-        print(f"ta_{index_name} index is created and up to date...")

tubearchivist/home/src/ta/config.py

@@ -0,0 +1,276 @@
+"""
+Functionality:
+- read and write config
+- load config variables into redis
+"""
+
+import json
+import os
+import re
+
+from celery.schedules import crontab
+from home.src.ta.ta_redis import RedisArchivist
+
+
+class AppConfig:
+    """handle user settings and application variables"""
+
+    def __init__(self, user_id=False):
+        self.user_id = user_id
+        self.config = self.get_config()
+        self.colors = self.get_colors()
+
+    def get_config(self):
+        """get config from default file or redis if changed"""
+        config = self.get_config_redis()
+        if not config:
+            config = self.get_config_file()
+
+        if self.user_id:
+            key = f"{self.user_id}:page_size"
+            page_size = RedisArchivist().get_message(key)["status"]
+            if page_size:
+                config["archive"]["page_size"] = page_size
+
+        config["application"].update(self.get_config_env())
+        return config
+
+    def get_config_file(self):
+        """read the defaults from config.json"""
+        with open("home/config.json", "r", encoding="utf-8") as f:
+            config_file = json.load(f)
+
+        config_file["application"].update(self.get_config_env())
+
+        return config_file
+
+    @staticmethod
+    def get_config_env():
+        """read environment application variables"""
+        host_uid_env = os.environ.get("HOST_UID")
+        if host_uid_env:
+            host_uid = int(host_uid_env)
+        else:
+            host_uid = False
+
+        host_gid_env = os.environ.get("HOST_GID")
+        if host_gid_env:
+            host_gid = int(host_gid_env)
+        else:
+            host_gid = False
+
+        es_pass = os.environ.get("ELASTIC_PASSWORD")
+        es_user = os.environ.get("ELASTIC_USER", default="elastic")
+
+        application = {
+            "REDIS_HOST": os.environ.get("REDIS_HOST"),
+            "es_url": os.environ.get("ES_URL"),
+            "es_auth": (es_user, es_pass),
+            "HOST_UID": host_uid,
+            "HOST_GID": host_gid,
+        }
+
+        return application
+
+    @staticmethod
+    def get_config_redis():
+        """read config json set from redis to overwrite defaults"""
+        config = RedisArchivist().get_message("config")
+        if not list(config.values())[0]:
+            return False
+
+        return config
+
+    def update_config(self, form_post):
+        """update config values from settings form"""
+        updated = []
+        for key, value in form_post.items():
+            if not value and not isinstance(value, int):
+                continue
+
+            if value in ["0", 0]:
+                to_write = False
+            elif value == "1":
+                to_write = True
+            else:
+                to_write = value
+
+            config_dict, config_value = key.split("_", maxsplit=1)
+            self.config[config_dict][config_value] = to_write
+            updated.append((config_value, to_write))
+
+        RedisArchivist().set_message("config", self.config, expire=False)
+        return updated
+
+    @staticmethod
+    def set_user_config(form_post, user_id):
+        """set values in redis for user settings"""
+        for key, value in form_post.items():
+            if not value:
+                continue
+
+            message = {"status": value}
+            redis_key = f"{user_id}:{key}"
+            RedisArchivist().set_message(redis_key, message, expire=False)
+
+    def get_colors(self):
+        """overwrite config if user has set custom values"""
+        colors = False
+        if self.user_id:
+            col_dict = RedisArchivist().get_message(f"{self.user_id}:colors")
+            colors = col_dict["status"]
+
+        if not colors:
+            colors = self.config["application"]["colors"]
+
+        self.config["application"]["colors"] = colors
+        return colors
+
+    def load_new_defaults(self):
+        """check config.json for missing defaults"""
+        default_config = self.get_config_file()
+        redis_config = self.get_config_redis()
+
+        # check for customizations
+        if not redis_config:
+            return
+
+        needs_update = False
+
+        for key, value in default_config.items():
+            # missing whole main key
+            if key not in redis_config:
+                redis_config.update({key: value})
+                needs_update = True
+                continue
+
+            # missing nested values
+            for sub_key, sub_value in value.items():
+                if sub_key not in redis_config[key].keys():
+                    redis_config[key].update({sub_key: sub_value})
+                    needs_update = True
+
+        if needs_update:
+            RedisArchivist().set_message("config", redis_config, expire=False)
+
+
+class ScheduleBuilder:
+    """build schedule dicts for beat"""
+
+    SCHEDULES = {
+        "update_subscribed": "0 8 *",
+        "download_pending": "0 16 *",
+        "check_reindex": "0 12 *",
+        "thumbnail_check": "0 17 *",
+        "run_backup": "0 18 0",
+    }
+    CONFIG = ["check_reindex_days", "run_backup_rotate"]
+
+    def __init__(self):
+        self.config = AppConfig().config
+
+    def update_schedule_conf(self, form_post):
+        """process form post"""
+        print("processing form, restart container for changes to take effect")
+        redis_config = self.config
+        for key, value in form_post.items():
+            if key in self.SCHEDULES and value:
+                try:
+                    to_write = self.value_builder(key, value)
+                except ValueError:
+                    print(f"failed: {key} {value}")
+                    mess_dict = {
+                        "status": "message:setting",
+                        "level": "error",
+                        "title": "Scheduler update failed.",
+                        "message": "Invalid schedule input",
+                    }
+                    RedisArchivist().set_message("message:setting", mess_dict)
+                    return
+
+                redis_config["scheduler"][key] = to_write
+            if key in self.CONFIG and value:
+                redis_config["scheduler"][key] = int(value)
+        RedisArchivist().set_message("config", redis_config, expire=False)
+        mess_dict = {
+            "status": "message:setting",
+            "level": "info",
+            "title": "Scheduler changed.",
+            "message": "Please restart container for changes to take effect",
+        }
+        RedisArchivist().set_message("message:setting", mess_dict)
+
+    def value_builder(self, key, value):
+        """validate single cron form entry and return cron dict"""
+        print(f"change schedule for {key} to {value}")
+        if value == "0":
+            # deactivate this schedule
+            return False
+        if re.search(r"[\d]{1,2}\/[\d]{1,2}", value):
+            # number/number cron format will fail in celery
+            print("number/number schedule formatting not supported")
+            raise ValueError
+
+        keys = ["minute", "hour", "day_of_week"]
+        if value == "auto":
+            # set to sensible default
+            values = self.SCHEDULES[key].split()
+        else:
+            values = value.split()
+
+        if len(keys) != len(values):
+            print(f"failed to parse {value} for {key}")
+            raise ValueError("invalid input")
+
+        to_write = dict(zip(keys, values))
+        self._validate_cron(to_write)
+
+        return to_write
+
+    @staticmethod
+    def _validate_cron(to_write):
+        """validate all fields, raise value error for impossible schedule"""
+        all_hours = list(re.split(r"\D+", to_write["hour"]))
+        for hour in all_hours:
+            if hour.isdigit() and int(hour) > 23:
+                print("hour can not be greater than 23")
+                raise ValueError("invalid input")
+
+        all_days = list(re.split(r"\D+", to_write["day_of_week"]))
+        for day in all_days:
+            if day.isdigit() and int(day) > 6:
+                print("day can not be greater than 6")
+                raise ValueError("invalid input")
+
+        if not to_write["minute"].isdigit():
+            print("too frequent: only number in minutes are supported")
+            raise ValueError("invalid input")
+
+        if int(to_write["minute"]) > 59:
+            print("minutes can not be greater than 59")
+            raise ValueError("invalid input")
+
+    def build_schedule(self):
+        """build schedule dict as expected by app.conf.beat_schedule"""
+        schedule_dict = {}
+
+        for schedule_item in self.SCHEDULES:
+            item_conf = self.config["scheduler"][schedule_item]
+            if not item_conf:
+                continue
+
+            minute = item_conf["minute"]
+            hour = item_conf["hour"]
+            day_of_week = item_conf["day_of_week"]
+            schedule_name = f"schedule_{schedule_item}"
+            to_add = {
+                schedule_name: {
+                    "task": schedule_item,
+                    "schedule": crontab(
+                        minute=minute, hour=hour, day_of_week=day_of_week
+                    ),
+                }
+            }
+            schedule_dict.update(to_add)
+
+        return schedule_dict

tubearchivist/home/src/ta/helper.py

@@ -3,31 +3,15 @@ Loose collection of helper functions
 - don't import AppConfig class here to avoid circular imports
 """
 
-import json
-import os
+import random
 import re
 import string
 import subprocess
 import unicodedata
+from datetime import datetime
 from urllib.parse import parse_qs, urlparse
 
-import redis
-import requests
-import yt_dlp as youtube_dl
-
-
-def get_total_hits(index, es_url, es_auth, match_field):
-    """get total hits from index"""
-    headers = {"Content-type": "application/json"}
-    data = {"query": {"match": {match_field: True}}}
-    payload = json.dumps(data)
-    url = f"{es_url}/{index}/_search?filter_path=hits.total"
-    request = requests.post(url, data=payload, headers=headers, auth=es_auth)
-    if not request.ok:
-        print(request.text)
-    total_json = json.loads(request.text)
-    total_hits = total_json["hits"]["total"]["value"]
-    return total_hits
+import yt_dlp
 
 
 def clean_string(file_name):
@@ -53,6 +37,74 @@ def ignore_filelist(filelist):
     return cleaned
 
 
+def randomizor(length):
+    """generate random alpha numeric string"""
+    pool = string.digits + string.ascii_letters
+    return "".join(random.choice(pool) for i in range(length))
+
+
+def requests_headers():
+    """build header with random user agent for requests outside of yt-dlp"""
+
+    chrome_versions = (
+        "90.0.4430.212",
+        "90.0.4430.24",
+        "90.0.4430.70",
+        "90.0.4430.72",
+        "90.0.4430.85",
+        "90.0.4430.93",
+        "91.0.4472.101",
+        "91.0.4472.106",
+        "91.0.4472.114",
+        "91.0.4472.124",
+        "91.0.4472.164",
+        "91.0.4472.19",
+        "91.0.4472.77",
+        "92.0.4515.107",
+        "92.0.4515.115",
+        "92.0.4515.131",
+        "92.0.4515.159",
+        "92.0.4515.43",
+        "93.0.4556.0",
+        "93.0.4577.15",
+        "93.0.4577.63",
+        "93.0.4577.82",
+        "94.0.4606.41",
+        "94.0.4606.54",
+        "94.0.4606.61",
+        "94.0.4606.71",
+        "94.0.4606.81",
+        "94.0.4606.85",
+        "95.0.4638.17",
+        "95.0.4638.50",
+        "95.0.4638.54",
+        "95.0.4638.69",
+        "95.0.4638.74",
+        "96.0.4664.18",
+        "96.0.4664.45",
+        "96.0.4664.55",
+        "96.0.4664.93",
+        "97.0.4692.20",
+    )
+    template = (
+        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
+        + "AppleWebKit/537.36 (KHTML, like Gecko) "
+        + f"Chrome/{random.choice(chrome_versions)} Safari/537.36"
+    )
+
+    return {"User-Agent": template}
+
+
+def date_praser(timestamp):
+    """return formatted date string"""
+    if isinstance(timestamp, int):
+        date_obj = datetime.fromtimestamp(timestamp)
+    elif isinstance(timestamp, str):
+        date_obj = datetime.strptime(timestamp, "%Y-%m-%d")
+
+    return datetime.strftime(date_obj, "%d %b, %Y")
+
+
 class UrlListParser:
     """take a multi line string and detect valid youtube ids"""
 
@@ -70,7 +122,7 @@ class UrlListParser:
                 # is not a url
                 id_type = self.find_valid_id(url)
                 youtube_id = url
-            elif "youtube.com" not in url:
+            elif "youtube.com" not in url and "youtu.be" not in url:
                 raise ValueError(f"{url} is not a youtube link")
             elif parsed.path:
                 # is a url
@@ -121,7 +173,7 @@ class UrlListParser:
             id_type = "video"
         elif str_len == 24:
             id_type = "channel"
-        elif str_len == 34:
+        elif str_len in [34, 18] or id_str in ["LL", "WL"]:
             id_type = "playlist"
         else:
             # unable to parse
@@ -139,7 +191,7 @@ class UrlListParser:
             "extract_flat": True,
             "playlistend": 0,
         }
-        url_info = youtube_dl.YoutubeDL(obs).extract_info(url, download=False)
+        url_info = yt_dlp.YoutubeDL(obs).extract_info(url, download=False)
         try:
             channel_id = url_info["channel_id"]
         except KeyError as error:
@@ -149,143 +201,6 @@ class UrlListParser:
         return channel_id
 
 
-class RedisArchivist:
-    """collection of methods to interact with redis"""
-
-    REDIS_HOST = os.environ.get("REDIS_HOST")
-    REDIS_PORT = os.environ.get("REDIS_PORT")
-    NAME_SPACE = "ta:"
-
-    if not REDIS_PORT:
-        REDIS_PORT = 6379
-
-    def __init__(self):
-        self.redis_connection = redis.Redis(
-            host=self.REDIS_HOST, port=self.REDIS_PORT
-        )
-
-    def set_message(self, key, message, expire=True):
-        """write new message to redis"""
-        self.redis_connection.execute_command(
-            "JSON.SET", self.NAME_SPACE + key, ".", json.dumps(message)
-        )
-
-        if expire:
-            self.redis_connection.execute_command(
-                "EXPIRE", self.NAME_SPACE + key, 20
-            )
-
-    def get_message(self, key):
-        """get message dict from redis"""
-        reply = self.redis_connection.execute_command(
-            "JSON.GET", self.NAME_SPACE + key
-        )
-        if reply:
-            json_str = json.loads(reply)
-        else:
-            json_str = {"status": False}
-
-        return json_str
-
-    def del_message(self, key):
-        """delete key from redis"""
-        response = self.redis_connection.execute_command(
-            "DEL", self.NAME_SPACE + key
-        )
-        return response
-
-    def get_lock(self, lock_key):
-        """handle lock for task management"""
-        redis_lock = self.redis_connection.lock(self.NAME_SPACE + lock_key)
-        return redis_lock
-
-    def get_dl_message(self, cache_dir):
-        """get latest download progress message if available"""
-        reply = self.redis_connection.execute_command(
-            "JSON.GET", self.NAME_SPACE + "progress:download"
-        )
-        if reply:
-            json_str = json.loads(reply)
-        elif json_str := self.monitor_cache_dir(cache_dir):
-            json_str = self.monitor_cache_dir(cache_dir)
-        else:
-            json_str = {"status": False}
-
-        return json_str
-
-    @staticmethod
-    def monitor_cache_dir(cache_dir):
-        """
-        look at download cache dir directly as alternative progress info
-        """
-        dl_cache = os.path.join(cache_dir, "download")
-        all_cache_file = os.listdir(dl_cache)
-        cache_file = ignore_filelist(all_cache_file)
-        if cache_file:
-            filename = cache_file[0][12:].replace("_", " ").split(".")[0]
-            mess_dict = {
-                "status": "downloading",
-                "level": "info",
-                "title": "Downloading: " + filename,
-                "message": "",
-            }
-        else:
-            return False
-
-        return mess_dict
-
-
-class RedisQueue:
-    """dynamically interact with the download queue in redis"""
-
-    REDIS_HOST = os.environ.get("REDIS_HOST")
-    REDIS_PORT = os.environ.get("REDIS_PORT")
-    NAME_SPACE = "ta:"
-
-    if not REDIS_PORT:
-        REDIS_PORT = 6379
-
-    def __init__(self, key):
-        self.key = self.NAME_SPACE + key
-        self.conn = redis.Redis(host=self.REDIS_HOST, port=self.REDIS_PORT)
-
-    def get_all(self):
-        """return all elements in list"""
-        result = self.conn.execute_command("LRANGE", self.key, 0, -1)
-        all_elements = [i.decode() for i in result]
-        return all_elements
-
-    def add_list(self, to_add):
-        """add list to queue"""
-        self.conn.execute_command("RPUSH", self.key, *to_add)
-
-    def add_priority(self, to_add):
-        """add single video to front of queue"""
-        self.clear_item(to_add)
-        self.conn.execute_command("LPUSH", self.key, to_add)
-
-    def get_next(self):
-        """return next element in the queue, False if none"""
-        result = self.conn.execute_command("LPOP", self.key)
-        if not result:
-            return False
-
-        next_element = result.decode()
-        return next_element
-
-    def clear(self):
-        """delete list from redis"""
-        self.conn.execute_command("DEL", self.key)
-
-    def clear_item(self, to_clear):
-        """remove single item from list if it's there"""
-        self.conn.execute_command("LREM", self.key, 0, to_clear)
-
-    def trim(self, size):
-        """trim the queue based on settings amount"""
-        self.conn.execute_command("LTRIM", self.key, 0, size)
-
-
 class DurationConverter:
     """
     using ffmpeg to get and parse duration from filepath
@@ -308,7 +223,11 @@ class DurationConverter:
             capture_output=True,
             check=True,
         )
-        duration_sec = int(float(duration.stdout.decode().strip()))
+        duration_raw = duration.stdout.decode().strip()
+        if duration_raw == "N/A":
+            return 0
+
+        duration_sec = int(float(duration_raw))
         return duration_sec
 
     @staticmethod

tubearchivist/home/src/ta/ta_redis.py

@@ -0,0 +1,161 @@
+"""
+functionality:
+- interact with redis
+- hold temporary download queue in redis
+"""
+
+import json
+import os
+
+import redis
+from home.src.ta.helper import ignore_filelist
+
+
+class RedisBase:
+    """connection base for redis"""
+
+    REDIS_HOST = os.environ.get("REDIS_HOST")
+    REDIS_PORT = os.environ.get("REDIS_PORT") or 6379
+    NAME_SPACE = "ta:"
+
+    def __init__(self):
+        self.conn = redis.Redis(host=self.REDIS_HOST, port=self.REDIS_PORT)
+
+
+class RedisArchivist(RedisBase):
+    """collection of methods to interact with redis"""
+
+    CHANNELS = [
+        "download",
+        "add",
+        "rescan",
+        "subchannel",
+        "subplaylist",
+        "playlistscan",
+        "setting",
+    ]
+
+    def set_message(self, key, message, expire=True):
+        """write new message to redis"""
+        self.conn.execute_command(
+            "JSON.SET", self.NAME_SPACE + key, ".", json.dumps(message)
+        )
+
+        if expire:
+            if isinstance(expire, bool):
+                secs = 20
+            else:
+                secs = expire
+            self.conn.execute_command("EXPIRE", self.NAME_SPACE + key, secs)
+
+    def get_message(self, key):
+        """get message dict from redis"""
+        reply = self.conn.execute_command("JSON.GET", self.NAME_SPACE + key)
+        if reply:
+            json_str = json.loads(reply)
+        else:
+            json_str = {"status": False}
+
+        return json_str
+
+    def list_items(self, query):
+        """list all matches"""
+        reply = self.conn.execute_command(
+            "KEYS", self.NAME_SPACE + query + "*"
+        )
+        all_matches = [i.decode().lstrip(self.NAME_SPACE) for i in reply]
+        all_results = []
+        for match in all_matches:
+            json_str = self.get_message(match)
+            all_results.append(json_str)
+
+        return all_results
+
+    def del_message(self, key):
+        """delete key from redis"""
+        response = self.conn.execute_command("DEL", self.NAME_SPACE + key)
+        return response
+
+    def get_lock(self, lock_key):
+        """handle lock for task management"""
+        redis_lock = self.conn.lock(self.NAME_SPACE + lock_key)
+        return redis_lock
+
+    def get_progress(self):
+        """get a list of all progress messages"""
+        all_messages = []
+        for channel in self.CHANNELS:
+            key = "message:" + channel
+            reply = self.conn.execute_command(
+                "JSON.GET", self.NAME_SPACE + key
+            )
+            if reply:
+                json_str = json.loads(reply)
+                all_messages.append(json_str)
+
+        return all_messages
+
+    @staticmethod
+    def monitor_cache_dir(cache_dir):
+        """
+        look at download cache dir directly as alternative progress info
+        """
+        dl_cache = os.path.join(cache_dir, "download")
+        all_cache_file = os.listdir(dl_cache)
+        cache_file = ignore_filelist(all_cache_file)
+        if cache_file:
+            filename = cache_file[0][12:].replace("_", " ").split(".")[0]
+            mess_dict = {
+                "status": "message:download",
+                "level": "info",
+                "title": "Downloading: " + filename,
+                "message": "",
+            }
+        else:
+            return False
+
+        return mess_dict
+
+
+class RedisQueue(RedisBase):
+    """dynamically interact with the download queue in redis"""
+
+    def __init__(self):
+        super().__init__()
+        self.key = self.NAME_SPACE + "dl_queue"
+
+    def get_all(self):
+        """return all elements in list"""
+        result = self.conn.execute_command("LRANGE", self.key, 0, -1)
+        all_elements = [i.decode() for i in result]
+        return all_elements
+
+    def add_list(self, to_add):
+        """add list to queue"""
+        self.conn.execute_command("RPUSH", self.key, *to_add)
+
+    def add_priority(self, to_add):
+        """add single video to front of queue"""
+        self.clear_item(to_add)
+        self.conn.execute_command("LPUSH", self.key, to_add)
+
+    def get_next(self):
+        """return next element in the queue, False if none"""
+        result = self.conn.execute_command("LPOP", self.key)
+        if not result:
+            return False
+
+        next_element = result.decode()
+        return next_element
+
+    def clear(self):
+        """delete list from redis"""
+        self.conn.execute_command("DEL", self.key)
+
+    def clear_item(self, to_clear):
+        """remove single item from list if it's there"""
+        self.conn.execute_command("LREM", self.key, 0, to_clear)
+
+    def trim(self, size):
+        """trim the queue based on settings amount"""
+        self.conn.execute_command("LTRIM", self.key, 0, size)

tubearchivist/home/tasks.py

@@ -2,49 +2,80 @@
 Functionality:
 - initiate celery app
 - collect tasks
+- user config changes won't get applied here
+  because tasks are initiated at application start
 """
 
 import os
 
 from celery import Celery, shared_task
-from home.src.config import AppConfig
-from home.src.download import ChannelSubscription, PendingList, VideoDownloader
-from home.src.helper import RedisArchivist, RedisQueue, UrlListParser
-from home.src.index_management import backup_all_indexes, restore_from_backup
-from home.src.reindex import (
+from home.apps import StartupCheck
+from home.src.download.queue import PendingList
+from home.src.download.subscriptions import (
+    ChannelSubscription,
+    PlaylistSubscription,
+)
+from home.src.download.thumbnails import ThumbManager, validate_thumbnails
+from home.src.download.yt_dlp_handler import VideoDownloader
+from home.src.es.index_setup import backup_all_indexes, restore_from_backup
+from home.src.index.channel import YoutubeChannel
+from home.src.index.filesystem import (
     ManualImport,
     reindex_old_documents,
     scan_filesystem,
 )
-from home.src.thumbnails import ThumbManager
+from home.src.ta.config import AppConfig, ScheduleBuilder
+from home.src.ta.helper import UrlListParser
+from home.src.ta.ta_redis import RedisArchivist, RedisQueue
 
 CONFIG = AppConfig().config
 REDIS_HOST = os.environ.get("REDIS_HOST")
-REDIS_PORT = os.environ.get("REDIS_PORT")
+REDIS_PORT = os.environ.get("REDIS_PORT") or 6379
 
-if not REDIS_PORT:
-    REDIS_PORT = 6379
-
-os.environ.setdefault("DJANGO_SETTINGS_MODULE", "home.settings")
+os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings")
 app = Celery("tasks", broker=f"redis://{REDIS_HOST}:{REDIS_PORT}")
 app.config_from_object("django.conf:settings", namespace="ta:")
 app.autodiscover_tasks()
+app.conf.timezone = os.environ.get("TZ") or "UTC"
 
 
-@shared_task
+@shared_task(name="update_subscribed")
 def update_subscribed():
     """look for missing videos and add to pending"""
-    channel_handler = ChannelSubscription()
-    missing_videos = channel_handler.find_missing()
-    if missing_videos:
-        pending_handler = PendingList()
-        all_videos_added = pending_handler.add_to_pending(missing_videos)
-        ThumbManager().download_vid(all_videos_added)
-    # check if reindex is needed
-    check_reindex.delay()
+    message = {
+        "status": "message:rescan",
+        "level": "info",
+        "title": "Rescanning channels and playlists.",
+        "message": "Looking for new videos.",
+    }
+    RedisArchivist().set_message("message:rescan", message)
+
+    have_lock = False
+    my_lock = RedisArchivist().get_lock("rescan")
+
+    try:
+        have_lock = my_lock.acquire(blocking=False)
+        if have_lock:
+            channel_handler = ChannelSubscription()
+            missing_from_channels = channel_handler.find_missing()
+            playlist_handler = PlaylistSubscription()
+            missing_from_playlists = playlist_handler.find_missing()
+            missing = missing_from_channels + missing_from_playlists
+            if missing:
+                youtube_ids = [{"type": "video", "url": i} for i in missing]
+                pending_handler = PendingList(youtube_ids=youtube_ids)
+                pending_handler.parse_url_list()
+                pending_handler.add_to_pending()
+
+        else:
+            print("Did not acquire rescan lock.")
+
+    finally:
+        if have_lock:
+            my_lock.release()
 
 
-@shared_task
+@shared_task(name="download_pending")
 def download_pending():
     """download latest pending videos"""
     have_lock = False
@@ -67,7 +98,7 @@ def download_pending():
 @shared_task
 def download_single(youtube_id):
     """start download single video now"""
-    queue = RedisQueue("dl_queue")
+    queue = RedisQueue()
     queue.add_priority(youtube_id)
     print("Added to queue with priority: " + youtube_id)
     # start queue if needed
@@ -77,6 +108,13 @@ def download_single(youtube_id):
     try:
         have_lock = my_lock.acquire(blocking=False)
         if have_lock:
+            mess_dict = {
+                "status": "message:download",
+                "level": "info",
+                "title": "Download single video",
+                "message": "processing",
+            }
+            RedisArchivist().set_message("message:download", mess_dict)
             VideoDownloader().run_queue()
         else:
             print("Download queue already running.")
@@ -90,13 +128,12 @@ def download_single(youtube_id):
 @shared_task
 def extrac_dl(youtube_ids):
     """parse list passed and add to pending"""
-    pending_handler = PendingList()
-    missing_videos = pending_handler.parse_url_list(youtube_ids)
-    all_videos_added = pending_handler.add_to_pending(missing_videos)
-    ThumbManager().download_vid(all_videos_added)
+    pending_handler = PendingList(youtube_ids=youtube_ids)
+    pending_handler.parse_url_list()
+    pending_handler.add_to_pending()
 
 
-@shared_task
+@shared_task(name="check_reindex")
 def check_reindex():
     """run the reindex main command"""
     reindex_old_documents()
@@ -124,17 +161,17 @@ def run_manual_import():
             my_lock.release()
 
 
-@shared_task
-def run_backup():
+@shared_task(name="run_backup")
+def run_backup(reason="auto"):
     """called from settings page, dump backup to zip file"""
-    backup_all_indexes()
+    backup_all_indexes(reason)
     print("backup finished")
 
 
 @shared_task
-def run_restore_backup():
+def run_restore_backup(filename):
     """called from settings page, dump backup to zip file"""
-    restore_from_backup()
+    restore_from_backup(filename)
     print("index restore finished")
 
 
@@ -144,7 +181,7 @@ def kill_dl(task_id):
         app.control.revoke(task_id, terminate=True)
 
     _ = RedisArchivist().del_message("dl_queue_id")
-    RedisQueue("dl_queue").clear()
+    RedisQueue().clear()
 
     # clear cache
     cache_dir = os.path.join(CONFIG["application"]["cache_dir"], "download")
@@ -154,18 +191,25 @@ def kill_dl(task_id):
 
     # notify
     mess_dict = {
-        "status": "downloading",
+        "status": "message:download",
         "level": "error",
-        "title": "Brutally killing download queue",
-        "message": "",
+        "title": "Canceling download process",
+        "message": "Canceling download queue now.",
     }
-    RedisArchivist().set_message("progress:download", mess_dict)
+    RedisArchivist().set_message("message:download", mess_dict)
 
 
 @shared_task
 def rescan_filesystem():
     """check the media folder for mismatches"""
     scan_filesystem()
+    validate_thumbnails()
+
+
+@shared_task(name="thumbnail_check")
+def thumbnail_check():
+    """validate thumbnails"""
+    validate_thumbnails()
 
 
 @shared_task
@@ -179,22 +223,56 @@ def re_sync_thumbs():
 @shared_task
 def subscribe_to(url_str):
     """take a list of urls to subscribe to"""
-    youtube_ids = UrlListParser(url_str).process_list()
-    for youtube_id in youtube_ids:
-        if youtube_id["type"] == "video":
-            to_sub = youtube_id["url"]
-            vid_details = PendingList().get_youtube_details(to_sub)
+    to_subscribe_list = UrlListParser(url_str).process_list()
+    counter = 1
+    for item in to_subscribe_list:
+        to_sub_id = item["url"]
+        if item["type"] == "playlist":
+            new_thumbs = PlaylistSubscription().process_url_str([item])
+            if new_thumbs:
+                ThumbManager().download_playlist(new_thumbs)
+            continue
+
+        if item["type"] == "video":
+            vid_details = PendingList().get_youtube_details(to_sub_id)
             channel_id_sub = vid_details["channel_id"]
-        elif youtube_id["type"] == "channel":
-            channel_id_sub = youtube_id["url"]
+        elif item["type"] == "channel":
+            channel_id_sub = to_sub_id
         else:
-            raise ValueError("failed to subscribe to: " + youtube_id)
+            raise ValueError("failed to subscribe to: " + to_sub_id)
 
         ChannelSubscription().change_subscribe(
             channel_id_sub, channel_subscribed=True
         )
-        print("subscribed to: " + channel_id_sub)
-        # notify
-        RedisArchivist().set_message(
-            "progress:subscribe", {"status": "subscribing"}
-        )
+        # notify for channels
+        message = {
+            "status": "message:subchannel",
+            "level": "info",
+            "title": "Subscribing to Channels",
+            "message": f"Processing {counter} of {len(to_subscribe_list)}",
+        }
+        RedisArchivist().set_message("message:subchannel", message=message)
+        counter = counter + 1
+
+
+@shared_task
+def index_channel_playlists(channel_id):
+    """add all playlists of channel to index"""
+    channel = YoutubeChannel(channel_id)
+    # notify
+    mess_dict = {
+        "status": "message:playlistscan",
+        "level": "info",
+        "title": "Looking for playlists",
+        "message": f'Scanning channel "{channel.youtube_id}" in progress',
+    }
+    RedisArchivist().set_message("message:playlistscan", mess_dict)
+    channel.index_channel_playlists()
+
+
+try:
+    app.conf.beat_schedule = ScheduleBuilder().build_schedule()
+except KeyError:
+    # update path from v0.0.8 to v0.0.9 to load new defaults
+    StartupCheck().sync_redis_state()
+    app.conf.beat_schedule = ScheduleBuilder().build_schedule()

tubearchivist/home/templates/home/about.html

@@ -1,17 +1,19 @@
 {% extends "home/base.html" %}
 {% load static %}
 {% block content %}
-<div class="title-bar">
-    <h1>About The Tube Archivist</h1>
-</div>
-<div class="about-section">
-    <h2>Useful Links</h2>
-    <p>This project is in active and constant development, take a look at the <a href="https://github.com/bbilly1/tubearchivist#roadmap" target="_blank">roadmap</a> for a overview.</p>
-    <p>For any questions on what a button or a function does, You can find the up-to-date user documentation on <a href="https://github.com/bbilly1/tubearchivist/wiki" target="_blank">Github</a>.</p>
-    <p>All contributions are welcome: Open an <a href="https://github.com/bbilly1/tubearchivist/issues" target="_blank">issue</a> for any bugs and errors, start a <a href="https://github.com/bbilly1/tubearchivist/discussions" target="_blank">discussion</a> for anything that will require a more indepth look. The <a href="https://github.com/bbilly1/tubearchivist/blob/master/CONTRIBUTING.md" target="_blank">contributing</a> page is a good place to get started.</p>
-</div>
-<div class="about-section">
-    <h2>Donate</h2>
-    <p>Here are <a href="https://github.com/bbilly1/tubearchivist#donate" target="_blank">some links</a>, if you want to buy the developer a coffee. Thank you for your support!</p>
+<div class="boxed-content">
+    <div class="title-bar">
+        <h1>About The Tube Archivist</h1>
+    </div>
+    <div class="about-section">
+        <h2>Useful Links</h2>
+        <p>This project is in active and constant development, take a look at the <a href="https://github.com/tubearchivist/tubearchivist#roadmap" target="_blank">roadmap</a> for a overview.</p>
+        <p>For any questions on what a button or a function does, You can find the up-to-date user documentation on <a href="https://github.com/tubearchivist/tubearchivist/wiki" target="_blank">Github</a>.</p>
+        <p>All contributions are welcome: Open an <a href="https://github.com/tubearchivist/tubearchivist/issues" target="_blank">issue</a> for any bugs and errors, join us on <a href="https://discord.gg/AFwz8nE7BK" target="_blank">Discord</a> to discuss details. The <a href="https://github.com/tubearchivist/tubearchivist/blob/master/CONTRIBUTING.md" target="_blank">contributing</a> page is a good place to get started.</p>
+    </div>
+    <div class="about-section">
+        <h2>Donate</h2>
+        <p>Here are <a href="https://github.com/tubearchivist/tubearchivist#donate" target="_blank">some links</a>, if you want to buy the developer a coffee. Thank you for your support!</p>
+    </div>
 </div>
 {% endblock content %}

tubearchivist/home/templates/home/base.html

@@ -6,7 +6,17 @@
     <meta http-equiv="X-UA-Compatible" content="IE=edge">
     <meta name="viewport" content="width=device-width, initial-scale=1.0">
     <link rel="stylesheet" type="text/css" href="{% static 'css/style.css' %}">
-    <link rel="shortcut icon" type="image/png" href="{% static 'favicon.ico' %}"/>
+    <link rel="apple-touch-icon" sizes="180x180" href="{% static 'favicon/apple-touch-icon.png' %}">
+    <link rel="icon" type="image/png" sizes="32x32" href="{% static 'favicon/favicon-32x32.png' %}">
+    <link rel="icon" type="image/png" sizes="16x16" href="{% static 'favicon/favicon-16x16.png' %}">
+    <link rel="manifest" href="{% static 'favicon/site.webmanifest' %}">
+    <link rel="mask-icon" href="{% static 'favicon/safari-pinned-tab.svg' %}" color="#01202e">
+    <link rel="shortcut icon" href="{% static 'favicon/favicon.ico' %}">
+    <meta name="apple-mobile-web-app-title" content="TubeArchivist">
+    <meta name="application-name" content="TubeArchivist">
+    <meta name="msapplication-TileColor" content="#01202e">
+    <meta name="msapplication-config" content="{% static 'favicon/browserconfig.xml' %}">
+    <meta name="theme-color" content="#01202e">
     {% if title %}
         <title>TA | {{ title }}</title>
     {% else %}
@@ -18,44 +28,54 @@
         <link rel="stylesheet" href="{% static 'css/light.css' %}">
     {% endif %}
     <script type="text/javascript" src="{% static 'script.js' %}"></script>
+    {% if cast %}
+        <script type="text/javascript" src="https://www.gstatic.com/cv/js/sender/v1/cast_sender.js?loadCastFramework=1"></script>
+        <script id="cast-script" type="text/javascript" src="{% static 'cast-videos.js' %}"></script>
+    {% endif %}
 </head>
 <body>
-    <div class="boxed-content">
-        <div class="top-banner">
-            <a href="{% url 'home' %}">
-                {% if colors == 'dark' %}
-                    <img src="{% static 'img/banner-tube-archivist-dark.png' %}" alt="tube-archivist-banner">
-                {% endif %}
-                {% if colors == 'light' %}
-                    <img src="{% static 'img/banner-tube-archivist-light.png' %}" alt="tube-archivist-banner">
-                {% endif %}
-            </a>
-        </div>
-        <div class="top-nav">
-            <div class="nav-items">
+    <div class="main-content">
+        <div class="boxed-content">
+            <div class="top-banner">
                 <a href="{% url 'home' %}">
-                    <div class="nav-item">home</div>
-                </a>
-                <a href="{% url 'channel' %}">
-                    <div class="nav-item">channels</div>
-                </a>
-                <a href="{% url 'downloads' %}">
-                    <div class="nav-item">downloads</div>
+                    {% if colors == 'dark' %}
+                        <img src="{% static 'img/banner-tube-archivist-dark.png' %}" alt="tube-archivist-banner">
+                    {% endif %}
+                    {% if colors == 'light' %}
+                        <img src="{% static 'img/banner-tube-archivist-light.png' %}" alt="tube-archivist-banner">
+                    {% endif %}
                 </a>
             </div>
-            <div class="nav-icons">
-                <a href="{% url 'about' %}">
-                    <img src="{% static 'img/icon-help.svg' %}" alt="help-icon" title="About">
-                </a>
-                <a href="{% url 'settings' %}">
-                    <img src="{% static 'img/icon-gear.svg' %}" alt="gear-icon" title="Settings">
-                </a>
-                <a href="{% url 'logout' %}">
-                    <img src="{% static 'img/icon-exit.svg' %}" alt="exit-icon" title="Logout">
-                </a>
+            <div class="top-nav">
+                <div class="nav-items">
+                    <a href="{% url 'home' %}">
+                        <div class="nav-item">home</div>
+                    </a>
+                    <a href="{% url 'channel' %}">
+                        <div class="nav-item">channels</div>
+                    </a>
+                    <a href="{% url 'playlist' %}">
+                        <div class="nav-item">playlists</div>
+                    </a>
+                    <a href="{% url 'downloads' %}">
+                        <div class="nav-item">downloads</div>
+                    </a>
+                </div>
+                <div class="nav-icons">
+                    <a href="{% url 'search' %}">
+                        <img src="{% static 'img/icon-search.svg' %}" alt="search-icon" title="Search">
+                    </a>
+                    <a href="{% url 'settings' %}">
+                        <img src="{% static 'img/icon-gear.svg' %}" alt="gear-icon" title="Settings">
+                    </a>
+                    <a href="{% url 'logout' %}">
+                        <img class="alert-hover" src="{% static 'img/icon-exit.svg' %}" alt="exit-icon" title="Logout">
+                    </a>
+                </div>
             </div>
         </div>
         {% block content %}{% endblock %}
+        <div class="boxed-content">
             <div class="pagination">
                 {% if pagination %}
                     {% if pagination.current_page > 1 %}
@@ -89,18 +109,31 @@
                     {% endif %}
                     {% if pagination.last_page > 0 %}
                         {% if pagination.search_get %}
-                        <a class="pagination-item" href="?page={{ pagination.last_page }}&search={{ pagination.search_get }}">Last ({{ pagination.last_page }})</a>
+                        <a class="pagination-item" href="?page={{ pagination.last_page }}&search={{ pagination.search_get }}">
+                            {% if pagination.max_hits %}
+                                Max ({{ pagination.last_page }})
+                            {% else %}
+                                Last ({{ pagination.last_page }})
+                            {% endif %}
+                        </a>
                         {% else %}
-                        <a class="pagination-item" href="?page={{ pagination.last_page }}">Last ({{ pagination.last_page }})</a>
+                        <a class="pagination-item" href="?page={{ pagination.last_page }}">
+                            {% if pagination.max_hits %}
+                                Max ({{ pagination.last_page }})
+                            {% else %}
+                                Last ({{ pagination.last_page }})
+                            {% endif %}
+                        </a>
                         {% endif %}
                     {% endif %}
                 {% endif %}
             </div>
+        </div>
     </div>
     <div class="footer">
         <div class="boxed-content">
-            <span>© 2021 The Tube Archivist v0.0.7 | <a href="https://github.com/bbilly1/tubearchivist" target="_blank">Github</a> | <a href="https://hub.docker.com/r/bbilly1/tubearchivist" target="_blank">Docker Hub</a></span>
+            <span>© 2021 - <script type="text/javascript">document.write(new Date().getFullYear());</script> TubeArchivist {{ version }} </span><span><a href="{% url 'about' %}">About</a> | <a href="https://github.com/tubearchivist/tubearchivist" target="_blank">GitHub</a> | <a href="https://hub.docker.com/r/bbilly1/tubearchivist" target="_blank">Docker Hub</a> | <a href="https://discord.gg/AFwz8nE7BK" target="_blank">Discord</a> | <a href="https://www.reddit.com/r/TubeArchivist/">Reddit</a></span>
         </div>
     </div>
 </body>
-</html>
+</html>

tubearchivist/home/templates/home/channel.html

@@ -2,16 +2,13 @@
 {% load static %}
 {% load humanize %}
 {% block content %}
-<div class="title-bar">
-    <h1>Channels</h1>
-</div>
-<div class="info-box info-box-2">
-    <div class="icon-text">
-        {% if running == "subscribing" %}
-            <p>Subscribing in progress, refresh.</p>
-        {% else %}
-            <img id="add-icon" onclick="showForm()" src="{% static 'img/icon-add.svg' %}" alt="add-icon">
-            <p>Subscribe to Channels</p>
+<div class="boxed-content">
+    <div class="title-split">
+        <div class="title-bar">
+            <h1>Channels</h1>
+        </div>
+        <div class="title-split-form">
+            <img id="animate-icon" onclick="showForm()" src="{% static 'img/icon-add.svg' %}" alt="add-icon" title="Subscribe to Channels">
             <div class="show-form">
                 <form id="hidden-form" action="/channel/" method="post">
                     {% csrf_token %}
@@ -19,83 +16,69 @@
                     <button type="submit">Subscribe</button>
                 </form>
             </div>
-        {% endif %}
-    </div>
-    <div class="search-form icon-text">
-        <div class="search-icon">
-            <img src="{% static 'img/icon-search.svg' %}" alt="search-icon" onclick="showSearch()">
-            <p>Search your Channels</p>
         </div>
-        <form onSubmit="return channelRedirect();" id="search-box">
-            {% csrf_token %}
-            {{ search_form }}
-            <datalist id="resultBox">
-            </datalist>
-        </form>
     </div>
-</div>
-<div class="view-controls">
-    <div class="toggle">
-        <span>Show only subscribed channels:</span>
-        <div class="toggleBox">
-            <input 
-                id="show_subed_only" onclick="toggleCheckbox(this)" type="checkbox"
-                {% if show_subed_only %}
-                    checked
+    <div id="notifications" data="channel"></div>
+    <div class="view-controls">
+        <div class="toggle">
+            <span>Show subscribed only:</span>
+            <div class="toggleBox">
+                <input id="show_subed_only" onclick="toggleCheckbox(this)" type="checkbox" {% if show_subed_only %}checked{% endif %}>
+                {% if not show_subed_only %}
+                    <label for="" class="ofbtn">Off</label>
+                {% else %}
+                    <label for="" class="onbtn">On</label>
                 {% endif %}
-                >
-            <label for="" class="onbtn">On</label>
-            <label for="" class="ofbtn">Off</label>
+            </div>
+        </div>
+        <div class="view-icons">
+            <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="channel" data-value="grid" alt="grid view">
+            <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="channel" data-value="list" alt="list view">
         </div>
     </div>
-    <div class="view-icons">
-        <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="channel" data-value="grid" alt="grid view">
-        <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="channel" data-value="list" alt="list view">
-    </div>
-</div>
-<h2>Total matching channels: {{ max_hits }}</h2>
-<div class="channel-list {{ view_style }}">
-    {% if channels %}
-        {% for channel in channels %}
-        <div class="channel-item {{ view_style }}">
-            <div class="channel-banner {{ view_style }}">
-                <a href="{% url 'channel_id' channel.source.channel_id %}">
-                    <img src="/cache/channels/{{ channel.source.channel_id }}_banner.jpg" alt="{{ channel.source.channel_id }}-banner">
-                </a>
-            </div>
-            <div class="info-box info-box-2 {{ view_style }}">
-                <div class="info-box-item">
-                    <div class="round-img">
+    <h2>Total matching channels: {{ max_hits }}</h2>
+    <div class="channel-list {{ view_style }}">
+        {% if results %}
+            {% for channel in results %}
+                <div class="channel-item {{ view_style }}">
+                    <div class="channel-banner {{ view_style }}">
                         <a href="{% url 'channel_id' channel.source.channel_id %}">
-                            <img src="/cache/channels/{{ channel.source.channel_id }}_thumb.jpg" alt="channel-thumb">
+                            <img src="/cache/channels/{{ channel.source.channel_id }}_banner.jpg" alt="{{ channel.source.channel_id }}-banner">
                         </a>
                     </div>
-                    <div>
-                        <h3><a href="{% url 'channel_id' channel.source.channel_id %}">{{ channel.source.channel_name }}</a></h3>
-                        {% if channel.source.channel_subs >= 1000000 %}
-                            <p>Subscribers: {{ channel.source.channel_subs|intword }}</p>
-                        {% else %}
-                            <p>Subscribers: {{ channel.source.channel_subs|intcomma }}</p>
-                        {% endif %}
+                    <div class="info-box info-box-2 {{ view_style }}">
+                        <div class="info-box-item">
+                            <div class="round-img">
+                                <a href="{% url 'channel_id' channel.source.channel_id %}">
+                                    <img src="/cache/channels/{{ channel.source.channel_id }}_thumb.jpg" alt="channel-thumb">
+                                </a>
+                            </div>
+                            <div>
+                                <h3><a href="{% url 'channel_id' channel.source.channel_id %}">{{ channel.source.channel_name }}</a></h3>
+                                {% if channel.source.channel_subs >= 1000000 %}
+                                    <p>Subscribers: {{ channel.source.channel_subs|intword }}</p>
+                                {% else %}
+                                    <p>Subscribers: {{ channel.source.channel_subs|intcomma }}</p>
+                                {% endif %}
+                            </div>
+                        </div>
+                        <div class="info-box-item">
+                            <div>
+                                <p>Last refreshed: {{ channel.source.channel_last_refresh }}</p>
+                                {% if channel.source.channel_subscribed %}
+                                    <button class="unsubscribe" type="button" id="{{ channel.source.channel_id }}" onclick="unsubscribe(this.id)" title="Unsubscribe from {{ channel.source.channel_name }}">Unsubscribe</button>
+                                {% else %}
+                                    <button type="button" id="{{ channel.source.channel_id }}" onclick="subscribe(this.id)" title="Subscribe to {{ channel.source.channel_name }}">Subscribe</button>
+                                {% endif %}
+                            </div>
+                        </div>
                     </div>
                 </div>
-                <div class="info-box-item">
-                    <div>
-                        <p>Last refreshed: {{ channel.source.channel_last_refresh }}</p>
-                        <p>Subscribed: 
-                        {% if channel.source.channel_subscribed %}
-                            <button type="button" id="{{ channel.source.channel_id }}" onclick="unsubscribe(this.id)" title="Unsubscribe from {{ channel.source.channel_name }}">Unsubscribe</button>
-                        {% else %}
-                            false
-                        {% endif %}
-                        </p>
-                    </div>
-                </div>
-            </div>
-        </div>
-        {% endfor %}
-    {% else %}
-        <h2>No channels found...</h2>
-    {% endif %}
+            {% endfor %}
+        {% else %}
+            <h2>No channels found...</h2>
+        {% endif %}
+    </div>
 </div>
+<script type="text/javascript" src="{% static 'progress.js' %}"></script>
 {% endblock content %}

tubearchivist/home/templates/home/channel_id.html

@@ -2,141 +2,191 @@
 {% block content %}
 {% load static %}
 {% load humanize %}
-<div class="title-bar">
-    <h1>Channel: {{ channel_info.channel_name }}</h1>
-</div>
-<div class="channel-banner">
-    <a href="/channel/{{ channel_info.channel_id }}/"><img src="/cache/channels/{{ channel_info.channel_id }}_banner.jpg" alt="channel_banner"></a>
-</div>
-<div class="info-box info-box-3">
-    <div class="info-box-item">
-        <div class="round-img">
-            <a href="{% url 'channel_id' channel_info.channel_id %}">
-                <img src="/cache/channels/{{ channel_info.channel_id }}_thumb.jpg" alt="channel-thumb">
-            </a>
-        </div>
-        <div>
-            <h3><a href="{% url 'channel_id' channel_info.channel_id %}">{{ channel_info.channel_name }}</a></h3>
-            {% if channel_info.channel_subs >= 1000000 %}
-                <span>Subscribers: {{ channel_info.channel_subs|intword }}</span>
-            {% else %}
-                <span>Subscribers: {{ channel_info.channel_subs|intcomma }}</span>
-            {% endif %}
-        </div>
+<div class="boxed-content">
+    <div class="channel-banner">
+        <a href="/channel/{{ channel_info.channel_id }}/"><img src="/cache/channels/{{ channel_info.channel_id }}_banner.jpg" alt="channel_banner"></a>
     </div>
-    <div class="info-box-item">
-        <div>
-            <p>Last refreshed: {{ channel_info.channel_last_refresh }}</p>
-            <p>Subscribed: 
-            {% if channel_info.channel_subscribed %}
-                <button type="button" id="{{ channel_info.channel_id }}" onclick="unsubscribe(this.id)" title="Unsubscribe from {{ channel_info.channel_name }}">Unsubscribe</button>
-            {% else %}
-                false
-            {% endif %}
-            </p>
-            <p>Channel id: {{ channel_info.channel_id }}</p>
-            <button onclick="deleteConfirm()" id="delete-item">Delete Channel</button>
-            <div class="delete-confirm" id="delete-button">
-                <span>Delete {{ channel_info.channel_name }} including all videos? </span><button class="danger-button" onclick="deleteChannel(this)" data-id="{{ channel_info.channel_id }}">Delete</button> <button onclick="cancelDelete()">Cancel</button>
+    <div id="notifications" data="channel_id"></div>
+    <div class="info-box info-box-3">
+        <div class="info-box-item">
+            <div class="round-img">
+                <a href="{% url 'channel_id' channel_info.channel_id %}">
+                    <img src="/cache/channels/{{ channel_info.channel_id }}_thumb.jpg" alt="channel-thumb">
+                </a>
+            </div>
+            <div>
+                <h3><a href="{% url 'channel_id' channel_info.channel_id %}">{{ channel_info.channel_name }}</a></h3>
+                {% if channel_info.channel_subs >= 1000000 %}
+                    <p>Subscribers: {{ channel_info.channel_subs|intword }}</p>
+                {% else %}
+                    <p>Subscribers: {{ channel_info.channel_subs|intcomma }}</p>
+                {% endif %}
+                {% if channel_info.channel_subscribed %}
+                    <button class="unsubscribe" type="button" id="{{ channel_info.channel_id }}" onclick="unsubscribe(this.id)" title="Unsubscribe from {{ channel_info.channel_name }}">Unsubscribe</button>
+                {% else %}
+                    <button type="button" id="{{ channel_info.channel_id }}" onclick="subscribe(this.id)" title="Subscribe to {{ channel_info.channel_name }}">Subscribe</button>
+                {% endif %}
             </div>
         </div>
-    </div>
-    <div class="info-box-item">
-        <div>
-            {% if channel_info.channel_views >= 1000000 %}
-                <p>Channel views: {{ channel_info.channel_views|intword }}</p>
-            {% elif channel_info.channel_views > 0 %}
-                <p>Channel views: {{ channel_info.channel_views|intcomma }}</p>
-            {% endif %}
-            {% if max_hits %}
-                <p>Total Videos archived: {{ max_hits }}</p>
-                <p>Watched: <button title="Mark all videos from {{ channel_info.channel_name }} as watched" type="button" id="{{ channel_info.channel_id }}" onclick="isWatched(this.id)">Mark as watched</button></p>
-            {% endif %}
-        </div>
-    </div>
-</div>
-{% if channel_info.channel_description %}
-    <div class="info-box-item description-box">
-        <p>Description: <button onclick="textReveal()" id="text-reveal-button">Show</button></p>
-        <div id="text-reveal" class="description-text">
-            {{ channel_info.channel_description|linebreaks }}
-        </div>
-    </div>
-{% endif %}
-<div id="player" class="video-player"></div>
-<div class="sort">
-    <p>Sort by <span class="settings-current">{{ sort_by }}</span>
-        <select name="sort" id="sort" onchange="sortChange(this.value)">
-            <option value="" disabled selected> -- change sort by -- </option>
-            <option value="published">date published</option>
-            <option value="downloaded">date downloaded</option>
-            <option value="views">views</option>
-            <option value="likes">likes</option>
-        </select>
-        <select name="sord-order" id="sort-order" onchange="sortChange(this.value)">
-            {% if sort_order == "asc" %}
-                <option value="asc" selected>asc</option>
-            {% else %}
-                <option value="asc">asc</option>
-            {% endif %}
-            {% if sort_order == "desc" %}
-                <option value="desc" selected>desc</option>
-            {% else %}
-                <option value="desc">desc</option>
-            {% endif %}
-        </select>
-    </p>
-</div>
-<div class="view-controls">
-    <div class="toggle">
-        <span>Hide watched videos:</span>
-        <div class="toggleBox">
-            <input 
-                id="hide_watched" onclick="toggleCheckbox(this)" type="checkbox"
-                {% if hide_watched %}
-                    checked
+        <div class="info-box-item">
+            <div>
+                <p>Last refreshed: {{ channel_info.channel_last_refresh }}</p>
+                {% if channel_info.channel_active %}
+                    <p>Youtube: <a href="https://www.youtube.com/channel/{{ channel_info.channel_id }}" target="_blank">Active</a></p>
+                {% else %}
+                    <p>Youtube: Deactivated</p>
                 {% endif %}
-                >
-            <label for="" class="onbtn">On</label>
-            <label for="" class="ofbtn">Off</label>
-        </div>
-    </div>
-    <div class="view-icons">
-        <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="grid" alt="grid view">
-        <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="list" alt="list view">
-    </div>
-</div>
-<div class="video-list {{ view_style }}">
-    {% if videos %}
-        {% for video in videos %}
-            <div class="video-item {{ view_style }}">
-                <a href="#player" data-src="/media/{{ video.source.media_url }}" data-thumb="/cache/{{ video.source.vid_thumb_url }}" data-title="{{ video.source.title }}" data-channel="{{ video.source.channel.channel_name }}" data-id="{{ video.source.youtube_id }}" onclick="createPlayer(this)">
-                    <div class="video-thumb-wrap {{ view_style }}">
-                        <div class="video-thumb">
-                            <img src="/cache/{{ video.source.vid_thumb_url }}" alt="video-thumb">
-                        </div>
-                        <div class="video-play">
-                            <img src="{% static 'img/icon-play.svg' %}" alt="play-icon">
-                        </div>
-                    </div>
-                </a>
-                <div class="video-desc {{ view_style }}">
-                    <div class="video-desc-player" id="video-info-{{ video.source.youtube_id }}">
-                        {% if video.source.player.watched %}
-                            <img src="{% static 'img/icon-seen.svg' %}" alt="seen-icon" id="{{ video.source.youtube_id }}" onclick="isUnwatched(this.id)" class="seen-icon" title="Mark as unwatched">
-                        {% else %}
-                            <img src="{% static 'img/icon-unseen.svg' %}" alt="unseen-icon" id="{{ video.source.youtube_id }}" onclick="isWatched(this.id)" class="unseen-icon" title="Mark as watched.">
-                        {% endif %}
-                        <span>{{ video.source.published }} | {{ video.source.player.duration_str }}</span>
-                    </div>
-                    <div>
-                        <a class="video-more" href="{% url 'video' video.source.youtube_id %}"><h2>{{ video.source.title }}</h2></a>
-                    </div>
+                <button onclick="deleteConfirm()" id="delete-item">Delete Channel</button>
+                <div class="delete-confirm" id="delete-button">
+                    <span>Delete {{ channel_info.channel_name }} including all videos? </span><button class="danger-button" onclick="deleteChannel(this)" data-id="{{ channel_info.channel_id }}">Delete</button> <button onclick="cancelDelete()">Cancel</button>
                 </div>
             </div>
-        {% endfor %}
-    {% else %}
-        <h2>No videos found...</h2>
+        </div>
+        <div class="info-box-item">
+            <div>
+                {% if channel_info.channel_views >= 1000000 %}
+                    <p>Channel views: {{ channel_info.channel_views|intword }}</p>
+                {% elif channel_info.channel_views > 0 %}
+                    <p>Channel views: {{ channel_info.channel_views|intcomma }}</p>
+                {% endif %}
+                {% if max_hits %}
+                    <p>Total Videos archived: {{ max_hits }}</p>
+                    <p>Watched: <button title="Mark all videos from {{ channel_info.channel_name }} as watched" type="button" id="watched-button" data-id="{{ channel_info.channel_id }}" onclick="isWatchedButton(this)">Mark as watched</button></p>
+                {% endif %}
+                <button {% if channel_info.channel_overwrites %} class="danger-button"{% endif %} onclick="showOverwrite()" title="Overwrite settings for channel {{ channel_info.channel_name }}">Configure</button>
+                <a href="/playlist/?search={{ channel_info.channel_id }}" title="Show all playlists belonging to {{ channel_info.channel_name }}"><button>Show Playlists</button></a>
+            </div>
+        </div>
+    </div>
+    <div id="overwrite-form" class="info-box{% if not channel_info.channel_overwrites %} hidden-overwrite{% endif %}">
+        <div class="info-box-item">
+            <form class="overwrite-form" action="/channel/{{ channel_info.channel_id }}/" method="POST">
+                {% csrf_token %}
+                <div class="overwrite-form-item">
+                    <p>Download format: <span class="settings-current">
+                        {% if channel_info.channel_overwrites.download_format %}
+                            {{ channel_info.channel_overwrites.download_format }}
+                        {% else %}
+                            False
+                        {% endif %}</span></p>
+                    {{ channel_overwrite_form.download_format }}<br>
+                </div>
+                <div class="overwrite-form-item">
+                    <p>Auto delete watched videos after x days: <span class="settings-current">
+                        {% if channel_info.channel_overwrites.autodelete_days %}
+                            {{ channel_info.channel_overwrites.autodelete_days }}
+                        {% else %}
+                            False
+                        {% endif %}</span></p>
+                    {{ channel_overwrite_form.autodelete_days }}<br>
+                </div>
+                <div class="overwrite-form-item">
+                    <p>Index playlists: <span class="settings-current">
+                        {% if channel_info.channel_overwrites.index_playlists %}
+                            {{ channel_info.channel_overwrites.index_playlists }}
+                        {% else %}
+                            False
+                        {% endif %}</span></p>
+                    {{ channel_overwrite_form.index_playlists }}<br>
+                </div>
+                <div class="overwrite-form-item">
+                    <p>Enable <a href="https://sponsor.ajay.app/" target="_blank">SponsorBlock</a>: <span class="settings-current">
+                        {% if channel_info.channel_overwrites.integrate_sponsorblock %}
+                            {{ channel_info.channel_overwrites.integrate_sponsorblock }}
+                        {% elif channel_info.channel_overwrites.integrate_sponsorblock == False %}
+                            Disabled
+                        {% else %}
+                            False
+                        {% endif %}</span></p>
+                    {{ channel_overwrite_form.integrate_sponsorblock }}<br>
+                </div>
+                <button type="submit">Save Channel Overwrites</button>
+            </form>
+        </div>
+    </div>
+    {% if channel_info.channel_description %}
+        <div class="info-box-item description-box">
+            <p>Description: <button onclick="textReveal()" id="text-reveal-button">Show</button></p>
+            <div id="text-reveal" class="description-text">
+                {{ channel_info.channel_description|linebreaks }}
+            </div>
+        </div>
     {% endif %}
+    <div class="view-controls">
+        <div class="toggle">
+            <span>Hide watched videos:</span>
+            <div class="toggleBox">
+                <input id="hide_watched" onclick="toggleCheckbox(this)" type="checkbox" {% if hide_watched %}checked{% endif %}>
+                {% if not hide_watched %}
+                    <label for="" class="ofbtn">Off</label>
+                {% else %}
+                    <label for="" class="onbtn">On</label>
+                {% endif %}
+            </div>
+        </div>
+        <div class="sort">
+            <div id="hidden-form">
+                <span>Sort by:</span>
+                <select name="sort" id="sort" onchange="sortChange(this.value)">
+                    <option value="published" {% if sort_by == "published" %}selected{% endif %}>date published</option>
+                    <option value="downloaded" {% if sort_by == "downloaded" %}selected{% endif %}>date downloaded</option>
+                    <option value="views" {% if sort_by == "views" %}selected{% endif %}>views</option>
+                    <option value="likes" {% if sort_by == "likes" %}selected{% endif %}>likes</option>
+                </select>
+                <select name="sord-order" id="sort-order" onchange="sortChange(this.value)">
+                    <option value="asc" {% if sort_order == "asc" %}selected{% endif %}>asc</option>
+                    <option value="desc" {% if sort_order == "desc" %}selected{% endif %}>desc</option>
+                </select>
+            </div>
+        </div>
+        <div class="view-icons">
+            <img src="{% static 'img/icon-sort.svg' %}" alt="sort-icon" onclick="showForm()" id="animate-icon">
+            <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="grid" alt="grid view">
+            <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="list" alt="list view">
+        </div>
+    </div>
 </div>
+<div id="player" class="player-wrapper"></div>
+<div class="boxed-content">
+    <div class="video-list {{ view_style }}">
+        {% if results %}
+            {% for video in results %}
+                <div class="video-item {{ view_style }}">
+                    <a href="#player" data-id="{{ video.source.youtube_id }}" onclick="createPlayer(this)">
+                        <div class="video-thumb-wrap {{ view_style }}">
+                            <div class="video-thumb">
+                                <img src="/cache/{{ video.source.vid_thumb_url }}" alt="video-thumb">
+                                {% if video.source.player.progress %}
+                                    <div class="video-progress-bar" id="progress-{{ video.source.youtube_id }}" style="width: {{video.source.player.progress}}%;"></div>
+                                {% else %}
+                                    <div class="video-progress-bar" id="progress-{{ video.source.youtube_id }}" style="width: 0%;"></div>
+                                {% endif %}
+                            </div>
+                            <div class="video-play">
+                                <img src="{% static 'img/icon-play.svg' %}" alt="play-icon">
+                            </div>
+                        </div>
+                    </a>
+                    <div class="video-desc {{ view_style }}">
+                        <div class="video-desc-player" id="video-info-{{ video.source.youtube_id }}">
+                            {% if video.source.player.watched %}
+                                <img src="{% static 'img/icon-seen.svg' %}" alt="seen-icon" data-id="{{ video.source.youtube_id }}" data-status="watched" onclick="updateVideoWatchStatus(this)" class="watch-button" title="Mark as unwatched">
+                            {% else %}
+                                <img src="{% static 'img/icon-unseen.svg' %}" alt="unseen-icon" data-id="{{ video.source.youtube_id }}" data-status="unwatched" onclick="updateVideoWatchStatus(this)" class="watch-button" title="Mark as watched">
+                            {% endif %}
+                            <span>{{ video.source.published }} | {{ video.source.player.duration_str }}</span>
+                        </div>
+                        <div>
+                            <a class="video-more" href="{% url 'video' video.source.youtube_id %}"><h2>{{ video.source.title }}</h2></a>
+                        </div>
+                    </div>
+                </div>
+            {% endfor %}
+        {% else %}
+            <h2>No videos found...</h2>
+            <p>Try going to the <a href="{% url 'downloads' %}">downloads page</a> to start the scan and download tasks.</p>
+        {% endif %}
+    </div>
+</div>
+<script type="text/javascript" src="{% static 'progress.js' %}"></script>
 {% endblock content %}

tubearchivist/home/templates/home/downloads.html

@@ -1,89 +1,92 @@
 {% extends "home/base.html" %}
 {% load static %}
 {% block content %}
-<div class="title-bar">
-    <h1>Downloads</h1>
-</div>
-<div id="downloadMessage"></div>
-<div id="downloadControl"></div>
-<div class="info-box info-box-3">
-    <div class="icon-text">
-        <img id="rescan-icon" onclick="rescanPending()" src="{% static 'img/icon-rescan.svg' %}" alt="rescan-icon">
-        <p>Rescan subscriptions</p>
+<div class="boxed-content">
+    <div class="title-bar">
+        <h1>Downloads</h1>
     </div>
-    <div class="icon-text">
-        <img id="download-icon" onclick="dlPending()" src="{% static 'img/icon-download.svg' %}" alt="download-icon">
-        <p>Start download</p>
-    </div>
-    <div class="icon-text">
-        <img id="add-icon" onclick="showForm()" src="{% static 'img/icon-add.svg' %}" alt="add-icon">
-        <p>Add to download queue</p>
-        <div class="show-form">
-            <form id='hidden-form' action="/downloads/" method="post">
-                {% csrf_token %}
-                <!-- <textarea rows="4" placeholder="Enter Video Urls or IDs here..." id="vid-url" name="vid-url"></textarea> -->
-                {{ add_form }}
-                <button type="submit">Add to download queue</button>
-            </form>
+    <div id="notifications" data="download"></div>
+    <div id="downloadControl"></div>
+    <div class="info-box info-box-3">
+        <div class="icon-text">
+            <img id="rescan-icon" onclick="rescanPending()" src="{% static 'img/icon-rescan.svg' %}" alt="rescan-icon">
+            <p>Rescan subscriptions</p>
         </div>
-    </div>
-</div>
-<div class="view-controls">
-    <div class="toggle">
-        <span>Show only ignored videos:</span>
-        <div class="toggleBox">
-            <input 
-                id="show_ignored_only" onclick="toggleCheckbox(this)" type="checkbox"
-                {% if show_ignored_only %}
-                    checked
-                {% endif %}
-                >
-            <label for="" class="onbtn">On</label>
-            <label for="" class="ofbtn">Off</label>
+        <div class="icon-text">
+            <img id="download-icon" onclick="dlPending()" src="{% static 'img/icon-download.svg' %}" alt="download-icon">
+            <p>Start download</p>
         </div>
-    </div>
-    <div class="view-icons">
-        <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="downloads" data-value="grid" alt="grid view">
-        <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="downloads" data-value="list" alt="list view">
-    </div>
-</div>
-{% if show_ignored_only %}
-    <h2>Ignored from download</h2>
-{% else %}
-    <h2>Download queue</h2>
-{% endif %}
-<h3>Total videos: {{ max_hits }}</h3>
-<div class="dl-list {{ view_style }}">
-    {% if all_video_hits %}
-        {% for video in all_video_hits %}
-            <div class="dl-item {{ view_style }}" id="dl-{{ video.youtube_id }}">
-                <div class="dl-thumb {{ view_style }}">
-                    <img src="/cache/{{ video.vid_thumb_url }}" alt="video_thumb">
-                    {% if show_ignored_only %}
-                        <span>ignored</span>
-                    {% else %}
-                        <span>queued</span>
-                    {% endif %}
-                </div>
-                <div class="dl-desc {{ view_style }}">
-                    <h3>{{ video.title }}</h3>
-                    {% if video.channel_indexed %}
-                        <a href="{% url 'channel_id' video.channel_id %}">{{ video.channel_name }}</a>
-                    {% else %}
-                        <span>{{ video.channel_name }}</span>
-                    {% endif %}
-                    <p>Published: {{ video.published }} | Duration: {{ video.duration }} | {{ video.youtube_id }}</p>
-                    {% if show_ignored_only %}
-                        <button data-id="{{ video.youtube_id }}" onclick="forgetIgnore(this)">Forget</button>
-                        <button data-id="{{ video.youtube_id }}" onclick="addSingle(this)">Add to queue</button>
-                    {% else %}
-                        <button data-id="{{ video.youtube_id }}" onclick="toIgnore(this)">Ignore</button>
-                        <button id="{{ video.youtube_id }}" data-id="{{ video.youtube_id }}" onclick="downloadNow(this)">Download now</button>
-                    {% endif %}
-                </div>
+        <div class="icon-text">
+            <img id="animate-icon" onclick="showForm()" src="{% static 'img/icon-add.svg' %}" alt="add-icon">
+            <p>Add to download queue</p>
+            <div class="show-form">
+                <form id='hidden-form' action="/downloads/" method="post">
+                    {% csrf_token %}
+                    {{ add_form }}
+                    <button type="submit">Add to download queue</button>
+                </form>
             </div>
-        {% endfor %}
-    {% endif %}
+        </div>
+    </div>
+    <div class="view-controls">
+        <div class="toggle">
+            <span>Show only ignored videos:</span>
+            <div class="toggleBox">
+                <input id="show_ignored_only" onclick="toggleCheckbox(this)" type="checkbox" {% if show_ignored_only %}checked{% endif %}>
+                {% if not show_ignored_only %}
+                    <label for="" class="ofbtn">Off</label>
+                {% else %}
+                    <label for="" class="onbtn">On</label>
+                {% endif %}
+            </div>
+        </div>
+        <div class="view-icons">
+            <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="downloads" data-value="grid" alt="grid view">
+            <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="downloads" data-value="list" alt="list view">
+        </div>
+    </div>
+    <div class="title-split">
+        {% if show_ignored_only %}
+            <h2>Ignored from download</h2>
+            <button onclick="deleteQueue(this)" data-id="ignore" title="Delete all previously ignored videos from the queue">Delete all ignored</button>
+        {% else %}
+            <h2>Download queue</h2>
+            <button onclick="deleteQueue(this)" data-id="pending" title="Delete all pending videos from the queue">Delete all queued</button>
+        {% endif %}
+    </div>
+    <h3>Total videos: {{ max_hits }}{% if max_hits == 10000 %}+{% endif %}</h3>
+    <div class="dl-list {{ view_style }}">
+        {% if results %}
+            {% for video in results %}
+                <div class="dl-item {{ view_style }}" id="dl-{{ video.source.youtube_id }}">
+                    <div class="dl-thumb {{ view_style }}">
+                        <img src="/cache/{{ video.source.vid_thumb_url }}" alt="video_thumb">
+                        {% if show_ignored_only %}
+                            <span>ignored</span>
+                        {% else %}
+                            <span>queued</span>
+                        {% endif %}
+                    </div>
+                    <div class="dl-desc {{ view_style }}">
+                        <h3>{{ video.source.title }}</h3>
+                        {% if video.source.channel_indexed %}
+                            <a href="{% url 'channel_id' video.source.channel_id %}">{{ video.source.channel_name }}</a>
+                        {% else %}
+                            <span>{{ video.source.channel_name }}</span>
+                        {% endif %}
+                        <p>Published: {{ video.source.published }} | Duration: {{ video.source.duration }} | {{ video.source.youtube_id }}</p>
+                        {% if show_ignored_only %}
+                            <button data-id="{{ video.source.youtube_id }}" onclick="forgetIgnore(this)">Forget</button>
+                            <button data-id="{{ video.source.youtube_id }}" onclick="addSingle(this)">Add to queue</button>
+                        {% else %}
+                            <button data-id="{{ video.source.youtube_id }}" onclick="toIgnore(this)">Ignore</button>
+                            <button id="{{ video.source.youtube_id }}" data-id="{{ video.source.youtube_id }}" onclick="downloadNow(this)">Download now</button>
+                        {% endif %}
+                    </div>
+                </div>
+            {% endfor %}
+        {% endif %}
+    </div>
 </div>
 <script type="text/javascript" src="{% static 'progress.js' %}"></script>
 {% endblock content %}

tubearchivist/home/templates/home/home.html

@@ -1,95 +1,125 @@
 {% extends "home/base.html" %}
 {% block content %}
 {% load static %}
-<div class="title-bar">
-    <h1>Recent Videos</h1>
-</div>
-<div class="info-box info-box-2">
-    <div class="sort">
-        <p>Sort by <span class="settings-current">{{ sort_by }}</span>
-            <select name="sort" id="sort" onchange="sortChange(this.value)">
-                <option value="" disabled selected> -- change sort by -- </option>
-                <option value="published">date published</option>
-                <option value="downloaded">date downloaded</option>
-                <option value="views">views</option>
-                <option value="likes">likes</option>
-            </select>
-            <select name="sord-order" id="sort-order" onchange="sortChange(this.value)">
-                {% if sort_order == "asc" %}
-                    <option value="asc" selected>asc</option>
-                {% else %}
-                    <option value="asc">asc</option>
-                {% endif %}
-                {% if sort_order == "desc" %}
-                    <option value="desc" selected>desc</option>
-                {% else %}
-                    <option value="desc">desc</option>
-                {% endif %}
-            </select>
-        </p>
-    </div>
-    <div class="search-form icon-text">
-        <div class="search-icon">
-            <img src="{% static 'img/icon-search.svg' %}" alt="search-icon" onclick="showSearch()">
+<div class="boxed-content">
+    {% if continue_vids %}
+        <div class="title-bar">
+            <h1>Continue Watching</h1>
         </div>
-        <form action="/" method="POST" id="search-box">
-            {% csrf_token %}
-            {{ search_form }}
-        </form>
-    </div>
-</div>
-<div class="view-controls">
-    <div class="toggle">
-        <span>Hide watched videos:</span>
-        <div class="toggleBox">
-            <input 
-                id="hide_watched" onclick="toggleCheckbox(this)" type="checkbox"
-                {% if hide_watched %}
-                    checked
-                {% endif %}
-                >
-            <label for="" class="onbtn">On</label>
-            <label for="" class="ofbtn">Off</label>
-        </div>
-    </div>
-    <div class="view-icons">
-        <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="grid" alt="grid view">
-        <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="list" alt="list view">
-    </div>
-</div>
-<div id="player" class="video-player"></div>
-<div class="video-list {{ view_style }}">
-    {% if videos %}
-        {% for video in videos %}
-            <div class="video-item {{ view_style }}">
-                <a href="#player" data-src="/media/{{ video.source.media_url }}" data-thumb="/cache/{{ video.source.vid_thumb_url }}" data-title="{{ video.source.title }}" data-channel="{{ video.source.channel.channel_name }}" data-id="{{ video.source.youtube_id }}" onclick="createPlayer(this)">
-                    <div class="video-thumb-wrap {{ view_style }}">
-                        <div class="video-thumb">
-                            <img src="/cache/{{ video.source.vid_thumb_url }}" alt="video-thumb">
+        <div class="video-list {{ view_style }}">
+            {% for video in continue_vids %}
+                <div class="video-item {{ view_style }}">
+                    <a href="#player" data-id="{{ video.source.youtube_id }}" onclick="createPlayer(this)">
+                        <div class="video-thumb-wrap {{ view_style }}">
+                            <div class="video-thumb">
+                                <img src="/cache/{{ video.source.vid_thumb_url }}" alt="video-thumb">
+                                {% if video.source.player.progress %}
+                                    <div class="video-progress-bar" id="progress-{{ video.source.youtube_id }}" style="width: {{video.source.player.progress}}%;"></div>
+                                {% else %}
+                                    <div class="video-progress-bar" id="progress-{{ video.source.youtube_id }}" style="width: 0%;"></div>
+                                {% endif %}
+                            </div>
+                            <div class="video-play">
+                                <img src="{% static 'img/icon-play.svg' %}" alt="play-icon">
+                            </div>
                         </div>
-                        <div class="video-play">
-                            <img src="{% static 'img/icon-play.svg' %}" alt="play-icon">
+                    </a>
+                    <div class="video-desc {{ view_style }}">
+                        <div class="video-desc-player" id="video-info-{{ video.source.youtube_id }}">
+                            {% if video.source.player.watched %}
+                                <img src="{% static 'img/icon-seen.svg' %}" alt="seen-icon" data-id="{{ video.source.youtube_id }}" data-status="watched" onclick="updateVideoWatchStatus(this)" class="watch-button" title="Mark as unwatched">
+                            {% else %}
+                                <img src="{% static 'img/icon-unseen.svg' %}" alt="unseen-icon" data-id="{{ video.source.youtube_id }}" data-status="unwatched" onclick="updateVideoWatchStatus(this)" class="watch-button" title="Mark as watched">
+                            {% endif %}
+                            <span>{{ video.source.published }} | {{ video.source.player.duration_str }}</span>
+                        </div>
+                        <div>
+                            <a href="{% url 'channel_id' video.source.channel.channel_id %}"><h3>{{ video.source.channel.channel_name }}</h3></a>
+                            <a class="video-more" href="{% url 'video' video.source.youtube_id %}"><h2>{{ video.source.title }}</h2></a>
                         </div>
-                    </div>
-                </a>
-                <div class="video-desc {{ view_style }}">
-                    <div class="video-desc-player" id="video-info-{{ video.source.youtube_id }}">
-                        {% if video.source.player.watched %}
-                            <img src="{% static 'img/icon-seen.svg' %}" alt="seen-icon" id="{{ video.source.youtube_id }}" onclick="isUnwatched(this.id)" class="seen-icon" title="Mark as unwatched">
-                        {% else %}
-                            <img src="{% static 'img/icon-unseen.svg' %}" alt="unseen-icon" id="{{ video.source.youtube_id }}" onclick="isWatched(this.id)" class="unseen-icon" title="Mark as watched.">
-                        {% endif %}
-                        <span>{{ video.source.published }} | {{ video.source.player.duration_str }}</span>
-                    </div>
-                    <div>
-                        <a href="{% url 'channel_id' video.source.channel.channel_id %}"><h3>{{ video.source.channel.channel_name }}</h3></a>
-                        <a class="video-more" href="{% url 'video' video.source.youtube_id %}"><h2>{{ video.source.title }}</h2></a>
                     </div>
                 </div>
-            </div>
-        {% endfor %}
-    {% else %}
-        <h2>No videos found...</h2>
+            {% endfor %}
+        </div>
     {% endif %}
+    <div class="title-bar">
+        <h1>Recent Videos</h1>
+    </div>
+    <div class="view-controls">
+        <div class="toggle">
+            <span>Hide watched:</span>
+            <div class="toggleBox">
+                <input id="hide_watched" onclick="toggleCheckbox(this)" type="checkbox" {% if hide_watched %}checked{% endif %}>
+                {% if not hide_watched %}
+                    <label for="" class="ofbtn">Off</label>
+                {% else %}
+                    <label for="" class="onbtn">On</label>
+                {% endif %}
+            </div>
+        </div>
+        <div class="sort">
+            <div id="hidden-form">
+                <span>Sort by:</span>
+                <select name="sort" id="sort" onchange="sortChange(this.value)">
+                    <option value="published" {% if sort_by == "published" %}selected{% endif %}>date published</option>
+                    <option value="downloaded" {% if sort_by == "downloaded" %}selected{% endif %}>date downloaded</option>
+                    <option value="views" {% if sort_by == "views" %}selected{% endif %}>views</option>
+                    <option value="likes" {% if sort_by == "likes" %}selected{% endif %}>likes</option>
+                </select>
+                <select name="sord-order" id="sort-order" onchange="sortChange(this.value)">
+                    <option value="asc" {% if sort_order == "asc" %}selected{% endif %}>asc</option>
+                    <option value="desc" {% if sort_order == "desc" %}selected{% endif %}>desc</option>
+                </select>
+            </div>
+        </div>
+        <div class="view-icons">
+            <img src="{% static 'img/icon-sort.svg' %}" alt="sort-icon" onclick="showForm()" id="animate-icon">
+            <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="grid" alt="grid view">
+            <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="list" alt="list view">
+        </div>
+    </div>
+</div>
+<div id="player" class="player-wrapper"></div>
+<div class="boxed-content">
+    <div class="video-list {{ view_style }}">
+        {% if results %}
+            {% for video in results %}
+                <div class="video-item {{ view_style }}">
+                    <a href="#player" data-id="{{ video.source.youtube_id }}" onclick="createPlayer(this)">
+                        <div class="video-thumb-wrap {{ view_style }}">
+                            <div class="video-thumb">
+                                <img src="/cache/{{ video.source.vid_thumb_url }}" alt="video-thumb">
+                                {% if video.source.player.progress %}
+                                    <div class="video-progress-bar" id="progress-{{ video.source.youtube_id }}" style="width: {{video.source.player.progress}}%;"></div>
+                                {% else %}
+                                    <div class="video-progress-bar" id="progress-{{ video.source.youtube_id }}" style="width: 0%;"></div>
+                                {% endif %}
+                            </div>
+                            <div class="video-play">
+                                <img src="{% static 'img/icon-play.svg' %}" alt="play-icon">
+                            </div>
+                        </div>
+                    </a>
+                    <div class="video-desc {{ view_style }}">
+                        <div class="video-desc-player" id="video-info-{{ video.source.youtube_id }}">
+                            {% if video.source.player.watched %}
+                                <img src="{% static 'img/icon-seen.svg' %}" alt="seen-icon" data-id="{{ video.source.youtube_id }}" data-status="watched" onclick="updateVideoWatchStatus(this)" class="watch-button" title="Mark as unwatched">
+                            {% else %}
+                                <img src="{% static 'img/icon-unseen.svg' %}" alt="unseen-icon" data-id="{{ video.source.youtube_id }}" data-status="unwatched" onclick="updateVideoWatchStatus(this)" class="watch-button" title="Mark as watched">
+                            {% endif %}
+                            <span>{{ video.source.published }} | {{ video.source.player.duration_str }}</span>
+                        </div>
+                        <div>
+                            <a href="{% url 'channel_id' video.source.channel.channel_id %}"><h3>{{ video.source.channel.channel_name }}</h3></a>
+                            <a class="video-more" href="{% url 'video' video.source.youtube_id %}"><h2>{{ video.source.title }}</h2></a>
+                        </div>
+                    </div>
+                </div>
+            {% endfor %}
+        {% else %}
+            <h2>No videos found...</h2>
+            <p>If you've already added a channel or playlist, try going to the <a href="{% url 'downloads' %}">downloads page</a> to start the scan and download tasks.</p>
+        {% endif %}
+    </div>
 </div>
 {% endblock content %}

tubearchivist/home/templates/home/login.html

@@ -7,7 +7,17 @@
     <meta name="viewport" content="width=device-width, initial-scale=1.0">
     <title>TA | Welcome</title>
     <link rel="stylesheet" type="text/css" href="{% static 'css/style.css' %}">
-    <link rel="shortcut icon" type="image/png" href="{% static 'favicon.ico' %}"/>
+    <link rel="apple-touch-icon" sizes="180x180" href="{% static 'favicon/apple-touch-icon.png' %}">
+    <link rel="icon" type="image/png" sizes="32x32" href="{% static 'favicon/favicon-32x32.png' %}">
+    <link rel="icon" type="image/png" sizes="16x16" href="{% static 'favicon/favicon-16x16.png' %}">
+    <link rel="manifest" href="{% static 'favicon/site.webmanifest' %}">
+    <link rel="mask-icon" href="{% static 'favicon/safari-pinned-tab.svg' %}" color="#01202e">
+    <link rel="shortcut icon" href="{% static 'favicon/favicon.ico' %}">
+    <meta name="apple-mobile-web-app-title" content="TubeArchivist">
+    <meta name="application-name" content="TubeArchivist">
+    <meta name="msapplication-TileColor" content="#01202e">
+    <meta name="msapplication-config" content="{% static 'favicon/browserconfig.xml' %}">
+    <meta name="theme-color" content="#01202e">
     {% if colors == "dark" %}
         <link rel="stylesheet" href="{% static 'css/dark.css' %}">
     {% else %}
@@ -29,13 +39,13 @@
         {% endif %}
         <form action="/login/" method="POST" name="login">
             {% csrf_token %}
-            {% for field in form %}
-                {{ field }}<br>
-            {% endfor %}
+            {{ form.username }}<br>
+            {{ form.password }}<br>
+            <p>Remember me: {{ form.remember_me }}</p>
             <input type="hidden" name="next" value="{{ request.GET.next }}" />
             <button type="submit">Login</button>
         </form>
-        <p class="login-links"><span><a href="https://github.com/bbilly1/tubearchivist" target="_blank">Github</a></span> <span><a href="https://github.com/bbilly1/tubearchivist#donate" target="_blank">Donate</a></span></p>
+        <p class="login-links"><span><a href="https://github.com/tubearchivist/tubearchivist" target="_blank">Github</a></span> <span><a href="https://github.com/tubearchivist/tubearchivist#donate" target="_blank">Donate</a></span></p>
     </div>
     <div class="footer-colors">
         <div class="col-1"></div>

tubearchivist/home/templates/home/playlist.html

@@ -0,0 +1,65 @@
+{% extends "home/base.html" %}
+{% load static %}
+{% block content %}
+<div class="boxed-content">
+    <div class="title-split">
+        <div class="title-bar">
+            <h1>Playlists</h1>
+        </div>
+        <div class="title-split-form">
+            <img id="animate-icon" onclick="showForm()" src="{% static 'img/icon-add.svg' %}" alt="add-icon" title="Subscribe to Playlists">
+            <div class="show-form">
+                <form id="hidden-form" action="/playlist/" method="post">
+                    {% csrf_token %}
+                    {{ subscribe_form }}
+                    <button type="submit">Subscribe</button>
+                </form>
+            </div>
+        </div>
+    </div>
+    <div id="notifications" data="playlist"></div>
+    <div class="view-controls">
+        <div class="toggle">
+            <span>Show subscribed only:</span>
+            <div class="toggleBox">
+                <input id="show_subed_only" onclick="toggleCheckbox(this)" type="checkbox" {% if show_subed_only %}checked{% endif %}>
+                {% if not show_subed_only %}
+                    <label for="" class="ofbtn">Off</label>
+                {% else %}
+                    <label for="" class="onbtn">On</label>
+                {% endif %}
+            </div>
+        </div>
+        <div class="view-icons">
+            <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="playlist" data-value="grid" alt="grid view">
+            <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="playlist" data-value="list" alt="list view">
+        </div>
+    </div>
+    <div class="playlist-list {{ view_style }}">
+        {% if results %}
+            {% for playlist in results %}
+            <div class="playlist-item {{ view_style }}">
+                <div class="playlist-thumbnail">
+                    <a href="{% url 'playlist_id' playlist.source.playlist_id %}">
+                        <img src="/cache/playlists/{{ playlist.source.playlist_id }}.jpg" alt="{{ playlist.source.playlist_id }}-thumbnail">
+                    </a>
+                </div>
+                <div class="playlist-desc {{ view_style }}">
+                    <a href="{% url 'channel_id' playlist.source.playlist_channel_id %}"><h3>{{ playlist.source.playlist_channel }}</h3></a>
+                    <a href="{% url 'playlist_id' playlist.source.playlist_id %}"><h2>{{ playlist.source.playlist_name }}</h2></a>
+                    <p>Last refreshed: {{ playlist.source.playlist_last_refresh }}</p>
+                    {% if playlist.source.playlist_subscribed %}
+                        <button class="unsubscribe" type="button" id="{{ playlist.source.playlist_id }}" onclick="unsubscribe(this.id)" title="Unsubscribe from {{ playlist.source.playlist_name }}">Unsubscribe</button>
+                    {% else %}
+                        <button type="button" id="{{ playlist.source.playlist_id }}" onclick="subscribe(this.id)" title="Subscribe to {{ playlist.source.playlist_name }}">Subscribe</button>
+                    {% endif %}
+                </div>
+            </div>
+            {% endfor %}
+        {% else %}
+            <h2>No playlists found...</h2>
+        {% endif %}
+    </div>
+</div>
+<script type="text/javascript" src="{% static 'progress.js' %}"></script>
+{% endblock content %}

tubearchivist/home/templates/home/playlist_id.html

@@ -0,0 +1,126 @@
+{% extends "home/base.html" %}
+{% load static %}
+{% load humanize %}
+{% block content %}
+<div class="boxed-content">
+    <div class="title-bar">
+        <h1>{{ playlist_info.playlist_name }}</h1>
+    </div>
+    <div class="info-box info-box-3">
+        <div class="info-box-item">
+            <div class="round-img">
+                <a href="{% url 'channel_id' channel_info.channel_id %}">
+                    <img src="/cache/channels/{{ channel_info.channel_id }}_thumb.jpg" alt="channel-thumb">
+                </a>
+            </div>
+            <div>
+                <h3><a href="{% url 'channel_id' channel_info.channel_id %}">{{ channel_info.channel_name }}</a></h3>
+                {% if channel_info.channel_subs >= 1000000 %}
+                    <span>Subscribers: {{ channel_info.channel_subs|intword }}</span>
+                {% else %}
+                    <span>Subscribers: {{ channel_info.channel_subs|intcomma }}</span>
+                {% endif %}
+            </div>
+        </div>
+        <div class="info-box-item">
+            <div>
+                <p>Last refreshed: {{ playlist_info.playlist_last_refresh }}</p>
+                <p>Playlist:
+                    {% if playlist_info.playlist_subscribed %}
+                        <button class="unsubscribe" type="button" id="{{ playlist_info.playlist_id }}" onclick="unsubscribe(this.id)" title="Unsubscribe from {{ playlist_info.playlist_name }}">Unsubscribe</button>
+                    {% else %}
+                        <button type="button" id="{{ playlist_info.playlist_id }}" onclick="subscribe(this.id)" title="Subscribe to {{ playlist_info.playlist_name }}">Subscribe</button>
+                    {% endif %}
+                </p>
+                {% if playlist_info.playlist_active %}
+                    <p>Youtube: <a href="https://www.youtube.com/playlist?list={{ playlist_info.playlist_id }}" target="_blank">Active</a></p>
+                {% else %}
+                    <p>Youtube: Deactivated</p>
+                {% endif %}
+                <button onclick="deleteConfirm()" id="delete-item">Delete Playlist</button>
+                <div class="delete-confirm" id="delete-button">
+                    <span>Delete {{ playlist_info.playlist_name }}?</span>
+                    <button onclick="deletePlaylist(this)" data-action="metadata" data-id="{{ playlist_info.playlist_id }}">Delete metadata</button>
+                    <button onclick="deletePlaylist(this)" data-action="all" class="danger-button" data-id="{{ playlist_info.playlist_id }}">Delete all</button><br>
+                    <button onclick="cancelDelete()">Cancel</button>
+                </div>
+            </div>
+        </div>
+        <div class="info-box-item">
+            <div>
+                {% if max_hits %}
+                    <p>Total Videos archived: {{ max_hits }}/{{ playlist_info.playlist_entries|length }}</p>
+                    <p>Watched: <button title="Mark all videos from {{ playlist_info.playlist_name }} as watched" type="button" id="watched-button" data-id="{{ playlist_info.playlist_id }}" onclick="isWatchedButton(this)">Mark as watched</button></p>
+                {% endif %}
+            </div>
+        </div>
+    </div>
+    {% if playlist_info.playlist_description %}
+        <div class="info-box-item description-box">
+            <p>Description: <button onclick="textReveal()" id="text-reveal-button">Show</button></p>
+            <div id="text-reveal" class="description-text">
+                {{ playlist_info.playlist_description|linebreaks }}
+            </div>
+        </div>
+    {% endif %}
+    <div class="view-controls">
+        <div class="toggle">
+            <span>Hide watched videos:</span>
+            <div class="toggleBox">
+                <input id="hide_watched" onclick="toggleCheckbox(this)" type="checkbox" {% if hide_watched %}checked{% endif %}>
+                {% if not hide_watched %}
+                    <label for="" class="ofbtn">Off</label>
+                {% else %}
+                    <label for="" class="onbtn">On</label>
+                {% endif %}
+            </div>
+        </div>
+        <div class="view-icons">
+            <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="grid" alt="grid view">
+            <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="list" alt="list view">
+        </div>
+    </div>
+</div>
+<div id="player" class="player-wrapper"></div>
+<div class="boxed-content">
+    <div class="video-list {{ view_style }}">
+        {% if results %}
+            {% for video in results %}
+                <div class="video-item {{ view_style }}">
+                    <a href="#player" data-id="{{ video.source.youtube_id }}" onclick="createPlayer(this)">
+                        <div class="video-thumb-wrap {{ view_style }}">
+                            <div class="video-thumb">
+                                <img src="/cache/{{ video.source.vid_thumb_url }}" alt="video-thumb">
+                                {% if video.source.player.progress %}
+                                    <div class="video-progress-bar" id="progress-{{ video.source.youtube_id }}" style="width: {{video.source.player.progress}}%;"></div>
+                                {% else %}
+                                    <div class="video-progress-bar" id="progress-{{ video.source.youtube_id }}" style="width: 0%;"></div>
+                                {% endif %}
+                            </div>
+                            <div class="video-play">
+                                <img src="{% static 'img/icon-play.svg' %}" alt="play-icon">
+                            </div>
+                        </div>
+                    </a>
+                    <div class="video-desc {{ view_style }}">
+                        <div class="video-desc-player" id="video-info-{{ video.source.youtube_id }}">
+                            {% if video.source.player.watched %}
+                                <img src="{% static 'img/icon-seen.svg' %}" alt="seen-icon" data-id="{{ video.source.youtube_id }}" data-status="watched" onclick="updateVideoWatchStatus(this)" class="watch-button" title="Mark as unwatched">
+                            {% else %}
+                                <img src="{% static 'img/icon-unseen.svg' %}" alt="unseen-icon" data-id="{{ video.source.youtube_id }}" data-status="unwatched" onclick="updateVideoWatchStatus(this)" class="watch-button" title="Mark as watched">
+                            {% endif %}
+                            <span>{{ video.source.published }} | {{ video.source.player.duration_str }}</span>
+                        </div>
+                        <div>
+                            <a class="video-more" href="{% url 'video' video.source.youtube_id %}"><h2>{{ video.source.title }}</h2></a>
+                        </div>
+                    </div>
+                </div>
+            {% endfor %}
+        {% else %}
+            <h2>No videos found...</h2>
+            <p>Try going to the <a href="{% url 'downloads' %}">downloads page</a> to start the scan and download tasks.</p>
+        {% endif %}
+    </div>
+</div>
+{% endblock content %}

tubearchivist/home/templates/home/search.html

@@ -0,0 +1,30 @@
+{% extends "home/base.html" %}
+{% block content %}
+<div id="player" class="player-wrapper"></div>
+<div class="boxed-content">
+    <div class="title-bar">
+        <h1>Search your Archive</h1>
+    </div>
+    <div class="multi-search-box">
+        {{ search_form }}
+    </div>
+    <div class="multi-search-result">
+        <h2>Video Results</h2>
+        <div id="video-results" class="video-list {{ all_styles.home }}">
+            <p>No videos found.</p>
+        </div>
+    </div>
+    <div class="multi-search-result">
+        <h2>Channel Results</h2>
+        <div id="channel-results" class="channel-list {{ all_styles.channel }}">
+            <p>No channels found.</p>
+        </div>
+    </div>
+    <div class="multi-search-result">
+        <h2>Playlist Results</h2>
+        <div id="playlist-results" class="playlist-list {{ all_styles.playlist }}">
+            <p>No playlists found.</p>
+        </div>
+    </div>
+</div>
+{% endblock content %}

tubearchivist/home/templates/home/settings.html

@@ -1,154 +1,336 @@
 {% extends "home/base.html" %}
+{% load static %}
 {% block content %}
-<div class="title-bar">
-    <h1>User Configurations</h1>
-</div>
-<form action="/settings/" method="POST" name="user-update">
-    {% csrf_token %}
-    <div class="settings-group">
-        <h2>Color scheme</h2>
-        <div class="settings-item">
-            <p>Current color scheme: <span class="settings-current">{{ config.application.colors }}</span></p>
-            <i>Select your preferred color scheme between dark and light mode.</i><br>
-            {{ user_form.colors }}
-        </div>
-    </div>
-    <div class="settings-group">
-        <h2>Archive View</h2>
-        <div class="settings-item">
-            <p>Current page size: <span class="settings-current">{{ config.archive.page_size }}</span></p>
-            <i>Result of videos showing in archive page</i><br>
-            {{ user_form.page_size }}
-        </div>
-    </div>
-    <button type="submit" name="user-settings">Update User Configurations</button>
-</form>
-<div class="title-bar">
-    <h1>Application Configurations</h1>
-</div>
-<form action="/settings/" method="POST" name="application-update">
-    {% csrf_token %}
-    <div class="settings-group">
-        <h2 id="subscriptions">Subscriptions</h2>
-        <div class="settings-item">
-            <p>Current channel page size: <span class="settings-current">{{ config.subscriptions.channel_size }}</span></p>
-            <i>Recent videos to check on check pending, max recommended 50.</i><br>
-            {{ app_form.subscriptions_channel_size }}
-        </div>
-        <div class="settings-item">
-            <p>Auto scan subscribed channels:</p>
-            <i>Coming soon</i>
-        </div>
-        <div class="settings-item">
-            <p>Auto download pending:</p>
-            <i>Coming soon</i>
-        </div>
-    </div>
-    <div class="settings-group">
-        <h2 id="downloads">Downloads</h2>
-        <div class="settings-item">
-            <p>Current download limit: <span class="settings-current">{{ config.downloads.limit_count }}</span></p>
-            <i>Limit the number of videos getting downloaded on every run. 0 (zero) to deactivate.</i><br>
-            {{ app_form.downloads_limit_count }}
-        </div>
-        <div class="settings-item">
-            <p>Current download speed limit: <span class="settings-current">{{ config.downloads.limit_speed }}</span></p>
-            <i>Limit download speed. 0 (zero) to deactivate.</i><br>
-            {{ app_form.downloads_limit_speed }}
-        </div>
-        <div class="settings-item">
-            <p>Current throttled rate limit: <span class="settings-current">{{ config.downloads.throttledratelimit }}</span></p>
-            <i>Assume the download is being throttled below this speed and restart. 0 (zero) to deactivate, e.g. 100KB/sec</i><br>
-            {{ app_form.downloads_throttledratelimit }}
-        </div>
-        <div class="settings-item">
-            <p>Current scraping sleep interval: <span class="settings-current">{{ config.downloads.sleep_interval }}</p>
-            <i>Seconds to sleep between calls to YouTube. Might be necessary to avoid throttling. Recommended 3.</i><br>
-            {{ app_form.downloads_sleep_interval }}
-        </div>
-        <div class="settings-item">
-            <p>External downloader:</p>
-            <i>Coming soon</i>
-        </div>
-    </div>
-    <div class="settings-group">
-        <h2 id="format">Download Format</h2>
-        <div class="settings-item">
-            <p>Limit video and audio quality format for yt-dlp.<br>
-                Currently: <span class="settings-current">{{ config.downloads.format }}</span>
-            </p>
-            <p>Example configurations:</p>
-            <ul>
-                <li><span class="settings-current">bestvideo[height<=720]+bestaudio/best[height<=720]</span>: best audio and max video height of 720p.</li>
-                <li><span class="settings-current">bestvideo[height<=1080]+bestaudio/best[height<=1080]</span>: best audio and max video height of 1080p.</li>
-                <li><span class="settings-current">bestvideo[height<=1080][VCODEC=avc1]+bestaudio[ACODEC=mp4a]/mp4</span>: Max 1080p video height with iOS compatible video and audio codecs.</li>
-                <li><span class="settings-current">0</span>: deactivate and download the best quality possible as decided by yt-dlp.</li>
-            </ul>
-            <i>Make sure your custom format gets merged into a single file. Check out the <a href="https://github.com/yt-dlp/yt-dlp#format-selection" target="_blank">documentation</a> for valid configurations.</i><br>
-            {{ app_form.downloads_format }}
-            <br>
-        </div>
-        <div class="settings-item">
-            <p>Current metadata embed setting: <span class="settings-current">{{ config.downloads.add_metadata }}</span></p>
-            <i>Metadata is not embedded into the downloaded files by default.</i><br>
-            {{ app_form.downloads_add_metadata }}
-        </div>
-        <div class="settings-item">
-            <p>Current thumbnail embed setting: <span class="settings-current">{{ config.downloads.add_thumbnail }}</span></p>
-            <i>Embed thumbnail into the mediafile.</i><br>
-            {{ app_form.downloads_add_thumbnail }}
-        </div>
-    </div>
-    <button type="submit" name="application-settings">Update Application Configurations</button>
-</form>
-<div class="title-bar">
-    <h1>Actions</h1>
-</div>
-<div class="settings-group">
-    <h2>Manual media files import.</h2>
-    <p>Add files to the <span class="settings-current">cache/import</span> folder. Make sure to follow the instructions in the Github <a href="https://github.com/bbilly1/tubearchivist/wiki/Settings" target="_blank">Wiki</a>.</p>
-    <div id="manual-import">
-        <button onclick="manualImport()">Start import</button>
-    </div>
-</div>
-<div class="settings-group">
-    <h2>Embed thumbnails into media file.</h2>
-    <p>Set extracted youtube thumbnail as cover art of the media file.</p>
-    <div id="re-embed">
-        <button onclick="reEmbed()">Start process</button>
-    </div>
-</div>
-<div class="settings-group">
-    <h2>Backup database</h2>
-    <p>Export your database to a zip file stored at <span class="settings-current">cache/backup</span>.</p>
-    <div id="db-backup">
-        <button onclick="dbBackup()">Start backup</button>
-    </div>
-</div>
-<div class="settings-group">
-    <h2>Restore from backup</h2>
-    <p><span class="danger-zone">Danger Zone</span>: This will replace your existing index with the backup.</p>
-    <p>Add the backup zip file to the <span class="settings-current">cache/backup</span> folder.</p>
-    <div id="db-restore">
-        <button onclick="dbRestore()">Restore backup</button>
-    </div>
-</div>
-<div class="settings-group">
-    <h2>Rescan filesystem</h2>
-    <p><span class="danger-zone">Danger Zone</span>: This will delete the metadata of deleted videos from the filesystem.</p>
-    <p>Rescan your media folder looking for missing videos and clean up index. More infos on the Github <a href="https://github.com/bbilly1/tubearchivist/wiki/Settings" target="_blank">Wiki</a>.</p>
-    <div id="fs-rescan">
-        <button onclick="fsRescan()">Rescan filesystem</button>
-    </div>
-</div>
-{% if request.user.is_superuser %}
+<div class="boxed-content">
+    <div id="notifications" data="setting"></div>
     <div class="title-bar">
-        <h1>Users</h1>
+        <h1>User Configurations</h1>
+    </div>
+    <form action="/settings/" method="POST" name="user-update">
+        {% csrf_token %}
+        <div class="settings-group">
+            <h2>Color scheme</h2>
+            <div class="settings-item">
+                <p>Current color scheme: <span class="settings-current">{{ config.application.colors }}</span></p>
+                <i>Select your preferred color scheme between dark and light mode.</i><br>
+                {{ user_form.colors }}
+            </div>
+        </div>
+        <div class="settings-group">
+            <h2>Archive View</h2>
+            <div class="settings-item">
+                <p>Current page size: <span class="settings-current">{{ config.archive.page_size }}</span></p>
+                <i>Result of videos showing in archive page</i><br>
+                {{ user_form.page_size }}
+            </div>
+        </div>
+        <button type="submit" name="user-settings">Update User Configurations</button>
+    </form>
+    <div class="title-bar">
+        <h1>Application Configurations</h1>
+    </div>
+    <form action="/settings/" method="POST" name="application-update">
+        {% csrf_token %}
+        <div class="settings-group">
+            <h2 id="subscriptions">Subscriptions</h2>
+            <div class="settings-item">
+                <p>Current page size: <span class="settings-current">{{ config.subscriptions.channel_size }}</span></p>
+                <i>Recent videos for channels and playlist to check when running <b>Rescan subscriptions</b>, max recommended 50.</i><br>
+                {{ app_form.subscriptions_channel_size }}
+            </div>
+        </div>
+        <div class="settings-group">
+            <h2 id="downloads">Downloads</h2>
+            <div class="settings-item">
+                <p>Current download limit: <span class="settings-current">{{ config.downloads.limit_count }}</span></p>
+                <i>Limit the number of videos getting downloaded on every run. 0 (zero) to deactivate.</i><br>
+                {{ app_form.downloads_limit_count }}
+            </div>
+            <div class="settings-item">
+                <p>Current download speed limit in KB/s: <span class="settings-current">{{ config.downloads.limit_speed }}</span></p>
+                <i>Limit download speed. 0 (zero) to deactivate, e.g. 1000 (1MB/s). Speeds are in KB/s. Setting takes effect on new download jobs or application restart.</i><br>
+                {{ app_form.downloads_limit_speed }}
+            </div>
+            <div class="settings-item">
+                <p>Current throttled rate limit in KB/s: <span class="settings-current">{{ config.downloads.throttledratelimit }}</span></p>
+                <i>Download will restart if speeds drop below specified amount. 0 (zero) to deactivate, e.g. 100. Speeds are in KB/s.</i><br>
+                {{ app_form.downloads_throttledratelimit }}
+            </div>
+            <div class="settings-item">
+                <p>Current scraping sleep interval: <span class="settings-current">{{ config.downloads.sleep_interval }}</p>
+                <i>Seconds to sleep between calls to YouTube. Might be necessary to avoid throttling. Recommended 3.</i><br>
+                {{ app_form.downloads_sleep_interval }}
+            </div>
+            <div class="settings-item">
+                <p><span class="danger-zone">Danger Zone</span>: Current auto delete watched videos: <span class="settings-current">{{ config.downloads.autodelete_days }}</span></p>
+                <i>Auto delete watched videos after x days, 0 (zero) to deactivate:</i><br>
+                {{ app_form.downloads_autodelete_days }}
+            </div>
+        </div>
+        <div class="settings-group">
+            <h2 id="format">Download Format</h2>
+            <div class="settings-item">
+                <p>Limit video and audio quality format for yt-dlp.<br>
+                    Currently: <span class="settings-current">{{ config.downloads.format }}</span>
+                </p>
+                <p>Example configurations:</p>
+                <ul>
+                    <li><span class="settings-current">bestvideo[height<=720]+bestaudio/best[height<=720]</span>: best audio and max video height of 720p.</li>
+                    <li><span class="settings-current">bestvideo[height<=1080]+bestaudio/best[height<=1080]</span>: best audio and max video height of 1080p.</li>
+                    <li><span class="settings-current">bestvideo[height<=1080][VCODEC=avc1]+bestaudio[ACODEC=mp4a]/mp4</span>: Max 1080p video height with iOS compatible video and audio codecs.</li>
+                    <li><span class="settings-current">0</span>: deactivate and download the best quality possible as decided by yt-dlp.</li>
+                </ul>
+                <i>Make sure your custom format gets merged into a single file. Check out the <a href="https://github.com/yt-dlp/yt-dlp#format-selection" target="_blank">documentation</a> for valid configurations.</i><br>
+                {{ app_form.downloads_format }}
+                <br>
+            </div>
+            <div class="settings-item">
+                <p>Current metadata embed setting: <span class="settings-current">{{ config.downloads.add_metadata }}</span></p>
+                <i>Metadata is not embedded into the downloaded files by default.</i><br>
+                {{ app_form.downloads_add_metadata }}
+            </div>
+            <div class="settings-item">
+                <p>Current thumbnail embed setting: <span class="settings-current">{{ config.downloads.add_thumbnail }}</span></p>
+                <i>Embed thumbnail into the mediafile.</i><br>
+                {{ app_form.downloads_add_thumbnail }}
+            </div>
+        </div>
+        <div class="settings-group">
+            <h2 id="format">Subtitles</h2>
+            <div class="settings-item">
+                <p>Subtitles download setting: <span class="settings-current">{{ config.downloads.subtitle }}</span><br>
+                <i>Choose which subtitles to download, add comma separated two letter language ISO code,<br>
+                    e.g. <span class="settings-current">en, de</span></i><br>
+                {{ app_form.downloads_subtitle }}</p>
+            </div>
+            <div class="settings-item">
+                <p>Subtitle source settings: <span class="settings-current">{{ config.downloads.subtitle_source }}</span></p>
+                <i>Download only user generated, or also less accurate auto generated subtitles.</i><br>
+                {{ app_form.downloads_subtitle_source }}
+            </div>
+            <div class="settings-item">
+                <p>Index and make subtitles searchable: <span class="settings-current">{{ config.downloads.subtitle_index }}</span></p>
+                <i>Store subtitle lines in Elasticsearch. Not recommended for low-end hardware.</i><br>
+                {{ app_form.downloads_subtitle_index }}
+            </div>
+        </div>
+        <div class="settings-group">
+            <h2 id="format">Cookie</h2>
+            <div class="settings-item">
+                <p>Import YouTube cookie: <span class="settings-current">{{ config.downloads.cookie_import }}</span><br></p>
+                <i>Place your cookie file named <span class="settings-current">cookies.google.txt</span> in <span class="settings-current">cache/import</span> before enabling. Instructions in the <a href="https://github.com/tubearchivist/tubearchivist/wiki/Settings" target="_blank">Wiki.</a></i><br>
+                {{ app_form.downloads_cookie_import }}<br>
+                {% if config.downloads.cookie_import %}
+                    <div id="cookieMessage">
+                        <button onclick="handleCookieValidate()" type="button" id="cookieButton">Validate Cookie File</button>
+                    </div>
+                {% endif %}
+            </div>
+        </div>
+        <div class="settings-group">
+            <h2 id="integrations">Integrations</h2>
+            <div class="settings-item">
+                <p>API token: <button type="button" onclick="textReveal()" id="text-reveal-button">Show</button></p>
+                <div id="text-reveal" class="description-text">
+                    <p>{{ api_token }}</p>
+                    <button class="danger-button" type="button" onclick="resetToken()">Revoke</button>
+                </div>
+            </div>
+            <div class="settings-item">
+                <p>Integrate with <a href="https://returnyoutubedislike.com/" target="_blank">returnyoutubedislike.com</a> to get dislikes and average ratings back: <span class="settings-current">{{ config.downloads.integrate_ryd }}</span></p>
+                <i>Before activating that, make sure you have a scraping sleep interval of at least 3 secs set to avoid ratelimiting issues.</i><br>
+                {{ app_form.downloads_integrate_ryd }}
+            </div>
+            <div class="settings-item">
+                <p>Integrate with <a href="https://sponsor.ajay.app/" target="_blank">SponsorBlock</a> to get sponsored timestamps: <span class="settings-current">{{ config.downloads.integrate_sponsorblock }}</span></p>
+                <i>Before activating that, make sure you have a scraping sleep interval of at least 3 secs set to avoid ratelimiting issues.</i><br>
+                {{ app_form.downloads_integrate_sponsorblock }}
+            </div>
+            <div class="settings-item">
+                <p>Current Cast integration: <span class="settings-current">{{ config.application.enable_cast }}</span></p>
+                <i>Enabling Cast will load an additional JS library from Google. HTTPS and a supported browser are required for this integration.</i><br>
+                {{ app_form.application_enable_cast }}
+            </div>
+        </div>
+        <button type="submit" name="application-settings">Update Application Configurations</button>
+    </form>
+    <div class="title-bar">
+        <h1>Scheduler Setup</h1>
+        <div class="settings-group">
+            <p>Schedule settings expect a cron like format, where the first value is minute, second is hour and third is day of the week.</p>
+            <p>Examples:</p>
+            <ul>
+                <li><span class="settings-current">0 15 *</span>: Run task every day at 15:00 in the afternoon.</li>
+                <li><span class="settings-current">30 8 */2</span>: Run task every second day of the week (Sun, Tue, Thu, Sat) at 08:30 in the morning.</li>
+                <li><span class="settings-current">auto</span>: Sensible default.</li>
+                <li><span class="settings-current">0</span>: (zero), deactivate that task.</li>
+            </ul>
+            <p>Note:</p>
+            <ul>
+                <li>Changes in the scheduler settings require a container restart to take effect.</li>
+                <li>Avoid an unnecessary frequent schedule to not get blocked by YouTube. For that reason <i>*</i> or <i>wildcard</i> for minutes is not supported.</li>
+            </ul>
+        </div>
+    </div>
+    <form action="/settings/" method="POST" name="scheduler-update">
+        {% csrf_token %}
+        <div class="settings-group">
+            <h2>Rescan Subscriptions</h2>
+            <div class="settings-item">
+                <p>Current rescan schedule: <span class="settings-current">
+                    {% if config.scheduler.update_subscribed %}
+                        {% for key, value in config.scheduler.update_subscribed.items %}
+                            {{ value }}
+                        {% endfor %}
+                    {% else %}
+                        False
+                    {% endif %}
+                </span></p>
+                <p>Periodically rescan your subscriptions:</p>
+                {{ scheduler_form.update_subscribed }}
+            </div>
+        </div>
+        <div class="settings-group">
+            <h2>Start download</h2>
+            <div class="settings-item">
+                <p>Current Download schedule: <span class="settings-current">
+                    {% if config.scheduler.download_pending %}
+                        {% for key, value in config.scheduler.download_pending.items %}
+                            {{ value }}
+                        {% endfor %}
+                    {% else %}
+                        False
+                    {% endif %}
+                </span></p>
+                <p>Automatic video download schedule:</p>
+                {{ scheduler_form.download_pending }}
+            </div>
+        </div>
+        <div class="settings-group">
+            <h2>Refresh Metadata</h2>
+            <div class="settings-item">
+                <p>Current Metadata refresh schedule: <span class="settings-current">
+                    {% if config.scheduler.check_reindex %}
+                        {% for key, value in config.scheduler.check_reindex.items %}
+                            {{ value }}
+                        {% endfor %}
+                    {% else %}
+                        False
+                    {% endif %}
+                </span></p>
+                <p>Daily schedule to refresh metadata from YouTube:</p>
+                {{ scheduler_form.check_reindex }}
+            </div>
+            <div class="settings-item">
+                <p>Current refresh for metadata older than x days: <span class="settings-current">{{ config.scheduler.check_reindex_days }}</span></p>
+                <p>Refresh older than x days, recommended 90:</p>
+                {{ scheduler_form.check_reindex_days }}
+            </div>
+        </div>
+        <div class="settings-group">
+            <h2>Thumbnail check</h2>
+            <div class="settings-item">
+                <p>Current thumbnail check schedule: <span class="settings-current">
+                    {% if config.scheduler.thumbnail_check %}
+                        {% for key, value in config.scheduler.thumbnail_check.items %}
+                            {{ value }}
+                        {% endfor %}
+                    {% else %}
+                        False
+                    {% endif %}
+                </span></p>
+                <p>Periodically check and cleanup thumbnails:</p>
+                {{ scheduler_form.thumbnail_check }}
+            </div>
+        </div>
+        <div class="settings-group">
+            <h2>Index backup</h2>
+            <div class="settings-item">
+                <p>Current index backup schedule: <span class="settings-current">
+                    {% if config.scheduler.run_backup %}
+                        {% for key, value in config.scheduler.run_backup.items %}
+                            {{ value }}
+                        {% endfor %}
+                    {% else %}
+                        False
+                    {% endif %}
+                </span></p>
+                <p>Automatically backup metadata to a zip file:</p>
+                {{ scheduler_form.run_backup }}
+            </div>
+            <div class="settings-item">
+                <p>Current backup files to keep: <span class="settings-current">{{ config.scheduler.run_backup_rotate }}</span></p>
+                <p>Max auto backups to keep:</p>
+                {{ scheduler_form.run_backup_rotate }}
+            </div>
+        </div>
+        <button type="submit" name="scheduler-settings">Update Scheduler Settings</button>
+    </form>
+    <div class="title-bar">
+        <h1>Actions</h1>
     </div>
     <div class="settings-group">
-        <h2>User Management</h2>
-        <p>Access the admin interface for basic user management functionality like adding and deleting users, changing passwords and more.</p>
-        <a href="/admin/"><button>Admin Interface</button></a>
+        <h2>Manual media files import.</h2>
+        <p>Add files to the <span class="settings-current">cache/import</span> folder. Make sure to follow the instructions in the Github <a href="https://github.com/tubearchivist/tubearchivist/wiki/Settings" target="_blank">Wiki</a>.</p>
+        <div id="manual-import">
+            <button onclick="manualImport()">Start import</button>
+        </div>
     </div>
-{% endif %}
+    <div class="settings-group">
+        <h2>Embed thumbnails into media file.</h2>
+        <p>Set extracted youtube thumbnail as cover art of the media file.</p>
+        <div id="re-embed">
+            <button onclick="reEmbed()">Start process</button>
+        </div>
+    </div>
+    <div class="settings-group">
+        <h2>Backup database</h2>
+        <p>Export your database to a zip file stored at <span class="settings-current">cache/backup</span>.</p>
+        <div id="db-backup">
+            <button onclick="dbBackup()">Start backup</button>
+        </div>
+    </div>
+    <div class="settings-group">
+        <h2>Restore from backup</h2>
+        <p><span class="danger-zone">Danger Zone</span>: This will replace your existing index with the backup.</p>
+        <p>Restore from available backup files from <span class="settings-current">cache/backup</span>.</p>
+        {% if available_backups %}
+            <div class="backup-grid-row">
+                <span></span>
+                <span>Timestamp</span>
+                <span>Source</span>
+                <span>Filename</span>
+            </div>
+            {% for backup in available_backups %}
+                <div class="backup-grid-row" id="{{ backup.filename }}">
+                    <button onclick="dbRestore(this)" data-id="{{ backup.filename }}">Restore</button>
+                    <span>{{ backup.timestamp }}</span>
+                    <span>{{ backup.reason }}</span>
+                    <span>{{ backup.filename }}</span>
+                </div>
+            {% endfor %}
+        {% else %}
+            <p>No backups found.</p>
+        {% endif %}
+    </div>
+    <div class="settings-group">
+        <h2>Rescan filesystem</h2>
+        <p><span class="danger-zone">Danger Zone</span>: This will delete the metadata of deleted videos from the filesystem.</p>
+        <p>Rescan your media folder looking for missing videos and clean up index. More infos on the Github <a href="https://github.com/tubearchivist/tubearchivist/wiki/Settings" target="_blank">Wiki</a>.</p>
+        <div id="fs-rescan">
+            <button onclick="fsRescan()">Rescan filesystem</button>
+        </div>
+    </div>
+    {% if request.user.is_superuser %}
+        <div class="title-bar">
+            <h1>Users</h1>
+        </div>
+        <div class="settings-group">
+            <h2>User Management</h2>
+            <p>Access the admin interface for basic user management functionality like adding and deleting users, changing passwords and more.</p>
+            <a href="/admin/"><button>Admin Interface</button></a>
+        </div>
+    {% endif %}
+</div>
+<script type="text/javascript" src="{% static 'progress.js' %}"></script>
 {% endblock content %}

tubearchivist/home/templates/home/video.html

@@ -2,76 +2,130 @@
 {% block content %}
 {% load static %}
 {% load humanize %}
-<div class="title-bar">
-    <h1>Video: {{ video.title }}</h1>
+<div class="video-main"></div>
+<div class="notifications" id="notifications"></div>
+<div class="sponsorblock" id="sponsorblock">
+    {% if video.sponsorblock.is_enabled %}
+        {% if video.sponsorblock.segments|length == 0 %}
+            <h4>This video doesn't have any sponsor segments added. To add a segment go to <u><a href="https://www.youtube.com/watch?v={{ video.youtube_id }}">this video on YouTube</a></u> and add a segment using the <u><a href="https://sponsor.ajay.app/">SponsorBlock</a></u> extension.</h4>
+        {% elif video.sponsorblock.has_unlocked %}
+            <h4>This video has unlocked sponsor segments. Go to <u><a href="https://www.youtube.com/watch?v={{ video.youtube_id }}">this video on YouTube</a></u> and vote on the segments using the <u><a href="https://sponsor.ajay.app/">SponsorBlock</a></u> extension.</h4>
+        {% endif %}
+    {% endif %}
 </div>
-<div class="video-main">
-    <video 
-        src="/media/{{ video.media_url }}" 
-        poster="/cache/{{ video.vid_thumb_url }}" controls preload="false" 
-        type='video/mp4' width="100%" playsinline>
-    </video>
-</div>
-<div class="info-box info-box-3 padding-box">
-    <div class="info-box-item">
-        <div class="round-img">
-            <a href="{% url 'channel_id' video.channel.channel_id %}">
-                <img src="/cache/channels/{{ video.channel.channel_id }}_thumb.jpg" alt="channel-thumb">
-            </a>
-        </div>
-        <div>
-            <h3><a href="{% url 'channel_id' video.channel.channel_id %}">{{ video.channel.channel_name }}</a></h3>
-            {% if video.channel.channel_subs >= 1000000 %}
-                <p>Subscribers: {{ video.channel.channel_subs|intword }}</p>
-            {% else %}
-                <p>Subscribers: {{ video.channel.channel_subs|intcomma }}</p>
-            {% endif %}
-        </div>
+<div class="boxed-content">
+    <div class="title-bar">
+        {% if cast %}
+            <google-cast-launcher id="castbutton"></google-cast-launcher>
+        {% endif %}
+        <h1 id="video-title">{{ video.title }}</h1>
     </div>
-    <div class="info-box-item">
-        <div>
-            <p>Published: {{ video.published }}</p>
-            <p>Last refreshed: {{ video.vid_last_refresh }}</p>
-            <p class="video-info-watched">Watched:
-                {% if video.player.watched %}
-                    <img src="{% static 'img/icon-seen.svg' %}" alt="seen-icon" id="{{ video.youtube_id }}" onclick="isUnwatched(this.id)" class="seen-icon" title="Mark as unwatched">
+    <div class="info-box info-box-3">
+        <div class="info-box-item">
+            <div class="round-img">
+                <a href="{% url 'channel_id' video.channel.channel_id %}">
+                    <img src="/cache/channels/{{ video.channel.channel_id }}_thumb.jpg" alt="channel-thumb">
+                </a>
+            </div>
+            <div>
+                <h3><a href="{% url 'channel_id' video.channel.channel_id %}">{{ video.channel.channel_name }}</a></h3>
+                {% if video.channel.channel_subs >= 1000000 %}
+                    <p>Subscribers: {{ video.channel.channel_subs|intword }}</p>
                 {% else %}
-                    <img src="{% static 'img/icon-unseen.svg' %}" alt="unseen-icon" id="{{ video.youtube_id }}" onclick="isWatched(this.id)" class="unseen-icon" title="Mark as watched.">
+                    <p>Subscribers: {{ video.channel.channel_subs|intcomma }}</p>
+                {% endif %}
+            </div>
+        </div>
+        <div class="info-box-item">
+            <div>
+                <p>Published: {{ video.published }}</p>
+                <p>Last refreshed: {{ video.vid_last_refresh }}</p>
+                <p class="video-info-watched">Watched:
+                    {% if video.player.watched %}
+                        <img src="{% static 'img/icon-seen.svg' %}" alt="seen-icon" data-id="{{ video.youtube_id }}" data-status="watched" onclick="updateVideoWatchStatus(this)" class="watch-button" title="Mark as unwatched">
+                    {% else %}
+                        <img src="{% static 'img/icon-unseen.svg' %}" alt="unseen-icon" data-id="{{ video.youtube_id }}" data-status="unwatched" onclick="updateVideoWatchStatus(this)" class="watch-button" title="Mark as watched">
+                    {% endif %}
+                </p>
+                {% if video.active %}
+                    <p>Youtube: <a href="https://www.youtube.com/watch?v={{ video.youtube_id }}" target="_blank">Active</a></p>
+                {% else %}
+                    <p>Youtube: Deactivated</p>
+                {% endif %}
+                <a download="" href="/media/{{ video.media_url }}"><button id="download-item">Download File</button></a>
+                <button onclick="deleteConfirm()" id="delete-item">Delete Video</button>
+                <div class="delete-confirm" id="delete-button">
+                    <span>Are you sure? </span><button class="danger-button" onclick="deleteVideo(this)" data-id="{{ video.youtube_id }}" data-redirect = "{{ video.channel.channel_id }}">Delete</button> <button onclick="cancelDelete()">Cancel</button>
+                </div>
+            </div>
+        </div>
+        <div class="info-box-item">
+            <div>
+                <p class="thumb-icon"><img src="{% static 'img/icon-eye.svg' %}" alt="views">: {{ video.stats.view_count|intcomma }}</p>
+                <p class="thumb-icon like"><img src="{% static 'img/icon-thumb.svg' %}" alt="thumbs-up">: {{ video.stats.like_count|intcomma }}</p>
+                {% if video.stats.dislike_count %}
+                    <p class="thumb-icon"><img class="dislike" src="{% static 'img/icon-thumb.svg' %}" alt="thumbs-down">: {{ video.stats.dislike_count|intcomma }}</p>
+                {% endif %}
+                {% if video.stats.average_rating %}
+                    <p class="rating-stars">Rating: 
+                        {% for star in video.stats.average_rating %}
+                            <img src="/static/img/icon-star-{{ star }}.svg" alt="{{ star }}">
+                        {% endfor %}
+                    </p>
                 {% endif %}
-            </p>
-            {% if video.active %}
-                <p>Youtube: Active</p>
-            {% else %}
-                <p>Youtube: Deactivated</p>
-            {% endif %}
-            <p>Video ID: {{ video.youtube_id }}</p>
-            <button onclick="deleteConfirm()" id="delete-item">Delete Video</button>
-            <div class="delete-confirm" id="delete-button">
-                <span>Are you sure? </span><button class="danger-button" onclick="deleteVideo(this)" data-id="{{ video.youtube_id }}" data-redirect = "{{ video.channel.channel_id }}">Delete</button> <button onclick="cancelDelete()">Cancel</button>
             </div>
         </div>
     </div>
-    <div class="info-box-item">
-        <div>
-            <p>Views: {{ video.stats.view_count|intcomma }}</p>
-            <p class="thumb-icon like"><img src="{% static 'img/icon-thumb.svg' %}" alt="thumbs-up">: {{ video.stats.like_count|intcomma }}</p>
-            <p class="thumb-icon dislike"><img src="{% static 'img/icon-thumb.svg' %}" alt="thumbs-down">: {{ video.stats.dislike_count|intcomma }}</p>
-            {% if video.stats.average_rating %}
-                <p class="rating-stars">Rating: 
-                    {% for star in video.stats.average_rating %}
-                        <img src="/static/img/icon-star-{{ star }}.svg" alt="{{ star }}">
-                    {% endfor %}
-                </p>
-            {% endif %}
+    {% if video.description %}
+        <div class="info-box-item description-box">
+            <p>Description: <button onclick="textReveal()" id="text-reveal-button">Show</button></p>
+            <div id="text-reveal" class="description-text">
+                {{ video.description|linebreaks }}
+            </div>
         </div>
-    </div>
+    {% endif %}
+    {% if playlist_nav %}
+        {% for playlist_item in playlist_nav %}
+            <div class="playlist-wrap">
+                <a href="{% url 'playlist_id' playlist_item.playlist_meta.playlist_id %}">
+                    <h3>Playlist [{{ playlist_item.playlist_meta.current_idx|add:"1" }}]: {{ playlist_item.playlist_meta.playlist_name }}</h3>
+                </a>
+                <div class="playlist-nav">    
+                    <div class="playlist-nav-item">
+                        {% if playlist_item.playlist_previous %}
+                            <a href="{% url 'video' playlist_item.playlist_previous.youtube_id %}">
+                                <img src="/cache/{{ playlist_item.playlist_previous.vid_thumb }}" alt="previous thumbnail">
+                            </a>
+                            <div class="playlist-desc">
+                                <p>Previous:</p>
+                                <a href="{% url 'video' playlist_item.playlist_previous.youtube_id %}">
+                                    <h3>[{{ playlist_item.playlist_previous.idx|add:"1" }}] {{ playlist_item.playlist_previous.title }}</h3>
+                                </a>
+                            </div>
+                        {% endif %}
+                    </div>
+                    <div class="playlist-nav-item">
+                        {% if playlist_item.playlist_next %}
+                            <div class="playlist-desc">
+                                <p>Next:</p>
+                                <a href="{% url 'video' playlist_item.playlist_next.youtube_id %}">
+                                    <h3>[{{ playlist_item.playlist_next.idx|add:"1" }}] {{ playlist_item.playlist_next.title }}</h3>
+                                </a>
+                            </div>
+                            <a href="{% url 'video' playlist_item.playlist_next.youtube_id %}">
+                                <img src="/cache/{{ playlist_item.playlist_next.vid_thumb }}" alt="previous thumbnail">
+                            </a>
+                        {% endif %}
+                    </div>
+                </div>
+            </div>
+        {% endfor %}
+    {% endif %}
 </div>
-{% if video.description %}
-    <div class="info-box-item description-box">
-        <p>Description: <button onclick="textReveal()" id="text-reveal-button">Show</button></p>
-        <div id="text-reveal" class="description-text">
-            {{ video.description|linebreaks }}
-        </div>
-    </div>
-{% endif %}
-{% endblock content %}
+<script>
+    var videoData = getVideoData('{{ video.youtube_id }}');
+    sponsorBlock = videoData.data.sponsorblock;
+    var videoProgress = getVideoProgress('{{ video.youtube_id }}').position;
+    window.onload = insertVideoTag(videoData, videoProgress);
+</script>
+{% endblock content %}

tubearchivist/home/urls.py

@@ -11,6 +11,9 @@ from home.views import (
     DownloadView,
     HomeView,
     LoginView,
+    PlaylistIdView,
+    PlaylistView,
+    SearchView,
     SettingsView,
     VideoView,
     process,
@@ -32,10 +35,10 @@ urlpatterns = [
     ),
     path("settings/", login_required(SettingsView.as_view()), name="settings"),
     path("process/", login_required(process), name="process"),
-    path("downloads/progress/", login_required(progress), name="progress"),
+    path("progress/", login_required(progress), name="progress"),
     path("channel/", login_required(ChannelView.as_view()), name="channel"),
     path(
-        "channel/<slug:channel_id_detail>/",
+        "channel/<slug:channel_id>/",
         login_required(ChannelIdView.as_view()),
         name="channel_id",
     ),
@@ -44,4 +47,11 @@ urlpatterns = [
         login_required(VideoView.as_view()),
         name="video",
     ),
+    path("playlist/", login_required(PlaylistView.as_view()), name="playlist"),
+    path(
+        "playlist/<slug:playlist_id>/",
+        login_required(PlaylistIdView.as_view()),
+        name="playlist_id",
+    ),
+    path("search/", login_required(SearchView.as_view()), name="search"),
 ]

tubearchivist/requirements.txt

@@ -1,9 +1,12 @@
-beautifulsoup4==4.10.0
-celery==5.1.2
-Django==3.2.8
-Pillow==8.4.0
-redis==3.5.3
-requests==2.26.0
+beautifulsoup4==4.11.1
+celery==5.2.6
+Django==4.0.4
+django-cors-headers==3.11.0
+djangorestframework==3.13.1
+Pillow==9.1.0
+redis==4.2.2
+requests==2.27.1
+ryd-client==0.0.3
 uWSGI==2.0.20
-whitenoise==5.3.0
-yt_dlp==2021.10.22
+whitenoise==6.0.0
+yt_dlp==2022.4.8

tubearchivist/static/.jshintrc

@@ -0,0 +1,3 @@
+{
+    "esversion": 6
+}

tubearchivist/static/cast-videos.js

@@ -0,0 +1,148 @@
+function initializeCastApi() {
+    cast.framework.CastContext.getInstance().setOptions({
+        receiverApplicationId: chrome.cast.media.DEFAULT_MEDIA_RECEIVER_APP_ID, // Use built in reciver app on cast device, see https://developers.google.com/cast/docs/styled_receiver if you want to be able to add a theme, splash screen or watermark. Has a $5 one time fee.
+        autoJoinPolicy: chrome.cast.AutoJoinPolicy.ORIGIN_SCOPED
+    });
+
+    var player = new cast.framework.RemotePlayer();
+    var playerController = new cast.framework.RemotePlayerController(player);
+
+    // Add event listerner to check if a connection to a cast device is initiated
+    playerController.addEventListener(
+        cast.framework.RemotePlayerEventType.IS_CONNECTED_CHANGED, function() { 
+            castConnectionChange(player) 
+        }
+    );
+    playerController.addEventListener(
+        cast.framework.RemotePlayerEventType.CURRENT_TIME_CHANGED, function() { 
+            castVideoProgress(player) 
+        }
+    );
+    playerController.addEventListener(
+        cast.framework.RemotePlayerEventType.IS_PAUSED_CHANGED, function() { 
+            castVideoPaused(player) 
+        }
+    );
+}
+
+
+function castConnectionChange(player) {
+    // If cast connection is initialized start cast
+    if (player.isConnected) {
+        // console.log("Cast Connected.");
+        castStart();
+    } else if (!player.isConnected) {
+        // console.log("Cast Disconnected.");
+    }
+}
+
+function castVideoProgress(player) {
+    var videoId = getVideoPlayerVideoId();
+    if (player.mediaInfo.contentId.includes(videoId)) {
+        var currentTime = player.currentTime;
+        var duration = player.duration;
+        if ((currentTime % 10) <= 1.0 && currentTime != 0 && duration != 0) { // Check progress every 10 seconds or else progress is checked a few times a second
+            postVideoProgress(videoId, currentTime);
+            setProgressBar(videoId, currentTime, duration);
+            if (!getVideoPlayerWatchStatus()) { // Check if video is already marked as watched
+                if (watchedThreshold(currentTime, duration)) {
+                    isWatched(videoId);
+                }
+            }
+        }
+    }
+}
+
+function castVideoPaused(player) {
+    var videoId = getVideoPlayerVideoId();
+    var currentTime = player.currentTime;
+    var duration = player.duration;
+    if (player.mediaInfo != null) {
+        if (player.mediaInfo.contentId.includes(videoId)) {
+            if (currentTime != 0 && duration != 0) {
+                postVideoProgress(videoId, currentTime);
+            }
+        }
+    }
+}
+
+function castStart() {
+    var castSession = cast.framework.CastContext.getInstance().getCurrentSession();
+    // Check if there is already media playing on the cast target to prevent recasting on page reload or switching to another video page
+    if (!castSession.getMediaSession()) { 
+        var videoId = getVideoPlayerVideoId();
+        var videoData = getVideoData(videoId);
+        var contentId = getURL() + videoData.data.media_url;
+        var contentTitle = videoData.data.title;
+        var contentImage = getURL() + videoData.data.vid_thumb_url;
+
+        contentType = 'video/mp4'; // Set content type, only videos right now so it is hard coded
+        contentCurrentTime = getVideoPlayerCurrentTime(); // Get video's current position
+        contentActiveSubtitle = [];
+        // Check if a subtitle is turned on.
+        for (var i = 0; i < getVideoPlayer().textTracks.length; i++) {
+            if (getVideoPlayer().textTracks[i].mode == "showing") {
+                contentActiveSubtitle =[i + 1];
+            }
+        }
+        contentSubtitles = [];
+        var videoSubtitles = videoData.data.subtitles; // Array of subtitles
+        if (typeof(videoSubtitles) != 'undefined' && videoData.config.downloads.subtitle) {
+            for (var i = 0; i < videoSubtitles.length; i++) {
+                subtitle = new chrome.cast.media.Track(i, chrome.cast.media.TrackType.TEXT);
+                subtitle.trackContentId = videoSubtitles[i].media_url;
+                subtitle.trackContentType = 'text/vtt';
+                subtitle.subtype = chrome.cast.media.TextTrackType.SUBTITLES;
+                subtitle.name = videoSubtitles[i].name;
+                subtitle.language = videoSubtitles[i].lang;
+                subtitle.customData = null;
+                contentSubtitles.push(subtitle);
+            }
+        }
+
+        mediaInfo = new chrome.cast.media.MediaInfo(contentId, contentType); // Create MediaInfo var that contains url and content type
+        // mediaInfo.streamType = chrome.cast.media.StreamType.BUFFERED; // Set type of stream, BUFFERED, LIVE, OTHER
+        mediaInfo.metadata = new chrome.cast.media.GenericMediaMetadata(); // Create metadata var and add it to MediaInfo
+        mediaInfo.metadata.title = contentTitle.replace("&amp;", "&"); // Set the video title
+        mediaInfo.metadata.images = [new chrome.cast.Image(contentImage)]; // Set the video thumbnail
+        // mediaInfo.textTrackStyle = new chrome.cast.media.TextTrackStyle();
+        mediaInfo.tracks = contentSubtitles;
+
+        var request = new chrome.cast.media.LoadRequest(mediaInfo); // Create request with the previously set MediaInfo. 
+        // request.queueData = new chrome.cast.media.QueueData(); // See https://developers.google.com/cast/docs/reference/web_sender/chrome.cast.media.QueueData for playlist support.
+        request.currentTime = shiftCurrentTime(contentCurrentTime); // Set video start position based on the browser video position
+        request.activeTrackIds = contentActiveSubtitle; // Set active subtitle based on video player
+        // request.autoplay = false; // Set content to auto play, true by default
+        castSession.loadMedia(request).then(
+            function() { 
+                castSuccessful();
+            }, 
+            function() { 
+                castFailed(errorCode); 
+            }
+        ); // Send request to cast device
+    }   
+}
+
+function shiftCurrentTime(contentCurrentTime) { // Shift media back 3 seconds to prevent missing some of the content
+    if (contentCurrentTime > 5) {
+        return(contentCurrentTime - 3);
+    } else {
+        return(0);
+    }
+}
+
+function castSuccessful() {
+    // console.log('Cast Successful.');
+    getVideoPlayer().pause(); // Pause browser video on successful cast
+}
+
+function castFailed(errorCode) {
+    console.log('Error code: ' + errorCode);
+}
+
+window['__onGCastApiAvailable'] = function(isAvailable) {
+    if (isAvailable) {
+        initializeCastApi();
+    }
+}

tubearchivist/static/css/style.css

@@ -17,6 +17,16 @@ font-family: 'Sen-Regular';
 
 html {
     height: 100%;
+    scrollbar-color: var(--accent-font-dark) #0000;
+}
+
+::-webkit-scrollbar {
+    width: 7px;
+}
+
+::-webkit-scrollbar-thumb {
+    background: var(--accent-font-dark);
+    border-radius: 10px;
 }
 
 body {
@@ -52,6 +62,13 @@ h3 {
     color: var(--accent-font-light);
 }
 
+h4 {
+    font-size: 0.7em;
+    margin-bottom: 7px;
+    font-family: Sen-Regular, sans-serif;
+    color: var(--accent-font-light);
+}
+
 p, i, li {
     font-family: Sen-Regular, sans-serif;
     margin-bottom: 10px;
@@ -62,7 +79,7 @@ ul {
     margin-left: 20px;
 }
 
-td, span, label {
+td, th, span, label {
     font-family: Sen-Regular, sans-serif;
     color: var(--main-font);
 }
@@ -102,6 +119,14 @@ button:hover {
     color: var(--main-bg);
 }
 
+.unsubscribe {
+    background-color: var(--accent-font-light);
+}
+
+.unsubscribe:hover {
+    background-color: var(--accent-font-dark);
+}
+
 .boxed-content {
     max-width: 1000px;
     width: 80%;
@@ -118,7 +143,6 @@ button:hover {
 
 .top-banner {
     text-align: center;
-    margin-top: 10px;
 }
 
 .top-banner img {
@@ -148,8 +172,8 @@ button:hover {
     position: relative;
     width: 70px;
     height: 30px;
-    background-color: var(--accent-font-dark);
-    border-color: var(--accent-font-dark);
+    background-color: var(--accent-font-light);
+    border-color: var(--accent-font-light);
     appearance: none;
     border-radius: 15px;
     transition: 0.4s;
@@ -158,8 +182,8 @@ button:hover {
 }
 
 .toggleBox > input:checked[type="checkbox"] {
-    background-color: var(--accent-font-light);
-    border-color: var(--accent-font-light);
+    background-color: var(--accent-font-dark);
+    border-color: var(--accent-font-dark);
 }
 
 .toggleBox > input[type="checkbox"]::before {
@@ -193,14 +217,17 @@ button:hover {
 }
 
 .toggleBox > .onbtn {
-    bottom: 15px;
-    left: 15px;
+    right: 70%;
+    top: 45%;
+    transform: translate(50%,-50%);
     font-family: Sen-Regular, sans-serif;
+
 }
 
 .toggleBox > .ofbtn {
-    bottom: 15px;
-    right: 15px;
+    left: 37%;
+    top: 45%;
+    transform: translate(50%,-50%);
     font-family: Sen-Regular, sans-serif;
     color: var(--main-font);
 }
@@ -210,6 +237,10 @@ button:hover {
     display: none;
 }
 
+.delete-confirm button {
+    margin: 3px 0;
+}
+
 .danger-button {
     background-color: var(--highlight-error);
 }
@@ -221,7 +252,7 @@ button:hover {
 /* navigation */
 .top-nav {
     display: block;
-    padding: 15px 0;
+    padding: 5px 0;
     position: relative;
 }
 
@@ -254,15 +285,27 @@ button:hover {
     filter: var(--img-filter);
 }
 
+#castbutton {
+    float: right;
+    width: 40px;
+    padding: 0 5px;
+    --disconnected-color: var(--accent-font-dark);
+    --connected-color: var(--accent-font-light);
+}
+
+.alert-hover:hover {
+    filter: var(--img-filter-error);
+}
+
 /* top of page */
 .title-bar {
-    padding: 20px 0;
+    padding-top: 30px;
 }
 
 .sort {
     display: flex;
     flex-wrap: wrap;
-    align-content: center;
+    align-items: center;
 }
 
 .padding-box {
@@ -277,23 +320,9 @@ button:hover {
     width: 50%;
 }
 
-.search-form input {
-    width: 90%;
-}
-
-.search-icon {
-    flex: 0 0 40px;
-}
-
-.search-icon img {
-    width: 100%;
-    cursor: pointer;
-    filter: var(--img-filter);
-}
-
 .view-controls {
     display: grid;
-    grid-template-columns: 1fr 1fr;
+    grid-template-columns: 1fr auto auto;
     border-bottom: 2px solid;
     border-color: var(--accent-font-dark);
     margin: 15px 0;
@@ -311,12 +340,6 @@ button:hover {
     filter: var(--img-filter);
 }
 
-#search-box {
-    display: none;
-    flex: auto;
-    padding: 10px;
-}
-
 #hidden-form {
     display: none;
 }
@@ -328,49 +351,56 @@ button:hover {
 
 
 /* video player */
-.video-player {
-    position: relative;
+.player-wrapper {
+    background-color: var(--highlight-bg);
     margin: 20px 0;
 }
 
-.video-player video {
-    max-height: 70vh;
+.video-player {
+    display: grid;
+    align-content: space-evenly;
+    height: 100vh;
+}
+
+.notifications {
+    text-align: center;
+    width: 80%;
+    margin: auto;
+}
+
+.sponsorblock {
+    text-align: center;
+    width: 80%;
+    margin: auto;
+}
+
+.video-player video,
+.video-main video {
+    max-height: 80vh;
+    width: 90%;
+    max-width: 1500px;
+    margin: 0 auto;
+    display: block;
 }
 
 .player-title img {
     width: 30px;
+    margin: 10px 10px 10px 0;
 }
 
-.player-title:hover {
-    opacity: 1;
-}
-
-.player-title {
-    opacity: 0;
-    background-color: var(--highlight-bg-transparent);
-    position: absolute;
-    top: 0;
-    z-index: 1;
-    width: 100%;
-    padding: 20px 0;
-}
-
-.player-title > * {
-    margin-left: 20px;
-}
-
-
 /* video list */
-.video-list.grid {
+.video-list {
     display: grid;
-    grid-template-columns: 1fr 1fr 1fr;
     grid-gap: 1rem;
+    margin-top: 1rem;
+}
+
+.video-list.grid {
+    grid-template-columns: 1fr 1fr 1fr;
 }
 
 .video-list.list {
-    display: grid;
     grid-template-columns: unset;
-    grid-gap: 1rem;
 }
 
 .video-item {
@@ -382,8 +412,17 @@ button:hover {
     grid-template-columns: 25% auto;
 }
 
+.video-progress-bar {
+    position: absolute;
+    background-color: var(--accent-font-dark);
+    height: 7px;
+    left: 0;
+    bottom: 3px;
+}
+
 .video-thumb img {
     width: 100%;
+    position: relative;
 }
 
 .video-play img {
@@ -441,19 +480,30 @@ button:hover {
     text-align: left;
 }
 
-.video-desc h3 {
+.video-desc h3,
+.player-title h3 {
     font-size: 0.9em;
     text-transform: uppercase;
 }
 
+.player-stats {
+    float: right;
+    display: flex;
+    align-items: center;
+    margin-top: 10px;
+}
+
+.player-stats span {
+    margin: 0 5px;
+}
+
 .video-desc-player {
     margin-bottom: 8px;
     display: flex;
     align-items: center;
 }
 
-.unseen-icon,
-.seen-icon,
+.watch-button,
 .close-button {
     cursor: pointer;
     filter: var(--img-filter);
@@ -480,10 +530,31 @@ button:hover {
 
 
 /* info box */
+.title-split {
+    display: grid;
+    grid-template-columns: 1fr 1fr;
+}
+
+.title-split img {
+    width: 40px;
+    filter: var(--img-filter);
+    cursor: pointer;
+    margin: 0 10px;
+}
+
+.title-split-form {
+    padding-top: 30px;
+    display: flex;
+}
+
 .info-box {
     display: grid;
     grid-gap: 1rem;
-    margin: 1rem 0;
+    margin-top: 1rem;
+}
+
+.description-box {
+    margin-top: 1rem;
 }
 
 .info-box-3 {
@@ -514,6 +585,28 @@ button:hover {
     margin-bottom: 10px;
 }
 
+.overwrite-form {
+    display: grid;
+    grid-template-columns: 1fr 1fr;
+    width: 100%;
+}
+
+.overwrite-form button {
+    width: 200px;
+}
+
+.overwrite-form-item {
+    margin-bottom: 1rem;
+}
+
+.overwrite-form-item input {
+    width: 90%;
+}
+
+.hidden-overwrite {
+    display: none;
+}
+
 /* login */
 .login-page {
     display: flex;
@@ -542,6 +635,10 @@ button:hover {
     min-width: 200px;
 }
 
+#id_remember_me {
+    min-width: unset;
+}
+
 .login-page button,
 .login-page .danger-zone {
     width: 210px;
@@ -578,9 +675,15 @@ button:hover {
 }
 
 /* video page */
+.video-main {
+    margin: 1rem 0;
+}
+
 .video-main video {
     max-height: 70vh;
+    margin-bottom: 1rem;
 }
+
 .video-info-watched {
     display: flex;
     align-items: center;
@@ -598,10 +701,49 @@ button:hover {
     filter: var(--img-filter);
 }
 
-.thumb-icon.dislike img {
+.dislike {
     transform: rotate(180deg);
 }
 
+.playlist-wrap {
+    background-color: var(--highlight-bg);
+    margin: 1rem 0;
+    padding: 1rem;
+}
+
+.playlist-wrap > a > h3 {
+    text-align: center;
+}
+
+.playlist-nav {
+    display: grid;
+    grid-template-columns: 1fr 1fr;
+    margin-bottom: 10px;
+}
+
+.playlist-nav-item {
+    display: flex;
+    justify-content: space-between;
+}
+
+.playlist-nav-item img {
+    width: 200px;
+}
+
+.playlist-desc {
+    padding: 5px;
+    width: 100%;
+}
+
+/* multi search page */
+.multi-search-box input {
+    width: 100%;
+}
+
+.multi-search-result {
+    padding: 1rem 0;
+}
+
 /* channel overview page */
 .channel-list.list {
     display: block;
@@ -614,7 +756,7 @@ button:hover {
 }
 
 .channel-item.list {
-    padding: 20px 0;
+    padding-bottom: 1rem;
 }
 
 .channel-item.grid > .info-box {
@@ -622,6 +764,7 @@ button:hover {
 }
 
 .channel-banner img {
+    margin-top: 1rem;
     width: 100%;
 }
 
@@ -638,6 +781,51 @@ button:hover {
     transform: translateX(-30%);
 }
 
+/* playlist overview page */
+.playlist-list.list {
+    display: grid;
+    grid-template-columns: unset;
+    gap: 1rem;
+}
+
+.playlist-list.grid {
+    display: grid;
+    grid-template-columns: 1fr 1fr 1fr;
+    gap: 1rem;
+}
+
+.playlist-item {
+    overflow: hidden;
+}
+
+.playlist-item.list {
+    display: flex;
+}
+
+.playlist-thumbnail img {
+    width: 100%;
+}
+
+.playlist-desc.grid {
+    padding: 10px;
+    height: 100%;
+    background-color: var(--highlight-bg);
+}
+
+.playlist-desc.list {
+    width: 100%;
+    padding: 10px;
+    height: unset;
+    background-color: var(--highlight-bg);
+    display: flex;
+    flex-wrap: wrap;
+    align-content: center;
+}
+
+.playlist-desc.list > a,
+.playlist-desc.list > p {
+    width: 100%;
+}
 
 /* download page */
 .icon-text {
@@ -721,7 +909,7 @@ button:hover {
 .dl-control-icons {
     display: flex;
     justify-content: center;
-    padding: 10px 0;
+    padding-bottom: 10px;
 }
 
 .dl-control-icons img {
@@ -738,22 +926,28 @@ button:hover {
     filter: var(--img-filter-error);
 }
 
+.title-split {
+    display: flex;
+    justify-content: space-between;
+}
+
 /* status message */
-.download-progress {
+.notification {
     background-color: var(--highlight-bg);
     text-align: center;
     padding: 30px 0 15px 0;
+    margin: 1rem 0;
 }
 
-.download-progress.info {
+.notification.info {
     background-color: var(--highlight-bg);
 }
 
-.download-progress.error {
+.notification.error {
     background-color: var(--highlight-error);
 }
 
-.download-progress.error h3 {
+.notification.error h3 {
     color: #fff;
 }
 
@@ -778,6 +972,19 @@ button:hover {
     padding: 3px;
 }
 
+.backup-grid-row {
+    display: grid;
+    grid-template-columns: 10% 10% 10% auto;
+    align-items: center;
+    padding: 5px 10px;
+    border-bottom: solid 1px;
+    border-color: var(--main-font);
+}
+
+.backup-grid-row > span {
+    margin-left: 10px;
+}
+
 /* about */
 .about-section {
     padding: 20px 0;
@@ -849,13 +1056,15 @@ button:hover {
     }
     .video-list.grid,
     .dl-list.grid,
-    .channel-list.grid {
+    .channel-list.grid,
+    .playlist-list.grid {
         grid-template-columns: 1fr 1fr;
     }
     .dl-thumb.list {
         width: 35%;
     }
-    .video-item.list {
+    .video-item.list,
+    .playlist-item.list {
         display: grid;
         grid-template-columns: 35% auto;
     }
@@ -865,6 +1074,23 @@ button:hover {
     .two-col > div {
         width: 100%;
     }
+    .top-nav {
+        flex-wrap: wrap-reverse;
+        display: flex;
+    }
+    .nav-icons {
+        width: 100%;
+        justify-content: center;
+        position: unset;
+        transform: unset;
+    }
+    .video-player {
+        height: unset;
+        padding: 20px 0
+    }
+    .video-player video {
+        width: 90%;
+    }
 }
 
 /* phone */
@@ -875,9 +1101,17 @@ button:hover {
     .video-list.grid,
     .dl-list.grid,
     .channel-list.grid,
-    .video-item.list {
+    .video-item.list,
+    .playlist-list.list,
+    .playlist-list.grid,
+    .info-box-2,
+    .info-box-3,
+    .overwrite-form {
         grid-template-columns: 1fr;
     }
+    .playlist-item.list {
+        display: block;
+    }
     .video-desc.grid {
         height: unset;
         display: flex;
@@ -889,34 +1123,28 @@ button:hover {
     .footer {
         text-align: center;
     }
+    .footer .boxed-content span {
+        width: 100%;
+        display: block;
+    }
     .toggle {
         flex-wrap: wrap;
     }
-    .top-nav {
-        flex-wrap: wrap-reverse;
-        display: flex;
-    }
-    .nav-icons {
-        width: 100%;
-        justify-content: center;
-        position: unset;
-        transform: unset;
+    .nav-items {
+        display: grid;
+        grid-template-columns: 1fr 1fr;
     }
     .nav-item {
-        padding: unset;
+        padding: 5px 0;
         margin: 15px;
+        text-align: center;
     }
     .sort {
         display: block;
     }
     .sort select {
-        width: 100%;
         margin: unset;
     }
-    .info-box-2,
-    .info-box-3 {
-        grid-template-columns: 1fr;
-    }
     .description-box {
         display: block;
     }
@@ -929,4 +1157,24 @@ button:hover {
     .dl-desc {
         padding: 15px;
     }
+    .backup-grid-row {
+        display: flex;
+        flex-wrap: wrap;
+        padding: 10px 0;
+        justify-content: center;
+    }
+    .backup-grid-row span {
+        padding: 5px 0;
+    }
+    .playlist-nav {
+        display: block;
+        grid-template-columns: unset;
+    }
+    .playlist-nav-item {
+        display: block;
+        justify-content: unset;
+    }
+    .playlist-nav-item img {
+        width: 100%;
+    }
 }