bumping versions for new release

* Added helper function in deploy.sh to run same tests as github actions locally

.dockerignore

@@ -17,5 +17,8 @@ venv/
 # Unneeded graphics
 assets/*
 
+# Unneeded docs
+docs/*
+
 # for local testing only
 testing.sh

.github/workflows/lint_python.yml

@@ -0,0 +1,23 @@
+name: lint_python
+on: [pull_request, push]
+jobs:
+  lint_python:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/setup-python@v2
+      - run: pip install --upgrade pip wheel
+      - run: pip install bandit black codespell flake8 flake8-bugbear
+                         flake8-comprehensions isort
+      - run: bandit --recursive  --skip B105,B108,B404,B603,B607 .
+      - run: black --check --diff --line-length 79 .
+      - run: codespell
+      - run: flake8 . --count --max-complexity=12 --max-line-length=79
+                      --show-source --statistics
+      - run: isort --check-only --line-length 79 --profile black .
+      # - run: pip install -r tubearchivist/requirements.txt
+      # - run: mkdir --parents --verbose .mypy_cache
+      # - run: mypy --ignore-missing-imports --install-types --non-interactive .
+      # - run: python3 tubearchivist/manage.py test || true
+      # - run: shopt -s globstar && pyupgrade --py36-plus **/*.py || true
+      # - run: safety check

CONTRIBUTING.md

@@ -0,0 +1,52 @@
+## Contributing to Tube Archivist
+
+Welcome, and thanks for showing interest in improving Tube Archivist!  
+If you haven't already, the best place to start is the README. This will give you an overview on what the project is all about.
+
+## Report a bug
+
+If you notice something is not working as expected, check to see if it has been previously reported in the [open issues](https://github.com/bbilly1/tubearchivist/issues).
+If it has not yet been disclosed, go ahead and create an issue.  
+If the issue doesn't move forward due to a lack of response, I assume it's solved and will close it after some time to keep the list fresh. 
+
+## Wiki
+
+WIP: The wiki is where all user functions are explained in detail. These pages are mirrored into the **docs** folder of the repo. This allows for pull requests and all other features like regular code. Make any changes there, and I'll sync them  with the wiki tab.
+
+## Development Environment
+
+I have learned the hard way, that working on a dockerized application outside of docker is very error prone and in general not a good idea. So if you want to test your changes, it's best to run them in a docker testing environment.  
+
+This is my setup I have landed on, YMMV:
+- Clone the repo, work on it with your favorite code editor in your local filesystem. *testing* branch is the where all the changes are happening, might be unstable and is WIP.
+- Then I have a VM on KVM hypervisor running standard Ubuntu Server LTS with docker installed. The VM keeps my projects separate and offers convenient snapshot functionality. The VM also offers ways to simulate lowend environments by limiting CPU cores and memory. But you could also just run docker on your host system.
+- Additionally to the required services as listed in the example docker-compose file, the **Dev Tools** of [Kibana](https://www.elastic.co/guide/en/kibana/current/docker.html) are invaluable for running and testing Elasticsearch queries.
+- The `Dockerfile` is structured in a way that the actual application code is in the last layer so rebuilding the image with only code changes utilizes the build cache for everything else and will take just 2-3 secs.
+- Take a look at the `deploy.sh` file. I have my local DNS resolve `tubearchivist.local` to the IP of the VM for convenience. To deploy the latest changes and rebuild the application to the testing VM run:
+```bash
+./deploy.sh test
+```
+- The command above will also copy the file `tubarchivist/testing.sh` into the working folder of the container. Running this script will install additional debugging tools I regularly use in testing. 
+- This `deploy.sh` file is not meant to be universally usable for every possible environment but could serve as an idea on how to automatically rebuild containers to test changes - customize to your liking. 
+
+## Implementing a new feature
+
+Do you see anything on the roadmap that you would like to take a closer look at but you are not sure, what's the best way to tackle that? Or anything not on there yet you'd like to implement but are not sure how? Open up an issue and we try to find a solution together.
+
+## Making changes
+
+To fix a bug or implement a feature, fork the repository and make all changes to the testing branch. When ready, create a pull request.
+
+## Releases
+
+Everything on the master branch is what's in the latest release and is what you get in your container when you `pull` either the *:latest* tag or the newest named version. If you want to test the newest changes and improvements, clone the repository and build the docker container with the Dockerfile from the testing branch.
+
+## Code formatting and linting
+
+To keep things clean and consistent for everybody, there is a github action setup to lint and check the changes. You can test your code locally first if you want. For example if you made changes in the **download** module, run
+
+```shell
+./deploy.sh validate tubearchivist/home/src/download.py
+```
+
+to validate your changes. If you omit the path, all the project files will get checked. This is subject to change as the codebase improves. 

Dockerfile

@@ -7,10 +7,20 @@ ENV PYTHONUNBUFFERED 1
 # install distro packages needed
 RUN apt-get clean && apt-get -y update && apt-get -y install --no-install-recommends \
     build-essential \
-    ffmpeg \
     nginx \
     curl && rm -rf /var/lib/apt/lists/*
 
+# get newest patched ffmpeg and ffprobe builds
+RUN curl -s https://api.github.com/repos/yt-dlp/FFmpeg-Builds/releases/latest \
+    | grep browser_download_url \
+    | grep linux64 \
+    | grep ffmpeg-n \
+    | cut -d '"' -f 4 \
+    | xargs curl -L --output ffmpeg.tar.xz && \
+    tar -xf ffmpeg.tar.xz --strip-components=2 --no-anchored -C /usr/bin/ "ffmpeg" && \
+    tar -xf ffmpeg.tar.xz --strip-components=2 --no-anchored -C /usr/bin/ "ffprobe" && \
+    rm ffmpeg.tar.xz
+
 # copy config files
 COPY nginx.conf /etc/nginx/conf.d/
 
@@ -34,7 +44,7 @@ VOLUME /youtube
 
 # start
 WORKDIR /app
-EXPOSE 80
+EXPOSE 8000
 
 RUN chmod +x ./run.sh
 

README.md

@@ -1,10 +1,23 @@
 ![Tube Archivist](assets/tube-archivist-banner.jpg?raw=true "Tube Archivist Banner")  
 
-<center><h1>Your self hosted Youtube media server</h1></center>
+<center><h1>Your self hosted YouTube media server</h1></center>
 
+## Table of contents:
+* [Wiki](https://github.com/bbilly1/tubearchivist/wiki) for a detailed documentation
+* [Core functionality](#core-functionality)
+* [Screenshots](#screenshots)
+* [Problem Tube Archivist tries to solve](#problem-tube-archivist-tries-to-solve)
+* [Installing and updating](#installing-and-updating)
+* [Getting Started](#getting-started)
+* [Potential pitfalls](#potential-pitfalls)
+* [Roadmap](#roadmap)
+* [Known limitations](#known-limitations)
+* [Donate](#donate)
+
+------------------------
 
 ## Core functionality
-* Subscribe to your favourite Youtube channels
+* Subscribe to your favorite YouTube channels
 * Download Videos using **yt-dlp**
 * Index and make videos searchable
 * Play videos
@@ -27,10 +40,10 @@
 *Downloads Page*
   
 ## Problem Tube Archivist tries to solve
-Once your Youtube video collection grows, it becomes hard to search and find a specific video. That's where Tube Archivist comes in: By indexing your video collection with metadata from Youtube, you can organize, search and enjoy your archived Youtube videos without hassle offline through a convenient web interface.
+Once your YouTube video collection grows, it becomes hard to search and find a specific video. That's where Tube Archivist comes in: By indexing your video collection with metadata from YouTube, you can organize, search and enjoy your archived YouTube videos without hassle offline through a convenient web interface.
 
-## Installation
+## Installing and updating
-Take a look at the example `docker-compose.yml` file provided. Tube Archivist depends on three main components split up into seperate docker containers:  
+Take a look at the example `docker-compose.yml` file provided. Tube Archivist depends on three main components split up into separate docker containers:  
 
 ### Tube Archivist
 The main Python application that displays and serves your video collection, built with Django.
@@ -38,51 +51,34 @@ The main Python application that displays and serves your video collection, buil
   - Needs a mandatory volume for the video archive at **/youtube**
   - And another recommended volume to save the cache for thumbnails and artwork at **/cache**.
   - The environment variables `ES_URL` and `REDIS_HOST` are needed to tell Tube Archivist where Elasticsearch and Redis respectively are located.
-  - The environment variables `HOST_UID` and `HOST_GID` allowes Tube Archivist to `chown` the video files to the main host system user instead of the container user.
+  - The environment variables `HOST_UID` and `HOST_GID` allows Tube Archivist to `chown` the video files to the main host system user instead of the container user.
 
 ### Elasticsearch
 Stores video meta data and makes everything searchable. Also keeps track of the download queue.
-  - Needs to be accessable over the default port `9200`
+  - Needs to be accessible over the default port `9200`
   - Needs a volume at **/usr/share/elasticsearch/data** to store data
 
 Follow the [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html) for additional installation details.
 
 ### Redis JSON
-Functions as a cache and temporary link between the application and the filesystem. Used to store and display messages and configuration variables.
+Functions as a cache and temporary link between the application and the file system. Used to store and display messages and configuration variables.
-  - Needs to be accessable over the default port `6379`
+  - Needs to be accessible over the default port `6379`
   - Takes an optional volume at **/data** to make your configuration changes permanent.
 
-## Getting Started
+### Redis on a custom port
-1. Go through the **settings** page and look at the available options. Particularly set *Download Format* to your desired video quality before downloading. **Tube Archivist** downloads the best available quality by default.
+For some architectures it might be required to run Redis JSON on a nonstandard port. To for example change the Redis port to **6380**, set the following values:
-2. Subscribe to some of your favourite Youtube channels on the **channels** page. 
+- Set the environment variable `REDIS_PORT=6380` to the *tubearchivist* service.
-3. On the **downloads** page, click on *Rescan subscriptions* to add videos from the subscribed channels to your Download queue or click on *Add to download queue* to manually add Video IDs, links, channels or playlists.
+- For the *archivist-redis* service, change the ports to `6380:6380`
-4. Click on *Download queue* and let Tube Archivist to it's thing. 
+- Additionally set the following value to the *archivist-redis* service: `command: --port 6380 --loadmodule /usr/lib/redis/modules/rejson.so`
-5. Enjoy your archived collection!
-  
-## Import your existing library
-So far this depends on the video you are trying to import to be still available on youtube to get the metadata. Add the files you like to import to the */cache/import* folder. Then start the process from the settings page *Manual media files import*. Make sure to follow one of the two methods below.
-
-### Method 1:
-Add a matching *.json* file with the media file. Both files need to have the same base name, for example:
-- For the media file: \<base-name>.mp4
-- For the JSON file: \<base-name>.info.json
-- Alternate JSON file: \<base-name>.json
-
-**Tube Archivist** then looks for the 'id' key within the JSON file to identify the video.
-
-### Method 2:
-Detect the Youtube ID from filename, this accepts the default yt-dlp naming convention for file names like:
-- \<base-name>[\<youtube-id>].mp4
-- The Youtube ID in square brackets at the end of the filename is the crucial part.
-
-### Some notes:
-- This will **consume** the files you put into the import folder: Files will get converted to mp4 if needed (this might take a long time...) and moved to the archive, *.json* files will get deleted upon completion to avoid having doublicates on the next run.
-- Maybe start with a subset of your files to import to make sure everything goes well...
-- Follow the logs to monitor progress and errors: `docker-compose logs -f tubearchivist`.
 
+### Updating Tube Archivist
+You will see the current version number of **Tube Archivist** in the footer of the interface so you can compare it with the latest release to make sure you are running the *latest and greatest*.  
+* There can be breaking changes between updates, particularly as the application grows, new environment variables or settings might be required for you to set in the your docker-compose file. Any breaking changes will be marked in the **release notes**.  
+* All testing and development is done with the Elasticsearch version number as mentioned in the provided *docker-compose.yml* file. This will be updated when a new release of Elasticsearch is available. Running an older version of Elasticsearch is most likely not going to result in any issues, but it's still recommended to run the same version as mentioned.
 
 ## Potential pitfalls
-**Elastic Search** in Docker requires the kernel setting of the host machine `vm.max_map_count` to be set to least 262144.
+### vm.max_map_count
+**Elastic Search** in Docker requires the kernel setting of the host machine `vm.max_map_count` to be set to at least 262144.
 
 To temporary set the value run:  
 ```
@@ -94,21 +90,39 @@ To apply the change permanently depends on your host operating system:
 - On Arch based systems create a file */etc/sysctl.d/max_map_count.conf* with the content `vm.max_map_count = 262144`. 
 - On any other platform look up in the documentation on how to pass kernel parameters.
 
+### Permissions for elasticsearch
+If you see a message similar to `AccessDeniedException[/usr/share/elasticsearch/data/nodes]` when initially starting elasticsearch, that means the container is not allowed to write files to the volume.  
+That's most likely the case when you run `docker-compose` as an unprivileged user. To fix that issue, shutdown the container and on your host machine run:
+```
+chown 1000:0 /path/to/mount/point
+```
+This will match the permissions with the **UID** and **GID** of elasticsearch within the container and should fix the issue.
 
+## Getting Started
+1. Go through the **settings** page and look at the available options. Particularly set *Download Format* to your desired video quality before downloading. **Tube Archivist** downloads the best available quality by default.
+2. Subscribe to some of your favorite YouTube channels on the **channels** page. 
+3. On the **downloads** page, click on *Rescan subscriptions* to add videos from the subscribed channels to your Download queue or click on *Add to download queue* to manually add Video IDs, links, channels or playlists.
+4. Click on *Start download* and let **Tube Archivist** to it's thing. 
+5. Enjoy your archived collection!
+  
 ## Roadmap
-This should be considered as a **minimal viable product**, there is an exstensive list of future functions and improvements planned.
+This should be considered as a **minimal viable product**, there is an extensive list of future functions and improvements planned.
 
 ### Functionality
-- [ ] Access controll
+- [ ] Access control
 - [ ] User roles
 - [ ] Delete videos and channel
 - [ ] Create playlists
-- [ ] Backup and restore
 - [ ] Podcast mode to serve channel as mp3
 - [ ] Implement [PyFilesystem](https://github.com/PyFilesystem/pyfilesystem2) for flexible video storage
-- [X] Scan your filesystem to index already downloaded videos [2021-09-14]
+- [ ] Un-ignore videos
+- [ ] Add thumbnail embed option
+- [X] Dynamic download queue [2021-09-26]
+- [X] Backup and restore [2021-09-22]
+- [X] Scan your file system to index already downloaded videos [2021-09-14]
 
 ### UI
+- [ ] Create a github wiki for user documentation
 - [ ] Show similar videos on video page
 - [ ] Multi language support
 - [ ] Grid and list view for both channel and video list pages
@@ -118,4 +132,12 @@ This should be considered as a **minimal viable product**, there is an exstensiv
 ## Known limitations
 - Video files created by Tube Archivist need to be **mp4** video files for best browser compatibility.
 - Every limitation of **yt-dlp** will also be present in Tube Archivist. If **yt-dlp** can't download or extract a video for any reason, Tube Archivist won't be able to either.
-- For now this is meant to be run in a trusted network environment.
+- For now this is meant to be run in a trusted network environment. There is *no* security.
+
+
+## Donate
+The best donation to **Tube Archivist** is your time, take a look at the [contribution page](CONTRIBUTING.md) to get started.  
+Second best way to support the development is to provide for caffeinated beverages:
+* [Paypal.me](https://paypal.me/bbilly1) for a one time coffee
+* [Paypal Subscription](https://www.paypal.com/webapps/billing/plans/subscribe?plan_id=P-03770005GR991451KMFGVPMQ) for a monthly coffee
+* [co-fi.com](https://ko-fi.com/bbilly1) for an alternative platform

deploy.sh

@@ -52,18 +52,80 @@ function sync_test {
 }
 
 
+# run same tests and checks as with github action but locally
+# takes filename to validate as optional argument
+function validate {
+
+    if [[ $1 ]]; then
+        check_path="$1"
+    else
+        check_path="."
+    fi
+
+    echo "run validate on $check_path"
+    
+    echo "running bandit"
+    bandit --recursive --skip B105,B108,B404,B603,B607 "$check_path"
+    echo "running black"
+    black --diff --color --check -l 79 "$check_path"
+    echo "running codespell"
+    codespell --skip="./.git" "$check_path"
+    echo "running flake8"
+    flake8 "$check_path" --count --max-complexity=12 --max-line-length=79 \
+        --show-source --statistics
+    echo "running isort"
+    isort --check-only --diff --profile black -l 79 "$check_path"
+    printf "    \n> all validations passed\n"
+
+}
+
+
 function sync_docker {
 
+    # check things
+    if [[ $(git branch --show-current) != 'master' ]]; then
+        echo 'you are not on master, dummy!'
+        return
+    fi
+
     if [[ $(systemctl is-active docker) != 'active' ]]; then
         echo "starting docker"
         sudo systemctl start docker
     fi
 
-    sudo docker build -t bbilly1/tubearchivist:latest .
+    echo "latest tags:"
+    git tag
+
+    echo "latest docker images:"
+    sudo docker image ls bbilly1/tubearchivist
+
+    printf "\ncreate new version:\n"
+    read -r VERSION
+
+    # start build
+    sudo docker build -t bbilly1/tubearchivist:latest -t bbilly1/tubearchivist:"$VERSION" .
+
+    printf "\nlatest images:\n"
+    sudo docker image ls bbilly1/tubearchivist
+
+    echo "continue?"
+    read -rn 1
+
+    # push to docker
+    echo "pushing latest:"
     sudo docker push bbilly1/tubearchivist:latest
+    echo "pushing $VERSION"
+    sudo docker push bbilly1/tubearchivist:"$VERSION"
+
+    # create release tag
+    echo "commits since last version:"
+    git log "$(git describe --tags --abbrev=0)"..HEAD --oneline
+    git tag -a "$VERSION" -m "new release version $VERSION"
+    git push all "$VERSION"
 
 }
 
+
 # check package versions in requirements.txt for updates
 python version_check.py
 
@@ -72,10 +134,12 @@ if [[ $1 == "blackhole" ]]; then
     sync_blackhole
 elif [[ $1 == "test" ]]; then
     sync_test
+elif [[ $1 == "validate" ]]; then
+    validate "$2"
 elif [[ $1 == "docker" ]]; then
     sync_docker
 else
-    echo "valid options are: blackhole | test | docker"
+    echo "valid options are: blackhole | test | validate | docker"
 fi
 
 

docker-compose.yml

@@ -29,7 +29,7 @@ services:
     depends_on:
       - archivist-es
   archivist-es:
-    image: docker.elastic.co/elasticsearch/elasticsearch:7.14.1
+    image: docker.elastic.co/elasticsearch/elasticsearch:7.15.0
     container_name: archivist-es
     restart: always
     environment:

docs/Channels.md

@@ -0,0 +1,23 @@
+# Channels Overview and Channel Detail Page
+
+The channels are organized on two different levels:
+
+## Channels Overview
+Accessible at `/channel/` of your Tube Archivist, the **Overview Page** shows a list of all channels you have indexed. 
+- You can filter that list to show or hide subscribed channels from the drop down menu. Clicking on the channel banner or the channel name will direct you to the *Channel Detail Page*.
+- If you are subscribed to a channel a *Unsubscribe* button will show.
+
+The **Subscribe to Channels** button <img src="assets/icon-add.png?raw=true" alt="add icon" width="20px" style="margin:0 5px;"> opens a text field to subscribe to a channel. You have a few options:
+- Enter the YouTube channel ID, a 25 character alphanumeric string. For example *UCBa659QWEk1AI4Tg--mrJ2A*
+- Enter the URL to the channel page on YouTube. For example *https://www.youtube.com/channel/UCBa659QWEk1AI4Tg--mrJ2A*
+- Enter the video URL for any video and let Tube Archivist extract the channel ID for you. For example *https://www.youtube.com/watch?v=2tdiKTSdE9Y*
+- Add one per line.
+- **Note**: Adding a link to a YouTube channel name is not yet supported, for example: *https://www.youtube.com/c/TomScottGo* will fail.
+
+The search icon <img src="assets/icon-search.png?raw=true" alt="search icon" width="20px" style="margin:0 5px;"> opens a text box to search for indexed channel names. Possible matches will show as you type. 
+
+## Channel Detail
+Each channel will get a dedicated channel detail page accessible at `/channel/<channel-id>/` of your Tube Archivist. This page shows all the videos you have downloaded from this channel plus additional metadata. 
+- If you are subscribed to the channel, an *Unsubscribe* button will show.
+- You can *Show* the channel description, that matches with the *About* tab on YouTube.
+- The **Mark as Watched** button will mark all videos of this channel as watched.

docs/Downloads.md

@@ -0,0 +1,35 @@
+# Downloads Page
+Accessible at `/downloads/` of your Tube Archivist, this page handles all the download functionality.
+
+
+## Rescan Subscriptions
+The **Rescan Subscriptions** icon <img src="assets/icon-rescan.png?raw=true" alt="rescan icon" width="20px" style="margin:0 5px;"> will start a background task to look for new videos from the channels you are subscribed to. You can define the channel page size on the [settings page](Settings#subscriptions). With the default channel page size, expect this process to take around 2-3 seconds for each channel you are subscribed to. A status message will show the progress.
+
+Then for every video found, **Tube Archivist** will skip the video if it has already been downloaded or if you added it to the *ignored* list before. All the other videos will get added to the download queue. Expect this to take around 1 second for each video as **Tube Archivist** needs to grab some additional metadata. New videos will get added at the bottom of the download queue.
+
+## Download Queue
+The **Start Download** icon <img src="assets/icon-download.png?raw=true" alt="download icon" width="20px" style="margin:0 5px;"> will start the download process starting from the top of the queue. Take a look at the relevant settings on the [Settings Page](Settings#downloads). Once the process started, a progress message will show with additional details and controls: 
+- The stop icon <img src="assets/icon-stop.png?raw=true" alt="stop icon" width="20px" style="margin:0 5px;"> will gracefully stop the download process, once the current video has been finished successfully.
+- The cancel icon <img src="assets/icon-close-red.png?raw=true" alt="close icon" width="20px" style="margin:0 5px;"> is equivalent to killing the process and will stop the download immediately. Any leftover files will get deleted, the canceled video will still be available in the download queue.
+
+## Add to Download Queue
+The **Add to Download Queue** icon <img src="assets/icon-add.png?raw=true" alt="add icon" width="20px" style="margin:0 5px;"> opens a text field to manually add videos to the download queue. You have a few options:
+- Add a link to a YouTube video. For example *https://www.youtube.com/watch?v=2tdiKTSdE9Y*.
+- Add a YouTube video ID. For example *2tdiKTSdE9Y*.
+- Add a link to a YouTube video by providing the shortened URL, for example *https://youtu.be/2tdiKTSdE9Y*.
+- Add a Channel ID or Channel URL to add every available video to the download queue. This will ignore the channel page size as described before and is meant for an initial download of the whole channel. You can still ignore selected videos before starting the download.
+- Add a playlist ID or URL to add every available video in the list to the download queue, for example *https://www.youtube.com/playlist?list=PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha* or *PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha*. Note that when you add a link to a video in a playlist, Tube Archivist assumes you want to download only the specific video and not the whole playlist, for example *https://www.youtube.com/watch?v=CINVwWHlzTY&list=PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha* will only add one video *CINVwWHlzTY* to the queue.
+- Add one link per line.
+- **Note**: Adding a link to a YouTube channel name is not yet supported, for example: *https://www.youtube.com/c/TomScottGo* will fail.
+
+## The Download Queue
+Below the three buttons you find the download queue. New items will get added at the bottom of the queue, the next video to download once you click on **Start Download** will be the first in the list.
+
+Every video in the download queue has two buttons:
+- **Ignore**: This will remove that video from the download queue and this video will not get added again, even when you **Rescan Subscriptions**.
+- **Download now**: This will give priority to this video. If the download process is already running, the prioritized video will get downloaded as soon as the current video is finished. If there is no download process running, this will start downloading this single video and stop after that.
+
+You can flip the view by activating **Show Only Ignored Videos**. This will show all videos you have previously *ignored*.  
+Every video in the ignored list has two buttons:
+- **Forget**: This will delete the item form the ignored list.
+- **Add to Queue**: This will add the ignored video back to the download queue.

docs/Home.md

@@ -0,0 +1,29 @@
+# Tube Archivist Wiki
+
+**WIP**: This is work in progress!
+
+Welcome to the official Tube Archivist Wiki. This is an up-to-date documentation of user functionality.
+
+Table of contents:
+* [Main](Main): Tube Archivist landing page
+* [Channels](Channels): Browse your channels, handle subscriptions
+* [Downloads](Downloads): Scanning subscriptions, handle download queue
+* [Settings](Settings): All the configuration options
+
+## Getting Started
+1. [Subscribe](Channels#channels-overview) to some of your favourite YouTube channels.
+2. [Scan](Downloads#rescan-subscriptions) subscriptions to add the latest videos to the download queue.
+3. [Add](Downloads#add-to-download-queue) additional videos, channels or playlist - ignore the ones you don't want to download.
+4. [Download](Downloads#download-queue) and let **Tube Archivist** do it's thing.
+5. Sit back and enjoy your archived and indexed collection!
+
+## General Navigation
+* Clicking on the channel name or the channel icon brings you to the dedicated channel page to show videos from that channel.
+* Clicking on a video title brings you to the dedicated video page and shows additional details.
+* Clicking on a video thumbnail opens the video player and starts streaming the selected video.
+* Hover over the playing video to show additional control options.
+
+
+An empty checkbox icon <img src="assets/icon-unseen.png?raw=true" alt="unseen icon" width="20px" style="margin:0 5px;"> will show for videos you haven't marked as watched. Click on it and the icon will change to a filled checkbox <img src="assets/icon-seen.png?raw=true" alt="seen icon" width="20px" style="margin:0 5px;"> indicating it as watched.
+
+When available the <img src="assets/icon-gridview.png?raw=true" alt="gridview icon" width="20px" style="margin:0 5px;"> gridview icon will display the list in a grid, the <img src="assets/icon-listview.png?raw=true" alt="listview icon" width="20px" style="margin:0 5px;"> listview icon will arrange the items in a list.

docs/Main.md

@@ -0,0 +1,10 @@
+# Tube Archivist Home Page Functionality
+
+This is the landing page, when you first open **Tube Archivist**. You have a few options to sort and filter that view:
+- With the **Sort Order** you can select how the "Recent Videos" are sorted: 
+  - **Date Published**: Sorts the list by date when the video was published on YouTube, newest on top.
+  - **Date Downloaded**: Sorts the list based on when you have downloaded the video to your archive, newest on top.
+- With **Hide Watched** you can filter out videos you have already marked as watched to only show unwatched videos. 
+- You can use those two options together to for example filter the list to *Hide Watched* videos **and** sort by date downloaded.
+
+Additionally the search icon <img src="assets/icon-search.png?raw=true" alt="search icon" width="20px" style="margin:0 5px;"> opens a text field to search your collection.

docs/Settings.md

@@ -0,0 +1,60 @@
+# Settings Page
+Accessible at `/settings/` of your **Tube Archivist**, this page holds all the configurations and additional functionality related to the database.
+
+Click on **Update Settings** at the bottom of the form to apply your configurations.
+
+## Color scheme
+Switch between the easy on the eyes dark theme and the burning bright theme.
+
+## Archive View
+- **Page Size**: Defines how many results get displayed on a given page. Same value goes for all archive views.
+
+## Subscriptions
+Settings related to the channel management.
+- **Channel Page Size**: Defines how many pages will get analyzed by **Tube Archivist** each time you click on *Rescan Subscriptions*. The default page size used by yt-dlp is **50**, that's also the recommended value to set here. Any value higher will slow down the rescan process, for example if you set the value to 51, that means yt-dlp will have to go through 2 pages of results instead of 1 and by that doubling the time that process takes.
+
+## Downloads
+Settings related to the download process.
+- **Download Limit**: Stop the download process after downloading the set quantity of videos.
+- **Download Speed Limit**: Set your download speed limit in KB/s. This will pass the option `--limit-rate` to yt-dlp.
+- **Sleep Interval**: Time in seconds to sleep between requests to YouTube. It's a good idea to set this to **3** seconds. Might be necessary to avoid throttling.
+
+## Download Format
+Additional settings passed to yt-dlp.
+- **Format**: This controls which streams get downloaded and is equivalent to passing `--format` to yt-dlp. Use one of the recommended one or look at the documentation of [yt-dlp](https://github.com/yt-dlp/yt-dlp#format-selection). Please note: The option `--merge-output-format mp4` is automatically passed to yt-dlp to guarantee browser compatibility.
+- **Embed Metadata**: This saves the available tags directly into the media file by passing `--embed-metadata` to yt-dlp.
+
+
+# Actions
+Additional database functionality.
+
+## Manual Media Files Import
+So far this depends on the video you are trying to import to be still available on YouTube to get the metadata. Add the files you like to import to the */cache/import* folder. Then start the process from the settings page *Manual Media Files Import*. Make sure to follow one of the two methods below.
+
+### Method 1:
+Add a matching *.json* file with the media file. Both files need to have the same base name, for example:
+- For the media file: \<base-name>.mp4
+- For the JSON file: \<base-name>.info.json
+- Alternate JSON file: \<base-name>.json
+
+**Tube Archivist** then looks for the 'id' key within the JSON file to identify the video.
+
+### Method 2:
+Detect the YouTube ID from filename, this accepts the default yt-dlp naming convention for file names like:
+- \<base-name>[\<youtube-id>].mp4
+- The YouTube ID in square brackets at the end of the filename is the crucial part.
+
+### Some notes:
+- This will **consume** the files you put into the import folder: Files will get converted to mp4 if needed (this might take a long time...) and moved to the archive, *.json* files will get deleted upon completion to avoid having duplicates on the next run.
+- Maybe start with a subset of your files to import to make sure everything goes well...
+- Follow the logs to monitor progress and errors: `docker-compose logs -f tubearchivist`.
+
+## Backup Database
+This will backup your metadata into a zip file. The file will get stored at *cache/backup* and will contain the necessary files to restore the Elasticsearch index formatted **nd-json** files plus a complete export of the index in a set of conventional **json** files.  
+
+BE AWARE: This will **not** backup any media files, just the metadata from the Elasticsearch.
+
+## Restore From Backup
+The restore functionality will expect the same zip file in *cache/backup* as created from the **Backup database** function. This will recreate the index from the snapshot. If there are multiple backup files in the folder, the newest one will take priority. 
+
+BE AWARE: This will **replace** your current index with the one from the backup file. This won't restore any media files.

run.sh

@@ -14,7 +14,7 @@ until curl "$ES_URL" -fs; do
 done
 
 python manage.py migrate
-python manage.py collectstatic
+python manage.py collectstatic --noinput -c
 nginx &
 celery -A home.tasks worker --loglevel=INFO &
 uwsgi --ini uwsgi.ini

tubearchivist/config/asgi.py

@@ -11,6 +11,6 @@ import os
 
 from django.core.asgi import get_asgi_application
 
-os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings')
+os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings")
 
 application = get_asgi_application()

tubearchivist/config/settings.py

@@ -10,8 +10,8 @@ For the full list of settings and their values, see
 https://docs.djangoproject.com/en/3.2/ref/settings/
 """
 
-from pathlib import Path
 from os import environ
+from pathlib import Path
 
 # Build paths inside the project like this: BASE_DIR / 'subdir'.
 BASE_DIR = Path(__file__).resolve().parent.parent
@@ -21,67 +21,67 @@ BASE_DIR = Path(__file__).resolve().parent.parent
 # See https://docs.djangoproject.com/en/3.2/howto/deployment/checklist/
 
 # SECURITY WARNING: keep the secret key used in production secret!
-SECRET_KEY = 'Fvid^aUL6LohRZz*kZFvq85B&JW&kB9o*#jdzWsdWE8*XkCLR8'
+SECRET_KEY = "Fvid^aUL6LohRZz*kZFvq85B&JW&kB9o*#jdzWsdWE8*XkCLR8"
 
 # SECURITY WARNING: don't run with debug turned on in production!
-DEBUG = bool(environ.get('DJANGO_DEBUG'))
+DEBUG = bool(environ.get("DJANGO_DEBUG"))
 
-ALLOWED_HOSTS = ['*']
+ALLOWED_HOSTS = ["*"]
 
 
 # Application definition
 
 INSTALLED_APPS = [
-    'home.apps.HomeConfig',
+    "home.apps.HomeConfig",
-    'django.contrib.admin',
+    "django.contrib.admin",
-    'django.contrib.auth',
+    "django.contrib.auth",
-    'django.contrib.contenttypes',
+    "django.contrib.contenttypes",
-    'django.contrib.sessions',
+    "django.contrib.sessions",
-    'django.contrib.messages',
+    "django.contrib.messages",
-    'whitenoise.runserver_nostatic',
+    "whitenoise.runserver_nostatic",
-    'django.contrib.staticfiles',
+    "django.contrib.staticfiles",
-    'django.contrib.humanize'
+    "django.contrib.humanize",
 ]
 
 MIDDLEWARE = [
-    'django.middleware.security.SecurityMiddleware',
+    "django.middleware.security.SecurityMiddleware",
-    'django.contrib.sessions.middleware.SessionMiddleware',
+    "django.contrib.sessions.middleware.SessionMiddleware",
-    'whitenoise.middleware.WhiteNoiseMiddleware',
+    "whitenoise.middleware.WhiteNoiseMiddleware",
-    'django.middleware.common.CommonMiddleware',
+    "django.middleware.common.CommonMiddleware",
-    'django.middleware.csrf.CsrfViewMiddleware',
+    "django.middleware.csrf.CsrfViewMiddleware",
-    'django.contrib.auth.middleware.AuthenticationMiddleware',
+    "django.contrib.auth.middleware.AuthenticationMiddleware",
-    'django.contrib.messages.middleware.MessageMiddleware',
+    "django.contrib.messages.middleware.MessageMiddleware",
-    'django.middleware.clickjacking.XFrameOptionsMiddleware',
+    "django.middleware.clickjacking.XFrameOptionsMiddleware",
 ]
 
-ROOT_URLCONF = 'config.urls'
+ROOT_URLCONF = "config.urls"
 
 TEMPLATES = [
     {
-        'BACKEND': 'django.template.backends.django.DjangoTemplates',
+        "BACKEND": "django.template.backends.django.DjangoTemplates",
-        'DIRS': [],
+        "DIRS": [],
-        'APP_DIRS': True,
+        "APP_DIRS": True,
-        'OPTIONS': {
+        "OPTIONS": {
-            'context_processors': [
+            "context_processors": [
-                'django.template.context_processors.debug',
+                "django.template.context_processors.debug",
-                'django.template.context_processors.request',
+                "django.template.context_processors.request",
-                'django.contrib.auth.context_processors.auth',
+                "django.contrib.auth.context_processors.auth",
-                'django.contrib.messages.context_processors.messages',
+                "django.contrib.messages.context_processors.messages",
             ],
         },
     },
 ]
 
-WSGI_APPLICATION = 'config.wsgi.application'
+WSGI_APPLICATION = "config.wsgi.application"
 
 
 # Database
 # https://docs.djangoproject.com/en/3.2/ref/settings/#databases
 
 DATABASES = {
-    'default': {
+    "default": {
-        'ENGINE': 'django.db.backends.sqlite3',
+        "ENGINE": "django.db.backends.sqlite3",
-        'NAME': BASE_DIR / 'db.sqlite3',
+        "NAME": BASE_DIR / "db.sqlite3",
     }
 }
 
@@ -91,16 +91,16 @@ DATABASES = {
 
 AUTH_PASSWORD_VALIDATORS = [
     {
-        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
+        "NAME": "django.contrib.auth.password_validation.UserAttributeSimilarityValidator",  # noqa: E501
     },
     {
-        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
+        "NAME": "django.contrib.auth.password_validation.MinimumLengthValidator",  # noqa: E501
     },
     {
-        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
+        "NAME": "django.contrib.auth.password_validation.CommonPasswordValidator",  # noqa: E501
     },
     {
-        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
+        "NAME": "django.contrib.auth.password_validation.NumericPasswordValidator",  # noqa: E501
     },
 ]
 
@@ -108,9 +108,9 @@ AUTH_PASSWORD_VALIDATORS = [
 # Internationalization
 # https://docs.djangoproject.com/en/3.2/topics/i18n/
 
-LANGUAGE_CODE = 'en-us'
+LANGUAGE_CODE = "en-us"
 
-TIME_ZONE = 'UTC'
+TIME_ZONE = "UTC"
 
 USE_I18N = True
 
@@ -122,7 +122,7 @@ USE_TZ = True
 # Static files (CSS, JavaScript, Images)
 # https://docs.djangoproject.com/en/3.2/howto/static-files/
 
-STATIC_URL = '/static/'
+STATIC_URL = "/static/"
 
 # STATICFILES_DIRS = [
 #     str(BASE_DIR.joinpath('static')),
@@ -130,15 +130,15 @@ STATIC_URL = '/static/'
 # ]
 
 # STATIC_URL = '/static/'
-STATICFILES_DIRS = (str(BASE_DIR.joinpath('static')),)
+STATICFILES_DIRS = (str(BASE_DIR.joinpath("static")),)
 # MEDIA_ROOT = str(BASE_DIR.joinpath('media'))
 # MEDIA_URL = '/media/'
 
-STATIC_ROOT = str(BASE_DIR.joinpath('staticfiles'))
+STATIC_ROOT = str(BASE_DIR.joinpath("staticfiles"))
 
-STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'
+STATICFILES_STORAGE = "whitenoise.storage.CompressedManifestStaticFilesStorage"
 
 # Default primary key field type
 # https://docs.djangoproject.com/en/3.2/ref/settings/#default-auto-field
 
-DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'
+DEFAULT_AUTO_FIELD = "django.db.models.BigAutoField"

tubearchivist/config/urls.py

@@ -14,9 +14,9 @@ Including another URLconf
     2. Add a URL to urlpatterns:  path('blog/', include('blog.urls'))
 """
 from django.contrib import admin
-from django.urls import path, include
+from django.urls import include, path
 
 urlpatterns = [
-    path('', include('home.urls')),
+    path("", include("home.urls")),
-    path('admin/', admin.site.urls),
+    path("admin/", admin.site.urls),
 ]

tubearchivist/config/wsgi.py

@@ -11,6 +11,6 @@ import os
 
 from django.core.wsgi import get_wsgi_application
 
-os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings')
+os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings")
 
 application = get_wsgi_application()

tubearchivist/home/__init__.py

@@ -1,40 +1,5 @@
-""" handle startup """
+""" handle celery startup """
-
-import os
-
-from home.src.config import AppConfig
-from home.src.helper import set_message
-from home.src.index_management import index_check
 
 from .tasks import app as celery_app
 
-
+__all__ = ("celery_app",)
-def sync_redis_state():
-    """ make sure redis gets the config.json values """
-    print('sync redis')
-    config = AppConfig().config
-    sort_order = config['archive']['sort']
-    set_message('sort_order', sort_order, expire=False)
-    hide_watched = bool(int(config['archive']['hide_watched']))
-    set_message('hide_watched', hide_watched, expire=False)
-    show_subed_only = bool(int(config['archive']['show_subed_only']))
-    set_message('show_subed_only', show_subed_only, expire=False)
-
-
-def make_folders():
-    """ make needed folders here to avoid letting docker messing it up """
-    folders = ['download', 'channels', 'videos', 'import']
-    config = AppConfig().config
-    cache_dir = config['application']['cache_dir']
-    for folder in folders:
-        folder_path = os.path.join(cache_dir, folder)
-        try:
-            os.makedirs(folder_path)
-        except FileExistsError:
-            continue
-
-
-__all__ = ('celery_app',)
-make_folders()
-sync_redis_state()
-index_check()

tubearchivist/home/admin.py

@@ -1,3 +1,3 @@
-from django.contrib import admin
+from django.contrib import admin  # noqa: F401 - Unused import
 
 # Register your models here.

tubearchivist/home/apps.py

@@ -1,6 +1,42 @@
+"""handle custom startup functions"""
+
+import os
+
 from django.apps import AppConfig
+from home.src.config import AppConfig as ArchivistConfig
+from home.src.helper import RedisArchivist
+from home.src.index_management import index_check
+
+
+def make_folders():
+    """make needed cache folders here so docker doesn't mess it up"""
+    folders = ["download", "channels", "videos", "import", "backup"]
+    config = ArchivistConfig().config
+    cache_dir = config["application"]["cache_dir"]
+    for folder in folders:
+        folder_path = os.path.join(cache_dir, folder)
+        try:
+            os.makedirs(folder_path)
+        except FileExistsError:
+            continue
+
+
+def release_lock():
+    """make sure there are no leftover locks set in redis on container start"""
+    all_locks = ["manual_import", "downloading", "dl_queue", "dl_queue_id"]
+    for lock in all_locks:
+        response = RedisArchivist().del_message(lock)
+        if response:
+            print("deleted leftover key from redis: " + lock)
 
 
 class HomeConfig(AppConfig):
-    default_auto_field = 'django.db.models.BigAutoField'
+    """call startup funcs"""
-    name = 'home'
+
+    default_auto_field = "django.db.models.BigAutoField"
+    name = "home"
+
+    def ready(self):
+        release_lock()
+        index_check()
+        make_folders()

tubearchivist/home/config.json

@@ -1,20 +1,24 @@
 {
     "archive": {
         "sort": "published",
-        "hide_watched": false,
-        "show_subed_only": false,
         "page_size": 12
     },
+    "default_view": {
+        "home": "grid",
+        "channel": "list",
+        "downloads": "list"
+    },
     "subscriptions": {
         "auto_search": false,
         "auto_download": false,
         "channel_size": 50
     },
     "downloads": {
-        "limit_count": 5,
+        "limit_count": false,
         "limit_speed": false,
         "sleep_interval": 3,
-        "format": false
+        "format": false,
+        "add_metadata": false
     },
     "application": {
         "cache_dir": "/cache",
@@ -22,4 +26,4 @@
         "file_template": "%(id)s_%(title)s.mp4",
         "colors": "dark"
     }
-}
+}

tubearchivist/home/models.py

@@ -1,3 +1,3 @@
-from django.db import models
+from django.db import models  # noqa: F401 - Unused import
 
 # Create your models here.

tubearchivist/home/src/config.py

@@ -8,53 +8,96 @@ Functionality:
 import json
 import os
 
-from home.src.helper import get_message, set_message
+from home.src.helper import RedisArchivist
 
 
 class AppConfig:
-    """ handle user settings and application variables """
+    """handle user settings and application variables"""
 
     def __init__(self):
         self.config = self.get_config()
 
     def get_config(self):
-        """ get config from default file or redis if changed """
+        """get config from default file or redis if changed"""
         config = self.get_config_redis()
         if not config:
-            with open('home/config.json', 'r', encoding="utf-8") as f:
+            config = self.get_config_file()
-                config_str = f.read()
-                config = json.loads(config_str)
 
-        config['application']['REDIS_HOST'] = os.environ.get('REDIS_HOST')
+        config["application"].update(self.get_config_env())
-        config['application']['es_url'] = os.environ.get('ES_URL')
-        config['application']['HOST_UID'] = int(os.environ.get('HOST_UID'))
-        config['application']['HOST_GID'] = int(os.environ.get('HOST_GID'))
         return config
 
+    def get_config_file(self):
+        """read the defaults from config.json"""
+        with open("home/config.json", "r", encoding="utf-8") as f:
+            config_str = f.read()
+            config_file = json.loads(config_str)
+
+        config_file["application"].update(self.get_config_env())
+
+        return config_file
+
+    @staticmethod
+    def get_config_env():
+        """read environment application variables"""
+        application = {
+            "REDIS_HOST": os.environ.get("REDIS_HOST"),
+            "es_url": os.environ.get("ES_URL"),
+            "HOST_UID": int(os.environ.get("HOST_UID")),
+            "HOST_GID": int(os.environ.get("HOST_GID")),
+        }
+
+        return application
+
     @staticmethod
     def get_config_redis():
-        """ read config json set from redis to overwrite defaults """
+        """read config json set from redis to overwrite defaults"""
-        config = get_message('config')
+        config = RedisArchivist().get_message("config")
         if not list(config.values())[0]:
             return False
 
         return config
 
     def update_config(self, form_post):
-        """ update config values from settings form """
+        """update config values from settings form"""
         config = self.config
         for key, value in form_post.items():
             to_write = value[0]
             if len(to_write):
-                if to_write == '0':
+                if to_write == "0":
                     to_write = False
+                elif to_write == "1":
+                    to_write = True
                 elif to_write.isdigit():
                     to_write = int(to_write)
 
-                config_dict, config_value = key.split('.')
+                config_dict, config_value = key.split(".")
                 config[config_dict][config_value] = to_write
 
-        with open('home/config.json', 'w', encoding="utf-8") as f:
+        RedisArchivist().set_message("config", config, expire=False)
-            f.write(json.dumps(config))
 
-        set_message('config', config, expire=False)
+    def load_new_defaults(self):
+        """check config.json for missing defaults"""
+        default_config = self.get_config_file()
+        redis_config = self.get_config_redis()
+
+        # check for customizations
+        if not redis_config:
+            return
+
+        needs_update = False
+
+        for key, value in default_config.items():
+            # missing whole main key
+            if key not in redis_config:
+                redis_config.update({key: value})
+                needs_update = True
+                continue
+
+            # missing nested values
+            for sub_key, sub_value in value.items():
+                if sub_key not in redis_config[key].keys():
+                    redis_config[key].update({sub_key: sub_value})
+                    needs_update = True
+
+        if needs_update:
+            RedisArchivist().set_message("config", redis_config, expire=False)

tubearchivist/home/src/download.py

@@ -6,30 +6,34 @@ Functionality:
 """
 
 import json
-import shutil
 import os
-
+import shutil
 from datetime import datetime
 from time import sleep
 
 import requests
 import yt_dlp as youtube_dl
-
-from home.src.index import YoutubeChannel, index_new_video
 from home.src.config import AppConfig
-from home.src.helper import clean_string, DurationConverter, set_message
+from home.src.helper import (
+    DurationConverter,
+    RedisArchivist,
+    RedisQueue,
+    clean_string,
+    ignore_filelist,
+)
+from home.src.index import YoutubeChannel, index_new_video
 
 
 class PendingList:
-    """ manage the pending videos list """
+    """manage the pending videos list"""
 
     CONFIG = AppConfig().config
-    ES_URL = CONFIG['application']['es_url']
+    ES_URL = CONFIG["application"]["es_url"]
-    VIDEOS = CONFIG['application']['videos']
+    VIDEOS = CONFIG["application"]["videos"]
 
     @staticmethod
     def parse_url_list(youtube_ids):
-        """ extract youtube ids from list """
+        """extract youtube ids from list"""
         missing_videos = []
         for entry in youtube_ids:
             # notify
@@ -37,31 +41,31 @@ class PendingList:
                 "status": "pending",
                 "level": "info",
                 "title": "Adding to download queue.",
-                "message": 'Extracting lists'
+                "message": "Extracting lists",
             }
-            set_message('progress:download', mess_dict)
+            RedisArchivist().set_message("progress:download", mess_dict)
             # extract
-            url = entry['url']
+            url = entry["url"]
-            url_type = entry['type']
+            url_type = entry["type"]
-            if url_type == 'video':
+            if url_type == "video":
                 missing_videos.append(url)
-            elif url_type == 'channel':
+            elif url_type == "channel":
                 youtube_ids = ChannelSubscription().get_last_youtube_videos(
                     url, limit=False
                 )
                 missing_videos = missing_videos + youtube_ids
-            elif url_type == 'playlist':
+            elif url_type == "playlist":
                 youtube_ids = playlist_extractor(url)
                 missing_videos = missing_videos + youtube_ids
 
         return missing_videos
 
     def add_to_pending(self, missing_videos):
-        """ build the bulk json data from pending """
+        """build the bulk json data from pending"""
         # check if channel is indexed
         channel_handler = ChannelSubscription()
         all_indexed = channel_handler.get_channels(subscribed_only=False)
-        all_channel_ids = [i['channel_id'] for i in all_indexed]
+        all_channel_ids = [i["channel_id"] for i in all_indexed]
         # check if already there
         all_downloaded = self.get_all_downloaded()
         # loop
@@ -72,18 +76,18 @@ class PendingList:
             elif isinstance(video, tuple):
                 youtube_id = video[0]
             if youtube_id in all_downloaded:
-                # skip already downlaoded
+                # skip already downloaded
                 continue
             video = self.get_youtube_details(youtube_id)
             # skip on download error
             if not video:
                 continue
 
-            if video['channel_id'] in all_channel_ids:
+            if video["channel_id"] in all_channel_ids:
-                video['channel_indexed'] = True
+                video["channel_indexed"] = True
             else:
-                video['channel_indexed'] = False
+                video["channel_indexed"] = False
-            video['status'] = "pending"
+            video["status"] = "pending"
             action = {"create": {"_id": youtube_id, "_index": "ta_download"}}
             bulk_list.append(json.dumps(action))
             bulk_list.append(json.dumps(video))
@@ -92,258 +96,268 @@ class PendingList:
                 "status": "pending",
                 "level": "info",
                 "title": "Adding to download queue.",
-                "message": 'Processing IDs...'
+                "message": "Processing IDs...",
             }
-            set_message('progress:download', mess_dict)
+            RedisArchivist().set_message("progress:download", mess_dict)
         # add last newline
-        bulk_list.append('\n')
+        bulk_list.append("\n")
-        query_str = '\n'.join(bulk_list)
+        query_str = "\n".join(bulk_list)
-        headers = {'Content-type': 'application/x-ndjson'}
+        headers = {"Content-type": "application/x-ndjson"}
-        url = self.ES_URL + '/_bulk'
+        url = self.ES_URL + "/_bulk"
         request = requests.post(url, data=query_str, headers=headers)
         if not request.ok:
             print(request)
 
     @staticmethod
     def get_youtube_details(youtube_id):
-        """ get details from youtubedl for single pending video """
+        """get details from youtubedl for single pending video"""
         obs = {
-            'default_search': 'ytsearch',
+            "default_search": "ytsearch",
-            'quiet': True,
+            "quiet": True,
-            'skip_download': True,
+            "skip_download": True,
         }
         try:
             vid = youtube_dl.YoutubeDL(obs).extract_info(youtube_id)
         except youtube_dl.utils.DownloadError:
-            print('failed to extract info for: ' + youtube_id)
+            print("failed to extract info for: " + youtube_id)
             return False
         # parse response
-        seconds = vid['duration']
+        seconds = vid["duration"]
         duration_str = DurationConverter.get_str(seconds)
-        upload_date = vid['upload_date']
+        upload_date = vid["upload_date"]
         upload_dt = datetime.strptime(upload_date, "%Y%m%d")
         published = upload_dt.strftime("%Y-%m-%d")
         # build dict
         youtube_details = {
             "youtube_id": youtube_id,
-            "channel_name": vid['channel'],
+            "channel_name": vid["channel"],
-            "vid_thumb_url": vid['thumbnail'],
+            "vid_thumb_url": vid["thumbnail"],
-            "title": vid['title'],
+            "title": vid["title"],
-            "channel_id": vid['channel_id'],
+            "channel_id": vid["channel_id"],
             "duration": duration_str,
             "published": published,
-            "timestamp": int(datetime.now().strftime("%s"))
+            "timestamp": int(datetime.now().strftime("%s")),
         }
         return youtube_details
 
     def get_all_pending(self):
-        """ get a list of all pending videos in ta_download """
+        """get a list of all pending videos in ta_download"""
-        headers = {'Content-type': 'application/json'}
+        headers = {"Content-type": "application/json"}
         # get PIT ID
-        url = self.ES_URL + '/ta_download/_pit?keep_alive=1m'
+        url = self.ES_URL + "/ta_download/_pit?keep_alive=1m"
         response = requests.post(url)
         json_data = json.loads(response.text)
-        pit_id = json_data['id']
+        pit_id = json_data["id"]
         # query
         data = {
-            "size": 50, "query": {"match_all": {}},
+            "size": 50,
+            "query": {"match_all": {}},
             "pit": {"id": pit_id, "keep_alive": "1m"},
-            "sort": [{"timestamp": {"order": "desc"}}]
+            "sort": [{"timestamp": {"order": "asc"}}],
         }
         query_str = json.dumps(data)
-        url = self.ES_URL + '/_search'
+        url = self.ES_URL + "/_search"
         all_pending = []
         all_ignore = []
         while True:
             response = requests.get(url, data=query_str, headers=headers)
             json_data = json.loads(response.text)
-            all_hits = json_data['hits']['hits']
+            all_hits = json_data["hits"]["hits"]
             if all_hits:
                 for hit in all_hits:
-                    youtube_id = hit['_source']['youtube_id']
+                    youtube_id = hit["_source"]["youtube_id"]
-                    status = hit['_source']['status']
+                    status = hit["_source"]["status"]
-                    if status == 'pending':
+                    if status == "pending":
-                        all_pending.append(hit['_source'])
+                        all_pending.append(hit["_source"])
-                    elif status == 'ignore':
+                    elif status == "ignore":
                         all_ignore.append(youtube_id)
-                    search_after = hit['sort']
+                    search_after = hit["sort"]
                 # update search_after with last hit data
-                data['search_after'] = search_after
+                data["search_after"] = search_after
                 query_str = json.dumps(data)
             else:
                 break
         # clean up PIT
         query_str = json.dumps({"id": pit_id})
-        requests.delete(self.ES_URL + '/_pit', data=query_str, headers=headers)
+        requests.delete(self.ES_URL + "/_pit", data=query_str, headers=headers)
         return all_pending, all_ignore
 
     def get_all_indexed(self):
-        """ get a list of all videos indexed """
+        """get a list of all videos indexed"""
-        headers = {'Content-type': 'application/json'}
+        headers = {"Content-type": "application/json"}
         # get PIT ID
-        url = self.ES_URL + '/ta_video/_pit?keep_alive=1m'
+        url = self.ES_URL + "/ta_video/_pit?keep_alive=1m"
         response = requests.post(url)
         json_data = json.loads(response.text)
-        pit_id = json_data['id']
+        pit_id = json_data["id"]
         # query
         data = {
-            "size": 500, "query": {"match_all": {}},
+            "size": 500,
+            "query": {"match_all": {}},
             "pit": {"id": pit_id, "keep_alive": "1m"},
-            "sort": [{"published": {"order": "desc"}}]
+            "sort": [{"published": {"order": "desc"}}],
         }
         query_str = json.dumps(data)
-        url = self.ES_URL + '/_search'
+        url = self.ES_URL + "/_search"
         all_indexed = []
         while True:
             response = requests.get(url, data=query_str, headers=headers)
             json_data = json.loads(response.text)
-            all_hits = json_data['hits']['hits']
+            all_hits = json_data["hits"]["hits"]
             if all_hits:
                 for hit in all_hits:
                     all_indexed.append(hit)
-                    search_after = hit['sort']
+                    search_after = hit["sort"]
                 # update search_after with last hit data
-                data['search_after'] = search_after
+                data["search_after"] = search_after
                 query_str = json.dumps(data)
             else:
                 break
         # clean up PIT
         query_str = json.dumps({"id": pit_id})
-        requests.delete(self.ES_URL + '/_pit', data=query_str, headers=headers)
+        requests.delete(self.ES_URL + "/_pit", data=query_str, headers=headers)
         return all_indexed
 
     def get_all_downloaded(self):
-        """ get a list of all videos in archive """
+        """get a list of all videos in archive"""
-        all_channel_folders = os.listdir(self.VIDEOS)
+        channel_folders = os.listdir(self.VIDEOS)
+        all_channel_folders = ignore_filelist(channel_folders)
         all_downloaded = []
         for channel_folder in all_channel_folders:
             channel_path = os.path.join(self.VIDEOS, channel_folder)
-            all_videos = os.listdir(channel_path)
+            videos = os.listdir(channel_path)
+            all_videos = ignore_filelist(videos)
             youtube_vids = [i[9:20] for i in all_videos]
             for youtube_id in youtube_vids:
                 all_downloaded.append(youtube_id)
         return all_downloaded
 
     def delete_from_pending(self, youtube_id):
-        """ delete the youtube_id from ta_download """
+        """delete the youtube_id from ta_download"""
-        url = f'{self.ES_URL}/ta_download/_doc/{youtube_id}'
+        url = f"{self.ES_URL}/ta_download/_doc/{youtube_id}"
         response = requests.delete(url)
         if not response.ok:
             print(response.text)
 
     def ignore_from_pending(self, ignore_list):
-        """ build the bulk query string """
+        """build the bulk query string"""
 
         stamp = int(datetime.now().strftime("%s"))
         bulk_list = []
 
         for youtube_id in ignore_list:
             action = {"update": {"_id": youtube_id, "_index": "ta_download"}}
-            source = {"doc": {"status": 'ignore', "timestamp": stamp}}
+            source = {"doc": {"status": "ignore", "timestamp": stamp}}
             bulk_list.append(json.dumps(action))
             bulk_list.append(json.dumps(source))
 
         # add last newline
-        bulk_list.append('\n')
+        bulk_list.append("\n")
-        query_str = '\n'.join(bulk_list)
+        query_str = "\n".join(bulk_list)
 
-        headers = {'Content-type': 'application/x-ndjson'}
+        headers = {"Content-type": "application/x-ndjson"}
-        url = self.ES_URL + '/_bulk'
+        url = self.ES_URL + "/_bulk"
         request = requests.post(url, data=query_str, headers=headers)
         mess_dict = {
             "status": "ignore",
             "level": "info",
             "title": "Added to ignore list",
-            "message": ''
+            "message": "",
         }
-        set_message('progress:download', mess_dict)
+        RedisArchivist().set_message("progress:download", mess_dict)
         if not request.ok:
             print(request)
 
 
 class ChannelSubscription:
-    """ manage the list of channels subscribed """
+    """manage the list of channels subscribed"""
 
     def __init__(self):
         config = AppConfig().config
-        self.es_url = config['application']['es_url']
+        self.es_url = config["application"]["es_url"]
-        self.channel_size = config['subscriptions']['channel_size']
+        self.channel_size = config["subscriptions"]["channel_size"]
 
     def get_channels(self, subscribed_only=True):
-        """ get a list of all channels subscribed to """
+        """get a list of all channels subscribed to"""
-        headers = {'Content-type': 'application/json'}
+        headers = {"Content-type": "application/json"}
         # get PIT ID
-        url = self.es_url + '/ta_channel/_pit?keep_alive=1m'
+        url = self.es_url + "/ta_channel/_pit?keep_alive=1m"
         response = requests.post(url)
         json_data = json.loads(response.text)
-        pit_id = json_data['id']
+        pit_id = json_data["id"]
         # query
         if subscribed_only:
             data = {
                 "query": {"term": {"channel_subscribed": {"value": True}}},
-                "size": 50, "pit": {"id": pit_id, "keep_alive": "1m"},
+                "size": 50,
-                "sort": [{"channel_name.keyword": {"order": "asc"}}]
+                "pit": {"id": pit_id, "keep_alive": "1m"},
+                "sort": [{"channel_name.keyword": {"order": "asc"}}],
             }
         else:
             data = {
                 "query": {"match_all": {}},
-                "size": 50, "pit": {"id": pit_id, "keep_alive": "1m"},
+                "size": 50,
-                "sort": [{"channel_name.keyword": {"order": "asc"}}]
+                "pit": {"id": pit_id, "keep_alive": "1m"},
+                "sort": [{"channel_name.keyword": {"order": "asc"}}],
             }
         query_str = json.dumps(data)
-        url = self.es_url + '/_search'
+        url = self.es_url + "/_search"
         all_channels = []
         while True:
             response = requests.get(url, data=query_str, headers=headers)
             json_data = json.loads(response.text)
-            all_hits = json_data['hits']['hits']
+            all_hits = json_data["hits"]["hits"]
             if all_hits:
                 for hit in all_hits:
-                    source = hit['_source']
+                    source = hit["_source"]
-                    search_after = hit['sort']
+                    search_after = hit["sort"]
                     all_channels.append(source)
                 # update search_after with last hit data
-                data['search_after'] = search_after
+                data["search_after"] = search_after
                 query_str = json.dumps(data)
             else:
                 break
         # clean up PIT
         query_str = json.dumps({"id": pit_id})
-        requests.delete(self.es_url + '/_pit', data=query_str, headers=headers)
+        requests.delete(self.es_url + "/_pit", data=query_str, headers=headers)
         return all_channels
 
     def get_last_youtube_videos(self, channel_id, limit=True):
-        """ get a list of last videos from channel """
+        """get a list of last videos from channel"""
-        url = f'https://www.youtube.com/channel/{channel_id}/videos'
+        url = f"https://www.youtube.com/channel/{channel_id}/videos"
         obs = {
-            'default_search': 'ytsearch', 'quiet': True,
+            "default_search": "ytsearch",
-            'skip_download': True, 'extract_flat': True
+            "quiet": True,
+            "skip_download": True,
+            "extract_flat": True,
         }
         if limit:
-            obs['playlistend'] = self.channel_size
+            obs["playlistend"] = self.channel_size
         chan = youtube_dl.YoutubeDL(obs).extract_info(url, download=False)
-        last_videos = [(i['id'], i['title']) for i in chan['entries']]
+        last_videos = [(i["id"], i["title"]) for i in chan["entries"]]
         return last_videos
 
     def find_missing(self):
-        """ add missing videos from subscribed channels to pending """
+        """add missing videos from subscribed channels to pending"""
         all_channels = self.get_channels()
         pending_handler = PendingList()
         all_pending, all_ignore = pending_handler.get_all_pending()
-        all_pending_ids = [i['youtube_id'] for i in all_pending]
+        all_pending_ids = [i["youtube_id"] for i in all_pending]
         all_downloaded = pending_handler.get_all_downloaded()
         to_ignore = all_pending_ids + all_ignore + all_downloaded
         missing_videos = []
         counter = 1
         for channel in all_channels:
-            channel_id = channel['channel_id']
+            channel_id = channel["channel_id"]
             last_videos = self.get_last_youtube_videos(channel_id)
-            set_message('progress:download', {
+            RedisArchivist().set_message(
-                "status": "rescan",
+                "progress:download",
-                "level": "info",
+                {
-                "title": "Rescanning: Looking for new videos.",
+                    "status": "rescan",
-                "message": f'Progress: {counter}/{len(all_channels)}'
+                    "level": "info",
-                }
+                    "title": "Rescanning: Looking for new videos.",
+                    "message": f"Progress: {counter}/{len(all_channels)}",
+                },
             )
             for video in last_videos:
                 youtube_id = video[0]
@@ -354,22 +368,22 @@ class ChannelSubscription:
         return missing_videos
 
     def change_subscribe(self, channel_id, channel_subscribed):
-        """ subscribe or unsubscribe from channel and update """
+        """subscribe or unsubscribe from channel and update"""
         if not isinstance(channel_subscribed, bool):
-            print('invalid status, should be bool')
+            print("invalid status, should be bool")
             return
-        headers = {'Content-type': 'application/json'}
+        headers = {"Content-type": "application/json"}
         channel_handler = YoutubeChannel(channel_id)
         channel_dict = channel_handler.channel_dict
-        channel_dict['channel_subscribed'] = channel_subscribed
+        channel_dict["channel_subscribed"] = channel_subscribed
         if channel_subscribed:
             # handle subscribe
-            url = self.es_url + '/ta_channel/_doc/' + channel_id
+            url = self.es_url + "/ta_channel/_doc/" + channel_id
             payload = json.dumps(channel_dict)
             print(channel_dict)
         else:
-            url = self.es_url + '/ta_channel/_update/' + channel_id
+            url = self.es_url + "/ta_channel/_update/" + channel_id
-            payload = json.dumps({'doc': channel_dict})
+            payload = json.dumps({"doc": channel_dict})
         # update channel
         request = requests.post(url, data=payload, headers=headers)
         if not request.ok:
@@ -379,127 +393,167 @@ class ChannelSubscription:
 
 
 def playlist_extractor(playlist_id):
-    """ return youtube_ids from a playlist_id """
+    """return youtube_ids from a playlist_id"""
-    url = 'https://www.youtube.com/playlist?list=' + playlist_id
+    url = "https://www.youtube.com/playlist?list=" + playlist_id
     obs = {
-        'default_search': 'ytsearch', 'quiet': True, 'ignoreerrors': True,
+        "default_search": "ytsearch",
-        'skip_download': True, 'extract_flat': True
+        "quiet": True,
+        "ignoreerrors": True,
+        "skip_download": True,
+        "extract_flat": True,
     }
     playlist = youtube_dl.YoutubeDL(obs).extract_info(url, download=False)
-    playlist_vids = [(i['id'], i['title']) for i in playlist['entries']]
+    playlist_vids = [(i["id"], i["title"]) for i in playlist["entries"]]
     return playlist_vids
 
 
 class VideoDownloader:
-    """ handle the video download functionality """
+    """
+    handle the video download functionality
+    if not initiated with list, take from queue
+    """
 
-    def __init__(self, youtube_id_list):
+    def __init__(self, youtube_id_list=False):
         self.youtube_id_list = youtube_id_list
         self.config = AppConfig().config
 
-    def download_list(self):
+    def run_queue(self):
-        """ download the list of youtube_ids """
+        """setup download queue in redis loop until no more items"""
-        limit_count = self.config['downloads']['limit_count']
+        queue = RedisQueue("dl_queue")
-        if limit_count:
+
-            self.youtube_id_list = self.youtube_id_list[:limit_count]
+        limit_queue = self.config["downloads"]["limit_count"]
+        if limit_queue:
+            queue.trim(limit_queue - 1)
+
+        while True:
+            youtube_id = queue.get_next()
+            if not youtube_id:
+                break
 
-        for youtube_id in self.youtube_id_list:
             try:
                 self.dl_single_vid(youtube_id)
             except youtube_dl.utils.DownloadError:
-                print('failed to download ' + youtube_id)
+                print("failed to download " + youtube_id)
                 continue
             vid_dict = index_new_video(youtube_id)
             self.move_to_archive(vid_dict)
             self.delete_from_pending(youtube_id)
-            if self.config['downloads']['sleep_interval']:
+
-                sleep(self.config['downloads']['sleep_interval'])
+    @staticmethod
+    def add_pending():
+        """add pending videos to download queue"""
+        all_pending, _ = PendingList().get_all_pending()
+        to_add = [i["youtube_id"] for i in all_pending]
+        queue = RedisQueue("dl_queue")
+        queue.add_list(to_add)
 
     @staticmethod
     def progress_hook(response):
-        """ process the progress_hooks from youtube_dl """
+        """process the progress_hooks from youtube_dl"""
         # title
-        filename = response['filename'][12:].replace('_', ' ')
+        filename = response["filename"][12:].replace("_", " ")
         title = "Downloading: " + os.path.split(filename)[-1]
         # message
         try:
-            percent = response['_percent_str']
+            percent = response["_percent_str"]
-            size = response['_total_bytes_str']
+            size = response["_total_bytes_str"]
-            speed = response['_speed_str']
+            speed = response["_speed_str"]
-            eta = response['_eta_str']
+            eta = response["_eta_str"]
-            message = f'{percent} of {size} at {speed} - time left: {eta}'
+            message = f"{percent} of {size} at {speed} - time left: {eta}"
         except KeyError:
-            message = ''
+            message = ""
         mess_dict = {
             "status": "downloading",
             "level": "info",
             "title": title,
-            "message": message
+            "message": message,
         }
-        set_message('progress:download', mess_dict)
+        RedisArchivist().set_message("progress:download", mess_dict)
 
     def dl_single_vid(self, youtube_id):
-        """ download single video """
+        """download single video"""
         obs = {
-            'default_search': 'ytsearch',
+            "default_search": "ytsearch",
-            'merge_output_format': 'mp4', 'restrictfilenames': True,
+            "merge_output_format": "mp4",
-            'outtmpl': (self.config['application']['cache_dir'] +
+            "restrictfilenames": True,
-                        '/download/' +
+            "outtmpl": (
-                        self.config['application']['file_template']),
+                self.config["application"]["cache_dir"]
-            'progress_hooks': [self.progress_hook],
+                + "/download/"
-            'quiet': True, 'continuedl': True, 'retries': 3
+                + self.config["application"]["file_template"]
+            ),
+            "progress_hooks": [self.progress_hook],
+            "quiet": True,
+            "continuedl": True,
+            "retries": 3,
         }
-        if self.config['downloads']['format']:
+        if self.config["downloads"]["format"]:
-            obs['format'] = self.config['downloads']['format']
+            obs["format"] = self.config["downloads"]["format"]
-        if self.config['downloads']['limit_speed']:
+        if self.config["downloads"]["limit_speed"]:
-            obs['ratelimit'] = self.config['downloads']['limit_speed'] * 1024
+            obs["ratelimit"] = self.config["downloads"]["limit_speed"] * 1024
         external = False
         if external:
-            obs['external_downloader'] = 'aria2c'
+            obs["external_downloader"] = "aria2c"
+
+        postprocessors = []
+
+        if self.config["downloads"]["add_metadata"]:
+            postprocessors.append(
+                {
+                    "key": "FFmpegMetadata",
+                    "add_chapters": True,
+                    "add_metadata": True,
+                }
+            )
+
+        obs["postprocessors"] = postprocessors
+
         # check if already in cache to continue from there
-        cache_dir = self.config['application']['cache_dir']
+        cache_dir = self.config["application"]["cache_dir"]
-        all_cached = os.listdir(cache_dir + '/download/')
+        cached = os.listdir(cache_dir + "/download/")
+        all_cached = ignore_filelist(cached)
         for file_name in all_cached:
             if youtube_id in file_name:
-                obs['outtmpl'] = cache_dir + '/download/' + file_name
+                obs["outtmpl"] = cache_dir + "/download/" + file_name
         with youtube_dl.YoutubeDL(obs) as ydl:
             try:
                 ydl.download([youtube_id])
             except youtube_dl.utils.DownloadError:
-                print('retry failed download: ' + youtube_id)
+                print("retry failed download: " + youtube_id)
                 sleep(10)
                 ydl.download([youtube_id])
 
     def move_to_archive(self, vid_dict):
-        """ move downloaded video from cache to archive """
+        """move downloaded video from cache to archive"""
-        videos = self.config['application']['videos']
+        videos = self.config["application"]["videos"]
-        channel_name = vid_dict['channel']['channel_name']
+        channel_name = vid_dict["channel"]["channel_name"]
         channel_name_clean = clean_string(channel_name)
-        media_url = vid_dict['media_url']
+        media_url = vid_dict["media_url"]
-        youtube_id = vid_dict['youtube_id']
+        youtube_id = vid_dict["youtube_id"]
         # make archive folder
-        videos = self.config['application']['videos']
+        videos = self.config["application"]["videos"]
         new_folder = os.path.join(videos, channel_name_clean)
         os.makedirs(new_folder, exist_ok=True)
         # find real filename
-        cache_dir = self.config['application']['cache_dir']
+        cache_dir = self.config["application"]["cache_dir"]
-        for file_str in os.listdir(cache_dir + '/download'):
+        cached = os.listdir(cache_dir + "/download/")
+        all_cached = ignore_filelist(cached)
+        for file_str in all_cached:
             if youtube_id in file_str:
                 old_file = file_str
-        old_file_path = os.path.join(cache_dir, 'download', old_file)
+        old_file_path = os.path.join(cache_dir, "download", old_file)
         new_file_path = os.path.join(videos, media_url)
         # move and fix permission
         shutil.move(old_file_path, new_file_path)
         os.chown(
             new_file_path,
-            self.config['application']['HOST_UID'],
+            self.config["application"]["HOST_UID"],
-            self.config['application']['HOST_GID']
+            self.config["application"]["HOST_GID"],
         )
 
     def delete_from_pending(self, youtube_id):
-        """ delete downloaded video from pending index if its there """
+        """delete downloaded video from pending index if its there"""
-        es_url = self.config['application']['es_url']
+        es_url = self.config["application"]["es_url"]
-        url = f'{es_url}/ta_download/_doc/{youtube_id}'
+        url = f"{es_url}/ta_download/_doc/{youtube_id}"
         response = requests.delete(url)
         if not response.ok and not response.status_code == 404:
             print(response.text)

tubearchivist/home/src/helper.py

@@ -10,120 +10,203 @@ import string
 import subprocess
 import unicodedata
 
-import requests
 import redis
-
+import requests
-REDIS_HOST = os.environ.get('REDIS_HOST')
 
 
 def get_total_hits(index, es_url, match_field):
-    """ get total hits from index """
+    """get total hits from index"""
-    headers = {'Content-type': 'application/json'}
+    headers = {"Content-type": "application/json"}
     data = {"query": {"match": {match_field: True}}}
     payload = json.dumps(data)
-    url = f'{es_url}/{index}/_search?filter_path=hits.total'
+    url = f"{es_url}/{index}/_search?filter_path=hits.total"
     request = requests.post(url, data=payload, headers=headers)
     if not request.ok:
         print(request.text)
     total_json = json.loads(request.text)
-    total_hits = total_json['hits']['total']['value']
+    total_hits = total_json["hits"]["total"]["value"]
     return total_hits
 
 
 def clean_string(file_name):
-    """ clean string to only asci characters """
+    """clean string to only asci characters"""
     whitelist = "-_.() " + string.ascii_letters + string.digits
-    normalized = unicodedata.normalize('NFKD', file_name)
+    normalized = unicodedata.normalize("NFKD", file_name)
-    ascii_only = normalized.encode('ASCII', 'ignore').decode().strip()
+    ascii_only = normalized.encode("ASCII", "ignore").decode().strip()
-    white_listed = ''.join(c for c in ascii_only if c in whitelist)
+    white_listed = "".join(c for c in ascii_only if c in whitelist)
-    cleaned = re.sub(r'[ ]{2,}', ' ', white_listed)
+    cleaned = re.sub(r"[ ]{2,}", " ", white_listed)
+    return cleaned
+
+
+def ignore_filelist(filelist):
+    """ignore temp files for os.listdir sanitizer"""
+    to_ignore = ["Icon\r\r", "Temporary Items", "Network Trash Folder"]
+    cleaned = []
+    for file_name in filelist:
+        if file_name.startswith(".") or file_name in to_ignore:
+            continue
+
+        cleaned.append(file_name)
+
     return cleaned
 
 
 def process_url_list(url_str):
-    """ parse url_list to find valid youtube video or channel ids """
+    """parse url_list to find valid youtube video or channel ids"""
-    to_replace = ['watch?v=', 'playlist?list=']
+    to_replace = ["watch?v=", "playlist?list="]
-    url_list = re.split('\n+', url_str[0])
+    url_list = re.split("\n+", url_str[0])
     youtube_ids = []
     for url in url_list:
-        url_clean = url.strip().strip('/').split('/')[-1]
+        if "/c/" in url or "/user/" in url:
+            raise ValueError("user name is not unique, use channel ID")
+
+        url_clean = url.strip().strip("/").split("/")[-1]
         for i in to_replace:
-            url_clean = url_clean.replace(i, '')
+            url_clean = url_clean.replace(i, "")
-        url_no_param = url_clean.split('&')[0]
+        url_no_param = url_clean.split("&")[0]
         str_len = len(url_no_param)
         if str_len == 11:
-            link_type = 'video'
+            link_type = "video"
         elif str_len == 24:
-            link_type = 'channel'
+            link_type = "channel"
         elif str_len == 34:
-            link_type = 'playlist'
+            link_type = "playlist"
         else:
             # unable to parse
-            raise ValueError('not a valid url: ' + url)
+            raise ValueError("not a valid url: " + url)
 
         youtube_ids.append({"url": url_no_param, "type": link_type})
 
     return youtube_ids
 
 
-def set_message(key, message, expire=True):
+class RedisArchivist:
-    """ write new message to redis """
+    """collection of methods to interact with redis"""
-    redis_connection = redis.Redis(host=REDIS_HOST)
+
-    redis_connection.execute_command(
+    REDIS_HOST = os.environ.get("REDIS_HOST")
-        'JSON.SET', key, '.', json.dumps(message)
+    REDIS_PORT = os.environ.get("REDIS_PORT")
-    )
+
-    if expire:
+    if not REDIS_PORT:
-        redis_connection.execute_command('EXPIRE', key, 20)
+        REDIS_PORT = 6379
+
+    def __init__(self):
+        self.redis_connection = redis.Redis(
+            host=self.REDIS_HOST, port=self.REDIS_PORT
+        )
+
+    def set_message(self, key, message, expire=True):
+        """write new message to redis"""
+        self.redis_connection.execute_command(
+            "JSON.SET", key, ".", json.dumps(message)
+        )
+
+        if expire:
+            self.redis_connection.execute_command("EXPIRE", key, 20)
+
+    def get_message(self, key):
+        """get message dict from redis"""
+        reply = self.redis_connection.execute_command("JSON.GET", key)
+        if reply:
+            json_str = json.loads(reply)
+        else:
+            json_str = {"status": False}
+
+        return json_str
+
+    def del_message(self, key):
+        """delete key from redis"""
+        response = self.redis_connection.execute_command("DEL", key)
+        return response
+
+    def get_lock(self, lock_key):
+        """handle lock for task management"""
+        redis_lock = self.redis_connection.lock(lock_key)
+        return redis_lock
+
+    def get_dl_message(self, cache_dir):
+        """get latest download progress message if available"""
+        reply = self.redis_connection.execute_command(
+            "JSON.GET", "progress:download"
+        )
+        if reply:
+            json_str = json.loads(reply)
+        elif json_str := self.monitor_cache_dir(cache_dir):
+            json_str = self.monitor_cache_dir(cache_dir)
+        else:
+            json_str = {"status": False}
+
+        return json_str
+
+    @staticmethod
+    def monitor_cache_dir(cache_dir):
+        """
+        look at download cache dir directly as alternative progress info
+        """
+        dl_cache = os.path.join(cache_dir, "download")
+        all_cache_file = os.listdir(dl_cache)
+        cache_file = ignore_filelist(all_cache_file)
+        if cache_file:
+            filename = cache_file[0][12:].replace("_", " ").split(".")[0]
+            mess_dict = {
+                "status": "downloading",
+                "level": "info",
+                "title": "Downloading: " + filename,
+                "message": "",
+            }
+        else:
+            return False
+
+        return mess_dict
 
 
-def get_message(key):
+class RedisQueue:
-    """ get any message from JSON key """
+    """dynamically interact with the download queue in redis"""
-    redis_connection = redis.Redis(host=REDIS_HOST)
-    reply = redis_connection.execute_command('JSON.GET', key)
-    if reply:
-        json_str = json.loads(reply)
-    else:
-        json_str = {"status": False}
-    return json_str
 
+    REDIS_HOST = os.environ.get("REDIS_HOST")
+    REDIS_PORT = os.environ.get("REDIS_PORT")
 
-def get_dl_message(cache_dir):
+    if not REDIS_PORT:
-    """ get latest message if available """
+        REDIS_PORT = 6379
-    redis_connection = redis.Redis(host=REDIS_HOST)
-    reply = redis_connection.execute_command('JSON.GET', 'progress:download')
-    if reply:
-        json_str = json.loads(reply)
-    elif json_str := monitor_cache_dir(cache_dir):
-        json_str = monitor_cache_dir(cache_dir)
-    else:
-        json_str = {"status": False}
-    return json_str
 
+    def __init__(self, key):
+        self.key = key
+        self.conn = redis.Redis(host=self.REDIS_HOST, port=self.REDIS_PORT)
 
-def get_lock(lock_key):
+    def get_all(self):
-    """ handle lock for task management """
+        """return all elements in list"""
-    redis_lock = redis.Redis(host=REDIS_HOST).lock(lock_key)
+        result = self.conn.execute_command("LRANGE", self.key, 0, -1)
-    return redis_lock
+        all_elements = [i.decode() for i in result]
+        return all_elements
 
+    def add_list(self, to_add):
+        """add list to queue"""
+        self.conn.execute_command("RPUSH", self.key, *to_add)
 
-def monitor_cache_dir(cache_dir):
+    def add_priority(self, to_add):
-    """
+        """add single video to front of queue"""
-    look at download cache dir directly as alterative progress info
+        self.clear_item(to_add)
-    """
+        self.conn.execute_command("LPUSH", self.key, to_add)
-    dl_cache = os.path.join(cache_dir, 'download')
-    cache_file = os.listdir(dl_cache)
-    if cache_file:
-        filename = cache_file[0][12:].replace('_', ' ').split('.')[0]
-        mess_dict = {
-            "status": "downloading",
-            "level": "info",
-            "title": "Downloading: " + filename,
-            "message": ""
-        }
-    else:
-        return False
 
-    return mess_dict
+    def get_next(self):
+        """return next element in the queue, False if none"""
+        result = self.conn.execute_command("LPOP", self.key)
+        if not result:
+            return False
+
+        next_element = result.decode()
+        return next_element
+
+    def clear(self):
+        """delete list from redis"""
+        self.conn.execute_command("DEL", self.key)
+
+    def clear_item(self, to_clear):
+        """remove single item from list if it's there"""
+        self.conn.execute_command("LREM", self.key, 0, to_clear)
+
+    def trim(self, size):
+        """trim the queue based on settings amount"""
+        self.conn.execute_command("LTRIM", self.key, 0, size)
 
 
 class DurationConverter:
@@ -133,27 +216,37 @@ class DurationConverter:
 
     @staticmethod
     def get_sec(file_path):
-        """ read duration from file """
+        """read duration from file"""
-        duration = subprocess.run([
+        duration = subprocess.run(
-            "ffprobe", "-v", "error", "-show_entries", "format=duration",
+            [
-            "-of", "default=noprint_wrappers=1:nokey=1", file_path
+                "ffprobe",
-        ], capture_output=True, check=True)
+                "-v",
+                "error",
+                "-show_entries",
+                "format=duration",
+                "-of",
+                "default=noprint_wrappers=1:nokey=1",
+                file_path,
+            ],
+            capture_output=True,
+            check=True,
+        )
         duration_sec = int(float(duration.stdout.decode().strip()))
         return duration_sec
 
     @staticmethod
     def get_str(duration_sec):
-        """ takes duration in sec and returns clean string """
+        """takes duration in sec and returns clean string"""
         hours = duration_sec // 3600
         minutes = (duration_sec - (hours * 3600)) // 60
         secs = duration_sec - (hours * 3600) - (minutes * 60)
 
         duration_str = str()
         if hours:
-            duration_str = str(hours).zfill(2) + ':'
+            duration_str = str(hours).zfill(2) + ":"
         if minutes:
-            duration_str = duration_str + str(minutes).zfill(2) + ':'
+            duration_str = duration_str + str(minutes).zfill(2) + ":"
         else:
-            duration_str = duration_str + '00:'
+            duration_str = duration_str + "00:"
         duration_str = duration_str + str(secs).zfill(2)
         return duration_str

tubearchivist/home/src/index.py

@@ -6,27 +6,24 @@ Functionality:
 """
 
 import json
+import os
 import re
-
 from datetime import datetime
 from time import sleep
-import os
 
 import requests
 import yt_dlp as youtube_dl
-
 from bs4 import BeautifulSoup
-
 from home.src.config import AppConfig
-from home.src.helper import clean_string, DurationConverter
+from home.src.helper import DurationConverter, clean_string, process_url_list
 
 
 class YoutubeChannel:
-    """ represents a single youtube channel """
+    """represents a single youtube channel"""
 
     CONFIG = AppConfig().config
-    ES_URL = CONFIG['application']['es_url']
+    ES_URL = CONFIG["application"]["es_url"]
-    CACHE_DIR = CONFIG['application']['cache_dir']
+    CACHE_DIR = CONFIG["application"]["cache_dir"]
 
     def __init__(self, channel_id):
         self.channel_id = channel_id
@@ -35,193 +32,187 @@ class YoutubeChannel:
         self.channel_dict = self.build_channel_dict()
 
     def build_channel_dict(self, scrape=False):
-        """ combine the dicts build from extracted json payload """
+        """combine the dicts build from extracted json payload"""
         if scrape:
             channel_dict = False
         else:
             channel_dict = self.get_es_channel()
         if not channel_dict:
-            print('scrape data from youtube')
+            print("scrape data from youtube")
             self.scrape_channel()
             channel_dict = self.parse_channel_main()
             channel_dict.update(self.parse_channel_meta())
-            self.source = 'scraped'
+            self.source = "scraped"
         return channel_dict
 
     def get_es_channel(self):
-        """ get from elastic search first if possible """
+        """get from elastic search first if possible"""
         channel_id = self.channel_id
-        url = f'{self.ES_URL}/ta_channel/_doc/{channel_id}'
+        url = f"{self.ES_URL}/ta_channel/_doc/{channel_id}"
         response = requests.get(url)
         if response.ok:
-            channel_source = response.json()['_source']
+            channel_source = response.json()["_source"]
-            self.source = 'elastic'
+            self.source = "elastic"
             return channel_source
         return False
 
     def scrape_channel(self):
-        """ scrape channel page for additional infos """
+        """scrape channel page for additional infos"""
         channel_id = self.channel_id
-        url = f'https://www.youtube.com/channel/{channel_id}/about?hl=en'
+        url = f"https://www.youtube.com/channel/{channel_id}/about?hl=en"
-        cookies = {
+        cookies = {"CONSENT": "YES+xxxxxxxxxxxxxxxxxxxxxxxxxxx"}
-            'CONSENT': 'YES+xxxxxxxxxxxxxxxxxxxxxxxxxxx'
-        }
         response = requests.get(url, cookies=cookies)
         if response.ok:
             channel_page = response.text
         else:
-            print(f'failed to extract channel info for: {channel_id}')
+            print(f"failed to extract channel info for: {channel_id}")
             raise ConnectionError
-        soup = BeautifulSoup(channel_page, 'html.parser')
+        soup = BeautifulSoup(channel_page, "html.parser")
         # load script into json
-        all_scripts = soup.find('body').find_all('script')
+        all_scripts = soup.find("body").find_all("script")
         for script in all_scripts:
-            if 'var ytInitialData = ' in str(script):
+            if "var ytInitialData = " in str(script):
                 script_content = str(script)
                 break
         # extract payload
-        script_content = script_content.split('var ytInitialData = ')[1]
+        script_content = script_content.split("var ytInitialData = ")[1]
-        json_raw = script_content.rstrip(';</script>')
+        json_raw = script_content.rstrip(";</script>")
         json_data = json.loads(json_raw)
         # add to self
         self.json_data = json_data
 
     def parse_channel_main(self):
-        """ extract maintab values from scraped channel json data """
+        """extract maintab values from scraped channel json data"""
-        main_tab = self.json_data['header']['c4TabbedHeaderRenderer']
+        main_tab = self.json_data["header"]["c4TabbedHeaderRenderer"]
-        channel_name = main_tab['title']
+        channel_name = main_tab["title"]
         last_refresh = int(datetime.now().strftime("%s"))
         # channel_subs
         try:
-            sub_text_simple = main_tab['subscriberCountText']['simpleText']
+            sub_text_simple = main_tab["subscriberCountText"]["simpleText"]
-            sub_text = sub_text_simple.split(' ')[0]
+            sub_text = sub_text_simple.split(" ")[0]
-            if sub_text[-1] == 'K':
+            if sub_text[-1] == "K":
-                channel_subs = int(float(sub_text.replace('K', ''))*1000)
+                channel_subs = int(float(sub_text.replace("K", "")) * 1000)
-            elif sub_text[-1] == 'M':
+            elif sub_text[-1] == "M":
-                channel_subs = int(float(sub_text.replace('M', ''))*1000000)
+                channel_subs = int(float(sub_text.replace("M", "")) * 1000000)
             elif int(sub_text) >= 0:
                 channel_subs = int(sub_text)
             else:
-                message = f'{sub_text} not dealt with'
+                message = f"{sub_text} not dealt with"
                 print(message)
         except KeyError:
             channel_subs = 0
         # banner
         try:
-            all_banners = main_tab['banner']['thumbnails']
+            all_banners = main_tab["banner"]["thumbnails"]
-            banner = sorted(all_banners, key=lambda k: k['width'])[-1]['url']
+            banner = sorted(all_banners, key=lambda k: k["width"])[-1]["url"]
         except KeyError:
             banner = False
         # build and return dict
         main_channel_dict = {
-            'channel_active': True,
+            "channel_active": True,
-            'channel_last_refresh': last_refresh,
+            "channel_last_refresh": last_refresh,
-            'channel_subs': channel_subs,
+            "channel_subs": channel_subs,
-            'channel_banner_url': banner,
+            "channel_banner_url": banner,
-            'channel_name': channel_name,
+            "channel_name": channel_name,
-            'channel_id': self.channel_id
+            "channel_id": self.channel_id,
         }
         return main_channel_dict
 
     def parse_channel_meta(self):
-        """ extract meta tab values from channel payload """
+        """extract meta tab values from channel payload"""
         # meta tab
         json_data = self.json_data
-        meta_tab = json_data['metadata']['channelMetadataRenderer']
+        meta_tab = json_data["metadata"]["channelMetadataRenderer"]
-        description = meta_tab['description']
+        description = meta_tab["description"]
-        all_thumbs = meta_tab['avatar']['thumbnails']
+        all_thumbs = meta_tab["avatar"]["thumbnails"]
-        thumb_url = sorted(all_thumbs, key=lambda k: k['width'])[-1]['url']
+        thumb_url = sorted(all_thumbs, key=lambda k: k["width"])[-1]["url"]
         # stats tab
-        renderer = 'twoColumnBrowseResultsRenderer'
+        renderer = "twoColumnBrowseResultsRenderer"
-        all_tabs = json_data['contents'][renderer]['tabs']
+        all_tabs = json_data["contents"][renderer]["tabs"]
         for tab in all_tabs:
-            if 'tabRenderer' in tab.keys():
+            if "tabRenderer" in tab.keys():
-                if tab['tabRenderer']['title'] == 'About':
+                if tab["tabRenderer"]["title"] == "About":
-                    about_tab = (tab['tabRenderer']['content']
+                    about_tab = tab["tabRenderer"]["content"][
-                                 ['sectionListRenderer']['contents'][0]
+                        "sectionListRenderer"
-                                 ['itemSectionRenderer']['contents'][0]
+                    ]["contents"][0]["itemSectionRenderer"]["contents"][0][
-                                 ['channelAboutFullMetadataRenderer'])
+                        "channelAboutFullMetadataRenderer"
+                    ]
                     break
         try:
-            channel_views_text = about_tab['viewCountText']['simpleText']
+            channel_views_text = about_tab["viewCountText"]["simpleText"]
             channel_views = int(re.sub(r"\D", "", channel_views_text))
         except KeyError:
             channel_views = 0
 
         meta_channel_dict = {
-            'channel_description': description,
+            "channel_description": description,
-            'channel_thumb_url': thumb_url,
+            "channel_thumb_url": thumb_url,
-            'channel_views': channel_views
+            "channel_views": channel_views,
         }
 
         return meta_channel_dict
 
     def upload_to_es(self):
-        """ upload channel data to elastic search """
+        """upload channel data to elastic search"""
-        url = f'{self.ES_URL}/ta_channel/_doc/{self.channel_id}'
+        url = f"{self.ES_URL}/ta_channel/_doc/{self.channel_id}"
         response = requests.put(url, json=self.channel_dict)
-        print(f'added {self.channel_id} to es')
+        print(f"added {self.channel_id} to es")
         if not response.ok:
             print(response.text)
 
     def clear_cache(self):
-        """ delete banner and thumb from cache if available """
+        """delete banner and thumb from cache if available"""
-        channel_cache = os.path.join(self.CACHE_DIR, 'channels')
+        channel_cache = os.path.join(self.CACHE_DIR, "channels")
-        thumb = os.path.join(channel_cache, self.channel_id + '_thumb.jpg')
+        thumb = os.path.join(channel_cache, self.channel_id + "_thumb.jpg")
-        banner = os.path.join(channel_cache, self.channel_id + '_banner.jpg')
+        banner = os.path.join(channel_cache, self.channel_id + "_banner.jpg")
         if os.path.exists(thumb):
             os.remove(thumb)
         if os.path.exists(banner):
             os.remove(banner)
 
     def sync_to_videos(self):
-        """ sync new channel_dict to all videos of channel """
+        """sync new channel_dict to all videos of channel"""
-        headers = {'Content-type': 'application/json'}
+        headers = {"Content-type": "application/json"}
         channel_id = self.channel_id
         # add ingest pipeline
         processors = []
         for field, value in self.channel_dict.items():
             line = {"set": {"field": "channel." + field, "value": value}}
             processors.append(line)
-        data = {
+        data = {"description": channel_id, "processors": processors}
-            "description": channel_id,
-            "processors": processors
-        }
         payload = json.dumps(data)
-        url = self.ES_URL + '/_ingest/pipeline/' + channel_id
+        url = self.ES_URL + "/_ingest/pipeline/" + channel_id
         request = requests.put(url, data=payload, headers=headers)
         if not request.ok:
             print(request.text)
         # apply pipeline
-        data = {
+        data = {"query": {"match": {"channel.channel_id": channel_id}}}
-            "query": {"match": {"channel.channel_id": channel_id}}
-        }
         payload = json.dumps(data)
-        url = self.ES_URL + '/ta_video/_update_by_query?pipeline=' + channel_id
+        url = self.ES_URL + "/ta_video/_update_by_query?pipeline=" + channel_id
         request = requests.post(url, data=payload, headers=headers)
         if not request.ok:
             print(request.text)
 
     def get_total_hits(self):
-        """ get total channels indexed """
+        """get total channels indexed"""
-        headers = {'Content-type': 'application/json'}
+        headers = {"Content-type": "application/json"}
         data = {"query": {"match_all": {}}}
         payload = json.dumps(data)
-        url = f'{self.ES_URL}/ta_channel/_search?filter_path=hits.total'
+        url = f"{self.ES_URL}/ta_channel/_search?filter_path=hits.total"
         request = requests.post(url, data=payload, headers=headers)
         if not request.ok:
             print(request.text)
-        total_hits = json.loads(request.text)['hits']['total']['value']
+        total_hits = json.loads(request.text)["hits"]["total"]["value"]
         return total_hits
 
 
 class YoutubeVideo:
-    """ represents a signle youtube video """
+    """represents a single youtube video"""
 
     CONFIG = AppConfig().config
-    ES_URL = CONFIG['application']['es_url']
+    ES_URL = CONFIG["application"]["es_url"]
-    CACHE_DIR = CONFIG['application']['cache_dir']
+    CACHE_DIR = CONFIG["application"]["cache_dir"]
-    VIDEOS = CONFIG['application']['videos']
+    VIDEOS = CONFIG["application"]["videos"]
 
     def __init__(self, youtube_id):
         self.youtube_id = youtube_id
@@ -229,8 +220,8 @@ class YoutubeVideo:
         self.vid_dict = self.get_wrapper()
 
     def get_wrapper(self):
-        """ wrapper to loop around youtube_dl to retry on failure """
+        """wrapper to loop around youtube_dl to retry on failure"""
-        print(f'get video data for {self.youtube_id}')
+        print(f"get video data for {self.youtube_id}")
         for i in range(3):
             try:
                 vid_dict = self.get_youtubedl_vid_data()
@@ -244,63 +235,63 @@ class YoutubeVideo:
         return vid_dict
 
     def get_youtubedl_vid_data(self):
-        """ parse youtubedl extract info """
+        """parse youtubedl extract info"""
         youtube_id = self.youtube_id
         obs = {
-            'quiet': True,
+            "quiet": True,
-            'default_search': 'ytsearch',
+            "default_search": "ytsearch",
-            'skip_download': True
+            "skip_download": True,
         }
         try:
             vid = youtube_dl.YoutubeDL(obs).extract_info(youtube_id)
         except (
-                youtube_dl.utils.ExtractorError,
+            youtube_dl.utils.ExtractorError,
-                youtube_dl.utils.DownloadError
+            youtube_dl.utils.DownloadError,
-                ):
+        ):
-            print('failed to get info for ' + youtube_id)
+            print("failed to get info for " + youtube_id)
             return False
         # extract
-        self.channel_id = vid['channel_id']
+        self.channel_id = vid["channel_id"]
-        upload_date = vid['upload_date']
+        upload_date = vid["upload_date"]
         upload_date_time = datetime.strptime(upload_date, "%Y%m%d")
         published = upload_date_time.strftime("%Y-%m-%d")
         last_refresh = int(datetime.now().strftime("%s"))
         # likes
         try:
-            like_count = vid['like_count']
+            like_count = vid["like_count"]
         except KeyError:
             like_count = 0
         try:
-            dislike_count = vid['dislike_count']
+            dislike_count = vid["dislike_count"]
         except KeyError:
             dislike_count = 0
         # build dicts
         stats = {
-            "view_count": vid['view_count'],
+            "view_count": vid["view_count"],
             "like_count": like_count,
             "dislike_count": dislike_count,
-            "average_rating": vid['average_rating']
+            "average_rating": vid["average_rating"],
         }
         vid_basic = {
-            "title": vid['title'],
+            "title": vid["title"],
-            "description": vid['description'],
+            "description": vid["description"],
-            "category": vid['categories'],
+            "category": vid["categories"],
-            "vid_thumb_url": vid['thumbnail'],
+            "vid_thumb_url": vid["thumbnail"],
-            "tags": vid['tags'],
+            "tags": vid["tags"],
             "published": published,
             "stats": stats,
             "vid_last_refresh": last_refresh,
             "date_downloaded": last_refresh,
             "youtube_id": youtube_id,
             "active": True,
-            "channel": False
+            "channel": False,
         }
 
         return vid_basic
 
     def add_player(self, missing_vid):
-        """ add player information for new videos """
+        """add player information for new videos"""
-        cache_path = self.CACHE_DIR + '/download/'
+        cache_path = self.CACHE_DIR + "/download/"
         videos = self.VIDEOS
 
         if missing_vid:
@@ -321,24 +312,24 @@ class YoutubeVideo:
         player = {
             "watched": False,
             "duration": duration,
-            "duration_str": duration_str
+            "duration_str": duration_str,
         }
-        self.vid_dict['player'] = player
+        self.vid_dict["player"] = player
 
     def build_file_path(self, channel_name):
-        """ build media_url from where file will be located """
+        """build media_url from where file will be located"""
         clean_channel_name = clean_string(channel_name)
-        timestamp = self.vid_dict['published'].replace('-', '')
+        timestamp = self.vid_dict["published"].replace("-", "")
-        youtube_id = self.vid_dict['youtube_id']
+        youtube_id = self.vid_dict["youtube_id"]
-        title = self.vid_dict['title']
+        title = self.vid_dict["title"]
         clean_title = clean_string(title)
-        filename = f'{timestamp}_{youtube_id}_{clean_title}.mp4'
+        filename = f"{timestamp}_{youtube_id}_{clean_title}.mp4"
         media_url = os.path.join(clean_channel_name, filename)
-        self.vid_dict['media_url'] = media_url
+        self.vid_dict["media_url"] = media_url
 
     def get_es_data(self):
-        """ get current data from elastic search """
+        """get current data from elastic search"""
-        url = self.ES_URL + '/ta_video/_doc/' + self.youtube_id
+        url = self.ES_URL + "/ta_video/_doc/" + self.youtube_id
         response = requests.get(url)
         if not response.ok:
             print(response.text)
@@ -346,48 +337,120 @@ class YoutubeVideo:
         return es_vid_dict
 
     def upload_to_es(self):
-        """ upload channel data to elastic search """
+        """upload channel data to elastic search"""
-        url = f'{self.ES_URL}/ta_video/_doc/{self.youtube_id}'
+        url = f"{self.ES_URL}/ta_video/_doc/{self.youtube_id}"
         response = requests.put(url, json=self.vid_dict)
         if not response.ok:
             print(response.text)
 
     def delete_cache(self):
-        """ delete thumbnail from cache if exist """
+        """delete thumbnail from cache if exist"""
-        video_cache = os.path.join(self.CACHE_DIR, 'videos')
+        video_cache = os.path.join(self.CACHE_DIR, "videos")
-        thumb = os.path.join(video_cache, self.youtube_id + '.jpg')
+        thumb = os.path.join(video_cache, self.youtube_id + ".jpg")
         if os.path.exists(thumb):
             os.remove(thumb)
 
     def deactivate(self):
-        """ deactivate document on extractor error """
+        """deactivate document on extractor error"""
         youtube_id = self.youtube_id
-        headers = {'Content-type': 'application/json'}
+        headers = {"Content-type": "application/json"}
-        url = f'{self.ES_URL}/ta_video/_update/{youtube_id}'
+        url = f"{self.ES_URL}/ta_video/_update/{youtube_id}"
         data = {"script": "ctx._source.active = false"}
         json_str = json.dumps(data)
         response = requests.post(url, data=json_str, headers=headers)
-        print(f'deactivated {youtube_id}')
+        print(f"deactivated {youtube_id}")
         if not response.ok:
             print(response.text)
 
 
+class WatchState:
+    """handle watched checkbox for videos and channels"""
+
+    CONFIG = AppConfig().config
+    ES_URL = CONFIG["application"]["es_url"]
+    HEADERS = {"Content-type": "application/json"}
+
+    def __init__(self, youtube_id):
+        self.youtube_id = youtube_id
+        self.stamp = int(datetime.now().strftime("%s"))
+
+    def mark_as_watched(self):
+        """update es with new watched value"""
+        url_type = self.dedect_type()
+        if url_type == "video":
+            self.mark_vid_watched()
+        elif url_type == "channel":
+            self.mark_channel_watched()
+
+        print(f"marked {self.youtube_id} as watched")
+
+    def dedect_type(self):
+        """find youtube id type"""
+        url_process = process_url_list([self.youtube_id])
+        url_type = url_process[0]["type"]
+
+        return url_type
+
+    def mark_vid_watched(self):
+        """change watched status of single video"""
+        url = self.ES_URL + "/ta_video/_update/" + self.youtube_id
+        data = {
+            "doc": {"player": {"watched": True, "watched_date": self.stamp}}
+        }
+        payload = json.dumps(data)
+        request = requests.post(url, data=payload, headers=self.HEADERS)
+        if not request.ok:
+            print(request.text)
+
+    def mark_channel_watched(self):
+        """change watched status of every video in channel"""
+        es_url = self.ES_URL
+        headers = self.HEADERS
+        youtube_id = self.youtube_id
+        # create pipeline
+        data = {
+            "description": youtube_id,
+            "processors": [
+                {"set": {"field": "player.watched", "value": True}},
+                {"set": {"field": "player.watched_date", "value": self.stamp}},
+            ],
+        }
+        payload = json.dumps(data)
+        url = f"{es_url}/_ingest/pipeline/{youtube_id}"
+        request = requests.put(url, data=payload, headers=headers)
+        if not request.ok:
+            print(request.text)
+            raise ValueError("failed to post ingest pipeline")
+
+        # apply pipeline
+        must_list = [
+            {"term": {"channel.channel_id": {"value": youtube_id}}},
+            {"term": {"player.watched": {"value": False}}},
+        ]
+        data = {"query": {"bool": {"must": must_list}}}
+        payload = json.dumps(data)
+        url = f"{es_url}/ta_video/_update_by_query?pipeline={youtube_id}"
+        request = requests.post(url, data=payload, headers=headers)
+        if not request.ok:
+            print(request.text)
+
+
 def index_new_video(youtube_id, missing_vid=False):
-    """ combine video and channel classes for new video index """
+    """combine video and channel classes for new video index"""
     vid_handler = YoutubeVideo(youtube_id)
     if not vid_handler.vid_dict:
-        raise ValueError('failed to get metadata for ' + youtube_id)
+        raise ValueError("failed to get metadata for " + youtube_id)
 
     channel_handler = YoutubeChannel(vid_handler.channel_id)
     # add filepath to vid_dict
-    channel_name = channel_handler.channel_dict['channel_name']
+    channel_name = channel_handler.channel_dict["channel_name"]
     vid_handler.build_file_path(channel_name)
     # add channel and player to video
     vid_handler.add_player(missing_vid)
-    vid_handler.vid_dict['channel'] = channel_handler.channel_dict
+    vid_handler.vid_dict["channel"] = channel_handler.channel_dict
     # add new channel to es
-    if channel_handler.source == 'scraped':
+    if channel_handler.source == "scraped":
-        channel_handler.channel_dict['channel_subscribed'] = False
+        channel_handler.channel_dict["channel_subscribed"] = False
         channel_handler.upload_to_es()
     # upload video to es
     vid_handler.upload_to_es()

tubearchivist/home/src/index_management.py

@@ -8,19 +8,18 @@ Functionality:
 
 import json
 import os
-
+import zipfile
 from datetime import datetime
 
 import requests
-
 from home.src.config import AppConfig
-
+from home.src.helper import ignore_filelist
 
 # expected mapping and settings
 INDEX_CONFIG = [
     {
-        'index_name': 'channel',
+        "index_name": "channel",
-        'expected_map': {
+        "expected_map": {
             "channel_id": {
                 "type": "keyword",
             },
@@ -30,53 +29,34 @@ INDEX_CONFIG = [
                     "keyword": {
                         "type": "keyword",
                         "ignore_above": 256,
-                        "normalizer": "to_lower"
+                        "normalizer": "to_lower",
                     },
                     "search_as_you_type": {
                         "type": "search_as_you_type",
                         "doc_values": False,
-                        "max_shingle_size": 3
+                        "max_shingle_size": 3,
-                    }
+                    },
-                }
+                },
             },
-            "channel_banner_url": {
+            "channel_banner_url": {"type": "keyword", "index": False},
-                "type": "keyword",
+            "channel_thumb_url": {"type": "keyword", "index": False},
-                "index": False
+            "channel_description": {"type": "text"},
-            },
+            "channel_last_refresh": {"type": "date", "format": "epoch_second"},
-            "channel_thumb_url": {
-                "type": "keyword",
-                "index": False
-            },
-            "channel_description": {
-                "type": "text"
-            },
-            "channel_last_refresh": {
-                "type": "date",
-                "format": "epoch_second"
-            }
         },
-        'expected_set': {
+        "expected_set": {
             "analysis": {
                 "normalizer": {
-                    "to_lower": {
+                    "to_lower": {"type": "custom", "filter": ["lowercase"]}
-                        "type": "custom",
-                        "filter": ["lowercase"]
-                    }
                 }
             },
-            "number_of_replicas": "0"
+            "number_of_replicas": "0",
-        }
+        },
     },
     {
-        'index_name': 'video',
+        "index_name": "video",
-        'expected_map': {
+        "expected_map": {
-            "vid_thumb_url": {
+            "vid_thumb_url": {"type": "text", "index": False},
-                "type": "text",
+            "date_downloaded": {"type": "date"},
-                "index": False
-            },
-            "date_downloaded": {
-                "type": "date"
-            },
             "channel": {
                 "properties": {
                     "channel_id": {
@@ -88,127 +68,92 @@ INDEX_CONFIG = [
                             "keyword": {
                                 "type": "keyword",
                                 "ignore_above": 256,
-                                "normalizer": "to_lower"
+                                "normalizer": "to_lower",
                             },
                             "search_as_you_type": {
                                 "type": "search_as_you_type",
                                 "doc_values": False,
-                                "max_shingle_size": 3
+                                "max_shingle_size": 3,
-                            }
+                            },
-                        }
+                        },
-                    },
-                    "channel_banner_url": {
-                        "type": "keyword",
-                        "index": False
-                    },
-                    "channel_thumb_url": {
-                        "type": "keyword",
-                        "index": False
-                    },
-                    "channel_description": {
-                        "type": "text"
                     },
+                    "channel_banner_url": {"type": "keyword", "index": False},
+                    "channel_thumb_url": {"type": "keyword", "index": False},
+                    "channel_description": {"type": "text"},
                     "channel_last_refresh": {
                         "type": "date",
-                        "format": "epoch_second"
+                        "format": "epoch_second",
-                    }
+                    },
                 }
             },
-            "description": {
+            "description": {"type": "text"},
-                "type": "text"
+            "media_url": {"type": "keyword", "index": False},
-            },
-            "media_url": {
-                "type": "keyword",
-                "index": False
-            },
             "title": {
                 "type": "text",
                 "fields": {
                     "keyword": {
                         "type": "keyword",
                         "ignore_above": 256,
-                        "normalizer": "to_lower"
+                        "normalizer": "to_lower",
                     },
                     "search_as_you_type": {
                         "type": "search_as_you_type",
                         "doc_values": False,
-                        "max_shingle_size": 3
+                        "max_shingle_size": 3,
-                    }
+                    },
-                }
+                },
-            },
-            "vid_last_refresh": {
-                "type": "date"
-            },
-            "youtube_id": {
-                "type": "keyword"
-            },
-            "published": {
-                "type": "date"
             },
+            "vid_last_refresh": {"type": "date"},
+            "youtube_id": {"type": "keyword"},
+            "published": {"type": "date"},
         },
-        'expected_set': {
+        "expected_set": {
             "analysis": {
                 "normalizer": {
-                    "to_lower": {
+                    "to_lower": {"type": "custom", "filter": ["lowercase"]}
-                        "type": "custom",
-                        "filter": ["lowercase"]
-                    }
                 }
             },
-            "number_of_replicas": "0"
+            "number_of_replicas": "0",
-        }
+        },
     },
     {
-        'index_name': 'download',
+        "index_name": "download",
-        'expected_map': {
+        "expected_map": {
-            "timestamp": {
+            "timestamp": {"type": "date"},
-                "type": "date"
+            "channel_id": {"type": "keyword"},
-            },
-            "channel_id": {
-                "type": "keyword"
-            },
             "channel_name": {
                 "type": "text",
                 "fields": {
                     "keyword": {
                         "type": "keyword",
                         "ignore_above": 256,
-                        "normalizer": "to_lower"
+                        "normalizer": "to_lower",
                     }
-                }
+                },
-            },
-            "status": {
-                "type": "keyword"
             },
+            "status": {"type": "keyword"},
             "title": {
                 "type": "text",
                 "fields": {
                     "keyword": {
                         "type": "keyword",
                         "ignore_above": 256,
-                        "normalizer": "to_lower"
+                        "normalizer": "to_lower",
                     }
-                }
+                },
             },
-            "vid_thumb_url": {
+            "vid_thumb_url": {"type": "keyword"},
-                "type": "keyword"
+            "youtube_id": {"type": "keyword"},
-            },
-            "youtube_id": {
-                "type": "keyword"
-            }
         },
-        'expected_set': {
+        "expected_set": {
             "analysis": {
                 "normalizer": {
-                    "to_lower": {
+                    "to_lower": {"type": "custom", "filter": ["lowercase"]}
-                        "type": "custom",
-                        "filter": ["lowercase"]
-                    }
                 }
             },
-            "number_of_replicas": "0"
+            "number_of_replicas": "0",
-        }
+        },
-    }
+    },
 ]
 
 
@@ -218,8 +163,8 @@ class ElasticIndex:
     """
 
     CONFIG = AppConfig().config
-    ES_URL = CONFIG['application']['es_url']
+    ES_URL = CONFIG["application"]["es_url"]
-    HEADERS = {'Content-type': 'application/json'}
+    HEADERS = {"Content-type": "application/json"}
 
     def __init__(self, index_name, expected_map, expected_set):
         self.index_name = index_name
@@ -228,14 +173,14 @@ class ElasticIndex:
         self.exists, self.details = self.index_exists()
 
     def index_exists(self):
-        """ check if index already exists and return mapping if it does """
+        """check if index already exists and return mapping if it does"""
         index_name = self.index_name
-        url = f'{self.ES_URL}/ta_{index_name}'
+        url = f"{self.ES_URL}/ta_{index_name}"
         response = requests.get(url)
         exists = response.ok
 
         if exists:
-            details = response.json()[f'ta_{index_name}']
+            details = response.json()[f"ta_{index_name}"]
         else:
             details = False
 
@@ -260,19 +205,19 @@ class ElasticIndex:
         return False
 
     def validate_mappings(self):
-        """ check if all mappings are as expected """
+        """check if all mappings are as expected"""
 
         expected_map = self.expected_map
-        now_map = self.details['mappings']['properties']
+        now_map = self.details["mappings"]["properties"]
 
         for key, value in expected_map.items():
             # nested
-            if list(value.keys()) == ['properties']:
+            if list(value.keys()) == ["properties"]:
-                for key_n, value_n in value['properties'].items():
+                for key_n, value_n in value["properties"].items():
-                    if key_n not in now_map[key]['properties'].keys():
+                    if key_n not in now_map[key]["properties"].keys():
                         print(key_n, value_n)
                         return True
-                    if not value_n == now_map[key]['properties'][key_n]:
+                    if not value_n == now_map[key]["properties"][key_n]:
                         print(key_n, value_n)
                         return True
 
@@ -289,9 +234,9 @@ class ElasticIndex:
         return False
 
     def validate_settings(self):
-        """ check if all settings are as expected """
+        """check if all settings are as expected"""
 
-        now_set = self.details['settings']['index']
+        now_set = self.details["settings"]["index"]
 
         for key, value in self.expected_set.items():
             if key not in now_set.keys():
@@ -305,53 +250,46 @@ class ElasticIndex:
         return False
 
     def rebuild_index(self):
-        """ rebuild with new mapping """
+        """rebuild with new mapping"""
         # backup
-        self.reindex('backup')
+        self.reindex("backup")
         # delete original
         self.delete_index(backup=False)
         # create new
         self.create_blank()
-        self.reindex('restore')
+        self.reindex("restore")
         # delete backup
         self.delete_index()
 
     def reindex(self, method):
-        """ create on elastic search """
+        """create on elastic search"""
         index_name = self.index_name
-        if method == 'backup':
+        if method == "backup":
-            source = f'ta_{index_name}'
+            source = f"ta_{index_name}"
-            destination = f'ta_{index_name}_backup'
+            destination = f"ta_{index_name}_backup"
-        elif method == 'restore':
+        elif method == "restore":
-            source = f'ta_{index_name}_backup'
+            source = f"ta_{index_name}_backup"
-            destination = f'ta_{index_name}'
+            destination = f"ta_{index_name}"
 
-        query = {
+        query = {"source": {"index": source}, "dest": {"index": destination}}
-            "source": {
-                "index": source
-            },
-            "dest": {
-                "index": destination
-            }
-        }
         data = json.dumps(query)
-        url = self.ES_URL + '/_reindex?refresh=true'
+        url = self.ES_URL + "/_reindex?refresh=true"
         response = requests.post(url=url, data=data, headers=self.HEADERS)
         if not response.ok:
             print(response.text)
 
     def delete_index(self, backup=True):
-        """ delete index passed as argument """
+        """delete index passed as argument"""
         if backup:
-            url = f'{self.ES_URL}/ta_{self.index_name}_backup'
+            url = f"{self.ES_URL}/ta_{self.index_name}_backup"
         else:
-            url = f'{self.ES_URL}/ta_{self.index_name}'
+            url = f"{self.ES_URL}/ta_{self.index_name}"
         response = requests.delete(url)
         if not response.ok:
             print(response.text)
 
     def create_blank(self):
-        """ apply new mapping and settings for blank new index """
+        """apply new mapping and settings for blank new index"""
         expected_map = self.expected_map
         expected_set = self.expected_set
         # stich payload
@@ -361,7 +299,7 @@ class ElasticIndex:
         if expected_map:
             payload.update({"mappings": {"properties": expected_map}})
         # create
-        url = f'{self.ES_URL}/ta_{self.index_name}'
+        url = f"{self.ES_URL}/ta_{self.index_name}"
         data = json.dumps(payload)
         response = requests.put(url=url, data=data, headers=self.HEADERS)
         if not response.ok:
@@ -369,133 +307,202 @@ class ElasticIndex:
 
 
 class ElasticBackup:
-    """ dump index to nd-json files for later bulk import """
+    """dump index to nd-json files for later bulk import"""
 
     def __init__(self, index_config):
         self.config = AppConfig().config
         self.index_config = index_config
-        self.timestamp = datetime.now().strftime('%Y%m%d')
+        self.timestamp = datetime.now().strftime("%Y%m%d")
+        self.backup_files = []
 
     def get_all_documents(self, index_name):
-        """ export all documents of a single index """
+        """export all documents of a single index"""
-        headers = {'Content-type': 'application/json'}
+        headers = {"Content-type": "application/json"}
-        es_url = self.config['application']['es_url']
+        es_url = self.config["application"]["es_url"]
         # get PIT ID
-        url = f'{es_url}/ta_{index_name}/_pit?keep_alive=1m'
+        url = f"{es_url}/ta_{index_name}/_pit?keep_alive=1m"
         response = requests.post(url)
         json_data = json.loads(response.text)
-        pit_id = json_data['id']
+        pit_id = json_data["id"]
         # build query
         data = {
             "query": {"match_all": {}},
-            "size": 100, "pit": {"id": pit_id, "keep_alive": "1m"},
+            "size": 100,
-            "sort": [ {"_id": {"order": "asc"}} ]
+            "pit": {"id": pit_id, "keep_alive": "1m"},
+            "sort": [{"_id": {"order": "asc"}}],
         }
         query_str = json.dumps(data)
-        url = es_url + '/_search'
+        url = es_url + "/_search"
         # loop until nothing left
         all_results = []
         while True:
             response = requests.get(url, data=query_str, headers=headers)
             json_data = json.loads(response.text)
-            all_hits = json_data['hits']['hits']
+            all_hits = json_data["hits"]["hits"]
             if all_hits:
                 for hit in all_hits:
-                    search_after = hit['sort']
+                    search_after = hit["sort"]
                     all_results.append(hit)
                 # update search_after with last hit data
-                data['search_after'] = search_after
+                data["search_after"] = search_after
                 query_str = json.dumps(data)
             else:
                 break
         # clean up PIT
         query_str = json.dumps({"id": pit_id})
-        requests.delete(es_url + '/_pit', data=query_str, headers=headers)
+        requests.delete(es_url + "/_pit", data=query_str, headers=headers)
 
         return all_results
 
     @staticmethod
     def build_bulk(all_results):
-        """ build bulk query data from all_results """
+        """build bulk query data from all_results"""
         bulk_list = []
 
         for document in all_results:
-            document_id = document['_id']
+            document_id = document["_id"]
-            es_index = document['_index']
+            es_index = document["_index"]
-            action = { "index" : { "_index": es_index, "_id": document_id } }
+            action = {"index": {"_index": es_index, "_id": document_id}}
-            source = document['_source']
+            source = document["_source"]
             bulk_list.append(json.dumps(action))
             bulk_list.append(json.dumps(source))
 
         # add last newline
-        bulk_list.append('\n')
+        bulk_list.append("\n")
-        file_content = '\n'.join(bulk_list)
+        file_content = "\n".join(bulk_list)
 
         return file_content
 
-    def write_json_file(self, file_content, index_name):
+    def write_es_json(self, file_content, index_name):
-        """ write json file to disk """
+        """write nd-json file for es _bulk API to disk"""
-        cache_dir = self.config['application']['cache_dir']
+        cache_dir = self.config["application"]["cache_dir"]
-        file_name = f'ta_{index_name}-{self.timestamp}.json'
+        file_name = f"es_{index_name}-{self.timestamp}.json"
-        file_path = os.path.join(cache_dir, file_name)
+        file_path = os.path.join(cache_dir, "backup", file_name)
-        with open(file_path, 'w', encoding='utf-8') as f:
+        with open(file_path, "w", encoding="utf-8") as f:
             f.write(file_content)
 
+        self.backup_files.append(file_path)
+
+    def write_ta_json(self, all_results, index_name):
+        """write generic json file to disk"""
+        cache_dir = self.config["application"]["cache_dir"]
+        file_name = f"ta_{index_name}-{self.timestamp}.json"
+        file_path = os.path.join(cache_dir, "backup", file_name)
+        to_write = [i["_source"] for i in all_results]
+        file_content = json.dumps(to_write)
+        with open(file_path, "w", encoding="utf-8") as f:
+            f.write(file_content)
+
+        self.backup_files.append(file_path)
+
+    def zip_it(self):
+        """pack it up into single zip file"""
+        cache_dir = self.config["application"]["cache_dir"]
+        file_name = f"ta_backup-{self.timestamp}.zip"
+        backup_folder = os.path.join(cache_dir, "backup")
+        backup_file = os.path.join(backup_folder, file_name)
+
+        with zipfile.ZipFile(
+            backup_file, "w", compression=zipfile.ZIP_DEFLATED
+        ) as zip_f:
+            for backup_file in self.backup_files:
+                zip_f.write(backup_file, os.path.basename(backup_file))
+
+        # cleanup
+        for backup_file in self.backup_files:
+            os.remove(backup_file)
+
     def post_bulk_restore(self, file_name):
-        """ send bulk to es """
+        """send bulk to es"""
-        cache_dir = self.config['application']['cache_dir']
+        cache_dir = self.config["application"]["cache_dir"]
-        es_url = self.config['application']['es_url']
+        es_url = self.config["application"]["es_url"]
-        headers = {'Content-type': 'application/x-ndjson'}
+        headers = {"Content-type": "application/x-ndjson"}
         file_path = os.path.join(cache_dir, file_name)
 
-        with open(file_path, 'r', encoding='utf-8') as f:
+        with open(file_path, "r", encoding="utf-8") as f:
             query_str = f.read()
 
-        url = es_url + '/_bulk'
+        if not query_str.strip():
+            return
+
+        url = es_url + "/_bulk"
         request = requests.post(url, data=query_str, headers=headers)
         if not request.ok:
             print(request.text)
 
-    def restore_from_file(self):
+    def unpack_zip_backup(self):
-        """ restore all available backup files """
+        """extract backup zip and return filelist"""
-        cache_dir = self.config['application']['cache_dir']
+        cache_dir = self.config["application"]["cache_dir"]
+        backup_dir = os.path.join(cache_dir, "backup")
+        backup_files = os.listdir(backup_dir)
+        all_backup_files = ignore_filelist(backup_files)
         all_available_backups = [
-            i for i in os.listdir(cache_dir) if
+            i
-            i.startswith('ta_') and i.endswith('.json')
+            for i in all_backup_files
+            if i.startswith("ta_") and i.endswith(".zip")
         ]
-        for file_name in all_available_backups:
+        all_available_backups.sort()
+        newest_backup = all_available_backups[-1]
+        file_path = os.path.join(backup_dir, newest_backup)
+
+        with zipfile.ZipFile(file_path, "r") as z:
+            zip_content = z.namelist()
+            z.extractall(backup_dir)
+
+        return zip_content
+
+    def restore_json_files(self, zip_content):
+        """go through the unpacked files and restore"""
+
+        cache_dir = self.config["application"]["cache_dir"]
+        backup_dir = os.path.join(cache_dir, "backup")
+
+        for json_f in zip_content:
+
+            file_name = os.path.join(backup_dir, json_f)
+
+            if not json_f.startswith("es_") or not json_f.endswith(".json"):
+                os.remove(file_name)
+                continue
+
+            print("restoring: " + json_f)
             self.post_bulk_restore(file_name)
+            os.remove(file_name)
 
 
 def backup_all_indexes():
-    """ backup all es indexes to disk """
+    """backup all es indexes to disk"""
     backup_handler = ElasticBackup(INDEX_CONFIG)
 
     for index in backup_handler.index_config:
-        index_name = index['index_name']
+        index_name = index["index_name"]
         all_results = backup_handler.get_all_documents(index_name)
         file_content = backup_handler.build_bulk(all_results)
-        backup_handler.write_json_file(file_content, index_name)
+        backup_handler.write_es_json(file_content, index_name)
+        backup_handler.write_ta_json(all_results, index_name)
+
+    backup_handler.zip_it()
 
 
 def restore_from_backup():
-    """ restore indexes from backup file """
+    """restore indexes from backup file"""
     # delete
     index_check(force_restore=True)
     # recreate
     backup_handler = ElasticBackup(INDEX_CONFIG)
-    backup_handler.restore_from_file()
+    zip_content = backup_handler.unpack_zip_backup()
+    backup_handler.restore_json_files(zip_content)
 
 
 def index_check(force_restore=False):
-    """ check if all indexes are created and have correct mapping """
+    """check if all indexes are created and have correct mapping"""
 
     backed_up = False
 
     for index in INDEX_CONFIG:
-        index_name = index['index_name']
+        index_name = index["index_name"]
-        expected_map = index['expected_map']
+        expected_map = index["expected_map"]
-        expected_set = index['expected_set']
+        expected_set = index["expected_set"]
         handler = ElasticIndex(index_name, expected_map, expected_set)
         # force restore
         if force_restore:
@@ -505,7 +512,7 @@ def index_check(force_restore=False):
 
         # create new
         if not handler.exists:
-            print(f'create new blank index with name ta_{index_name}...')
+            print(f"create new blank index with name ta_{index_name}...")
             handler.create_blank()
             continue
 
@@ -514,13 +521,13 @@ def index_check(force_restore=False):
         if rebuild:
             # make backup before rebuild
             if not backed_up:
-                print('running backup first')
+                print("running backup first")
                 backup_all_indexes()
                 backed_up = True
 
-            print(f'applying new mappings to index ta_{index_name}...')
+            print(f"applying new mappings to index ta_{index_name}...")
             handler.rebuild_index()
             continue
 
         # else all good
-        print(f'ta_{index_name} index is created and up to date...')
+        print(f"ta_{index_name} index is created and up to date...")

tubearchivist/home/src/reindex.py

@@ -8,38 +8,32 @@ Functionality:
 import json
 import os
 import re
-import subprocess
 import shutil
-
+import subprocess
 from datetime import datetime
-from time import sleep
 from math import ceil
+from time import sleep
 
 import requests
-
-from home.src.download import ChannelSubscription, PendingList, VideoDownloader
 from home.src.config import AppConfig
-from home.src.index import (
+from home.src.download import ChannelSubscription, PendingList, VideoDownloader
-    YoutubeChannel,
-    YoutubeVideo,
-    index_new_video
-)
 from home.src.helper import (
-    get_total_hits,
+    RedisArchivist,
     clean_string,
-    set_message,
+    get_total_hits,
-    get_message
+    ignore_filelist,
 )
+from home.src.index import YoutubeChannel, YoutubeVideo, index_new_video
 
 
 class Reindex:
-    """ check for outdated documents and refresh data from youtube """
+    """check for outdated documents and refresh data from youtube"""
 
     def __init__(self):
         # config
         config = AppConfig().config
-        self.sleep_interval = config['downloads']['sleep_interval']
+        self.sleep_interval = config["downloads"]["sleep_interval"]
-        self.es_url = config['application']['es_url']
+        self.es_url = config["application"]["es_url"]
         self.refresh_interval = 90
         # scan
         self.video_daily, self.channel_daily = self.get_daily()
@@ -47,20 +41,18 @@ class Reindex:
         self.all_channel_ids = False
 
     def get_daily(self):
-        """ get daily refresh values """
+        """get daily refresh values"""
-        total_videos = get_total_hits(
+        total_videos = get_total_hits("ta_video", self.es_url, "active")
-            'ta_video', self.es_url, 'active'
-        )
         video_daily = ceil(total_videos / self.refresh_interval * 1.2)
         total_channels = get_total_hits(
-            'ta_channel', self.es_url, 'channel_active'
+            "ta_channel", self.es_url, "channel_active"
         )
         channel_daily = ceil(total_channels / self.refresh_interval * 1.2)
         return (video_daily, channel_daily)
 
     def get_outdated_vids(self):
-        """ get daily videos to refresh """
+        """get daily videos to refresh"""
-        headers = {'Content-type': 'application/json'}
+        headers = {"Content-type": "application/json"}
         now = int(datetime.now().strftime("%s"))
         now_3m = now - 3 * 30 * 24 * 60 * 60
         size = self.video_daily
@@ -70,24 +62,25 @@ class Reindex:
                 "bool": {
                     "must": [
                         {"match": {"active": True}},
-                        {"range": {"vid_last_refresh": {"lte": now_3m}}}
+                        {"range": {"vid_last_refresh": {"lte": now_3m}}},
                     ]
                 }
             },
-            "sort": [{"vid_last_refresh": {"order": "asc"}}], "_source": False
+            "sort": [{"vid_last_refresh": {"order": "asc"}}],
+            "_source": False,
         }
         query_str = json.dumps(data)
-        url = self.es_url + '/ta_video/_search'
+        url = self.es_url + "/ta_video/_search"
         response = requests.get(url, data=query_str, headers=headers)
         if not response.ok:
             print(response.text)
         response_dict = json.loads(response.text)
-        all_youtube_ids = [i['_id'] for i in response_dict['hits']['hits']]
+        all_youtube_ids = [i["_id"] for i in response_dict["hits"]["hits"]]
         return all_youtube_ids
 
     def get_outdated_channels(self):
-        """ get daily channels to refresh """
+        """get daily channels to refresh"""
-        headers = {'Content-type': 'application/json'}
+        headers = {"Content-type": "application/json"}
         now = int(datetime.now().strftime("%s"))
         now_3m = now - 3 * 30 * 24 * 60 * 60
         size = self.channel_daily
@@ -97,52 +90,50 @@ class Reindex:
                 "bool": {
                     "must": [
                         {"match": {"channel_active": True}},
-                        {"range": {"channel_last_refresh": {"lte": now_3m}}}
+                        {"range": {"channel_last_refresh": {"lte": now_3m}}},
                     ]
                 }
             },
             "sort": [{"channel_last_refresh": {"order": "asc"}}],
-            "_source": False
+            "_source": False,
         }
         query_str = json.dumps(data)
-        url = self.es_url + '/ta_channel/_search'
+        url = self.es_url + "/ta_channel/_search"
         response = requests.get(url, data=query_str, headers=headers)
         if not response.ok:
             print(response.text)
         response_dict = json.loads(response.text)
-        all_channel_ids = [i['_id'] for i in response_dict['hits']['hits']]
+        all_channel_ids = [i["_id"] for i in response_dict["hits"]["hits"]]
         return all_channel_ids
 
     def check_outdated(self):
-        """ add missing vids and channels """
+        """add missing vids and channels"""
         self.all_youtube_ids = self.get_outdated_vids()
         self.all_channel_ids = self.get_outdated_channels()
 
     def rescrape_all_channels(self):
-        """ sync new data from channel to all matching videos """
+        """sync new data from channel to all matching videos"""
         sleep_interval = self.sleep_interval
         channel_sub_handler = ChannelSubscription()
-        all_channels = channel_sub_handler.get_channels(
+        all_channels = channel_sub_handler.get_channels(subscribed_only=False)
-            subscribed_only=False
+        all_channel_ids = [i["channel_id"] for i in all_channels]
-        )
-        all_channel_ids = [i['channel_id'] for i in all_channels]
 
         counter = 1
         for channel_id in all_channel_ids:
-            message = f'Progress: {counter}/{len(all_channels)}'
+            message = f"Progress: {counter}/{len(all_channels)}"
             mess_dict = {
                 "status": "scraping",
                 "level": "info",
                 "title": "Scraping all youtube channels",
-                "message": message
+                "message": message,
             }
-            set_message('progress:download', mess_dict)
+            RedisArchivist().set_message("progress:download", mess_dict)
             channel_index = YoutubeChannel(channel_id)
-            subscribed = channel_index.channel_dict['channel_subscribed']
+            subscribed = channel_index.channel_dict["channel_subscribed"]
             channel_index.channel_dict = channel_index.build_channel_dict(
                 scrape=True
             )
-            channel_index.channel_dict['channel_subscribed'] = subscribed
+            channel_index.channel_dict["channel_subscribed"] = subscribed
             channel_index.upload_to_es()
             channel_index.sync_to_videos()
             counter = counter + 1
@@ -151,7 +142,7 @@ class Reindex:
 
     @staticmethod
     def reindex_single_video(youtube_id):
-        """ refresh data for single video """
+        """refresh data for single video"""
         vid_handler = YoutubeVideo(youtube_id)
         if not vid_handler.vid_dict:
             # stop if deactivated
@@ -159,42 +150,42 @@ class Reindex:
             return
 
         es_vid_dict = vid_handler.get_es_data()
-        player = es_vid_dict['_source']['player']
+        player = es_vid_dict["_source"]["player"]
-        date_downloaded = es_vid_dict['_source']['date_downloaded']
+        date_downloaded = es_vid_dict["_source"]["date_downloaded"]
-        channel_dict = es_vid_dict['_source']['channel']
+        channel_dict = es_vid_dict["_source"]["channel"]
-        channel_name = channel_dict['channel_name']
+        channel_name = channel_dict["channel_name"]
         vid_handler.build_file_path(channel_name)
         # add to vid_dict
-        vid_handler.vid_dict['player'] = player
+        vid_handler.vid_dict["player"] = player
-        vid_handler.vid_dict['date_downloaded'] = date_downloaded
+        vid_handler.vid_dict["date_downloaded"] = date_downloaded
-        vid_handler.vid_dict['channel'] = channel_dict
+        vid_handler.vid_dict["channel"] = channel_dict
         # update
         vid_handler.upload_to_es()
         vid_handler.delete_cache()
 
     @staticmethod
     def reindex_single_channel(channel_id):
-        """ refresh channel data and sync to videos """
+        """refresh channel data and sync to videos"""
         channel_handler = YoutubeChannel(channel_id)
-        subscribed = channel_handler.channel_dict['channel_subscribed']
+        subscribed = channel_handler.channel_dict["channel_subscribed"]
         channel_handler.channel_dict = channel_handler.build_channel_dict(
             scrape=True
         )
-        channel_handler.channel_dict['channel_subscribed'] = subscribed
+        channel_handler.channel_dict["channel_subscribed"] = subscribed
         channel_handler.upload_to_es()
         channel_handler.sync_to_videos()
         channel_handler.clear_cache()
 
     def reindex(self):
-        """ reindex what's needed """
+        """reindex what's needed"""
         # videos
-        print(f'reindexing {len(self.all_youtube_ids)} videos')
+        print(f"reindexing {len(self.all_youtube_ids)} videos")
         for youtube_id in self.all_youtube_ids:
             self.reindex_single_video(youtube_id)
             if self.sleep_interval:
                 sleep(self.sleep_interval)
         # channels
-        print(f'reindexing {len(self.all_channel_ids)} channels')
+        print(f"reindexing {len(self.all_channel_ids)} channels")
         for channel_id in self.all_channel_ids:
             self.reindex_single_channel(channel_id)
             if self.sleep_interval:
@@ -202,28 +193,31 @@ class Reindex:
 
 
 class FilesystemScanner:
-    """ handle scanning and fixing from filesystem """
+    """handle scanning and fixing from filesystem"""
 
     CONFIG = AppConfig().config
-    ES_URL = CONFIG['application']['es_url']
+    ES_URL = CONFIG["application"]["es_url"]
-    VIDEOS = CONFIG['application']['videos']
+    VIDEOS = CONFIG["application"]["videos"]
 
     def __init__(self):
         self.all_downloaded = self.get_all_downloaded()
         self.all_indexed = self.get_all_indexed()
-        self.missmatch = None
+        self.mismatch = None
         self.to_rename = None
         self.to_index = None
         self.to_delete = None
 
     def get_all_downloaded(self):
-        """ get a list of all video files downloaded """
+        """get a list of all video files downloaded"""
-        all_channels = os.listdir(self.VIDEOS)
+        channels = os.listdir(self.VIDEOS)
+        all_channels = ignore_filelist(channels)
         all_channels.sort()
         all_downloaded = []
         for channel_name in all_channels:
             channel_path = os.path.join(self.VIDEOS, channel_name)
-            for video in os.listdir(channel_path):
+            videos = os.listdir(channel_path)
+            all_videos = ignore_filelist(videos)
+            for video in all_videos:
                 youtube_id = video[9:20]
                 all_downloaded.append((channel_name, video, youtube_id))
 
@@ -231,26 +225,26 @@ class FilesystemScanner:
 
     @staticmethod
     def get_all_indexed():
-        """ get a list of all indexed videos """
+        """get a list of all indexed videos"""
         index_handler = PendingList()
         all_indexed_raw = index_handler.get_all_indexed()
         all_indexed = []
         for video in all_indexed_raw:
-            youtube_id = video['_id']
+            youtube_id = video["_id"]
-            media_url = video['_source']['media_url']
+            media_url = video["_source"]["media_url"]
-            published = video['_source']['published']
+            published = video["_source"]["published"]
-            title = video['_source']['title']
+            title = video["_source"]["title"]
             all_indexed.append((youtube_id, media_url, published, title))
         return all_indexed
 
     def list_comarison(self):
-        """ compare the lists to figure out what to do """
+        """compare the lists to figure out what to do"""
         self.find_unindexed()
         self.find_missing()
         self.find_bad_media_url()
 
     def find_unindexed(self):
-        """ find video files without a matching document indexed """
+        """find video files without a matching document indexed"""
         all_indexed_ids = [i[0] for i in self.all_indexed]
         to_index = []
         for downloaded in self.all_downloaded:
@@ -260,7 +254,7 @@ class FilesystemScanner:
         self.to_index = to_index
 
     def find_missing(self):
-        """ find indexed videos without matching media file """
+        """find indexed videos without matching media file"""
         all_downloaded_ids = [i[2] for i in self.all_downloaded]
         to_delete = []
         for video in self.all_indexed:
@@ -271,7 +265,7 @@ class FilesystemScanner:
         self.to_delete = to_delete
 
     def find_bad_media_url(self):
-        """ rename media files not matching the indexed title """
+        """rename media files not matching the indexed title"""
         to_fix = []
         to_rename = []
         for downloaded in self.all_downloaded:
@@ -282,8 +276,8 @@ class FilesystemScanner:
                 if indexed_id == downloaded_id:
                     # found it
                     title_c = clean_string(title)
-                    pub = published.replace('-', '')
+                    pub = published.replace("-", "")
-                    expected_filename = f'{pub}_{indexed_id}_{title_c}.mp4'
+                    expected_filename = f"{pub}_{indexed_id}_{title_c}.mp4"
                     new_url = os.path.join(channel, expected_filename)
                     if expected_filename != filename:
                         # file to rename
@@ -296,83 +290,84 @@ class FilesystemScanner:
 
                     break
 
-        self.missmatch = to_fix
+        self.mismatch = to_fix
         self.to_rename = to_rename
 
     def rename_files(self):
-        """ rename media files as identified by find_bad_media_url """
+        """rename media files as identified by find_bad_media_url"""
         for bad_filename in self.to_rename:
             channel, filename, expected_filename = bad_filename
             old_path = os.path.join(self.VIDEOS, channel, filename)
             new_path = os.path.join(self.VIDEOS, channel, expected_filename)
             os.rename(old_path, new_path)
 
-    def send_missmatch_bulk(self):
+    def send_mismatch_bulk(self):
-        """ build bulk update """
+        """build bulk update"""
         bulk_list = []
-        for video_missmatch in self.missmatch:
+        for video_mismatch in self.mismatch:
-            youtube_id, media_url = video_missmatch
+            youtube_id, media_url = video_mismatch
-            action = {"update": {"_id": youtube_id, "_index": 'ta_video'}}
+            action = {"update": {"_id": youtube_id, "_index": "ta_video"}}
             source = {"doc": {"media_url": media_url}}
             bulk_list.append(json.dumps(action))
             bulk_list.append(json.dumps(source))
         # add last newline
-        bulk_list.append('\n')
+        bulk_list.append("\n")
-        query_str = '\n'.join(bulk_list)
+        query_str = "\n".join(bulk_list)
         # make the call
-        headers = {'Content-type': 'application/x-ndjson'}
+        headers = {"Content-type": "application/x-ndjson"}
-        url = self.ES_URL + '/_bulk'
+        url = self.ES_URL + "/_bulk"
         request = requests.post(url, data=query_str, headers=headers)
         if not request.ok:
             print(request.text)
 
     def delete_from_index(self):
-        """ find indexed but deleted mediafile """
+        """find indexed but deleted mediafile"""
         for indexed in self.to_delete:
             youtube_id, _ = indexed
-            url = self.ES_URL + '/ta_video/_doc/' + youtube_id
+            url = self.ES_URL + "/ta_video/_doc/" + youtube_id
             request = requests.delete(url)
             if not request.ok:
                 print(request.text)
 
 
 class ManualImport:
-    """ import and indexing existing video files """
+    """import and indexing existing video files"""
 
     CONFIG = AppConfig().config
-    CACHE_DIR = CONFIG['application']['cache_dir']
+    CACHE_DIR = CONFIG["application"]["cache_dir"]
-    IMPORT_DIR = os.path.join(CACHE_DIR, 'import')
+    IMPORT_DIR = os.path.join(CACHE_DIR, "import")
 
     def __init__(self):
         self.identified = self.import_folder_parser()
 
     def import_folder_parser(self):
-        """ detect files in import folder """
+        """detect files in import folder"""
-
+        import_files = os.listdir(self.IMPORT_DIR)
-        to_import = os.listdir(self.IMPORT_DIR)
+        to_import = ignore_filelist(import_files)
         to_import.sort()
-        video_files = [i for i in to_import if not i.endswith('.json')]
+        video_files = [i for i in to_import if not i.endswith(".json")]
 
         identified = []
 
         for file_path in video_files:
 
-            file_dict = {'video_file': file_path}
+            file_dict = {"video_file": file_path}
             file_name, _ = os.path.splitext(file_path)
 
             matching_json = [
-                i for i in to_import if i.startswith(file_name)
+                i
-                and i.endswith('.json')
+                for i in to_import
+                if i.startswith(file_name) and i.endswith(".json")
             ]
             if matching_json:
                 json_file = matching_json[0]
                 youtube_id = self.extract_id_from_json(json_file)
-                file_dict.update({'json_file': json_file})
+                file_dict.update({"json_file": json_file})
             else:
                 youtube_id = self.extract_id_from_filename(file_name)
-                file_dict.update({'json_file': False})
+                file_dict.update({"json_file": False})
 
-            file_dict.update({'youtube_id': youtube_id})
+            file_dict.update({"youtube_id": youtube_id})
             identified.append(file_dict)
 
         return identified
@@ -383,33 +378,33 @@ class ManualImport:
         look at the file name for the youtube id
         expects filename ending in [<youtube_id>].<ext>
         """
-        id_search = re.search(r'\[([a-zA-Z0-9_-]{11})\]$', file_name)
+        id_search = re.search(r"\[([a-zA-Z0-9_-]{11})\]$", file_name)
         if id_search:
             youtube_id = id_search.group(1)
             return youtube_id
 
-        print('failed to extract youtube id for: ' + file_name)
+        print("failed to extract youtube id for: " + file_name)
         raise Exception
 
     def extract_id_from_json(self, json_file):
-        """ open json file and extract id """
+        """open json file and extract id"""
-        json_path = os.path.join(self.CACHE_DIR, 'import', json_file)
+        json_path = os.path.join(self.CACHE_DIR, "import", json_file)
-        with open(json_path, 'r', encoding='utf-8') as f:
+        with open(json_path, "r", encoding="utf-8") as f:
             json_content = f.read()
 
-        youtube_id = json.loads(json_content)['id']
+        youtube_id = json.loads(json_content)["id"]
 
         return youtube_id
 
     def process_import(self):
-        """ go through identified media files """
+        """go through identified media files"""
 
         for media_file in self.identified:
-            json_file = media_file['json_file']
+            json_file = media_file["json_file"]
-            video_file = media_file['video_file']
+            video_file = media_file["video_file"]
-            youtube_id = media_file['youtube_id']
+            youtube_id = media_file["youtube_id"]
 
-            video_path = os.path.join(self.CACHE_DIR, 'import', video_file)
+            video_path = os.path.join(self.CACHE_DIR, "import", video_file)
 
             self.move_to_cache(video_path, youtube_id)
 
@@ -421,41 +416,49 @@ class ManualImport:
             if os.path.exists(video_path):
                 os.remove(video_path)
             if json_file:
-                json_path = os.path.join(self.CACHE_DIR, 'import', json_file)
+                json_path = os.path.join(self.CACHE_DIR, "import", json_file)
                 os.remove(json_path)
 
     def move_to_cache(self, video_path, youtube_id):
-        """ move identified video file to cache, convert to mp4 """
+        """move identified video file to cache, convert to mp4"""
         file_name = os.path.split(video_path)[-1]
         video_file, ext = os.path.splitext(file_name)
 
         # make sure youtube_id is in filename
-        if not youtube_id in video_file:
+        if youtube_id not in video_file:
-            video_file = f'{video_file}_{youtube_id}'
+            video_file = f"{video_file}_{youtube_id}"
 
         # move, convert if needed
-        if ext == '.mp4':
+        if ext == ".mp4":
             new_file = video_file + ext
-            dest_path = os.path.join(self.CACHE_DIR, 'download', new_file)
+            dest_path = os.path.join(self.CACHE_DIR, "download", new_file)
             shutil.move(video_path, dest_path)
         else:
-            print(f'processing with ffmpeg: {video_file}')
+            print(f"processing with ffmpeg: {video_file}")
-            new_file = video_file + '.mp4'
+            new_file = video_file + ".mp4"
-            dest_path = os.path.join(self.CACHE_DIR, 'download', new_file)
+            dest_path = os.path.join(self.CACHE_DIR, "download", new_file)
             subprocess.run(
-                ["ffmpeg", "-i", video_path, dest_path,
+                [
-                "-loglevel", "warning", "-stats"], check=True
+                    "ffmpeg",
+                    "-i",
+                    video_path,
+                    dest_path,
+                    "-loglevel",
+                    "warning",
+                    "-stats",
+                ],
+                check=True,
             )
 
 
 def scan_filesystem():
-    """ grouped function to delete and update index """
+    """grouped function to delete and update index"""
     filesystem_handler = FilesystemScanner()
     filesystem_handler.list_comarison()
     if filesystem_handler.to_rename:
         filesystem_handler.rename_files()
-    if filesystem_handler.missmatch:
+    if filesystem_handler.mismatch:
-        filesystem_handler.send_missmatch_bulk()
+        filesystem_handler.send_mismatch_bulk()
     if filesystem_handler.to_delete:
         filesystem_handler.delete_from_index()
     if filesystem_handler.to_index:
@@ -465,10 +468,10 @@ def scan_filesystem():
 
 
 def reindex_old_documents():
-    """ daily refresh of old documents """
+    """daily refresh of old documents"""
     # check needed last run
     now = int(datetime.now().strftime("%s"))
-    last_reindex = get_message('last_reindex')
+    last_reindex = RedisArchivist().get_message("last_reindex")
     if isinstance(last_reindex, int) and now - last_reindex < 60 * 60 * 24:
         return
     # continue if needed
@@ -476,4 +479,4 @@ def reindex_old_documents():
     reindex_handler.check_outdated()
     reindex_handler.reindex()
     # set timestamp
-    set_message('last_reindex', now, expire=False)
+    RedisArchivist().set_message("last_reindex", now, expire=False)

tubearchivist/home/src/searching.py

@@ -9,21 +9,19 @@ Functionality:
 import math
 import os
 import urllib.parse
-
 from datetime import datetime
 
 import requests
-
-from PIL import Image
-
 from home.src.config import AppConfig
+from home.src.helper import ignore_filelist
+from PIL import Image
 
 
 class SearchHandler:
-    """ search elastic search """
+    """search elastic search"""
 
     CONFIG = AppConfig().config
-    CACHE_DIR = CONFIG['application']['cache_dir']
+    CACHE_DIR = CONFIG["application"]["cache_dir"]
 
     def __init__(self, url, data, cache=True):
         self.max_hits = None
@@ -32,15 +30,15 @@ class SearchHandler:
         self.cache = cache
 
     def get_data(self):
-        """ get the data """
+        """get the data"""
         if self.data:
             response = requests.get(self.url, json=self.data).json()
         else:
             response = requests.get(self.url).json()
 
-        if 'hits' in response.keys():
+        if "hits" in response.keys():
-            self.max_hits = response['hits']['total']['value']
+            self.max_hits = response["hits"]["total"]["value"]
-            return_value = response['hits']['hits']
+            return_value = response["hits"]["hits"]
         else:
             # simulate list for single result to reuse rest of class
             return_value = [response]
@@ -53,13 +51,13 @@ class SearchHandler:
         all_channels = []
         for idx, hit in enumerate(return_value):
             return_value[idx] = self.hit_cleanup(hit)
-            if hit['_index'] == 'ta_video':
+            if hit["_index"] == "ta_video":
                 video_dict, channel_dict = self.vid_cache_link(hit)
                 if video_dict not in all_videos:
                     all_videos.append(video_dict)
                 if channel_dict not in all_channels:
                     all_channels.append(channel_dict)
-            elif hit['_index'] == 'ta_channel':
+            elif hit["_index"] == "ta_channel":
                 channel_dict = self.channel_cache_link(hit)
                 if channel_dict not in all_channels:
                     all_channels.append(channel_dict)
@@ -72,52 +70,50 @@ class SearchHandler:
 
     @staticmethod
     def vid_cache_link(hit):
-        """ download thumbnails into chache """
+        """download thumbnails into cache"""
-        vid_thumb = hit['source']['vid_thumb_url']
+        vid_thumb = hit["source"]["vid_thumb_url"]
-        youtube_id = hit['source']['youtube_id']
+        youtube_id = hit["source"]["youtube_id"]
-        channel_id_hit = hit['source']['channel']['channel_id']
+        channel_id_hit = hit["source"]["channel"]["channel_id"]
-        chan_thumb = hit['source']['channel']['channel_thumb_url']
+        chan_thumb = hit["source"]["channel"]["channel_thumb_url"]
         try:
-            chan_banner = hit['source']['channel']['channel_banner_url']
+            chan_banner = hit["source"]["channel"]["channel_banner_url"]
         except KeyError:
             chan_banner = False
-        video_dict = {
+        video_dict = {"youtube_id": youtube_id, "vid_thumb": vid_thumb}
-            'youtube_id': youtube_id,
-            'vid_thumb': vid_thumb
-        }
         channel_dict = {
-            'channel_id': channel_id_hit,
+            "channel_id": channel_id_hit,
-            'chan_thumb': chan_thumb,
+            "chan_thumb": chan_thumb,
-            'chan_banner': chan_banner
+            "chan_banner": chan_banner,
         }
         return video_dict, channel_dict
 
     @staticmethod
     def channel_cache_link(hit):
-        """ build channel thumb links """
+        """build channel thumb links"""
-        channel_id_hit = hit['source']['channel_id']
+        channel_id_hit = hit["source"]["channel_id"]
-        chan_thumb = hit['source']['channel_thumb_url']
+        chan_thumb = hit["source"]["channel_thumb_url"]
         try:
-            chan_banner = hit['source']['channel_banner_url']
+            chan_banner = hit["source"]["channel_banner_url"]
         except KeyError:
             chan_banner = False
         channel_dict = {
-            'channel_id': channel_id_hit,
+            "channel_id": channel_id_hit,
-            'chan_thumb': chan_thumb,
+            "chan_thumb": chan_thumb,
-            'chan_banner': chan_banner
+            "chan_banner": chan_banner,
         }
         return channel_dict
 
     def cache_dl_vids(self, all_videos):
-        """ video thumbs links for cache """
+        """video thumbs links for cache"""
-        vid_cache = os.path.join(self.CACHE_DIR, 'videos')
+        vid_cache = os.path.join(self.CACHE_DIR, "videos")
-        all_vid_cached = os.listdir(vid_cache)
+        vid_cached = os.listdir(vid_cache)
+        all_vid_cached = ignore_filelist(vid_cached)
         # videos
         for video_dict in all_videos:
-            youtube_id = video_dict['youtube_id']
+            youtube_id = video_dict["youtube_id"]
-            if not youtube_id + '.jpg' in all_vid_cached:
+            if not youtube_id + ".jpg" in all_vid_cached:
-                cache_path = os.path.join(vid_cache, youtube_id + '.jpg')
+                cache_path = os.path.join(vid_cache, youtube_id + ".jpg")
-                thumb_url = video_dict['vid_thumb']
+                thumb_url = video_dict["vid_thumb"]
                 img_raw = requests.get(thumb_url, stream=True).raw
                 img = Image.open(img_raw)
                 width, height = img.size
@@ -128,66 +124,101 @@ class SearchHandler:
                 img.convert("RGB").save(cache_path)
 
     def cache_dl_chan(self, all_channels):
-        """ download channel thumbs """
+        """download channel thumbs"""
-        chan_cache = os.path.join(self.CACHE_DIR, 'channels')
+        chan_cache = os.path.join(self.CACHE_DIR, "channels")
-        all_chan_cached = os.listdir(chan_cache)
+        chan_cached = os.listdir(chan_cache)
+        all_chan_cached = ignore_filelist(chan_cached)
         for channel_dict in all_channels:
-            channel_id_cache = channel_dict['channel_id']
+            channel_id_cache = channel_dict["channel_id"]
-            channel_banner_url = channel_dict['chan_banner']
+            channel_banner_url = channel_dict["chan_banner"]
-            channel_banner = channel_id_cache + '_banner.jpg'
+            channel_banner = channel_id_cache + "_banner.jpg"
-            channel_thumb_url = channel_dict['chan_thumb']
+            channel_thumb_url = channel_dict["chan_thumb"]
-            channel_thumb = channel_id_cache + '_thumb.jpg'
+            channel_thumb = channel_id_cache + "_thumb.jpg"
             # thumb
             if channel_thumb_url and channel_thumb not in all_chan_cached:
                 cache_path = os.path.join(chan_cache, channel_thumb)
                 img_raw = requests.get(channel_thumb_url, stream=True).content
-                with open(cache_path, 'wb') as f:
+                with open(cache_path, "wb") as f:
                     f.write(img_raw)
             # banner
             if channel_banner_url and channel_banner not in all_chan_cached:
                 cache_path = os.path.join(chan_cache, channel_banner)
                 img_raw = requests.get(channel_banner_url, stream=True).content
-                with open(cache_path, 'wb') as f:
+                with open(cache_path, "wb") as f:
                     f.write(img_raw)
 
     @staticmethod
     def hit_cleanup(hit):
-        """ clean up and parse data from a single hit """
+        """clean up and parse data from a single hit"""
-        hit['source'] = hit.pop('_source')
+        hit["source"] = hit.pop("_source")
-        hit_keys = hit['source'].keys()
+        hit_keys = hit["source"].keys()
-        if 'media_url' in hit_keys:
+        if "media_url" in hit_keys:
-            parsed_url = urllib.parse.quote(hit['source']['media_url'])
+            parsed_url = urllib.parse.quote(hit["source"]["media_url"])
-            hit['source']['media_url'] = parsed_url
+            hit["source"]["media_url"] = parsed_url
 
-        if 'published' in hit_keys:
+        if "published" in hit_keys:
-            published = hit['source']['published']
+            published = hit["source"]["published"]
             date_pub = datetime.strptime(published, "%Y-%m-%d")
             date_str = datetime.strftime(date_pub, "%d %b, %Y")
-            hit['source']['published'] = date_str
+            hit["source"]["published"] = date_str
 
-        if 'vid_last_refresh' in hit_keys:
+        if "vid_last_refresh" in hit_keys:
-            vid_last_refresh = hit['source']['vid_last_refresh']
+            vid_last_refresh = hit["source"]["vid_last_refresh"]
             date_refresh = datetime.fromtimestamp(vid_last_refresh)
             date_str = datetime.strftime(date_refresh, "%d %b, %Y")
-            hit['source']['vid_last_refresh'] = date_str
+            hit["source"]["vid_last_refresh"] = date_str
 
-        if 'channel_last_refresh' in hit_keys:
+        if "channel_last_refresh" in hit_keys:
-            refreshed = hit['source']['channel_last_refresh']
+            refreshed = hit["source"]["channel_last_refresh"]
             date_refresh = datetime.fromtimestamp(refreshed)
             date_str = datetime.strftime(date_refresh, "%d %b, %Y")
-            hit['source']['channel_last_refresh'] = date_str
+            hit["source"]["channel_last_refresh"] = date_str
 
-        if 'channel' in hit_keys:
+        if "channel" in hit_keys:
-            channel_keys = hit['source']['channel'].keys()
+            channel_keys = hit["source"]["channel"].keys()
-            if 'channel_last_refresh' in channel_keys:
+            if "channel_last_refresh" in channel_keys:
-                refreshed = hit['source']['channel']['channel_last_refresh']
+                refreshed = hit["source"]["channel"]["channel_last_refresh"]
                 date_refresh = datetime.fromtimestamp(refreshed)
                 date_str = datetime.strftime(date_refresh, "%d %b, %Y")
-                hit['source']['channel']['channel_last_refresh'] = date_str
+                hit["source"]["channel"]["channel_last_refresh"] = date_str
 
         return hit
 
 
+class SearchForm:
+    """build query from search form data"""
+
+    CONFIG = AppConfig().config
+    ES_URL = CONFIG["application"]["es_url"]
+
+    def search_channels(self, search_query):
+        """fancy searching channels as you type"""
+        url = self.ES_URL + "/ta_channel/_search"
+        data = {
+            "size": 10,
+            "query": {
+                "multi_match": {
+                    "query": search_query,
+                    "type": "bool_prefix",
+                    "fields": [
+                        "channel_name.search_as_you_type",
+                        "channel_name._2gram",
+                        "channel_name._3gram",
+                    ],
+                }
+            },
+        }
+        look_up = SearchHandler(url, data, cache=False)
+        search_results = look_up.get_data()
+        return {"results": search_results}
+
+    @staticmethod
+    def search_videos():
+        """searching for videos"""
+        # TBD palceholder for now
+        return False
+
+
 class Pagination:
     """
     figure out the pagination based on page size and total_hits
@@ -195,13 +226,13 @@ class Pagination:
 
     def __init__(self, page_get, search_get=False):
         config = AppConfig().config
-        self.page_size = config['archive']['page_size']
+        self.page_size = config["archive"]["page_size"]
         self.page_get = page_get
         self.search_get = search_get
         self.pagination = self.first_guess()
 
     def first_guess(self):
-        """ build first guess before api call """
+        """build first guess before api call"""
         page_get = self.page_get
         if page_get in [0, 1]:
             page_from = 0
@@ -216,22 +247,22 @@ class Pagination:
             "page_size": self.page_size,
             "page_from": page_from,
             "prev_pages": prev_pages,
-            "current_page": page_get
+            "current_page": page_get,
         }
         if self.search_get:
             pagination.update({"search_get": self.search_get})
         return pagination
 
     def validate(self, total_hits):
-        """ validate pagination with total_hits after making api call """
+        """validate pagination with total_hits after making api call"""
         page_get = self.page_get
         max_pages = math.ceil(total_hits / self.page_size)
         if page_get < max_pages and max_pages > 1:
-            self.pagination['last_page'] = max_pages
+            self.pagination["last_page"] = max_pages
         else:
-            self.pagination['last_page'] = False
+            self.pagination["last_page"] = False
         next_pages = [
             i for i in range(page_get + 1, page_get + 6) if 1 < i < max_pages
         ]
 
-        self.pagination['next_pages'] = next_pages
+        self.pagination["next_pages"] = next_pages

tubearchivist/home/tasks.py

@@ -7,29 +7,28 @@ Functionality:
 import os
 
 from celery import Celery, shared_task
-
-from home.src.download import (
-    PendingList,
-    ChannelSubscription,
-    VideoDownloader
-)
 from home.src.config import AppConfig
-from home.src.reindex import reindex_old_documents, ManualImport
+from home.src.download import ChannelSubscription, PendingList, VideoDownloader
-from home.src.helper import get_lock
+from home.src.helper import RedisArchivist, RedisQueue
-
+from home.src.index_management import backup_all_indexes, restore_from_backup
+from home.src.reindex import ManualImport, reindex_old_documents
 
 CONFIG = AppConfig().config
-REDIS_HOST = CONFIG['application']['REDIS_HOST']
+REDIS_HOST = os.environ.get("REDIS_HOST")
+REDIS_PORT = os.environ.get("REDIS_PORT")
 
-os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'home.settings')
+if not REDIS_PORT:
-app = Celery('tasks', broker='redis://' + REDIS_HOST)
+    REDIS_PORT = 6379
-app.config_from_object('django.conf:settings', namespace='CELERY')
+
+os.environ.setdefault("DJANGO_SETTINGS_MODULE", "home.settings")
+app = Celery("tasks", broker=f"redis://{REDIS_HOST}:{REDIS_PORT}")
+app.config_from_object("django.conf:settings", namespace="CELERY")
 app.autodiscover_tasks()
 
 
 @shared_task
 def update_subscribed():
-    """ look for missing videos and add to pending """
+    """look for missing videos and add to pending"""
     channel_handler = ChannelSubscription()
     missing_videos = channel_handler.find_missing()
     if missing_videos:
@@ -41,27 +40,53 @@ def update_subscribed():
 
 @shared_task
 def download_pending():
-    """ download latest pending videos """
+    """download latest pending videos"""
-    pending_handler = PendingList()
+
-    pending_vids = pending_handler.get_all_pending()[0]
+    have_lock = False
-    to_download = [i['youtube_id'] for i in pending_vids]
+    my_lock = RedisArchivist().get_lock("downloading")
-    to_download.reverse()
+
-    if to_download:
+    try:
-        download_handler = VideoDownloader(to_download)
+        have_lock = my_lock.acquire(blocking=False)
-        download_handler.download_list()
+        if have_lock:
+            downloader = VideoDownloader()
+            downloader.add_pending()
+            downloader.run_queue()
+        else:
+            print("Did not acquire download lock.")
+
+    finally:
+        if have_lock:
+            my_lock.release()
 
 
 @shared_task
 def download_single(youtube_id):
-    """ start download single video now """
+    """start download single video now"""
-    to_download = [youtube_id]
+
-    download_handler = VideoDownloader(to_download)
+    queue = RedisQueue("dl_queue")
-    download_handler.download_list()
+    queue.add_priority(youtube_id)
+    print("Added to queue with priority: " + youtube_id)
+
+    # start queue if needed
+    have_lock = False
+    my_lock = RedisArchivist().get_lock("downloading")
+
+    try:
+        have_lock = my_lock.acquire(blocking=False)
+        if have_lock:
+            VideoDownloader().run_queue()
+        else:
+            print("Download queue already running.")
+
+    finally:
+        # release if only single run
+        if have_lock and not queue.get_next():
+            my_lock.release()
 
 
 @shared_task
 def extrac_dl(youtube_ids):
-    """ parse list passed and add to pending """
+    """parse list passed and add to pending"""
     pending_handler = PendingList()
     missing_videos = pending_handler.parse_url_list(youtube_ids)
     pending_handler.add_to_pending(missing_videos)
@@ -69,17 +94,17 @@ def extrac_dl(youtube_ids):
 
 @shared_task
 def check_reindex():
-    """ run the reindex main command """
+    """run the reindex main command"""
     reindex_old_documents()
 
 
 @shared_task
 def run_manual_import():
-    """ called from settings page, to go through import folder """
+    """called from settings page, to go through import folder"""
 
-    print('starting media file import')
+    print("starting media file import")
     have_lock = False
-    my_lock = get_lock('manual_import')
+    my_lock = RedisArchivist().get_lock("manual_import")
 
     try:
         have_lock = my_lock.acquire(blocking=False)
@@ -93,3 +118,39 @@ def run_manual_import():
     finally:
         if have_lock:
             my_lock.release()
+
+
+@shared_task
+def run_backup():
+    """called from settings page, dump backup to zip file"""
+    backup_all_indexes()
+    print("backup finished")
+
+
+@shared_task
+def run_restore_backup():
+    """called from settings page, dump backup to zip file"""
+    restore_from_backup()
+    print("index restore finished")
+
+
+def kill_dl(task_id):
+    """kill download worker task by ID"""
+    app.control.revoke(task_id, terminate=True)
+    _ = RedisArchivist().del_message("dl_queue_id")
+    RedisQueue("dl_queue").clear()
+
+    # clear cache
+    cache_dir = os.path.join(CONFIG["application"]["cache_dir"], "download")
+    for cached in os.listdir(cache_dir):
+        to_delete = os.path.join(cache_dir, cached)
+        os.remove(to_delete)
+
+    # notify
+    mess_dict = {
+        "status": "downloading",
+        "level": "error",
+        "title": "Brutally killing download queue",
+        "message": "",
+    }
+    RedisArchivist().set_message("progress:download", mess_dict)

tubearchivist/home/templates/home/about.html

@@ -5,35 +5,13 @@
     <h1>About The Tube Archivist</h1>
 </div>
 <div class="about-section">
-    <h2>Getting started</h2>
+    <h2>Useful Links</h2>
-    <ol>
+    <p>This project is in active and constant development, take a look at the <a href="https://github.com/bbilly1/tubearchivist#roadmap" target="_blank">roadmap</a> for a overview.</p>
-        <li>Before adding some videos to the download queue, take a look at the download format settings on your <a href="{% url 'settings' %}#format">settings</a> page and set your desired download quality.</li>
+    <p>For any questions on what a button or a function does, You can find the up-to-date user documentation on <a href="https://github.com/bbilly1/tubearchivist/wiki" target="_blank">Github</a>.</p>
-        <li>While at the settings page also set the value for the Subscriptions <a href="{% url 'settings' %}#subscriptions">page size</a>. This will define the max amount of videos that will get added once you run <i>Rescan Subscriptions</i>. yt-dlp by default uses 50 video page size. Any value higher than that will significantly slow down the scanning process.</li>
+    <p>All contributions are welcome: Open an <a href="https://github.com/bbilly1/tubearchivist/issues" target="_blank">issue</a> for any bugs and errors, start a <a href="https://github.com/bbilly1/tubearchivist/discussions" target="_blank">discussion</a> for anything that will require a more indepth look. The <a href="https://github.com/bbilly1/tubearchivist/blob/master/CONTRIBUTING.md" target="_blank">contributing</a> page is a good place to get started.</p>
-        <li>Subscribe to some of your favourite youtube channels in the <a href="{% url 'channel' %}">channel</a> page. To subscribe to a channel you have a few options:
-            <ul>
-                <li>Enter the Youtube channel ID, a 25 character alphanumeric string. For example <span class="settings-current">UCBa659QWEk1AI4Tg--mrJ2A</span></li>
-                <li>Enter the url to the channel page on youtube. For example <span class="settings-current">https://www.youtube.com/channel/UCBa659QWEk1AI4Tg--mrJ2A</span></li>
-                <li>Enter the video url for any video and let <i>Tube Archivist</i> extract the channel ID for you. For example <span class="settings-current">https://www.youtube.com/watch?v=2tdiKTSdE9Y</span></li>
-            </ul>
-        </li>
-        <li>After that head over to the <a href="{% url 'downloads' %}">Downloads</a> Page. Once you click on the <i>Rescan Subscriptions</i> button, <b>Tube Archivist</b> will go through all your subscribed channels and look for new videos. Initially these are the latest videos depending on the page size as described above.</li>
-        <li>Click on <span class="settings-current">ignore</span> for any video you don't want to download or on <span class="settings-current">Download now</span> for any video you want to start downloading now independently from the queue.</li>
-        <li>Click on the <span class="settings-current">Download queue</span> button to start downloading. </b>Tube Archivist</b> will then start the queue from the bottom and download as many videos as defined in <a href="{% url 'settings' %}#downloads">download limit</a>. There is currently no good way to cancel the download process.</li>
-    </ol>
 </div>
 <div class="about-section">
-    <h2>Additional</h2>
+    <h2>Donate</h2>
-    <ul>
+    <p>Here are <a href="https://github.com/bbilly1/tubearchivist#donate" target="_blank">some links</a>, if you want to buy the developer a coffee. Thank you for your support!</p>
-        <li>You can manually add videos to the queue on the <a href="{% url 'downloads' %}">download page</a> by clicking on <i>Add to download queue</i> and then entering a list of videos. Add one link per line. You have a few options:
-            <ul>
-                <li>Add a link to a Youtube video. For example <span class="settings-current">https://www.youtube.com/watch?v=2tdiKTSdE9Y</span>.</li>
-                <li>Add a Youtube video ID. For example <span class="settings-current">2tdiKTSdE9Y</span>.</li>
-                <li>Add a link to a Youtube video by providing the shortened url, for example <span class="settings-current">https://youtu.be/2tdiKTSdE9Y</span>.</li>
-                <li>Add a Channel ID or Channel Url to add every available video to the download queue. This will ignore the page size as described above. You can still ignore selected videos before starting the download.</li>
-                <li>Add a playlist ID or url to add every available video in the list to the download queue, for example <span class="settings-current">https://www.youtube.com/playlist?list=PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha</span> or <span class="settings-current">PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha</span>. Note that when you add a link to a video in a playlist, <b>Tube Archivist</b> assumes you want to download only the specific video and not the whole playlist, vor example <span class="settings-current">https://www.youtube.com/watch?v=CINVwWHlzTY&list=PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha</span> will only add one video <span class="settings-current">CINVwWHlzTY</span> to the queue.</li>
-            </ul>
-        </li>
-        <li>As you watch videos, mark them as watched by clicking on the unseen icon <span class="about-icon"><img src="{% static 'img/icon-unseen.svg' %}" alt="unseen-icon" class="unseen-icon"></span>. Once marked as watched, the icon will change to <span class="about-icon"><img src="{% static 'img/icon-seen.svg' %}" alt="seen-icon" class="seen-icon"></span>.</li>
-    </ul>
 </div>
-{% endblock content %}
+{% endblock content %}

tubearchivist/home/templates/home/base.html

@@ -96,7 +96,7 @@
     </div>
     <div class="footer">
         <div class="boxed-content">
-            <span>© 2021 The Tube Archivist | <a href="https://github.com/bbilly1/tubearchivist" target="_blank">Github</a> | <a href="https://hub.docker.com/r/bbilly1/tubearchivist" target="_blank">Docker Hub</a></span>
+            <span>© 2021 The Tube Archivist v0.0.5 | <a href="https://github.com/bbilly1/tubearchivist" target="_blank">Github</a> | <a href="https://hub.docker.com/r/bbilly1/tubearchivist" target="_blank">Docker Hub</a></span>
         </div>
     </div>
 </body>

tubearchivist/home/templates/home/channel.html

@@ -30,27 +30,38 @@
         </form>
     </div>
 </div>
-<div class="padding-box">
+<div class="view-controls">
-    <h2>Total matching channels: {{ max_hits }}</h2>
+    <div class="toggle">
-    <span>Change show / hide subscribed only </span><span class="settings-current">{{ show_subed_only }}</span>
+        <span>Show only subscribed channels:</span>
-    <select name="watched" id="watched" onchange="showSubscribedOnly(this.value)">
+        <div class="toggleBox">
-        <option value="" disabled selected> -- change -- </option>
+            <input 
-        <option value="0">show all channels</option>
+                id="show_subed_only" onclick="toggleCheckbox(this)" type="checkbox"
-        <option value="1">show subscribed channels only</option>
+                {% if show_subed_only %}
-    </select>
+                    checked
+                {% endif %}
+                >
+            <label for="" class="onbtn">On</label>
+            <label for="" class="ofbtn">Off</label>
+        </div>
+    </div>
+    <div class="view-icons">
+        <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="channel" data-value="grid" alt="grid view">
+        <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="channel" data-value="list" alt="list view">
+    </div>
 </div>
-<div>
+<h2>Total matching channels: {{ max_hits }}</h2>
+<div class="channel-list {{ view_style }}">
     {% if channels %}
         {% for channel in channels %}
-        <div class="channel-item">
+        <div class="channel-item {{ view_style }}">
             {% if channel.source.channel_banner_url %}
-                <div class="channel-banner">
+                <div class="channel-banner {{ view_style }}">
                     <a href="{% url 'channel_id' channel.source.channel_id %}">
                         <img src="/cache/channels/{{ channel.source.channel_id }}_banner.jpg" alt="{{ channel.source.channel_id }}-banner">
                     </a>
                 </div>
             {% endif %}
-            <div class="info-box info-box-2">
+            <div class="info-box info-box-2 {{ view_style }}">
                 <div class="info-box-item">
                     <div class="round-img">
                         <a href="{% url 'channel_id' channel.source.channel_id %}">

tubearchivist/home/templates/home/channel_id.html

@@ -10,7 +10,7 @@
         <a href="/channel/{{ channel_info.channel_id }}/"><img src="/cache/channels/{{ channel_info.channel_id }}_banner.jpg" alt="channel_banner"></a>
     {% endif %}
 </div>
-<div class="info-box info-box-3 padding-box">
+<div class="info-box info-box-3">
     <div class="info-box-item">
         <div class="round-img">
             <a href="{% url 'channel_id' channel_info.channel_id %}">
@@ -61,14 +61,18 @@
         </div>
     </div>
 {% endif %}
+<div class="view-icons">
+    <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="grid" alt="grid view">
+    <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="list" alt="list view">
+</div>
 <div id="player" class="video-player"></div>
 <h2>Videos</h2>
-<div class="video-list">
+<div class="video-list {{ view_style }}">
     {% if videos %}
         {% for video in videos %}
-            <div class="video-item">
+            <div class="video-item {{ view_style }}">
                 <a href="#player" data-src="/media/{{ video.source.media_url }}" data-thumb="/cache/videos/{{ video.source.youtube_id }}.jpg" data-title="{{ video.source.title }}" data-channel="{{ video.source.channel.channel_name }}" data-id="{{ video.source.youtube_id }}" onclick="createPlayer(this)">
-                    <div class="video-thumb-wrap">
+                    <div class="video-thumb-wrap {{ view_style }}">
                         <div class="video-thumb">
                             <img src="/cache/videos/{{ video.source.youtube_id }}.jpg" alt="video-thumb">
                         </div>
@@ -77,7 +81,7 @@
                         </div>
                     </div>
                 </a>
-                <div class="video-desc">
+                <div class="video-desc {{ view_style }}">
                     <div class="video-desc-player" id="video-info-{{ video.source.youtube_id }}">
                         {% if video.source.player.watched %}
                             <img src="{% static 'img/icon-seen.svg' %}" alt="seen-icon" id="{{ video.source.youtube_id }}" class="seen-icon">

tubearchivist/home/templates/home/downloads.html

@@ -5,14 +5,15 @@
     <h1>Downloads</h1>
 </div>
 <div id="downloadMessage"></div>
-<div class="info-box info-box-3 padding-box">
+<div id="downloadControl"></div>
+<div class="info-box info-box-3">
     <div class="icon-text">
         <img id="rescan-icon" onclick="rescanPending()" src="{% static 'img/icon-rescan.svg' %}" alt="rescan-icon">
         <p>Rescan subscriptions</p>
     </div>
     <div class="icon-text">
         <img id="download-icon" onclick="dlPending()" src="{% static 'img/icon-download.svg' %}" alt="download-icon">
-        <p>Download queue</p>
+        <p>Start download</p>
     </div>
     <div class="icon-text">
         <img id="add-icon" onclick="showForm()" src="{% static 'img/icon-add.svg' %}" alt="add-icon">
@@ -26,30 +27,60 @@
         </div>
     </div>
 </div>
-<h2>Download queue</h2>
+<div class="view-controls">
-<div>
+    <div class="toggle">
-    {% if pending %}
+        <span>Show only ignored videos:</span>
-        <h3>Total pending downloads: {{ pending|length }}</h3>
+        <div class="toggleBox">
-        {% for video in pending %}
+            <input 
-            <div class="dl-item" id="dl-{{ video.youtube_id }}">
+                id="show_ignored_only" onclick="toggleCheckbox(this)" type="checkbox"
-                <div class="dl-thumb">
+                {% if show_ignored_only %}
+                    checked
+                {% endif %}
+                >
+            <label for="" class="onbtn">On</label>
+            <label for="" class="ofbtn">Off</label>
+        </div>
+    </div>
+    <div class="view-icons">
+        <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="downloads" data-value="grid" alt="grid view">
+        <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="downloads" data-value="list" alt="list view">
+    </div>
+</div>
+{% if show_ignored_only %}
+    <h2>Ignored from download</h2>
+{% else %}
+    <h2>Download queue</h2>
+{% endif %}
+<h3>Total videos: {{ max_hits }}</h3>
+<div class="dl-list {{ view_style }}">
+    {% if all_video_hits %}
+        {% for video in all_video_hits %}
+            <div class="dl-item {{ view_style }}" id="dl-{{ video.youtube_id }}">
+                <div class="dl-thumb {{ view_style }}">
                     <img src="{{ video.vid_thumb_url }}" alt="video_thumb">
                 </div>
-                <div class="dl-desc">
+                <div class="dl-desc {{ view_style }}">
-                    <h3>{{ video.title }}</h3>
+                    {% if show_ignored_only %}
+                        <h3>Ignore: {{ video.title }}</h3>
+                    {% else %}
+                        <h3>Download: {{ video.title }}</h3>
+                    {% endif %}
                     {% if video.channel_indexed %}
                         <a href="{% url 'channel_id' video.channel_id %}">{{ video.channel_name }}</a>
                     {% else %}
                         <span>{{ video.channel_name }}</span>
                     {% endif %}
                     <p>Published: {{ video.published }} | Duration: {{ video.duration }} | {{ video.youtube_id }}</p>
-                    <button data-id="{{ video.youtube_id }}" onclick="toIgnore(this)">Ignore</button>
+                    {% if show_ignored_only %}
-                    <button data-id="{{ video.youtube_id }}" onclick="downloadNow(this)">Download now</button>
+                        <button data-id="{{ video.youtube_id }}" onclick="forgetIgnore(this)">Forget</button>
+                        <button data-id="{{ video.youtube_id }}" onclick="addSingle(this)">Add to queue</button>
+                    {% else %}
+                        <button data-id="{{ video.youtube_id }}" onclick="toIgnore(this)">Ignore</button>
+                        <button id="{{ video.youtube_id }}" data-id="{{ video.youtube_id }}" onclick="downloadNow(this)">Download now</button>
+                    {% endif %}
                 </div>
             </div>
         {% endfor %}
-    {% else %}
-        <h3>No pending downloads</h3>
     {% endif %}
 </div>
 <script type="text/javascript" src="{% static 'progress.js' %}"></script>

tubearchivist/home/templates/home/home.html

@@ -13,13 +13,6 @@
                 <option value="downloaded">date downloaded</option>
             </select>
         </p>
-        <p>Hide watched videos <span class="settings-current">{{ hide_watched }}</span>
-            <select name="watched" id="watched" onchange="hideWatched(this.value)">
-                <option value="" disabled selected> -- change hide watched -- </option>
-                <option value="0">show watched videos</option>
-                <option value="1">hide watched videos</option>
-            </select>
-        </p>
     </div>
     <div class="search-form icon-text">
         <div class="search-icon">
@@ -31,13 +24,32 @@
         </form>
     </div>
 </div>
+<div class="view-controls">
+    <div class="toggle">
+        <span>Hide watched videos:</span>
+        <div class="toggleBox">
+            <input 
+                id="hide_watched" onclick="toggleCheckbox(this)" type="checkbox"
+                {% if hide_watched %}
+                    checked
+                {% endif %}
+                >
+            <label for="" class="onbtn">On</label>
+            <label for="" class="ofbtn">Off</label>
+        </div>
+    </div>
+    <div class="view-icons">
+        <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="grid" alt="grid view">
+        <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="list" alt="list view">
+    </div>
+</div>
 <div id="player" class="video-player"></div>
-<div class="video-list">
+<div class="video-list {{ view_style }}">
     {% if videos %}
         {% for video in videos %}
-            <div class="video-item">
+            <div class="video-item {{ view_style }}">
                 <a href="#player" data-src="/media/{{ video.source.media_url }}" data-thumb="/cache/videos/{{ video.source.youtube_id }}.jpg" data-title="{{ video.source.title }}" data-channel="{{ video.source.channel.channel_name }}" data-id="{{ video.source.youtube_id }}" onclick="createPlayer(this)">
-                    <div class="video-thumb-wrap">
+                    <div class="video-thumb-wrap {{ view_style }}">
                         <div class="video-thumb">
                             <img src="/cache/videos/{{ video.source.youtube_id }}.jpg" alt="video-thumb">
                         </div>
@@ -46,7 +58,7 @@
                         </div>
                     </div>
                 </a>
-                <div class="video-desc">
+                <div class="video-desc {{ view_style }}">
                     <div class="video-desc-player" id="video-info-{{ video.source.youtube_id }}">
                         {% if video.source.player.watched %}
                             <img src="{% static 'img/icon-seen.svg' %}" alt="seen-icon" id="{{ video.source.youtube_id }}" class="seen-icon">

tubearchivist/home/templates/home/settings.html

@@ -9,7 +9,7 @@
         <h2>Color scheme</h2>
         <div class="settings-item">
             <p>Current color scheme: <span class="settings-current">{{ config.application.colors }}</span></p>
-            <i>Select yout prefered color scheme between dark and light mode.</i><br>
+            <i>Select your preferred color scheme between dark and light mode.</i><br>
             <select name="application.colors" id="application.colors">
                 <option value="" disabled selected> -- change color scheme -- </option>
                 <option value="dark">dark mode</option>
@@ -19,33 +19,6 @@
     </div>
     <div class="settings-group">
         <h2>Archive View</h2>
-        <div class="settings-item">
-            <p>Current default Sort: <span class="settings-current">{{ config.archive.sort }}</span></p>
-            <i>Change how the home view and channels view sorts by defaults.</i><br>
-            <select name="archive.sort" id="archive.sort"">
-                <option value="" disabled selected> -- change sort order -- </option>
-                <option value="published">date published</option>
-                <option value="downloaded">date downloaded</option>
-            </select>
-        </div>
-        <div class="settings-item">
-            <p>Current default hide watched: <span class="settings-current">{{ config.archive.hide_watched }}</span></p>
-            <i>Show or hide watched videos by default.</i><br>
-            <select name="archive.hide_watched" id="archive.hide_watched"">
-                <option value="" disabled selected> -- change visibility -- </option>
-                <option value="0">show watched</option>
-                <option value="1">hide watched</option>
-            </select>
-        </div>
-        <div class="settings-item">
-            <p>Current default show subscribed channels only: <span class="settings-current">{{ config.archive.show_subed_only }}</span></p>
-            <i>Show subscribed channels only by default.</i><br>
-            <select name="archive.show_subed_only" id="archive.show_subed_only"">
-                <option value="" disabled selected> -- change visibility -- </option>
-                <option value="0">show subscribed only</option>
-                <option value="1">hide not subscribed</option>
-            </select>
-        </div>
         <div class="settings-item">
             <p>Current page size: <span class="settings-current">{{ config.archive.page_size }}</span></p>
             <i>Result of videos showing in archive page</i><br>
@@ -55,7 +28,7 @@
     <div class="settings-group">
         <h2 id="subscriptions">Subscriptions</h2>
         <div class="settings-item">
-            <p>Currend channel page size: <span class="settings-current">{{ config.subscriptions.channel_size }}</span></p>
+            <p>Current channel page size: <span class="settings-current">{{ config.subscriptions.channel_size }}</span></p>
             <i>Recent videos to check on check pending, max recommended 50.</i><br>
             <input type="number" name="subscriptions.channel_size" id="subscriptions.channel_size">
         </div>
@@ -72,17 +45,17 @@
         <h2 id="downloads">Downloads</h2>
         <div class="settings-item">
             <p>Current download limit: <span class="settings-current">{{ config.downloads.limit_count }}</span></p>
-            <i>Limit the count of videos getting downloaded on every run. 0 (zero) to deactivate.</i><br>
+            <i>Limit the number of videos getting downloaded on every run. 0 (zero) to deactivate.</i><br>
             <input type="number" name="downloads.limit_count" id="downloads.limit_count">
         </div>
         <div class="settings-item">
-            <p>Currend Download speed limit: <span class="settings-current">{{ config.downloads.limit_speed }}</span></p>
+            <p>Current download speed limit: <span class="settings-current">{{ config.downloads.limit_speed }}</span></p>
             <i>Limit download speed. 0 (zero) to deactivate.</i><br>
             <input type="number" name="downloads.limit_speed" id="downloads.limit_speed"><span>KB/sec</span>
         </div>
         <div class="settings-item">
             <p>Current scraping sleep interval: <span class="settings-current">{{ config.downloads.sleep_interval }}</p>
-            <i>Seconds to sleep between calls to youtube. Might be necessary to avoid throttling. Recommended 3.</i><br>
+            <i>Seconds to sleep between calls to YouTube. Might be necessary to avoid throttling. Recommended 3.</i><br>
             <input type="number" name="downloads.sleep_interval" id="downloads.sleep_interval">
         </div>
         <div class="settings-item">
@@ -106,6 +79,15 @@
             <input type="text" name="downloads.format" id="downloads.format">
             <br>
         </div>
+        <div class="settings-item">
+            <p>Current metadata embed setting: <span class="settings-current">{{ config.downloads.add_metadata }}</span></p>
+            <i>Metadata is not embedded into the downloaded files by default.</i><br>
+            <select name="downloads.add_metadata" id="downloads.add_metadata"">
+                <option value="" disabled selected> -- change metadata embed -- </option>
+                <option value="0">don't embed metadata</option>
+                <option value="1">embed metadata</option>
+            </select>
+        </div>
     </div>
     <button type="submit">Update Settings</button>
 </form>
@@ -114,17 +96,28 @@
 </div>
 <div class="settings-group">
     <h2>Manual media files import.</h2>
-    <p>Add files to the cache/import folder. Make sure to follow the instructions on <a href="https://github.com/bbilly1/tubearchivist#import-your-existing-library" target="_blank">Github</a>.</p>
+    <p>Add files to the <span class="settings-current">cache/import</span> folder. Make sure to follow the instructions on <a href="https://github.com/bbilly1/tubearchivist#import-your-existing-library" target="_blank">Github</a>.</p>
     <div id="manual-import">
         <button onclick="manualImport()">Start import</button>
     </div>
 </div>
+<div class="settings-group">
+    <h2>Backup database</h2>
+    <p>Export your database to a zip file stored at <span class="settings-current">cache/backup</span>.</p>
+    <div id="db-backup">
+        <button onclick="dbBackup()">Start backup</button>
+    </div>
+</div>
+<div class="settings-group">
+    <h2>Restore from backup</h2>
+    <p><span class="danger-zone">Danger Zone</span>: This will replace your existing index with the backup.</p>
+    <p>Add the backup zip file to the <span class="settings-current">cache/backup</span> folder.</p>
+    <div id="db-restore">
+        <button onclick="dbRestore()">Restore backup</button>
+    </div>
+</div>
 <div class="settings-group">
     <p>Rescan filesystem.</p>
     <i>Coming soon</i>
 </div>
-<div class="settings-group">
+{% endblock content %}
-    <p>Backup database.</p>
-    <i>Coming soon</i>
-</div>
-{% endblock content %}

tubearchivist/home/tests.py

@@ -1,3 +1,3 @@
-from django.test import TestCase
+from django.test import TestCase  # noqa: F401 - Unused import
 
 # Create your tests here.

tubearchivist/home/urls.py

@@ -1,27 +1,30 @@
 """ all home app urls """
 
 from django.urls import path
-
 from home.views import (
-    HomeView,
+    AboutView,
-    DownloadView,
-    ChannelView,
     ChannelIdView,
-    VideoView,
+    ChannelView,
+    DownloadView,
+    HomeView,
     SettingsView,
-    AboutView
+    VideoView,
 )
 
 from . import views
 
 urlpatterns = [
-    path('', HomeView.as_view(), name='home'),
+    path("", HomeView.as_view(), name="home"),
-    path('about/', AboutView.as_view(), name='about'),
+    path("about/", AboutView.as_view(), name="about"),
-    path('downloads/', DownloadView.as_view(), name='downloads'),
+    path("downloads/", DownloadView.as_view(), name="downloads"),
-    path('settings/', SettingsView.as_view(), name='settings'),
+    path("settings/", SettingsView.as_view(), name="settings"),
-    path('process/', views.process, name='process'),
+    path("process/", views.process, name="process"),
-    path('downloads/progress', views.progress, name='progress'),
+    path("downloads/progress", views.progress, name="progress"),
-    path('channel/', ChannelView.as_view(), name='channel'),
+    path("channel/", ChannelView.as_view(), name="channel"),
-    path('channel/<slug:channel_id_detail>/', ChannelIdView.as_view(), name='channel_id'),
+    path(
-    path('video/<slug:video_id>/', VideoView.as_view(), name='video')
+        "channel/<slug:channel_id_detail>/",
+        ChannelIdView.as_view(),
+        name="channel_id",
+    ),
+    path("video/<slug:video_id>/", VideoView.as_view(), name="video"),
 ]

tubearchivist/home/views.py

@@ -1,62 +1,56 @@
 """
 Functionality:
 - all views for home app
-- process post data recieved from frontend via ajax
+- process post data received from frontend via ajax
 """
 
-import urllib.parse
 import json
-
+import urllib.parse
-from datetime import datetime
 from time import sleep
 
-import requests
-
-from django.shortcuts import render, redirect
 from django.http import JsonResponse
-from django.views import View
+from django.shortcuts import redirect, render
 from django.utils.http import urlencode
-
+from django.views import View
-from home.src.download import PendingList, ChannelSubscription
-from home.src.searching import SearchHandler, Pagination
 from home.src.config import AppConfig
-from home.src.helper import (
+from home.src.download import ChannelSubscription, PendingList
-    process_url_list,
+from home.src.helper import RedisArchivist, RedisQueue, process_url_list
-    get_dl_message,
+from home.src.index import WatchState
-    get_message,
+from home.src.searching import Pagination, SearchForm, SearchHandler
-    set_message
-)
 from home.tasks import (
-    update_subscribed,
     download_pending,
-    extrac_dl,
     download_single,
-    run_manual_import
+    extrac_dl,
+    kill_dl,
+    run_backup,
+    run_manual_import,
+    run_restore_backup,
+    update_subscribed,
 )
 
 
 class HomeView(View):
-    """ resolves to /
+    """resolves to /
     handle home page and video search post functionality
     """
 
     CONFIG = AppConfig().config
-    ES_URL = CONFIG['application']['es_url']
+    ES_URL = CONFIG["application"]["es_url"]
 
     def get(self, request):
-        """ return home search results """
+        """return home search results"""
-        colors, sort_order, hide_watched = self.read_config()
+        colors, view_style, sort_order, hide_watched = self.read_config()
         # handle search
-        search_get = request.GET.get('search', False)
+        search_get = request.GET.get("search", False)
         if search_get:
             search_encoded = urllib.parse.quote(search_get)
         else:
             search_encoded = False
         # define page size
-        page_get = int(request.GET.get('page', 0))
+        page_get = int(request.GET.get("page", 0))
         pagination_handler = Pagination(page_get, search_encoded)
 
-        url = self.ES_URL + '/ta_video/_search'
+        url = self.ES_URL + "/ta_video/_search"
 
         data = self.build_data(
             pagination_handler, sort_order, search_get, hide_watched
@@ -67,160 +61,205 @@ class HomeView(View):
         max_hits = search.max_hits
         pagination_handler.validate(max_hits)
         context = {
-            'videos': videos_hits,
+            "videos": videos_hits,
-            'pagination': pagination_handler.pagination,
+            "pagination": pagination_handler.pagination,
-            'sortorder': sort_order,
+            "sortorder": sort_order,
-            'hide_watched': hide_watched,
+            "hide_watched": hide_watched,
-            'colors': colors
+            "colors": colors,
+            "view_style": view_style,
         }
-        return render(request, 'home/home.html', context)
+        return render(request, "home/home.html", context)
 
     @staticmethod
     def build_data(pagination_handler, sort_order, search_get, hide_watched):
-        """ build the data dict for the search query """
+        """build the data dict for the search query"""
-        page_size = pagination_handler.pagination['page_size']
+        page_size = pagination_handler.pagination["page_size"]
-        page_from = pagination_handler.pagination['page_from']
+        page_from = pagination_handler.pagination["page_from"]
         data = {
-            "size": page_size, "from": page_from, "query": {"match_all": {}},
+            "size": page_size,
+            "from": page_from,
+            "query": {"match_all": {}},
             "sort": [
                 {"published": {"order": "desc"}},
-                {"date_downloaded": {"order": "desc"}}
+                {"date_downloaded": {"order": "desc"}},
-            ]
+            ],
         }
         # define sort
-        if sort_order == 'downloaded':
+        if sort_order == "downloaded":
-            del data['sort'][0]
+            del data["sort"][0]
         if search_get:
-            del data['sort']
+            del data["sort"]
         if hide_watched:
-            data['query'] = {"term": {"player.watched": {"value": False}}}
+            data["query"] = {"term": {"player.watched": {"value": False}}}
         if search_get:
             query = {
                 "multi_match": {
                     "query": search_get,
                     "fields": ["title", "channel.channel_name", "tags"],
                     "type": "cross_fields",
-                    "operator": "and"
+                    "operator": "and",
                 }
             }
-            data['query'] = query
+            data["query"] = query
 
         return data
 
     @staticmethod
     def read_config():
-        """ read needed values from redis """
+        """read needed values from redis"""
         config_handler = AppConfig().config
-        colors = config_handler['application']['colors']
+        colors = config_handler["application"]["colors"]
-        sort_order = get_message('sort_order')
+        view_style = config_handler["default_view"]["home"]
-        hide_watched = get_message('hide_watched')
+        sort_order = RedisArchivist().get_message("sort_order")
-        return colors, sort_order, hide_watched
+        if not sort_order:
+            sort_order = "published"
+        hide_watched = RedisArchivist().get_message("hide_watched")
+        return colors, view_style, sort_order, hide_watched
 
     @staticmethod
     def post(request):
-        """ handle post from search form """
+        """handle post from search form"""
         post_data = dict(request.POST)
-        search_query = post_data['videoSearch'][0]
+        search_query = post_data["videoSearch"][0]
-        search_url = '/?' + urlencode({'search': search_query})
+        search_url = "/?" + urlencode({"search": search_query})
         return redirect(search_url, permanent=True)
 
 
 class AboutView(View):
-    """ resolves to /about/
+    """resolves to /about/
     show helpful how to information
     """
 
     @staticmethod
     def get(request):
-        """ handle http get """
+        """handle http get"""
         config = AppConfig().config
-        colors = config['application']['colors']
+        colors = config["application"]["colors"]
-        context = {
+        context = {"title": "About", "colors": colors}
-            'title': 'About',
+        return render(request, "home/about.html", context)
-            'colors': colors
-        }
-        return render(request, 'home/about.html', context)
 
 
 class DownloadView(View):
-    """ resolves to /download/
+    """resolves to /download/
     takes POST for downloading youtube links
     """
 
-    @staticmethod
+    def get(self, request):
-    def get(request):
+        """handle get requests"""
-        """ handle get requests """
         config = AppConfig().config
-        colors = config['application']['colors']
+        colors = config["application"]["colors"]
-        pending_handler = PendingList()
+        view_style = config["default_view"]["downloads"]
-        all_pending, _ = pending_handler.get_all_pending()
+        ignored = RedisArchivist().get_message("show_ignored_only")
+        show_ignored_only = ignored["status"]
+
+        page_get = int(request.GET.get("page", 0))
+        pagination_handler = Pagination(page_get)
+
+        url = config["application"]["es_url"] + "/ta_download/_search"
+        data = self.build_data(pagination_handler, show_ignored_only)
+        search = SearchHandler(url, data, cache=False)
+
+        videos_hits = search.get_data()
+        max_hits = search.max_hits
+
+        if videos_hits:
+            all_video_hits = [i["source"] for i in videos_hits]
+            pagination_handler.validate(max_hits)
+            pagination = pagination_handler.pagination
+        else:
+            all_video_hits = False
+            pagination = False
+
         context = {
-            'pending': all_pending,
+            "all_video_hits": all_video_hits,
-            'title': 'Downloads',
+            "max_hits": max_hits,
-            'colors': colors
+            "pagination": pagination,
+            "title": "Downloads",
+            "colors": colors,
+            "show_ignored_only": show_ignored_only,
+            "view_style": view_style,
         }
-        return render(request, 'home/downloads.html', context)
+        return render(request, "home/downloads.html", context)
+
+    @staticmethod
+    def build_data(pagination_handler, show_ignored_only):
+        """build data dict for search"""
+        page_size = pagination_handler.pagination["page_size"]
+        page_from = pagination_handler.pagination["page_from"]
+        if show_ignored_only:
+            filter_view = "ignore"
+        else:
+            filter_view = "pending"
+
+        data = {
+            "size": page_size,
+            "from": page_from,
+            "query": {"term": {"status": {"value": filter_view}}},
+            "sort": [{"timestamp": {"order": "asc"}}],
+        }
+        return data
 
     @staticmethod
     def post(request):
-        """ handle post requests """
+        """handle post requests"""
         download_post = dict(request.POST)
-        if 'vid-url' in download_post.keys():
+        if "vid-url" in download_post.keys():
-            url_str = download_post['vid-url']
+            url_str = download_post["vid-url"]
-            print('adding to queue')
+            try:
-            youtube_ids = process_url_list(url_str)
+                youtube_ids = process_url_list(url_str)
-            if not youtube_ids:
+            except ValueError:
                 # failed to process
-                print(url_str)
+                print(f"failed to parse: {url_str}")
                 mess_dict = {
                     "status": "downloading",
                     "level": "error",
-                    "title": 'Failed to extract links.',
+                    "title": "Failed to extract links.",
-                    "message": ''
+                    "message": "Not a video, channel or playlist ID or URL",
                 }
-                set_message('progress:download', mess_dict)
+                RedisArchivist().set_message("progress:download", mess_dict)
-                return redirect('downloads')
+                return redirect("downloads")
 
             print(youtube_ids)
             extrac_dl.delay(youtube_ids)
 
         sleep(2)
-        return redirect('downloads', permanent=True)
+        return redirect("downloads", permanent=True)
 
 
 class ChannelIdView(View):
-    """ resolves to /chanel/<channel-id>/
+    """resolves to /channel/<channel-id>/
     display single channel page from channel_id
     """
 
     def get(self, request, channel_id_detail):
-        """ get method """
+        """get method"""
-        es_url, colors = self.read_config()
+        es_url, colors, view_style = self.read_config()
         context = self.get_channel_videos(request, channel_id_detail, es_url)
-        context.update({'colors': colors})
+        context.update({"colors": colors, "view_style": view_style})
-        return render(request, 'home/channel_id.html', context)
+        return render(request, "home/channel_id.html", context)
 
     @staticmethod
     def read_config():
-        """ read config file """
+        """read config file"""
         config = AppConfig().config
-        es_url = config['application']['es_url']
+        es_url = config["application"]["es_url"]
-        colors = config['application']['colors']
+        colors = config["application"]["colors"]
-        return es_url, colors
+        view_style = config["default_view"]["home"]
+        return es_url, colors, view_style
 
     def get_channel_videos(self, request, channel_id_detail, es_url):
-        """ get channel from video index """
+        """get channel from video index"""
-        page_get = int(request.GET.get('page', 0))
+        page_get = int(request.GET.get("page", 0))
         pagination_handler = Pagination(page_get)
         # get data
-        url = es_url + '/ta_video/_search'
+        url = es_url + "/ta_video/_search"
         data = self.build_data(pagination_handler, channel_id_detail)
         search = SearchHandler(url, data)
         videos_hits = search.get_data()
         max_hits = search.max_hits
         if max_hits:
-            channel_info = videos_hits[0]['source']['channel']
+            channel_info = videos_hits[0]["source"]["channel"]
-            channel_name = channel_info['channel_name']
+            channel_name = channel_info["channel_name"]
             pagination_handler.validate(max_hits)
             pagination = pagination_handler.pagination
         else:
@@ -232,368 +271,390 @@ class ChannelIdView(View):
             pagination = False
 
         context = {
-            'channel_info': channel_info,
+            "channel_info": channel_info,
-            'videos': videos_hits,
+            "videos": videos_hits,
-            'max_hits': max_hits,
+            "max_hits": max_hits,
-            'pagination': pagination,
+            "pagination": pagination,
-            'title': 'Channel: ' + channel_name,
+            "title": "Channel: " + channel_name,
         }
 
         return context
 
     @staticmethod
     def build_data(pagination_handler, channel_id_detail):
-        """ build data dict for search """
+        """build data dict for search"""
-        page_size = pagination_handler.pagination['page_size']
+        page_size = pagination_handler.pagination["page_size"]
-        page_from = pagination_handler.pagination['page_from']
+        page_from = pagination_handler.pagination["page_from"]
         data = {
-            "size": page_size, "from": page_from,
+            "size": page_size,
+            "from": page_from,
             "query": {
                 "term": {"channel.channel_id": {"value": channel_id_detail}}
             },
             "sort": [
                 {"published": {"order": "desc"}},
-                {"date_downloaded": {"order": "desc"}}
+                {"date_downloaded": {"order": "desc"}},
-            ]
+            ],
         }
         return data
 
     @staticmethod
     def get_channel_info(channel_id_detail, es_url):
-        """ get channel info from channel index if no videos """
+        """get channel info from channel index if no videos"""
-        url = f'{es_url}/ta_channel/_doc/{channel_id_detail}'
+        url = f"{es_url}/ta_channel/_doc/{channel_id_detail}"
         data = False
         search = SearchHandler(url, data)
         channel_data = search.get_data()
-        channel_info = channel_data[0]['source']
+        channel_info = channel_data[0]["source"]
-        channel_name = channel_info['channel_name']
+        channel_name = channel_info["channel_name"]
         return channel_info, channel_name
 
 
 class ChannelView(View):
-    """ resolves to /channel/
+    """resolves to /channel/
     handle functionality for channel overview page, subscribe to channel,
     search as you type for channel name
     """
 
     def get(self, request):
-        """ handle http get requests """
+        """handle http get requests"""
-        es_url, colors = self.read_config()
+        es_url, colors, view_style = self.read_config()
-        page_get = int(request.GET.get('page', 0))
+        page_get = int(request.GET.get("page", 0))
         pagination_handler = Pagination(page_get)
-        page_size = pagination_handler.pagination['page_size']
+        page_size = pagination_handler.pagination["page_size"]
-        page_from = pagination_handler.pagination['page_from']
+        page_from = pagination_handler.pagination["page_from"]
         # get
-        url = es_url + '/ta_channel/_search'
+        url = es_url + "/ta_channel/_search"
         data = {
-            "size": page_size, "from": page_from, "query": {"match_all": {}},
+            "size": page_size,
-            "sort": [{"channel_name.keyword": {"order": "asc"}}]
+            "from": page_from,
+            "query": {"match_all": {}},
+            "sort": [{"channel_name.keyword": {"order": "asc"}}],
         }
-        show_subed_only = get_message('show_subed_only')
+        show_subed_only = RedisArchivist().get_message("show_subed_only")
         if show_subed_only:
-            data['query'] = {"term": {"channel_subscribed": {"value": True}}}
+            data["query"] = {"term": {"channel_subscribed": {"value": True}}}
         search = SearchHandler(url, data)
         channel_hits = search.get_data()
         max_hits = search.max_hits
         pagination_handler.validate(search.max_hits)
         context = {
-            'channels': channel_hits,
+            "channels": channel_hits,
-            'max_hits': max_hits,
+            "max_hits": max_hits,
-            'pagination': pagination_handler.pagination,
+            "pagination": pagination_handler.pagination,
-            'show_subed_only': show_subed_only,
+            "show_subed_only": show_subed_only,
-            'title': 'Channels',
+            "title": "Channels",
-            'colors': colors
+            "colors": colors,
+            "view_style": view_style,
         }
-        return render(request, 'home/channel.html', context)
+        return render(request, "home/channel.html", context)
 
     @staticmethod
     def read_config():
-        """ read config file """
+        """read config file"""
         config = AppConfig().config
-        es_url = config['application']['es_url']
+        es_url = config["application"]["es_url"]
-        colors = config['application']['colors']
+        colors = config["application"]["colors"]
-        return es_url, colors
+        view_style = config["default_view"]["channel"]
+        return es_url, colors, view_style
 
     def post(self, request):
-        """ handle http post requests """
+        """handle http post requests"""
         subscriptions_post = dict(request.POST)
         print(subscriptions_post)
         subscriptions_post = dict(request.POST)
-        if 'subscribe' in subscriptions_post.keys():
+        if "subscribe" in subscriptions_post.keys():
-            sub_str = subscriptions_post['subscribe']
+            sub_str = subscriptions_post["subscribe"]
             try:
                 youtube_ids = process_url_list(sub_str)
                 self.subscribe_to(youtube_ids)
             except ValueError:
-                print('parsing subscribe ids failed!')
+                print("parsing subscribe ids failed!")
                 print(sub_str)
 
         sleep(1)
-        return redirect('channel', permanent=True)
+        return redirect("channel", permanent=True)
 
     @staticmethod
     def subscribe_to(youtube_ids):
-        """ process the subscribe ids """
+        """process the subscribe ids"""
         for youtube_id in youtube_ids:
-            if youtube_id['type'] == 'video':
+            if youtube_id["type"] == "video":
-                to_sub = youtube_id['url']
+                to_sub = youtube_id["url"]
                 vid_details = PendingList().get_youtube_details(to_sub)
-                channel_id_sub = vid_details['channel_id']
+                channel_id_sub = vid_details["channel_id"]
-            elif youtube_id['type'] == 'channel':
+            elif youtube_id["type"] == "channel":
-                channel_id_sub = youtube_id['url']
+                channel_id_sub = youtube_id["url"]
             else:
-                raise ValueError('failed to subscribe to: ' + youtube_id)
+                raise ValueError("failed to subscribe to: " + youtube_id)
 
             ChannelSubscription().change_subscribe(
                 channel_id_sub, channel_subscribed=True
             )
-            print('subscribed to: ' + channel_id_sub)
+            print("subscribed to: " + channel_id_sub)
 
 
 class VideoView(View):
-    """ resolves to /video/<video-id>/
+    """resolves to /video/<video-id>/
     display details about a single video
     """
 
     def get(self, request, video_id):
-        """ get single video """
+        """get single video"""
         es_url, colors = self.read_config()
-        url = f'{es_url}/ta_video/_doc/{video_id}'
+        url = f"{es_url}/ta_video/_doc/{video_id}"
         data = None
         look_up = SearchHandler(url, data)
         video_hit = look_up.get_data()
-        video_data = video_hit[0]['source']
+        video_data = video_hit[0]["source"]
-        video_title = video_data['title']
+        video_title = video_data["title"]
-        context = {
+        context = {"video": video_data, "title": video_title, "colors": colors}
-            'video': video_data,
+        return render(request, "home/video.html", context)
-            'title': video_title,
-            'colors': colors
-        }
-        return render(request, 'home/video.html', context)
 
     @staticmethod
     def read_config():
-        """ read config file """
+        """read config file"""
         config = AppConfig().config
-        es_url = config['application']['es_url']
+        es_url = config["application"]["es_url"]
-        colors = config['application']['colors']
+        colors = config["application"]["colors"]
         return es_url, colors
 
 
 class SettingsView(View):
-    """ resolves to /settings/
+    """resolves to /settings/
     handle the settings page, display current settings,
     take post request from the form to update settings
     """
 
     @staticmethod
     def get(request):
-        """ read and display current settings """
+        """read and display current settings"""
         config = AppConfig().config
-        colors = config['application']['colors']
+        colors = config["application"]["colors"]
 
-        context = {
+        context = {"title": "Settings", "config": config, "colors": colors}
-            'title': 'Settings',
-            'config': config,
-            'colors': colors
-        }
 
-        return render(request, 'home/settings.html', context)
+        return render(request, "home/settings.html", context)
 
     @staticmethod
     def post(request):
-        """ handle form post to update settings """
+        """handle form post to update settings"""
         form_post = dict(request.POST)
-        del form_post['csrfmiddlewaretoken']
+        del form_post["csrfmiddlewaretoken"]
         print(form_post)
         config_handler = AppConfig()
         config_handler.update_config(form_post)
 
-        return redirect('settings', permanent=True)
+        return redirect("settings", permanent=True)
 
 
 def progress(request):
     # pylint: disable=unused-argument
-    """ endpoint for download progress ajax calls """
+    """endpoint for download progress ajax calls"""
     config = AppConfig().config
-    cache_dir = config['application']['cache_dir']
+    cache_dir = config["application"]["cache_dir"]
-    json_data = get_dl_message(cache_dir)
+    json_data = RedisArchivist().get_dl_message(cache_dir)
     return JsonResponse(json_data)
 
 
 def process(request):
-    """ handle all the buttons calls via POST ajax """
+    """handle all the buttons calls via POST ajax"""
-    if request.method == 'POST':
+    if request.method == "POST":
         post_dict = json.loads(request.body.decode())
         post_handler = PostData(post_dict)
-        if post_handler.to_do:
+        if post_handler.to_exec:
             task_result = post_handler.run_task()
             return JsonResponse(task_result)
 
-    return JsonResponse({'success': False})
+    return JsonResponse({"success": False})
 
 
 class PostData:
-    """ generic post handler from process route """
+    """
-
+    map frontend http post values to backend funcs
-    CONFIG = AppConfig().config
+    handover long running tasks to celery
-    ES_URL = CONFIG['application']['es_url']
+    """
-
-    VALID_KEYS = [
-        "watched", "rescan_pending", "ignore", "dl_pending",
-        "unsubscribe", "sort_order", "hide_watched", "show_subed_only",
-        "channel-search", "video-search", "dlnow", "manual-import"
-    ]
 
     def __init__(self, post_dict):
         self.post_dict = post_dict
-        self.to_do = self.validate()
+        self.to_exec, self.exec_val = list(post_dict.items())[0]
-
-    def validate(self):
-        """ validate the post_dict """
-        to_do = []
-        for key, value in self.post_dict.items():
-            if key in self.VALID_KEYS:
-                task_item = {'task': key, 'status': value}
-                print(task_item)
-                to_do.append(task_item)
-            else:
-                print(key + ' not a valid key')
-
-        return to_do
 
     def run_task(self):
-        """ run through the tasks to do """
+        """execute and return task result"""
-        for item in self.to_do:
+        to_exec = self.exec_map()
-            task = item['task']
+        task_result = to_exec()
-            if task == 'watched':
+        return task_result
-                youtube_id = item['status']
-                self.parse_watched(youtube_id)
-            elif task == 'rescan_pending':
-                print('rescan subscribed channels')
-                update_subscribed.delay()
-            elif task == 'ignore':
-                print('ignore video')
-                handler = PendingList()
-                ignore_list = item['status']
-                handler.ignore_from_pending([ignore_list])
-            elif task == 'dl_pending':
-                print('download pending')
-                download_pending.delay()
-            elif task == 'unsubscribe':
-                channel_id_unsub = item['status']
-                print('unsubscribe from ' + channel_id_unsub)
-                ChannelSubscription().change_subscribe(
-                    channel_id_unsub, channel_subscribed=False
-                )
-            elif task == 'sort_order':
-                sort_order = item['status']
-                set_message('sort_order', sort_order, expire=False)
-            elif task == 'hide_watched':
-                hide_watched = bool(int(item['status']))
-                print(item['status'])
-                set_message('hide_watched', hide_watched, expire=False)
-            elif task == 'show_subed_only':
-                show_subed_only = bool(int(item['status']))
-                print(show_subed_only)
-                set_message('show_subed_only', show_subed_only, expire=False)
-            elif task == 'channel-search':
-                search_query = item['status']
-                print('searching for: ' + search_query)
-                search_results = self.search_channels(search_query)
-                return search_results
-            elif task == 'video-search':
-                search_query = item['status']
-                print('searching for: ' + search_query)
-                search_results = self.search_videos(search_query)
-                return search_results
-            elif task == 'dlnow':
-                youtube_id = item['status']
-                print('downloading: ' + youtube_id)
-                download_single(youtube_id)
-            elif task == 'manual-import':
-                print('starting manual import')
-                run_manual_import.delay()
-        return {'success': True}
 
-    def search_channels(self, search_query):
+    def exec_map(self):
-        """ fancy searching channels as you type """
+        """map dict key and return function to execute"""
-        url = self.ES_URL + '/ta_channel/_search'
+        exec_map = {
-        data = {
+            "watched": self.watched,
-            "size": 10,
+            "change_view": self.change_view,
-            "query": {
+            "rescan_pending": self.rescan_pending,
-                "multi_match": {
+            "ignore": self.ignore,
-                    "query": search_query,
+            "dl_pending": self.dl_pending,
-                    "type": "bool_prefix",
+            "queue": self.queue_handler,
-                    "fields": [
+            "unsubscribe": self.unsubscribe,
-                        "channel_name.search_as_you_type",
+            "sort_order": self.sort_order,
-                        "channel_name._2gram",
+            "hide_watched": self.hide_watched,
-                        "channel_name._3gram"
+            "show_subed_only": self.show_subed_only,
-                    ]
+            "dlnow": self.dlnow,
-                }
+            "show_ignored_only": self.show_ignored_only,
-            }
+            "forgetIgnore": self.forget_ignore,
+            "addSingle": self.add_single,
+            "manual-import": self.manual_import,
+            "db-backup": self.db_backup,
+            "db-restore": self.db_restore,
+            "channel-search": self.channel_search,
         }
-        look_up = SearchHandler(url, data, cache=False)
-        search_results = look_up.get_data()
-        return {'results': search_results}
 
-    def search_videos(self, search_query):
+        return exec_map[self.to_exec]
-        """ fancy searching videos as you type """
-        url = self.ES_URL + '/ta_video/_search'
-        data = {
-            "size": 10,
-            "query": {
-                "multi_match": {
-                    "query": search_query,
-                    "type": "bool_prefix",
-                    "fields": [
-                        "title.search_as_you_type",
-                        "title._2gram",
-                        "title._3gram"
-                    ]
-                }
-            }
-        }
-        look_up = SearchHandler(url, data, cache=False)
-        search_results = look_up.get_data()
-        return {'results': search_results}
 
-    def parse_watched(self, youtube_id):
+    def watched(self):
-        """ marked as watched based on id type """
+        """mark as watched"""
-        es_url = self.ES_URL
+        WatchState(self.exec_val).mark_as_watched()
-        id_type = process_url_list([youtube_id])[0]['type']
+        return {"success": True}
-        stamp = int(datetime.now().strftime("%s"))
+
-        if id_type == 'video':
+    def change_view(self):
-            stamp = int(datetime.now().strftime("%s"))
+        """process view changes in home, channel, and downloads"""
-            url = self.ES_URL + '/ta_video/_update/' + youtube_id
+        origin, new_view = self.exec_val.split(":")
-            source = {
+        print(f"change view on page {origin} to {new_view}")
-                "doc": {"player": {"watched": True, "watched_date": stamp}}
+        update_dict = {f"default_view.{origin}": [new_view]}
-            }
+        AppConfig().update_config(update_dict)
-            request = requests.post(url, json=source)
+        return {"success": True}
-            if not request.ok:
+
-                print(request.text)
+    @staticmethod
-        elif id_type == 'channel':
+    def rescan_pending():
-            headers = {'Content-type': 'application/json'}
+        """look for new items in subscribed channels"""
-            data = {
+        print("rescan subscribed channels")
-                "description": youtube_id,
+        update_subscribed.delay()
-                "processors": [
+        return {"success": True}
-                    {"set": {"field": "player.watched", "value": True}},
+
-                    {"set": {"field": "player.watched_date", "value": stamp}}
+    def ignore(self):
-                ]
+        """ignore from download queue"""
-            }
+        id_to_ignore = self.exec_val
-            payload = json.dumps(data)
+        print("ignore video " + id_to_ignore)
-            url = es_url + '/_ingest/pipeline/' + youtube_id
+        handler = PendingList()
-            request = requests.put(url, data=payload, headers=headers)
+        handler.ignore_from_pending([id_to_ignore])
-            if not request.ok:
+        # also clear from redis queue
-                print(request.text)
+        RedisQueue("dl_queue").clear_item(id_to_ignore)
-            # apply pipeline
+        return {"success": True}
-            must_list = [
+
-                {"term": {"channel.channel_id": {"value": youtube_id}}},
+    @staticmethod
-                {"term": {"player.watched": {"value": False}}}
+    def dl_pending():
-            ]
+        """start the download queue"""
-            data = {"query": {"bool": {"must": must_list}}}
+        print("download pending")
-            payload = json.dumps(data)
+        running = download_pending.delay()
-            url = f'{es_url}/ta_video/_update_by_query?pipeline={youtube_id}'
+        task_id = running.id
-            request = requests.post(url, data=payload, headers=headers)
+        print("set task id: " + task_id)
-            if not request.ok:
+        RedisArchivist().set_message("dl_queue_id", task_id, expire=False)
-                print(request.text)
+        return {"success": True}
+
+    def queue_handler(self):
+        """queue controls from frontend"""
+        to_execute = self.exec_val
+        if to_execute == "stop":
+            print("stopping download queue")
+            RedisQueue("dl_queue").clear()
+        elif to_execute == "kill":
+            task_id = RedisArchivist().get_message("dl_queue_id")
+            print("brutally killing " + task_id)
+            kill_dl(task_id)
+
+        return {"success": True}
+
+    def unsubscribe(self):
+        """unsubscribe from channel"""
+        channel_id_unsub = self.exec_val
+        print("unsubscribe from " + channel_id_unsub)
+        ChannelSubscription().change_subscribe(
+            channel_id_unsub, channel_subscribed=False
+        )
+        return {"success": True}
+
+    def sort_order(self):
+        """change the sort between published to downloaded"""
+        sort_order = self.exec_val
+        RedisArchivist().set_message("sort_order", sort_order, expire=False)
+        return {"success": True}
+
+    def hide_watched(self):
+        """toggle if to show watched vids or not"""
+        hide_watched = bool(int(self.exec_val))
+        print(f"hide watched: {hide_watched}")
+        RedisArchivist().set_message(
+            "hide_watched", hide_watched, expire=False
+        )
+        return {"success": True}
+
+    def show_subed_only(self):
+        """show or hide subscribed channels only on channels page"""
+        show_subed_only = bool(int(self.exec_val))
+        print(f"show subed only: {show_subed_only}")
+        RedisArchivist().set_message(
+            "show_subed_only", show_subed_only, expire=False
+        )
+        return {"success": True}
+
+    def dlnow(self):
+        """start downloading single vid now"""
+        youtube_id = self.exec_val
+        print("downloading: " + youtube_id)
+        running = download_single.delay(youtube_id=youtube_id)
+        task_id = running.id
+        print("set task id: " + task_id)
+        RedisArchivist().set_message("dl_queue_id", task_id, expire=False)
+        return {"success": True}
+
+    def show_ignored_only(self):
+        """switch view on /downloads/ to show ignored only"""
+        show_value = self.exec_val
+        print(f"Filter download view ignored only: {show_value}")
+        RedisArchivist().set_message(
+            "show_ignored_only", {"status": show_value}, expire=False
+        )
+        return {"success": True}
+
+    def forget_ignore(self):
+        """delete from ta_download index"""
+        youtube_id = self.exec_val
+        print("forgetting from download index: " + youtube_id)
+        PendingList().delete_from_pending(youtube_id)
+        return {"success": True}
+
+    def add_single(self):
+        """add single youtube_id to download queue"""
+        youtube_id = self.exec_val
+        print("add vid to dl queue: " + youtube_id)
+        PendingList().delete_from_pending(youtube_id)
+        youtube_ids = process_url_list([youtube_id])
+        extrac_dl.delay(youtube_ids)
+        return {"success": True}
+
+    @staticmethod
+    def manual_import():
+        """run manual import from settings page"""
+        print("starting manual import")
+        run_manual_import.delay()
+        return {"success": True}
+
+    @staticmethod
+    def db_backup():
+        """backup es to zip from settings page"""
+        print("backing up database")
+        run_backup.delay()
+        return {"success": True}
+
+    @staticmethod
+    def db_restore():
+        """restore es zip from settings page"""
+        print("restoring index from backup zip")
+        run_restore_backup.delay()
+        return {"success": True}
+
+    def channel_search(self):
+        """search for channel name as_you_type"""
+        search_query = self.exec_val
+        print("searching for: " + search_query)
+        search_results = SearchForm().search_channels(search_query)
+        return search_results

tubearchivist/manage.py

@@ -5,8 +5,9 @@ import sys
 
 
 def main():
+    # pylint: disable=import-outside-toplevel
     """Run administrative tasks."""
-    os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings')
+    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings")
     try:
         from django.core.management import execute_from_command_line
     except ImportError as exc:
@@ -18,5 +19,5 @@ def main():
     execute_from_command_line(sys.argv)
 
 
-if __name__ == '__main__':
+if __name__ == "__main__":
     main()

tubearchivist/requirements.txt

@@ -6,4 +6,4 @@ redis==3.5.3
 requests==2.26.0
 uWSGI==2.0.19.1
 whitenoise==5.3.0
-yt_dlp==2021.9.2
+yt_dlp==2021.9.25

tubearchivist/static/css/dark.css

@@ -7,4 +7,5 @@
     --accent-font-dark: #259485;
     --accent-font-light: #97d4c8;
     --img-filter: invert(50%) sepia(9%) saturate(2940%) hue-rotate(122deg) brightness(94%) contrast(90%);
+    --img-filter-error: invert(16%) sepia(60%) saturate(3717%) hue-rotate(349deg) brightness(86%) contrast(120%);
 }

tubearchivist/static/css/light.css

@@ -7,4 +7,5 @@
     --accent-font-dark: #259485;
     --accent-font-light: #35b399;
     --img-filter: invert(50%) sepia(9%) saturate(2940%) hue-rotate(122deg) brightness(94%) contrast(90%);
+    --img-filter-error: invert(83%) sepia(35%) saturate(1238%) hue-rotate(297deg) brightness(103%) contrast(97%);
 }

tubearchivist/static/css/style.css

@@ -138,6 +138,73 @@ button:hover {
     text-decoration: underline;
 }
 
+/* toggle on-off */
+.toggle {
+    display: flex;
+    align-items: center;
+}
+
+.toggleBox > input[type="checkbox"] {
+    position: relative;
+    width: 70px;
+    height: 30px;
+    background-color: var(--accent-font-dark);
+    border-color: var(--accent-font-dark);
+    appearance: none;
+    border-radius: 15px;
+    transition: 0.4s;
+    box-shadow: inset 0 0 5px rgba(0, 0, 0, 0.2);
+    cursor: pointer;
+}
+
+.toggleBox > input:checked[type="checkbox"] {
+    background-color: var(--accent-font-light);
+    border-color: var(--accent-font-light);
+}
+
+.toggleBox > input[type="checkbox"]::before {
+    z-index: 2;
+    position: absolute;
+    content: "";
+    left: 0;
+    top: 0;
+    width: 30px;
+    height: 30px;
+    background-color: white;
+    border-radius: 50%;
+    transform: scale(1.1);
+    transition: 0.4s;
+}
+
+.toggleBox > input:checked[type="checkbox"]::before {
+    left: 40px;
+}
+
+.toggleBox {
+    margin-left: 10px;
+    position: relative;
+    display: inline;
+}
+
+.toggleBox > label {
+    position: absolute;
+    color: var(--main-font);
+    pointer-events: none;
+}
+
+.toggleBox > .onbtn {
+    bottom: 15px;
+    left: 15px;
+    font-family: Sen-Regular, sans-serif;
+}
+
+.toggleBox > .ofbtn {
+    bottom: 15px;
+    right: 15px;
+    font-family: Sen-Regular, sans-serif;
+    color: var(--main-font);
+}
+
 /* navigation */
 .top-nav {
     display: block;
@@ -176,7 +243,7 @@ button:hover {
 
 /* top of page */
 .title-bar {
-    padding: 25px 0;
+    padding: 20px 0;
 }
 
 .sort {
@@ -211,6 +278,26 @@ button:hover {
     filter: var(--img-filter);
 }
 
+.view-controls {
+    display: grid;
+    grid-template-columns: 1fr 1fr;
+    border-bottom: 2px solid;
+    border-color: var(--accent-font-dark);
+    margin: 15px 0;
+}
+
+.view-icons {
+    display: flex;
+    justify-content: end;
+}
+
+.view-icons img {
+    width: 30px;
+    margin: 5px 10px;
+    cursor: pointer;
+    filter: var(--img-filter);
+}
+
 #search-box {
     display: none;
     flex: auto;
@@ -261,16 +348,27 @@ button:hover {
 
 
 /* video list */
-.video-list {
+.video-list.grid {
     display: grid;
     grid-template-columns: 1fr 1fr 1fr;
     grid-gap: 1rem;
 }
 
+.video-list.list {
+    display: grid;
+    grid-template-columns: unset;
+    grid-gap: 1rem;
+}
+
 .video-item {
     overflow: hidden;
 }
 
+.video-item.list {
+    display: grid;
+    grid-template-columns: 25% auto;
+}
+
 .video-thumb img {
     width: 100%;
 }
@@ -301,12 +399,25 @@ button:hover {
     padding: 8px;
 }
 
-.video-desc {
+.video-desc.grid {
     padding: 10px;
     height: 100%;
     background-color: var(--highlight-bg);
 }
 
+.video-desc.list {
+    padding: 10px;
+    height: unset;
+    background-color: var(--highlight-bg);
+    display: flex;
+    flex-wrap: wrap-reverse;
+    align-content: center;
+}
+
+.video-desc > div {
+    width: 100%;
+}
+
 .video-desc img {
     width: 20px;
     margin-right: 10px;
@@ -362,6 +473,7 @@ button:hover {
 .info-box {
     display: grid;
     grid-gap: 1rem;
+    margin: 1rem 0;
 }
 
 .info-box-3 {
@@ -417,14 +529,41 @@ button:hover {
 }
 
 /* channel overview page */
-.channel-item {
+.channel-list.list {
+    display: block;
+}
+
+.channel-list.grid {
+    display: grid;
+    grid-template-columns: 1fr 1fr 1fr;
+    gap: 1rem;
+}
+
+.channel-item.list {
     padding: 20px 0;
 }
 
+.channel-item.grid > .info-box {
+    display: block;
+}
+
 .channel-banner img {
     width: 100%;
 }
 
+.channel-banner.grid {
+    overflow: hidden;
+}
+
+.channel-banner.list img {
+    width: 100%;
+}
+
+.channel-banner.grid img {
+    width: 250%;
+    transform: translateX(-30%);
+}
+
 
 /* download page */
 .icon-text {
@@ -438,30 +577,74 @@ button:hover {
     cursor: pointer;
 }
 
-.dl-item {
+.dl-list.list {
+    display: block;
+}
+
+.dl-list.grid {
+    display: grid;
+    grid-template-columns: 1fr 1fr 1fr;
+    grid-gap: 1rem;
+}
+
+.dl-item.list {
     display: flex;
     margin: 15px 0;
     align-items: center;
     background-color: var(--highlight-bg);
 }
 
+.dl-item.grid {
+    display: flex;
+    flex-wrap: wrap;
+    background-color: var(--highlight-bg);
+}
+
 .dl-check {
     width: 30px;
 }
 
-.dl-thumb {
+.dl-thumb.list {
     width: 25%;
 }
 
+.dl-thumb.grid {
+    width: 100%;
+}
+
 .dl-item img {
     width: 100%;
 }
 
-.dl-desc {
+.dl-desc.list {
-    padding-left: 15px;
+    padding: 0 15px;
     width: 75%;
 }
 
+.dl-desc.grid {
+    padding: 15px;
+    width: 100%;
+}
+
+.dl-control-icons {
+    display: flex;
+    justify-content: center;
+    padding: 10px 0;
+}
+
+.dl-control-icons img {
+    width: 30px;
+    cursor: pointer;
+    margin: 5px;
+}
+
+#stop-icon {
+    filter: var(--img-filter);
+}
+
+#kill-icon {
+    filter: var(--img-filter-error);
+}
 
 /* status message */
 .download-progress {
@@ -493,6 +676,11 @@ button:hover {
     min-width: 300px;
 }
 
+.danger-zone {
+    background-color: var(--highlight-error);
+    padding: 3px;
+}
+
 /* about */
 .about-section {
     padding: 20px 0;
@@ -562,9 +750,18 @@ button:hover {
     .boxed-content {
         width: 90%;
     }
-    .video-list {
+    .video-list.grid,
+    .dl-list.grid,
+    .channel-list.grid {
         grid-template-columns: 1fr 1fr;
     }
+    .dl-thumb.list {
+        width: 35%;
+    }
+    .video-item.list {
+        display: grid;
+        grid-template-columns: 35% auto;
+    }
     .two-col {
         display: block;
     }
@@ -578,15 +775,26 @@ button:hover {
     * {
         word-wrap: anywhere;
     }
-    .video-list {
+    .video-list.grid,
+    .dl-list.grid,
+    .channel-list.grid,
+    .video-item.list {
         grid-template-columns: 1fr;
     }
+    .video-desc.grid {
+        height: unset;
+        display: flex;
+        flex-wrap: wrap-reverse;
+    }
     .boxed-content {
         width: 95%;
     }
     .footer {
         text-align: center;
     }
+    .toggle {
+        flex-wrap: wrap;
+    }
     .top-nav {
         flex-wrap: wrap-reverse;
         display: flex;

tubearchivist/static/img/icon-gridview.svg

@@ -0,0 +1,122 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   xmlns:dc="http://purl.org/dc/elements/1.1/"
+   xmlns:cc="http://creativecommons.org/ns#"
+   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+   xmlns:svg="http://www.w3.org/2000/svg"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   width="2000"
+   height="2000"
+   viewBox="0 0 529.16666 529.16735"
+   version="1.1"
+   id="svg8"
+   inkscape:version="0.92.4 (5da689c313, 2019-01-14)"
+   sodipodi:docname="Gridview.svg">
+  <defs
+     id="defs2" />
+  <sodipodi:namedview
+     id="base"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageopacity="0.0"
+     inkscape:pageshadow="2"
+     inkscape:zoom="0.35729063"
+     inkscape:cx="901.7564"
+     inkscape:cy="1021.9111"
+     inkscape:document-units="mm"
+     inkscape:current-layer="layer1"
+     showgrid="false"
+     units="px"
+     showguides="true"
+     inkscape:guide-bbox="true"
+     inkscape:window-width="1920"
+     inkscape:window-height="1017"
+     inkscape:window-x="-8"
+     inkscape:window-y="-8"
+     inkscape:window-maximized="1">
+    <sodipodi:guide
+       position="247.25932,291.92959"
+       orientation="1,0"
+       id="guide853"
+       inkscape:locked="false" />
+    <sodipodi:guide
+       position="337.22901,167.98535"
+       orientation="0,1"
+       id="guide855"
+       inkscape:locked="false" />
+    <sodipodi:guide
+       position="266.76325,305.4565"
+       orientation="0,1"
+       id="guide857"
+       inkscape:locked="false" />
+    <sodipodi:guide
+       position="257.79774,279.50371"
+       orientation="0,1"
+       id="guide861"
+       inkscape:locked="false" />
+  </sodipodi:namedview>
+  <metadata
+     id="metadata5">
+    <rdf:RDF>
+      <cc:Work
+         rdf:about="">
+        <dc:format>image/svg+xml</dc:format>
+        <dc:type
+           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+        <dc:title></dc:title>
+      </cc:Work>
+    </rdf:RDF>
+  </metadata>
+  <g
+     inkscape:label="Ebene 1"
+     inkscape:groupmode="layer"
+     id="layer1"
+     transform="translate(0,232.16736)">
+    <g
+       id="g873"
+       transform="matrix(1.3431799,0,0,1.3431799,-84.854433,26.13855)"
+       style="stroke:none">
+      <rect
+         ry="7.445024"
+         rx="7.445024"
+         y="-121.39048"
+         x="79.903137"
+         height="113.24854"
+         width="167.35619"
+         id="rect815"
+         style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.56300002;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:1.12600005, 0.56300002999999998;stroke-dashoffset:0;stroke-opacity:0.22508042;paint-order:markers fill stroke" />
+      <rect
+         ry="7.445024"
+         rx="7.445024"
+         y="-121.7837"
+         x="273.05484"
+         height="113.24854"
+         width="167.35619"
+         id="rect815-4"
+         style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.56300002;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:1.12600006, 0.56300004000000003;stroke-dashoffset:0;stroke-opacity:0.22508042;paint-order:markers fill stroke" />
+      <rect
+         ry="7.445024"
+         rx="7.445024"
+         y="17.882772"
+         x="79.903137"
+         height="113.24854"
+         width="167.35619"
+         id="rect815-2"
+         style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.56300002;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:1.12600006, 0.56300004000000003;stroke-dashoffset:0;stroke-opacity:0.22508042;paint-order:markers fill stroke" />
+      <rect
+         ry="7.445024"
+         rx="7.445024"
+         y="17.453594"
+         x="273.05484"
+         height="113.24854"
+         width="167.35619"
+         id="rect815-7"
+         style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.56300002;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:1.12600006, 0.56300004000000003;stroke-dashoffset:0;stroke-opacity:0.22508042;paint-order:markers fill stroke" />
+    </g>
+  </g>
+</svg>

tubearchivist/static/img/icon-listview.svg

@@ -0,0 +1,122 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   xmlns:dc="http://purl.org/dc/elements/1.1/"
+   xmlns:cc="http://creativecommons.org/ns#"
+   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+   xmlns:svg="http://www.w3.org/2000/svg"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   width="2000"
+   height="2000"
+   viewBox="0 0 529.16666 529.16735"
+   version="1.1"
+   id="svg8"
+   inkscape:version="0.92.4 (5da689c313, 2019-01-14)"
+   sodipodi:docname="Listview.svg">
+  <defs
+     id="defs2" />
+  <sodipodi:namedview
+     id="base"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageopacity="0.0"
+     inkscape:pageshadow="2"
+     inkscape:zoom="0.42053519"
+     inkscape:cx="851.82064"
+     inkscape:cy="1105.7974"
+     inkscape:document-units="mm"
+     inkscape:current-layer="g873"
+     showgrid="false"
+     units="px"
+     showguides="true"
+     inkscape:guide-bbox="true"
+     inkscape:window-width="1920"
+     inkscape:window-height="1017"
+     inkscape:window-x="-8"
+     inkscape:window-y="-8"
+     inkscape:window-maximized="1">
+    <sodipodi:guide
+       position="247.25932,291.92959"
+       orientation="1,0"
+       id="guide853"
+       inkscape:locked="false" />
+    <sodipodi:guide
+       position="337.22901,167.98535"
+       orientation="0,1"
+       id="guide855"
+       inkscape:locked="false" />
+    <sodipodi:guide
+       position="266.76325,305.4565"
+       orientation="0,1"
+       id="guide857"
+       inkscape:locked="false" />
+    <sodipodi:guide
+       position="257.79774,279.50371"
+       orientation="0,1"
+       id="guide861"
+       inkscape:locked="false" />
+    <sodipodi:guide
+       position="22.413775,336.67894"
+       orientation="1,0"
+       id="guide926"
+       inkscape:locked="false" />
+    <sodipodi:guide
+       position="503.71355,217.50031"
+       orientation="1,0"
+       id="guide928"
+       inkscape:locked="false" />
+  </sodipodi:namedview>
+  <metadata
+     id="metadata5">
+    <rdf:RDF>
+      <cc:Work
+         rdf:about="">
+        <dc:format>image/svg+xml</dc:format>
+        <dc:type
+           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+        <dc:title></dc:title>
+      </cc:Work>
+    </rdf:RDF>
+  </metadata>
+  <g
+     inkscape:label="Ebene 1"
+     inkscape:groupmode="layer"
+     id="layer1"
+     transform="translate(0,232.16736)">
+    <g
+       id="g873"
+       transform="matrix(1.3431799,0,0,1.3431799,-84.854433,26.13855)">
+      <rect
+         ry="7.445024"
+         rx="7.445024"
+         y="-121.34892"
+         x="79.944702"
+         height="70.107315"
+         width="358.24551"
+         id="rect815"
+         style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.64810181;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:1.29620369, 0.64810185;stroke-dashoffset:0;stroke-opacity:0.22508042;paint-order:markers fill stroke" />
+      <rect
+         ry="7.4450235"
+         rx="7.4450235"
+         y="-31.167784"
+         x="79.861153"
+         height="70.107315"
+         width="358.32907"
+         id="rect815-20"
+         style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.64817739;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:1.29635485, 0.64817743;stroke-dashoffset:0;stroke-opacity:0.22508042;paint-order:markers fill stroke" />
+      <rect
+         ry="7.4450235"
+         rx="7.4450231"
+         y="60.024326"
+         x="79.908524"
+         height="70.106117"
+         width="358.28171"
+         id="rect815-8"
+         style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.64812905;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:1.29625813, 0.64812907;stroke-dashoffset:0;stroke-opacity:0.22508042;paint-order:markers fill stroke" />
+    </g>
+  </g>
+</svg>

tubearchivist/static/img/icon-stop.svg

@@ -0,0 +1,67 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   xmlns:dc="http://purl.org/dc/elements/1.1/"
+   xmlns:cc="http://creativecommons.org/ns#"
+   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+   xmlns:svg="http://www.w3.org/2000/svg"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   width="500"
+   height="500"
+   viewBox="0 0 132.29197 132.29167"
+   version="1.1"
+   id="svg1303"
+   inkscape:version="0.92.4 (5da689c313, 2019-01-14)"
+   sodipodi:docname="Icons_stop.svg">
+  <defs
+     id="defs1297" />
+  <sodipodi:namedview
+     id="base"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageopacity="0.0"
+     inkscape:pageshadow="2"
+     inkscape:zoom="1.0105705"
+     inkscape:cx="43.182711"
+     inkscape:cy="168.09972"
+     inkscape:document-units="mm"
+     inkscape:current-layer="layer1"
+     showgrid="false"
+     units="px"
+     inkscape:window-width="1920"
+     inkscape:window-height="1017"
+     inkscape:window-x="-8"
+     inkscape:window-y="-8"
+     inkscape:window-maximized="1" />
+  <metadata
+     id="metadata1300">
+    <rdf:RDF>
+      <cc:Work
+         rdf:about="">
+        <dc:format>image/svg+xml</dc:format>
+        <dc:type
+           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+        <dc:title></dc:title>
+      </cc:Work>
+    </rdf:RDF>
+  </metadata>
+  <g
+     inkscape:label="Ebene 1"
+     inkscape:groupmode="layer"
+     id="layer1"
+     transform="translate(0,-164.70764)">
+    <rect
+       style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;paint-order:markers fill stroke"
+       id="rect836"
+       width="118.86465"
+       height="118.86465"
+       x="6.7136617"
+       y="171.42116"
+       rx="10.00003"
+       ry="10.00003" />
+  </g>
+</svg>

tubearchivist/static/progress.js

@@ -15,9 +15,13 @@ function checkMessage() {
     req.open('GET', '/downloads/progress', true);
     req.onload = function() {
         var dlProgress = req.response;
+        // var dlStatus = dlProgress['status'];
         if (dlProgress['status']) {
             buildDownloadMessage(dlProgress);
             handleInterval();
+            // if (dlStatus == 'downloading') {
+            //     buildDownloadIcons();
+            // };
         };
     };
     req.send();
@@ -70,4 +74,33 @@ function buildDownloadMessage(dlProgress) {
     message.appendChild(title);
     message.appendChild(messageText);
     box.appendChild(message);
+    if (dlStatus == 'downloading' && dlLevel != 'error') {
+        box.appendChild(buildDownloadIcons());
+    };
 };
+
+
+// add dl control icons
+function buildDownloadIcons() {
+    var iconBox = document.createElement('div');
+    iconBox.classList = 'dl-control-icons';
+    // stop icon
+    var stopIcon = document.createElement('img');
+    stopIcon.setAttribute('id', "stop-icon");
+    stopIcon.setAttribute('title', "Stop Download Queue");
+    stopIcon.setAttribute('src', "/static/img/icon-stop.svg");
+    stopIcon.setAttribute('alt', "stop icon");
+    stopIcon.setAttribute('onclick', 'stopQueue()');
+    // kill icon
+    var killIcon = document.createElement('img');
+    killIcon.setAttribute('id', "kill-icon");
+    killIcon.setAttribute('title', "Kill Download Queue");
+    killIcon.setAttribute('src', "/static/img/icon-close.svg");
+    killIcon.setAttribute('alt', "kill icon");
+    killIcon.setAttribute('onclick', 'killQueue()');
+    // stich together
+    iconBox.appendChild(stopIcon);
+    iconBox.appendChild(killIcon);
+    
+    return iconBox
+}

tubearchivist/static/script.js

@@ -8,24 +8,6 @@ function sortChange(sortValue) {
     }, 500);
 }
 
-function hideWatched(hideValue) {
-    var payload = JSON.stringify({'hide_watched': hideValue});
-    sendPost(payload);
-    setTimeout(function(){
-        location.reload();
-        return false;
-    }, 500);
-}
-
-function showSubscribedOnly(showValue) {
-    var payload = JSON.stringify({'show_subed_only': showValue});
-    sendPost(payload);
-    setTimeout(function(){
-        location.reload();
-        return false;
-    }, 500);
-}
-
 function isWatched(youtube_id) {
     var payload = JSON.stringify({'watched': youtube_id});
     sendPost(payload);
@@ -43,6 +25,32 @@ function unsubscribe(channel_id) {
     document.getElementById(channel_id).remove();
 }
 
+function changeView(image) {
+    var sourcePage = image.getAttribute("data-origin");
+    var newView = image.getAttribute("data-value");
+    var payload = JSON.stringify({'change_view': sourcePage + ":" + newView});
+    sendPost(payload);
+    setTimeout(function(){
+        location.reload();
+        return false;
+    }, 500);
+}
+
+function toggleCheckbox(checkbox) {
+    // pass checkbox id as key and checkbox.checked as value
+    var toggleId = checkbox.id;
+    var toggleVal = checkbox.checked;
+    var payloadDict = {};
+    payloadDict[toggleId] = toggleVal;
+    var payload = JSON.stringify(payloadDict);
+    sendPost(payload);
+    setTimeout(function(){
+        var currPage = window.location.pathname;
+        window.location.replace(currPage);
+        return false;
+    }, 500);
+}
+
 // download page buttons
 function rescanPending() {
     var payload = JSON.stringify({'rescan_pending': true});
@@ -72,13 +80,42 @@ function toIgnore(button) {
 function downloadNow(button) {
     var youtube_id = button.getAttribute('data-id');
     var payload = JSON.stringify({'dlnow': youtube_id});
-    animate('download-icon', 'bounce-img');
     sendPost(payload);
+    document.getElementById(youtube_id).remove();
     setTimeout(function(){
         handleInterval();
     }, 500);
 }
 
+function forgetIgnore(button) {
+    var youtube_id = button.getAttribute('data-id');
+    var payload = JSON.stringify({'forgetIgnore': youtube_id});
+    sendPost(payload);
+    document.getElementById("dl-" + youtube_id).remove();
+}
+
+function addSingle(button) {
+    var youtube_id = button.getAttribute('data-id');
+    var payload = JSON.stringify({'addSingle': youtube_id});
+    sendPost(payload);
+    document.getElementById("dl-" + youtube_id).remove();
+    setTimeout(function(){
+        handleInterval();
+    }, 500);
+}
+
+function stopQueue() {
+    var payload = JSON.stringify({'queue': 'stop'});
+    sendPost(payload);
+    document.getElementById('stop-icon').remove();
+}
+
+function killQueue() {
+    var payload = JSON.stringify({'queue': 'kill'});
+    sendPost(payload);
+    document.getElementById('kill-icon').remove();
+}
+
 // settings page buttons
 function manualImport() {
     var payload = JSON.stringify({'manual-import': true});
@@ -91,6 +128,28 @@ function manualImport() {
     toReplace.appendChild(message);
 }
 
+function dbBackup() {
+    var payload = JSON.stringify({'db-backup': true});
+    sendPost(payload)
+    // clear button
+    var message = document.createElement('p');
+    message.innerText = 'backing up archive';
+    var toReplace = document.getElementById('db-backup');
+    toReplace.innerHTML = '';
+    toReplace.appendChild(message);
+}
+
+function dbRestore() {
+    var payload = JSON.stringify({'db-restore': true});
+    sendPost(payload)
+    // clear button
+    var message = document.createElement('p');
+    message.innerText = 'restoring from backup';
+    var toReplace = document.getElementById('db-restore');
+    toReplace.innerHTML = '';
+    toReplace.appendChild(message);
+}
+
 // player
 function createPlayer(button) {
     var mediaUrl = button.getAttribute('data-src');

version_check.py

@@ -1,17 +1,17 @@
 #!/usr/bin/env python
 """ check requirements.txt for outdated packages """
 
-import sys
 import pathlib
+import sys
 
 import requests
 
 
 class Requirements:
-    """ handle requirements.txt """
+    """handle requirements.txt"""
 
-    FILE_PATH = 'tubearchivist/requirements.txt'
+    FILE_PATH = "tubearchivist/requirements.txt"
-    LOCK = '/tmp/tubearchivist-requirements.lock'
+    LOCK = "/tmp/tubearchivist-requirements.lock"
 
     def __init__(self):
         self.exists = self.checked_today()
@@ -19,58 +19,57 @@ class Requirements:
         self.all_updates = False
 
     def checked_today(self):
-        """ skip requirements check when lock file exists """
+        """skip requirements check when lock file exists"""
         exists = pathlib.Path(self.LOCK).exists()
         return exists
 
     def look_for_updates(self):
-        """ look through requirements and check for updates """
+        """look through requirements and check for updates"""
         self.all_requirements = self.get_dependencies()
         self.all_updates = self.check_packages()
 
     def get_dependencies(self):
-        """ read out requirements.txt """
+        """read out requirements.txt"""
 
         all_requirements = []
-        with open(self.FILE_PATH, 'r', encoding='utf-8') as f:
+        with open(self.FILE_PATH, "r", encoding="utf-8") as f:
             dependencies = f.readlines()
 
         for dependency in dependencies:
-            package, version = dependency.split('==')
+            package, version = dependency.split("==")
             all_requirements.append((package, version.strip()))
 
-        all_requirements.sort(key = lambda x: x[0].lower())
+        all_requirements.sort(key=lambda x: x[0].lower())
 
         return all_requirements
 
     def check_packages(self):
-        """ compare installed with remote version """
+        """compare installed with remote version"""
 
         total = len(self.all_requirements)
-        print(f'checking versions for {total} packages...')
+        print(f"checking versions for {total} packages...")
 
         all_updates = {}
 
         for dependency in self.all_requirements:
             package, version_installed = dependency
-            url = f'https://pypi.org/pypi/{package}/json'
+            url = f"https://pypi.org/pypi/{package}/json"
             response = requests.get(url).json()
-            version_remote = response['info']['version']
+            version_remote = response["info"]["version"]
-            homepage = response['info']['home_page']
+            homepage = response["info"]["home_page"]
             if version_remote != version_installed:
                 to_update = {
-                    package: {
+                    package: {"from": version_installed, "to": version_remote}
-                        "from": version_installed,
-                        "to": version_remote
-                    }
                 }
                 all_updates.update(to_update)
-                message = (f'update {package} {version_installed}' +
+                message = (
-                           f'==> {version_remote}\n    {homepage}')
+                    f"update {package} {version_installed}"
+                    + f"==> {version_remote}\n    {homepage}"
+                )
                 print(message)
 
         if not all_updates:
-            print('no updates found')
+            print("no updates found")
 
         # remember that
         pathlib.Path(self.LOCK).touch()
@@ -78,7 +77,7 @@ class Requirements:
         return all_updates
 
     def apply_updates(self):
-        """ update requirements.txt file with new versions """
+        """update requirements.txt file with new versions"""
 
         to_write = []
 
@@ -86,32 +85,33 @@ class Requirements:
             package, old_version = requirement
 
             if package in self.all_updates.keys():
-                package_version = self.all_updates[package]['to']
+                package_version = self.all_updates[package]["to"]
             else:
                 package_version = old_version
 
-            to_write.append(f'{package}=={package_version}\n')
+            to_write.append(f"{package}=={package_version}\n")
 
-        with open(self.FILE_PATH, 'w', encoding='utf-8') as f:
+        with open(self.FILE_PATH, "w", encoding="utf-8") as f:
             f.writelines(to_write)
 
-        print('requirements.txt updates')
+        print("requirements.txt updates")
 
 
 def main():
-    """ main to check for updates """
+    """main to check for updates"""
     handler = Requirements()
     if handler.exists:
         return
 
     handler.look_for_updates()
     if handler.all_updates:
-        input_response = input('\nupdate requirements.txt? [y/n] ')
+        input_response = input("\nupdate requirements.txt? [y/n] ")
-        if input_response == 'y':
+        if input_response == "y":
             handler.apply_updates()
         else:
-            print('cancle update...')
+            print("cancel update...")
             sys.exit(1)
 
+
 if __name__ == "__main__":
     main()