add note about channel playlists

* Update settings.html

Fixed the example as you wouldn't put KB/s at the end, reworded the explanation, and added "KB/s" at the end of the "Current throttled rate limit:"

Also added KB/s to the download speed limit, and added an example.

* Update settings.html

.dockerignore

@@ -17,5 +17,8 @@ venv/
 # Unneeded graphics
 assets/*
 
+# Unneeded docs
+docs/*
+
 # for local testing only
 testing.sh

.gitignore

@@ -3,6 +3,3 @@ __pycache__
 
 # django testing db
 db.sqlite3
-
-# frontend fonts
-*ttf.woff

CONTRIBUTING.md

@@ -6,7 +6,37 @@ If you haven't already, the best place to start is the README. This will give yo
 ## Report a bug
 
 If you notice something is not working as expected, check to see if it has been previously reported in the [open issues](https://github.com/bbilly1/tubearchivist/issues).
-If it has not yet been disclosed, go ahead and create an issue.
+If it has not yet been disclosed, go ahead and create an issue.  
+If the issue doesn't move forward due to a lack of response, I assume it's solved and will close it after some time to keep the list fresh. 
+
+## Wiki
+
+The wiki is where all user functions are explained in detail. These pages are mirrored into the **docs** folder of the repo. This allows for pull requests and all other features like regular code. Make any changes there, and I'll sync them with the wiki tab.
+
+## Development Environment
+
+I have learned the hard way, that working on a dockerized application outside of docker is very error prone and in general not a good idea. So if you want to test your changes, it's best to run them in a docker testing environment.  
+
+This is my setup I have landed on, YMMV:
+- Clone the repo, work on it with your favorite code editor in your local filesystem. *testing* branch is the where all the changes are happening, might be unstable and is WIP.
+- Then I have a VM on KVM hypervisor running standard Ubuntu Server LTS with docker installed. The VM keeps my projects separate and offers convenient snapshot functionality. The VM also offers ways to simulate lowend environments by limiting CPU cores and memory. But you could also just run docker on your host system.
+- The `Dockerfile` is structured in a way that the actual application code is in the last layer so rebuilding the image with only code changes utilizes the build cache for everything else and will take just 2-3 secs.
+- Take a look at the `deploy.sh` file. I have my local DNS resolve `tubearchivist.local` to the IP of the VM for convenience. To deploy the latest changes and rebuild the application to the testing VM run:
+```bash
+./deploy.sh test
+```
+- The command above will also copy the file `tubarchivist/testing.sh` into the working folder of the container. Running this script will install additional debugging tools I regularly use in testing. 
+- The `test` argument takes another optional argument to build for a specific architecture valid options are: `amd64`, `arm64` and `multi`, default is `amd64`.
+- This `deploy.sh` file is not meant to be universally usable for every possible environment but could serve as an idea on how to automatically rebuild containers to test changes - customize to your liking. 
+
+## Working with Elasticsearch
+Additionally to the required services as listed in the example docker-compose file, the **Dev Tools** of [Kibana](https://www.elastic.co/guide/en/kibana/current/docker.html) are invaluable for running and testing Elasticsearch queries.  
+
+If you want to run queries in on the Elasticsearch container directly from your host with for example `curl` or something like *postman*, you might want to **publish** the port 9200 instead of just **exposing** it.
+
+## Implementing a new feature
+
+Do you see anything on the roadmap that you would like to take a closer look at but you are not sure, what's the best way to tackle that? Or anything not on there yet you'd like to implement but are not sure how? Open up an issue and we try to find a solution together.
 
 ## Making changes
 

Dockerfile

@@ -1,16 +1,30 @@
 # build the tube archivist image from default python slim image
 
 FROM python:3.9.7-slim-bullseye
+ARG TARGETPLATFORM
 
 ENV PYTHONUNBUFFERED 1
 
 # install distro packages needed
 RUN apt-get clean && apt-get -y update && apt-get -y install --no-install-recommends \
     build-essential \
-    ffmpeg \
     nginx \
     curl && rm -rf /var/lib/apt/lists/*
 
+# get newest patched ffmpeg and ffprobe builds for amd64 fall back to repo ffmpeg for arm64
+RUN if [ "$TARGETPLATFORM" = "linux/amd64" ] ; then \
+    curl -s https://api.github.com/repos/yt-dlp/FFmpeg-Builds/releases/latest \
+        | grep browser_download_url \
+        | grep linux64-gpl-4.4.tar.xz \
+        | cut -d '"' -f 4 \
+        | xargs curl -L --output ffmpeg.tar.xz && \
+        tar -xf ffmpeg.tar.xz --strip-components=2 --no-anchored -C /usr/bin/ "ffmpeg" && \
+        tar -xf ffmpeg.tar.xz --strip-components=2 --no-anchored -C /usr/bin/ "ffprobe" && \
+        rm ffmpeg.tar.xz \
+    ; elif [ "$TARGETPLATFORM" = "linux/arm64" ] ; then \
+        apt-get -y update && apt-get -y install --no-install-recommends ffmpeg && rm -rf /var/lib/apt/lists/* \
+    ; fi
+
 # copy config files
 COPY nginx.conf /etc/nginx/conf.d/
 

README.md

@@ -2,6 +2,19 @@
 
 <center><h1>Your self hosted YouTube media server</h1></center>
 
+## Table of contents:
+* [Wiki](https://github.com/bbilly1/tubearchivist/wiki) for a detailed documentation
+* [Core functionality](#core-functionality)
+* [Screenshots](#screenshots)
+* [Problem Tube Archivist tries to solve](#problem-tube-archivist-tries-to-solve)
+* [Installing and updating](#installing-and-updating)
+* [Getting Started](#getting-started)
+* [Potential pitfalls](#potential-pitfalls)
+* [Roadmap](#roadmap)
+* [Known limitations](#known-limitations)
+* [Donate](#donate)
+
+------------------------
 
 ## Core functionality
 * Subscribe to your favorite YouTube channels
@@ -29,16 +42,18 @@
 ## Problem Tube Archivist tries to solve
 Once your YouTube video collection grows, it becomes hard to search and find a specific video. That's where Tube Archivist comes in: By indexing your video collection with metadata from YouTube, you can organize, search and enjoy your archived YouTube videos without hassle offline through a convenient web interface.
 
-## Installation
+## Installing and updating
 Take a look at the example `docker-compose.yml` file provided. Tube Archivist depends on three main components split up into separate docker containers:  
 
 ### Tube Archivist
 The main Python application that displays and serves your video collection, built with Django.
   - Serves the interface on port `8000`
-  - Needs a mandatory volume for the video archive at **/youtube**
+  - Needs a volume for the video archive at **/youtube**
-  - And another recommended volume to save the cache for thumbnails and artwork at **/cache**.
+  - And another volume to save application data at **/cache**.
   - The environment variables `ES_URL` and `REDIS_HOST` are needed to tell Tube Archivist where Elasticsearch and Redis respectively are located.
-  - The environment variables `HOST_UID` and `HOST_GID` allows Tube Archivist to `chown` the video files to the main host system user instead of the container user.
+  - The environment variables `HOST_UID` and `HOST_GID` allows Tube Archivist to `chown` the video files to the main host system user instead of the container user. Those two variables are optional, not setting them will disable that functionality. That might be needed if the underlying filesystem doesn't support `chown` like *NFS*. 
+  - Change the environment variables `TA_USERNAME` and `TA_PASSWORD` to create the initial credentials. 
+  - `ELASTIC_PASSWORD` is for the password for Elasticsearch. The environment variable `ELASTIC_USER` is optional, should you want to change the username from the default *elastic*.
 
 ### Elasticsearch
 Stores video meta data and makes everything searchable. Also keeps track of the download queue.
@@ -50,42 +65,24 @@ Follow the [documentation](https://www.elastic.co/guide/en/elasticsearch/referen
 ### Redis JSON
 Functions as a cache and temporary link between the application and the file system. Used to store and display messages and configuration variables.
   - Needs to be accessible over the default port `6379`
-  - Takes an optional volume at **/data** to make your configuration changes permanent.
+  - Needs a volume at **/data** to make your configuration changes permanent.
 
-## Getting Started
+### Redis on a custom port
-1. Go through the **settings** page and look at the available options. Particularly set *Download Format* to your desired video quality before downloading. **Tube Archivist** downloads the best available quality by default.
+For some architectures it might be required to run Redis JSON on a nonstandard port. To for example change the Redis port to **6380**, set the following values:
-2. Subscribe to some of your favorite YouTube channels on the **channels** page. 
+- Set the environment variable `REDIS_PORT=6380` to the *tubearchivist* service.
-3. On the **downloads** page, click on *Rescan subscriptions* to add videos from the subscribed channels to your Download queue or click on *Add to download queue* to manually add Video IDs, links, channels or playlists.
+- For the *archivist-redis* service, change the ports to `6380:6380`
-4. Click on *Download queue* and let Tube Archivist to it's thing. 
+- Additionally set the following value to the *archivist-redis* service: `command: --port 6380 --loadmodule /usr/lib/redis/modules/rejson.so`
-5. Enjoy your archived collection!
-  
-## Import your existing library
-So far this depends on the video you are trying to import to be still available on YouTube to get the metadata. Add the files you like to import to the */cache/import* folder. Then start the process from the settings page *Manual media files import*. Make sure to follow one of the two methods below.
 
-### Method 1:
+### Updating Tube Archivist
-Add a matching *.json* file with the media file. Both files need to have the same base name, for example:
+You will see the current version number of **Tube Archivist** in the footer of the interface so you can compare it with the latest release to make sure you are running the *latest and greatest*.  
-- For the media file: \<base-name>.mp4
+* There can be breaking changes between updates, particularly as the application grows, new environment variables or settings might be required for you to set in the your docker-compose file. *Always* check the **release notes**: Any breaking changes will be marked there.  
-- For the JSON file: \<base-name>.info.json
+* All testing and development is done with the Elasticsearch version number as mentioned in the provided *docker-compose.yml* file. This will be updated when a new release of Elasticsearch is available. Running an older version of Elasticsearch is most likely not going to result in any issues, but it's still recommended to run the same version as mentioned.
-- Alternate JSON file: \<base-name>.json
 
-**Tube Archivist** then looks for the 'id' key within the JSON file to identify the video.
+### Alternative installation instructions:
+- **arm64**: The Tube Archivist container is multi arch, so is Elasticsearch. RedisJSON doesn't offer arm builds, you can use `bbilly1/rejson`, an unofficial rebuild for arm64.
+- **Synology**: There is a [discussion thread](https://github.com/bbilly1/tubearchivist/discussions/48) with Synology installation instructions.
+- **Unraid**: The three containers needed are all in the Community Applications. First install `TubeArchivist RedisJSON` followed by `TubeArchivist ES`, and finally you can install `TubeArchivist`. If you have unraid specific issues, report those to the [support thread](https://forums.unraid.net/topic/114073-support-crocs-tube-archivist/ "support thread").
 
-### Method 2:
-Detect the YouTube ID from filename, this accepts the default yt-dlp naming convention for file names like:
-- \<base-name>[\<youtube-id>].mp4
-- The YouTube ID in square brackets at the end of the filename is the crucial part.
-
-### Some notes:
-- This will **consume** the files you put into the import folder: Files will get converted to mp4 if needed (this might take a long time...) and moved to the archive, *.json* files will get deleted upon completion to avoid having duplicates on the next run.
-- Maybe start with a subset of your files to import to make sure everything goes well...
-- Follow the logs to monitor progress and errors: `docker-compose logs -f tubearchivist`.
-
-## Backup and restore
-From the settings page you can backup your metadata into a zip file. The file will get stored at *cache/backup* and will contain the necessary files to restore the Elasticsearch index formatted **nd-json** files as well a complete export of the index in a set of conventional **json** files.  
-
-The restore functionality will expect the same zip file in *cache/backup* and will recreate the index from the snapshot.  
-
-BE AWARE: This will **replace** your current index with the one from the backup file.
 
 ## Potential pitfalls
 ### vm.max_map_count
@@ -109,30 +106,55 @@ chown 1000:0 /path/to/mount/point
 ```
 This will match the permissions with the **UID** and **GID** of elasticsearch within the container and should fix the issue.
 
+### Disk usage
+The Elasticsearch index will turn to *read only* if the disk usage of the container goes above 95% until the usage drops below 90% again. Similar to that, TubeArchivist will become all sorts of messed up when running out of disk space. There are some error messages in the logs when that happens, but it's best to make sure to have enough disk space before starting to download.
+
+## Getting Started
+1. Go through the **settings** page and look at the available options. Particularly set *Download Format* to your desired video quality before downloading. **Tube Archivist** downloads the best available quality by default. To support iOS or MacOS and some other browsers a compatible format must be specified. For example:
+```
+bestvideo[VCODEC=avc1]+bestaudio[ACODEC=mp4a]/mp4
+```
+2. Subscribe to some of your favorite YouTube channels on the **channels** page. 
+3. On the **downloads** page, click on *Rescan subscriptions* to add videos from the subscribed channels to your Download queue or click on *Add to download queue* to manually add Video IDs, links, channels or playlists.
+4. Click on *Start download* and let **Tube Archivist** to it's thing. 
+5. Enjoy your archived collection!
+  
 ## Roadmap
 This should be considered as a **minimal viable product**, there is an extensive list of future functions and improvements planned.
 
 ### Functionality
-- [ ] Access control
 - [ ] User roles
-- [ ] Delete videos and channel
 - [ ] Create playlists
 - [ ] Podcast mode to serve channel as mp3
 - [ ] Implement [PyFilesystem](https://github.com/PyFilesystem/pyfilesystem2) for flexible video storage
-- [ ] Dynamic download queue
+- [ ] Optional automatic deletion of watched items after a specified time
-- [ ] Un-ignore videos
+- [ ] Subtitle download & indexing
+- [X] Access control [2021-11-01]
+- [X] Delete videos and channel [2021-10-16]
+- [X] Add thumbnail embed option [2021-10-16]
+- [X] Un-ignore videos [2021-10-03]
+- [X] Dynamic download queue [2021-09-26]
 - [X] Backup and restore [2021-09-22]
 - [X] Scan your file system to index already downloaded videos [2021-09-14]
 
 ### UI
-- [ ] Create a github wiki for user documentation
 - [ ] Show similar videos on video page
 - [ ] Multi language support
-- [ ] Grid and list view for both channel and video list pages
 - [ ] Show total video downloaded vs total videos available in channel
+- [X] Grid and list view for both channel and video list pages [2021-10-03]
+- [X] Create a github wiki for user documentation [2021-10-03]
 
 
 ## Known limitations
 - Video files created by Tube Archivist need to be **mp4** video files for best browser compatibility.
 - Every limitation of **yt-dlp** will also be present in Tube Archivist. If **yt-dlp** can't download or extract a video for any reason, Tube Archivist won't be able to either.
-- For now this is meant to be run in a trusted network environment.
+- For now this is meant to be run in a trusted network environment. Not everything is properly authenticated.
+- There is currently no flexibility in naming of the media files.
+
+
+## Donate
+The best donation to **Tube Archivist** is your time, take a look at the [contribution page](CONTRIBUTING.md) to get started.  
+Second best way to support the development is to provide for caffeinated beverages:
+* [Paypal.me](https://paypal.me/bbilly1) for a one time coffee
+* [Paypal Subscription](https://www.paypal.com/webapps/billing/plans/subscribe?plan_id=P-03770005GR991451KMFGVPMQ) for a monthly coffee
+* [co-fi.com](https://ko-fi.com/bbilly1) for an alternative platform

deploy.sh

@@ -5,11 +5,19 @@
 # blackhole for local production
 # docker to publish
 
+# create builder:
+# docker buildx create --name tubearchivist
+# docker buildx use tubearchivist
+# docker buildx inspect --bootstrap
+
+# more details:
+# https://github.com/bbilly1/tubearchivist/issues/6
+
 set -e
 
 function sync_blackhole {
 
-    # docker commands need sudo
+    # docker commands need sudo, only build amd64
     host="blackhole.local"
 
     read -sp 'Password: ' remote_pw
@@ -23,7 +31,7 @@ function sync_blackhole {
         --exclude "db.sqlite3" \
         . -e ssh "$host":tubearchivist
 
-    echo "$PASS" | ssh "$host" 'sudo -S docker build -t bbilly1/tubearchivist:latest tubearchivist 2>/dev/null'
+    echo "$PASS" | ssh "$host" 'sudo -S docker buildx build --platform linux/amd64 -t bbilly1/tubearchivist:latest tubearchivist --load 2>/dev/null'
     echo "$PASS" | ssh "$host" 'sudo -S docker-compose up -d 2>/dev/null'
 
 }
@@ -31,6 +39,8 @@ function sync_blackhole {
 function sync_test {
 
     # docker commands don't need sudo in testing vm
+    # pass argument to build for specific platform
+
     host="tubearchivist.local"
 
     rsync -a --progress --delete-after \
@@ -43,7 +53,17 @@ function sync_test {
 
     rsync -r --progress --delete docker-compose.yml -e ssh "$host":docker
 
-    ssh "$host" 'docker build -t bbilly1/tubearchivist:latest tubearchivist'
+    if [[ $1 = "amd64" ]]; then
+        platform="linux/amd64"
+    elif [[ $1 = "arm64" ]]; then
+        platform="linux/arm64"
+    elif [[ $1 = "multi" ]]; then
+        platform="linux/amd64,linux/arm64"
+    else
+        platform="linux/amd64"
+    fi
+
+    ssh "$host" "docker buildx build --platform $platform -t bbilly1/tubearchivist:latest tubearchivist --load"
     ssh "$host" 'docker-compose -f docker/docker-compose.yml up -d'
 
     ssh "$host" 'docker cp tubearchivist/tubearchivist/testing.sh tubearchivist:/app/testing.sh'
@@ -102,19 +122,14 @@ function sync_docker {
     printf "\ncreate new version:\n"
     read -r VERSION
 
+    echo "build and push $VERSION?"
+    read -rn 1
+
     # start build
-    sudo docker build -t bbilly1/tubearchivist:latest -t bbilly1/tubearchivist:"$VERSION" .
+    sudo docker buildx build \
-
+        --platform linux/amd64,linux/arm64 \
-    printf "\nlatest images:\n"
+        -t bbilly1/tubearchivist:latest \
-    sudo docker image ls bbilly1/tubearchivist
+        -t bbilly1/tubearchivist:"$VERSION" --push .
-
-    read -s "Push?"
-
-    # push to docker
-    echo "pushing latest:"
-    sudo docker push bbilly1/tubearchivist:latest
-    echo "pushing $VERSION"
-    sudo docker push bbilly1/tubearchivist:"$VERSION"
 
     # create release tag
     echo "commits since last version:"
@@ -132,7 +147,7 @@ python version_check.py
 if [[ $1 == "blackhole" ]]; then
     sync_blackhole
 elif [[ $1 == "test" ]]; then
-    sync_test
+    sync_test "$2"
 elif [[ $1 == "validate" ]]; then
     validate "$2"
 elif [[ $1 == "docker" ]]; then

docker-compose.yml

@@ -15,6 +15,9 @@ services:
       - REDIS_HOST=archivist-redis
       - HOST_UID=1000
       - HOST_GID=1000
+      - TA_USERNAME=tubearchivist
+      - TA_PASSWORD=verysecret
+      - ELASTIC_PASSWORD=verysecret
     depends_on:
       - archivist-es
       - archivist-redis
@@ -22,17 +25,19 @@ services:
     image: redislabs/rejson:latest
     container_name: archivist-redis
     restart: always
-    ports:
+    expose:
-      - 6379:6379
+      - "6379"
     volumes:
       - ./volumes/tubearchivist/redis:/data
     depends_on:
       - archivist-es
   archivist-es:
-    image: docker.elastic.co/elasticsearch/elasticsearch:7.14.1
+    image: docker.elastic.co/elasticsearch/elasticsearch:7.15.2
     container_name: archivist-es
     restart: always
     environment:
+      - "xpack.security.enabled=true"
+      - "ELASTIC_PASSWORD=verysecret"
       - "discovery.type=single-node"
       - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
     ulimits:
@@ -41,5 +46,5 @@ services:
         hard: -1
     volumes:
       - ./volumes/tubearchivist/es:/usr/share/elasticsearch/data
-    ports:
+    expose:
-      - 9200:9200
+      - "9200"

docs/Channels.md

@@ -0,0 +1,26 @@
+# Channels Overview and Channel Detail Page
+
+The channels are organized on two different levels, similar as the playlists:
+
+## Channels Overview
+Accessible at `/channel/` of your Tube Archivist, the **Overview Page** shows a list of all channels you have indexed. 
+- You can filter that list to show or hide subscribed channels with the toggle. Clicking on the channel banner or the channel name will direct you to the *Channel Detail Page*.
+- If you are subscribed to a channel a *Unsubscribe* button will show, if you aren't subscribed, a *Subscribe* button will show instead. 
+
+The **Subscribe to Channels** button <img src="assets/icon-add.png?raw=true" alt="add icon" width="20px" style="margin:0 5px;"> opens a text field to subscribe to a channel. You have a few options:
+- Enter the YouTube channel ID, a 25 character alphanumeric string. For example *UCBa659QWEk1AI4Tg--mrJ2A*
+- Enter the URL to the channel page on YouTube. For example *https://www.youtube.com/channel/UCBa659QWEk1AI4Tg--mrJ2A*
+- Enter the channel name for example: *https://www.youtube.com/c/TomScottGo*.
+- Enter the video URL for any video and let Tube Archivist extract the channel ID for you. For example *https://www.youtube.com/watch?v=2tdiKTSdE9Y*
+- Add one per line.
+
+The search icon <img src="assets/icon-search.png?raw=true" alt="search icon" width="20px" style="margin:0 5px;"> opens a text box to search for indexed channel names. Possible matches will show as you type. 
+
+## Channel Detail
+Each channel will get a dedicated channel detail page accessible at `/channel/<channel-id>/` of your Tube Archivist. This page shows all the videos you have downloaded from this channel plus additional metadata. 
+- If you are subscribed to the channel, an *Unsubscribe* button will show, else the *Subscribe* button will show.
+- You can *Show* the channel description, that matches with the *About* tab on YouTube.
+- The **Mark as Watched** button will mark all videos of this channel as watched.
+- The button **Delete Channel** will delete the channel plus all videos of this channel, both media files and metadata.
+- The button **Find Playlists** will go out to youtube and look for playlists belonging to this channel and index all with at least one video downloaded. Only do this for channels where you care about playlists as this will slow down indexing new videos for having to check which playlist this belongs to.
+- The button **Show Playlists** will go to the [playlists](Playlists) page and filter the list to only show playlists from this channel.

docs/Downloads.md

@@ -0,0 +1,41 @@
+# Downloads Page
+Accessible at `/downloads/` of your Tube Archivist, this page handles all the download functionality.
+
+
+## Rescan Subscriptions
+The **Rescan Subscriptions** icon <img src="assets/icon-rescan.png?raw=true" alt="rescan icon" width="20px" style="margin:0 5px;"> will start a background task to look for new videos from the channels and playlists you are subscribed to. You can define the channel and playlist page size on the [settings page](Settings#subscriptions). With the default page size, expect this process to take around 2-3 seconds for each channel or playlist you are subscribed to. A status message will show the progress.
+
+Then for every video found, **Tube Archivist** will skip the video if it has already been downloaded or if you added it to the *ignored* list before. All the other videos will get added to the download queue. Expect this to take around 1 second for each video as **Tube Archivist** needs to grab some additional metadata. New videos will get added at the bottom of the download queue.
+
+## Download Queue
+The **Start Download** icon <img src="assets/icon-download.png?raw=true" alt="download icon" width="20px" style="margin:0 5px;"> will start the download process starting from the top of the queue. Take a look at the relevant settings on the [Settings Page](Settings#downloads). Once the process started, a progress message will show with additional details and controls: 
+- The stop icon <img src="assets/icon-stop.png?raw=true" alt="stop icon" width="20px" style="margin:0 5px;"> will gracefully stop the download process, once the current video has been finished successfully.
+- The cancel icon <img src="assets/icon-close-red.png?raw=true" alt="close icon" width="20px" style="margin:0 5px;"> is equivalent to killing the process and will stop the download immediately. Any leftover files will get deleted, the canceled video will still be available in the download queue.
+
+After downloading, Tube Archivist tries to add new videos to already indexed playlists.
+
+## Add to Download Queue
+The **Add to Download Queue** icon <img src="assets/icon-add.png?raw=true" alt="add icon" width="20px" style="margin:0 5px;"> opens a text field to manually add videos to the download queue. You have a few options:
+- Add a link to a YouTube video. For example *https://www.youtube.com/watch?v=2tdiKTSdE9Y*.
+- Add a YouTube video ID. For example *2tdiKTSdE9Y*.
+- Add a link to a YouTube video by providing the shortened URL, for example *https://youtu.be/2tdiKTSdE9Y*.
+- Add a Channel ID or Channel URL to add every available video to the download queue. This will ignore the channel page size as described before and is meant for an initial download of the whole channel. You can still ignore selected videos before starting the download.
+- Add a channel name like for example *https://www.youtube.com/c/TomScottGo*.
+- Add a playlist ID or URL to add every available video in the list to the download queue, for example *https://www.youtube.com/playlist?list=PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha* or *PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha*. Note that when you add a link to a video in a playlist, Tube Archivist assumes you want to download only the specific video and not the whole playlist, for example *https://www.youtube.com/watch?v=CINVwWHlzTY&list=PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha* will only add one video *CINVwWHlzTY* to the queue.
+- Add one link per line.
+
+## The Download Queue
+Below the three buttons you find the download queue. New items will get added at the bottom of the queue, the next video to download once you click on **Start Download** will be the first in the list.
+
+Every video in the download queue has two buttons:
+- **Ignore**: This will remove that video from the download queue and this video will not get added again, even when you **Rescan Subscriptions**.
+- **Download now**: This will give priority to this video. If the download process is already running, the prioritized video will get downloaded as soon as the current video is finished. If there is no download process running, this will start downloading this single video and stop after that.  
+
+The button **Delete all queued** will delete all pending videos from the download queue.
+
+You can flip the view by activating **Show Only Ignored Videos**. This will show all videos you have previously *ignored*.  
+Every video in the ignored list has two buttons:
+- **Forget**: This will delete the item form the ignored list.
+- **Add to Queue**: This will add the ignored video back to the download queue.  
+
+The button **Delete all ignored** will delete all videos you have previously ignored.

docs/Home.md

@@ -0,0 +1,29 @@
+# Tube Archivist Wiki
+Welcome to the official Tube Archivist Wiki. This is an up-to-date documentation of user functionality.
+
+Table of contents:
+* [Main](Main): Tube Archivist landing page
+* [Channels](Channels): Browse your channels, handle channel subscriptions
+* [Playlists](Playlists): Browse your indexed playlists, handle playlist subscriptions
+* [Downloads](Downloads): Scanning subscriptions, handle download queue
+* [Settings](Settings): All the configuration options
+* [Video](Video): All details of a single video and playlist navigation.
+* [Users](Users): User management admin interface
+
+## Getting Started
+1. [Subscribe](Channels#channels-overview) to some of your favourite YouTube channels.
+2. [Scan](Downloads#rescan-subscriptions) subscriptions to add the latest videos to the download queue.
+3. [Add](Downloads#add-to-download-queue) additional videos, channels or playlist - ignore the ones you don't want to download.
+4. [Download](Downloads#download-queue) and let **Tube Archivist** do it's thing.
+5. Sit back and enjoy your archived and indexed collection!
+
+## General Navigation
+* Clicking on the channel name or the channel icon brings you to the dedicated channel page to show videos from that channel.
+* Clicking on a video title brings you to the dedicated video page and shows additional details.
+* Clicking on a video thumbnail opens the video player and starts streaming the selected video.
+* Hover over the playing video to show additional control options.
+
+
+An empty checkbox icon <img src="assets/icon-unseen.png?raw=true" alt="unseen icon" width="20px" style="margin:0 5px;"> will show for videos you haven't marked as watched. Click on it and the icon will change to a filled checkbox <img src="assets/icon-seen.png?raw=true" alt="seen icon" width="20px" style="margin:0 5px;"> indicating it as watched - click again to revert.
+
+When available the <img src="assets/icon-gridview.png?raw=true" alt="gridview icon" width="20px" style="margin:0 5px;"> gridview icon will display the list in a grid, the <img src="assets/icon-listview.png?raw=true" alt="listview icon" width="20px" style="margin:0 5px;"> listview icon will arrange the items in a list.

docs/Main.md

@@ -0,0 +1,10 @@
+# Tube Archivist Home Page Functionality
+
+This is the landing page, when you first open **Tube Archivist**. You have a few options to sort and filter that view:
+- With the **Sort Order** you can select how the "Recent Videos" are sorted: 
+  - **Date Published**: Sorts the list by date when the video was published on YouTube, newest on top.
+  - **Date Downloaded**: Sorts the list based on when you have downloaded the video to your archive, newest on top.
+- With **Hide Watched** you can filter out videos you have already marked as watched to only show unwatched videos. 
+- You can use those two options together to for example filter the list to *Hide Watched* videos **and** sort by date downloaded.
+
+Additionally the search icon <img src="assets/icon-search.png?raw=true" alt="search icon" width="20px" style="margin:0 5px;"> opens a text field to search your collection.

docs/Playlists.md

@@ -0,0 +1,23 @@
+# Playlist Overview and Playlist Detail Page
+The playlists are organized in two different levels, similar as the channels:
+
+## Playlist Overview
+Accessible at `/playlist/` of your Tube Archivist, this **Overview Page** shows a list of all playlists you have indexed over all your channels.
+- You can filter that list to show only subscribed to playlists with the toggle.
+
+You can index playlists of a channel from the channel detail page as described [here](Channels#channel-detail).
+
+The **Subscribe to Playlist** button <img src="assets/icon-add.png?raw=true" alt="add icon" width="20px" style="margin:0 5px;"> opens a text field to subscribe to playlists. You have a few options:
+- Enter the YouTube playlist id, for example: *PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha*
+- Enter the Youtube dedicated playlist url, for example: *https://www.youtube.com/playlist?list=PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha*
+- Add one per line.
+- NOTE: It doesn't make sense to subscribe to a playlist if you are already subscribed the corresponding channel as this will slow down the **Rescan Subscriptions** [task](Downloads#rescan-subscriptions).
+
+The search icon <img src="assets/icon-search.png?raw=true" alt="search icon" width="20px" style="margin:0 5px;"> opens a text box to search for indexed playlists.
+
+## Playlist Detail
+Each playlist will get a dedicated playlist detail page accessible at `/playlist/<playlist-id>/` of your Tube Archivist. This page shows all the videos you have downloaded from this playlist.
+
+- If you are subscribed to the playlist, an Unsubscribe button will show, else the Subscribe button will show.
+- The Mark as Watched button will mark all videos of this playlist as watched.
+- The **Delete Playlist** button will give you the option to delete just the *metadata* which won't delete any media files or *delete all* which will delete metadata plus all videos belonging to this playlist.

docs/Settings.md

@@ -0,0 +1,73 @@
+# Settings Page
+Accessible at `/settings/` of your **Tube Archivist**, this page holds all the configurations and additional functionality related to the database.
+
+Click on **Update Settings** at the bottom of the form to apply your configurations.
+
+## Color scheme
+Switch between the easy on the eyes dark theme and the burning bright theme.
+
+## Archive View
+- **Page Size**: Defines how many results get displayed on a given page. Same value goes for all archive views.
+
+## Subscriptions
+Settings related to the channel management.
+- **Channel Page Size**: Defines how many pages will get analyzed by **Tube Archivist** each time you click on *Rescan Subscriptions*. The default page size used by yt-dlp is **50**, that's also the recommended value to set here. Any value higher will slow down the rescan process, for example if you set the value to 51, that means yt-dlp will have to go through 2 pages of results instead of 1 and by that doubling the time that process takes.
+
+## Downloads
+Settings related to the download process.
+- **Download Limit**: Stop the download process after downloading the set quantity of videos.
+- **Download Speed Limit**: Set your download speed limit in KB/s. This will pass the option `--limit-rate` to yt-dlp.
+- **Throttled Rate Limit**: Restart download if the download speed drops below this value in KB/s. This will pass the option `--throttled-rate` to yt-dlp. Using this option might have a negative effect if you have an unstable or slow internet connection.
+- **Sleep Interval**: Time in seconds to sleep between requests to YouTube. It's a good idea to set this to **3** seconds. Might be necessary to avoid throttling.
+
+## Download Format
+Additional settings passed to yt-dlp.
+- **Format**: This controls which streams get downloaded and is equivalent to passing `--format` to yt-dlp. Use one of the recommended one or look at the documentation of [yt-dlp](https://github.com/yt-dlp/yt-dlp#format-selection). Please note: The option `--merge-output-format mp4` is automatically passed to yt-dlp to guarantee browser compatibility. Similar to that, `--check-formats` is passed as well to check that the selected formats are actually downloadable.
+- **Embed Metadata**: This saves the available tags directly into the media file by passing `--embed-metadata` to yt-dlp.
+- **Embed Thumbnail**: This will save the thumbnail into the media file by passing `--embed-thumbnail` to yt-dlp.
+
+
+# Actions
+Additional database functionality.
+
+## Manual Media Files Import
+So far this depends on the video you are trying to import to be still available on YouTube to get the metadata. Add the files you like to import to the */cache/import* folder. Then start the process from the settings page *Manual Media Files Import*. Make sure to follow one of the two methods below.
+
+### Method 1:
+Add a matching *.json* file with the media file. Both files need to have the same base name, for example:
+- For the media file: \<base-name>.mp4
+- For the JSON file: \<base-name>.info.json
+- Alternate JSON file: \<base-name>.json
+
+**Tube Archivist** then looks for the 'id' key within the JSON file to identify the video.
+
+### Method 2:
+Detect the YouTube ID from filename, this accepts the default yt-dlp naming convention for file names like:
+- \<base-name>[\<youtube-id>].mp4
+- The YouTube ID in square brackets at the end of the filename is the crucial part.
+
+### Some notes:
+- This will **consume** the files you put into the import folder: Files will get converted to mp4 if needed (this might take a long time...) and moved to the archive, *.json* files will get deleted upon completion to avoid having duplicates on the next run.
+- Maybe start with a subset of your files to import to make sure everything goes well...
+- Follow the logs to monitor progress and errors: `docker-compose logs -f tubearchivist`.
+
+## Embed thumbnails into media file
+This will write or overwrite all thumbnails in the media file using the downloaded thumbnail. This is only necessary if you didn't download the files with the option *Embed Thumbnail* enabled or want to make sure all media files get the newest thumbnail. Follow the docker-compose logs to monitor progress.
+
+## Backup Database
+This will backup your metadata into a zip file. The file will get stored at *cache/backup* and will contain the necessary files to restore the Elasticsearch index formatted **nd-json** files plus a complete export of the index in a set of conventional **json** files.  
+
+BE AWARE: This will **not** backup any media files, just the metadata from the Elasticsearch.
+
+## Restore From Backup
+The restore functionality will expect the same zip file in *cache/backup* as created from the **Backup database** function. This will recreate the index from the snapshot. If there are multiple backup files in the folder, the newest one will take priority. 
+
+BE AWARE: This will **replace** your current index with the one from the backup file. This won't restore any media files.
+
+## Rescan Filesystem
+This function will go through all your media files and looks at the whole index to try to find any issues:
+- Should the filename not match with the indexed media url, this will rename the video files correctly and update the index with the new link.
+- When you delete media files from the filesystem outside of the Tube Archivist interface, this will delete leftover metadata from the index.
+- When you have media files that are not indexed yet, this will grab the metadata from YouTube like it was a newly downloaded video. This can be useful when restoring from an older backup file with missing metadata but already downloaded mediafiles. NOTE: This only works if the media files are named in the same convention as Tube Archivist does, particularly the YouTube ID needs to be at the same index in the filename, alternatively see above for *Manual Media Files Import*.
+
+BE AWARE: There is no undo.

docs/Users.md

@@ -0,0 +1,20 @@
+# User Management
+
+For now, **Tube Archivist** is a single user application. You can create multiple users with different names and passwords, they will share the same videos and permissions but some interface configurations are on a per user basis. *More is on the roadmap*.
+
+## Superuser
+The first user gets created with the environment variables **TA_USERNAME** and **TA_PASSWORD** from your docker-compose file. That first user will automatically have *superuser* privileges. 
+
+## Admin Interface
+When logged in from your *superuser* account, you are able to access the admin interface from the settings page or at `/admin/`. This interface holds all functionality for user management.
+
+## Create additional users
+From the admin interface when you click on *Accounts* you will get a list of all users. From there you can create additional users by clicking on *Add Account*, provide a name and confirm password and click on *Save* to create the user.
+
+## Changing users
+You can delete or change permissions and password of a user by clicking on the username from the *Accounts* list page and follow the interface from there. Changing the password of the *superuser* here will overwrite the password originally set with the environment variables.
+
+## Reset
+Delete all user configurations by deleting the file `cache/db.sqlite3` and restart the container. This will create the superuser again from the environment variables.
+
+NOTE: Future improvements here will most likely require such a reset.

docs/Video.md

@@ -0,0 +1,11 @@
+# Video Page
+
+Every video downloaded gets a dedicated page accessible at `/video/<video-id>` of your Tube Archivist.
+
+Clicking on the channel name or the channel icon will bring you to the dedicated channel detail [page](Channels#channel-detail).
+
+The button **Delete Video** will delete that video including the media file.
+
+When available, a playlist navigation will show at the bottom. Clicking on the playlist name will bring you to the dedicated [Playlist Detail](Playlists#playlist-detail) page showing all videos downloaded from that playlist. The number in square brackets indicates the position of the current video in that playlist.
+
+Clicking on the next or previous video name or thumbnail will bring you to that dedicated video page.

run.sh

@@ -1,19 +1,38 @@
 #!/bin/bash
 # startup script inside the container for tubearchivist
 
+# check environment
+if [[ -z "$ELASTIC_USER" ]]; then
+    export ELASTIC_USER=elastic
+fi
+
+ENV_VARS=("TA_USERNAME" "TA_PASSWORD" "ELASTIC_PASSWORD" "ELASTIC_USER")
+for each in "${ENV_VARS[@]}"; do
+    if ! [[ -v $each ]]; then
+        echo "missing environment variable $each"
+        exit 1
+    fi
+done
+
+# wait for elasticsearch
 counter=0
-until curl "$ES_URL" -fs; do
+until curl -u "$ELASTIC_USER":"$ELASTIC_PASSWORD" "$ES_URL" -fs; do
     echo "waiting for elastic search to start"
     counter=$((counter+1))
     if [[ $counter -eq 12 ]]; then
-        # fail after 1 min
+        # fail after 2 min
         echo "failed to connect to elastic search, exiting..."
         exit 1
     fi
-    sleep 5
+    sleep 10
 done
 
+# start python application
+python manage.py makemigrations
 python manage.py migrate
+export DJANGO_SUPERUSER_PASSWORD=$TA_PASSWORD && \
+    python manage.py createsuperuser --noinput --name "$TA_USERNAME"
+
 python manage.py collectstatic --noinput -c
 nginx &
 celery -A home.tasks worker --loglevel=INFO &

tubearchivist/config/settings.py

@@ -10,9 +10,12 @@ For the full list of settings and their values, see
 https://docs.djangoproject.com/en/3.2/ref/settings/
 """
 
-from os import environ
+import hashlib
+from os import environ, path
 from pathlib import Path
 
+from home.src.config import AppConfig
+
 # Build paths inside the project like this: BASE_DIR / 'subdir'.
 BASE_DIR = Path(__file__).resolve().parent.parent
 
@@ -20,8 +23,8 @@ BASE_DIR = Path(__file__).resolve().parent.parent
 # Quick-start development settings - unsuitable for production
 # See https://docs.djangoproject.com/en/3.2/howto/deployment/checklist/
 
-# SECURITY WARNING: keep the secret key used in production secret!
+PW_HASH = hashlib.sha256(environ.get("TA_PASSWORD").encode())
-SECRET_KEY = "Fvid^aUL6LohRZz*kZFvq85B&JW&kB9o*#jdzWsdWE8*XkCLR8"
+SECRET_KEY = PW_HASH.hexdigest()
 
 # SECURITY WARNING: don't run with debug turned on in production!
 DEBUG = bool(environ.get("DJANGO_DEBUG"))
@@ -78,10 +81,12 @@ WSGI_APPLICATION = "config.wsgi.application"
 # Database
 # https://docs.djangoproject.com/en/3.2/ref/settings/#databases
 
+CACHE_DIR = AppConfig().config["application"]["cache_dir"]
+DB_PATH = path.join(CACHE_DIR, "db.sqlite3")
 DATABASES = {
     "default": {
         "ENGINE": "django.db.backends.sqlite3",
-        "NAME": BASE_DIR / "db.sqlite3",
+        "NAME": DB_PATH,
     }
 }
 
@@ -104,18 +109,16 @@ AUTH_PASSWORD_VALIDATORS = [
     },
 ]
 
+AUTH_USER_MODEL = "home.Account"
+
 
 # Internationalization
 # https://docs.djangoproject.com/en/3.2/topics/i18n/
 
 LANGUAGE_CODE = "en-us"
-
 TIME_ZONE = "UTC"
-
 USE_I18N = True
-
 USE_L10N = True
-
 USE_TZ = True
 
 
@@ -123,22 +126,14 @@ USE_TZ = True
 # https://docs.djangoproject.com/en/3.2/howto/static-files/
 
 STATIC_URL = "/static/"
-
-# STATICFILES_DIRS = [
-#     str(BASE_DIR.joinpath('static')),
-#     '/cache/'
-# ]
-
-# STATIC_URL = '/static/'
 STATICFILES_DIRS = (str(BASE_DIR.joinpath("static")),)
-# MEDIA_ROOT = str(BASE_DIR.joinpath('media'))
-# MEDIA_URL = '/media/'
-
 STATIC_ROOT = str(BASE_DIR.joinpath("staticfiles"))
-
 STATICFILES_STORAGE = "whitenoise.storage.CompressedManifestStaticFilesStorage"
 
 # Default primary key field type
 # https://docs.djangoproject.com/en/3.2/ref/settings/#default-auto-field
 
 DEFAULT_AUTO_FIELD = "django.db.models.BigAutoField"
+
+LOGIN_URL = "/login/"
+LOGOUT_REDIRECT_URL = "/login/"

tubearchivist/home/__init__.py

@@ -1,42 +1,5 @@
-""" handle startup """
+""" handle celery startup """
-
-import os
-
-from home.src.config import AppConfig
-from home.src.helper import set_message
-from home.src.index_management import index_check
 
 from .tasks import app as celery_app
 
-
-def sync_redis_state():
-    """make sure redis gets the config.json values"""
-    print("sync redis")
-    config_handler = AppConfig()
-    config_handler.load_new_defaults()
-    config = config_handler.config
-    sort_order = config["archive"]["sort"]
-    set_message("sort_order", sort_order, expire=False)
-    hide_watched = bool(int(config["archive"]["hide_watched"]))
-    set_message("hide_watched", hide_watched, expire=False)
-    show_subed_only = bool(int(config["archive"]["show_subed_only"]))
-    set_message("show_subed_only", show_subed_only, expire=False)
-
-
-def make_folders():
-    """make needed cache folders here so docker doesn't mess it up"""
-    folders = ["download", "channels", "videos", "import", "backup"]
-    config = AppConfig().config
-    cache_dir = config["application"]["cache_dir"]
-    for folder in folders:
-        folder_path = os.path.join(cache_dir, folder)
-        try:
-            os.makedirs(folder_path)
-        except FileExistsError:
-            continue
-
-
 __all__ = ("celery_app",)
-make_folders()
-sync_redis_state()
-index_check()

tubearchivist/home/admin.py

@@ -1,3 +1,36 @@
-from django.contrib import admin  # noqa: F401 - Unused import
+"""custom admin classes"""
 
-# Register your models here.
+from django.contrib import admin
+from django.contrib.auth.admin import UserAdmin as BaseUserAdmin
+
+from .models import Account
+
+
+class HomeAdmin(BaseUserAdmin):
+    """register in admin page"""
+
+    list_display = ("name", "is_staff", "is_superuser")
+    list_filter = ("is_superuser",)
+
+    fieldsets = (
+        (None, {"fields": ("is_staff", "is_superuser", "password")}),
+        ("Personal info", {"fields": ("name",)}),
+        ("Groups", {"fields": ("groups",)}),
+        ("Permissions", {"fields": ("user_permissions",)}),
+    )
+    add_fieldsets = (
+        (
+            None,
+            {"fields": ("is_staff", "is_superuser", "password1", "password2")},
+        ),
+        ("Personal info", {"fields": ("name",)}),
+        ("Groups", {"fields": ("groups",)}),
+        ("Permissions", {"fields": ("user_permissions",)}),
+    )
+
+    search_fields = ("name",)
+    ordering = ("name",)
+    filter_horizontal = ()
+
+
+admin.site.register(Account, HomeAdmin)

tubearchivist/home/apps.py

@@ -1,6 +1,59 @@
+"""handle custom startup functions"""
+
+import os
+
 from django.apps import AppConfig
+from home.src.config import AppConfig as ArchivistConfig
+from home.src.helper import RedisArchivist
+from home.src.index_management import index_check
+from home.src.thumbnails import validate_thumbnails
+
+
+def sync_redis_state():
+    """make sure redis gets new config.json values"""
+    print("sync redis")
+    config_handler = ArchivistConfig()
+    config_handler.load_new_defaults()
+
+
+def make_folders():
+    """make needed cache folders here so docker doesn't mess it up"""
+    folders = [
+        "download",
+        "channels",
+        "videos",
+        "playlists",
+        "import",
+        "backup",
+    ]
+    config = ArchivistConfig().config
+    cache_dir = config["application"]["cache_dir"]
+    for folder in folders:
+        folder_path = os.path.join(cache_dir, folder)
+        try:
+            os.makedirs(folder_path)
+        except FileExistsError:
+            continue
+
+
+def release_lock():
+    """make sure there are no leftover locks set in redis on container start"""
+    all_locks = ["manual_import", "downloading", "dl_queue", "dl_queue_id"]
+    for lock in all_locks:
+        response = RedisArchivist().del_message(lock)
+        if response:
+            print("deleted leftover key from redis: " + lock)
 
 
 class HomeConfig(AppConfig):
+    """call startup funcs"""
+
     default_auto_field = "django.db.models.BigAutoField"
     name = "home"
+
+    def ready(self):
+        release_lock()
+        index_check()
+        sync_redis_state()
+        make_folders()
+        validate_thumbnails()

tubearchivist/home/config.json

@@ -1,23 +1,30 @@
 {
     "archive": {
-        "sort": "published",
+        "sort_by": "published",
-        "hide_watched": false,
+        "sort_order": "desc",
-        "show_subed_only": false,
         "page_size": 12
     },
+    "default_view": {
+        "home": "grid",
+        "channel": "list",
+        "downloads": "list"
+    },
     "subscriptions": {
         "auto_search": false,
         "auto_download": false,
         "channel_size": 50
     },
     "downloads": {
-        "limit_count": 5,
+        "limit_count": false,
         "limit_speed": false,
         "sleep_interval": 3,
         "format": false,
-        "add_metadata": false
+        "add_metadata": false,
+        "add_thumbnail": false,
+        "throttledratelimit": false
     },
     "application": {
+        "app_root": "/app",
         "cache_dir": "/cache",
         "videos": "/youtube",
         "file_template": "%(id)s_%(title)s.mp4",

tubearchivist/home/forms.py

@@ -0,0 +1,137 @@
+"""functionality:
+- hold all form classes used in the views
+"""
+
+from django import forms
+from django.contrib.auth.forms import AuthenticationForm
+from django.forms.widgets import PasswordInput, TextInput
+
+
+class CustomAuthForm(AuthenticationForm):
+    """better styled login form"""
+
+    username = forms.CharField(
+        widget=TextInput(attrs={"placeholder": "Username"}), label=False
+    )
+    password = forms.CharField(
+        widget=PasswordInput(attrs={"placeholder": "Password"}), label=False
+    )
+
+
+class UserSettingsForm(forms.Form):
+    """user configurations values"""
+
+    CHOICES = [
+        ("", "-- change color scheme --"),
+        ("dark", "Dark"),
+        ("light", "Light"),
+    ]
+
+    colors = forms.ChoiceField(
+        widget=forms.Select, choices=CHOICES, required=False
+    )
+    page_size = forms.IntegerField(required=False)
+
+
+class ApplicationSettingsForm(forms.Form):
+    """handle all application settings"""
+
+    METADATA_CHOICES = [
+        ("", "-- change metadata embed --"),
+        ("0", "don't embed metadata"),
+        ("1", "embed metadata"),
+    ]
+
+    THUMBNAIL_CHOICES = [
+        ("", "-- change thumbnail embed --"),
+        ("0", "don't embed thumbnail"),
+        ("1", "embed thumbnail"),
+    ]
+
+    subscriptions_channel_size = forms.IntegerField(required=False)
+    downloads_limit_count = forms.IntegerField(required=False)
+    downloads_limit_speed = forms.IntegerField(required=False)
+    downloads_throttledratelimit = forms.IntegerField(required=False)
+    downloads_sleep_interval = forms.IntegerField(required=False)
+    downloads_format = forms.CharField(required=False)
+    downloads_add_metadata = forms.ChoiceField(
+        widget=forms.Select, choices=METADATA_CHOICES, required=False
+    )
+    downloads_add_thumbnail = forms.ChoiceField(
+        widget=forms.Select, choices=THUMBNAIL_CHOICES, required=False
+    )
+
+
+class VideoSearchForm(forms.Form):
+    """search videos form"""
+
+    searchInput = forms.CharField(
+        label="Search your videos",
+        widget=forms.TextInput(attrs={"autocomplete": "off"}),
+    )
+
+
+class ChannelSearchForm(forms.Form):
+    """search for channels"""
+
+    searchInput = forms.CharField(
+        label="",
+        widget=forms.TextInput(
+            attrs={
+                "oninput": "searchChannels(this.value)",
+                "autocomplete": "off",
+                "list": "resultBox",
+            }
+        ),
+    )
+
+
+class PlaylistSearchForm(forms.Form):
+    """search for playlists"""
+
+    searchInput = forms.CharField(
+        label="",
+        widget=forms.TextInput(attrs={"autocomplete": "off"}),
+    )
+
+
+class AddToQueueForm(forms.Form):
+    """text area form to add to downloads"""
+
+    vid_url = forms.CharField(
+        label=False,
+        widget=forms.Textarea(
+            attrs={
+                "rows": 4,
+                "placeholder": "Enter Video Urls or IDs here...",
+            }
+        ),
+    )
+
+
+class SubscribeToChannelForm(forms.Form):
+    """text area form to subscribe to multiple channels"""
+
+    subscribe = forms.CharField(
+        label=False,
+        widget=forms.Textarea(
+            attrs={
+                "rows": 3,
+                "placeholder": "Input channel ID, URL or Video of a channel",
+            }
+        ),
+    )
+
+
+class SubscribeToPlaylistForm(forms.Form):
+    """text area form to subscribe to multiple playlists"""
+
+    subscribe = forms.CharField(
+        label=False,
+        widget=forms.Textarea(
+            attrs={
+                "rows": 3,
+                "placeholder": "Input playlist IDs",
+            }
+        ),
+    )

tubearchivist/home/models.py

@@ -1,3 +1,53 @@
-from django.db import models  # noqa: F401 - Unused import
+"""custom models"""
+from django.contrib.auth.models import (
+    AbstractBaseUser,
+    BaseUserManager,
+    PermissionsMixin,
+)
+from django.db import models
 
-# Create your models here.
+
+class AccountManager(BaseUserManager):
+    """manage user creation methods"""
+
+    use_in_migrations = True
+
+    def _create_user(self, name, password, **extra_fields):
+        """create regular user private"""
+        values = [name, password]
+        field_value_map = dict(zip(self.model.REQUIRED_FIELDS, values))
+        for field_name, value in field_value_map.items():
+            if not value:
+                raise ValueError(f"The {field_name} value must be set")
+
+        user = self.model(name=name, **extra_fields)
+        user.set_password(password)
+        user.save(using=self._db)
+        return user
+
+    def create_user(self, name, password):
+        """create regular user public"""
+        return self._create_user(name, password)
+
+    def create_superuser(self, name, password, **extra_fields):
+        """create super user"""
+        extra_fields.setdefault("is_staff", True)
+        extra_fields.setdefault("is_superuser", True)
+
+        if extra_fields.get("is_staff") is not True:
+            raise ValueError("Superuser must have is_staff=True.")
+        if extra_fields.get("is_superuser") is not True:
+            raise ValueError("Superuser must have is_superuser=True.")
+
+        return self._create_user(name, password, **extra_fields)
+
+
+class Account(AbstractBaseUser, PermissionsMixin):
+    """handle account creation"""
+
+    name = models.CharField(max_length=150, unique=True)
+    is_staff = models.BooleanField(default=False)
+    objects = AccountManager()
+
+    USERNAME_FIELD = "name"
+    REQUIRED_FIELDS = ["password"]

tubearchivist/home/src/config.py

@@ -8,14 +8,16 @@ Functionality:
 import json
 import os
 
-from home.src.helper import get_message, set_message
+from home.src.helper import RedisArchivist
 
 
 class AppConfig:
     """handle user settings and application variables"""
 
-    def __init__(self):
+    def __init__(self, user_id=False):
+        self.user_id = user_id
         self.config = self.get_config()
+        self.colors = self.get_colors()
 
     def get_config(self):
         """get config from default file or redis if changed"""
@@ -23,6 +25,12 @@ class AppConfig:
         if not config:
             config = self.get_config_file()
 
+        if self.user_id:
+            key = f"{self.user_id}:page_size"
+            page_size = RedisArchivist().get_message(key)["status"]
+            if page_size:
+                config["archive"]["page_size"] = page_size
+
         config["application"].update(self.get_config_env())
         return config
 
@@ -39,11 +47,27 @@ class AppConfig:
     @staticmethod
     def get_config_env():
         """read environment application variables"""
+        host_uid_env = os.environ.get("HOST_UID")
+        if host_uid_env:
+            host_uid = int(host_uid_env)
+        else:
+            host_uid = False
+
+        host_gid_env = os.environ.get("HOST_GID")
+        if host_gid_env:
+            host_gid = int(host_gid_env)
+        else:
+            host_gid = False
+
+        es_pass = os.environ.get("ELASTIC_PASSWORD")
+        es_user = os.environ.get("ELASTIC_USER", default="elastic")
+
         application = {
             "REDIS_HOST": os.environ.get("REDIS_HOST"),
             "es_url": os.environ.get("ES_URL"),
-            "HOST_UID": int(os.environ.get("HOST_UID")),
+            "es_auth": (es_user, es_pass),
-            "HOST_GID": int(os.environ.get("HOST_GID")),
+            "HOST_UID": host_uid,
+            "HOST_GID": host_gid,
         }
 
         return application
@@ -51,7 +75,7 @@ class AppConfig:
     @staticmethod
     def get_config_redis():
         """read config json set from redis to overwrite defaults"""
-        config = get_message("config")
+        config = RedisArchivist().get_message("config")
         if not list(config.values())[0]:
             return False
 
@@ -70,10 +94,35 @@ class AppConfig:
                 elif to_write.isdigit():
                     to_write = int(to_write)
 
-                config_dict, config_value = key.split(".")
+                config_dict, config_value = key.split("_", maxsplit=1)
                 config[config_dict][config_value] = to_write
 
-        set_message("config", config, expire=False)
+        RedisArchivist().set_message("config", config, expire=False)
+
+    @staticmethod
+    def set_user_config(form_post, user_id):
+        """set values in redis for user settings"""
+        for key, value in form_post.items():
+            to_write = value[0]
+            if len(to_write):
+                if to_write.isdigit():
+                    to_write = int(to_write)
+                message = {"status": to_write}
+                redis_key = f"{user_id}:{key}"
+                RedisArchivist().set_message(redis_key, message, expire=False)
+
+    def get_colors(self):
+        """overwrite config if user has set custom values"""
+        colors = False
+        if self.user_id:
+            col_dict = RedisArchivist().get_message(f"{self.user_id}:colors")
+            colors = col_dict["status"]
+
+        if not colors:
+            colors = self.config["application"]["colors"]
+
+        self.config["application"]["colors"] = colors
+        return colors
 
     def load_new_defaults(self):
         """check config.json for missing defaults"""
@@ -100,4 +149,4 @@ class AppConfig:
                     needs_update = True
 
         if needs_update:
-            set_message("config", redis_config, expire=False)
+            RedisArchivist().set_message("config", redis_config, expire=False)

tubearchivist/home/src/download.py

@@ -14,8 +14,19 @@ from time import sleep
 import requests
 import yt_dlp as youtube_dl
 from home.src.config import AppConfig
-from home.src.helper import DurationConverter, clean_string, set_message
+from home.src.helper import (
-from home.src.index import YoutubeChannel, index_new_video
+    DurationConverter,
+    RedisArchivist,
+    RedisQueue,
+    clean_string,
+    ignore_filelist,
+)
+from home.src.index import (
+    IndexPaginate,
+    YoutubeChannel,
+    YoutubePlaylist,
+    index_new_video,
+)
 
 
 class PendingList:
@@ -23,8 +34,13 @@ class PendingList:
 
     CONFIG = AppConfig().config
     ES_URL = CONFIG["application"]["es_url"]
+    ES_AUTH = CONFIG["application"]["es_auth"]
     VIDEOS = CONFIG["application"]["videos"]
 
+    def __init__(self):
+        self.all_channel_ids = False
+        self.all_downloaded = False
+
     @staticmethod
     def parse_url_list(youtube_ids):
         """extract youtube ids from list"""
@@ -37,19 +53,21 @@ class PendingList:
                 "title": "Adding to download queue.",
                 "message": "Extracting lists",
             }
-            set_message("progress:download", mess_dict)
+            RedisArchivist().set_message("progress:download", mess_dict)
             # extract
             url = entry["url"]
             url_type = entry["type"]
             if url_type == "video":
                 missing_videos.append(url)
             elif url_type == "channel":
-                youtube_ids = ChannelSubscription().get_last_youtube_videos(
+                video_results = ChannelSubscription().get_last_youtube_videos(
                     url, limit=False
                 )
+                youtube_ids = [i[0] for i in video_results]
                 missing_videos = missing_videos + youtube_ids
             elif url_type == "playlist":
-                youtube_ids = playlist_extractor(url)
+                video_results = YoutubePlaylist(url).get_entries()
+                youtube_ids = [i["youtube_id"] for i in video_results]
                 missing_videos = missing_videos + youtube_ids
 
         return missing_videos
@@ -59,48 +77,62 @@ class PendingList:
         # check if channel is indexed
         channel_handler = ChannelSubscription()
         all_indexed = channel_handler.get_channels(subscribed_only=False)
-        all_channel_ids = [i["channel_id"] for i in all_indexed]
+        self.all_channel_ids = [i["channel_id"] for i in all_indexed]
         # check if already there
-        all_downloaded = self.get_all_downloaded()
+        self.all_downloaded = self.get_all_downloaded()
-        # loop
-        bulk_list = []
-        for video in missing_videos:
-            if isinstance(video, str):
-                youtube_id = video
-            elif isinstance(video, tuple):
-                youtube_id = video[0]
-            if youtube_id in all_downloaded:
-                # skip already downloaded
-                continue
-            video = self.get_youtube_details(youtube_id)
-            # skip on download error
-            if not video:
-                continue
 
-            if video["channel_id"] in all_channel_ids:
+        bulk_list, all_videos_added = self.build_bulk(missing_videos)
-                video["channel_indexed"] = True
-            else:
-                video["channel_indexed"] = False
-            video["status"] = "pending"
-            action = {"create": {"_id": youtube_id, "_index": "ta_download"}}
-            bulk_list.append(json.dumps(action))
-            bulk_list.append(json.dumps(video))
-            # notify
-            mess_dict = {
-                "status": "pending",
-                "level": "info",
-                "title": "Adding to download queue.",
-                "message": "Processing IDs...",
-            }
-            set_message("progress:download", mess_dict)
         # add last newline
         bulk_list.append("\n")
         query_str = "\n".join(bulk_list)
         headers = {"Content-type": "application/x-ndjson"}
         url = self.ES_URL + "/_bulk"
-        request = requests.post(url, data=query_str, headers=headers)
+        request = requests.post(
+            url, data=query_str, headers=headers, auth=self.ES_AUTH
+        )
         if not request.ok:
             print(request)
+            raise ValueError("failed to add video to download queue")
+
+        return all_videos_added
+
+    def build_bulk(self, missing_videos):
+        """build the bulk lists"""
+        bulk_list = []
+        all_videos_added = []
+        counter = 1
+        for youtube_id in missing_videos:
+            # check if already downloaded
+            if youtube_id in self.all_downloaded:
+                continue
+
+            video = self.get_youtube_details(youtube_id)
+            # skip on download error
+            if not video:
+                continue
+
+            channel_indexed = video["channel_id"] in self.all_channel_ids
+            video["channel_indexed"] = channel_indexed
+            thumb_url = video["vid_thumb_url"]
+            video["status"] = "pending"
+            action = {"create": {"_id": youtube_id, "_index": "ta_download"}}
+            bulk_list.append(json.dumps(action))
+            bulk_list.append(json.dumps(video))
+            all_videos_added.append((youtube_id, thumb_url))
+            # notify
+            progress = f"{counter}/{len(missing_videos)}"
+            mess_dict = {
+                "status": "pending",
+                "level": "info",
+                "title": "Adding new videos to download queue.",
+                "message": "Progress: " + progress,
+            }
+            RedisArchivist().set_message("progress:download", mess_dict)
+            if counter % 25 == 0:
+                print("adding to queue progress: " + progress)
+            counter = counter + 1
+
+        return bulk_list, all_videos_added
 
     @staticmethod
     def get_youtube_details(youtube_id):
@@ -108,16 +140,24 @@ class PendingList:
         obs = {
             "default_search": "ytsearch",
             "quiet": True,
-            "skip_download": True,
+            "check_formats": "selected",
+            "noplaylist": True,
+            "writethumbnail": True,
+            "simulate": True,
         }
         try:
             vid = youtube_dl.YoutubeDL(obs).extract_info(youtube_id)
         except youtube_dl.utils.DownloadError:
             print("failed to extract info for: " + youtube_id)
             return False
+        # stop if video is streaming live now
+        if vid["is_live"]:
+            return False
         # parse response
         seconds = vid["duration"]
         duration_str = DurationConverter.get_str(seconds)
+        if duration_str == "NA":
+            print(f"skip extracting duration for: {youtube_id}")
         upload_date = vid["upload_date"]
         upload_dt = datetime.strptime(upload_date, "%Y%m%d")
         published = upload_dt.strftime("%Y-%m-%d")
@@ -134,91 +174,47 @@ class PendingList:
         }
         return youtube_details
 
-    def get_all_pending(self):
+    @staticmethod
+    def get_all_pending():
         """get a list of all pending videos in ta_download"""
-        headers = {"Content-type": "application/json"}
-        # get PIT ID
-        url = self.ES_URL + "/ta_download/_pit?keep_alive=1m"
-        response = requests.post(url)
-        json_data = json.loads(response.text)
-        pit_id = json_data["id"]
-        # query
         data = {
-            "size": 50,
             "query": {"match_all": {}},
-            "pit": {"id": pit_id, "keep_alive": "1m"},
+            "sort": [{"timestamp": {"order": "asc"}}],
-            "sort": [{"timestamp": {"order": "desc"}}],
         }
-        query_str = json.dumps(data)
+        all_results = IndexPaginate("ta_download", data).get_results()
-        url = self.ES_URL + "/_search"
+
         all_pending = []
         all_ignore = []
-        while True:
+
-            response = requests.get(url, data=query_str, headers=headers)
+        for result in all_results:
-            json_data = json.loads(response.text)
+            if result["status"] == "pending":
-            all_hits = json_data["hits"]["hits"]
+                all_pending.append(result)
-            if all_hits:
+            elif result["status"] == "ignore":
-                for hit in all_hits:
+                all_ignore.append(result)
-                    youtube_id = hit["_source"]["youtube_id"]
+
-                    status = hit["_source"]["status"]
-                    if status == "pending":
-                        all_pending.append(hit["_source"])
-                    elif status == "ignore":
-                        all_ignore.append(youtube_id)
-                    search_after = hit["sort"]
-                # update search_after with last hit data
-                data["search_after"] = search_after
-                query_str = json.dumps(data)
-            else:
-                break
-        # clean up PIT
-        query_str = json.dumps({"id": pit_id})
-        requests.delete(self.ES_URL + "/_pit", data=query_str, headers=headers)
         return all_pending, all_ignore
 
-    def get_all_indexed(self):
+    @staticmethod
+    def get_all_indexed():
         """get a list of all videos indexed"""
-        headers = {"Content-type": "application/json"}
+
-        # get PIT ID
-        url = self.ES_URL + "/ta_video/_pit?keep_alive=1m"
-        response = requests.post(url)
-        json_data = json.loads(response.text)
-        pit_id = json_data["id"]
-        # query
         data = {
-            "size": 500,
             "query": {"match_all": {}},
-            "pit": {"id": pit_id, "keep_alive": "1m"},
             "sort": [{"published": {"order": "desc"}}],
         }
-        query_str = json.dumps(data)
+        all_indexed = IndexPaginate("ta_video", data).get_results()
-        url = self.ES_URL + "/_search"
+
-        all_indexed = []
-        while True:
-            response = requests.get(url, data=query_str, headers=headers)
-            json_data = json.loads(response.text)
-            all_hits = json_data["hits"]["hits"]
-            if all_hits:
-                for hit in all_hits:
-                    all_indexed.append(hit)
-                    search_after = hit["sort"]
-                # update search_after with last hit data
-                data["search_after"] = search_after
-                query_str = json.dumps(data)
-            else:
-                break
-        # clean up PIT
-        query_str = json.dumps({"id": pit_id})
-        requests.delete(self.ES_URL + "/_pit", data=query_str, headers=headers)
         return all_indexed
 
     def get_all_downloaded(self):
         """get a list of all videos in archive"""
-        all_channel_folders = os.listdir(self.VIDEOS)
+        channel_folders = os.listdir(self.VIDEOS)
+        all_channel_folders = ignore_filelist(channel_folders)
         all_downloaded = []
         for channel_folder in all_channel_folders:
             channel_path = os.path.join(self.VIDEOS, channel_folder)
-            all_videos = os.listdir(channel_path)
+            videos = os.listdir(channel_path)
+            all_videos = ignore_filelist(videos)
             youtube_vids = [i[9:20] for i in all_videos]
             for youtube_id in youtube_vids:
                 all_downloaded.append(youtube_id)
@@ -227,7 +223,19 @@ class PendingList:
     def delete_from_pending(self, youtube_id):
         """delete the youtube_id from ta_download"""
         url = f"{self.ES_URL}/ta_download/_doc/{youtube_id}"
-        response = requests.delete(url)
+        response = requests.delete(url, auth=self.ES_AUTH)
+        if not response.ok:
+            print(response.text)
+
+    def delete_pending(self, status):
+        """delete download queue based on status value"""
+        data = {"query": {"term": {"status": {"value": status}}}}
+        payload = json.dumps(data)
+        url = self.ES_URL + "/ta_download/_delete_by_query"
+        headers = {"Content-type": "application/json"}
+        response = requests.post(
+            url, data=payload, headers=headers, auth=self.ES_AUTH
+        )
         if not response.ok:
             print(response.text)
 
@@ -249,16 +257,19 @@ class PendingList:
 
         headers = {"Content-type": "application/x-ndjson"}
         url = self.ES_URL + "/_bulk"
-        request = requests.post(url, data=query_str, headers=headers)
+        request = requests.post(
+            url, data=query_str, headers=headers, auth=self.ES_AUTH
+        )
         mess_dict = {
             "status": "ignore",
             "level": "info",
             "title": "Added to ignore list",
             "message": "",
         }
-        set_message("progress:download", mess_dict)
+        RedisArchivist().set_message("progress:download", mess_dict)
         if not request.ok:
             print(request)
+            raise ValueError("failed to set video to ignore")
 
 
 class ChannelSubscription:
@@ -267,51 +278,22 @@ class ChannelSubscription:
     def __init__(self):
         config = AppConfig().config
         self.es_url = config["application"]["es_url"]
+        self.es_auth = config["application"]["es_auth"]
         self.channel_size = config["subscriptions"]["channel_size"]
 
-    def get_channels(self, subscribed_only=True):
+    @staticmethod
+    def get_channels(subscribed_only=True):
         """get a list of all channels subscribed to"""
-        headers = {"Content-type": "application/json"}
+        data = {
-        # get PIT ID
+            "sort": [{"channel_name.keyword": {"order": "asc"}}],
-        url = self.es_url + "/ta_channel/_pit?keep_alive=1m"
+        }
-        response = requests.post(url)
-        json_data = json.loads(response.text)
-        pit_id = json_data["id"]
-        # query
         if subscribed_only:
-            data = {
+            data["query"] = {"term": {"channel_subscribed": {"value": True}}}
-                "query": {"term": {"channel_subscribed": {"value": True}}},
-                "size": 50,
-                "pit": {"id": pit_id, "keep_alive": "1m"},
-                "sort": [{"channel_name.keyword": {"order": "asc"}}],
-            }
         else:
-            data = {
+            data["query"] = {"match_all": {}}
-                "query": {"match_all": {}},
+
-                "size": 50,
+        all_channels = IndexPaginate("ta_channel", data).get_results()
-                "pit": {"id": pit_id, "keep_alive": "1m"},
+
-                "sort": [{"channel_name.keyword": {"order": "asc"}}],
-            }
-        query_str = json.dumps(data)
-        url = self.es_url + "/_search"
-        all_channels = []
-        while True:
-            response = requests.get(url, data=query_str, headers=headers)
-            json_data = json.loads(response.text)
-            all_hits = json_data["hits"]["hits"]
-            if all_hits:
-                for hit in all_hits:
-                    source = hit["_source"]
-                    search_after = hit["sort"]
-                    all_channels.append(source)
-                # update search_after with last hit data
-                data["search_after"] = search_after
-                query_str = json.dumps(data)
-            else:
-                break
-        # clean up PIT
-        query_str = json.dumps({"id": pit_id})
-        requests.delete(self.es_url + "/_pit", data=query_str, headers=headers)
         return all_channels
 
     def get_last_youtube_videos(self, channel_id, limit=True):
@@ -334,20 +316,20 @@ class ChannelSubscription:
         all_channels = self.get_channels()
         pending_handler = PendingList()
         all_pending, all_ignore = pending_handler.get_all_pending()
-        all_pending_ids = [i["youtube_id"] for i in all_pending]
+        all_ids = [i["youtube_id"] for i in all_ignore + all_pending]
         all_downloaded = pending_handler.get_all_downloaded()
-        to_ignore = all_pending_ids + all_ignore + all_downloaded
+        to_ignore = all_ids + all_downloaded
         missing_videos = []
         counter = 1
         for channel in all_channels:
             channel_id = channel["channel_id"]
             last_videos = self.get_last_youtube_videos(channel_id)
-            set_message(
+            RedisArchivist().set_message(
                 "progress:download",
                 {
                     "status": "rescan",
                     "level": "info",
-                    "title": "Rescanning: Looking for new videos.",
+                    "title": "Scanning channels: Looking for new videos.",
                     "message": f"Progress: {counter}/{len(all_channels)}",
                 },
             )
@@ -377,52 +359,223 @@ class ChannelSubscription:
             url = self.es_url + "/ta_channel/_update/" + channel_id
             payload = json.dumps({"doc": channel_dict})
         # update channel
-        request = requests.post(url, data=payload, headers=headers)
+        request = requests.post(
+            url, data=payload, headers=headers, auth=self.es_auth
+        )
         if not request.ok:
             print(request.text)
+            raise ValueError("failed change subscribe status")
         # sync to videos
         channel_handler.sync_to_videos()
+        if channel_handler.source == "scraped":
+            channel_handler.get_channel_art()
 
 
-def playlist_extractor(playlist_id):
+class PlaylistSubscription:
-    """return youtube_ids from a playlist_id"""
+    """manage the playlist download functionality"""
-    url = "https://www.youtube.com/playlist?list=" + playlist_id
+
-    obs = {
+    def __init__(self):
-        "default_search": "ytsearch",
+        self.config = AppConfig().config
-        "quiet": True,
+
-        "ignoreerrors": True,
+    @staticmethod
-        "skip_download": True,
+    def get_playlists(subscribed_only=True):
-        "extract_flat": True,
+        """get a list of all active playlists"""
-    }
+        data = {
-    playlist = youtube_dl.YoutubeDL(obs).extract_info(url, download=False)
+            "sort": [{"playlist_channel.keyword": {"order": "desc"}}],
-    playlist_vids = [(i["id"], i["title"]) for i in playlist["entries"]]
+        }
-    return playlist_vids
+        data["query"] = {
+            "bool": {"must": [{"term": {"playlist_active": {"value": True}}}]}
+        }
+        if subscribed_only:
+            data["query"]["bool"]["must"].append(
+                {"term": {"playlist_subscribed": {"value": True}}}
+            )
+
+        all_playlists = IndexPaginate("ta_playlist", data).get_results()
+
+        return all_playlists
+
+    def process_url_str(self, new_playlists):
+        """process playlist subscribe form url_str"""
+        all_indexed = PendingList().get_all_indexed()
+        all_youtube_ids = [i["youtube_id"] for i in all_indexed]
+
+        new_thumbs = []
+
+        for playlist in new_playlists:
+            url_type = playlist["type"]
+            playlist_id = playlist["url"]
+            if not url_type == "playlist":
+                print(f"{playlist_id} not a playlist, skipping...")
+                continue
+
+            playlist_handler = YoutubePlaylist(
+                playlist_id, all_youtube_ids=all_youtube_ids
+            )
+            if not playlist_handler.get_es_playlist():
+                playlist_handler.get_playlist_dict()
+                playlist_handler.playlist_dict["playlist_subscribed"] = True
+                playlist_handler.upload_to_es()
+                playlist_handler.add_vids_to_playlist()
+                thumb = playlist_handler.playlist_dict["playlist_thumbnail"]
+                new_thumbs.append((playlist_id, thumb))
+                self.channel_validate(playlist_handler)
+            else:
+                self.change_subscribe(playlist_id, subscribe_status=True)
+
+            # notify
+            RedisArchivist().set_message(
+                "progress:subscribe", {"status": "subscribing"}
+            )
+
+        return new_thumbs
+
+    @staticmethod
+    def channel_validate(playlist_handler):
+        """make sure channel of playlist is there"""
+        channel_id = playlist_handler.playlist_dict["playlist_channel_id"]
+        channel_handler = YoutubeChannel(channel_id)
+        if channel_handler.source == "scraped":
+            channel_handler.channel_dict["channel_subscribed"] = False
+            channel_handler.upload_to_es()
+            channel_handler.get_channel_art()
+
+    def change_subscribe(self, playlist_id, subscribe_status):
+        """change the subscribe status of a playlist"""
+        es_url = self.config["application"]["es_url"]
+        es_auth = self.config["application"]["es_auth"]
+        playlist_handler = YoutubePlaylist(playlist_id)
+        playlist_handler.get_playlist_dict()
+        subed_now = playlist_handler.playlist_dict["playlist_subscribed"]
+
+        if subed_now == subscribe_status:
+            # status already as expected, do nothing
+            return False
+
+        # update subscribed status
+        headers = {"Content-type": "application/json"}
+        url = f"{es_url}/ta_playlist/_update/{playlist_id}"
+        payload = json.dumps(
+            {"doc": {"playlist_subscribed": subscribe_status}}
+        )
+        response = requests.post(
+            url, data=payload, headers=headers, auth=es_auth
+        )
+        if not response.ok:
+            print(response.text)
+            raise ValueError("failed to change subscribe status")
+
+        return True
+
+    @staticmethod
+    def get_to_ignore():
+        """get all youtube_ids already downloaded or ignored"""
+        pending_handler = PendingList()
+        all_pending, all_ignore = pending_handler.get_all_pending()
+        all_ids = [i["youtube_id"] for i in all_ignore + all_pending]
+        all_downloaded = pending_handler.get_all_downloaded()
+        to_ignore = all_ids + all_downloaded
+        return to_ignore
+
+    def find_missing(self):
+        """find videos in subscribed playlists not downloaded yet"""
+        all_playlists = [i["playlist_id"] for i in self.get_playlists()]
+        to_ignore = self.get_to_ignore()
+
+        missing_videos = []
+        counter = 1
+        for playlist_id in all_playlists:
+            size_limit = self.config["subscriptions"]["channel_size"]
+            playlist_handler = YoutubePlaylist(playlist_id)
+            playlist = playlist_handler.update_playlist()
+            if not playlist:
+                playlist_handler.deactivate()
+                continue
+
+            if size_limit:
+                playlist_entries = playlist["playlist_entries"][:size_limit]
+            else:
+                playlist_entries = playlist["playlist_entries"]
+            all_missing = [i for i in playlist_entries if not i["downloaded"]]
+
+            RedisArchivist().set_message(
+                "progress:download",
+                {
+                    "status": "rescan",
+                    "level": "info",
+                    "title": "Scanning playlists: Looking for new videos.",
+                    "message": f"Progress: {counter}/{len(all_playlists)}",
+                },
+            )
+            for video in all_missing:
+                youtube_id = video["youtube_id"]
+                if youtube_id not in to_ignore:
+                    missing_videos.append(youtube_id)
+            counter = counter + 1
+
+        return missing_videos
 
 
 class VideoDownloader:
-    """handle the video download functionality"""
+    """
+    handle the video download functionality
+    if not initiated with list, take from queue
+    """
 
-    def __init__(self, youtube_id_list):
+    def __init__(self, youtube_id_list=False):
         self.youtube_id_list = youtube_id_list
         self.config = AppConfig().config
+        self.channels = set()
 
-    def download_list(self):
+    def run_queue(self):
-        """download the list of youtube_ids"""
+        """setup download queue in redis loop until no more items"""
-        limit_count = self.config["downloads"]["limit_count"]
+        queue = RedisQueue("dl_queue")
-        if limit_count:
+
-            self.youtube_id_list = self.youtube_id_list[:limit_count]
+        limit_queue = self.config["downloads"]["limit_count"]
+        if limit_queue:
+            queue.trim(limit_queue - 1)
+
+        while True:
+            youtube_id = queue.get_next()
+            if not youtube_id:
+                break
 
-        for youtube_id in self.youtube_id_list:
             try:
                 self.dl_single_vid(youtube_id)
             except youtube_dl.utils.DownloadError:
                 print("failed to download " + youtube_id)
                 continue
             vid_dict = index_new_video(youtube_id)
+            self.channels.add(vid_dict["channel"]["channel_id"])
             self.move_to_archive(vid_dict)
             self.delete_from_pending(youtube_id)
-            if self.config["downloads"]["sleep_interval"]:
+
-                sleep(self.config["downloads"]["sleep_interval"])
+    @staticmethod
+    def add_pending():
+        """add pending videos to download queue"""
+        mess_dict = {
+            "status": "downloading",
+            "level": "info",
+            "title": "Looking for videos to download",
+            "message": "",
+        }
+        RedisArchivist().set_message("progress:download", mess_dict)
+        all_pending, _ = PendingList().get_all_pending()
+        to_add = [i["youtube_id"] for i in all_pending]
+        if not to_add:
+            # there is nothing pending
+            print("download queue is empty")
+            mess_dict = {
+                "status": "downloading",
+                "level": "error",
+                "title": "Download queue is empty",
+                "message": "",
+            }
+            RedisArchivist().set_message("progress:download", mess_dict)
+            return
+
+        queue = RedisQueue("dl_queue")
+        queue.add_list(to_add)
 
     @staticmethod
     def progress_hook(response):
@@ -445,10 +598,10 @@ class VideoDownloader:
             "title": title,
             "message": message,
         }
-        set_message("progress:download", mess_dict)
+        RedisArchivist().set_message("progress:download", mess_dict)
 
-    def dl_single_vid(self, youtube_id):
+    def build_obs(self):
-        """download single video"""
+        """build obs dictionary for yt-dlp"""
         obs = {
             "default_search": "ytsearch",
             "merge_output_format": "mp4",
@@ -459,14 +612,23 @@ class VideoDownloader:
                 + self.config["application"]["file_template"]
             ),
             "progress_hooks": [self.progress_hook],
+            "noprogress": True,
             "quiet": True,
             "continuedl": True,
             "retries": 3,
+            "writethumbnail": False,
+            "noplaylist": True,
+            "check_formats": "selected",
         }
         if self.config["downloads"]["format"]:
             obs["format"] = self.config["downloads"]["format"]
         if self.config["downloads"]["limit_speed"]:
             obs["ratelimit"] = self.config["downloads"]["limit_speed"] * 1024
+
+        throttle = self.config["downloads"]["throttledratelimit"]
+        if throttle:
+            obs["throttledratelimit"] = throttle * 1024
+
         external = False
         if external:
             obs["external_downloader"] = "aria2c"
@@ -482,14 +644,29 @@ class VideoDownloader:
                 }
             )
 
+        if self.config["downloads"]["add_thumbnail"]:
+            postprocessors.append(
+                {
+                    "key": "EmbedThumbnail",
+                    "already_have_thumbnail": True,
+                }
+            )
+            obs["writethumbnail"] = True
+
         obs["postprocessors"] = postprocessors
 
+        return obs
+
+    def dl_single_vid(self, youtube_id):
+        """download single video"""
+        dl_cache = self.config["application"]["cache_dir"] + "/download/"
+        obs = self.build_obs()
+
         # check if already in cache to continue from there
-        cache_dir = self.config["application"]["cache_dir"]
+        all_cached = ignore_filelist(os.listdir(dl_cache))
-        all_cached = os.listdir(cache_dir + "/download/")
         for file_name in all_cached:
             if youtube_id in file_name:
-                obs["outtmpl"] = cache_dir + "/download/" + file_name
+                obs["outtmpl"] = os.path.join(dl_cache, file_name)
         with youtube_dl.YoutubeDL(obs) as ydl:
             try:
                 ydl.download([youtube_id])
@@ -498,36 +675,93 @@ class VideoDownloader:
                 sleep(10)
                 ydl.download([youtube_id])
 
+        if obs["writethumbnail"]:
+            # webp files don't get cleaned up automatically
+            all_cached = ignore_filelist(os.listdir(dl_cache))
+            to_clean = [i for i in all_cached if not i.endswith(".mp4")]
+            for file_name in to_clean:
+                file_path = os.path.join(dl_cache, file_name)
+                os.remove(file_path)
+
     def move_to_archive(self, vid_dict):
         """move downloaded video from cache to archive"""
         videos = self.config["application"]["videos"]
-        channel_name = vid_dict["channel"]["channel_name"]
+        host_uid = self.config["application"]["HOST_UID"]
-        channel_name_clean = clean_string(channel_name)
+        host_gid = self.config["application"]["HOST_GID"]
-        media_url = vid_dict["media_url"]
+        channel_name = clean_string(vid_dict["channel"]["channel_name"])
-        youtube_id = vid_dict["youtube_id"]
+        # make archive folder with correct permissions
-        # make archive folder
+        new_folder = os.path.join(videos, channel_name)
-        videos = self.config["application"]["videos"]
+        if not os.path.exists(new_folder):
-        new_folder = os.path.join(videos, channel_name_clean)
+            os.makedirs(new_folder)
-        os.makedirs(new_folder, exist_ok=True)
+            if host_uid and host_gid:
+                os.chown(new_folder, host_uid, host_gid)
         # find real filename
         cache_dir = self.config["application"]["cache_dir"]
-        for file_str in os.listdir(cache_dir + "/download"):
+        all_cached = ignore_filelist(os.listdir(cache_dir + "/download/"))
-            if youtube_id in file_str:
+        for file_str in all_cached:
+            if vid_dict["youtube_id"] in file_str:
                 old_file = file_str
         old_file_path = os.path.join(cache_dir, "download", old_file)
-        new_file_path = os.path.join(videos, media_url)
+        new_file_path = os.path.join(videos, vid_dict["media_url"])
-        # move and fix permission
+        # move media file and fix permission
         shutil.move(old_file_path, new_file_path)
-        os.chown(
+        if host_uid and host_gid:
-            new_file_path,
+            os.chown(new_file_path, host_uid, host_gid)
-            self.config["application"]["HOST_UID"],
-            self.config["application"]["HOST_GID"],
-        )
 
     def delete_from_pending(self, youtube_id):
         """delete downloaded video from pending index if its there"""
         es_url = self.config["application"]["es_url"]
+        es_auth = self.config["application"]["es_auth"]
         url = f"{es_url}/ta_download/_doc/{youtube_id}"
-        response = requests.delete(url)
+        response = requests.delete(url, auth=es_auth)
         if not response.ok and not response.status_code == 404:
             print(response.text)
+
+    def add_subscribed_channels(self):
+        """add all channels subscribed to refresh"""
+        all_subscribed = PlaylistSubscription().get_playlists()
+        if not all_subscribed:
+            return
+
+        channel_ids = [i["playlist_channel_id"] for i in all_subscribed]
+        for channel_id in channel_ids:
+            self.channels.add(channel_id)
+
+        return
+
+    def validate_playlists(self):
+        """look for playlist needing to update"""
+        print("sync playlists")
+        self.add_subscribed_channels()
+        all_indexed = PendingList().get_all_indexed()
+        all_youtube_ids = [i["youtube_id"] for i in all_indexed]
+        c_counter = 1
+        for channel_id in self.channels:
+            playlists = YoutubeChannel(channel_id).get_indexed_playlists()
+            all_playlist_ids = [i["playlist_id"] for i in playlists]
+            p_counter = 1
+            for playlist_id in all_playlist_ids:
+                playlist_handler = YoutubePlaylist(
+                    playlist_id, all_youtube_ids=all_youtube_ids
+                )
+                playlist_dict = playlist_handler.update_playlist()
+                if not playlist_dict:
+                    playlist_handler.deactivate()
+                    continue
+
+                playlist_handler.add_vids_to_playlist()
+                # notify
+                title = (
+                    "Processing playlists for channels: "
+                    + f"{c_counter}/{len(self.channels)}"
+                )
+                message = f"Progress: {p_counter}/{len(all_playlist_ids)}"
+                mess_dict = {
+                    "status": "downloading",
+                    "level": "info",
+                    "title": title,
+                    "message": message,
+                }
+                RedisArchivist().set_message("progress:download", mess_dict)
+                p_counter = p_counter + 1
+            c_counter = c_counter + 1

tubearchivist/home/src/helper.py

@@ -9,20 +9,20 @@ import re
 import string
 import subprocess
 import unicodedata
+from urllib.parse import parse_qs, urlparse
 
 import redis
 import requests
-
+import yt_dlp as youtube_dl
-REDIS_HOST = os.environ.get("REDIS_HOST")
 
 
-def get_total_hits(index, es_url, match_field):
+def get_total_hits(index, es_url, es_auth, match_field):
     """get total hits from index"""
     headers = {"Content-type": "application/json"}
     data = {"query": {"match": {match_field: True}}}
     payload = json.dumps(data)
     url = f"{es_url}/{index}/_search?filter_path=hits.total"
-    request = requests.post(url, data=payload, headers=headers)
+    request = requests.post(url, data=payload, headers=headers, auth=es_auth)
     if not request.ok:
         print(request.text)
     total_json = json.loads(request.text)
@@ -40,88 +40,250 @@ def clean_string(file_name):
     return cleaned
 
 
-def process_url_list(url_str):
+def ignore_filelist(filelist):
-    """parse url_list to find valid youtube video or channel ids"""
+    """ignore temp files for os.listdir sanitizer"""
-    to_replace = ["watch?v=", "playlist?list="]
+    to_ignore = ["Icon\r\r", "Temporary Items", "Network Trash Folder"]
-    url_list = re.split("\n+", url_str[0])
+    cleaned = []
-    youtube_ids = []
+    for file_name in filelist:
-    for url in url_list:
+        if file_name.startswith(".") or file_name in to_ignore:
-        url_clean = url.strip().strip("/").split("/")[-1]
+            continue
-        for i in to_replace:
+
-            url_clean = url_clean.replace(i, "")
+        cleaned.append(file_name)
-        url_no_param = url_clean.split("&")[0]
+
-        str_len = len(url_no_param)
+    return cleaned
+
+
+class UrlListParser:
+    """take a multi line string and detect valid youtube ids"""
+
+    def __init__(self, url_str):
+        self.url_list = [i.strip() for i in url_str.split()]
+
+    def process_list(self):
+        """loop through the list"""
+        youtube_ids = []
+        for url in self.url_list:
+            parsed = urlparse(url)
+            print(f"processing: {url}")
+            print(parsed)
+            if not parsed.netloc:
+                # is not a url
+                id_type = self.find_valid_id(url)
+                youtube_id = url
+            elif "youtube.com" not in url and "youtu.be" not in url:
+                raise ValueError(f"{url} is not a youtube link")
+            elif parsed.path:
+                # is a url
+                youtube_id, id_type = self.detect_from_url(parsed)
+            else:
+                # not detected
+                raise ValueError(f"failed to detect {url}")
+
+            youtube_ids.append({"url": youtube_id, "type": id_type})
+
+        return youtube_ids
+
+    def detect_from_url(self, parsed):
+        """detect from parsed url"""
+        if parsed.netloc == "youtu.be":
+            # shortened
+            youtube_id = parsed.path.strip("/")
+            _ = self.find_valid_id(youtube_id)
+            return youtube_id, "video"
+
+        if parsed.query:
+            # detect from query string
+            query_parsed = parse_qs(parsed.query)
+            if "v" in query_parsed.keys():
+                youtube_id = query_parsed["v"][0]
+                _ = self.find_valid_id(youtube_id)
+                return youtube_id, "video"
+
+            if "list" in query_parsed.keys():
+                youtube_id = query_parsed["list"][0]
+                return youtube_id, "playlist"
+
+        if parsed.path.startswith("/channel/"):
+            # channel id in url
+            youtube_id = parsed.path.split("/")[2]
+            _ = self.find_valid_id(youtube_id)
+            return youtube_id, "channel"
+
+        # dedect channel with yt_dlp
+        youtube_id = self.extract_channel_name(parsed.geturl())
+        return youtube_id, "channel"
+
+    @staticmethod
+    def find_valid_id(id_str):
+        """dedect valid id from length of string"""
+        str_len = len(id_str)
         if str_len == 11:
-            link_type = "video"
+            id_type = "video"
         elif str_len == 24:
-            link_type = "channel"
+            id_type = "channel"
         elif str_len == 34:
-            link_type = "playlist"
+            id_type = "playlist"
         else:
             # unable to parse
-            raise ValueError("not a valid url: " + url)
+            raise ValueError("not a valid id_str: " + id_str)
 
-        youtube_ids.append({"url": url_no_param, "type": link_type})
+        return id_type
 
-    return youtube_ids
+    @staticmethod
-
+    def extract_channel_name(url):
-
+        """find channel id from channel name with yt-dlp help"""
-def set_message(key, message, expire=True):
+        obs = {
-    """write new message to redis"""
+            "default_search": "ytsearch",
-    redis_connection = redis.Redis(host=REDIS_HOST)
+            "quiet": True,
-    redis_connection.execute_command("JSON.SET", key, ".", json.dumps(message))
+            "skip_download": True,
-    if expire:
+            "extract_flat": True,
-        redis_connection.execute_command("EXPIRE", key, 20)
+            "playlistend": 0,
-
-
-def get_message(key):
-    """get any message from JSON key"""
-    redis_connection = redis.Redis(host=REDIS_HOST)
-    reply = redis_connection.execute_command("JSON.GET", key)
-    if reply:
-        json_str = json.loads(reply)
-    else:
-        json_str = {"status": False}
-    return json_str
-
-
-def get_dl_message(cache_dir):
-    """get latest message if available"""
-    redis_connection = redis.Redis(host=REDIS_HOST)
-    reply = redis_connection.execute_command("JSON.GET", "progress:download")
-    if reply:
-        json_str = json.loads(reply)
-    elif json_str := monitor_cache_dir(cache_dir):
-        json_str = monitor_cache_dir(cache_dir)
-    else:
-        json_str = {"status": False}
-    return json_str
-
-
-def get_lock(lock_key):
-    """handle lock for task management"""
-    redis_lock = redis.Redis(host=REDIS_HOST).lock(lock_key)
-    return redis_lock
-
-
-def monitor_cache_dir(cache_dir):
-    """
-    look at download cache dir directly as alternative progress info
-    """
-    dl_cache = os.path.join(cache_dir, "download")
-    cache_file = os.listdir(dl_cache)
-    if cache_file:
-        filename = cache_file[0][12:].replace("_", " ").split(".")[0]
-        mess_dict = {
-            "status": "downloading",
-            "level": "info",
-            "title": "Downloading: " + filename,
-            "message": "",
         }
-    else:
+        url_info = youtube_dl.YoutubeDL(obs).extract_info(url, download=False)
-        return False
+        try:
+            channel_id = url_info["channel_id"]
+        except KeyError as error:
+            print(f"failed to extract channel id from {url}")
+            raise ValueError from error
 
-    return mess_dict
+        return channel_id
+
+
+class RedisArchivist:
+    """collection of methods to interact with redis"""
+
+    REDIS_HOST = os.environ.get("REDIS_HOST")
+    REDIS_PORT = os.environ.get("REDIS_PORT")
+    NAME_SPACE = "ta:"
+
+    if not REDIS_PORT:
+        REDIS_PORT = 6379
+
+    def __init__(self):
+        self.redis_connection = redis.Redis(
+            host=self.REDIS_HOST, port=self.REDIS_PORT
+        )
+
+    def set_message(self, key, message, expire=True):
+        """write new message to redis"""
+        self.redis_connection.execute_command(
+            "JSON.SET", self.NAME_SPACE + key, ".", json.dumps(message)
+        )
+
+        if expire:
+            self.redis_connection.execute_command(
+                "EXPIRE", self.NAME_SPACE + key, 20
+            )
+
+    def get_message(self, key):
+        """get message dict from redis"""
+        reply = self.redis_connection.execute_command(
+            "JSON.GET", self.NAME_SPACE + key
+        )
+        if reply:
+            json_str = json.loads(reply)
+        else:
+            json_str = {"status": False}
+
+        return json_str
+
+    def del_message(self, key):
+        """delete key from redis"""
+        response = self.redis_connection.execute_command(
+            "DEL", self.NAME_SPACE + key
+        )
+        return response
+
+    def get_lock(self, lock_key):
+        """handle lock for task management"""
+        redis_lock = self.redis_connection.lock(self.NAME_SPACE + lock_key)
+        return redis_lock
+
+    def get_dl_message(self, cache_dir):
+        """get latest download progress message if available"""
+        reply = self.redis_connection.execute_command(
+            "JSON.GET", self.NAME_SPACE + "progress:download"
+        )
+        if reply:
+            json_str = json.loads(reply)
+        elif json_str := self.monitor_cache_dir(cache_dir):
+            json_str = self.monitor_cache_dir(cache_dir)
+        else:
+            json_str = {"status": False}
+
+        return json_str
+
+    @staticmethod
+    def monitor_cache_dir(cache_dir):
+        """
+        look at download cache dir directly as alternative progress info
+        """
+        dl_cache = os.path.join(cache_dir, "download")
+        all_cache_file = os.listdir(dl_cache)
+        cache_file = ignore_filelist(all_cache_file)
+        if cache_file:
+            filename = cache_file[0][12:].replace("_", " ").split(".")[0]
+            mess_dict = {
+                "status": "downloading",
+                "level": "info",
+                "title": "Downloading: " + filename,
+                "message": "",
+            }
+        else:
+            return False
+
+        return mess_dict
+
+
+class RedisQueue:
+    """dynamically interact with the download queue in redis"""
+
+    REDIS_HOST = os.environ.get("REDIS_HOST")
+    REDIS_PORT = os.environ.get("REDIS_PORT")
+    NAME_SPACE = "ta:"
+
+    if not REDIS_PORT:
+        REDIS_PORT = 6379
+
+    def __init__(self, key):
+        self.key = self.NAME_SPACE + key
+        self.conn = redis.Redis(host=self.REDIS_HOST, port=self.REDIS_PORT)
+
+    def get_all(self):
+        """return all elements in list"""
+        result = self.conn.execute_command("LRANGE", self.key, 0, -1)
+        all_elements = [i.decode() for i in result]
+        return all_elements
+
+    def add_list(self, to_add):
+        """add list to queue"""
+        self.conn.execute_command("RPUSH", self.key, *to_add)
+
+    def add_priority(self, to_add):
+        """add single video to front of queue"""
+        self.clear_item(to_add)
+        self.conn.execute_command("LPUSH", self.key, to_add)
+
+    def get_next(self):
+        """return next element in the queue, False if none"""
+        result = self.conn.execute_command("LPOP", self.key)
+        if not result:
+            return False
+
+        next_element = result.decode()
+        return next_element
+
+    def clear(self):
+        """delete list from redis"""
+        self.conn.execute_command("DEL", self.key)
+
+    def clear_item(self, to_clear):
+        """remove single item from list if it's there"""
+        self.conn.execute_command("LREM", self.key, 0, to_clear)
+
+    def trim(self, size):
+        """trim the queue based on settings amount"""
+        self.conn.execute_command("LTRIM", self.key, 0, size)
 
 
 class DurationConverter:
@@ -152,6 +314,10 @@ class DurationConverter:
     @staticmethod
     def get_str(duration_sec):
         """takes duration in sec and returns clean string"""
+        if not duration_sec:
+            # failed to extract
+            return "NA"
+
         hours = duration_sec // 3600
         minutes = (duration_sec - (hours * 3600)) // 60
         secs = duration_sec - (hours * 3600) - (minutes * 60)

tubearchivist/home/src/index.py

@@ -15,7 +15,8 @@ import requests
 import yt_dlp as youtube_dl
 from bs4 import BeautifulSoup
 from home.src.config import AppConfig
-from home.src.helper import DurationConverter, clean_string, process_url_list
+from home.src.helper import DurationConverter, UrlListParser, clean_string
+from home.src.thumbnails import ThumbManager
 
 
 class YoutubeChannel:
@@ -23,7 +24,9 @@ class YoutubeChannel:
 
     CONFIG = AppConfig().config
     ES_URL = CONFIG["application"]["es_url"]
+    ES_AUTH = CONFIG["application"]["es_auth"]
     CACHE_DIR = CONFIG["application"]["cache_dir"]
+    VIDEOS = CONFIG["application"]["videos"]
 
     def __init__(self, channel_id):
         self.channel_id = channel_id
@@ -49,7 +52,7 @@ class YoutubeChannel:
         """get from elastic search first if possible"""
         channel_id = self.channel_id
         url = f"{self.ES_URL}/ta_channel/_doc/{channel_id}"
-        response = requests.get(url)
+        response = requests.get(url, auth=self.ES_AUTH)
         if response.ok:
             channel_source = response.json()["_source"]
             self.source = "elastic"
@@ -61,7 +64,7 @@ class YoutubeChannel:
         channel_id = self.channel_id
         url = f"https://www.youtube.com/channel/{channel_id}/about?hl=en"
         cookies = {"CONSENT": "YES+xxxxxxxxxxxxxxxxxxxxxxxxxxx"}
-        response = requests.get(url, cookies=cookies)
+        response = requests.get(url, cookies=cookies, auth=self.ES_AUTH)
         if response.ok:
             channel_page = response.text
         else:
@@ -152,23 +155,23 @@ class YoutubeChannel:
 
         return meta_channel_dict
 
+    def get_channel_art(self):
+        """download channel art for new channels"""
+        channel_id = self.channel_id
+        channel_thumb = self.channel_dict["channel_thumb_url"]
+        channel_banner = self.channel_dict["channel_banner_url"]
+        ThumbManager().download_chan(
+            [(channel_id, channel_thumb, channel_banner)]
+        )
+
     def upload_to_es(self):
         """upload channel data to elastic search"""
         url = f"{self.ES_URL}/ta_channel/_doc/{self.channel_id}"
-        response = requests.put(url, json=self.channel_dict)
+        response = requests.put(url, json=self.channel_dict, auth=self.ES_AUTH)
         print(f"added {self.channel_id} to es")
         if not response.ok:
             print(response.text)
-
+            raise ValueError("failed to add channel to index")
-    def clear_cache(self):
-        """delete banner and thumb from cache if available"""
-        channel_cache = os.path.join(self.CACHE_DIR, "channels")
-        thumb = os.path.join(channel_cache, self.channel_id + "_thumb.jpg")
-        banner = os.path.join(channel_cache, self.channel_id + "_banner.jpg")
-        if os.path.exists(thumb):
-            os.remove(thumb)
-        if os.path.exists(banner):
-            os.remove(banner)
 
     def sync_to_videos(self):
         """sync new channel_dict to all videos of channel"""
@@ -182,28 +185,93 @@ class YoutubeChannel:
         data = {"description": channel_id, "processors": processors}
         payload = json.dumps(data)
         url = self.ES_URL + "/_ingest/pipeline/" + channel_id
-        request = requests.put(url, data=payload, headers=headers)
+        request = requests.put(
+            url, data=payload, headers=headers, auth=self.ES_AUTH
+        )
         if not request.ok:
             print(request.text)
         # apply pipeline
         data = {"query": {"match": {"channel.channel_id": channel_id}}}
         payload = json.dumps(data)
         url = self.ES_URL + "/ta_video/_update_by_query?pipeline=" + channel_id
-        request = requests.post(url, data=payload, headers=headers)
+        request = requests.post(
+            url, data=payload, headers=headers, auth=self.ES_AUTH
+        )
         if not request.ok:
             print(request.text)
 
-    def get_total_hits(self):
+    def get_folder_path(self):
-        """get total channels indexed"""
+        """get folder where media files get stored"""
+        channel_name = self.channel_dict["channel_name"]
+        folder_name = clean_string(channel_name)
+        folder_path = os.path.join(self.VIDEOS, folder_name)
+        return folder_path
+
+    def delete_es_videos(self):
+        """delete all channel documents from elasticsearch"""
         headers = {"Content-type": "application/json"}
-        data = {"query": {"match_all": {}}}
+        data = {
+            "query": {
+                "term": {"channel.channel_id": {"value": self.channel_id}}
+            }
+        }
         payload = json.dumps(data)
-        url = f"{self.ES_URL}/ta_channel/_search?filter_path=hits.total"
+        url = self.ES_URL + "/ta_video/_delete_by_query"
-        request = requests.post(url, data=payload, headers=headers)
+        response = requests.post(
-        if not request.ok:
+            url, data=payload, headers=headers, auth=self.ES_AUTH
-            print(request.text)
+        )
-        total_hits = json.loads(request.text)["hits"]["total"]["value"]
+        if not response.ok:
-        return total_hits
+            print(response.text)
+
+    def delete_channel(self):
+        """delete channel and all videos"""
+        print(f"deleting {self.channel_id} and all matching media files")
+        folder_path = self.get_folder_path()
+        print("delete all media files")
+        try:
+            all_videos = os.listdir(folder_path)
+            for video in all_videos:
+                video_path = os.path.join(folder_path, video)
+                os.remove(video_path)
+            os.rmdir(folder_path)
+        except FileNotFoundError:
+            print(f"no videos found for {folder_path}")
+
+        ThumbManager().delete_chan_thumb(self.channel_id)
+
+        print("delete indexed videos")
+        self.delete_es_videos()
+        url = self.ES_URL + "/ta_channel/_doc/" + self.channel_id
+        response = requests.delete(url, auth=self.ES_AUTH)
+        if not response.ok:
+            print(response.text)
+
+    def get_all_playlists(self):
+        """get all playlists owned by this channel"""
+        url = (
+            f"https://www.youtube.com/channel/{self.channel_id}"
+            + "/playlists?view=1&sort=dd&shelf_id=0"
+        )
+        obs = {
+            "quiet": True,
+            "skip_download": True,
+            "extract_flat": True,
+        }
+        playlists = youtube_dl.YoutubeDL(obs).extract_info(url)
+        all_entries = [(i["id"], i["title"]) for i in playlists["entries"]]
+
+        return all_entries
+
+    def get_indexed_playlists(self):
+        """get all indexed playlists from channel"""
+        data = {
+            "query": {
+                "term": {"playlist_channel_id": {"value": self.channel_id}}
+            },
+            "sort": [{"playlist_channel.keyword": {"order": "desc"}}],
+        }
+        all_playlists = IndexPaginate("ta_playlist", data).get_results()
+        return all_playlists
 
 
 class YoutubeVideo:
@@ -211,17 +279,19 @@ class YoutubeVideo:
 
     CONFIG = AppConfig().config
     ES_URL = CONFIG["application"]["es_url"]
+    ES_AUTH = CONFIG["application"]["es_auth"]
     CACHE_DIR = CONFIG["application"]["cache_dir"]
     VIDEOS = CONFIG["application"]["videos"]
 
     def __init__(self, youtube_id):
         self.youtube_id = youtube_id
         self.channel_id = None
-        self.vid_dict = self.get_wrapper()
+        self.vid_dict = None
 
-    def get_wrapper(self):
+    def get_vid_dict(self):
         """wrapper to loop around youtube_dl to retry on failure"""
         print(f"get video data for {self.youtube_id}")
+        vid_dict = False
         for i in range(3):
             try:
                 vid_dict = self.get_youtubedl_vid_data()
@@ -232,7 +302,7 @@ class YoutubeVideo:
             else:
                 break
 
-        return vid_dict
+        self.vid_dict = vid_dict
 
     def get_youtubedl_vid_data(self):
         """parse youtubedl extract info"""
@@ -241,6 +311,8 @@ class YoutubeVideo:
             "quiet": True,
             "default_search": "ytsearch",
             "skip_download": True,
+            "check_formats": "selected",
+            "noplaylist": True,
         }
         try:
             vid = youtube_dl.YoutubeDL(obs).extract_info(youtube_id)
@@ -330,25 +402,19 @@ class YoutubeVideo:
     def get_es_data(self):
         """get current data from elastic search"""
         url = self.ES_URL + "/ta_video/_doc/" + self.youtube_id
-        response = requests.get(url)
+        response = requests.get(url, auth=self.ES_AUTH)
         if not response.ok:
             print(response.text)
         es_vid_dict = json.loads(response.text)
         return es_vid_dict
 
     def upload_to_es(self):
-        """upload channel data to elastic search"""
+        """upload video data to elastic search"""
-        url = f"{self.ES_URL}/ta_video/_doc/{self.youtube_id}"
+        url = f"{self.ES_URL}/ta_video/_doc/{self.youtube_id}/?refresh=true"
-        response = requests.put(url, json=self.vid_dict)
+        response = requests.put(url, json=self.vid_dict, auth=self.ES_AUTH)
         if not response.ok:
             print(response.text)
-
+            raise ValueError("failed to add video to index")
-    def delete_cache(self):
-        """delete thumbnail from cache if exist"""
-        video_cache = os.path.join(self.CACHE_DIR, "videos")
-        thumb = os.path.join(video_cache, self.youtube_id + ".jpg")
-        if os.path.exists(thumb):
-            os.remove(thumb)
 
     def deactivate(self):
         """deactivate document on extractor error"""
@@ -357,17 +423,312 @@ class YoutubeVideo:
         url = f"{self.ES_URL}/ta_video/_update/{youtube_id}"
         data = {"script": "ctx._source.active = false"}
         json_str = json.dumps(data)
-        response = requests.post(url, data=json_str, headers=headers)
+        response = requests.post(
+            url, data=json_str, headers=headers, auth=self.ES_AUTH
+        )
         print(f"deactivated {youtube_id}")
         if not response.ok:
             print(response.text)
 
+    def delete_media_file(self):
+        """delete video file, meta data, thumbnails"""
+        # delete media file
+        es_vid_dict = self.get_es_data()
+        media_url = es_vid_dict["_source"]["media_url"]
+        print(f"delete {media_url} from file system")
+        to_delete = os.path.join(self.VIDEOS, media_url)
+        os.remove(to_delete)
+        # delete from index
+        url = f"{self.ES_URL}/ta_video/_doc/{self.youtube_id}"
+        response = requests.delete(url, auth=self.ES_AUTH)
+        if not response.ok:
+            print(response.text)
+        # delete thumbs from cache
+        ThumbManager().delete_vid_thumb(self.youtube_id)
+
+
+class YoutubePlaylist:
+    """represent a single playlist on YouTube"""
+
+    CONFIG = AppConfig().config
+    ES_URL = CONFIG["application"]["es_url"]
+    ES_AUTH = CONFIG["application"]["es_auth"]
+
+    def __init__(self, playlist_id, all_youtube_ids=False):
+        self.playlist_id = playlist_id
+        self.stamp = int(datetime.now().strftime("%s"))
+        self.all_youtube_ids = all_youtube_ids
+        self.playlist_dict = False
+
+    def get_playlist_dict(self, scrape=False):
+        """get data from es or youtube"""
+        print(f"get playlist with id {self.playlist_id}")
+
+        if scrape:
+            playlist_dict = self.get_youtube_playlist()
+            if not playlist_dict:
+                return False
+            playlist_dict["playlist_entries"] = self.get_entries()
+        else:
+            playlist_dict = self.get_es_playlist()
+            if not playlist_dict:
+                playlist_dict = self.get_youtube_playlist()
+                playlist_dict["playlist_entries"] = self.get_entries()
+
+        self.playlist_dict = playlist_dict
+        return True
+
+    def get_youtube_playlist(self):
+        """get meta data dict from youtube"""
+        url = "https://www.youtube.com/playlist?list=" + self.playlist_id
+        obs = {
+            "default_search": "ytsearch",
+            "quiet": True,
+            "skip_download": True,
+            "extract_flat": True,
+            "playlistend": 0,
+        }
+        try:
+            playlist = youtube_dl.YoutubeDL(obs).extract_info(
+                url, download=False
+            )
+        except (
+            youtube_dl.utils.ExtractorError,
+            youtube_dl.utils.DownloadError,
+        ):
+            print("failed to get info for " + self.playlist_id)
+            return False
+
+        playlist_es = {
+            "playlist_id": self.playlist_id,
+            "playlist_active": True,
+            "playlist_subscribed": False,
+            "playlist_name": playlist["title"],
+            "playlist_channel": playlist["channel"],
+            "playlist_channel_id": playlist["channel_id"],
+            "playlist_thumbnail": playlist["thumbnails"][-1]["url"],
+            "playlist_description": playlist["description"] or False,
+            "playlist_last_refresh": self.stamp,
+        }
+
+        return playlist_es
+
+    def get_es_playlist(self):
+        """get indexed data from es"""
+        url = f"{self.ES_URL}/ta_playlist/_doc/{self.playlist_id}"
+        response = requests.get(url, auth=self.ES_AUTH)
+        if response.ok:
+            return json.loads(response.text)["_source"]
+
+        return False
+
+    def get_entries(self, playlistend=False):
+        """get all videos in playlist"""
+        url = "https://www.youtube.com/playlist?list=" + self.playlist_id
+        obs = {
+            "default_search": "ytsearch",
+            "quiet": True,
+            "skip_download": True,
+            "extract_flat": True,
+        }
+        if playlistend:
+            obs["playlistend"] = playlistend
+
+        try:
+            playlist = youtube_dl.YoutubeDL(obs).extract_info(
+                url, download=False
+            )
+        except (
+            youtube_dl.utils.ExtractorError,
+            youtube_dl.utils.DownloadError,
+        ):
+            print("failed to get plealist entries for " + self.playlist_id)
+            return False
+
+        all_members = []
+        for idx, entry in enumerate(playlist["entries"]):
+            uploader = entry["uploader"]
+            youtube_id = entry["id"]
+            if self.all_youtube_ids:
+                downloaded = youtube_id in self.all_youtube_ids
+            else:
+                downloaded = False
+            if not uploader:
+                continue
+            to_append = {
+                "youtube_id": youtube_id,
+                "title": entry["title"],
+                "uploader": uploader,
+                "idx": idx,
+                "downloaded": downloaded,
+            }
+            all_members.append(to_append)
+
+        return all_members
+
+    def upload_to_es(self):
+        """add playlist to es with its entries"""
+        playlist = self.playlist_dict
+        url = f"{self.ES_URL}/ta_playlist/_doc/{self.playlist_id}"
+        response = requests.put(url, json=playlist, auth=self.ES_AUTH)
+        if not response.ok:
+            print(response.text)
+            raise ValueError("failed to add playlist to index")
+
+    def add_vids_to_playlist(self):
+        """sync the playlist id to videos"""
+        playlist_dict = self.playlist_dict
+        script = (
+            'if (!ctx._source.containsKey("playlist")) '
+            + "{ctx._source.playlist = [params.playlist]} "
+            + "else if (!ctx._source.playlist.contains(params.playlist)) "
+            + "{ctx._source.playlist.add(params.playlist)} "
+            + "else {ctx.op = 'none'}"
+        )
+
+        bulk_list = []
+        for entry in playlist_dict["playlist_entries"]:
+            youtube_id = entry["youtube_id"]
+            action = {"update": {"_id": youtube_id, "_index": "ta_video"}}
+            source = {
+                "script": {
+                    "source": script,
+                    "lang": "painless",
+                    "params": {"playlist": self.playlist_id},
+                }
+            }
+            bulk_list.append(json.dumps(action))
+            bulk_list.append(json.dumps(source))
+
+        # add last newline
+        bulk_list.append("\n")
+        query_str = "\n".join(bulk_list)
+        headers = {"Content-type": "application/x-ndjson"}
+        url = self.ES_URL + "/_bulk"
+        response = requests.post(
+            url, data=query_str, headers=headers, auth=self.ES_AUTH
+        )
+        if not response.ok:
+            print(response.text)
+
+    def update_playlist(self):
+        """update metadata for playlist with data from YouTube"""
+        subscribed = self.get_es_playlist()["playlist_subscribed"]
+        self.get_playlist_dict(scrape=True)
+        if not self.playlist_dict:
+            # return false to deactivate
+            return False
+
+        self.playlist_dict["playlist_subscribed"] = subscribed
+        self.upload_to_es()
+        return self.playlist_dict
+
+    def build_nav(self, youtube_id):
+        """find next and previous in playlist of a given youtube_id"""
+        all_entries_available = self.playlist_dict["playlist_entries"]
+        all_entries = [i for i in all_entries_available if i["downloaded"]]
+        current = [i for i in all_entries if i["youtube_id"] == youtube_id]
+        # stop if not found or playlist of 1
+        if not current or not len(all_entries) > 1:
+            return False
+
+        current_idx = all_entries.index(current[0])
+        if current_idx == 0:
+            previous_item = False
+        else:
+            previous_item = all_entries[current_idx - 1]
+            prev_thumb = ThumbManager().vid_thumb_path(
+                previous_item["youtube_id"]
+            )
+            previous_item["vid_thumb"] = prev_thumb
+
+        if current_idx == len(all_entries) - 1:
+            next_item = False
+        else:
+            next_item = all_entries[current_idx + 1]
+            next_thumb = ThumbManager().vid_thumb_path(next_item["youtube_id"])
+            next_item["vid_thumb"] = next_thumb
+
+        nav = {
+            "playlist_meta": {
+                "current_idx": current[0]["idx"],
+                "playlist_id": self.playlist_id,
+                "playlist_name": self.playlist_dict["playlist_name"],
+                "playlist_channel": self.playlist_dict["playlist_channel"],
+            },
+            "playlist_previous": previous_item,
+            "playlist_next": next_item,
+        }
+        return nav
+
+    def delete_metadata(self):
+        """delete metadata for playlist"""
+        script = (
+            "ctx._source.playlist.removeAll("
+            + "Collections.singleton(params.playlist)) "
+        )
+        data = {
+            "query": {
+                "term": {"playlist.keyword": {"value": self.playlist_id}}
+            },
+            "script": {
+                "source": script,
+                "lang": "painless",
+                "params": {"playlist": self.playlist_id},
+            },
+        }
+        payload = json.dumps(data)
+        url = f"{self.ES_URL}/ta_video/_update_by_query"
+        headers = {"Content-type": "application/json"}
+        response = requests.post(
+            url, data=payload, headers=headers, auth=self.ES_AUTH
+        )
+        if not response.ok:
+            print(response.text)
+
+        self.delete_playlist()
+
+    def delete_videos_playlist(self):
+        """delete playlist with all videos"""
+        print(f"delete playlist {self.playlist_id} with all videos")
+        self.get_playlist_dict()
+        all_youtube_id = [
+            i["youtube_id"]
+            for i in self.playlist_dict["playlist_entries"]
+            if i["downloaded"]
+        ]
+        for youtube_id in all_youtube_id:
+            YoutubeVideo(youtube_id).delete_media_file()
+
+        self.delete_playlist()
+
+    def delete_playlist(self):
+        """delete only playlist document"""
+        url = f"{self.ES_URL}/ta_playlist/_doc/{self.playlist_id}"
+        response = requests.delete(url, auth=self.ES_AUTH)
+        if not response.ok:
+            print(response.text)
+
+    def deactivate(self):
+        """deactivate document on extractor error"""
+        headers = {"Content-type": "application/json"}
+        url = f"{self.ES_URL}/ta_playlist/_update/{self.playlist_id}"
+        data = {"script": "ctx._source.playlist_active = false"}
+        json_str = json.dumps(data)
+        response = requests.post(
+            url, data=json_str, headers=headers, auth=self.ES_AUTH
+        )
+        print(f"deactivated {self.playlist_id}")
+        if not response.ok:
+            print(response.text)
+
 
 class WatchState:
     """handle watched checkbox for videos and channels"""
 
     CONFIG = AppConfig().config
     ES_URL = CONFIG["application"]["es_url"]
+    ES_AUTH = CONFIG["application"]["es_auth"]
     HEADERS = {"Content-type": "application/json"}
 
     def __init__(self, youtube_id):
@@ -381,63 +742,186 @@ class WatchState:
             self.mark_vid_watched()
         elif url_type == "channel":
             self.mark_channel_watched()
+        elif url_type == "playlist":
+            self.mark_playlist_watched()
 
         print(f"marked {self.youtube_id} as watched")
 
+    def mark_as_unwatched(self):
+        """revert watched state to false"""
+        url_type = self.dedect_type()
+        if url_type == "video":
+            self.mark_vid_watched(revert=True)
+
+        print(f"revert {self.youtube_id} as unwatched")
+
     def dedect_type(self):
         """find youtube id type"""
-        url_process = process_url_list([self.youtube_id])
+        print(self.youtube_id)
+        url_process = UrlListParser(self.youtube_id).process_list()
         url_type = url_process[0]["type"]
-
         return url_type
 
-    def mark_vid_watched(self):
+    def mark_vid_watched(self, revert=False):
         """change watched status of single video"""
         url = self.ES_URL + "/ta_video/_update/" + self.youtube_id
         data = {
             "doc": {"player": {"watched": True, "watched_date": self.stamp}}
         }
+        if revert:
+            data["doc"]["player"]["watched"] = False
+
         payload = json.dumps(data)
-        request = requests.post(url, data=payload, headers=self.HEADERS)
+        request = requests.post(
+            url, data=payload, headers=self.HEADERS, auth=self.ES_AUTH
+        )
         if not request.ok:
             print(request.text)
+            raise ValueError("failed to mark video as watched")
 
     def mark_channel_watched(self):
         """change watched status of every video in channel"""
-        es_url = self.ES_URL
-        headers = self.HEADERS
-        youtube_id = self.youtube_id
-        # create pipeline
         data = {
-            "description": youtube_id,
+            "query": {
-            "processors": [
+                "bool": {
-                {"set": {"field": "player.watched", "value": True}},
+                    "must": [
-                {"set": {"field": "player.watched_date", "value": self.stamp}},
+                        {
-            ],
+                            "term": {
+                                "channel.channel_id": {
+                                    "value": self.youtube_id
+                                }
+                            }
+                        },
+                        {"term": {"player.watched": {"value": False}}},
+                    ]
+                }
+            },
+            "script": {
+                "source": "ctx._source.player['watched'] = true",
+                "lang": "painless",
+            },
         }
         payload = json.dumps(data)
-        url = f"{es_url}/_ingest/pipeline/{youtube_id}"
+        url = f"{self.ES_URL}/ta_video/_update_by_query"
-        request = requests.put(url, data=payload, headers=headers)
+        request = requests.post(
+            url, data=payload, headers=self.HEADERS, auth=self.ES_AUTH
+        )
         if not request.ok:
             print(request.text)
-            raise ValueError("failed to post ingest pipeline")
+            raise ValueError("failed mark channel as watched")
 
-        # apply pipeline
+    def mark_playlist_watched(self):
-        must_list = [
+        """change watched state of all videos in playlist"""
-            {"term": {"channel.channel_id": {"value": youtube_id}}},
+        data = {
-            {"term": {"player.watched": {"value": False}}},
+            "query": {
-        ]
+                "bool": {
-        data = {"query": {"bool": {"must": must_list}}}
+                    "must": [
+                        {
+                            "term": {
+                                "playlist.keyword": {"value": self.youtube_id}
+                            }
+                        },
+                        {"term": {"player.watched": {"value": False}}},
+                    ]
+                }
+            },
+            "script": {
+                "source": "ctx._source.player['watched'] = true",
+                "lang": "painless",
+            },
+        }
         payload = json.dumps(data)
-        url = f"{es_url}/ta_video/_update_by_query?pipeline={youtube_id}"
+        url = f"{self.ES_URL}/ta_video/_update_by_query"
-        request = requests.post(url, data=payload, headers=headers)
+        request = requests.post(
+            url, data=payload, headers=self.HEADERS, auth=self.ES_AUTH
+        )
         if not request.ok:
             print(request.text)
+            raise ValueError("failed mark playlist as watched")
+
+
+class IndexPaginate:
+    """use search_after to go through whole index"""
+
+    CONFIG = AppConfig().config
+    ES_URL = CONFIG["application"]["es_url"]
+    ES_AUTH = CONFIG["application"]["es_auth"]
+    HEADERS = {"Content-type": "application/json"}
+    DEFAULT_SIZE = 500
+
+    def __init__(self, index_name, data, size=False):
+        self.index_name = index_name
+        self.data = data
+        self.pit_id = False
+        self.size = size
+
+    def get_results(self):
+        """get all results"""
+        self.get_pit()
+        self.validate_data()
+        all_results = self.run_loop()
+        self.clean_pit()
+        return all_results
+
+    def get_pit(self):
+        """get pit for index"""
+        url = f"{self.ES_URL}/{self.index_name}/_pit?keep_alive=10m"
+        response = requests.post(url, auth=self.ES_AUTH)
+        json_data = json.loads(response.text)
+
+        self.pit_id = json_data["id"]
+
+    def validate_data(self):
+        """add pit and size to data"""
+        if "sort" not in self.data.keys():
+            print(self.data)
+            raise ValueError("missing sort key in data")
+
+        size = self.size or self.DEFAULT_SIZE
+
+        self.data["size"] = size
+        self.data["pit"] = {"id": self.pit_id, "keep_alive": "10m"}
+
+    def run_loop(self):
+        """loop through results until last hit"""
+        query_str = json.dumps(self.data)
+        url = self.ES_URL + "/_search"
+
+        all_results = []
+        while True:
+            response = requests.get(
+                url, data=query_str, headers=self.HEADERS, auth=self.ES_AUTH
+            )
+            json_data = json.loads(response.text)
+            all_hits = json_data["hits"]["hits"]
+            if all_hits:
+                for hit in all_hits:
+                    source = hit["_source"]
+                    search_after = hit["sort"]
+                    all_results.append(source)
+                # update search_after with last hit data
+                self.data["search_after"] = search_after
+                query_str = json.dumps(self.data)
+            else:
+                break
+
+        return all_results
+
+    def clean_pit(self):
+        """delete pit from elastic search"""
+        query_str = json.dumps({"id": self.pit_id})
+        requests.delete(
+            self.ES_URL + "/_pit",
+            data=query_str,
+            headers=self.HEADERS,
+            auth=self.ES_AUTH,
+        )
 
 
 def index_new_video(youtube_id, missing_vid=False):
     """combine video and channel classes for new video index"""
     vid_handler = YoutubeVideo(youtube_id)
+    vid_handler.get_vid_dict()
     if not vid_handler.vid_dict:
         raise ValueError("failed to get metadata for " + youtube_id)
 
@@ -452,6 +936,7 @@ def index_new_video(youtube_id, missing_vid=False):
     if channel_handler.source == "scraped":
         channel_handler.channel_dict["channel_subscribed"] = False
         channel_handler.upload_to_es()
+        channel_handler.get_channel_art()
     # upload video to es
     vid_handler.upload_to_es()
     # return vid_dict for further processing

tubearchivist/home/src/index_management.py

@@ -13,6 +13,7 @@ from datetime import datetime
 
 import requests
 from home.src.config import AppConfig
+from home.src.helper import ignore_filelist
 
 # expected mapping and settings
 INDEX_CONFIG = [
@@ -105,6 +106,16 @@ INDEX_CONFIG = [
             "vid_last_refresh": {"type": "date"},
             "youtube_id": {"type": "keyword"},
             "published": {"type": "date"},
+            "playlist": {
+                "type": "text",
+                "fields": {
+                    "keyword": {
+                        "type": "keyword",
+                        "ignore_above": 256,
+                        "normalizer": "to_lower",
+                    }
+                },
+            },
         },
         "expected_set": {
             "analysis": {
@@ -153,6 +164,43 @@ INDEX_CONFIG = [
             "number_of_replicas": "0",
         },
     },
+    {
+        "index_name": "playlist",
+        "expected_map": {
+            "playlist_id": {"type": "keyword"},
+            "playlist_description": {"type": "text"},
+            "playlist_name": {
+                "type": "text",
+                "fields": {
+                    "keyword": {
+                        "type": "keyword",
+                        "ignore_above": 256,
+                        "normalizer": "to_lower",
+                    }
+                },
+            },
+            "playlist_channel": {
+                "type": "text",
+                "fields": {
+                    "keyword": {
+                        "type": "keyword",
+                        "ignore_above": 256,
+                        "normalizer": "to_lower",
+                    }
+                },
+            },
+            "playlist_channel_id": {"type": "keyword"},
+            "playlist_thumbnail": {"type": "keyword"},
+        },
+        "expected_set": {
+            "analysis": {
+                "normalizer": {
+                    "to_lower": {"type": "custom", "filter": ["lowercase"]}
+                }
+            },
+            "number_of_replicas": "0",
+        },
+    },
 ]
 
 
@@ -163,6 +211,7 @@ class ElasticIndex:
 
     CONFIG = AppConfig().config
     ES_URL = CONFIG["application"]["es_url"]
+    ES_AUTH = CONFIG["application"]["es_auth"]
     HEADERS = {"Content-type": "application/json"}
 
     def __init__(self, index_name, expected_map, expected_set):
@@ -175,7 +224,7 @@ class ElasticIndex:
         """check if index already exists and return mapping if it does"""
         index_name = self.index_name
         url = f"{self.ES_URL}/ta_{index_name}"
-        response = requests.get(url)
+        response = requests.get(url, auth=self.ES_AUTH)
         exists = response.ok
 
         if exists:
@@ -273,7 +322,9 @@ class ElasticIndex:
         query = {"source": {"index": source}, "dest": {"index": destination}}
         data = json.dumps(query)
         url = self.ES_URL + "/_reindex?refresh=true"
-        response = requests.post(url=url, data=data, headers=self.HEADERS)
+        response = requests.post(
+            url=url, data=data, headers=self.HEADERS, auth=self.ES_AUTH
+        )
         if not response.ok:
             print(response.text)
 
@@ -283,7 +334,7 @@ class ElasticIndex:
             url = f"{self.ES_URL}/ta_{self.index_name}_backup"
         else:
             url = f"{self.ES_URL}/ta_{self.index_name}"
-        response = requests.delete(url)
+        response = requests.delete(url, auth=self.ES_AUTH)
         if not response.ok:
             print(response.text)
 
@@ -300,7 +351,9 @@ class ElasticIndex:
         # create
         url = f"{self.ES_URL}/ta_{self.index_name}"
         data = json.dumps(payload)
-        response = requests.put(url=url, data=data, headers=self.HEADERS)
+        response = requests.put(
+            url=url, data=data, headers=self.HEADERS, auth=self.ES_AUTH
+        )
         if not response.ok:
             print(response.text)
 
@@ -318,9 +371,10 @@ class ElasticBackup:
         """export all documents of a single index"""
         headers = {"Content-type": "application/json"}
         es_url = self.config["application"]["es_url"]
+        es_auth = self.config["application"]["es_auth"]
         # get PIT ID
         url = f"{es_url}/ta_{index_name}/_pit?keep_alive=1m"
-        response = requests.post(url)
+        response = requests.post(url, auth=es_auth)
         json_data = json.loads(response.text)
         pit_id = json_data["id"]
         # build query
@@ -335,7 +389,9 @@ class ElasticBackup:
         # loop until nothing left
         all_results = []
         while True:
-            response = requests.get(url, data=query_str, headers=headers)
+            response = requests.get(
+                url, data=query_str, headers=headers, auth=es_auth
+            )
             json_data = json.loads(response.text)
             all_hits = json_data["hits"]["hits"]
             if all_hits:
@@ -349,7 +405,9 @@ class ElasticBackup:
                 break
         # clean up PIT
         query_str = json.dumps({"id": pit_id})
-        requests.delete(es_url + "/_pit", data=query_str, headers=headers)
+        requests.delete(
+            es_url + "/_pit", data=query_str, headers=headers, auth=es_auth
+        )
 
         return all_results
 
@@ -415,6 +473,7 @@ class ElasticBackup:
         """send bulk to es"""
         cache_dir = self.config["application"]["cache_dir"]
         es_url = self.config["application"]["es_url"]
+        es_auth = self.config["application"]["es_auth"]
         headers = {"Content-type": "application/x-ndjson"}
         file_path = os.path.join(cache_dir, file_name)
 
@@ -425,7 +484,9 @@ class ElasticBackup:
             return
 
         url = es_url + "/_bulk"
-        request = requests.post(url, data=query_str, headers=headers)
+        request = requests.post(
+            url, data=query_str, headers=headers, auth=es_auth
+        )
         if not request.ok:
             print(request.text)
 
@@ -433,9 +494,11 @@ class ElasticBackup:
         """extract backup zip and return filelist"""
         cache_dir = self.config["application"]["cache_dir"]
         backup_dir = os.path.join(cache_dir, "backup")
+        backup_files = os.listdir(backup_dir)
+        all_backup_files = ignore_filelist(backup_files)
         all_available_backups = [
             i
-            for i in os.listdir(backup_dir)
+            for i in all_backup_files
             if i.startswith("ta_") and i.endswith(".zip")
         ]
         all_available_backups.sort()
@@ -466,6 +529,15 @@ class ElasticBackup:
             self.post_bulk_restore(file_name)
             os.remove(file_name)
 
+    def index_exists(self, index_name):
+        """check if index already exists to skip"""
+        es_url = self.config["application"]["es_url"]
+        es_auth = self.config["application"]["es_auth"]
+        url = f"{es_url}/ta_{index_name}"
+        response = requests.get(url, auth=es_auth)
+
+        return response.ok
+
 
 def backup_all_indexes():
     """backup all es indexes to disk"""
@@ -473,6 +545,8 @@ def backup_all_indexes():
 
     for index in backup_handler.index_config:
         index_name = index["index_name"]
+        if not backup_handler.index_exists(index_name):
+            continue
         all_results = backup_handler.get_all_documents(index_name)
         file_content = backup_handler.build_bulk(all_results)
         backup_handler.write_es_json(file_content, index_name)

tubearchivist/home/src/reindex.py

@@ -18,12 +18,18 @@ import requests
 from home.src.config import AppConfig
 from home.src.download import ChannelSubscription, PendingList, VideoDownloader
 from home.src.helper import (
+    RedisArchivist,
     clean_string,
-    get_message,
     get_total_hits,
-    set_message,
+    ignore_filelist,
 )
-from home.src.index import YoutubeChannel, YoutubeVideo, index_new_video
+from home.src.index import (
+    YoutubeChannel,
+    YoutubePlaylist,
+    YoutubeVideo,
+    index_new_video,
+)
+from home.src.thumbnails import ThumbManager
 
 
 class Reindex:
@@ -34,28 +40,33 @@ class Reindex:
         config = AppConfig().config
         self.sleep_interval = config["downloads"]["sleep_interval"]
         self.es_url = config["application"]["es_url"]
+        self.es_auth = config["application"]["es_auth"]
         self.refresh_interval = 90
         # scan
-        self.video_daily, self.channel_daily = self.get_daily()
         self.all_youtube_ids = False
         self.all_channel_ids = False
+        self.all_playlist_ids = False
 
     def get_daily(self):
         """get daily refresh values"""
-        total_videos = get_total_hits("ta_video", self.es_url, "active")
+        total_videos = get_total_hits(
+            "ta_video", self.es_url, self.es_auth, "active"
+        )
         video_daily = ceil(total_videos / self.refresh_interval * 1.2)
         total_channels = get_total_hits(
-            "ta_channel", self.es_url, "channel_active"
+            "ta_channel", self.es_url, self.es_auth, "channel_active"
         )
         channel_daily = ceil(total_channels / self.refresh_interval * 1.2)
-        return (video_daily, channel_daily)
+        playlist_daily = get_total_hits(
+            "ta_playlist", self.es_url, self.es_auth, "playlist_active"
+        )
+        return (video_daily, channel_daily, playlist_daily)
 
-    def get_outdated_vids(self):
+    def get_outdated_vids(self, size):
         """get daily videos to refresh"""
         headers = {"Content-type": "application/json"}
         now = int(datetime.now().strftime("%s"))
         now_3m = now - 3 * 30 * 24 * 60 * 60
-        size = self.video_daily
         data = {
             "size": size,
             "query": {
@@ -71,19 +82,20 @@ class Reindex:
         }
         query_str = json.dumps(data)
         url = self.es_url + "/ta_video/_search"
-        response = requests.get(url, data=query_str, headers=headers)
+        response = requests.get(
+            url, data=query_str, headers=headers, auth=self.es_auth
+        )
         if not response.ok:
             print(response.text)
         response_dict = json.loads(response.text)
         all_youtube_ids = [i["_id"] for i in response_dict["hits"]["hits"]]
         return all_youtube_ids
 
-    def get_outdated_channels(self):
+    def get_outdated_channels(self, size):
         """get daily channels to refresh"""
         headers = {"Content-type": "application/json"}
         now = int(datetime.now().strftime("%s"))
         now_3m = now - 3 * 30 * 24 * 60 * 60
-        size = self.channel_daily
         data = {
             "size": size,
             "query": {
@@ -99,17 +111,50 @@ class Reindex:
         }
         query_str = json.dumps(data)
         url = self.es_url + "/ta_channel/_search"
-        response = requests.get(url, data=query_str, headers=headers)
+        response = requests.get(
+            url, data=query_str, headers=headers, auth=self.es_auth
+        )
         if not response.ok:
             print(response.text)
         response_dict = json.loads(response.text)
         all_channel_ids = [i["_id"] for i in response_dict["hits"]["hits"]]
         return all_channel_ids
 
+    def get_outdated_playlists(self, size):
+        """get daily outdated playlists to refresh"""
+        headers = {"Content-type": "application/json"}
+        now = int(datetime.now().strftime("%s"))
+        now_3m = now - 3 * 30 * 24 * 60 * 60
+        data = {
+            "size": size,
+            "query": {
+                "bool": {
+                    "must": [
+                        {"match": {"playlist_active": True}},
+                        {"range": {"playlist_last_refresh": {"lte": now_3m}}},
+                    ]
+                }
+            },
+            "sort": [{"playlist_last_refresh": {"order": "asc"}}],
+            "_source": False,
+        }
+        query_str = json.dumps(data)
+        url = self.es_url + "/ta_playlist/_search"
+        response = requests.get(
+            url, data=query_str, headers=headers, auth=self.es_auth
+        )
+        if not response.ok:
+            print(response.text)
+        response_dict = json.loads(response.text)
+        all_playlist_ids = [i["_id"] for i in response_dict["hits"]["hits"]]
+        return all_playlist_ids
+
     def check_outdated(self):
         """add missing vids and channels"""
-        self.all_youtube_ids = self.get_outdated_vids()
+        video_daily, channel_daily, playlist_daily = self.get_daily()
-        self.all_channel_ids = self.get_outdated_channels()
+        self.all_youtube_ids = self.get_outdated_vids(video_daily)
+        self.all_channel_ids = self.get_outdated_channels(channel_daily)
+        self.all_playlist_ids = self.get_outdated_playlists(playlist_daily)
 
     def rescrape_all_channels(self):
         """sync new data from channel to all matching videos"""
@@ -127,7 +172,7 @@ class Reindex:
                 "title": "Scraping all youtube channels",
                 "message": message,
             }
-            set_message("progress:download", mess_dict)
+            RedisArchivist().set_message("progress:download", mess_dict)
             channel_index = YoutubeChannel(channel_id)
             subscribed = channel_index.channel_dict["channel_subscribed"]
             channel_index.channel_dict = channel_index.build_channel_dict(
@@ -144,6 +189,7 @@ class Reindex:
     def reindex_single_video(youtube_id):
         """refresh data for single video"""
         vid_handler = YoutubeVideo(youtube_id)
+        vid_handler.get_vid_dict()
         if not vid_handler.vid_dict:
             # stop if deactivated
             vid_handler.deactivate()
@@ -154,14 +200,24 @@ class Reindex:
         date_downloaded = es_vid_dict["_source"]["date_downloaded"]
         channel_dict = es_vid_dict["_source"]["channel"]
         channel_name = channel_dict["channel_name"]
+        try:
+            playlist = es_vid_dict["_source"]["playlist"]
+        except KeyError:
+            playlist = False
+
         vid_handler.build_file_path(channel_name)
         # add to vid_dict
         vid_handler.vid_dict["player"] = player
         vid_handler.vid_dict["date_downloaded"] = date_downloaded
         vid_handler.vid_dict["channel"] = channel_dict
+        if playlist:
+            vid_handler.vid_dict["playlist"] = playlist
         # update
         vid_handler.upload_to_es()
-        vid_handler.delete_cache()
+        thumb_handler = ThumbManager()
+        thumb_handler.delete_vid_thumb(youtube_id)
+        to_download = (youtube_id, vid_handler.vid_dict["vid_thumb_url"])
+        thumb_handler.download_vid([to_download], notify=False)
 
     @staticmethod
     def reindex_single_channel(channel_id):
@@ -172,9 +228,31 @@ class Reindex:
             scrape=True
         )
         channel_handler.channel_dict["channel_subscribed"] = subscribed
+        # update
         channel_handler.upload_to_es()
         channel_handler.sync_to_videos()
-        channel_handler.clear_cache()
+        thumb_handler = ThumbManager()
+        thumb_handler.delete_chan_thumb(channel_id)
+        channel_thumb = channel_handler.channel_dict["channel_thumb_url"]
+        channel_banner = channel_handler.channel_dict["channel_banner_url"]
+        to_download = (channel_id, channel_thumb, channel_banner)
+        thumb_handler.download_chan([to_download])
+
+    @staticmethod
+    def reindex_single_playlist(playlist_id, all_indexed_ids):
+        """refresh playlist data"""
+        playlist_handler = YoutubePlaylist(
+            playlist_id, all_youtube_ids=all_indexed_ids
+        )
+        playlist = playlist_handler.update_playlist()
+        if not playlist:
+            playlist_handler.deactivate()
+            return
+
+        playlist_thumbnail = (playlist_id, playlist["playlist_thumbnail"])
+        thumb_handler = ThumbManager()
+        thumb_handler.download_playlist([playlist_thumbnail])
+        return
 
     def reindex(self):
         """reindex what's needed"""
@@ -190,6 +268,15 @@ class Reindex:
             self.reindex_single_channel(channel_id)
             if self.sleep_interval:
                 sleep(self.sleep_interval)
+        # playlist
+        print(f"reindexing {len(self.all_playlist_ids)} playlists")
+        if self.all_playlist_ids:
+            all_indexed = PendingList().get_all_indexed()
+            all_indexed_ids = [i["youtube_id"] for i in all_indexed]
+            for playlist_id in self.all_playlist_ids:
+                self.reindex_single_playlist(playlist_id, all_indexed_ids)
+                if self.sleep_interval:
+                    sleep(self.sleep_interval)
 
 
 class FilesystemScanner:
@@ -197,6 +284,7 @@ class FilesystemScanner:
 
     CONFIG = AppConfig().config
     ES_URL = CONFIG["application"]["es_url"]
+    ES_AUTH = CONFIG["application"]["es_auth"]
     VIDEOS = CONFIG["application"]["videos"]
 
     def __init__(self):
@@ -209,12 +297,15 @@ class FilesystemScanner:
 
     def get_all_downloaded(self):
         """get a list of all video files downloaded"""
-        all_channels = os.listdir(self.VIDEOS)
+        channels = os.listdir(self.VIDEOS)
+        all_channels = ignore_filelist(channels)
         all_channels.sort()
         all_downloaded = []
         for channel_name in all_channels:
             channel_path = os.path.join(self.VIDEOS, channel_name)
-            for video in os.listdir(channel_path):
+            videos = os.listdir(channel_path)
+            all_videos = ignore_filelist(videos)
+            for video in all_videos:
                 youtube_id = video[9:20]
                 all_downloaded.append((channel_name, video, youtube_id))
 
@@ -227,10 +318,10 @@ class FilesystemScanner:
         all_indexed_raw = index_handler.get_all_indexed()
         all_indexed = []
         for video in all_indexed_raw:
-            youtube_id = video["_id"]
+            youtube_id = video["youtube_id"]
-            media_url = video["_source"]["media_url"]
+            media_url = video["media_url"]
-            published = video["_source"]["published"]
+            published = video["published"]
-            title = video["_source"]["title"]
+            title = video["title"]
             all_indexed.append((youtube_id, media_url, published, title))
         return all_indexed
 
@@ -294,6 +385,7 @@ class FilesystemScanner:
         """rename media files as identified by find_bad_media_url"""
         for bad_filename in self.to_rename:
             channel, filename, expected_filename = bad_filename
+            print(f"renaming [{filename}] to [{expected_filename}]")
             old_path = os.path.join(self.VIDEOS, channel, filename)
             new_path = os.path.join(self.VIDEOS, channel, expected_filename)
             os.rename(old_path, new_path)
@@ -303,6 +395,7 @@ class FilesystemScanner:
         bulk_list = []
         for video_mismatch in self.mismatch:
             youtube_id, media_url = video_mismatch
+            print(f"{youtube_id}: fixing media url {media_url}")
             action = {"update": {"_id": youtube_id, "_index": "ta_video"}}
             source = {"doc": {"media_url": media_url}}
             bulk_list.append(json.dumps(action))
@@ -313,16 +406,19 @@ class FilesystemScanner:
         # make the call
         headers = {"Content-type": "application/x-ndjson"}
         url = self.ES_URL + "/_bulk"
-        request = requests.post(url, data=query_str, headers=headers)
+        request = requests.post(
+            url, data=query_str, headers=headers, auth=self.ES_AUTH
+        )
         if not request.ok:
             print(request.text)
 
     def delete_from_index(self):
         """find indexed but deleted mediafile"""
         for indexed in self.to_delete:
-            youtube_id, _ = indexed
+            youtube_id = indexed[0]
+            print(f"deleting {youtube_id} from index")
             url = self.ES_URL + "/ta_video/_doc/" + youtube_id
-            request = requests.delete(url)
+            request = requests.delete(url, auth=self.ES_AUTH)
             if not request.ok:
                 print(request.text)
 
@@ -339,8 +435,8 @@ class ManualImport:
 
     def import_folder_parser(self):
         """detect files in import folder"""
-
+        import_files = os.listdir(self.IMPORT_DIR)
-        to_import = os.listdir(self.IMPORT_DIR)
+        to_import = ignore_filelist(import_files)
         to_import.sort()
         video_files = [i for i in to_import if not i.endswith(".json")]
 
@@ -396,6 +492,8 @@ class ManualImport:
     def process_import(self):
         """go through identified media files"""
 
+        all_videos_added = []
+
         for media_file in self.identified:
             json_file = media_file["json_file"]
             video_file = media_file["video_file"]
@@ -408,6 +506,9 @@ class ManualImport:
             # identify and archive
             vid_dict = index_new_video(youtube_id)
             VideoDownloader([youtube_id]).move_to_archive(vid_dict)
+            youtube_id = vid_dict["youtube_id"]
+            thumb_url = vid_dict["vid_thumb_url"]
+            all_videos_added.append((youtube_id, thumb_url))
 
             # cleanup
             if os.path.exists(video_path):
@@ -416,6 +517,8 @@ class ManualImport:
                 json_path = os.path.join(self.CACHE_DIR, "import", json_file)
                 os.remove(json_path)
 
+        return all_videos_added
+
     def move_to_cache(self, video_path, youtube_id):
         """move identified video file to cache, convert to mp4"""
         file_name = os.path.split(video_path)[-1]
@@ -453,12 +556,16 @@ def scan_filesystem():
     filesystem_handler = FilesystemScanner()
     filesystem_handler.list_comarison()
     if filesystem_handler.to_rename:
+        print("renaming files")
         filesystem_handler.rename_files()
     if filesystem_handler.mismatch:
+        print("fixing media urls in index")
         filesystem_handler.send_mismatch_bulk()
     if filesystem_handler.to_delete:
+        print("delete metadata from index")
         filesystem_handler.delete_from_index()
     if filesystem_handler.to_index:
+        print("index new videos")
         for missing_vid in filesystem_handler.to_index:
             youtube_id = missing_vid[2]
             index_new_video(youtube_id, missing_vid=missing_vid)
@@ -468,7 +575,7 @@ def reindex_old_documents():
     """daily refresh of old documents"""
     # check needed last run
     now = int(datetime.now().strftime("%s"))
-    last_reindex = get_message("last_reindex")
+    last_reindex = RedisArchivist().get_message("last_reindex")
     if isinstance(last_reindex, int) and now - last_reindex < 60 * 60 * 24:
         return
     # continue if needed
@@ -476,4 +583,4 @@ def reindex_old_documents():
     reindex_handler.check_outdated()
     reindex_handler.reindex()
     # set timestamp
-    set_message("last_reindex", now, expire=False)
+    RedisArchivist().set_message("last_reindex", now, expire=False)

tubearchivist/home/src/searching.py

@@ -7,13 +7,13 @@ Functionality:
 """
 
 import math
-import os
 import urllib.parse
 from datetime import datetime
 
 import requests
 from home.src.config import AppConfig
-from PIL import Image
+from home.src.helper import RedisArchivist
+from home.src.thumbnails import ThumbManager
 
 
 class SearchHandler:
@@ -21,19 +21,21 @@ class SearchHandler:
 
     CONFIG = AppConfig().config
     CACHE_DIR = CONFIG["application"]["cache_dir"]
+    ES_AUTH = CONFIG["application"]["es_auth"]
 
-    def __init__(self, url, data, cache=True):
+    def __init__(self, url, data):
         self.max_hits = None
         self.url = url
         self.data = data
-        self.cache = cache
 
     def get_data(self):
         """get the data"""
         if self.data:
-            response = requests.get(self.url, json=self.data).json()
+            response = requests.get(
+                self.url, json=self.data, auth=self.ES_AUTH
+            ).json()
         else:
-            response = requests.get(self.url).json()
+            response = requests.get(self.url, auth=self.ES_AUTH).json()
 
         if "hits" in response.keys():
             self.max_hits = response["hits"]["total"]["value"]
@@ -60,10 +62,6 @@ class SearchHandler:
                 channel_dict = self.channel_cache_link(hit)
                 if channel_dict not in all_channels:
                     all_channels.append(channel_dict)
-        if self.cache:
-            # validate cache
-            self.cache_dl_vids(all_videos)
-            self.cache_dl_chan(all_channels)
 
         return return_value
 
@@ -102,48 +100,6 @@ class SearchHandler:
         }
         return channel_dict
 
-    def cache_dl_vids(self, all_videos):
-        """video thumbs links for cache"""
-        vid_cache = os.path.join(self.CACHE_DIR, "videos")
-        all_vid_cached = os.listdir(vid_cache)
-        # videos
-        for video_dict in all_videos:
-            youtube_id = video_dict["youtube_id"]
-            if not youtube_id + ".jpg" in all_vid_cached:
-                cache_path = os.path.join(vid_cache, youtube_id + ".jpg")
-                thumb_url = video_dict["vid_thumb"]
-                img_raw = requests.get(thumb_url, stream=True).raw
-                img = Image.open(img_raw)
-                width, height = img.size
-                if not width / height == 16 / 9:
-                    new_height = width / 16 * 9
-                    offset = (height - new_height) / 2
-                    img = img.crop((0, offset, width, height - offset))
-                img.convert("RGB").save(cache_path)
-
-    def cache_dl_chan(self, all_channels):
-        """download channel thumbs"""
-        chan_cache = os.path.join(self.CACHE_DIR, "channels")
-        all_chan_cached = os.listdir(chan_cache)
-        for channel_dict in all_channels:
-            channel_id_cache = channel_dict["channel_id"]
-            channel_banner_url = channel_dict["chan_banner"]
-            channel_banner = channel_id_cache + "_banner.jpg"
-            channel_thumb_url = channel_dict["chan_thumb"]
-            channel_thumb = channel_id_cache + "_thumb.jpg"
-            # thumb
-            if channel_thumb_url and channel_thumb not in all_chan_cached:
-                cache_path = os.path.join(chan_cache, channel_thumb)
-                img_raw = requests.get(channel_thumb_url, stream=True).content
-                with open(cache_path, "wb") as f:
-                    f.write(img_raw)
-            # banner
-            if channel_banner_url and channel_banner not in all_chan_cached:
-                cache_path = os.path.join(chan_cache, channel_banner)
-                img_raw = requests.get(channel_banner_url, stream=True).content
-                with open(cache_path, "wb") as f:
-                    f.write(img_raw)
-
     @staticmethod
     def hit_cleanup(hit):
         """clean up and parse data from a single hit"""
@@ -165,6 +121,17 @@ class SearchHandler:
             date_str = datetime.strftime(date_refresh, "%d %b, %Y")
             hit["source"]["vid_last_refresh"] = date_str
 
+        if "playlist_last_refresh" in hit_keys:
+            playlist_last_refresh = hit["source"]["playlist_last_refresh"]
+            date_refresh = datetime.fromtimestamp(playlist_last_refresh)
+            date_str = datetime.strftime(date_refresh, "%d %b, %Y")
+            hit["source"]["playlist_last_refresh"] = date_str
+
+        if "vid_thumb_url" in hit_keys:
+            youtube_id = hit["source"]["youtube_id"]
+            thumb_path = ThumbManager().vid_thumb_path(youtube_id)
+            hit["source"]["vid_thumb_url"] = thumb_path
+
         if "channel_last_refresh" in hit_keys:
             refreshed = hit["source"]["channel_last_refresh"]
             date_refresh = datetime.fromtimestamp(refreshed)
@@ -205,7 +172,7 @@ class SearchForm:
                 }
             },
         }
-        look_up = SearchHandler(url, data, cache=False)
+        look_up = SearchHandler(url, data)
         search_results = look_up.get_data()
         return {"results": search_results}
 
@@ -221,13 +188,23 @@ class Pagination:
     figure out the pagination based on page size and total_hits
     """
 
-    def __init__(self, page_get, search_get=False):
+    def __init__(self, page_get, user_id, search_get=False):
-        config = AppConfig().config
+        self.user_id = user_id
-        self.page_size = config["archive"]["page_size"]
+        self.page_size = self.get_page_size()
         self.page_get = page_get
         self.search_get = search_get
         self.pagination = self.first_guess()
 
+    def get_page_size(self):
+        """get default or user modified page_size"""
+        key = f"{self.user_id}:page_size"
+        page_size = RedisArchivist().get_message(key)["status"]
+        if not page_size:
+            config = AppConfig().config
+            page_size = config["archive"]["page_size"]
+
+        return page_size
+
     def first_guess(self):
         """build first guess before api call"""
         page_get = self.page_get

tubearchivist/home/src/thumbnails.py

@@ -0,0 +1,324 @@
+"""
+functionality:
+- handle download and caching for thumbnails
+"""
+
+import os
+from collections import Counter
+from time import sleep
+
+import home.src.download as download
+import requests
+from home.src.config import AppConfig
+from home.src.helper import RedisArchivist, ignore_filelist
+from mutagen.mp4 import MP4, MP4Cover
+from PIL import Image
+
+
+class ThumbManager:
+    """handle thumbnails related functions"""
+
+    CONFIG = AppConfig().config
+    MEDIA_DIR = CONFIG["application"]["videos"]
+    CACHE_DIR = CONFIG["application"]["cache_dir"]
+    VIDEO_DIR = os.path.join(CACHE_DIR, "videos")
+    CHANNEL_DIR = os.path.join(CACHE_DIR, "channels")
+    PLAYLIST_DIR = os.path.join(CACHE_DIR, "playlists")
+
+    def get_all_thumbs(self):
+        """get all video artwork already downloaded"""
+        all_thumb_folders = ignore_filelist(os.listdir(self.VIDEO_DIR))
+        all_thumbs = []
+        for folder in all_thumb_folders:
+            folder_path = os.path.join(self.VIDEO_DIR, folder)
+            if os.path.isfile(folder_path):
+                self.update_path(folder)
+                all_thumbs.append(folder_path)
+                continue
+                # raise exemption here in a future version
+                # raise FileExistsError("video cache dir has files inside")
+
+            all_folder_thumbs = ignore_filelist(os.listdir(folder_path))
+            all_thumbs.extend(all_folder_thumbs)
+
+        return all_thumbs
+
+    def update_path(self, file_name):
+        """reorganize thumbnails into folders as update path from v0.0.5"""
+        folder_name = file_name[0].lower()
+        folder_path = os.path.join(self.VIDEO_DIR, folder_name)
+        old_file = os.path.join(self.VIDEO_DIR, file_name)
+        new_file = os.path.join(folder_path, file_name)
+        os.makedirs(folder_path, exist_ok=True)
+        os.rename(old_file, new_file)
+
+    def get_needed_thumbs(self, missing_only=False):
+        """get a list of all missing thumbnails"""
+        all_thumbs = self.get_all_thumbs()
+        all_indexed = download.PendingList().get_all_indexed()
+        all_in_queue, all_ignored = download.PendingList().get_all_pending()
+
+        needed_thumbs = []
+        for video in all_indexed:
+            youtube_id = video["youtube_id"]
+            thumb_url = video["vid_thumb_url"]
+            if missing_only:
+                if youtube_id + ".jpg" not in all_thumbs:
+                    needed_thumbs.append((youtube_id, thumb_url))
+            else:
+                needed_thumbs.append((youtube_id, thumb_url))
+
+        for video in all_in_queue + all_ignored:
+            youtube_id = video["youtube_id"]
+            thumb_url = video["vid_thumb_url"]
+            if missing_only:
+                if youtube_id + ".jpg" not in all_thumbs:
+                    needed_thumbs.append((youtube_id, thumb_url))
+            else:
+                needed_thumbs.append((youtube_id, thumb_url))
+
+        return needed_thumbs
+
+    def get_missing_channels(self):
+        """get all channel artwork"""
+        all_channel_art = os.listdir(self.CHANNEL_DIR)
+        files = [i[0:24] for i in all_channel_art]
+        cached_channel_ids = [k for (k, v) in Counter(files).items() if v > 1]
+        channels = download.ChannelSubscription().get_channels(
+            subscribed_only=False
+        )
+
+        missing_channels = []
+        for channel in channels:
+            channel_id = channel["channel_id"]
+            if channel_id not in cached_channel_ids:
+                channel_banner = channel["channel_banner_url"]
+                channel_thumb = channel["channel_thumb_url"]
+                missing_channels.append(
+                    (channel_id, channel_thumb, channel_banner)
+                )
+
+        return missing_channels
+
+    def get_missing_playlists(self):
+        """get all missing playlist artwork"""
+        all_downloaded = ignore_filelist(os.listdir(self.PLAYLIST_DIR))
+        all_ids_downloaded = [i.replace(".jpg", "") for i in all_downloaded]
+
+        playlists = download.PlaylistSubscription().get_playlists(
+            subscribed_only=False
+        )
+
+        missing_playlists = []
+        for playlist in playlists:
+            playlist_id = playlist["playlist_id"]
+            if playlist_id not in all_ids_downloaded:
+                playlist_thumb = playlist["playlist_thumbnail"]
+                missing_playlists.append((playlist_id, playlist_thumb))
+
+        return missing_playlists
+
+    def get_raw_img(self, img_url, thumb_type):
+        """get raw image from youtube and handle 404"""
+        try:
+            app_root = self.CONFIG["application"]["app_root"]
+        except KeyError:
+            # lazy keyerror fix to not have to deal with a strange startup
+            # racing contition between the threads in HomeConfig.ready()
+            app_root = "/app"
+        default_map = {
+            "video": os.path.join(
+                app_root, "static/img/default-video-thumb.jpg"
+            ),
+            "icon": os.path.join(
+                app_root, "static/img/default-channel-icon.jpg"
+            ),
+            "banner": os.path.join(
+                app_root, "static/img/default-channel-banner.jpg"
+            ),
+        }
+        if img_url:
+            try:
+                response = requests.get(img_url, stream=True)
+            except ConnectionError:
+                sleep(5)
+                response = requests.get(img_url, stream=True)
+            if not response.ok and not response.status_code == 404:
+                print("retry thumbnail download for " + img_url)
+                sleep(5)
+                response = requests.get(img_url, stream=True)
+        else:
+            response = False
+        if not response or response.status_code == 404:
+            # use default
+            img_raw = Image.open(default_map[thumb_type])
+        else:
+            # use response
+            img_obj = response.raw
+            img_raw = Image.open(img_obj)
+
+        return img_raw
+
+    def download_vid(self, missing_thumbs, notify=True):
+        """download all missing thumbnails from list"""
+        print(f"downloading {len(missing_thumbs)} thumbnails")
+        counter = 1
+        for youtube_id, thumb_url in missing_thumbs:
+            folder_path = os.path.join(self.VIDEO_DIR, youtube_id[0].lower())
+            thumb_path = os.path.join(
+                self.CACHE_DIR, self.vid_thumb_path(youtube_id)
+            )
+
+            os.makedirs(folder_path, exist_ok=True)
+            img_raw = self.get_raw_img(thumb_url, "video")
+
+            width, height = img_raw.size
+            if not width / height == 16 / 9:
+                new_height = width / 16 * 9
+                offset = (height - new_height) / 2
+                img_raw = img_raw.crop((0, offset, width, height - offset))
+
+            img_raw.convert("RGB").save(thumb_path)
+
+            progress = f"{counter}/{len(missing_thumbs)}"
+            if notify:
+                mess_dict = {
+                    "status": "pending",
+                    "level": "info",
+                    "title": "Downloading Thumbnails",
+                    "message": "Progress: " + progress,
+                }
+                RedisArchivist().set_message("progress:download", mess_dict)
+
+            if counter % 25 == 0:
+                print("thumbnail progress: " + progress)
+            counter = counter + 1
+
+    def download_chan(self, missing_channels):
+        """download needed artwork for channels"""
+        print(f"downloading {len(missing_channels)} channel artwork")
+        for channel in missing_channels:
+            channel_id, channel_thumb, channel_banner = channel
+
+            thumb_path = os.path.join(
+                self.CHANNEL_DIR, channel_id + "_thumb.jpg"
+            )
+            img_raw = self.get_raw_img(channel_thumb, "icon")
+            img_raw.convert("RGB").save(thumb_path)
+
+            banner_path = os.path.join(
+                self.CHANNEL_DIR, channel_id + "_banner.jpg"
+            )
+            img_raw = self.get_raw_img(channel_banner, "banner")
+            img_raw.convert("RGB").save(banner_path)
+
+            mess_dict = {
+                "status": "pending",
+                "level": "info",
+                "title": "Adding to download queue.",
+                "message": "Downloading Channel Art...",
+            }
+            RedisArchivist().set_message("progress:download", mess_dict)
+
+    def download_playlist(self, missing_playlists):
+        """download needed artwork for playlists"""
+        print(f"downloading {len(missing_playlists)} playlist artwork")
+        for playlist in missing_playlists:
+            playlist_id, playlist_thumb_url = playlist
+            thumb_path = os.path.join(self.PLAYLIST_DIR, playlist_id + ".jpg")
+            img_raw = self.get_raw_img(playlist_thumb_url, "video")
+            img_raw.convert("RGB").save(thumb_path)
+
+            mess_dict = {
+                "status": "pending",
+                "level": "info",
+                "title": "Adding to download queue.",
+                "message": "Downloading Playlist Art...",
+            }
+            RedisArchivist().set_message("progress:download", mess_dict)
+
+    @staticmethod
+    def vid_thumb_path(youtube_id):
+        """build expected path for video thumbnail from youtube_id"""
+        folder_name = youtube_id[0].lower()
+        folder_path = os.path.join("videos", folder_name)
+        thumb_path = os.path.join(folder_path, youtube_id + ".jpg")
+        return thumb_path
+
+    def delete_vid_thumb(self, youtube_id):
+        """delete video thumbnail if exists"""
+        thumb_path = self.vid_thumb_path(youtube_id)
+        to_delete = os.path.join(self.CACHE_DIR, thumb_path)
+        if os.path.exists(to_delete):
+            os.remove(to_delete)
+
+    def delete_chan_thumb(self, channel_id):
+        """delete all artwork of channel"""
+        thumb = os.path.join(self.CHANNEL_DIR, channel_id + "_thumb.jpg")
+        banner = os.path.join(self.CHANNEL_DIR, channel_id + "_banner.jpg")
+        if os.path.exists(thumb):
+            os.remove(thumb)
+        if os.path.exists(banner):
+            os.remove(banner)
+
+    def cleanup_downloaded(self):
+        """find downloaded thumbnails without video indexed"""
+        all_thumbs = self.get_all_thumbs()
+        all_indexed = self.get_needed_thumbs()
+        all_needed_thumbs = [i[0] + ".jpg" for i in all_indexed]
+        for thumb in all_thumbs:
+            if thumb not in all_needed_thumbs:
+                # cleanup
+                youtube_id = thumb.rstrip(".jpg")
+                self.delete_vid_thumb(youtube_id)
+
+    def get_thumb_list(self):
+        """get list of mediafiles and matching thumbnails"""
+        all_indexed = download.PendingList().get_all_indexed()
+        video_list = []
+        for video in all_indexed:
+            youtube_id = video["youtube_id"]
+            media_url = os.path.join(self.MEDIA_DIR, video["media_url"])
+            thumb_path = os.path.join(
+                self.CACHE_DIR, self.vid_thumb_path(youtube_id)
+            )
+            video_list.append(
+                {
+                    "media_url": media_url,
+                    "thumb_path": thumb_path,
+                }
+            )
+
+        return video_list
+
+    @staticmethod
+    def write_all_thumbs(video_list):
+        """rewrite the thumbnail into media file"""
+
+        counter = 1
+        for video in video_list:
+            # loop through all videos
+            media_url = video["media_url"]
+            thumb_path = video["thumb_path"]
+
+            mutagen_vid = MP4(media_url)
+            with open(thumb_path, "rb") as f:
+                mutagen_vid["covr"] = [
+                    MP4Cover(f.read(), imageformat=MP4Cover.FORMAT_JPEG)
+                ]
+            mutagen_vid.save()
+            if counter % 50 == 0:
+                print(f"thumbnail write progress {counter}/{len(video_list)}")
+            counter = counter + 1
+
+
+def validate_thumbnails():
+    """check if all thumbnails are there and organized correctly"""
+    handler = ThumbManager()
+    thumbs_to_download = handler.get_needed_thumbs(missing_only=True)
+    handler.download_vid(thumbs_to_download)
+    missing_channels = handler.get_missing_channels()
+    handler.download_chan(missing_channels)
+    missing_playlists = handler.get_missing_playlists()
+    handler.download_playlist(missing_playlists)
+    handler.cleanup_downloaded()

tubearchivist/home/tasks.py

@@ -8,28 +8,53 @@ import os
 
 from celery import Celery, shared_task
 from home.src.config import AppConfig
-from home.src.download import ChannelSubscription, PendingList, VideoDownloader
+from home.src.download import (
-from home.src.helper import get_lock
+    ChannelSubscription,
+    PendingList,
+    PlaylistSubscription,
+    VideoDownloader,
+)
+from home.src.helper import RedisArchivist, RedisQueue, UrlListParser
+from home.src.index import YoutubeChannel, YoutubePlaylist
 from home.src.index_management import backup_all_indexes, restore_from_backup
-from home.src.reindex import ManualImport, reindex_old_documents
+from home.src.reindex import (
+    ManualImport,
+    reindex_old_documents,
+    scan_filesystem,
+)
+from home.src.thumbnails import ThumbManager, validate_thumbnails
 
 CONFIG = AppConfig().config
-REDIS_HOST = CONFIG["application"]["REDIS_HOST"]
+REDIS_HOST = os.environ.get("REDIS_HOST")
+REDIS_PORT = os.environ.get("REDIS_PORT")
+
+if not REDIS_PORT:
+    REDIS_PORT = 6379
 
 os.environ.setdefault("DJANGO_SETTINGS_MODULE", "home.settings")
-app = Celery("tasks", broker="redis://" + REDIS_HOST)
+app = Celery("tasks", broker=f"redis://{REDIS_HOST}:{REDIS_PORT}")
-app.config_from_object("django.conf:settings", namespace="CELERY")
+app.config_from_object("django.conf:settings", namespace="ta:")
 app.autodiscover_tasks()
 
 
 @shared_task
 def update_subscribed():
     """look for missing videos and add to pending"""
+    message = {
+        "status": "rescan",
+        "level": "info",
+        "title": "Start rescanning channels and playlists.",
+    }
+    RedisArchivist().set_message("progress:download", message)
     channel_handler = ChannelSubscription()
-    missing_videos = channel_handler.find_missing()
+    missing_from_channels = channel_handler.find_missing()
+    playlist_handler = PlaylistSubscription()
+    missing_from_playlists = playlist_handler.find_missing()
+    missing_videos = missing_from_channels + missing_from_playlists
     if missing_videos:
         pending_handler = PendingList()
-        pending_handler.add_to_pending(missing_videos)
+        all_videos_added = pending_handler.add_to_pending(missing_videos)
+        ThumbManager().download_vid(all_videos_added)
     # check if reindex is needed
     check_reindex.delay()
 
@@ -37,20 +62,45 @@ def update_subscribed():
 @shared_task
 def download_pending():
     """download latest pending videos"""
-    pending_handler = PendingList()
+    have_lock = False
-    pending_vids = pending_handler.get_all_pending()[0]
+    my_lock = RedisArchivist().get_lock("downloading")
-    to_download = [i["youtube_id"] for i in pending_vids]
+
-    to_download.reverse()
+    try:
-    if to_download:
+        have_lock = my_lock.acquire(blocking=False)
-        download_handler = VideoDownloader(to_download)
+        if have_lock:
-        download_handler.download_list()
+            downloader = VideoDownloader()
+            downloader.add_pending()
+            downloader.run_queue()
+            downloader.validate_playlists()
+        else:
+            print("Did not acquire download lock.")
+
+    finally:
+        if have_lock:
+            my_lock.release()
 
 
 @shared_task
 def download_single(youtube_id):
     """start download single video now"""
-    download_handler = VideoDownloader([youtube_id])
+    queue = RedisQueue("dl_queue")
-    download_handler.download_list()
+    queue.add_priority(youtube_id)
+    print("Added to queue with priority: " + youtube_id)
+    # start queue if needed
+    have_lock = False
+    my_lock = RedisArchivist().get_lock("downloading")
+
+    try:
+        have_lock = my_lock.acquire(blocking=False)
+        if have_lock:
+            VideoDownloader().run_queue()
+        else:
+            print("Download queue already running.")
+
+    finally:
+        # release if only single run
+        if have_lock and not queue.get_next():
+            my_lock.release()
 
 
 @shared_task
@@ -58,7 +108,8 @@ def extrac_dl(youtube_ids):
     """parse list passed and add to pending"""
     pending_handler = PendingList()
     missing_videos = pending_handler.parse_url_list(youtube_ids)
-    pending_handler.add_to_pending(missing_videos)
+    all_videos_added = pending_handler.add_to_pending(missing_videos)
+    ThumbManager().download_vid(all_videos_added)
 
 
 @shared_task
@@ -70,17 +121,17 @@ def check_reindex():
 @shared_task
 def run_manual_import():
     """called from settings page, to go through import folder"""
-
     print("starting media file import")
     have_lock = False
-    my_lock = get_lock("manual_import")
+    my_lock = RedisArchivist().get_lock("manual_import")
 
     try:
         have_lock = my_lock.acquire(blocking=False)
         if have_lock:
             import_handler = ManualImport()
             if import_handler.identified:
-                import_handler.process_import()
+                all_videos_added = import_handler.process_import()
+                ThumbManager().download_vid(all_videos_added)
         else:
             print("Did not acquire lock form import.")
 
@@ -101,3 +152,112 @@ def run_restore_backup():
     """called from settings page, dump backup to zip file"""
     restore_from_backup()
     print("index restore finished")
+
+
+def kill_dl(task_id):
+    """kill download worker task by ID"""
+    if task_id:
+        app.control.revoke(task_id, terminate=True)
+
+    _ = RedisArchivist().del_message("dl_queue_id")
+    RedisQueue("dl_queue").clear()
+
+    # clear cache
+    cache_dir = os.path.join(CONFIG["application"]["cache_dir"], "download")
+    for cached in os.listdir(cache_dir):
+        to_delete = os.path.join(cache_dir, cached)
+        os.remove(to_delete)
+
+    # notify
+    mess_dict = {
+        "status": "downloading",
+        "level": "error",
+        "title": "Brutally killing download queue",
+        "message": "",
+    }
+    RedisArchivist().set_message("progress:download", mess_dict)
+
+
+@shared_task
+def rescan_filesystem():
+    """check the media folder for mismatches"""
+    scan_filesystem()
+    validate_thumbnails()
+
+
+@shared_task
+def re_sync_thumbs():
+    """sync thumbnails to mediafiles"""
+    handler = ThumbManager()
+    video_list = handler.get_thumb_list()
+    handler.write_all_thumbs(video_list)
+
+
+@shared_task
+def subscribe_to(url_str):
+    """take a list of urls to subscribe to"""
+    to_subscribe_list = UrlListParser(url_str).process_list()
+    for item in to_subscribe_list:
+        to_sub_id = item["url"]
+        if item["type"] == "playlist":
+            new_thumbs = PlaylistSubscription().process_url_str([item])
+            if new_thumbs:
+                ThumbManager().download_playlist(new_thumbs)
+            continue
+
+        if item["type"] == "video":
+            vid_details = PendingList().get_youtube_details(to_sub_id)
+            channel_id_sub = vid_details["channel_id"]
+        elif item["type"] == "channel":
+            channel_id_sub = to_sub_id
+        else:
+            raise ValueError("failed to subscribe to: " + to_sub_id)
+
+        ChannelSubscription().change_subscribe(
+            channel_id_sub, channel_subscribed=True
+        )
+        # notify
+        RedisArchivist().set_message(
+            "progress:subscribe", {"status": "subscribing"}
+        )
+
+
+@shared_task
+def index_channel_playlists(channel_id):
+    """add all playlists of channel to index"""
+    channel_handler = YoutubeChannel(channel_id)
+    all_playlists = channel_handler.get_all_playlists()
+
+    if not all_playlists:
+        print(f"no playlists found for channel {channel_id}")
+        return
+
+    all_indexed = PendingList().get_all_indexed()
+    all_youtube_ids = [i["youtube_id"] for i in all_indexed]
+
+    for playlist_id, playlist_title in all_playlists:
+        print("add playlist: " + playlist_title)
+        playlist_handler = YoutubePlaylist(
+            playlist_id, all_youtube_ids=all_youtube_ids
+        )
+        playlist_handler.get_playlist_dict()
+        if not playlist_handler.playlist_dict:
+            # skip if not available
+            continue
+        # don't add if no videos downloaded
+        downloaded = [
+            i
+            for i in playlist_handler.playlist_dict["playlist_entries"]
+            if i["downloaded"]
+        ]
+        if not downloaded:
+            continue
+        playlist_handler.upload_to_es()
+        playlist_handler.add_vids_to_playlist()
+
+    if all_playlists:
+        handler = ThumbManager()
+        missing_playlists = handler.get_missing_playlists()
+        handler.download_playlist(missing_playlists)
+
+    return

tubearchivist/home/templates/home/about.html

@@ -5,35 +5,13 @@
     <h1>About The Tube Archivist</h1>
 </div>
 <div class="about-section">
-    <h2>Getting started</h2>
+    <h2>Useful Links</h2>
-    <ol>
+    <p>This project is in active and constant development, take a look at the <a href="https://github.com/bbilly1/tubearchivist#roadmap" target="_blank">roadmap</a> for a overview.</p>
-        <li>Before adding some videos to the download queue, take a look at the download format settings on your <a href="{% url 'settings' %}#format">settings</a> page and set your desired download quality.</li>
+    <p>For any questions on what a button or a function does, You can find the up-to-date user documentation on <a href="https://github.com/bbilly1/tubearchivist/wiki" target="_blank">Github</a>.</p>
-        <li>While at the settings page also set the value for the Subscriptions <a href="{% url 'settings' %}#subscriptions">page size</a>. This will define the max amount of videos that will get added once you run <i>Rescan Subscriptions</i>. By default, yt-dlp uses a 50 video page size. Any value higher than that will significantly slow down the scanning process.</li>
+    <p>All contributions are welcome: Open an <a href="https://github.com/bbilly1/tubearchivist/issues" target="_blank">issue</a> for any bugs and errors, start a <a href="https://github.com/bbilly1/tubearchivist/discussions" target="_blank">discussion</a> for anything that will require a more indepth look. The <a href="https://github.com/bbilly1/tubearchivist/blob/master/CONTRIBUTING.md" target="_blank">contributing</a> page is a good place to get started.</p>
-        <li>Subscribe to some of your favorite YouTube channels in the <a href="{% url 'channel' %}">channel</a> page. To subscribe to a channel you have a few options:
-            <ul>
-                <li>Enter the YouTube channel ID, a 25 character alphanumeric string. For example <span class="settings-current">UCBa659QWEk1AI4Tg--mrJ2A</span></li>
-                <li>Enter the URL to the channel page on YouTube. For example <span class="settings-current">https://www.youtube.com/channel/UCBa659QWEk1AI4Tg--mrJ2A</span></li>
-                <li>Enter the video URL for any video and let <i>Tube Archivist</i> extract the channel ID for you. For example <span class="settings-current">https://www.youtube.com/watch?v=2tdiKTSdE9Y</span></li>
-            </ul>
-        </li>
-        <li>After that head over to the <a href="{% url 'downloads' %}">Downloads</a> Page. Once you click on the <i>Rescan Subscriptions</i> button, <b>Tube Archivist</b> will go through all your subscribed channels and look for new videos. Initially these are the latest videos depending on the page size as described above.</li>
-        <li>Click on <span class="settings-current">ignore</span> for any video you don't want to download or on <span class="settings-current">Download now</span> for any video you want to start downloading now independently from the queue.</li>
-        <li>Click on the <span class="settings-current">Download queue</span> button to start downloading. </b>Tube Archivist</b> will then start the queue from the bottom and download as many videos as defined in <a href="{% url 'settings' %}#downloads">download limit</a>. There is currently no good way to cancel the download process.</li>
-    </ol>
 </div>
 <div class="about-section">
-    <h2>Additional</h2>
+    <h2>Donate</h2>
-    <ul>
+    <p>Here are <a href="https://github.com/bbilly1/tubearchivist#donate" target="_blank">some links</a>, if you want to buy the developer a coffee. Thank you for your support!</p>
-        <li>You can manually add videos to the queue on the <a href="{% url 'downloads' %}">download page</a> by clicking on <i>Add to download queue</i> and then entering a list of videos. Add one link per line. You have a few options:
-            <ul>
-                <li>Add a link to a YouTube video. For example <span class="settings-current">https://www.youtube.com/watch?v=2tdiKTSdE9Y</span>.</li>
-                <li>Add a YouTube video ID. For example <span class="settings-current">2tdiKTSdE9Y</span>.</li>
-                <li>Add a link to a YouTube video by providing the shortened URL, for example <span class="settings-current">https://youtu.be/2tdiKTSdE9Y</span>.</li>
-                <li>Add a Channel ID or Channel URL to add every available video to the download queue. This will ignore the page size as described above. You can still ignore selected videos before starting the download.</li>
-                <li>Add a playlist ID or URL to add every available video in the list to the download queue, for example <span class="settings-current">https://www.youtube.com/playlist?list=PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha</span> or <span class="settings-current">PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha</span>. Note that when you add a link to a video in a playlist, <b>Tube Archivist</b> assumes you want to download only the specific video and not the whole playlist, for example <span class="settings-current">https://www.youtube.com/watch?v=CINVwWHlzTY&list=PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha</span> will only add one video <span class="settings-current">CINVwWHlzTY</span> to the queue.</li>
-            </ul>
-        </li>
-        <li>As you watch videos, mark them as watched by clicking on the unseen icon <span class="about-icon"><img src="{% static 'img/icon-unseen.svg' %}" alt="unseen-icon" class="unseen-icon"></span>. Once marked as watched, the icon will change to <span class="about-icon"><img src="{% static 'img/icon-seen.svg' %}" alt="seen-icon" class="seen-icon"></span>.</li>
-    </ul>
 </div>
 {% endblock content %}

tubearchivist/home/templates/home/base.html

@@ -39,16 +39,22 @@
                 <a href="{% url 'channel' %}">
                     <div class="nav-item">channels</div>
                 </a>
+                <a href="{% url 'playlist' %}">
+                    <div class="nav-item">playlists</div>
+                </a>
                 <a href="{% url 'downloads' %}">
                     <div class="nav-item">downloads</div>
                 </a>
             </div>
             <div class="nav-icons">
                 <a href="{% url 'about' %}">
-                    <img src="{% static 'img/icon-help.svg' %}" alt="help-icon">
+                    <img src="{% static 'img/icon-help.svg' %}" alt="help-icon" title="About">
                 </a>
                 <a href="{% url 'settings' %}">
-                    <img src="{% static 'img/icon-gear.svg' %}" alt="gear-icon">
+                    <img src="{% static 'img/icon-gear.svg' %}" alt="gear-icon" title="Settings">
+                </a>
+                <a href="{% url 'logout' %}">
+                    <img src="{% static 'img/icon-exit.svg' %}" alt="exit-icon" title="Logout">
                 </a>
             </div>
         </div>
@@ -96,7 +102,7 @@
     </div>
     <div class="footer">
         <div class="boxed-content">
-            <span>© 2021 The Tube Archivist v0.0.3 | <a href="https://github.com/bbilly1/tubearchivist" target="_blank">Github</a> | <a href="https://hub.docker.com/r/bbilly1/tubearchivist" target="_blank">Docker Hub</a></span>
+            <span>© 2021 The Tube Archivist v0.0.8 | <a href="https://github.com/bbilly1/tubearchivist" target="_blank">Github</a> | <a href="https://hub.docker.com/r/bbilly1/tubearchivist" target="_blank">Docker Hub</a></span>
         </div>
     </div>
 </body>

tubearchivist/home/templates/home/channel.html

@@ -7,50 +7,63 @@
 </div>
 <div class="info-box info-box-2">
     <div class="icon-text">
-        <img id="add-icon" onclick="showForm()" src="{% static 'img/icon-add.svg' %}" alt="add-icon">
+        {% if running == "subscribing" %}
-        <p>Subscribe to Channels</p>
+            <p>Subscribing in progress, refresh.</p>
-        <div class="show-form">
+        {% else %}
-            <form id="hidden-form" action="/channel/" method="post">
+            <img id="add-icon" onclick="showForm()" src="{% static 'img/icon-add.svg' %}" alt="add-icon">
-                {% csrf_token %}
+            <p>Subscribe to Channels</p>
-                <textarea rows="3" placeholder="Input channel ID, channel URL or Video of a channel" id="subscribe" name="subscribe"></textarea>
+            <div class="show-form">
-                <button type="submit">Subscribe</button>
+                <form id="hidden-form" action="/channel/" method="post">
-            </form>
+                    {% csrf_token %}
-        </div>
+                    {{ subscribe_form }}
+                    <button type="submit">Subscribe</button>
+                </form>
+            </div>
+        {% endif %}
     </div>
     <div class="search-form icon-text">
         <div class="search-icon">
             <img src="{% static 'img/icon-search.svg' %}" alt="search-icon" onclick="showSearch()">
-            <p>Search Channels</p>
+            <p>Search your Channels</p>
         </div>
         <form onSubmit="return channelRedirect();" id="search-box">
             {% csrf_token %}
-            <input name="videoSearch" list="resultBox" type="text" id="searchInput" autocomplete="off" oninput="searchChannels(this.value)">
+            {{ search_form }}
             <datalist id="resultBox">
             </datalist>
         </form>
     </div>
 </div>
-<div class="padding-box">
+<div class="view-controls">
-    <h2>Total matching channels: {{ max_hits }}</h2>
+    <div class="toggle">
-    <span>Change show / hide subscribed only </span><span class="settings-current">{{ show_subed_only }}</span>
+        <span>Show only subscribed channels:</span>
-    <select name="watched" id="watched" onchange="showSubscribedOnly(this.value)">
+        <div class="toggleBox">
-        <option value="" disabled selected> -- change -- </option>
+            <input 
-        <option value="0">show all channels</option>
+                id="show_subed_only" onclick="toggleCheckbox(this)" type="checkbox"
-        <option value="1">show subscribed channels only</option>
+                {% if show_subed_only %}
-    </select>
+                    checked
+                {% endif %}
+                >
+            <label for="" class="onbtn">On</label>
+            <label for="" class="ofbtn">Off</label>
+        </div>
+    </div>
+    <div class="view-icons">
+        <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="channel" data-value="grid" alt="grid view">
+        <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="channel" data-value="list" alt="list view">
+    </div>
 </div>
-<div>
+<h2>Total matching channels: {{ max_hits }}</h2>
+<div class="channel-list {{ view_style }}">
     {% if channels %}
         {% for channel in channels %}
-        <div class="channel-item">
+        <div class="channel-item {{ view_style }}">
-            {% if channel.source.channel_banner_url %}
+            <div class="channel-banner {{ view_style }}">
-                <div class="channel-banner">
+                <a href="{% url 'channel_id' channel.source.channel_id %}">
-                    <a href="{% url 'channel_id' channel.source.channel_id %}">
+                    <img src="/cache/channels/{{ channel.source.channel_id }}_banner.jpg" alt="{{ channel.source.channel_id }}-banner">
-                        <img src="/cache/channels/{{ channel.source.channel_id }}_banner.jpg" alt="{{ channel.source.channel_id }}-banner">
+                </a>
-                    </a>
+            </div>
-                </div>
+            <div class="info-box info-box-2 {{ view_style }}">
-            {% endif %}
-            <div class="info-box info-box-2">
                 <div class="info-box-item">
                     <div class="round-img">
                         <a href="{% url 'channel_id' channel.source.channel_id %}">
@@ -73,7 +86,7 @@
                         {% if channel.source.channel_subscribed %}
                             <button type="button" id="{{ channel.source.channel_id }}" onclick="unsubscribe(this.id)" title="Unsubscribe from {{ channel.source.channel_name }}">Unsubscribe</button>
                         {% else %}
-                            false
+                            <button type="button" id="{{ channel.source.channel_id }}" onclick="subscribe(this.id)" title="Subscribe to {{ channel.source.channel_name }}">Subscribe</button>
                         {% endif %}
                         </p>
                     </div>

tubearchivist/home/templates/home/channel_id.html

@@ -2,15 +2,10 @@
 {% block content %}
 {% load static %}
 {% load humanize %}
-<div class="title-bar">
-    <h1>Channel: {{ channel_info.channel_name }}</h1>
-</div>
 <div class="channel-banner">
-    {% if channel_info.channel_banner_url %}
+    <a href="/channel/{{ channel_info.channel_id }}/"><img src="/cache/channels/{{ channel_info.channel_id }}_banner.jpg" alt="channel_banner"></a>
-        <a href="/channel/{{ channel_info.channel_id }}/"><img src="/cache/channels/{{ channel_info.channel_id }}_banner.jpg" alt="channel_banner"></a>
-    {% endif %}
 </div>
-<div class="info-box info-box-3 padding-box">
+<div class="info-box info-box-3">
     <div class="info-box-item">
         <div class="round-img">
             <a href="{% url 'channel_id' channel_info.channel_id %}">
@@ -33,25 +28,35 @@
             {% if channel_info.channel_subscribed %}
                 <button type="button" id="{{ channel_info.channel_id }}" onclick="unsubscribe(this.id)" title="Unsubscribe from {{ channel_info.channel_name }}">Unsubscribe</button>
             {% else %}
-                false
+                <button type="button" id="{{ channel_info.channel_id }}" onclick="subscribe(this.id)" title="Subscribe to {{ channel_info.channel_name }}">Subscribe</button>
             {% endif %}
             </p>
+            {% if channel_info.channel_active %}
+                <p>Youtube: <a href="https://www.youtube.com/channel/{{ channel_info.channel_id }}" target="_blank">Active</a></p>
+            {% else %}
+                <p>Youtube: Deactivated</p>
+            {% endif %}
+            <button onclick="deleteConfirm()" id="delete-item">Delete Channel</button>
+            <div class="delete-confirm" id="delete-button">
+                <span>Delete {{ channel_info.channel_name }} including all videos? </span><button class="danger-button" onclick="deleteChannel(this)" data-id="{{ channel_info.channel_id }}">Delete</button> <button onclick="cancelDelete()">Cancel</button>
+            </div>
         </div>
     </div>
-    {% if videos %}
     <div class="info-box-item">
         <div>
             {% if channel_info.channel_views >= 1000000 %}
                 <p>Channel views: {{ channel_info.channel_views|intword }}</p>
-            {% else %}
+            {% elif channel_info.channel_views > 0 %}
                 <p>Channel views: {{ channel_info.channel_views|intcomma }}</p>
             {% endif %}
-            <p>Total Videos archived: {{ max_hits }}</p>
+            {% if max_hits %}
-            <p>Watched: <button title="Mark all videos from {{ channel_info.channel_name }} as watched" type="button" id="{{ channel_info.channel_id }}" onclick="isWatched(this.id)">Mark as watched</button></p>
+                <p>Total Videos archived: {{ max_hits }}</p>
-            <p>Channel id: {{ channel_info.channel_id }}</p>
+                <p>Watched: <button title="Mark all videos from {{ channel_info.channel_name }} as watched" type="button" id="watched-button" data-id="{{ channel_info.channel_id }}" onclick="isWatchedButton(this)">Mark as watched</button></p>
+            {% endif %}
+            <button title="Search for playlists on YouTube for {{ channel_info.channel_name }}" type="button" id="find-playlists-button" data-id="{{ channel_info.channel_id }}" onclick="findPlaylists(this)">Find Playlists</button>
+            <a href="/playlist/?search={{ channel_info.channel_id }}" title="Show all playlists belonging to {{ channel_info.channel_name }}"><button>Show Playlists</button></a>
         </div>
     </div>
-    {% endif %}
 </div>
 {% if channel_info.channel_description %}
     <div class="info-box-item description-box">
@@ -62,25 +67,66 @@
     </div>
 {% endif %}
 <div id="player" class="video-player"></div>
-<h2>Videos</h2>
+<div class="sort">
-<div class="video-list">
+    <p>Sort by <span class="settings-current">{{ sort_by }}</span>
+        <select name="sort" id="sort" onchange="sortChange(this.value)">
+            <option value="" disabled selected> -- change sort by -- </option>
+            <option value="published">date published</option>
+            <option value="downloaded">date downloaded</option>
+            <option value="views">views</option>
+            <option value="likes">likes</option>
+        </select>
+        <select name="sord-order" id="sort-order" onchange="sortChange(this.value)">
+            {% if sort_order == "asc" %}
+                <option value="asc" selected>asc</option>
+            {% else %}
+                <option value="asc">asc</option>
+            {% endif %}
+            {% if sort_order == "desc" %}
+                <option value="desc" selected>desc</option>
+            {% else %}
+                <option value="desc">desc</option>
+            {% endif %}
+        </select>
+    </p>
+</div>
+<div class="view-controls">
+    <div class="toggle">
+        <span>Hide watched videos:</span>
+        <div class="toggleBox">
+            <input 
+                id="hide_watched" onclick="toggleCheckbox(this)" type="checkbox"
+                {% if hide_watched %}
+                    checked
+                {% endif %}
+                >
+            <label for="" class="onbtn">On</label>
+            <label for="" class="ofbtn">Off</label>
+        </div>
+    </div>
+    <div class="view-icons">
+        <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="grid" alt="grid view">
+        <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="list" alt="list view">
+    </div>
+</div>
+<div class="video-list {{ view_style }}">
     {% if videos %}
         {% for video in videos %}
-            <div class="video-item">
+            <div class="video-item {{ view_style }}">
-                <a href="#player" data-src="/media/{{ video.source.media_url }}" data-thumb="/cache/videos/{{ video.source.youtube_id }}.jpg" data-title="{{ video.source.title }}" data-channel="{{ video.source.channel.channel_name }}" data-id="{{ video.source.youtube_id }}" onclick="createPlayer(this)">
+                <a href="#player" data-src="/media/{{ video.source.media_url }}" data-thumb="/cache/{{ video.source.vid_thumb_url }}" data-title="{{ video.source.title }}" data-channel="{{ video.source.channel.channel_name }}" data-id="{{ video.source.youtube_id }}" onclick="createPlayer(this)">
-                    <div class="video-thumb-wrap">
+                    <div class="video-thumb-wrap {{ view_style }}">
                         <div class="video-thumb">
-                            <img src="/cache/videos/{{ video.source.youtube_id }}.jpg" alt="video-thumb">
+                            <img src="/cache/{{ video.source.vid_thumb_url }}" alt="video-thumb">
                         </div>
                         <div class="video-play">
                             <img src="{% static 'img/icon-play.svg' %}" alt="play-icon">
                         </div>
                     </div>
                 </a>
-                <div class="video-desc">
+                <div class="video-desc {{ view_style }}">
                     <div class="video-desc-player" id="video-info-{{ video.source.youtube_id }}">
                         {% if video.source.player.watched %}
-                            <img src="{% static 'img/icon-seen.svg' %}" alt="seen-icon" id="{{ video.source.youtube_id }}" class="seen-icon">
+                            <img src="{% static 'img/icon-seen.svg' %}" alt="seen-icon" id="{{ video.source.youtube_id }}" onclick="isUnwatched(this.id)" class="seen-icon" title="Mark as unwatched">
                         {% else %}
                             <img src="{% static 'img/icon-unseen.svg' %}" alt="unseen-icon" id="{{ video.source.youtube_id }}" onclick="isWatched(this.id)" class="unseen-icon" title="Mark as watched.">
                         {% endif %}

tubearchivist/home/templates/home/downloads.html

@@ -5,14 +5,15 @@
     <h1>Downloads</h1>
 </div>
 <div id="downloadMessage"></div>
-<div class="info-box info-box-3 padding-box">
+<div id="downloadControl"></div>
+<div class="info-box info-box-3">
     <div class="icon-text">
         <img id="rescan-icon" onclick="rescanPending()" src="{% static 'img/icon-rescan.svg' %}" alt="rescan-icon">
         <p>Rescan subscriptions</p>
     </div>
     <div class="icon-text">
         <img id="download-icon" onclick="dlPending()" src="{% static 'img/icon-download.svg' %}" alt="download-icon">
-        <p>Download queue</p>
+        <p>Start download</p>
     </div>
     <div class="icon-text">
         <img id="add-icon" onclick="showForm()" src="{% static 'img/icon-add.svg' %}" alt="add-icon">
@@ -20,22 +21,54 @@
         <div class="show-form">
             <form id='hidden-form' action="/downloads/" method="post">
                 {% csrf_token %}
-                <textarea rows="4" placeholder="Enter Video Urls or IDs here..." id="vid-url" name="vid-url"></textarea>
+                {{ add_form }}
                 <button type="submit">Add to download queue</button>
             </form>
         </div>
     </div>
 </div>
-<h2>Download queue</h2>
+<div class="view-controls">
-<div>
+    <div class="toggle">
-    {% if pending %}
+        <span>Show only ignored videos:</span>
-        <h3>Total pending downloads: {{ max_hits }}</h3>
+        <div class="toggleBox">
-        {% for video in pending %}
+            <input 
-            <div class="dl-item" id="dl-{{ video.youtube_id }}">
+                id="show_ignored_only" onclick="toggleCheckbox(this)" type="checkbox"
-                <div class="dl-thumb">
+                {% if show_ignored_only %}
-                    <img src="{{ video.vid_thumb_url }}" alt="video_thumb">
+                    checked
+                {% endif %}
+                >
+            <label for="" class="onbtn">On</label>
+            <label for="" class="ofbtn">Off</label>
+        </div>
+    </div>
+    <div class="view-icons">
+        <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="downloads" data-value="grid" alt="grid view">
+        <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="downloads" data-value="list" alt="list view">
+    </div>
+</div>
+<div class="title-split">
+    {% if show_ignored_only %}
+        <h2>Ignored from download</h2>
+        <button onclick="deleteQueue(this)" data-id="ignore" title="Delete all previously ignored videos from the queue">Delete all ignored</button>
+    {% else %}
+        <h2>Download queue</h2>
+        <button onclick="deleteQueue(this)" data-id="pending" title="Delete all pending videos from the queue">Delete all queued</button>
+    {% endif %}
+</div>
+<h3>Total videos: {{ max_hits }}</h3>
+<div class="dl-list {{ view_style }}">
+    {% if all_video_hits %}
+        {% for video in all_video_hits %}
+            <div class="dl-item {{ view_style }}" id="dl-{{ video.youtube_id }}">
+                <div class="dl-thumb {{ view_style }}">
+                    <img src="/cache/{{ video.vid_thumb_url }}" alt="video_thumb">
+                    {% if show_ignored_only %}
+                        <span>ignored</span>
+                    {% else %}
+                        <span>queued</span>
+                    {% endif %}
                 </div>
-                <div class="dl-desc">
+                <div class="dl-desc {{ view_style }}">
                     <h3>{{ video.title }}</h3>
                     {% if video.channel_indexed %}
                         <a href="{% url 'channel_id' video.channel_id %}">{{ video.channel_name }}</a>
@@ -43,13 +76,16 @@
                         <span>{{ video.channel_name }}</span>
                     {% endif %}
                     <p>Published: {{ video.published }} | Duration: {{ video.duration }} | {{ video.youtube_id }}</p>
-                    <button data-id="{{ video.youtube_id }}" onclick="toIgnore(this)">Ignore</button>
+                    {% if show_ignored_only %}
-                    <button id="{{ video.youtube_id }}" data-id="{{ video.youtube_id }}" onclick="downloadNow(this)">Download now</button>
+                        <button data-id="{{ video.youtube_id }}" onclick="forgetIgnore(this)">Forget</button>
+                        <button data-id="{{ video.youtube_id }}" onclick="addSingle(this)">Add to queue</button>
+                    {% else %}
+                        <button data-id="{{ video.youtube_id }}" onclick="toIgnore(this)">Ignore</button>
+                        <button id="{{ video.youtube_id }}" data-id="{{ video.youtube_id }}" onclick="downloadNow(this)">Download now</button>
+                    {% endif %}
                 </div>
             </div>
         {% endfor %}
-    {% else %}
-        <h3>No pending downloads</h3>
     {% endif %}
 </div>
 <script type="text/javascript" src="{% static 'progress.js' %}"></script>

tubearchivist/home/templates/home/home.html

@@ -6,18 +6,25 @@
 </div>
 <div class="info-box info-box-2">
     <div class="sort">
-        <p>Sort order from <span class="settings-current">{{ sortorder }}</span>
+        <p>Sort by <span class="settings-current">{{ sort_by }}</span>
             <select name="sort" id="sort" onchange="sortChange(this.value)">
-                <option value="" disabled selected> -- change sort order -- </option>
+                <option value="" disabled selected> -- change sort by -- </option>
                 <option value="published">date published</option>
                 <option value="downloaded">date downloaded</option>
+                <option value="views">views</option>
+                <option value="likes">likes</option>
             </select>
-        </p>
+            <select name="sord-order" id="sort-order" onchange="sortChange(this.value)">
-        <p>Hide watched videos <span class="settings-current">{{ hide_watched }}</span>
+                {% if sort_order == "asc" %}
-            <select name="watched" id="watched" onchange="hideWatched(this.value)">
+                    <option value="asc" selected>asc</option>
-                <option value="" disabled selected> -- change hide watched -- </option>
+                {% else %}
-                <option value="0">show watched videos</option>
+                    <option value="asc">asc</option>
-                <option value="1">hide watched videos</option>
+                {% endif %}
+                {% if sort_order == "desc" %}
+                    <option value="desc" selected>desc</option>
+                {% else %}
+                    <option value="desc">desc</option>
+                {% endif %}
             </select>
         </p>
     </div>
@@ -27,29 +34,48 @@
         </div>
         <form action="/" method="POST" id="search-box">
             {% csrf_token %}
-            <input name="videoSearch" list="resultBox" type="text" id="searchInput" autocomplete="off">
+            {{ search_form }}
         </form>
     </div>
 </div>
+<div class="view-controls">
+    <div class="toggle">
+        <span>Hide watched videos:</span>
+        <div class="toggleBox">
+            <input 
+                id="hide_watched" onclick="toggleCheckbox(this)" type="checkbox"
+                {% if hide_watched %}
+                    checked
+                {% endif %}
+                >
+            <label for="" class="onbtn">On</label>
+            <label for="" class="ofbtn">Off</label>
+        </div>
+    </div>
+    <div class="view-icons">
+        <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="grid" alt="grid view">
+        <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="list" alt="list view">
+    </div>
+</div>
 <div id="player" class="video-player"></div>
-<div class="video-list">
+<div class="video-list {{ view_style }}">
     {% if videos %}
         {% for video in videos %}
-            <div class="video-item">
+            <div class="video-item {{ view_style }}">
-                <a href="#player" data-src="/media/{{ video.source.media_url }}" data-thumb="/cache/videos/{{ video.source.youtube_id }}.jpg" data-title="{{ video.source.title }}" data-channel="{{ video.source.channel.channel_name }}" data-id="{{ video.source.youtube_id }}" onclick="createPlayer(this)">
+                <a href="#player" data-src="/media/{{ video.source.media_url }}" data-thumb="/cache/{{ video.source.vid_thumb_url }}" data-title="{{ video.source.title }}" data-channel="{{ video.source.channel.channel_name }}" data-id="{{ video.source.youtube_id }}" onclick="createPlayer(this)">
-                    <div class="video-thumb-wrap">
+                    <div class="video-thumb-wrap {{ view_style }}">
                         <div class="video-thumb">
-                            <img src="/cache/videos/{{ video.source.youtube_id }}.jpg" alt="video-thumb">
+                            <img src="/cache/{{ video.source.vid_thumb_url }}" alt="video-thumb">
                         </div>
                         <div class="video-play">
                             <img src="{% static 'img/icon-play.svg' %}" alt="play-icon">
                         </div>
                     </div>
                 </a>
-                <div class="video-desc">
+                <div class="video-desc {{ view_style }}">
                     <div class="video-desc-player" id="video-info-{{ video.source.youtube_id }}">
                         {% if video.source.player.watched %}
-                            <img src="{% static 'img/icon-seen.svg' %}" alt="seen-icon" id="{{ video.source.youtube_id }}" class="seen-icon">
+                            <img src="{% static 'img/icon-seen.svg' %}" alt="seen-icon" id="{{ video.source.youtube_id }}" onclick="isUnwatched(this.id)" class="seen-icon" title="Mark as unwatched">
                         {% else %}
                             <img src="{% static 'img/icon-unseen.svg' %}" alt="unseen-icon" id="{{ video.source.youtube_id }}" onclick="isWatched(this.id)" class="unseen-icon" title="Mark as watched.">
                         {% endif %}

tubearchivist/home/templates/home/login.html

@@ -0,0 +1,46 @@
+{% load static %}
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta http-equiv="X-UA-Compatible" content="IE=edge">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>TA | Welcome</title>
+    <link rel="stylesheet" type="text/css" href="{% static 'css/style.css' %}">
+    <link rel="shortcut icon" type="image/png" href="{% static 'favicon.ico' %}"/>
+    {% if colors == "dark" %}
+        <link rel="stylesheet" href="{% static 'css/dark.css' %}">
+    {% else %}
+        <link rel="stylesheet" href="{% static 'css/light.css' %}">
+    {% endif %}
+</head>
+<body>
+    <div class="boxed-content login-page">
+        {% if colors == 'dark' %}
+            <img src="{% static 'img/logo-tube-archivist-dark.png' %}" alt="tube-archivist-logo">
+        {% endif %}
+        {% if colors == 'light' %}
+            <img src="{% static 'img/logo-tube-archivist-light.png' %}" alt="tube-archivist-banner">
+        {% endif %}
+        <h1>Tube Archivist</h1>
+        <h2>Your Self Hosted YouTube Media Server</h2>
+        {% if form_error %}
+            <p class="danger-zone">Failed to login.</p>
+        {% endif %}
+        <form action="/login/" method="POST" name="login">
+            {% csrf_token %}
+            {% for field in form %}
+                {{ field }}<br>
+            {% endfor %}
+            <input type="hidden" name="next" value="{{ request.GET.next }}" />
+            <button type="submit">Login</button>
+        </form>
+        <p class="login-links"><span><a href="https://github.com/bbilly1/tubearchivist" target="_blank">Github</a></span> <span><a href="https://github.com/bbilly1/tubearchivist#donate" target="_blank">Donate</a></span></p>
+    </div>
+    <div class="footer-colors">
+        <div class="col-1"></div>
+        <div class="col-2"></div>
+        <div class="col-3"></div>
+    </div>
+</body>
+</html>

tubearchivist/home/templates/home/playlist.html

@@ -0,0 +1,80 @@
+{% extends "home/base.html" %}
+{% load static %}
+{% block content %}
+<div class="title-bar">
+    <h1>Playlists</h1>
+</div>
+<div class="info-box info-box-2">
+    <div class="icon-text">
+        {% if running == "subscribing" %}
+            <p>Subscribing in progress, refresh.</p>
+        {% else %}
+            <img id="add-icon" onclick="showForm()" src="{% static 'img/icon-add.svg' %}" alt="add-icon">
+            <p>Subscribe to Playlist</p>
+            <div class="show-form">
+                <form id="hidden-form" action="/playlist/" method="post">
+                    {% csrf_token %}
+                    {{ subscribe_form }}
+                    <button type="submit">Subscribe</button>
+                </form>
+            </div>
+        {% endif %}
+    </div>
+    <div class="search-form icon-text">
+        <div class="search-icon">
+            <img src="{% static 'img/icon-search.svg' %}" alt="search-icon" onclick="showSearch()">
+            <p>Search your Playlists</p>
+        </div>
+        <form action="/playlist/" method="POST" id="search-box">
+            {% csrf_token %}
+            {{ search_form }}
+        </form>
+    </div>
+</div>
+<div class="view-controls">
+    <div class="toggle">
+        <span>Show only subscribed playlists:</span>
+        <div class="toggleBox">
+            <input 
+                id="show_subed_only" onclick="toggleCheckbox(this)" type="checkbox"
+                {% if show_subed_only %}
+                    checked
+                {% endif %}
+                >
+            <label for="" class="onbtn">On</label>
+            <label for="" class="ofbtn">Off</label>
+        </div>
+    </div>
+    <div class="view-icons">
+        <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="playlist" data-value="grid" alt="grid view">
+        <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="playlist" data-value="list" alt="list view">
+    </div>
+</div>
+<div class="playlist-list {{ view_style }}">
+    {% if playlists %}
+        {% for playlist in playlists %}
+        <div class="playlist-item {{ view_style }}">
+            <div class="playlist-thumbnail">
+                <a href="{% url 'playlist_id' playlist.source.playlist_id %}">
+                    <img src="/cache/playlists/{{ playlist.source.playlist_id }}.jpg" alt="{{ playlist.source.playlist_id }}-thumbnail">
+                </a>
+            </div>
+            <div class="playlist-desc {{ view_style }}">
+                <a href="{% url 'channel_id' playlist.source.playlist_channel_id %}"><h3>{{ playlist.source.playlist_channel }}</h3></a>
+                <a href="{% url 'playlist_id' playlist.source.playlist_id %}"><h2>{{ playlist.source.playlist_name }}</h2></a>
+                <p>Last refreshed: {{ playlist.source.playlist_last_refresh }}</p>
+                <p>Subscribed: 
+                    {% if playlist.source.playlist_subscribed %}
+                        <button type="button" id="{{ playlist.source.playlist_id }}" onclick="unsubscribe(this.id)" title="Unsubscribe from {{ playlist.source.playlist_name }}">Unsubscribe</button>
+                    {% else %}
+                        <button type="button" id="{{ playlist.source.playlist_id }}" onclick="subscribe(this.id)" title="Subscribe to {{ playlist.source.playlist_name }}">Subscribe</button>
+                    {% endif %}
+                </p>
+            </div>
+        </div>
+        {% endfor %}
+    {% else %}
+        <h2>No playlists found...</h2>
+    {% endif %}
+</div>
+{% endblock content %}

tubearchivist/home/templates/home/playlist_id.html

@@ -0,0 +1,114 @@
+{% extends "home/base.html" %}
+{% load static %}
+{% load humanize %}
+{% block content %}
+<div class="title-bar">
+    <h1>{{ playlist_info.playlist_name }}</h1>
+</div>
+<div class="info-box info-box-3">
+    <div class="info-box-item">
+        <div class="round-img">
+            <a href="{% url 'channel_id' channel_info.channel_id %}">
+                <img src="/cache/channels/{{ channel_info.channel_id }}_thumb.jpg" alt="channel-thumb">
+            </a>
+        </div>
+        <div>
+            <h3><a href="{% url 'channel_id' channel_info.channel_id %}">{{ channel_info.channel_name }}</a></h3>
+            {% if channel_info.channel_subs >= 1000000 %}
+                <span>Subscribers: {{ channel_info.channel_subs|intword }}</span>
+            {% else %}
+                <span>Subscribers: {{ channel_info.channel_subs|intcomma }}</span>
+            {% endif %}
+        </div>
+    </div>
+    <div class="info-box-item">
+        <div>
+            <p>Last refreshed: {{ playlist_info.playlist_last_refresh }}</p>
+            <p>Subscribed: 
+                {% if playlist_info.playlist_subscribed %}
+                    <button type="button" id="{{ playlist_info.playlist_id }}" onclick="unsubscribe(this.id)" title="Unsubscribe from {{ playlist_info.playlist_name }}">Unsubscribe</button>
+                {% else %}
+                    <button type="button" id="{{ playlist_info.playlist_id }}" onclick="subscribe(this.id)" title="Subscribe to {{ playlist_info.playlist_name }}">Subscribe</button>
+                {% endif %}
+            </p>
+            {% if playlist_info.playlist_active %}
+                <p>Youtube: <a href="https://www.youtube.com/playlist?list={{ playlist_info.playlist_id }}" target="_blank">Active</a></p>
+            {% else %}
+                <p>Youtube: Deactivated</p>
+            {% endif %}
+            <button onclick="deleteConfirm()" id="delete-item">Delete Playlist</button>
+            <div class="delete-confirm" id="delete-button">
+                <span>Delete {{ playlist_info.playlist_name }}?</span>
+                <button onclick="deletePlaylist(this)" data-action="metadata" data-id="{{ playlist_info.playlist_id }}">Delete metadata</button>
+                <button onclick="deletePlaylist(this)" data-action="all" class="danger-button" data-id="{{ playlist_info.playlist_id }}">Delete all</button><br>
+                <button onclick="cancelDelete()">Cancel</button>
+            </div>
+        </div>
+    </div>
+    <div class="info-box-item">
+        <div>
+            {% if max_hits %}
+            <p>Total Videos archived: {{ max_hits }}/{{ playlist_info.playlist_entries|length }}</p>
+            <p>Watched: <button title="Mark all videos from {{ playlist_info.playlist_name }} as watched" type="button" id="watched-button" data-id="{{ playlist_info.playlist_id }}" onclick="isWatchedButton(this)">Mark as watched</button></p>
+            {% endif %}
+        </div>
+    </div>
+</div>
+{% if playlist_info.playlist_description %}
+    <div class="info-box-item description-box">
+        <p>Description: <button onclick="textReveal()" id="text-reveal-button">Show</button></p>
+        <div id="text-reveal" class="description-text">
+            {{ playlist_info.playlist_description|linebreaks }}
+        </div>
+    </div>
+{% endif %}
+<div id="player" class="video-player"></div>
+<div class="view-controls">
+    <div class="toggle">
+        <span>Hide watched videos:</span>
+        <div class="toggleBox">
+            <input id="hide_watched" onclick="toggleCheckbox(this)" type="checkbox"
+            {% if hide_watched %}checked{% endif %}>
+            <label for="" class="onbtn">On</label>
+            <label for="" class="ofbtn">Off</label>
+        </div>
+    </div>
+    <div class="view-icons">
+        <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="grid" alt="grid view">
+        <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="list" alt="list view">
+    </div>
+</div>
+<div class="video-list {{ view_style }}">
+    {% if videos %}
+        {% for video in videos %}
+            <div class="video-item {{ view_style }}">
+                <a href="#player" data-src="/media/{{ video.source.media_url }}" data-thumb="/cache/{{ video.source.vid_thumb_url }}" data-title="{{ video.source.title }}" data-channel="{{ video.source.channel.channel_name }}" data-id="{{ video.source.youtube_id }}" onclick="createPlayer(this)">
+                    <div class="video-thumb-wrap {{ view_style }}">
+                        <div class="video-thumb">
+                            <img src="/cache/{{ video.source.vid_thumb_url }}" alt="video-thumb">
+                        </div>
+                        <div class="video-play">
+                            <img src="{% static 'img/icon-play.svg' %}" alt="play-icon">
+                        </div>
+                    </div>
+                </a>
+                <div class="video-desc {{ view_style }}">
+                    <div class="video-desc-player" id="video-info-{{ video.source.youtube_id }}">
+                        {% if video.source.player.watched %}
+                            <img src="{% static 'img/icon-seen.svg' %}" alt="seen-icon" id="{{ video.source.youtube_id }}" onclick="isUnwatched(this.id)" class="seen-icon" title="Mark as unwatched">
+                        {% else %}
+                            <img src="{% static 'img/icon-unseen.svg' %}" alt="unseen-icon" id="{{ video.source.youtube_id }}" onclick="isWatched(this.id)" class="unseen-icon" title="Mark as watched.">
+                        {% endif %}
+                        <span>{{ video.source.published }} | {{ video.source.player.duration_str }}</span>
+                    </div>
+                    <div>
+                        <a class="video-more" href="{% url 'video' video.source.youtube_id %}"><h2>{{ video.source.title }}</h2></a>
+                    </div>
+                </div>
+            </div>
+        {% endfor %}
+    {% else %}
+        <h2>No videos found...</h2>
+    {% endif %}
+</div>
+{% endblock content %}

tubearchivist/home/templates/home/settings.html

@@ -1,63 +1,39 @@
 {% extends "home/base.html" %}
 {% block content %}
 <div class="title-bar">
-    <h1>Settings</h1>
+    <h1>User Configurations</h1>
 </div>
-<form action="/settings/" method="POST" name="settings-update">
+<form action="/settings/" method="POST" name="user-update">
     {% csrf_token %}
     <div class="settings-group">
         <h2>Color scheme</h2>
         <div class="settings-item">
             <p>Current color scheme: <span class="settings-current">{{ config.application.colors }}</span></p>
             <i>Select your preferred color scheme between dark and light mode.</i><br>
-            <select name="application.colors" id="application.colors">
+            {{ user_form.colors }}
-                <option value="" disabled selected> -- change color scheme -- </option>
-                <option value="dark">dark mode</option>
-                <option value="light">light mode</option>
-            </select>
         </div>
     </div>
     <div class="settings-group">
         <h2>Archive View</h2>
-        <div class="settings-item">
-            <p>Current default Sort: <span class="settings-current">{{ config.archive.sort }}</span></p>
-            <i>Change how the home view and channels view sorts by defaults.</i><br>
-            <select name="archive.sort" id="archive.sort"">
-                <option value="" disabled selected> -- change sort order -- </option>
-                <option value="published">date published</option>
-                <option value="downloaded">date downloaded</option>
-            </select>
-        </div>
-        <div class="settings-item">
-            <p>Current default hide watched: <span class="settings-current">{{ config.archive.hide_watched }}</span></p>
-            <i>Show or hide watched videos by default.</i><br>
-            <select name="archive.hide_watched" id="archive.hide_watched"">
-                <option value="" disabled selected> -- change visibility -- </option>
-                <option value="0">show watched</option>
-                <option value="1">hide watched</option>
-            </select>
-        </div>
-        <div class="settings-item">
-            <p>Current default show subscribed channels only: <span class="settings-current">{{ config.archive.show_subed_only }}</span></p>
-            <i>Show subscribed channels only by default.</i><br>
-            <select name="archive.show_subed_only" id="archive.show_subed_only"">
-                <option value="" disabled selected> -- change visibility -- </option>
-                <option value="0">show subscribed only</option>
-                <option value="1">hide not subscribed</option>
-            </select>
-        </div>
         <div class="settings-item">
             <p>Current page size: <span class="settings-current">{{ config.archive.page_size }}</span></p>
             <i>Result of videos showing in archive page</i><br>
-            <input type="number" name="archive.page_size" id="archive.page_size">
+            {{ user_form.page_size }}
         </div>
     </div>
+    <button type="submit" name="user-settings">Update User Configurations</button>
+</form>
+<div class="title-bar">
+    <h1>Application Configurations</h1>
+</div>
+<form action="/settings/" method="POST" name="application-update">
+    {% csrf_token %}
     <div class="settings-group">
         <h2 id="subscriptions">Subscriptions</h2>
         <div class="settings-item">
-            <p>Current channel page size: <span class="settings-current">{{ config.subscriptions.channel_size }}</span></p>
+            <p>Current page size: <span class="settings-current">{{ config.subscriptions.channel_size }}</span></p>
-            <i>Recent videos to check on check pending, max recommended 50.</i><br>
+            <i>Recent videos for channels and playlist to check when running <b>Rescan subscriptions</b>, max recommended 50.</i><br>
-            <input type="number" name="subscriptions.channel_size" id="subscriptions.channel_size">
+            {{ app_form.subscriptions_channel_size }}
         </div>
         <div class="settings-item">
             <p>Auto scan subscribed channels:</p>
@@ -73,17 +49,22 @@
         <div class="settings-item">
             <p>Current download limit: <span class="settings-current">{{ config.downloads.limit_count }}</span></p>
             <i>Limit the number of videos getting downloaded on every run. 0 (zero) to deactivate.</i><br>
-            <input type="number" name="downloads.limit_count" id="downloads.limit_count">
+            {{ app_form.downloads_limit_count }}
         </div>
         <div class="settings-item">
-            <p>Current download speed limit: <span class="settings-current">{{ config.downloads.limit_speed }}</span></p>
+            <p>Current download speed limit in KB/s: <span class="settings-current">{{ config.downloads.limit_speed }}</span></p>
-            <i>Limit download speed. 0 (zero) to deactivate.</i><br>
+            <i>Limit download speed. 0 (zero) to deactivate, e.g. 1000 (1MB/s). Speeds are in KB/s.</i><br>
-            <input type="number" name="downloads.limit_speed" id="downloads.limit_speed"><span>KB/sec</span>
+            {{ app_form.downloads_limit_speed }}
+        </div>
+        <div class="settings-item">
+            <p>Current throttled rate limit in KB/s: <span class="settings-current">{{ config.downloads.throttledratelimit }}</span></p>
+            <i>Download will restart if speeds drop below specified amount. 0 (zero) to deactivate, e.g. 100. Speeds are in KB/s.</i><br>
+            {{ app_form.downloads_throttledratelimit }}
         </div>
         <div class="settings-item">
             <p>Current scraping sleep interval: <span class="settings-current">{{ config.downloads.sleep_interval }}</p>
             <i>Seconds to sleep between calls to YouTube. Might be necessary to avoid throttling. Recommended 3.</i><br>
-            <input type="number" name="downloads.sleep_interval" id="downloads.sleep_interval">
+            {{ app_form.downloads_sleep_interval }}
         </div>
         <div class="settings-item">
             <p>External downloader:</p>
@@ -100,34 +81,43 @@
             <ul>
                 <li><span class="settings-current">bestvideo[height<=720]+bestaudio/best[height<=720]</span>: best audio and max video height of 720p.</li>
                 <li><span class="settings-current">bestvideo[height<=1080]+bestaudio/best[height<=1080]</span>: best audio and max video height of 1080p.</li>
+                <li><span class="settings-current">bestvideo[height<=1080][VCODEC=avc1]+bestaudio[ACODEC=mp4a]/mp4</span>: Max 1080p video height with iOS compatible video and audio codecs.</li>
                 <li><span class="settings-current">0</span>: deactivate and download the best quality possible as decided by yt-dlp.</li>
             </ul>
             <i>Make sure your custom format gets merged into a single file. Check out the <a href="https://github.com/yt-dlp/yt-dlp#format-selection" target="_blank">documentation</a> for valid configurations.</i><br>
-            <input type="text" name="downloads.format" id="downloads.format">
+            {{ app_form.downloads_format }}
             <br>
         </div>
         <div class="settings-item">
             <p>Current metadata embed setting: <span class="settings-current">{{ config.downloads.add_metadata }}</span></p>
             <i>Metadata is not embedded into the downloaded files by default.</i><br>
-            <select name="downloads.add_metadata" id="downloads.add_metadata"">
+            {{ app_form.downloads_add_metadata }}
-                <option value="" disabled selected> -- change metadata embed -- </option>
+        </div>
-                <option value="0">don't embed metadata</option>
+        <div class="settings-item">
-                <option value="1">embed metadata</option>
+            <p>Current thumbnail embed setting: <span class="settings-current">{{ config.downloads.add_thumbnail }}</span></p>
-            </select>
+            <i>Embed thumbnail into the mediafile.</i><br>
+            {{ app_form.downloads_add_thumbnail }}
         </div>
     </div>
-    <button type="submit">Update Settings</button>
+    <button type="submit" name="application-settings">Update Application Configurations</button>
 </form>
 <div class="title-bar">
     <h1>Actions</h1>
 </div>
 <div class="settings-group">
     <h2>Manual media files import.</h2>
-    <p>Add files to the <span class="settings-current">cache/import</span> folder. Make sure to follow the instructions on <a href="https://github.com/bbilly1/tubearchivist#import-your-existing-library" target="_blank">Github</a>.</p>
+    <p>Add files to the <span class="settings-current">cache/import</span> folder. Make sure to follow the instructions in the Github <a href="https://github.com/bbilly1/tubearchivist/wiki/Settings" target="_blank">Wiki</a>.</p>
     <div id="manual-import">
         <button onclick="manualImport()">Start import</button>
     </div>
 </div>
+<div class="settings-group">
+    <h2>Embed thumbnails into media file.</h2>
+    <p>Set extracted youtube thumbnail as cover art of the media file.</p>
+    <div id="re-embed">
+        <button onclick="reEmbed()">Start process</button>
+    </div>
+</div>
 <div class="settings-group">
     <h2>Backup database</h2>
     <p>Export your database to a zip file stored at <span class="settings-current">cache/backup</span>.</p>
@@ -144,7 +134,21 @@
     </div>
 </div>
 <div class="settings-group">
-    <p>Rescan filesystem.</p>
+    <h2>Rescan filesystem</h2>
-    <i>Coming soon</i>
+    <p><span class="danger-zone">Danger Zone</span>: This will delete the metadata of deleted videos from the filesystem.</p>
+    <p>Rescan your media folder looking for missing videos and clean up index. More infos on the Github <a href="https://github.com/bbilly1/tubearchivist/wiki/Settings" target="_blank">Wiki</a>.</p>
+    <div id="fs-rescan">
+        <button onclick="fsRescan()">Rescan filesystem</button>
+    </div>
 </div>
+{% if request.user.is_superuser %}
+    <div class="title-bar">
+        <h1>Users</h1>
+    </div>
+    <div class="settings-group">
+        <h2>User Management</h2>
+        <p>Access the admin interface for basic user management functionality like adding and deleting users, changing passwords and more.</p>
+        <a href="/admin/"><button>Admin Interface</button></a>
+    </div>
+{% endif %}
 {% endblock content %}

tubearchivist/home/templates/home/video.html

@@ -2,17 +2,15 @@
 {% block content %}
 {% load static %}
 {% load humanize %}
-<div class="title-bar">
-    <h1>Video: {{ video.title }}</h1>
-</div>
 <div class="video-main">
     <video 
         src="/media/{{ video.media_url }}" 
-        poster="/cache/videos/{{ video.youtube_id }}.jpg" controls preload="false" 
+        poster="/cache/{{ video.vid_thumb_url }}" controls preload="false" 
-        type='video/mp4' width="100%">
+        type='video/mp4' width="100%" playsinline>
     </video>
+    <h1>{{ video.title }}</h1>
 </div>
-<div class="info-box info-box-3 padding-box">
+<div class="info-box info-box-3">
     <div class="info-box-item">
         <div class="round-img">
             <a href="{% url 'channel_id' video.channel.channel_id %}">
@@ -34,17 +32,20 @@
             <p>Last refreshed: {{ video.vid_last_refresh }}</p>
             <p class="video-info-watched">Watched:
                 {% if video.player.watched %}
-                    <img src="{% static 'img/icon-seen.svg' %}" alt="seen-icon" class="seen-icon" id="{{ video.youtube_id }}">
+                    <img src="{% static 'img/icon-seen.svg' %}" alt="seen-icon" id="{{ video.youtube_id }}" onclick="isUnwatched(this.id)" class="seen-icon" title="Mark as unwatched">
                 {% else %}
                     <img src="{% static 'img/icon-unseen.svg' %}" alt="unseen-icon" id="{{ video.youtube_id }}" onclick="isWatched(this.id)" class="unseen-icon" title="Mark as watched.">
                 {% endif %}
             </p>
             {% if video.active %}
-                <p>Youtube: Active</p>
+                <p>Youtube: <a href="https://www.youtube.com/watch?v={{ video.youtube_id }}" target="_blank">Active</a></p>
             {% else %}
                 <p>Youtube: Deactivated</p>
             {% endif %}
-            <p>Video ID: {{ video.youtube_id }}</p>
+            <button onclick="deleteConfirm()" id="delete-item">Delete Video</button>
+            <div class="delete-confirm" id="delete-button">
+                <span>Are you sure? </span><button class="danger-button" onclick="deleteVideo(this)" data-id="{{ video.youtube_id }}" data-redirect = "{{ video.channel.channel_id }}">Delete</button> <button onclick="cancelDelete()">Cancel</button>
+            </div>
         </div>
     </div>
     <div class="info-box-item">
@@ -52,7 +53,13 @@
             <p>Views: {{ video.stats.view_count|intcomma }}</p>
             <p class="thumb-icon like"><img src="{% static 'img/icon-thumb.svg' %}" alt="thumbs-up">: {{ video.stats.like_count|intcomma }}</p>
             <p class="thumb-icon dislike"><img src="{% static 'img/icon-thumb.svg' %}" alt="thumbs-down">: {{ video.stats.dislike_count|intcomma }}</p>
-            <p>Rating: {{ video.stats.average_rating }}</p>
+            {% if video.stats.average_rating %}
+                <p class="rating-stars">Rating: 
+                    {% for star in video.stats.average_rating %}
+                        <img src="/static/img/icon-star-{{ star }}.svg" alt="{{ star }}">
+                    {% endfor %}
+                </p>
+            {% endif %}
         </div>
     </div>
 </div>
@@ -64,4 +71,41 @@
         </div>
     </div>
 {% endif %}
+{% if playlist_nav %}
+    {% for playlist_item in playlist_nav %}
+        <div class="playlist-wrap">
+            <a href="{% url 'playlist_id' playlist_item.playlist_meta.playlist_id %}">
+                <h3>Playlist [{{ playlist_item.playlist_meta.current_idx|add:"1" }}]: {{ playlist_item.playlist_meta.playlist_name }}</h3>
+            </a>
+            <div class="playlist-nav">    
+                <div class="playlist-nav-item">
+                    {% if playlist_item.playlist_previous %}
+                        <a href="{% url 'video' playlist_item.playlist_previous.youtube_id %}">
+                            <img src="/cache/{{ playlist_item.playlist_previous.vid_thumb }}" alt="previous thumbnail">
+                        </a>
+                        <div class="playlist-desc">
+                            <p>Previous:</p>
+                            <a href="{% url 'video' playlist_item.playlist_previous.youtube_id %}">
+                                <h3>[{{ playlist_item.playlist_previous.idx|add:"1" }}] {{ playlist_item.playlist_previous.title }}</h3>
+                            </a>
+                        </div>
+                    {% endif %}
+                </div>
+                <div class="playlist-nav-item">
+                    {% if playlist_item.playlist_next %}
+                        <div class="playlist-desc">
+                            <p>Next:</p>
+                            <a href="{% url 'video' playlist_item.playlist_next.youtube_id %}">
+                                <h3>[{{ playlist_item.playlist_next.idx|add:"1" }}] {{ playlist_item.playlist_next.title }}</h3>
+                            </a>
+                        </div>
+                        <a href="{% url 'video' playlist_item.playlist_next.youtube_id %}">
+                            <img src="/cache/{{ playlist_item.playlist_next.vid_thumb }}" alt="previous thumbnail">
+                        </a>
+                    {% endif %}
+                </div>
+            </div>
+        </div>
+    {% endfor %}
+{% endif %}
 {% endblock content %}

tubearchivist/home/tests.py

@@ -1,3 +0,0 @@
-from django.test import TestCase  # noqa: F401 - Unused import
-
-# Create your tests here.

tubearchivist/home/tests/test_views.py

@@ -0,0 +1,42 @@
+"""All test classes"""
+
+from django.test import TestCase
+
+
+class URLTests(TestCase):
+    """test if all expected URL are there"""
+
+    def test_home_view(self):
+        """check homepage"""
+        response = self.client.get("/")
+        self.assertEqual(response.status_code, 200)
+
+    def test_about_view(self):
+        """check about page"""
+        response = self.client.get("/about/")
+        self.assertEqual(response.status_code, 200)
+
+    def test_downloads_view(self):
+        """check downloads page"""
+        response = self.client.get("/downloads/")
+        self.assertEqual(response.status_code, 200)
+
+    def test_channel_view(self):
+        """check channel page"""
+        response = self.client.get("/channel/")
+        self.assertEqual(response.status_code, 200)
+
+    def test_settings_view(self):
+        """check settings page"""
+        response = self.client.get("/settings/")
+        self.assertEqual(response.status_code, 200)
+
+    def test_progress_view(self):
+        """check ajax progress endpoint"""
+        response = self.client.get("/downloads/progress/")
+        self.assertEqual(response.status_code, 200)
+
+    def test_process_view(self):
+        """check process ajax endpoint"""
+        response = self.client.get("/process/")
+        self.assertEqual(response.status_code, 200)

tubearchivist/home/urls.py

@@ -1,5 +1,8 @@
 """ all home app urls """
 
+from django.conf import settings
+from django.contrib.auth.decorators import login_required
+from django.contrib.auth.views import LogoutView
 from django.urls import path
 from home.views import (
     AboutView,
@@ -7,24 +10,46 @@ from home.views import (
     ChannelView,
     DownloadView,
     HomeView,
+    LoginView,
+    PlaylistIdView,
+    PlaylistView,
     SettingsView,
     VideoView,
+    process,
+    progress,
 )
 
-from . import views
-
 urlpatterns = [
-    path("", HomeView.as_view(), name="home"),
+    path("", login_required(HomeView.as_view()), name="home"),
+    path("login/", LoginView.as_view(), name="login"),
+    path(
+        "logout/",
+        LogoutView.as_view(),
+        {"next_page": settings.LOGOUT_REDIRECT_URL},
+        name="logout",
+    ),
     path("about/", AboutView.as_view(), name="about"),
-    path("downloads/", DownloadView.as_view(), name="downloads"),
+    path(
-    path("settings/", SettingsView.as_view(), name="settings"),
+        "downloads/", login_required(DownloadView.as_view()), name="downloads"
-    path("process/", views.process, name="process"),
+    ),
-    path("downloads/progress", views.progress, name="progress"),
+    path("settings/", login_required(SettingsView.as_view()), name="settings"),
-    path("channel/", ChannelView.as_view(), name="channel"),
+    path("process/", login_required(process), name="process"),
+    path("downloads/progress/", login_required(progress), name="progress"),
+    path("channel/", login_required(ChannelView.as_view()), name="channel"),
     path(
         "channel/<slug:channel_id_detail>/",
-        ChannelIdView.as_view(),
+        login_required(ChannelIdView.as_view()),
         name="channel_id",
     ),
-    path("video/<slug:video_id>/", VideoView.as_view(), name="video"),
+    path(
+        "video/<slug:video_id>/",
+        login_required(VideoView.as_view()),
+        name="video",
+    ),
+    path("playlist/", login_required(PlaylistView.as_view()), name="playlist"),
+    path(
+        "playlist/<slug:playlist_id_detail>/",
+        login_required(PlaylistIdView.as_view()),
+        name="playlist_id",
+    ),
 ]

tubearchivist/requirements.txt

@@ -1,9 +1,9 @@
 beautifulsoup4==4.10.0
-celery==5.1.2
+celery==5.2.1
-Django==3.2.7
+Django==3.2.9
-Pillow==8.3.2
+Pillow==8.4.0
 redis==3.5.3
 requests==2.26.0
-uWSGI==2.0.19.1
+uWSGI==2.0.20
 whitenoise==5.3.0
-yt_dlp==2021.9.2
+yt_dlp==2021.11.10

tubearchivist/static/css/dark.css

@@ -2,9 +2,11 @@
     --main-bg: #00202f;
     --highlight-bg: #00293b;
     --highlight-error: #990202;
+    --highlight-error-light: #c44343;
     --highlight-bg-transparent: #00293baf;
     --main-font: #eeeeee;
     --accent-font-dark: #259485;
     --accent-font-light: #97d4c8;
     --img-filter: invert(50%) sepia(9%) saturate(2940%) hue-rotate(122deg) brightness(94%) contrast(90%);
+    --img-filter-error: invert(16%) sepia(60%) saturate(3717%) hue-rotate(349deg) brightness(86%) contrast(120%);
 }

tubearchivist/static/css/light.css

@@ -1,10 +1,12 @@
 :root {
     --main-bg: #eeeeee;
     --highlight-bg: #d9e0d9;
-    --highlight-error: #f79c9c;
+    --highlight-error: #990202;
+    --highlight-error-light: #c44343;
     --highlight-bg-transparent: #00293baf;
     --main-font: #00202f;
     --accent-font-dark: #259485;
     --accent-font-light: #35b399;
     --img-filter: invert(50%) sepia(9%) saturate(2940%) hue-rotate(122deg) brightness(94%) contrast(90%);
+    --img-filter-error: invert(16%) sepia(60%) saturate(3717%) hue-rotate(349deg) brightness(86%) contrast(120%);
 }

tubearchivist/static/css/style.css

@@ -1,12 +1,12 @@
 @font-face {
 font-family: 'Sen-Bold';
-    src:  url('../font/Sen-Bold.ttf.woff');
+    src:  url('../font/Sen-Bold.woff');
     font-family: 'Sen-Bold';
 }
 
 @font-face {
 font-family: 'Sen-Regular';
-    src:  url('../font/Sen-Regular.ttf.woff');
+    src:  url('../font/Sen-Regular.woff');
     font-family: 'Sen-Regular';
 }
 
@@ -62,7 +62,7 @@ ul {
     margin-left: 20px;
 }
 
-td, span {
+td, span, label {
     font-family: Sen-Regular, sans-serif;
     color: var(--main-font);
 }
@@ -118,7 +118,6 @@ button:hover {
 
 .top-banner {
     text-align: center;
-    margin-top: 10px;
 }
 
 .top-banner img {
@@ -138,10 +137,94 @@ button:hover {
     text-decoration: underline;
 }
 
+/* toggle on-off */
+.toggle {
+    display: flex;
+    align-items: center;
+}
+
+.toggleBox > input[type="checkbox"] {
+    position: relative;
+    width: 70px;
+    height: 30px;
+    background-color: var(--accent-font-dark);
+    border-color: var(--accent-font-dark);
+    appearance: none;
+    border-radius: 15px;
+    transition: 0.4s;
+    box-shadow: inset 0 0 5px rgba(0, 0, 0, 0.2);
+    cursor: pointer;
+}
+
+.toggleBox > input:checked[type="checkbox"] {
+    background-color: var(--accent-font-light);
+    border-color: var(--accent-font-light);
+}
+
+.toggleBox > input[type="checkbox"]::before {
+    z-index: 2;
+    position: absolute;
+    content: "";
+    left: 0;
+    top: 0;
+    width: 30px;
+    height: 30px;
+    background-color: white;
+    border-radius: 50%;
+    transform: scale(1.1);
+    transition: 0.4s;
+}
+
+.toggleBox > input:checked[type="checkbox"]::before {
+    left: 40px;
+}
+
+.toggleBox {
+    margin-left: 10px;
+    position: relative;
+    display: inline;
+}
+
+.toggleBox > label {
+    position: absolute;
+    color: var(--main-font);
+    pointer-events: none;
+}
+
+.toggleBox > .onbtn {
+    bottom: 15px;
+    left: 15px;
+    font-family: Sen-Regular, sans-serif;
+}
+
+.toggleBox > .ofbtn {
+    bottom: 15px;
+    right: 15px;
+    font-family: Sen-Regular, sans-serif;
+    color: var(--main-font);
+}
+
+/* delete button */
+.delete-confirm {
+    display: none;
+}
+
+.delete-confirm button {
+    margin: 3px 0;
+}
+
+.danger-button {
+    background-color: var(--highlight-error);
+}
+
+.danger-button:hover {
+    background-color: var(--highlight-error-light);
+}
+
 /* navigation */
 .top-nav {
     display: block;
-    padding: 15px 0;
+    padding: 5px 0;
     position: relative;
 }
 
@@ -170,13 +253,13 @@ button:hover {
 
 .nav-icons img {
     width: 40px;
-    padding: 0 10px;
+    padding: 0 5px;
     filter: var(--img-filter);
 }
 
 /* top of page */
 .title-bar {
-    padding: 25px 0;
+    padding: 20px 0;
 }
 
 .sort {
@@ -211,6 +294,26 @@ button:hover {
     filter: var(--img-filter);
 }
 
+.view-controls {
+    display: grid;
+    grid-template-columns: 1fr 1fr;
+    border-bottom: 2px solid;
+    border-color: var(--accent-font-dark);
+    margin: 15px 0;
+}
+
+.view-icons {
+    display: flex;
+    justify-content: end;
+}
+
+.view-icons img {
+    width: 30px;
+    margin: 5px 10px;
+    cursor: pointer;
+    filter: var(--img-filter);
+}
+
 #search-box {
     display: none;
     flex: auto;
@@ -261,16 +364,27 @@ button:hover {
 
 
 /* video list */
-.video-list {
+.video-list.grid {
     display: grid;
     grid-template-columns: 1fr 1fr 1fr;
     grid-gap: 1rem;
 }
 
+.video-list.list {
+    display: grid;
+    grid-template-columns: unset;
+    grid-gap: 1rem;
+}
+
 .video-item {
     overflow: hidden;
 }
 
+.video-item.list {
+    display: grid;
+    grid-template-columns: 25% auto;
+}
+
 .video-thumb img {
     width: 100%;
 }
@@ -301,12 +415,25 @@ button:hover {
     padding: 8px;
 }
 
-.video-desc {
+.video-desc.grid {
     padding: 10px;
     height: 100%;
     background-color: var(--highlight-bg);
 }
 
+.video-desc.list {
+    padding: 10px;
+    height: unset;
+    background-color: var(--highlight-bg);
+    display: flex;
+    flex-wrap: wrap-reverse;
+    align-content: center;
+}
+
+.video-desc > div {
+    width: 100%;
+}
+
 .video-desc img {
     width: 20px;
     margin-right: 10px;
@@ -329,15 +456,12 @@ button:hover {
 }
 
 .unseen-icon,
+.seen-icon,
 .close-button {
     cursor: pointer;
     filter: var(--img-filter);
 }
 
-.seen-icon {
-    filter: var(--img-filter);
-}
-
 .video-more {
     text-decoration: underline;
     text-align: right;
@@ -364,6 +488,10 @@ button:hover {
     grid-gap: 1rem;
 }
 
+.description-box {
+    margin-top: 1rem;
+}
+
 .info-box-3 {
     grid-template-columns: 1fr 1fr 1fr;
 }
@@ -392,10 +520,79 @@ button:hover {
     margin-bottom: 10px;
 }
 
+/* login */
+.login-page {
+    display: flex;
+    flex-wrap: wrap;
+    justify-content: center;
+    text-align: center;
+    align-content: center;
+}
+
+.login-page > * {
+    width: 100%;
+}
+
+.login-page img {
+    width: 100%;
+    max-width: 200px;
+    max-height: 200px;
+    margin-bottom: 40px;
+}
+
+.login-page form {
+    margin: 30px 0;
+}
+
+.login-page input {
+    min-width: 200px;
+}
+
+.login-page button,
+.login-page .danger-zone {
+    width: 210px;
+    margin-top: 5px;
+}
+
+.login-links a {
+    text-decoration: underline;
+    margin: 30px 0;
+    padding: 20px;
+}
+
+.footer-colors {
+    grid-row-start: 2;
+    grid-row-end: 3;
+    display: flex;
+}
+
+.footer-colors div {
+    padding: 20px 0;
+    width: 33.33%;
+}
+
+.col-1 {
+    background-color: var(--highlight-bg);
+}
+
+.col-2 {
+    background-color: var(--accent-font-dark);
+}
+
+.col-3 {
+    background-color: var(--accent-font-light);
+}
+
 /* video page */
+.video-main {
+    margin: 1rem 0;
+}
+
 .video-main video {
     max-height: 70vh;
+    margin-bottom: 1rem;
 }
+
 .video-info-watched {
     display: flex;
     align-items: center;
@@ -406,7 +603,8 @@ button:hover {
     margin-left: 5px;
 }
 
-.thumb-icon img {
+.thumb-icon img,
+.rating-stars img {
     width: 20px;
     margin: 0;
     filter: var(--img-filter);
@@ -416,15 +614,117 @@ button:hover {
     transform: rotate(180deg);
 }
 
-/* channel overview page */
+.playlist-wrap {
-.channel-item {
+    background-color: var(--highlight-bg);
-    padding: 20px 0;
+    margin: 1rem 0;
+    padding: 1rem;
 }
 
-.channel-banner img {
+.playlist-wrap > a > h3 {
+    text-align: center;
+}
+
+.playlist-nav {
+    display: grid;
+    grid-template-columns: 1fr 1fr;
+    margin-bottom: 10px;
+}
+
+.playlist-nav-item {
+    display: flex;
+    justify-content: space-between;
+}
+
+.playlist-nav-item img {
+    width: 200px;
+}
+
+.playlist-desc {
+    padding: 5px;
     width: 100%;
 }
 
+/* channel overview page */
+.channel-list.list {
+    display: block;
+}
+
+.channel-list.grid {
+    display: grid;
+    grid-template-columns: 1fr 1fr 1fr;
+    gap: 1rem;
+}
+
+.channel-item.list {
+    padding-bottom: 1rem;
+}
+
+.channel-item.grid > .info-box {
+    display: block;
+}
+
+.channel-banner img {
+    margin-top: 1rem;
+    width: 100%;
+}
+
+.channel-banner.grid {
+    overflow: hidden;
+}
+
+.channel-banner.list img {
+    width: 100%;
+}
+
+.channel-banner.grid img {
+    width: 250%;
+    transform: translateX(-30%);
+}
+
+/* playlist overview page */
+.playlist-list.list {
+    display: grid;
+    grid-template-columns: unset;
+    gap: 1rem;
+}
+
+.playlist-list.grid {
+    display: grid;
+    grid-template-columns: 1fr 1fr 1fr;
+    gap: 1rem;
+}
+
+.playlist-item {
+    overflow: hidden;
+}
+
+.playlist-item.list {
+    display: flex;
+}
+
+.playlist-thumbnail img {
+    width: 100%;
+}
+
+.playlist-desc.grid {
+    padding: 10px;
+    height: 100%;
+    background-color: var(--highlight-bg);
+}
+
+.playlist-desc.list {
+    width: 100%;
+    padding: 10px;
+    height: unset;
+    background-color: var(--highlight-bg);
+    display: flex;
+    flex-wrap: wrap;
+    align-content: center;
+}
+
+.playlist-desc.list > * {
+    width: 100%;
+}
 
 /* download page */
 .icon-text {
@@ -438,36 +738,104 @@ button:hover {
     cursor: pointer;
 }
 
-.dl-item {
+.dl-list.list {
+    display: block;
+}
+
+.dl-list.grid {
+    display: grid;
+    grid-template-columns: 1fr 1fr 1fr;
+    grid-gap: 1rem;
+}
+
+.dl-item.list {
     display: flex;
     margin: 15px 0;
     align-items: center;
     background-color: var(--highlight-bg);
 }
 
+.dl-item.grid {
+    display: flex;
+    flex-wrap: wrap;
+    background-color: var(--highlight-bg);
+}
+
 .dl-check {
     width: 30px;
 }
 
+.dl-item:hover > .dl-thumb span {
+    opacity: 1;
+}
+
 .dl-thumb {
+    position: relative;
+}
+
+.dl-thumb span {
+    position: absolute;
+    top: 5px;
+    left: 5px;
+    background-color: var(--accent-font-light);
+    padding: 5px;
+    opacity: 0;
+    transition: 300ms ease-in-out;
+}
+
+.dl-thumb.list {
     width: 25%;
 }
 
+.dl-thumb.grid {
+    width: 100%;
+}
+
 .dl-item img {
     width: 100%;
 }
 
-.dl-desc {
+.dl-desc.list {
-    padding-left: 15px;
+    padding: 0 15px;
     width: 75%;
 }
 
+.dl-desc.grid {
+    padding: 15px;
+    width: 100%;
+}
+
+.dl-control-icons {
+    display: flex;
+    justify-content: center;
+    padding: 10px 0;
+}
+
+.dl-control-icons img {
+    width: 30px;
+    cursor: pointer;
+    margin: 5px;
+}
+
+#stop-icon {
+    filter: var(--img-filter);
+}
+
+#kill-icon {
+    filter: var(--img-filter-error);
+}
+
+.title-split {
+    display: flex;
+    justify-content: space-between;
+}
 
 /* status message */
 .download-progress {
     background-color: var(--highlight-bg);
     text-align: center;
     padding: 30px 0 15px 0;
+    margin-bottom: 1rem;
 }
 
 .download-progress.info {
@@ -478,6 +846,10 @@ button:hover {
     background-color: var(--highlight-error);
 }
 
+.download-progress.error h3 {
+    color: #fff;
+}
+
 /* settings */
 .settings-group {
     background-color: var(--highlight-bg);
@@ -495,6 +867,7 @@ button:hover {
 
 .danger-zone {
     background-color: var(--highlight-error);
+    color: #fff;
     padding: 3px;
 }
 
@@ -567,31 +940,26 @@ button:hover {
     .boxed-content {
         width: 90%;
     }
-    .video-list {
+    .video-list.grid,
+    .dl-list.grid,
+    .channel-list.grid,
+    .playlist-list.grid {
         grid-template-columns: 1fr 1fr;
     }
+    .dl-thumb.list {
+        width: 35%;
+    }
+    .video-item.list,
+    .playlist-item.list {
+        display: grid;
+        grid-template-columns: 35% auto;
+    }
     .two-col {
         display: block;
     }
     .two-col > div {
         width: 100%;
     }
-}
-
-/* phone */
-@media screen and (max-width: 600px) {
-    * {
-        word-wrap: anywhere;
-    }
-    .video-list {
-        grid-template-columns: 1fr;
-    }
-    .boxed-content {
-        width: 95%;
-    }
-    .footer {
-        text-align: center;
-    }
     .top-nav {
         flex-wrap: wrap-reverse;
         display: flex;
@@ -602,15 +970,51 @@ button:hover {
         position: unset;
         transform: unset;
     }
+}
+
+/* phone */
+@media screen and (max-width: 600px) {
+    * {
+        word-wrap: anywhere;
+    }
+    .video-list.grid,
+    .dl-list.grid,
+    .channel-list.grid,
+    .video-item.list,
+    .playlist-list.list,
+    .playlist-list.grid {
+        grid-template-columns: 1fr;
+    }
+    .playlist-item.list {
+        display: block;
+    }
+    .video-desc.grid {
+        height: unset;
+        display: flex;
+        flex-wrap: wrap-reverse;
+    }
+    .boxed-content {
+        width: 95%;
+    }
+    .footer {
+        text-align: center;
+    }
+    .toggle {
+        flex-wrap: wrap;
+    }
+    .nav-items {
+        display: grid;
+        grid-template-columns: 1fr 1fr;
+    }
     .nav-item {
-        padding: unset;
+        padding: 5px 0;
         margin: 15px;
+        text-align: center;
     }
     .sort {
         display: block;
     }
     .sort select {
-        width: 100%;
         margin: unset;
     }
     .info-box-2,
@@ -629,4 +1033,15 @@ button:hover {
     .dl-desc {
         padding: 15px;
     }
+    .playlist-nav {
+        display: block;
+        grid-template-columns: unset;
+    }
+    .playlist-nav-item {
+        display: block;
+        justify-content: unset;
+    }
+    .playlist-nav-item img {
+        width: 100%;
+    }
 }

tubearchivist/static/font/OFL_License.txt

@@ -0,0 +1,94 @@
+Copyright (c) 2015, Kosal Sen, Philatype (<http://philatype.com>),
+with Reserved Font Name Sen.
+
+This Font Software is licensed under the SIL Open Font License, Version 1.1.
+This license is copied below, and is also available with a FAQ at:
+http://scripts.sil.org/OFL
+
+
+-----------------------------------------------------------
+SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
+-----------------------------------------------------------
+
+PREAMBLE
+The goals of the Open Font License (OFL) are to stimulate worldwide
+development of collaborative font projects, to support the font creation
+efforts of academic and linguistic communities, and to provide a free and
+open framework in which fonts may be shared and improved in partnership
+with others.
+
+The OFL allows the licensed fonts to be used, studied, modified and
+redistributed freely as long as they are not sold by themselves. The
+fonts, including any derivative works, can be bundled, embedded, 
+redistributed and/or sold with any software provided that any reserved
+names are not used by derivative works. The fonts and derivatives,
+however, cannot be released under any other type of license. The
+requirement for fonts to remain under this license does not apply
+to any document created using the fonts or their derivatives.
+
+DEFINITIONS
+"Font Software" refers to the set of files released by the Copyright
+Holder(s) under this license and clearly marked as such. This may
+include source files, build scripts and documentation.
+
+"Reserved Font Name" refers to any names specified as such after the
+copyright statement(s).
+
+"Original Version" refers to the collection of Font Software components as
+distributed by the Copyright Holder(s).
+
+"Modified Version" refers to any derivative made by adding to, deleting,
+or substituting -- in part or in whole -- any of the components of the
+Original Version, by changing formats or by porting the Font Software to a
+new environment.
+
+"Author" refers to any designer, engineer, programmer, technical
+writer or other person who contributed to the Font Software.
+
+PERMISSION & CONDITIONS
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of the Font Software, to use, study, copy, merge, embed, modify,
+redistribute, and sell modified and unmodified copies of the Font
+Software, subject to the following conditions:
+
+1) Neither the Font Software nor any of its individual components,
+in Original or Modified Versions, may be sold by itself.
+
+2) Original or Modified Versions of the Font Software may be bundled,
+redistributed and/or sold with any software, provided that each copy
+contains the above copyright notice and this license. These can be
+included either as stand-alone text files, human-readable headers or
+in the appropriate machine-readable metadata fields within text or
+binary files as long as those fields can be easily viewed by the user.
+
+3) No Modified Version of the Font Software may use the Reserved Font
+Name(s) unless explicit written permission is granted by the corresponding
+Copyright Holder. This restriction only applies to the primary font name as
+presented to the users.
+
+4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
+Software shall not be used to promote, endorse or advertise any
+Modified Version, except to acknowledge the contribution(s) of the
+Copyright Holder(s) and the Author(s) or with their explicit written
+permission.
+
+5) The Font Software, modified or unmodified, in part or in whole,
+must be distributed entirely under this license, and must not be
+distributed under any other license. The requirement for fonts to
+remain under this license does not apply to any document created
+using the Font Software.
+
+TERMINATION
+This license becomes null and void if any of the above conditions are
+not met.
+
+DISCLAIMER
+THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
+OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
+COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
+DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
+OTHER DEALINGS IN THE FONT SOFTWARE.

tubearchivist/static/img/icon-exit.svg

@@ -0,0 +1,67 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   xmlns:dc="http://purl.org/dc/elements/1.1/"
+   xmlns:cc="http://creativecommons.org/ns#"
+   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+   xmlns:svg="http://www.w3.org/2000/svg"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   width="500"
+   height="500"
+   viewBox="0 0 132.29197 132.29167"
+   version="1.1"
+   id="svg1303"
+   inkscape:version="0.92.4 (5da689c313, 2019-01-14)"
+   sodipodi:docname="Icons_exit.svg">
+  <defs
+     id="defs1297" />
+  <sodipodi:namedview
+     id="base"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageopacity="0.0"
+     inkscape:pageshadow="2"
+     inkscape:zoom="0.66442107"
+     inkscape:cx="161.45413"
+     inkscape:cy="207.61753"
+     inkscape:document-units="mm"
+     inkscape:current-layer="layer1"
+     showgrid="false"
+     units="px"
+     inkscape:window-width="1169"
+     inkscape:window-height="893"
+     inkscape:window-x="729"
+     inkscape:window-y="13"
+     inkscape:window-maximized="0" />
+  <metadata
+     id="metadata1300">
+    <rdf:RDF>
+      <cc:Work
+         rdf:about="">
+        <dc:format>image/svg+xml</dc:format>
+        <dc:type
+           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+        <dc:title></dc:title>
+      </cc:Work>
+    </rdf:RDF>
+  </metadata>
+  <g
+     inkscape:label="Ebene 1"
+     inkscape:groupmode="layer"
+     id="layer1"
+     transform="translate(0,-164.70764)">
+    <g
+       id="g855"
+       transform="matrix(1.9016362,0,0,1.9016362,-197.93838,-58.9418)">
+      <path
+         inkscape:connector-curvature="0"
+         id="rect1208"
+         style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;paint-order:markers fill stroke"
+         d="m 124.57603,151.92962 c -0.0433,2.30016 2.0751,4.19245 4.75007,4.24278 l 30.26401,0.43007 -6.00195,5.78023 3.43246,3.56154 10.2778,-9.9006 0.002,0.002 3.5183,-3.3908 -3.42991,-3.564 -9.8737,-10.24989 -3.51834,3.39083 5.84388,6.06803 -30.35875,-0.43185 c -2.67494,-0.0503 -4.86301,1.76094 -4.90629,4.06112 z m -17.65039,-32.01644 v 64.95883 h 7.44347 v -58.27707 h 26.3896 v 18.5229 h 7.44296 v -25.20466 z m 33.83307,39.75416 v 25.20467 h 7.44296 v -25.20467 z" />
+    </g>
+  </g>
+</svg>

tubearchivist/static/img/icon-gridview.svg

@@ -0,0 +1,122 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   xmlns:dc="http://purl.org/dc/elements/1.1/"
+   xmlns:cc="http://creativecommons.org/ns#"
+   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+   xmlns:svg="http://www.w3.org/2000/svg"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   width="2000"
+   height="2000"
+   viewBox="0 0 529.16666 529.16735"
+   version="1.1"
+   id="svg8"
+   inkscape:version="0.92.4 (5da689c313, 2019-01-14)"
+   sodipodi:docname="Gridview.svg">
+  <defs
+     id="defs2" />
+  <sodipodi:namedview
+     id="base"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageopacity="0.0"
+     inkscape:pageshadow="2"
+     inkscape:zoom="0.35729063"
+     inkscape:cx="901.7564"
+     inkscape:cy="1021.9111"
+     inkscape:document-units="mm"
+     inkscape:current-layer="layer1"
+     showgrid="false"
+     units="px"
+     showguides="true"
+     inkscape:guide-bbox="true"
+     inkscape:window-width="1920"
+     inkscape:window-height="1017"
+     inkscape:window-x="-8"
+     inkscape:window-y="-8"
+     inkscape:window-maximized="1">
+    <sodipodi:guide
+       position="247.25932,291.92959"
+       orientation="1,0"
+       id="guide853"
+       inkscape:locked="false" />
+    <sodipodi:guide
+       position="337.22901,167.98535"
+       orientation="0,1"
+       id="guide855"
+       inkscape:locked="false" />
+    <sodipodi:guide
+       position="266.76325,305.4565"
+       orientation="0,1"
+       id="guide857"
+       inkscape:locked="false" />
+    <sodipodi:guide
+       position="257.79774,279.50371"
+       orientation="0,1"
+       id="guide861"
+       inkscape:locked="false" />
+  </sodipodi:namedview>
+  <metadata
+     id="metadata5">
+    <rdf:RDF>
+      <cc:Work
+         rdf:about="">
+        <dc:format>image/svg+xml</dc:format>
+        <dc:type
+           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+        <dc:title></dc:title>
+      </cc:Work>
+    </rdf:RDF>
+  </metadata>
+  <g
+     inkscape:label="Ebene 1"
+     inkscape:groupmode="layer"
+     id="layer1"
+     transform="translate(0,232.16736)">
+    <g
+       id="g873"
+       transform="matrix(1.3431799,0,0,1.3431799,-84.854433,26.13855)"
+       style="stroke:none">
+      <rect
+         ry="7.445024"
+         rx="7.445024"
+         y="-121.39048"
+         x="79.903137"
+         height="113.24854"
+         width="167.35619"
+         id="rect815"
+         style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.56300002;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:1.12600005, 0.56300002999999998;stroke-dashoffset:0;stroke-opacity:0.22508042;paint-order:markers fill stroke" />
+      <rect
+         ry="7.445024"
+         rx="7.445024"
+         y="-121.7837"
+         x="273.05484"
+         height="113.24854"
+         width="167.35619"
+         id="rect815-4"
+         style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.56300002;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:1.12600006, 0.56300004000000003;stroke-dashoffset:0;stroke-opacity:0.22508042;paint-order:markers fill stroke" />
+      <rect
+         ry="7.445024"
+         rx="7.445024"
+         y="17.882772"
+         x="79.903137"
+         height="113.24854"
+         width="167.35619"
+         id="rect815-2"
+         style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.56300002;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:1.12600006, 0.56300004000000003;stroke-dashoffset:0;stroke-opacity:0.22508042;paint-order:markers fill stroke" />
+      <rect
+         ry="7.445024"
+         rx="7.445024"
+         y="17.453594"
+         x="273.05484"
+         height="113.24854"
+         width="167.35619"
+         id="rect815-7"
+         style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.56300002;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:1.12600006, 0.56300004000000003;stroke-dashoffset:0;stroke-opacity:0.22508042;paint-order:markers fill stroke" />
+    </g>
+  </g>
+</svg>

tubearchivist/static/img/icon-listview.svg

@@ -0,0 +1,122 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   xmlns:dc="http://purl.org/dc/elements/1.1/"
+   xmlns:cc="http://creativecommons.org/ns#"
+   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+   xmlns:svg="http://www.w3.org/2000/svg"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   width="2000"
+   height="2000"
+   viewBox="0 0 529.16666 529.16735"
+   version="1.1"
+   id="svg8"
+   inkscape:version="0.92.4 (5da689c313, 2019-01-14)"
+   sodipodi:docname="Listview.svg">
+  <defs
+     id="defs2" />
+  <sodipodi:namedview
+     id="base"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageopacity="0.0"
+     inkscape:pageshadow="2"
+     inkscape:zoom="0.42053519"
+     inkscape:cx="851.82064"
+     inkscape:cy="1105.7974"
+     inkscape:document-units="mm"
+     inkscape:current-layer="g873"
+     showgrid="false"
+     units="px"
+     showguides="true"
+     inkscape:guide-bbox="true"
+     inkscape:window-width="1920"
+     inkscape:window-height="1017"
+     inkscape:window-x="-8"
+     inkscape:window-y="-8"
+     inkscape:window-maximized="1">
+    <sodipodi:guide
+       position="247.25932,291.92959"
+       orientation="1,0"
+       id="guide853"
+       inkscape:locked="false" />
+    <sodipodi:guide
+       position="337.22901,167.98535"
+       orientation="0,1"
+       id="guide855"
+       inkscape:locked="false" />
+    <sodipodi:guide
+       position="266.76325,305.4565"
+       orientation="0,1"
+       id="guide857"
+       inkscape:locked="false" />
+    <sodipodi:guide
+       position="257.79774,279.50371"
+       orientation="0,1"
+       id="guide861"
+       inkscape:locked="false" />
+    <sodipodi:guide
+       position="22.413775,336.67894"
+       orientation="1,0"
+       id="guide926"
+       inkscape:locked="false" />
+    <sodipodi:guide
+       position="503.71355,217.50031"
+       orientation="1,0"
+       id="guide928"
+       inkscape:locked="false" />
+  </sodipodi:namedview>
+  <metadata
+     id="metadata5">
+    <rdf:RDF>
+      <cc:Work
+         rdf:about="">
+        <dc:format>image/svg+xml</dc:format>
+        <dc:type
+           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+        <dc:title></dc:title>
+      </cc:Work>
+    </rdf:RDF>
+  </metadata>
+  <g
+     inkscape:label="Ebene 1"
+     inkscape:groupmode="layer"
+     id="layer1"
+     transform="translate(0,232.16736)">
+    <g
+       id="g873"
+       transform="matrix(1.3431799,0,0,1.3431799,-84.854433,26.13855)">
+      <rect
+         ry="7.445024"
+         rx="7.445024"
+         y="-121.34892"
+         x="79.944702"
+         height="70.107315"
+         width="358.24551"
+         id="rect815"
+         style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.64810181;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:1.29620369, 0.64810185;stroke-dashoffset:0;stroke-opacity:0.22508042;paint-order:markers fill stroke" />
+      <rect
+         ry="7.4450235"
+         rx="7.4450235"
+         y="-31.167784"
+         x="79.861153"
+         height="70.107315"
+         width="358.32907"
+         id="rect815-20"
+         style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.64817739;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:1.29635485, 0.64817743;stroke-dashoffset:0;stroke-opacity:0.22508042;paint-order:markers fill stroke" />
+      <rect
+         ry="7.4450235"
+         rx="7.4450231"
+         y="60.024326"
+         x="79.908524"
+         height="70.106117"
+         width="358.28171"
+         id="rect815-8"
+         style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.64812905;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:1.29625813, 0.64812907;stroke-dashoffset:0;stroke-opacity:0.22508042;paint-order:markers fill stroke" />
+    </g>
+  </g>
+</svg>

tubearchivist/static/img/icon-star-empty.svg

@@ -0,0 +1,65 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   width="500"
+   height="500"
+   viewBox="0 0 132.29197 132.29167"
+   version="1.1"
+   id="svg1303"
+   inkscape:version="1.1.1 (3bf5ae0d25, 2021-09-20)"
+   sodipodi:docname="icon-star-empty.svg"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:svg="http://www.w3.org/2000/svg"
+   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+   xmlns:cc="http://creativecommons.org/ns#"
+   xmlns:dc="http://purl.org/dc/elements/1.1/">
+  <defs
+     id="defs1297" />
+  <sodipodi:namedview
+     id="base"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageopacity="0.0"
+     inkscape:pageshadow="2"
+     inkscape:zoom="2.4005076"
+     inkscape:cx="40.40812"
+     inkscape:cy="211.20533"
+     inkscape:document-units="mm"
+     inkscape:current-layer="layer1"
+     showgrid="false"
+     units="px"
+     inkscape:window-width="3840"
+     inkscape:window-height="2112"
+     inkscape:window-x="0"
+     inkscape:window-y="48"
+     inkscape:window-maximized="1"
+     inkscape:pagecheckerboard="0" />
+  <metadata
+     id="metadata1300">
+    <rdf:RDF>
+      <cc:Work
+         rdf:about="">
+        <dc:format>image/svg+xml</dc:format>
+        <dc:type
+           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+        <dc:title />
+      </cc:Work>
+    </rdf:RDF>
+  </metadata>
+  <g
+     inkscape:label="Ebene 1"
+     inkscape:groupmode="layer"
+     id="layer1"
+     transform="translate(0,-164.70764)">
+    <path
+       id="path1255"
+       style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;paint-order:markers fill stroke"
+       inkscape:transform-center-y="-6.4703827"
+       d="M 250 21.158203 L 189.78906 194.37891 L 6.4375 198.11523 L 152.57617 308.9082 L 99.470703 484.43945 L 250 379.69141 L 400.5293 484.43945 L 347.42383 308.9082 L 493.5625 198.11523 L 310.21094 194.37891 L 250 21.158203 z M 250 96.408203 L 291.69141 216.34766 L 418.64453 218.93555 L 317.45703 295.65039 L 354.22852 417.18945 L 250 344.66211 L 145.77148 417.18945 L 182.54297 295.65039 L 81.355469 218.93555 L 208.30859 216.34766 L 250 96.408203 z "
+       transform="matrix(0.26458394,0,0,0.26458394,0,164.70749)" />
+  </g>
+</svg>

tubearchivist/static/img/icon-star-full.svg

@@ -0,0 +1,74 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   xmlns:dc="http://purl.org/dc/elements/1.1/"
+   xmlns:cc="http://creativecommons.org/ns#"
+   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+   xmlns:svg="http://www.w3.org/2000/svg"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   width="500"
+   height="500"
+   viewBox="0 0 132.29197 132.29167"
+   version="1.1"
+   id="svg1303"
+   inkscape:version="0.92.4 (5da689c313, 2019-01-14)"
+   sodipodi:docname="Icons_star.svg">
+  <defs
+     id="defs1297" />
+  <sodipodi:namedview
+     id="base"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageopacity="0.0"
+     inkscape:pageshadow="2"
+     inkscape:zoom="1.0105705"
+     inkscape:cx="61.891881"
+     inkscape:cy="148.25167"
+     inkscape:document-units="mm"
+     inkscape:current-layer="layer1"
+     showgrid="false"
+     units="px"
+     inkscape:window-width="1920"
+     inkscape:window-height="1017"
+     inkscape:window-x="-8"
+     inkscape:window-y="-8"
+     inkscape:window-maximized="1" />
+  <metadata
+     id="metadata1300">
+    <rdf:RDF>
+      <cc:Work
+         rdf:about="">
+        <dc:format>image/svg+xml</dc:format>
+        <dc:type
+           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+        <dc:title />
+      </cc:Work>
+    </rdf:RDF>
+  </metadata>
+  <g
+     inkscape:label="Ebene 1"
+     inkscape:groupmode="layer"
+     id="layer1"
+     transform="translate(0,-164.70764)">
+    <path
+       sodipodi:type="star"
+       style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;paint-order:markers fill stroke"
+       id="path1255"
+       sodipodi:sides="5"
+       sodipodi:cx="66.145981"
+       sodipodi:cy="237.32387"
+       sodipodi:r1="67.758904"
+       sodipodi:r2="27.103561"
+       sodipodi:arg1="-1.5707963"
+       sodipodi:arg2="-0.9424778"
+       inkscape:flatsided="false"
+       inkscape:rounded="-3.46945e-18"
+       inkscape:randomized="0"
+       d="m 66.145983,169.56496 15.931071,45.83167 48.511476,0.98859 L 91.923,245.69933 105.97366,292.14197 66.145981,264.42743 26.318295,292.14197 40.368962,245.69933 1.7034347,216.38521 50.214907,215.39663 Z"
+       inkscape:transform-center-y="-6.4703827" />
+  </g>
+</svg>

tubearchivist/static/img/icon-star-half.svg

@@ -0,0 +1,66 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   width="500"
+   height="500"
+   viewBox="0 0 132.29197 132.29167"
+   version="1.1"
+   id="svg1303"
+   inkscape:version="1.1.1 (3bf5ae0d25, 2021-09-20)"
+   sodipodi:docname="icon-star-half.svg"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:svg="http://www.w3.org/2000/svg"
+   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+   xmlns:cc="http://creativecommons.org/ns#"
+   xmlns:dc="http://purl.org/dc/elements/1.1/">
+  <defs
+     id="defs1297" />
+  <sodipodi:namedview
+     id="base"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageopacity="0.0"
+     inkscape:pageshadow="2"
+     inkscape:zoom="1.0105705"
+     inkscape:cx="44.034533"
+     inkscape:cy="168.22181"
+     inkscape:document-units="mm"
+     inkscape:current-layer="layer1"
+     showgrid="false"
+     units="px"
+     inkscape:window-width="3840"
+     inkscape:window-height="2078"
+     inkscape:window-x="0"
+     inkscape:window-y="82"
+     inkscape:window-maximized="1"
+     inkscape:pagecheckerboard="0" />
+  <metadata
+     id="metadata1300">
+    <rdf:RDF>
+      <cc:Work
+         rdf:about="">
+        <dc:format>image/svg+xml</dc:format>
+        <dc:type
+           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+        <dc:title />
+      </cc:Work>
+    </rdf:RDF>
+  </metadata>
+  <g
+     inkscape:label="Ebene 1"
+     inkscape:groupmode="layer"
+     id="layer1"
+     transform="translate(0,-164.70764)">
+    <path
+       id="path1255"
+       style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;paint-order:markers fill stroke"
+       inkscape:transform-center-x="1.1220981e-06"
+       inkscape:transform-center-y="-6.4703827"
+       d="M 250 21.158203 L 189.78906 194.37891 L 6.4375 198.11523 L 152.57617 308.9082 L 99.470703 484.43945 L 250 379.69141 L 400.5293 484.43945 L 347.42383 308.9082 L 493.5625 198.11523 L 310.21094 194.37891 L 250 21.158203 z M 250 96.408203 L 291.69141 216.34766 L 418.64453 218.93555 L 317.45703 295.65039 L 354.22852 417.18945 L 250 344.66211 L 250 96.408203 z "
+       transform="matrix(0.26458394,0,0,0.26458394,0,164.70749)" />
+  </g>
+</svg>

tubearchivist/static/img/icon-stop.svg

@@ -0,0 +1,67 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   xmlns:dc="http://purl.org/dc/elements/1.1/"
+   xmlns:cc="http://creativecommons.org/ns#"
+   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+   xmlns:svg="http://www.w3.org/2000/svg"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   width="500"
+   height="500"
+   viewBox="0 0 132.29197 132.29167"
+   version="1.1"
+   id="svg1303"
+   inkscape:version="0.92.4 (5da689c313, 2019-01-14)"
+   sodipodi:docname="Icons_stop.svg">
+  <defs
+     id="defs1297" />
+  <sodipodi:namedview
+     id="base"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageopacity="0.0"
+     inkscape:pageshadow="2"
+     inkscape:zoom="1.0105705"
+     inkscape:cx="43.182711"
+     inkscape:cy="168.09972"
+     inkscape:document-units="mm"
+     inkscape:current-layer="layer1"
+     showgrid="false"
+     units="px"
+     inkscape:window-width="1920"
+     inkscape:window-height="1017"
+     inkscape:window-x="-8"
+     inkscape:window-y="-8"
+     inkscape:window-maximized="1" />
+  <metadata
+     id="metadata1300">
+    <rdf:RDF>
+      <cc:Work
+         rdf:about="">
+        <dc:format>image/svg+xml</dc:format>
+        <dc:type
+           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+        <dc:title></dc:title>
+      </cc:Work>
+    </rdf:RDF>
+  </metadata>
+  <g
+     inkscape:label="Ebene 1"
+     inkscape:groupmode="layer"
+     id="layer1"
+     transform="translate(0,-164.70764)">
+    <rect
+       style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;paint-order:markers fill stroke"
+       id="rect836"
+       width="118.86465"
+       height="118.86465"
+       x="6.7136617"
+       y="171.42116"
+       rx="10.00003"
+       ry="10.00003" />
+  </g>
+</svg>

tubearchivist/static/progress.js

@@ -15,9 +15,13 @@ function checkMessage() {
     req.open('GET', '/downloads/progress', true);
     req.onload = function() {
         var dlProgress = req.response;
+        // var dlStatus = dlProgress['status'];
         if (dlProgress['status']) {
             buildDownloadMessage(dlProgress);
             handleInterval();
+            // if (dlStatus == 'downloading') {
+            //     buildDownloadIcons();
+            // };
         };
     };
     req.send();
@@ -28,7 +32,7 @@ function handleInterval() {
     var watchDownload = setInterval(function() {
         var req = new XMLHttpRequest();
         req.responseType = 'json';
-        req.open('GET', '/downloads/progress', true);
+        req.open('GET', '/downloads/progress/', true);
         req.onload = function() {
             var dlProgress = req.response;
             if (dlProgress['status']) {
@@ -70,4 +74,33 @@ function buildDownloadMessage(dlProgress) {
     message.appendChild(title);
     message.appendChild(messageText);
     box.appendChild(message);
+    if (dlStatus == 'downloading' && dlLevel != 'error') {
+        box.appendChild(buildDownloadIcons());
+    };
 };
+
+
+// add dl control icons
+function buildDownloadIcons() {
+    var iconBox = document.createElement('div');
+    iconBox.classList = 'dl-control-icons';
+    // stop icon
+    var stopIcon = document.createElement('img');
+    stopIcon.setAttribute('id', "stop-icon");
+    stopIcon.setAttribute('title', "Stop Download Queue");
+    stopIcon.setAttribute('src', "/static/img/icon-stop.svg");
+    stopIcon.setAttribute('alt', "stop icon");
+    stopIcon.setAttribute('onclick', 'stopQueue()');
+    // kill icon
+    var killIcon = document.createElement('img');
+    killIcon.setAttribute('id', "kill-icon");
+    killIcon.setAttribute('title', "Kill Download Queue");
+    killIcon.setAttribute('src', "/static/img/icon-close.svg");
+    killIcon.setAttribute('alt', "kill icon");
+    killIcon.setAttribute('onclick', 'killQueue()');
+    // stich together
+    iconBox.appendChild(stopIcon);
+    iconBox.appendChild(killIcon);
+    
+    return iconBox
+}

tubearchivist/static/script.js

@@ -8,24 +8,6 @@ function sortChange(sortValue) {
     }, 500);
 }
 
-function hideWatched(hideValue) {
-    var payload = JSON.stringify({'hide_watched': hideValue});
-    sendPost(payload);
-    setTimeout(function(){
-        location.reload();
-        return false;
-    }, 500);
-}
-
-function showSubscribedOnly(showValue) {
-    var payload = JSON.stringify({'show_subed_only': showValue});
-    sendPost(payload);
-    setTimeout(function(){
-        location.reload();
-        return false;
-    }, 500);
-}
-
 function isWatched(youtube_id) {
     var payload = JSON.stringify({'watched': youtube_id});
     sendPost(payload);
@@ -33,14 +15,74 @@ function isWatched(youtube_id) {
     seenIcon.setAttribute('src', "/static/img/icon-seen.svg");
     seenIcon.setAttribute('alt', 'seen-icon');
     seenIcon.setAttribute('id', youtube_id);
+    seenIcon.setAttribute('title', "Mark as unwatched");
+    seenIcon.setAttribute('onclick', "isUnwatched(this.id)");
     seenIcon.classList = 'seen-icon';
     document.getElementById(youtube_id).replaceWith(seenIcon);
 }
 
-function unsubscribe(channel_id) {
+function isWatchedButton(button) {
-    var payload = JSON.stringify({'unsubscribe': channel_id});
+    youtube_id = button.getAttribute("data-id");
+    var payload = JSON.stringify({'watched': youtube_id});
+    button.remove();
     sendPost(payload);
-    document.getElementById(channel_id).remove();
+    setTimeout(function(){
+        location.reload();
+        return false;
+    }, 1000);
+}
+
+function isUnwatched(youtube_id) {
+    var payload = JSON.stringify({'un_watched': youtube_id});
+    sendPost(payload);
+    var unseenIcon = document.createElement('img');
+    unseenIcon.setAttribute('src', "/static/img/icon-unseen.svg");
+    unseenIcon.setAttribute('alt', 'unseen-icon');
+    unseenIcon.setAttribute('id', youtube_id);
+    unseenIcon.setAttribute('title', "Mark as watched");
+    unseenIcon.setAttribute('onclick', "isWatched(this.id)");
+    unseenIcon.classList = 'unseen-icon';
+    document.getElementById(youtube_id).replaceWith(unseenIcon);
+}
+
+function unsubscribe(id_unsub) {
+    console.log(payload);
+    var payload = JSON.stringify({'unsubscribe': id_unsub});
+    sendPost(payload);
+    document.getElementById(id_unsub).replaceWith("You are unsubscribed.");
+}
+
+function subscribe(id_sub) {
+    var payload = JSON.stringify({'subscribe': id_sub});
+    console.log(payload);
+    sendPost(payload);
+    document.getElementById(id_sub).replaceWith("You are subscribed.");
+}
+
+function changeView(image) {
+    var sourcePage = image.getAttribute("data-origin");
+    var newView = image.getAttribute("data-value");
+    var payload = JSON.stringify({'change_view': sourcePage + ":" + newView});
+    sendPost(payload);
+    setTimeout(function(){
+        location.reload();
+        return false;
+    }, 500);
+}
+
+function toggleCheckbox(checkbox) {
+    // pass checkbox id as key and checkbox.checked as value
+    var toggleId = checkbox.id;
+    var toggleVal = checkbox.checked;
+    var payloadDict = {};
+    payloadDict[toggleId] = toggleVal;
+    var payload = JSON.stringify(payloadDict);
+    sendPost(payload);
+    setTimeout(function(){
+        var currPage = window.location.pathname;
+        window.location.replace(currPage);
+        return false;
+    }, 500);
 }
 
 // download page buttons
@@ -79,6 +121,46 @@ function downloadNow(button) {
     }, 500);
 }
 
+function forgetIgnore(button) {
+    var youtube_id = button.getAttribute('data-id');
+    var payload = JSON.stringify({'forgetIgnore': youtube_id});
+    sendPost(payload);
+    document.getElementById("dl-" + youtube_id).remove();
+}
+
+function addSingle(button) {
+    var youtube_id = button.getAttribute('data-id');
+    var payload = JSON.stringify({'addSingle': youtube_id});
+    sendPost(payload);
+    document.getElementById("dl-" + youtube_id).remove();
+    setTimeout(function(){
+        handleInterval();
+    }, 500);
+}
+
+function deleteQueue(button) {
+    var to_delete = button.getAttribute('data-id');
+    var payload = JSON.stringify({'deleteQueue': to_delete});
+    console.log(payload);
+    sendPost(payload);
+    setTimeout(function(){
+        location.reload();
+        return false;
+    }, 1000);
+}
+
+function stopQueue() {
+    var payload = JSON.stringify({'queue': 'stop'});
+    sendPost(payload);
+    document.getElementById('stop-icon').remove();
+}
+
+function killQueue() {
+    var payload = JSON.stringify({'queue': 'kill'});
+    sendPost(payload);
+    document.getElementById('kill-icon').remove();
+}
+
 // settings page buttons
 function manualImport() {
     var payload = JSON.stringify({'manual-import': true});
@@ -91,6 +173,17 @@ function manualImport() {
     toReplace.appendChild(message);
 }
 
+function reEmbed() {
+    var payload = JSON.stringify({'re-embed': true});
+    sendPost(payload);
+    // clear button
+    var message = document.createElement('p');
+    message.innerText = 'processing thumbnails';
+    var toReplace = document.getElementById('re-embed');
+    toReplace.innerHTML = '';
+    toReplace.appendChild(message);
+}
+
 function dbBackup() {
     var payload = JSON.stringify({'db-backup': true});
     sendPost(payload)
@@ -104,7 +197,7 @@ function dbBackup() {
 
 function dbRestore() {
     var payload = JSON.stringify({'db-restore': true});
-    sendPost(payload)
+    sendPost(payload);
     // clear button
     var message = document.createElement('p');
     message.innerText = 'restoring from backup';
@@ -113,6 +206,78 @@ function dbRestore() {
     toReplace.appendChild(message);
 }
 
+function fsRescan() {
+    var payload = JSON.stringify({'fs-rescan': true});
+    sendPost(payload);
+    // clear button
+    var message = document.createElement('p');
+    message.innerText = 'File system scan in progress';
+    var toReplace = document.getElementById('fs-rescan');
+    toReplace.innerHTML = '';
+    toReplace.appendChild(message);
+}
+
+function findPlaylists(button) {
+    var channel_id = button.getAttribute("data-id");
+    var payload = JSON.stringify({'find-playlists': channel_id});
+    sendPost(payload);
+    // clear button
+    var message = document.createElement('p');
+    message.innerText = 'Scraping for playlists in progress';
+    document.getElementById("find-playlists-button").replaceWith(message);
+}
+
+// delete from file system
+function deleteConfirm() {
+    to_show = document.getElementById("delete-button");
+    document.getElementById("delete-item").style.display = 'none';
+    to_show.style.display = "block";
+}
+
+function deleteVideo(button) {
+    var to_delete = button.getAttribute("data-id");
+    var to_redirect = button.getAttribute("data-redirect");
+    var payload = JSON.stringify({"delete-video": to_delete});
+    sendPost(payload);
+    setTimeout(function(){
+        var redirect = "/channel/" + to_redirect;
+        window.location.replace(redirect);
+        return false;
+    }, 1000);
+}
+
+function deleteChannel(button) {
+    var to_delete = button.getAttribute("data-id");
+    var payload = JSON.stringify({"delete-channel": to_delete});
+    sendPost(payload);
+    setTimeout(function(){
+        window.location.replace("/channel/");
+        return false;
+    }, 1000);
+}
+
+function deletePlaylist(button) {
+    var playlist_id = button.getAttribute("data-id");
+    var playlist_action = button.getAttribute("data-action");
+    var payload = JSON.stringify({
+        "delete-playlist": {
+            "playlist-id": playlist_id,
+            "playlist-action": playlist_action
+        }
+    });
+    console.log(payload);
+    sendPost(payload);
+    setTimeout(function(){
+        window.location.replace("/playlist/");
+        return false;
+    }, 1000);
+}
+
+function cancelDelete() {
+    document.getElementById("delete-button").style.display = 'none';
+    document.getElementById("delete-item").style.display = 'block';
+}
+
 // player
 function createPlayer(button) {
     var mediaUrl = button.getAttribute('data-src');
@@ -133,6 +298,7 @@ function createPlayer(button) {
     videoPlayer.setAttribute('controls', true);
     videoPlayer.setAttribute('autoplay', true);
     videoPlayer.setAttribute('width', '100%');
+    videoPlayer.setAttribute('playsinline', true);
     videoPlayer.setAttribute('poster', mediaThumb);
     playerElement.appendChild(videoPlayer);
     // title bar
@@ -274,7 +440,7 @@ function showSearch() {
     } else {
         searchBox.style.display = "";
     }
-    var inputBox = document.getElementById('searchInput');
+    var inputBox = document.getElementById('id_searchInput');
     inputBox.focus();
 }
 

uwsgi.ini

@@ -1,8 +1,8 @@
 [uwsgi]
-; chdir = /app
+module = config.wsgi:application
-module=config.wsgi:application
+master = True
-master=True
+pidfile = /tmp/project-master.pid
-pidfile=/tmp/project-master.pid
+vacuum = True
-vacuum=True
+max-requests = 5000
-max-requests=5000
+socket = :8080
-socket = :8080
+buffer-size = 8192