fix new screenshot links in readme

Never have to update this again!

CONTRIBUTING.md

@@ -11,7 +11,7 @@ If the issue doesn't move forward due to a lack of response, I assume it's solve
 
 ## Wiki
 
-WIP: The wiki is where all user functions are explained in detail. These pages are mirrored into the **docs** folder of the repo. This allows for pull requests and all other features like regular code. Make any changes there, and I'll sync them  with the wiki tab.
+The wiki is where all user functions are explained in detail. These pages are mirrored into the **docs** folder of the repo. This allows for pull requests and all other features like regular code. Make any changes there, and I'll sync them with the wiki tab.
 
 ## Development Environment
 
@@ -20,8 +20,7 @@ I have learned the hard way, that working on a dockerized application outside of
 This is my setup I have landed on, YMMV:
 - Clone the repo, work on it with your favorite code editor in your local filesystem. *testing* branch is the where all the changes are happening, might be unstable and is WIP.
 - Then I have a VM on KVM hypervisor running standard Ubuntu Server LTS with docker installed. The VM keeps my projects separate and offers convenient snapshot functionality. The VM also offers ways to simulate lowend environments by limiting CPU cores and memory. But you could also just run docker on your host system.
-- Additionally to the required services as listed in the example docker-compose file, the **Dev Tools** of [Kibana](https://www.elastic.co/guide/en/kibana/current/docker.html) are invaluable for running and testing Elasticsearch queries.
-- The `Dockerfile` is structured in a way that the actual application code is in the last layer so rebuilding the image with only code changes utilizes the build cache for everything else and will take just 2-3 secs.
+- The `Dockerfile` is structured in a way that the actual application code is in the last layer so rebuilding the image with only code changes utilizes the build cache for everything else and will just take a few seconds.
 - Take a look at the `deploy.sh` file. I have my local DNS resolve `tubearchivist.local` to the IP of the VM for convenience. To deploy the latest changes and rebuild the application to the testing VM run:
 ```bash
 ./deploy.sh test
@@ -30,6 +29,11 @@ This is my setup I have landed on, YMMV:
 - The `test` argument takes another optional argument to build for a specific architecture valid options are: `amd64`, `arm64` and `multi`, default is `amd64`.
 - This `deploy.sh` file is not meant to be universally usable for every possible environment but could serve as an idea on how to automatically rebuild containers to test changes - customize to your liking. 
 
+## Working with Elasticsearch
+Additionally to the required services as listed in the example docker-compose file, the **Dev Tools** of [Kibana](https://www.elastic.co/guide/en/kibana/current/docker.html) are invaluable for running and testing Elasticsearch queries.  
+
+If you want to run queries in on the Elasticsearch container directly from your host with for example `curl` or something like *postman*, you might want to **publish** the port 9200 instead of just **exposing** it.
+
 ## Implementing a new feature
 
 Do you see anything on the roadmap that you would like to take a closer look at but you are not sure, what's the best way to tackle that? Or anything not on there yet you'd like to implement but are not sure how? Open up an issue and we try to find a solution together.
@@ -40,7 +44,12 @@ To fix a bug or implement a feature, fork the repository and make all changes to
 
 ## Releases
 
-Everything on the master branch is what's in the latest release and is what you get in your container when you `pull` either the *:latest* tag or the newest named version. If you want to test the newest changes and improvements, clone the repository and build the docker container with the Dockerfile from the testing branch.
+There are three different docker tags:
+- **latest**: As the name implies is the latest multiarch release for regular usage.
+- **unstable**: Intermediate amd64 builds for quick testing and improved collaboration. Don't mix with a *latest* installation, for your testing environment only. This is untested and WIP and will have breaking changes between commits that might require a reset to resolve. 
+- **semantic versioning**: There will be a handful named version tags that will also have a matching release and tag on github.
+
+If you want to see what's in your container, checkout the matching release tag. A merge to **master** usually means a *latest* or *unstable* release. If you want to preview changes in your testing environment, pull the *unstable* tag or clone the repository and build the docker container with the Dockerfile from the **testing** branch.
 
 ## Code formatting and linting
 

Dockerfile

@@ -1,6 +1,6 @@
 # build the tube archivist image from default python slim image
 
-FROM python:3.9.7-slim-bullseye
+FROM python:3.10.1-slim-bullseye
 ARG TARGETPLATFORM
 
 ENV PYTHONUNBUFFERED 1
@@ -25,9 +25,6 @@ RUN if [ "$TARGETPLATFORM" = "linux/amd64" ] ; then \
         apt-get -y update && apt-get -y install --no-install-recommends ffmpeg && rm -rf /var/lib/apt/lists/* \
     ; fi
 
-# copy config files
-COPY nginx.conf /etc/nginx/conf.d/
-
 # make folders
 RUN mkdir /cache
 RUN mkdir /youtube
@@ -37,6 +34,9 @@ RUN mkdir /app
 COPY ./tubearchivist/requirements.txt /requirements.txt
 RUN pip install --no-cache-dir -r requirements.txt --src /usr/local/src
 
+# copy config files
+COPY nginx.conf /etc/nginx/conf.d/
+
 # copy application into container
 COPY ./tubearchivist /app
 COPY ./run.sh /app

README.md

@@ -7,6 +7,7 @@
 * [Core functionality](#core-functionality)
 * [Screenshots](#screenshots)
 * [Problem Tube Archivist tries to solve](#problem-tube-archivist-tries-to-solve)
+* [Connect](#connect)
 * [Installing and updating](#installing-and-updating)
 * [Getting Started](#getting-started)
 * [Potential pitfalls](#potential-pitfalls)
@@ -42,16 +43,34 @@
 ## Problem Tube Archivist tries to solve
 Once your YouTube video collection grows, it becomes hard to search and find a specific video. That's where Tube Archivist comes in: By indexing your video collection with metadata from YouTube, you can organize, search and enjoy your archived YouTube videos without hassle offline through a convenient web interface.
 
+## Connect
+- [Discord](https://discord.gg/AFwz8nE7BK): Connect with us on our brand new Discord server.
+- [r/TubeArchivist](https://www.reddit.com/r/TubeArchivist/): Join our brand new Subreddit.
+
 ## Installing and updating
-Take a look at the example `docker-compose.yml` file provided. Tube Archivist depends on three main components split up into separate docker containers:  
+Take a look at the example `docker-compose.yml` file provided. Use the *latest* or the named semantic version tag. The *unstable* tag is for intermediate testing and as the name implies, is **unstable** and not be used on your main installation but in a [testing environment](CONTRIBUTING.md).  
+
+Tube Archivist depends on three main components split up into separate docker containers:  
 
 ### Tube Archivist
 The main Python application that displays and serves your video collection, built with Django.
   - Serves the interface on port `8000`
-  - Needs a mandatory volume for the video archive at **/youtube**
-  - And another recommended volume to save the cache for thumbnails and artwork at **/cache**.
+  - Needs a volume for the video archive at **/youtube**
+  - And another volume to save application data at **/cache**.
   - The environment variables `ES_URL` and `REDIS_HOST` are needed to tell Tube Archivist where Elasticsearch and Redis respectively are located.
-  - The environment variables `HOST_UID` and `HOST_GID` allows Tube Archivist to `chown` the video files to the main host system user instead of the container user.
+  - The environment variables `HOST_UID` and `HOST_GID` allows Tube Archivist to `chown` the video files to the main host system user instead of the container user. Those two variables are optional, not setting them will disable that functionality. That might be needed if the underlying filesystem doesn't support `chown` like *NFS*. 
+  - Change the environment variables `TA_USERNAME` and `TA_PASSWORD` to create the initial credentials. 
+  - `ELASTIC_PASSWORD` is for the password for Elasticsearch. The environment variable `ELASTIC_USER` is optional, should you want to change the username from the default *elastic*.
+  - For the scheduler to know what time it is, set your timezone with the `TZ` environment variable, defaults to *UTC*.
+
+### Port collisions
+If you have a collision on port `8000`, best solution is to use dockers *HOST_PORT* and *CONTAINER_PORT* distinction: To for example change the interface to port 9000 use `9000:8000` in your docker-compose file.  
+
+Should that not be an option, the Tube Archivist container takes these two additional environment variables:
+- **TA_PORT**: To actually change the port where nginx listens, make sure to also change the ports value in your docker-compose file.
+- **TA_UWSGI_PORT**: To change the default uwsgi port 8080 used for container internal networking between uwsgi serving the django application and nginx.  
+
+Changing any of these two environment variables will change the files *nginx.conf* and *uwsgi.ini* at startup using `sed` in your container.
 
 ### Elasticsearch
 Stores video meta data and makes everything searchable. Also keeps track of the download queue.
@@ -63,7 +82,7 @@ Follow the [documentation](https://www.elastic.co/guide/en/elasticsearch/referen
 ### Redis JSON
 Functions as a cache and temporary link between the application and the file system. Used to store and display messages and configuration variables.
   - Needs to be accessible over the default port `6379`
-  - Takes an optional volume at **/data** to make your configuration changes permanent.
+  - Needs a volume at **/data** to make your configuration changes permanent.
 
 ### Redis on a custom port
 For some architectures it might be required to run Redis JSON on a nonstandard port. To for example change the Redis port to **6380**, set the following values:
@@ -77,10 +96,10 @@ You will see the current version number of **Tube Archivist** in the footer of t
 * All testing and development is done with the Elasticsearch version number as mentioned in the provided *docker-compose.yml* file. This will be updated when a new release of Elasticsearch is available. Running an older version of Elasticsearch is most likely not going to result in any issues, but it's still recommended to run the same version as mentioned.
 
 ### Alternative installation instructions:
-- **arm64**: Newest Tube Archivist container is multi arch, so is Elasticsearch. RedisJSON doesn't offer arm builds, you can use `bbilly1/rejson`, an unofficial rebuild for arm64.
-  - NOTE: This is untested, looking for feedback.
+- **arm64**: The Tube Archivist container is multi arch, so is Elasticsearch. RedisJSON doesn't offer arm builds, you can use `bbilly1/rejson`, an unofficial rebuild for arm64.
 - **Synology**: There is a [discussion thread](https://github.com/bbilly1/tubearchivist/discussions/48) with Synology installation instructions.
-- **Unraid**: There is a [template](https://github.com/pairofcrocs/unraid-templates/) and forum [thread](https://forums.unraid.net/topic/114073-support-crocs-tube-archivist/) with additional information.
+- **Unraid**: The three containers needed are all in the Community Applications. First install `TubeArchivist RedisJSON` followed by `TubeArchivist ES`, and finally you can install `TubeArchivist`. If you have unraid specific issues, report those to the [support thread](https://forums.unraid.net/topic/114073-support-crocs-tube-archivist/ "support thread").
+
 
 ## Potential pitfalls
 ### vm.max_map_count
@@ -104,8 +123,14 @@ chown 1000:0 /path/to/mount/point
 ```
 This will match the permissions with the **UID** and **GID** of elasticsearch within the container and should fix the issue.
 
+### Disk usage
+The Elasticsearch index will turn to *read only* if the disk usage of the container goes above 95% until the usage drops below 90% again. Similar to that, TubeArchivist will become all sorts of messed up when running out of disk space. There are some error messages in the logs when that happens, but it's best to make sure to have enough disk space before starting to download.
+
 ## Getting Started
-1. Go through the **settings** page and look at the available options. Particularly set *Download Format* to your desired video quality before downloading. **Tube Archivist** downloads the best available quality by default.
+1. Go through the **settings** page and look at the available options. Particularly set *Download Format* to your desired video quality before downloading. **Tube Archivist** downloads the best available quality by default. To support iOS or MacOS and some other browsers a compatible format must be specified. For example:
+```
+bestvideo[VCODEC=avc1]+bestaudio[ACODEC=mp4a]/mp4
+```
 2. Subscribe to some of your favorite YouTube channels on the **channels** page. 
 3. On the **downloads** page, click on *Rescan subscriptions* to add videos from the subscribed channels to your Download queue or click on *Add to download queue* to manually add Video IDs, links, channels or playlists.
 4. Click on *Start download* and let **Tube Archivist** to it's thing. 
@@ -115,11 +140,18 @@ This will match the permissions with the **UID** and **GID** of elasticsearch wi
 This should be considered as a **minimal viable product**, there is an extensive list of future functions and improvements planned.
 
 ### Functionality
-- [ ] Access control
 - [ ] User roles
-- [ ] Create playlists
 - [ ] Podcast mode to serve channel as mp3
 - [ ] Implement [PyFilesystem](https://github.com/PyFilesystem/pyfilesystem2) for flexible video storage
+- [ ] Implement [Apprise](https://github.com/caronc/apprise) for notifications
+- [ ] Subtitle download & indexing
+- [ ] User created playlists
+- [ ] Auto play or play next link
+- [ ] SSO / LDAP support
+- [X] Auto rescan and auto download on a schedule [2021-12-17]
+- [X] Optional automatic deletion of watched items after a specified time [2021-12-17]
+- [X] Create playlists [2021-11-27]
+- [X] Access control [2021-11-01]
 - [X] Delete videos and channel [2021-10-16]
 - [X] Add thumbnail embed option [2021-10-16]
 - [X] Un-ignore videos [2021-10-03]
@@ -128,6 +160,7 @@ This should be considered as a **minimal viable product**, there is an extensive
 - [X] Scan your file system to index already downloaded videos [2021-09-14]
 
 ### UI
+- [ ] Fancy advanced unified search interface
 - [ ] Show similar videos on video page
 - [ ] Multi language support
 - [ ] Show total video downloaded vs total videos available in channel
@@ -136,9 +169,10 @@ This should be considered as a **minimal viable product**, there is an extensive
 
 
 ## Known limitations
-- Video files created by Tube Archivist need to be **mp4** video files for best browser compatibility.
+- Video files created by Tube Archivist need to be playable in your browser of choice. Not every codec is compatible with every browser and might require some testing with format selection. 
 - Every limitation of **yt-dlp** will also be present in Tube Archivist. If **yt-dlp** can't download or extract a video for any reason, Tube Archivist won't be able to either.
-- For now this is meant to be run in a trusted network environment. There is *no* security.
+- For now this is meant to be run in a trusted network environment. Not everything is properly authenticated.
+- There is currently no flexibility in naming of the media files.
 
 
 ## Donate

deploy.sh

@@ -3,13 +3,17 @@
 # deploy all needed project files to different servers:
 # test for local vm for testing
 # blackhole for local production
-# docker to publish
+# unstable to publish intermediate releases
+# docker to publish regular release
 
 # create builder:
 # docker buildx create --name tubearchivist
 # docker buildx use tubearchivist
 # docker buildx inspect --bootstrap
 
+# more details:
+# https://github.com/bbilly1/tubearchivist/issues/6
+
 set -e
 
 function sync_blackhole {
@@ -48,7 +52,8 @@ function sync_test {
         --exclude "db.sqlite3" \
         . -e ssh "$host":tubearchivist
 
-    rsync -r --progress --delete docker-compose.yml -e ssh "$host":docker
+    # uncomment or copy your own docker-compose file
+    # rsync -r --progress --delete docker-compose.yml -e ssh "$host":docker
 
     if [[ $1 = "amd64" ]]; then
         platform="linux/amd64"
@@ -97,6 +102,22 @@ function validate {
 }
 
 
+# publish unstable tag to docker
+function sync_unsable {
+
+    if [[ $(systemctl is-active docker) != 'active' ]]; then
+        echo "starting docker"
+        sudo systemctl start docker
+    fi
+
+    # start amd64 build
+    sudo docker buildx build \
+        --platform linux/amd64 \
+        -t bbilly1/tubearchivist:unstable --push .
+
+}
+
+
 function sync_docker {
 
     # check things
@@ -119,20 +140,15 @@ function sync_docker {
     printf "\ncreate new version:\n"
     read -r VERSION
 
-    # start build
-    sudo docker buildx build --platform linux/amd64,linux/arm64 -t bbilly1/tubearchivist:latest -t bbilly1/tubearchivist:"$VERSION" .
-
-    printf "\nlatest images:\n"
-    sudo docker image ls bbilly1/tubearchivist
-
-    echo "continue?"
+    echo "build and push $VERSION?"
     read -rn 1
 
-    # push to docker
-    echo "pushing multiarch latest and $VERSION:"
-    sudo docker buildx build --platform linux/amd64,linux/arm64 -t bbilly1/tubearchivist:latest -t bbilly1/tubearchivist:"$VERSION" --push .
-    #sudo docker push bbilly1/tubearchivist:latest
-    #sudo docker push bbilly1/tubearchivist:"$VERSION"
+    # start build
+    sudo docker buildx build \
+        --platform linux/amd64,linux/arm64 \
+        -t bbilly1/tubearchivist:latest \
+        -t bbilly1/tubearchivist:unstable \
+        -t bbilly1/tubearchivist:"$VERSION" --push .
 
     # create release tag
     echo "commits since last version:"
@@ -155,8 +171,10 @@ elif [[ $1 == "validate" ]]; then
     validate "$2"
 elif [[ $1 == "docker" ]]; then
     sync_docker
+elif [[ $1 == "unstable" ]]; then
+    sync_unsable
 else
-    echo "valid options are: blackhole | test | validate | docker"
+    echo "valid options are: blackhole | test | validate | docker | unstable"
 fi
 
 

docker-compose.yml

@@ -15,6 +15,10 @@ services:
       - REDIS_HOST=archivist-redis
       - HOST_UID=1000
       - HOST_GID=1000
+      - TA_USERNAME=tubearchivist
+      - TA_PASSWORD=verysecret
+      - ELASTIC_PASSWORD=verysecret
+      - TZ=America/New_York
     depends_on:
       - archivist-es
       - archivist-redis
@@ -22,17 +26,19 @@ services:
     image: redislabs/rejson:latest
     container_name: archivist-redis
     restart: always
-    ports:
-      - 6379:6379
+    expose:
+      - "6379"
     volumes:
       - ./volumes/tubearchivist/redis:/data
     depends_on:
       - archivist-es
   archivist-es:
-    image: docker.elastic.co/elasticsearch/elasticsearch:7.15.1
+    image: docker.elastic.co/elasticsearch/elasticsearch:7.16.2
     container_name: archivist-es
     restart: always
     environment:
+      - "xpack.security.enabled=true"
+      - "ELASTIC_PASSWORD=verysecret"
       - "discovery.type=single-node"
       - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
     ulimits:
@@ -41,5 +47,5 @@ services:
         hard: -1
     volumes:
       - ./volumes/tubearchivist/es:/usr/share/elasticsearch/data
-    ports:
-      - 9200:9200
+    expose:
+      - "9200"

docs/Channels.md

@@ -1,24 +1,26 @@
 # Channels Overview and Channel Detail Page
 
-The channels are organized on two different levels:
+The channels are organized on two different levels, similar as the [playlists](Playlists):
 
 ## Channels Overview
 Accessible at `/channel/` of your Tube Archivist, the **Overview Page** shows a list of all channels you have indexed. 
-- You can filter that list to show or hide subscribed channels from the drop down menu. Clicking on the channel banner or the channel name will direct you to the *Channel Detail Page*.
-- If you are subscribed to a channel a *Unsubscribe* button will show.
+- You can filter that list to show or hide subscribed channels with the toggle. Clicking on the channel banner or the channel name will direct you to the *Channel Detail Page*.
+- If you are subscribed to a channel a *Unsubscribe* button will show, if you aren't subscribed, a *Subscribe* button will show instead. 
 
 The **Subscribe to Channels** button <img src="assets/icon-add.png?raw=true" alt="add icon" width="20px" style="margin:0 5px;"> opens a text field to subscribe to a channel. You have a few options:
 - Enter the YouTube channel ID, a 25 character alphanumeric string. For example *UCBa659QWEk1AI4Tg--mrJ2A*
 - Enter the URL to the channel page on YouTube. For example *https://www.youtube.com/channel/UCBa659QWEk1AI4Tg--mrJ2A*
+- Enter the channel name for example: *https://www.youtube.com/c/TomScottGo*.
 - Enter the video URL for any video and let Tube Archivist extract the channel ID for you. For example *https://www.youtube.com/watch?v=2tdiKTSdE9Y*
 - Add one per line.
-- **Note**: Adding a link to a YouTube channel name is not yet supported, for example: *https://www.youtube.com/c/TomScottGo* will fail.
 
-The search icon <img src="assets/icon-search.png?raw=true" alt="search icon" width="20px" style="margin:0 5px;"> opens a text box to search for indexed channel names. Possible matches will show as you type. 
+You can search your indexed channels by clicking on the search icon <img src="assets/icon-search.png?raw=true" alt="search icon" width="20px" style="margin:0 5px;">. This will open a dedicated page.
 
 ## Channel Detail
 Each channel will get a dedicated channel detail page accessible at `/channel/<channel-id>/` of your Tube Archivist. This page shows all the videos you have downloaded from this channel plus additional metadata. 
-- If you are subscribed to the channel, an *Unsubscribe* button will show.
+- If you are subscribed to the channel, an *Unsubscribe* button will show, else the *Subscribe* button will show.
 - You can *Show* the channel description, that matches with the *About* tab on YouTube.
 - The **Mark as Watched** button will mark all videos of this channel as watched.
-- The button **Delete Channel** will delete the channel plus all videos of this channel, both media files and metadata.
+- The button **Delete Channel** will delete the channel plus all videos of this channel, both media files and metadata additionally this will also delete playlists metadata belonging to that channel.
+- The button **Find Playlists** will go out to youtube and look for playlists belonging to this channel and index all with at least one video downloaded. Only do this for channels where you care about playlists as this will slow down indexing new videos for having to check which playlist this belongs to.
+- The button **Show Playlists** will go to the [playlists](Playlists) page and filter the list to only show playlists from this channel.

docs/Downloads.md

@@ -3,33 +3,41 @@ Accessible at `/downloads/` of your Tube Archivist, this page handles all the do
 
 
 ## Rescan Subscriptions
-The **Rescan Subscriptions** icon <img src="assets/icon-rescan.png?raw=true" alt="rescan icon" width="20px" style="margin:0 5px;"> will start a background task to look for new videos from the channels you are subscribed to. You can define the channel page size on the [settings page](Settings#subscriptions). With the default channel page size, expect this process to take around 2-3 seconds for each channel you are subscribed to. A status message will show the progress.
+The **Rescan Subscriptions** icon <img src="assets/icon-rescan.png?raw=true" alt="rescan icon" width="20px" style="margin:0 5px;"> will start a background task to look for new videos from the channels and playlists you are subscribed to. You can define the channel and playlist page size on the [settings page](Settings#subscriptions). With the default page size, expect this process to take around 2-3 seconds for each channel or playlist you are subscribed to. A status message will show the progress.
 
-Then for every video found, **Tube Archivist** will skip the video if it has already been downloaded or if you added it to the *ignored* list before. All the other videos will get added to the download queue. Expect this to take around 1 second for each video as **Tube Archivist** needs to grab some additional metadata. New videos will get added at the bottom of the download queue.
+Then for every video found, **Tube Archivist** will skip the video if it has already been downloaded or if you added it to the *ignored* list before. All the other videos will get added to the download queue. Expect this to take around 2 seconds for each video as **Tube Archivist** needs to grab some additional metadata. New videos will get added at the bottom of the download queue.
 
 ## Download Queue
 The **Start Download** icon <img src="assets/icon-download.png?raw=true" alt="download icon" width="20px" style="margin:0 5px;"> will start the download process starting from the top of the queue. Take a look at the relevant settings on the [Settings Page](Settings#downloads). Once the process started, a progress message will show with additional details and controls: 
 - The stop icon <img src="assets/icon-stop.png?raw=true" alt="stop icon" width="20px" style="margin:0 5px;"> will gracefully stop the download process, once the current video has been finished successfully.
 - The cancel icon <img src="assets/icon-close-red.png?raw=true" alt="close icon" width="20px" style="margin:0 5px;"> is equivalent to killing the process and will stop the download immediately. Any leftover files will get deleted, the canceled video will still be available in the download queue.
 
+After downloading, Tube Archivist tries to add new videos to already indexed playlists.
+
 ## Add to Download Queue
 The **Add to Download Queue** icon <img src="assets/icon-add.png?raw=true" alt="add icon" width="20px" style="margin:0 5px;"> opens a text field to manually add videos to the download queue. You have a few options:
 - Add a link to a YouTube video. For example *https://www.youtube.com/watch?v=2tdiKTSdE9Y*.
 - Add a YouTube video ID. For example *2tdiKTSdE9Y*.
 - Add a link to a YouTube video by providing the shortened URL, for example *https://youtu.be/2tdiKTSdE9Y*.
 - Add a Channel ID or Channel URL to add every available video to the download queue. This will ignore the channel page size as described before and is meant for an initial download of the whole channel. You can still ignore selected videos before starting the download.
-- Add a playlist ID or URL to add every available video in the list to the download queue, for example *https://www.youtube.com/playlist?list=PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha* or *PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha*. Note that when you add a link to a video in a playlist, Tube Archivist assumes you want to download only the specific video and not the whole playlist, for example *https://www.youtube.com/watch?v=CINVwWHlzTY&list=PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha* will only add one video *CINVwWHlzTY* to the queue.
+- Add a channel name like for example *https://www.youtube.com/c/TomScottGo*.
+- Add a playlist ID or URL to add every available video in the list to the download queue, for example *https://www.youtube.com/playlist?list=PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha* or *PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha*. 
+  - Note: When adding a playlist to the queue, this playlist will automatically get [indexed](Playlists#playlist-detail).
+  - Note: When you add a link to a video in a playlist, Tube Archivist assumes you want to download only the specific video and not the whole playlist, for example *https://www.youtube.com/watch?v=CINVwWHlzTY&list=PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha* will only add one video *CINVwWHlzTY* to the queue.
 - Add one link per line.
-- **Note**: Adding a link to a YouTube channel name is not yet supported, for example: *https://www.youtube.com/c/TomScottGo* will fail.
 
 ## The Download Queue
 Below the three buttons you find the download queue. New items will get added at the bottom of the queue, the next video to download once you click on **Start Download** will be the first in the list.
 
 Every video in the download queue has two buttons:
 - **Ignore**: This will remove that video from the download queue and this video will not get added again, even when you **Rescan Subscriptions**.
-- **Download now**: This will give priority to this video. If the download process is already running, the prioritized video will get downloaded as soon as the current video is finished. If there is no download process running, this will start downloading this single video and stop after that.
+- **Download now**: This will give priority to this video. If the download process is already running, the prioritized video will get downloaded as soon as the current video is finished. If there is no download process running, this will start downloading this single video and stop after that.  
+
+The button **Delete all queued** will delete all pending videos from the download queue.
 
 You can flip the view by activating **Show Only Ignored Videos**. This will show all videos you have previously *ignored*.  
 Every video in the ignored list has two buttons:
 - **Forget**: This will delete the item form the ignored list.
-- **Add to Queue**: This will add the ignored video back to the download queue.
+- **Add to Queue**: This will add the ignored video back to the download queue.  
+
+The button **Delete all ignored** will delete all videos you have previously ignored.

docs/Home.md

@@ -2,10 +2,12 @@
 Welcome to the official Tube Archivist Wiki. This is an up-to-date documentation of user functionality.
 
 Table of contents:
-* [Main](Main): Tube Archivist landing page
-* [Channels](Channels): Browse your channels, handle subscriptions
+* [Channels](Channels): Browse your channels, handle channel subscriptions
+* [Playlists](Playlists): Browse your indexed playlists, handle playlist subscriptions
 * [Downloads](Downloads): Scanning subscriptions, handle download queue
 * [Settings](Settings): All the configuration options
+* [Video](Video): All details of a single video and playlist navigation.
+* [Users](Users): User management admin interface
 
 ## Getting Started
 1. [Subscribe](Channels#channels-overview) to some of your favourite YouTube channels.
@@ -18,9 +20,9 @@ Table of contents:
 * Clicking on the channel name or the channel icon brings you to the dedicated channel page to show videos from that channel.
 * Clicking on a video title brings you to the dedicated video page and shows additional details.
 * Clicking on a video thumbnail opens the video player and starts streaming the selected video.
-* Hover over the playing video to show additional control options.
+* Clicking on the search icon <img src="assets/icon-search.png?raw=true" alt="gridview icon" width="20px" style="margin:0 5px;"> will open a dedicated search page to search over your complete index.
 
 
 An empty checkbox icon <img src="assets/icon-unseen.png?raw=true" alt="unseen icon" width="20px" style="margin:0 5px;"> will show for videos you haven't marked as watched. Click on it and the icon will change to a filled checkbox <img src="assets/icon-seen.png?raw=true" alt="seen icon" width="20px" style="margin:0 5px;"> indicating it as watched - click again to revert.
 
-When available the <img src="assets/icon-gridview.png?raw=true" alt="gridview icon" width="20px" style="margin:0 5px;"> gridview icon will display the list in a grid, the <img src="assets/icon-listview.png?raw=true" alt="listview icon" width="20px" style="margin:0 5px;"> listview icon will arrange the items in a list.
+When available the <img src="assets/icon-gridview.png?raw=true" alt="gridview icon" width="20px" style="margin:0 5px;"> gridview icon will display the list in a grid, the <img src="assets/icon-listview.png?raw=true" alt="listview icon" width="20px" style="margin:0 5px;"> listview icon will arrange the items in a list. The sort icon <img src="assets/icon-sort.png?raw=true" alt="listview icon" width="20px" style="margin:0 5px;"> will open additional sort options.

docs/Main.md

@@ -1,10 +0,0 @@
-# Tube Archivist Home Page Functionality
-
-This is the landing page, when you first open **Tube Archivist**. You have a few options to sort and filter that view:
-- With the **Sort Order** you can select how the "Recent Videos" are sorted: 
-  - **Date Published**: Sorts the list by date when the video was published on YouTube, newest on top.
-  - **Date Downloaded**: Sorts the list based on when you have downloaded the video to your archive, newest on top.
-- With **Hide Watched** you can filter out videos you have already marked as watched to only show unwatched videos. 
-- You can use those two options together to for example filter the list to *Hide Watched* videos **and** sort by date downloaded.
-
-Additionally the search icon <img src="assets/icon-search.png?raw=true" alt="search icon" width="20px" style="margin:0 5px;"> opens a text field to search your collection.

docs/Playlists.md

@@ -0,0 +1,23 @@
+# Playlist Overview and Playlist Detail Page
+The playlists are organized in two different levels, similar as the [channels](Channels):
+
+## Playlist Overview
+Accessible at `/playlist/` of your Tube Archivist, this **Overview Page** shows a list of all playlists you have indexed over all your channels.
+- You can filter that list to show only subscribed to playlists with the toggle.
+
+You can index playlists of a channel from the channel detail page as described [here](Channels#channel-detail).
+
+The **Subscribe to Playlist** button <img src="assets/icon-add.png?raw=true" alt="add icon" width="20px" style="margin:0 5px;"> opens a text field to subscribe to playlists. You have a few options:
+- Enter the YouTube playlist id, for example: *PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha*
+- Enter the Youtube dedicated playlist url, for example: *https://www.youtube.com/playlist?list=PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha*
+- Add one per line.
+- NOTE: It doesn't make sense to subscribe to a playlist if you are already subscribed the corresponding channel as this will slow down the **Rescan Subscriptions** [task](Downloads#rescan-subscriptions).
+
+You can search your indexed playlists by clicking on the search icon <img src="assets/icon-search.png?raw=true" alt="search icon" width="20px" style="margin:0 5px;">. This will open a dedicated page.
+
+## Playlist Detail
+Each playlist will get a dedicated playlist detail page accessible at `/playlist/<playlist-id>/` of your Tube Archivist. This page shows all the videos you have downloaded from this playlist.
+
+- If you are subscribed to the playlist, an Unsubscribe button will show, else the Subscribe button will show.
+- The Mark as Watched button will mark all videos of this playlist as watched.
+- The **Delete Playlist** button will give you the option to delete just the *metadata* which won't delete any media files or *delete all* which will delete metadata plus all videos belonging to this playlist.

docs/Settings.md

@@ -17,14 +17,53 @@ Settings related to the channel management.
 Settings related to the download process.
 - **Download Limit**: Stop the download process after downloading the set quantity of videos.
 - **Download Speed Limit**: Set your download speed limit in KB/s. This will pass the option `--limit-rate` to yt-dlp.
+- **Throttled Rate Limit**: Restart download if the download speed drops below this value in KB/s. This will pass the option `--throttled-rate` to yt-dlp. Using this option might have a negative effect if you have an unstable or slow internet connection.
 - **Sleep Interval**: Time in seconds to sleep between requests to YouTube. It's a good idea to set this to **3** seconds. Might be necessary to avoid throttling.
+- **Auto Delete Watched Videos**: Automatically delete videos marked as watched after selected days. If activated, checks your videos after download task is finished.
 
 ## Download Format
 Additional settings passed to yt-dlp.
-- **Format**: This controls which streams get downloaded and is equivalent to passing `--format` to yt-dlp. Use one of the recommended one or look at the documentation of [yt-dlp](https://github.com/yt-dlp/yt-dlp#format-selection). Please note: The option `--merge-output-format mp4` is automatically passed to yt-dlp to guarantee browser compatibility.
+- **Format**: This controls which streams get downloaded and is equivalent to passing `--format` to yt-dlp. Use one of the recommended one or look at the documentation of [yt-dlp](https://github.com/yt-dlp/yt-dlp#format-selection). Please note: The option `--merge-output-format mp4` is automatically passed to yt-dlp to guarantee browser compatibility. Similar to that, `--check-formats` is passed as well to check that the selected formats are actually downloadable.
 - **Embed Metadata**: This saves the available tags directly into the media file by passing `--embed-metadata` to yt-dlp.
 - **Embed Thumbnail**: This will save the thumbnail into the media file by passing `--embed-thumbnail` to yt-dlp.
 
+## Integrations
+All third party integrations of TubeArchivist will **always** be *opt in*.
+- **returnyoutubedislike.com**: This will get dislikes and average ratings for each video back by integarting with the API from [returnyoutubedislike.com](https://www.returnyoutubedislike.com/).
+
+# Scheduler Setup
+Schedule settings expect a cron like format, where the first value is minute, second is hour and third is day of the week. Day 0 is Sunday, day 1 is Monday etc.
+
+Examples:
+- **0 15 \***: Run task every day at 15:00 in the afternoon.
+- **30 8 \*/2**: Run task every second day of the week (Sun, Tue, Thu, Sat) at 08:30 in the morning.
+- **0 \*/3,8-17 \***: Execute every hour divisible by 3, and every hour during office hours (8 in the morning - 5 in the afternoon).
+- **0 8,16 \***: Execute every day at 8 in the morning and at 4 in the afternoon. 
+- **auto**: Sensible default.
+- **0**: (zero), deactivate that task.
+
+NOTE:
+- Changes in the scheduler settings require a container restart to take effect.
+- Cron format as *number*/*number* are none standard cron and are not supported by the scheduler, for example **0 0/12 \*** is invalid, use **0 \*/12 \*** instead.
+- Avoid an unnecessary frequent schedule to not get blocked by YouTube. For that reason * or wildcards for minutes are not supported.
+
+## Rescan Subscriptions
+That's the equivalent task as run from the downloads page looking through your channel and playlist and add missing videos to the download queue.
+
+## Start download
+Start downloading all videos currently in the download queue.
+
+## Refresh Metadata
+Rescan videos, channels and playlists on youtube and update metadata periodically. This will also deactivate an item and exclude it from future refreshes if the link on YouTube is no longer available. This task is meant to be run once per day, set your schedule accordingly. 
+
+The field **Refresh older than x days** takes a number where TubeArchivist will consider an item as *outdated*. This value is used to calculate how many items need to be refreshed today based on the total indexed. This will spread out the requests to YouTube. Sensible value here is **90** days. 
+
+## Thumbnail check
+This will check if all expected thumbnails are there and will delete any artwork without matching video.
+
+## Index backup
+Create a zip file of the metadata and select **Max auto backups to keep** to automatically delete old backups created from this task.
+
 
 # Actions
 Additional database functionality.
@@ -50,13 +89,20 @@ Detect the YouTube ID from filename, this accepts the default yt-dlp naming conv
 - Maybe start with a subset of your files to import to make sure everything goes well...
 - Follow the logs to monitor progress and errors: `docker-compose logs -f tubearchivist`.
 
+## Embed thumbnails into media file
+This will write or overwrite all thumbnails in the media file using the downloaded thumbnail. This is only necessary if you didn't download the files with the option *Embed Thumbnail* enabled or want to make sure all media files get the newest thumbnail. Follow the docker-compose logs to monitor progress.
+
 ## Backup Database
 This will backup your metadata into a zip file. The file will get stored at *cache/backup* and will contain the necessary files to restore the Elasticsearch index formatted **nd-json** files plus a complete export of the index in a set of conventional **json** files.  
 
 BE AWARE: This will **not** backup any media files, just the metadata from the Elasticsearch.
 
 ## Restore From Backup
-The restore functionality will expect the same zip file in *cache/backup* as created from the **Backup database** function. This will recreate the index from the snapshot. If there are multiple backup files in the folder, the newest one will take priority. 
+The restore functionality will expect the same zip file in *cache/backup* as created from the **Backup database** function. This will recreate the index from the snapshot. There will be a list of all available backup to choose from. The *source* tag can have these different values:
+- **manual**: For backups manually created from here on the settings page.
+- **auto**: For backups automatically created via a sceduled task.
+- **update**: For backups created after a Tube Archivist update due to changes in the index.
+- **False**: Undefined.
 
 BE AWARE: This will **replace** your current index with the one from the backup file. This won't restore any media files.
 

docs/Users.md

@@ -0,0 +1,20 @@
+# User Management
+
+For now, **Tube Archivist** is a single user application. You can create multiple users with different names and passwords, they will share the same videos and permissions but some interface configurations are on a per user basis. *More is on the roadmap*.
+
+## Superuser
+The first user gets created with the environment variables **TA_USERNAME** and **TA_PASSWORD** from your docker-compose file. That first user will automatically have *superuser* privileges. 
+
+## Admin Interface
+When logged in from your *superuser* account, you are able to access the admin interface from the settings page or at `/admin/`. This interface holds all functionality for user management.
+
+## Create additional users
+From the admin interface when you click on *Accounts* you will get a list of all users. From there you can create additional users by clicking on *Add Account*, provide a name and confirm password and click on *Save* to create the user.
+
+## Changing users
+You can delete or change permissions and password of a user by clicking on the username from the *Accounts* list page and follow the interface from there. Changing the password of the *superuser* here will overwrite the password originally set with the environment variables.
+
+## Reset
+Delete all user configurations by deleting the file `cache/db.sqlite3` and restart the container. This will create the superuser again from the environment variables.
+
+NOTE: Future improvements here will most likely require such a reset.

docs/Video.md

@@ -0,0 +1,11 @@
+# Video Page
+
+Every video downloaded gets a dedicated page accessible at `/video/<video-id>` of your Tube Archivist.
+
+Clicking on the channel name or the channel icon will bring you to the dedicated channel detail [page](Channels#channel-detail).
+
+The button **Delete Video** will delete that video including the media file.
+
+When available, a playlist navigation will show at the bottom. Clicking on the playlist name will bring you to the dedicated [Playlist Detail](Playlists#playlist-detail) page showing all videos downloaded from that playlist. The number in square brackets indicates the position of the current video in that playlist.
+
+Clicking on the next or previous video name or thumbnail will bring you to that dedicated video page.

nginx.conf

@@ -2,8 +2,16 @@ server {
 
     listen 8000;
 
-    location /cache/ {
-        alias /cache/;
+    location /cache/videos/ {
+        alias /cache/videos/;
+    }
+
+    location /cache/channels/ {
+        alias /cache/channels/;
+    }
+
+    location /cache/playlists/ {
+        alias /cache/playlists/;
     }
 
     location /media/ {

run.sh

@@ -1,20 +1,55 @@
 #!/bin/bash
 # startup script inside the container for tubearchivist
 
+# check environment
+if [[ -z "$DJANGO_DEBUG" ]]; then
+    export DJANGO_DEBUG=False
+fi
+
+if [[ -z "$ELASTIC_USER" ]]; then
+    export ELASTIC_USER=elastic
+fi
+
+ENV_VARS=("TA_USERNAME" "TA_PASSWORD" "ELASTIC_PASSWORD" "ELASTIC_USER")
+for each in "${ENV_VARS[@]}"; do
+    if ! [[ -v $each ]]; then
+        echo "missing environment variable $each"
+        exit 1
+    fi
+done
+
+# ugly nginx and uwsgi port overwrite with env vars
+if [[ -n "$TA_PORT" ]]; then
+    sed -i "s/8000/$TA_PORT/g" /etc/nginx/conf.d/nginx.conf
+fi
+
+if [[ -n "$TA_UWSGI_PORT" ]]; then
+    sed -i "s/8080/$TA_UWSGI_PORT/g" /etc/nginx/conf.d/nginx.conf
+    sed -i "s/8080/$TA_UWSGI_PORT/g" /app/uwsgi.ini
+fi
+
+# wait for elasticsearch
 counter=0
-until curl "$ES_URL" -fs; do
+until curl -u "$ELASTIC_USER":"$ELASTIC_PASSWORD" "$ES_URL" -fs; do
     echo "waiting for elastic search to start"
     counter=$((counter+1))
     if [[ $counter -eq 12 ]]; then
-        # fail after 1 min
+        # fail after 2 min
         echo "failed to connect to elastic search, exiting..."
         exit 1
     fi
-    sleep 5
+    sleep 10
 done
 
+# start python application
+python manage.py makemigrations
 python manage.py migrate
+export DJANGO_SUPERUSER_PASSWORD=$TA_PASSWORD && \
+    python manage.py createsuperuser --noinput --name "$TA_USERNAME"
+
 python manage.py collectstatic --noinput -c
 nginx &
 celery -A home.tasks worker --loglevel=INFO &
+celery -A home beat --loglevel=INFO \
+    -s "${BEAT_SCHEDULE_PATH:-/cache/celerybeat-schedule}" &
 uwsgi --ini uwsgi.ini

tubearchivist/config/settings.py

@@ -10,9 +10,12 @@ For the full list of settings and their values, see
 https://docs.djangoproject.com/en/3.2/ref/settings/
 """
 
-from os import environ
+import hashlib
+from os import environ, path
 from pathlib import Path
 
+from home.src.config import AppConfig
+
 # Build paths inside the project like this: BASE_DIR / 'subdir'.
 BASE_DIR = Path(__file__).resolve().parent.parent
 
@@ -20,8 +23,8 @@ BASE_DIR = Path(__file__).resolve().parent.parent
 # Quick-start development settings - unsuitable for production
 # See https://docs.djangoproject.com/en/3.2/howto/deployment/checklist/
 
-# SECURITY WARNING: keep the secret key used in production secret!
-SECRET_KEY = "Fvid^aUL6LohRZz*kZFvq85B&JW&kB9o*#jdzWsdWE8*XkCLR8"
+PW_HASH = hashlib.sha256(environ.get("TA_PASSWORD").encode())
+SECRET_KEY = PW_HASH.hexdigest()
 
 # SECURITY WARNING: don't run with debug turned on in production!
 DEBUG = bool(environ.get("DJANGO_DEBUG"))
@@ -78,10 +81,12 @@ WSGI_APPLICATION = "config.wsgi.application"
 # Database
 # https://docs.djangoproject.com/en/3.2/ref/settings/#databases
 
+CACHE_DIR = AppConfig().config["application"]["cache_dir"]
+DB_PATH = path.join(CACHE_DIR, "db.sqlite3")
 DATABASES = {
     "default": {
         "ENGINE": "django.db.backends.sqlite3",
-        "NAME": BASE_DIR / "db.sqlite3",
+        "NAME": DB_PATH,
     }
 }
 
@@ -104,18 +109,16 @@ AUTH_PASSWORD_VALIDATORS = [
     },
 ]
 
+AUTH_USER_MODEL = "home.Account"
+
 
 # Internationalization
 # https://docs.djangoproject.com/en/3.2/topics/i18n/
 
 LANGUAGE_CODE = "en-us"
-
-TIME_ZONE = "UTC"
-
+TIME_ZONE = environ.get("TZ") or "UTC"
 USE_I18N = True
-
 USE_L10N = True
-
 USE_TZ = True
 
 
@@ -123,22 +126,14 @@ USE_TZ = True
 # https://docs.djangoproject.com/en/3.2/howto/static-files/
 
 STATIC_URL = "/static/"
-
-# STATICFILES_DIRS = [
-#     str(BASE_DIR.joinpath('static')),
-#     '/cache/'
-# ]
-
-# STATIC_URL = '/static/'
 STATICFILES_DIRS = (str(BASE_DIR.joinpath("static")),)
-# MEDIA_ROOT = str(BASE_DIR.joinpath('media'))
-# MEDIA_URL = '/media/'
-
 STATIC_ROOT = str(BASE_DIR.joinpath("staticfiles"))
-
 STATICFILES_STORAGE = "whitenoise.storage.CompressedManifestStaticFilesStorage"
 
 # Default primary key field type
 # https://docs.djangoproject.com/en/3.2/ref/settings/#default-auto-field
 
 DEFAULT_AUTO_FIELD = "django.db.models.BigAutoField"
+
+LOGIN_URL = "/login/"
+LOGOUT_REDIRECT_URL = "/login/"

tubearchivist/home/admin.py

@@ -1,3 +1,36 @@
-from django.contrib import admin  # noqa: F401 - Unused import
+"""custom admin classes"""
 
-# Register your models here.
+from django.contrib import admin
+from django.contrib.auth.admin import UserAdmin as BaseUserAdmin
+
+from .models import Account
+
+
+class HomeAdmin(BaseUserAdmin):
+    """register in admin page"""
+
+    list_display = ("name", "is_staff", "is_superuser")
+    list_filter = ("is_superuser",)
+
+    fieldsets = (
+        (None, {"fields": ("is_staff", "is_superuser", "password")}),
+        ("Personal info", {"fields": ("name",)}),
+        ("Groups", {"fields": ("groups",)}),
+        ("Permissions", {"fields": ("user_permissions",)}),
+    )
+    add_fieldsets = (
+        (
+            None,
+            {"fields": ("is_staff", "is_superuser", "password1", "password2")},
+        ),
+        ("Personal info", {"fields": ("name",)}),
+        ("Groups", {"fields": ("groups",)}),
+        ("Permissions", {"fields": ("user_permissions",)}),
+    )
+
+    search_fields = ("name",)
+    ordering = ("name",)
+    filter_horizontal = ()
+
+
+admin.site.register(Account, HomeAdmin)

tubearchivist/home/apps.py

@@ -6,7 +6,6 @@ from django.apps import AppConfig
 from home.src.config import AppConfig as ArchivistConfig
 from home.src.helper import RedisArchivist
 from home.src.index_management import index_check
-from home.src.thumbnails import validate_thumbnails
 
 
 def sync_redis_state():
@@ -18,7 +17,14 @@ def sync_redis_state():
 
 def make_folders():
     """make needed cache folders here so docker doesn't mess it up"""
-    folders = ["download", "channels", "videos", "import", "backup"]
+    folders = [
+        "download",
+        "channels",
+        "videos",
+        "playlists",
+        "import",
+        "backup",
+    ]
     config = ArchivistConfig().config
     cache_dir = config["application"]["cache_dir"]
     for folder in folders:
@@ -31,7 +37,13 @@ def make_folders():
 
 def release_lock():
     """make sure there are no leftover locks set in redis on container start"""
-    all_locks = ["manual_import", "downloading", "dl_queue", "dl_queue_id"]
+    all_locks = [
+        "manual_import",
+        "downloading",
+        "dl_queue",
+        "dl_queue_id",
+        "rescan",
+    ]
     for lock in all_locks:
         response = RedisArchivist().del_message(lock)
         if response:
@@ -49,4 +61,3 @@ class HomeConfig(AppConfig):
         index_check()
         sync_redis_state()
         make_folders()
-        validate_thumbnails()

tubearchivist/home/config.json

@@ -1,12 +1,14 @@
 {
     "archive": {
-        "sort": "published",
+        "sort_by": "published",
+        "sort_order": "desc",
         "page_size": 12
     },
     "default_view": {
         "home": "grid",
         "channel": "list",
-        "downloads": "list"
+        "downloads": "list",
+        "playlist": "grid"
     },
     "subscriptions": {
         "auto_search": false,
@@ -17,9 +19,12 @@
         "limit_count": false,
         "limit_speed": false,
         "sleep_interval": 3,
+        "autodelete_days": false,
         "format": false,
         "add_metadata": false,
-        "add_thumbnail": false
+        "add_thumbnail": false,
+        "throttledratelimit": false,
+        "integrate_ryd": false
     },
     "application": {
         "app_root": "/app",
@@ -27,5 +32,14 @@
         "videos": "/youtube",
         "file_template": "%(id)s_%(title)s.mp4",
         "colors": "dark"
+    },
+    "scheduler": {
+        "update_subscribed": false,
+        "download_pending": false,
+        "check_reindex":  {"minute": "0", "hour": "12", "day_of_week": "*"},
+        "check_reindex_days": 90,
+        "thumbnail_check": {"minute": "0", "hour": "17", "day_of_week": "*"},
+        "run_backup": {"minute": "0", "hour": "8", "day_of_week": "0"},
+        "run_backup_rotate": 5
     }
 }

tubearchivist/home/forms.py

@@ -0,0 +1,152 @@
+"""functionality:
+- hold all form classes used in the views
+"""
+
+from django import forms
+from django.contrib.auth.forms import AuthenticationForm
+from django.forms.widgets import PasswordInput, TextInput
+
+
+class CustomAuthForm(AuthenticationForm):
+    """better styled login form"""
+
+    username = forms.CharField(
+        widget=TextInput(
+            attrs={
+                "placeholder": "Username",
+                "autofocus": True,
+                "autocomplete": True,
+            }
+        ),
+        label=False,
+    )
+    password = forms.CharField(
+        widget=PasswordInput(attrs={"placeholder": "Password"}), label=False
+    )
+    remember_me = forms.BooleanField(required=False)
+
+
+class UserSettingsForm(forms.Form):
+    """user configurations values"""
+
+    CHOICES = [
+        ("", "-- change color scheme --"),
+        ("dark", "Dark"),
+        ("light", "Light"),
+    ]
+
+    colors = forms.ChoiceField(
+        widget=forms.Select, choices=CHOICES, required=False
+    )
+    page_size = forms.IntegerField(required=False)
+
+
+class ApplicationSettingsForm(forms.Form):
+    """handle all application settings"""
+
+    METADATA_CHOICES = [
+        ("", "-- change metadata embed --"),
+        ("0", "don't embed metadata"),
+        ("1", "embed metadata"),
+    ]
+
+    THUMBNAIL_CHOICES = [
+        ("", "-- change thumbnail embed --"),
+        ("0", "don't embed thumbnail"),
+        ("1", "embed thumbnail"),
+    ]
+
+    RYD_CHOICES = [
+        ("", "-- change ryd integrations"),
+        ("0", "disable ryd integration"),
+        ("1", "enable ryd integration"),
+    ]
+
+    subscriptions_channel_size = forms.IntegerField(required=False)
+    downloads_limit_count = forms.IntegerField(required=False)
+    downloads_limit_speed = forms.IntegerField(required=False)
+    downloads_throttledratelimit = forms.IntegerField(required=False)
+    downloads_sleep_interval = forms.IntegerField(required=False)
+    downloads_autodelete_days = forms.IntegerField(required=False)
+    downloads_format = forms.CharField(required=False)
+    downloads_add_metadata = forms.ChoiceField(
+        widget=forms.Select, choices=METADATA_CHOICES, required=False
+    )
+    downloads_add_thumbnail = forms.ChoiceField(
+        widget=forms.Select, choices=THUMBNAIL_CHOICES, required=False
+    )
+    downloads_integrate_ryd = forms.ChoiceField(
+        widget=forms.Select, choices=RYD_CHOICES, required=False
+    )
+
+
+class SchedulerSettingsForm(forms.Form):
+    """handle scheduler settings"""
+
+    update_subscribed = forms.CharField(required=False)
+    download_pending = forms.CharField(required=False)
+    check_reindex = forms.CharField(required=False)
+    check_reindex_days = forms.IntegerField(required=False)
+    thumbnail_check = forms.CharField(required=False)
+    run_backup = forms.CharField(required=False)
+    run_backup_rotate = forms.IntegerField(required=False)
+
+
+class MultiSearchForm(forms.Form):
+    """multi search form for /search/"""
+
+    searchInput = forms.CharField(
+        label="",
+        widget=forms.TextInput(
+            attrs={
+                "autocomplete": "off",
+                "oninput": "searchMulti(this.value)",
+                "autofocus": True,
+            }
+        ),
+    )
+    home = forms.CharField(widget=forms.HiddenInput())
+    channel = forms.CharField(widget=forms.HiddenInput())
+    playlist = forms.CharField(widget=forms.HiddenInput())
+
+
+class AddToQueueForm(forms.Form):
+    """text area form to add to downloads"""
+
+    vid_url = forms.CharField(
+        label=False,
+        widget=forms.Textarea(
+            attrs={
+                "rows": 4,
+                "placeholder": "Enter Video Urls or IDs here...",
+            }
+        ),
+    )
+
+
+class SubscribeToChannelForm(forms.Form):
+    """text area form to subscribe to multiple channels"""
+
+    subscribe = forms.CharField(
+        label="Subscribe to channels",
+        widget=forms.Textarea(
+            attrs={
+                "rows": 3,
+                "placeholder": "Input channel ID, URL or Video of a channel",
+            }
+        ),
+    )
+
+
+class SubscribeToPlaylistForm(forms.Form):
+    """text area form to subscribe to multiple playlists"""
+
+    subscribe = forms.CharField(
+        label="Subscribe to playlists",
+        widget=forms.Textarea(
+            attrs={
+                "rows": 3,
+                "placeholder": "Input playlist IDs or URLs",
+            }
+        ),
+    )

tubearchivist/home/models.py

@@ -1,3 +1,53 @@
-from django.db import models  # noqa: F401 - Unused import
+"""custom models"""
+from django.contrib.auth.models import (
+    AbstractBaseUser,
+    BaseUserManager,
+    PermissionsMixin,
+)
+from django.db import models
 
-# Create your models here.
+
+class AccountManager(BaseUserManager):
+    """manage user creation methods"""
+
+    use_in_migrations = True
+
+    def _create_user(self, name, password, **extra_fields):
+        """create regular user private"""
+        values = [name, password]
+        field_value_map = dict(zip(self.model.REQUIRED_FIELDS, values))
+        for field_name, value in field_value_map.items():
+            if not value:
+                raise ValueError(f"The {field_name} value must be set")
+
+        user = self.model(name=name, **extra_fields)
+        user.set_password(password)
+        user.save(using=self._db)
+        return user
+
+    def create_user(self, name, password):
+        """create regular user public"""
+        return self._create_user(name, password)
+
+    def create_superuser(self, name, password, **extra_fields):
+        """create super user"""
+        extra_fields.setdefault("is_staff", True)
+        extra_fields.setdefault("is_superuser", True)
+
+        if extra_fields.get("is_staff") is not True:
+            raise ValueError("Superuser must have is_staff=True.")
+        if extra_fields.get("is_superuser") is not True:
+            raise ValueError("Superuser must have is_superuser=True.")
+
+        return self._create_user(name, password, **extra_fields)
+
+
+class Account(AbstractBaseUser, PermissionsMixin):
+    """handle account creation"""
+
+    name = models.CharField(max_length=150, unique=True)
+    is_staff = models.BooleanField(default=False)
+    objects = AccountManager()
+
+    USERNAME_FIELD = "name"
+    REQUIRED_FIELDS = ["password"]

tubearchivist/home/src/config.py

@@ -7,15 +7,19 @@ Functionality:
 
 import json
 import os
+import re
 
+from celery.schedules import crontab
 from home.src.helper import RedisArchivist
 
 
 class AppConfig:
     """handle user settings and application variables"""
 
-    def __init__(self):
+    def __init__(self, user_id=False):
+        self.user_id = user_id
         self.config = self.get_config()
+        self.colors = self.get_colors()
 
     def get_config(self):
         """get config from default file or redis if changed"""
@@ -23,6 +27,12 @@ class AppConfig:
         if not config:
             config = self.get_config_file()
 
+        if self.user_id:
+            key = f"{self.user_id}:page_size"
+            page_size = RedisArchivist().get_message(key)["status"]
+            if page_size:
+                config["archive"]["page_size"] = page_size
+
         config["application"].update(self.get_config_env())
         return config
 
@@ -39,11 +49,27 @@ class AppConfig:
     @staticmethod
     def get_config_env():
         """read environment application variables"""
+        host_uid_env = os.environ.get("HOST_UID")
+        if host_uid_env:
+            host_uid = int(host_uid_env)
+        else:
+            host_uid = False
+
+        host_gid_env = os.environ.get("HOST_GID")
+        if host_gid_env:
+            host_gid = int(host_gid_env)
+        else:
+            host_gid = False
+
+        es_pass = os.environ.get("ELASTIC_PASSWORD")
+        es_user = os.environ.get("ELASTIC_USER", default="elastic")
+
         application = {
             "REDIS_HOST": os.environ.get("REDIS_HOST"),
             "es_url": os.environ.get("ES_URL"),
-            "HOST_UID": int(os.environ.get("HOST_UID")),
-            "HOST_GID": int(os.environ.get("HOST_GID")),
+            "es_auth": (es_user, es_pass),
+            "HOST_UID": host_uid,
+            "HOST_GID": host_gid,
         }
 
         return application
@@ -70,11 +96,36 @@ class AppConfig:
                 elif to_write.isdigit():
                     to_write = int(to_write)
 
-                config_dict, config_value = key.split(".")
+                config_dict, config_value = key.split("_", maxsplit=1)
                 config[config_dict][config_value] = to_write
 
         RedisArchivist().set_message("config", config, expire=False)
 
+    @staticmethod
+    def set_user_config(form_post, user_id):
+        """set values in redis for user settings"""
+        for key, value in form_post.items():
+            to_write = value[0]
+            if len(to_write):
+                if to_write.isdigit():
+                    to_write = int(to_write)
+                message = {"status": to_write}
+                redis_key = f"{user_id}:{key}"
+                RedisArchivist().set_message(redis_key, message, expire=False)
+
+    def get_colors(self):
+        """overwrite config if user has set custom values"""
+        colors = False
+        if self.user_id:
+            col_dict = RedisArchivist().get_message(f"{self.user_id}:colors")
+            colors = col_dict["status"]
+
+        if not colors:
+            colors = self.config["application"]["colors"]
+
+        self.config["application"]["colors"] = colors
+        return colors
+
     def load_new_defaults(self):
         """check config.json for missing defaults"""
         default_config = self.get_config_file()
@@ -101,3 +152,107 @@ class AppConfig:
 
         if needs_update:
             RedisArchivist().set_message("config", redis_config, expire=False)
+
+
+class ScheduleBuilder:
+    """build schedule dicts for beat"""
+
+    SCHEDULES = {
+        "update_subscribed": "0 8 *",
+        "download_pending": "0 16 *",
+        "check_reindex": "0 12 *",
+        "thumbnail_check": "0 17 *",
+        "run_backup": "0 18 0",
+    }
+    CONFIG = ["check_reindex_days", "run_backup_rotate"]
+
+    def __init__(self):
+        self.config = AppConfig().config
+
+    def update_schedule_conf(self, form_post):
+        """process form post"""
+        print("processing form, restart container for changes to take effect")
+        redis_config = self.config
+        for key, value in form_post.items():
+            to_check = value[0]
+            if key in self.SCHEDULES and to_check:
+                try:
+                    to_write = self.value_builder(key, to_check)
+                except ValueError:
+                    print(f"failed: {key} {to_check}")
+                    mess_dict = {
+                        "status": "message:setting",
+                        "level": "error",
+                        "title": "Scheduler update failed.",
+                        "message": "Invalid schedule input",
+                    }
+                    RedisArchivist().set_message("message:setting", mess_dict)
+                    return
+
+                redis_config["scheduler"][key] = to_write
+            if key in self.CONFIG and to_check:
+                redis_config["scheduler"][key] = int(to_check)
+        RedisArchivist().set_message("config", redis_config, expire=False)
+        mess_dict = {
+            "status": "message:setting",
+            "level": "info",
+            "title": "Scheduler changed.",
+            "message": "Please restart container for changes to take effect",
+        }
+        RedisArchivist().set_message("message:setting", mess_dict)
+
+    def value_builder(self, key, to_check):
+        """validate single cron form entry and return cron dict"""
+        print(f"change schedule for {key} to {to_check}")
+        if to_check == "0":
+            # deactivate this schedule
+            return False
+        if re.search(r"[\d]{1,2}\/[\d]{1,2}", to_check):
+            # number/number cron format will fail in celery
+            print("number/number schedule formatting not supported")
+            raise ValueError
+
+        keys = ["minute", "hour", "day_of_week"]
+        if to_check == "auto":
+            # set to sensible default
+            values = self.SCHEDULES[key].split()
+        else:
+            values = to_check.split()
+
+        if len(keys) != len(values):
+            print(f"failed to parse {to_check} for {key}")
+            raise ValueError("invalid input")
+
+        to_write = dict(zip(keys, values))
+        try:
+            int(to_write["minute"])
+        except ValueError as error:
+            print("too frequent: only number in minutes are supported")
+            raise ValueError("invalid input") from error
+
+        return to_write
+
+    def build_schedule(self):
+        """build schedule dict as expected by app.conf.beat_schedule"""
+        schedule_dict = {}
+
+        for schedule_item in self.SCHEDULES:
+            item_conf = self.config["scheduler"][schedule_item]
+            if not item_conf:
+                continue
+
+            minute = item_conf["minute"]
+            hour = item_conf["hour"]
+            day_of_week = item_conf["day_of_week"]
+            schedule_name = f"schedule_{schedule_item}"
+            to_add = {
+                schedule_name: {
+                    "task": schedule_item,
+                    "schedule": crontab(
+                        minute=minute, hour=hour, day_of_week=day_of_week
+                    ),
+                }
+            }
+            schedule_dict.update(to_add)
+
+        return schedule_dict

tubearchivist/home/src/download.py

@@ -2,6 +2,7 @@
 Functionality:
 - handele the download queue
 - manage subscriptions to channels
+- manage subscriptions to playlists
 - downloading videos
 """
 
@@ -12,7 +13,7 @@ from datetime import datetime
 from time import sleep
 
 import requests
-import yt_dlp as youtube_dl
+import yt_dlp
 from home.src.config import AppConfig
 from home.src.helper import (
     DurationConverter,
@@ -21,7 +22,13 @@ from home.src.helper import (
     clean_string,
     ignore_filelist,
 )
-from home.src.index import YoutubeChannel, index_new_video
+from home.src.index import (
+    IndexPaginate,
+    YoutubeChannel,
+    YoutubePlaylist,
+    YoutubeVideo,
+    index_new_video,
+)
 
 
 class PendingList:
@@ -29,101 +36,127 @@ class PendingList:
 
     CONFIG = AppConfig().config
     ES_URL = CONFIG["application"]["es_url"]
+    ES_AUTH = CONFIG["application"]["es_auth"]
     VIDEOS = CONFIG["application"]["videos"]
 
-    @staticmethod
-    def parse_url_list(youtube_ids):
+    def __init__(self):
+        self.all_channel_ids = False
+        self.all_downloaded = False
+        self.missing_from_playlists = []
+
+    def parse_url_list(self, youtube_ids):
         """extract youtube ids from list"""
         missing_videos = []
         for entry in youtube_ids:
             # notify
             mess_dict = {
-                "status": "pending",
+                "status": "message:add",
                 "level": "info",
                 "title": "Adding to download queue.",
                 "message": "Extracting lists",
             }
-            RedisArchivist().set_message("progress:download", mess_dict)
+            RedisArchivist().set_message("message:add", mess_dict)
             # extract
             url = entry["url"]
             url_type = entry["type"]
             if url_type == "video":
                 missing_videos.append(url)
             elif url_type == "channel":
-                youtube_ids = ChannelSubscription().get_last_youtube_videos(
+                video_results = ChannelSubscription().get_last_youtube_videos(
                     url, limit=False
                 )
+                youtube_ids = [i[0] for i in video_results]
                 missing_videos = missing_videos + youtube_ids
             elif url_type == "playlist":
-                youtube_ids = playlist_extractor(url)
+                self.missing_from_playlists.append(entry)
+                video_results = YoutubePlaylist(url).get_entries()
+                youtube_ids = [i["youtube_id"] for i in video_results]
                 missing_videos = missing_videos + youtube_ids
 
         return missing_videos
 
-    def add_to_pending(self, missing_videos):
+    def add_to_pending(self, missing_videos, ignore=False):
         """build the bulk json data from pending"""
         # check if channel is indexed
         channel_handler = ChannelSubscription()
         all_indexed = channel_handler.get_channels(subscribed_only=False)
-        all_channel_ids = [i["channel_id"] for i in all_indexed]
+        self.all_channel_ids = [i["channel_id"] for i in all_indexed]
         # check if already there
-        all_downloaded = self.get_all_downloaded()
-        # loop
-        bulk_list = []
-        all_videos_added = []
-        for video in missing_videos:
-            if isinstance(video, str):
-                youtube_id = video
-            elif isinstance(video, tuple):
-                youtube_id = video[0]
-            if youtube_id in all_downloaded:
-                # skip already downloaded
-                continue
-            video = self.get_youtube_details(youtube_id)
-            # skip on download error
-            if not video:
-                continue
+        self.all_downloaded = self.get_all_downloaded()
 
-            if video["channel_id"] in all_channel_ids:
-                video["channel_indexed"] = True
-            else:
-                video["channel_indexed"] = False
-            thumb_url = video["vid_thumb_url"]
-            video["status"] = "pending"
-            action = {"create": {"_id": youtube_id, "_index": "ta_download"}}
-            bulk_list.append(json.dumps(action))
-            bulk_list.append(json.dumps(video))
-            all_videos_added.append((youtube_id, thumb_url))
-            # notify
-            mess_dict = {
-                "status": "pending",
-                "level": "info",
-                "title": "Adding to download queue.",
-                "message": "Processing IDs...",
-            }
-            RedisArchivist().set_message("progress:download", mess_dict)
+        bulk_list, all_videos_added = self.build_bulk(missing_videos, ignore)
         # add last newline
         bulk_list.append("\n")
         query_str = "\n".join(bulk_list)
         headers = {"Content-type": "application/x-ndjson"}
         url = self.ES_URL + "/_bulk"
-        request = requests.post(url, data=query_str, headers=headers)
+        request = requests.post(
+            url, data=query_str, headers=headers, auth=self.ES_AUTH
+        )
         if not request.ok:
             print(request)
+            raise ValueError("failed to add video to download queue")
 
         return all_videos_added
 
+    def build_bulk(self, missing_videos, ignore=False):
+        """build the bulk lists"""
+        bulk_list = []
+        all_videos_added = []
+
+        for idx, youtube_id in enumerate(missing_videos):
+            # check if already downloaded
+            if youtube_id in self.all_downloaded:
+                continue
+
+            video = self.get_youtube_details(youtube_id)
+            # skip on download error
+            if not video:
+                continue
+
+            channel_indexed = video["channel_id"] in self.all_channel_ids
+            video["channel_indexed"] = channel_indexed
+            if ignore:
+                video["status"] = "ignore"
+            else:
+                video["status"] = "pending"
+            action = {"create": {"_id": youtube_id, "_index": "ta_download"}}
+            bulk_list.append(json.dumps(action))
+            bulk_list.append(json.dumps(video))
+            all_videos_added.append((youtube_id, video["vid_thumb_url"]))
+            # notify
+            progress = f"{idx + 1}/{len(missing_videos)}"
+            mess_dict = {
+                "status": "message:add",
+                "level": "info",
+                "title": "Adding new videos to download queue.",
+                "message": "Progress: " + progress,
+            }
+            if idx + 1 == len(missing_videos):
+                RedisArchivist().set_message(
+                    "message:add", mess_dict, expire=4
+                )
+            else:
+                RedisArchivist().set_message("message:add", mess_dict)
+            if idx + 1 % 25 == 0:
+                print("adding to queue progress: " + progress)
+
+        return bulk_list, all_videos_added
+
     @staticmethod
     def get_youtube_details(youtube_id):
         """get details from youtubedl for single pending video"""
         obs = {
             "default_search": "ytsearch",
             "quiet": True,
-            "skip_download": True,
+            "check_formats": "selected",
+            "noplaylist": True,
+            "writethumbnail": True,
+            "simulate": True,
         }
         try:
-            vid = youtube_dl.YoutubeDL(obs).extract_info(youtube_id)
-        except youtube_dl.utils.DownloadError:
+            vid = yt_dlp.YoutubeDL(obs).extract_info(youtube_id)
+        except yt_dlp.utils.DownloadError:
             print("failed to extract info for: " + youtube_id)
             return False
         # stop if video is streaming live now
@@ -150,81 +183,36 @@ class PendingList:
         }
         return youtube_details
 
-    def get_all_pending(self):
+    @staticmethod
+    def get_all_pending():
         """get a list of all pending videos in ta_download"""
-        headers = {"Content-type": "application/json"}
-        # get PIT ID
-        url = self.ES_URL + "/ta_download/_pit?keep_alive=1m"
-        response = requests.post(url)
-        json_data = json.loads(response.text)
-        pit_id = json_data["id"]
-        # query
         data = {
-            "size": 50,
             "query": {"match_all": {}},
-            "pit": {"id": pit_id, "keep_alive": "1m"},
             "sort": [{"timestamp": {"order": "asc"}}],
         }
-        query_str = json.dumps(data)
-        url = self.ES_URL + "/_search"
+        all_results = IndexPaginate("ta_download", data).get_results()
+
         all_pending = []
         all_ignore = []
-        while True:
-            response = requests.get(url, data=query_str, headers=headers)
-            json_data = json.loads(response.text)
-            all_hits = json_data["hits"]["hits"]
-            if all_hits:
-                for hit in all_hits:
-                    status = hit["_source"]["status"]
-                    if status == "pending":
-                        all_pending.append(hit["_source"])
-                    elif status == "ignore":
-                        all_ignore.append(hit["_source"])
-                    search_after = hit["sort"]
-                # update search_after with last hit data
-                data["search_after"] = search_after
-                query_str = json.dumps(data)
-            else:
-                break
-        # clean up PIT
-        query_str = json.dumps({"id": pit_id})
-        requests.delete(self.ES_URL + "/_pit", data=query_str, headers=headers)
+
+        for result in all_results:
+            if result["status"] == "pending":
+                all_pending.append(result)
+            elif result["status"] == "ignore":
+                all_ignore.append(result)
+
         return all_pending, all_ignore
 
-    def get_all_indexed(self):
+    @staticmethod
+    def get_all_indexed():
         """get a list of all videos indexed"""
-        headers = {"Content-type": "application/json"}
-        # get PIT ID
-        url = self.ES_URL + "/ta_video/_pit?keep_alive=1m"
-        response = requests.post(url)
-        json_data = json.loads(response.text)
-        pit_id = json_data["id"]
-        # query
+
         data = {
-            "size": 500,
             "query": {"match_all": {}},
-            "pit": {"id": pit_id, "keep_alive": "1m"},
             "sort": [{"published": {"order": "desc"}}],
         }
-        query_str = json.dumps(data)
-        url = self.ES_URL + "/_search"
-        all_indexed = []
-        while True:
-            response = requests.get(url, data=query_str, headers=headers)
-            json_data = json.loads(response.text)
-            all_hits = json_data["hits"]["hits"]
-            if all_hits:
-                for hit in all_hits:
-                    all_indexed.append(hit)
-                    search_after = hit["sort"]
-                # update search_after with last hit data
-                data["search_after"] = search_after
-                query_str = json.dumps(data)
-            else:
-                break
-        # clean up PIT
-        query_str = json.dumps({"id": pit_id})
-        requests.delete(self.ES_URL + "/_pit", data=query_str, headers=headers)
+        all_indexed = IndexPaginate("ta_video", data).get_results()
+
         return all_indexed
 
     def get_all_downloaded(self):
@@ -244,7 +232,19 @@ class PendingList:
     def delete_from_pending(self, youtube_id):
         """delete the youtube_id from ta_download"""
         url = f"{self.ES_URL}/ta_download/_doc/{youtube_id}"
-        response = requests.delete(url)
+        response = requests.delete(url, auth=self.ES_AUTH)
+        if not response.ok:
+            print(response.text)
+
+    def delete_pending(self, status):
+        """delete download queue based on status value"""
+        data = {"query": {"term": {"status": {"value": status}}}}
+        payload = json.dumps(data)
+        url = self.ES_URL + "/ta_download/_delete_by_query"
+        headers = {"Content-type": "application/json"}
+        response = requests.post(
+            url, data=payload, headers=headers, auth=self.ES_AUTH
+        )
         if not response.ok:
             print(response.text)
 
@@ -266,16 +266,12 @@ class PendingList:
 
         headers = {"Content-type": "application/x-ndjson"}
         url = self.ES_URL + "/_bulk"
-        request = requests.post(url, data=query_str, headers=headers)
-        mess_dict = {
-            "status": "ignore",
-            "level": "info",
-            "title": "Added to ignore list",
-            "message": "",
-        }
-        RedisArchivist().set_message("progress:download", mess_dict)
+        request = requests.post(
+            url, data=query_str, headers=headers, auth=self.ES_AUTH
+        )
         if not request.ok:
             print(request)
+            raise ValueError("failed to set video to ignore")
 
 
 class ChannelSubscription:
@@ -284,51 +280,22 @@ class ChannelSubscription:
     def __init__(self):
         config = AppConfig().config
         self.es_url = config["application"]["es_url"]
+        self.es_auth = config["application"]["es_auth"]
         self.channel_size = config["subscriptions"]["channel_size"]
 
-    def get_channels(self, subscribed_only=True):
+    @staticmethod
+    def get_channels(subscribed_only=True):
         """get a list of all channels subscribed to"""
-        headers = {"Content-type": "application/json"}
-        # get PIT ID
-        url = self.es_url + "/ta_channel/_pit?keep_alive=1m"
-        response = requests.post(url)
-        json_data = json.loads(response.text)
-        pit_id = json_data["id"]
-        # query
+        data = {
+            "sort": [{"channel_name.keyword": {"order": "asc"}}],
+        }
         if subscribed_only:
-            data = {
-                "query": {"term": {"channel_subscribed": {"value": True}}},
-                "size": 50,
-                "pit": {"id": pit_id, "keep_alive": "1m"},
-                "sort": [{"channel_name.keyword": {"order": "asc"}}],
-            }
+            data["query"] = {"term": {"channel_subscribed": {"value": True}}}
         else:
-            data = {
-                "query": {"match_all": {}},
-                "size": 50,
-                "pit": {"id": pit_id, "keep_alive": "1m"},
-                "sort": [{"channel_name.keyword": {"order": "asc"}}],
-            }
-        query_str = json.dumps(data)
-        url = self.es_url + "/_search"
-        all_channels = []
-        while True:
-            response = requests.get(url, data=query_str, headers=headers)
-            json_data = json.loads(response.text)
-            all_hits = json_data["hits"]["hits"]
-            if all_hits:
-                for hit in all_hits:
-                    source = hit["_source"]
-                    search_after = hit["sort"]
-                    all_channels.append(source)
-                # update search_after with last hit data
-                data["search_after"] = search_after
-                query_str = json.dumps(data)
-            else:
-                break
-        # clean up PIT
-        query_str = json.dumps({"id": pit_id})
-        requests.delete(self.es_url + "/_pit", data=query_str, headers=headers)
+            data["query"] = {"match_all": {}}
+
+        all_channels = IndexPaginate("ta_channel", data).get_results()
+
         return all_channels
 
     def get_last_youtube_videos(self, channel_id, limit=True):
@@ -342,7 +309,7 @@ class ChannelSubscription:
         }
         if limit:
             obs["playlistend"] = self.channel_size
-        chan = youtube_dl.YoutubeDL(obs).extract_info(url, download=False)
+        chan = yt_dlp.YoutubeDL(obs).extract_info(url, download=False)
         last_videos = [(i["id"], i["title"]) for i in chan["entries"]]
         return last_videos
 
@@ -354,25 +321,28 @@ class ChannelSubscription:
         all_ids = [i["youtube_id"] for i in all_ignore + all_pending]
         all_downloaded = pending_handler.get_all_downloaded()
         to_ignore = all_ids + all_downloaded
+
         missing_videos = []
-        counter = 1
-        for channel in all_channels:
+
+        for idx, channel in enumerate(all_channels):
             channel_id = channel["channel_id"]
             last_videos = self.get_last_youtube_videos(channel_id)
-            RedisArchivist().set_message(
-                "progress:download",
-                {
-                    "status": "rescan",
-                    "level": "info",
-                    "title": "Rescanning: Looking for new videos.",
-                    "message": f"Progress: {counter}/{len(all_channels)}",
-                },
-            )
             for video in last_videos:
-                youtube_id = video[0]
-                if youtube_id not in to_ignore:
-                    missing_videos.append(youtube_id)
-            counter = counter + 1
+                if video[0] not in to_ignore:
+                    missing_videos.append(video[0])
+            # notify
+            message = {
+                "status": "message:rescan",
+                "level": "info",
+                "title": "Scanning channels: Looking for new videos.",
+                "message": f"Progress: {idx + 1}/{len(all_channels)}",
+            }
+            if idx + 1 == len(all_channels):
+                RedisArchivist().set_message(
+                    "message:rescan", message=message, expire=4
+                )
+            else:
+                RedisArchivist().set_message("message:rescan", message=message)
 
         return missing_videos
 
@@ -394,28 +364,165 @@ class ChannelSubscription:
             url = self.es_url + "/ta_channel/_update/" + channel_id
             payload = json.dumps({"doc": channel_dict})
         # update channel
-        request = requests.post(url, data=payload, headers=headers)
+        request = requests.post(
+            url, data=payload, headers=headers, auth=self.es_auth
+        )
         if not request.ok:
             print(request.text)
+            raise ValueError("failed change subscribe status")
         # sync to videos
         channel_handler.sync_to_videos()
         if channel_handler.source == "scraped":
             channel_handler.get_channel_art()
 
 
-def playlist_extractor(playlist_id):
-    """return youtube_ids from a playlist_id"""
-    url = "https://www.youtube.com/playlist?list=" + playlist_id
-    obs = {
-        "default_search": "ytsearch",
-        "quiet": True,
-        "ignoreerrors": True,
-        "skip_download": True,
-        "extract_flat": True,
-    }
-    playlist = youtube_dl.YoutubeDL(obs).extract_info(url, download=False)
-    playlist_vids = [(i["id"], i["title"]) for i in playlist["entries"]]
-    return playlist_vids
+class PlaylistSubscription:
+    """manage the playlist download functionality"""
+
+    def __init__(self):
+        self.config = AppConfig().config
+
+    @staticmethod
+    def get_playlists(subscribed_only=True):
+        """get a list of all active playlists"""
+        data = {
+            "sort": [{"playlist_channel.keyword": {"order": "desc"}}],
+        }
+        data["query"] = {
+            "bool": {"must": [{"term": {"playlist_active": {"value": True}}}]}
+        }
+        if subscribed_only:
+            data["query"]["bool"]["must"].append(
+                {"term": {"playlist_subscribed": {"value": True}}}
+            )
+
+        all_playlists = IndexPaginate("ta_playlist", data).get_results()
+
+        return all_playlists
+
+    def process_url_str(self, new_playlists, subscribed=True):
+        """process playlist subscribe form url_str"""
+        all_indexed = PendingList().get_all_indexed()
+        all_youtube_ids = [i["youtube_id"] for i in all_indexed]
+
+        new_thumbs = []
+        for idx, playlist in enumerate(new_playlists):
+            url_type = playlist["type"]
+            playlist_id = playlist["url"]
+            if not url_type == "playlist":
+                print(f"{playlist_id} not a playlist, skipping...")
+                continue
+
+            playlist_h = YoutubePlaylist(
+                playlist_id, all_youtube_ids=all_youtube_ids
+            )
+            if not playlist_h.get_es_playlist():
+                playlist_h.get_playlist_dict()
+                playlist_h.playlist_dict["playlist_subscribed"] = subscribed
+                playlist_h.upload_to_es()
+                playlist_h.add_vids_to_playlist()
+                thumb = playlist_h.playlist_dict["playlist_thumbnail"]
+                new_thumbs.append((playlist_id, thumb))
+                self.channel_validate(playlist_h)
+            else:
+                self.change_subscribe(playlist_id, subscribe_status=True)
+
+            # notify
+            message = {
+                "status": "message:subplaylist",
+                "level": "info",
+                "title": "Subscribing to Playlists",
+                "message": f"Processing {idx + 1} of {len(new_playlists)}",
+            }
+            RedisArchivist().set_message(
+                "message:subplaylist", message=message
+            )
+
+        return new_thumbs
+
+    @staticmethod
+    def channel_validate(playlist_handler):
+        """make sure channel of playlist is there"""
+        channel_id = playlist_handler.playlist_dict["playlist_channel_id"]
+        channel_handler = YoutubeChannel(channel_id)
+        if channel_handler.source == "scraped":
+            channel_handler.channel_dict["channel_subscribed"] = False
+            channel_handler.upload_to_es()
+            channel_handler.get_channel_art()
+
+    def change_subscribe(self, playlist_id, subscribe_status):
+        """change the subscribe status of a playlist"""
+        es_url = self.config["application"]["es_url"]
+        es_auth = self.config["application"]["es_auth"]
+        playlist_handler = YoutubePlaylist(playlist_id)
+        playlist_handler.get_playlist_dict()
+        subed_now = playlist_handler.playlist_dict["playlist_subscribed"]
+
+        if subed_now == subscribe_status:
+            # status already as expected, do nothing
+            return False
+
+        # update subscribed status
+        headers = {"Content-type": "application/json"}
+        url = f"{es_url}/ta_playlist/_update/{playlist_id}"
+        payload = json.dumps(
+            {"doc": {"playlist_subscribed": subscribe_status}}
+        )
+        response = requests.post(
+            url, data=payload, headers=headers, auth=es_auth
+        )
+        if not response.ok:
+            print(response.text)
+            raise ValueError("failed to change subscribe status")
+
+        return True
+
+    @staticmethod
+    def get_to_ignore():
+        """get all youtube_ids already downloaded or ignored"""
+        pending_handler = PendingList()
+        all_pending, all_ignore = pending_handler.get_all_pending()
+        all_ids = [i["youtube_id"] for i in all_ignore + all_pending]
+        all_downloaded = pending_handler.get_all_downloaded()
+        to_ignore = all_ids + all_downloaded
+        return to_ignore
+
+    def find_missing(self):
+        """find videos in subscribed playlists not downloaded yet"""
+        all_playlists = [i["playlist_id"] for i in self.get_playlists()]
+        to_ignore = self.get_to_ignore()
+
+        missing_videos = []
+        counter = 1
+        for playlist_id in all_playlists:
+            size_limit = self.config["subscriptions"]["channel_size"]
+            playlist_handler = YoutubePlaylist(playlist_id)
+            playlist = playlist_handler.update_playlist()
+            if not playlist:
+                playlist_handler.deactivate()
+                continue
+
+            if size_limit:
+                playlist_entries = playlist["playlist_entries"][:size_limit]
+            else:
+                playlist_entries = playlist["playlist_entries"]
+            all_missing = [i for i in playlist_entries if not i["downloaded"]]
+
+            message = {
+                "status": "message:rescan",
+                "level": "info",
+                "title": "Scanning playlists: Looking for new videos.",
+                "message": f"Progress: {counter}/{len(all_playlists)}",
+            }
+            RedisArchivist().set_message("message:rescan", message=message)
+
+            for video in all_missing:
+                youtube_id = video["youtube_id"]
+                if youtube_id not in to_ignore:
+                    missing_videos.append(youtube_id)
+            counter = counter + 1
+
+        return missing_videos
 
 
 class VideoDownloader:
@@ -427,6 +534,7 @@ class VideoDownloader:
     def __init__(self, youtube_id_list=False):
         self.youtube_id_list = youtube_id_list
         self.config = AppConfig().config
+        self.channels = set()
 
     def run_queue(self):
         """setup download queue in redis loop until no more items"""
@@ -443,28 +551,41 @@ class VideoDownloader:
 
             try:
                 self.dl_single_vid(youtube_id)
-            except youtube_dl.utils.DownloadError:
+            except yt_dlp.utils.DownloadError:
                 print("failed to download " + youtube_id)
                 continue
             vid_dict = index_new_video(youtube_id)
+            self.channels.add(vid_dict["channel"]["channel_id"])
             self.move_to_archive(vid_dict)
             self.delete_from_pending(youtube_id)
 
+        autodelete_days = self.config["downloads"]["autodelete_days"]
+        if autodelete_days:
+            print(f"auto delete older than {autodelete_days} days")
+            self.auto_delete_watched(autodelete_days)
+
     @staticmethod
     def add_pending():
         """add pending videos to download queue"""
+        mess_dict = {
+            "status": "message:download",
+            "level": "info",
+            "title": "Looking for videos to download",
+            "message": "Scanning your download queue.",
+        }
+        RedisArchivist().set_message("message:download", mess_dict)
         all_pending, _ = PendingList().get_all_pending()
         to_add = [i["youtube_id"] for i in all_pending]
         if not to_add:
             # there is nothing pending
             print("download queue is empty")
             mess_dict = {
-                "status": "downloading",
+                "status": "message:download",
                 "level": "error",
                 "title": "Download queue is empty",
-                "message": "",
+                "message": "Add some videos to the queue first.",
             }
-            RedisArchivist().set_message("progress:download", mess_dict)
+            RedisArchivist().set_message("message:download", mess_dict)
             return
 
         queue = RedisQueue("dl_queue")
@@ -472,10 +593,12 @@ class VideoDownloader:
 
     @staticmethod
     def progress_hook(response):
-        """process the progress_hooks from youtube_dl"""
+        """process the progress_hooks from yt_dlp"""
         # title
-        filename = response["filename"][12:].replace("_", " ")
-        title = "Downloading: " + os.path.split(filename)[-1]
+        path = os.path.split(response["filename"])[-1][12:]
+        filename = os.path.splitext(os.path.splitext(path)[0])[0]
+        filename_clean = filename.replace("_", " ")
+        title = "Downloading: " + filename_clean
         # message
         try:
             percent = response["_percent_str"]
@@ -484,14 +607,14 @@ class VideoDownloader:
             eta = response["_eta_str"]
             message = f"{percent} of {size} at {speed} - time left: {eta}"
         except KeyError:
-            message = ""
+            message = "processing"
         mess_dict = {
-            "status": "downloading",
+            "status": "message:download",
             "level": "info",
             "title": title,
             "message": message,
         }
-        RedisArchivist().set_message("progress:download", mess_dict)
+        RedisArchivist().set_message("message:download", mess_dict)
 
     def build_obs(self):
         """build obs dictionary for yt-dlp"""
@@ -510,14 +633,17 @@ class VideoDownloader:
             "continuedl": True,
             "retries": 3,
             "writethumbnail": False,
+            "noplaylist": True,
+            "check_formats": "selected",
         }
         if self.config["downloads"]["format"]:
             obs["format"] = self.config["downloads"]["format"]
         if self.config["downloads"]["limit_speed"]:
             obs["ratelimit"] = self.config["downloads"]["limit_speed"] * 1024
-        external = False
-        if external:
-            obs["external_downloader"] = "aria2c"
+
+        throttle = self.config["downloads"]["throttledratelimit"]
+        if throttle:
+            obs["throttledratelimit"] = throttle * 1024
 
         postprocessors = []
 
@@ -553,10 +679,10 @@ class VideoDownloader:
         for file_name in all_cached:
             if youtube_id in file_name:
                 obs["outtmpl"] = os.path.join(dl_cache, file_name)
-        with youtube_dl.YoutubeDL(obs) as ydl:
+        with yt_dlp.YoutubeDL(obs) as ydl:
             try:
                 ydl.download([youtube_id])
-            except youtube_dl.utils.DownloadError:
+            except yt_dlp.utils.DownloadError:
                 print("retry failed download: " + youtube_id)
                 sleep(10)
                 ydl.download([youtube_id])
@@ -579,7 +705,8 @@ class VideoDownloader:
         new_folder = os.path.join(videos, channel_name)
         if not os.path.exists(new_folder):
             os.makedirs(new_folder)
-            os.chown(new_folder, host_uid, host_gid)
+            if host_uid and host_gid:
+                os.chown(new_folder, host_uid, host_gid)
         # find real filename
         cache_dir = self.config["application"]["cache_dir"]
         all_cached = ignore_filelist(os.listdir(cache_dir + "/download/"))
@@ -590,12 +717,86 @@ class VideoDownloader:
         new_file_path = os.path.join(videos, vid_dict["media_url"])
         # move media file and fix permission
         shutil.move(old_file_path, new_file_path)
-        os.chown(new_file_path, host_uid, host_gid)
+        if host_uid and host_gid:
+            os.chown(new_file_path, host_uid, host_gid)
 
     def delete_from_pending(self, youtube_id):
         """delete downloaded video from pending index if its there"""
         es_url = self.config["application"]["es_url"]
+        es_auth = self.config["application"]["es_auth"]
         url = f"{es_url}/ta_download/_doc/{youtube_id}"
-        response = requests.delete(url)
+        response = requests.delete(url, auth=es_auth)
         if not response.ok and not response.status_code == 404:
             print(response.text)
+
+    def add_subscribed_channels(self):
+        """add all channels subscribed to refresh"""
+        all_subscribed = PlaylistSubscription().get_playlists()
+        if not all_subscribed:
+            return
+
+        channel_ids = [i["playlist_channel_id"] for i in all_subscribed]
+        for channel_id in channel_ids:
+            self.channels.add(channel_id)
+
+        return
+
+    def validate_playlists(self):
+        """look for playlist needing to update"""
+        print("sync playlists")
+        self.add_subscribed_channels()
+        all_indexed = PendingList().get_all_indexed()
+        all_youtube_ids = [i["youtube_id"] for i in all_indexed]
+        for id_c, channel_id in enumerate(self.channels):
+            playlists = YoutubeChannel(channel_id).get_indexed_playlists()
+            all_playlist_ids = [i["playlist_id"] for i in playlists]
+            for id_p, playlist_id in enumerate(all_playlist_ids):
+                playlist_handler = YoutubePlaylist(
+                    playlist_id, all_youtube_ids=all_youtube_ids
+                )
+                playlist_dict = playlist_handler.update_playlist()
+                if not playlist_dict:
+                    playlist_handler.deactivate()
+                    continue
+
+                playlist_handler.add_vids_to_playlist()
+                # notify
+                title = (
+                    "Processing playlists for channels: "
+                    + f"{id_c + 1}/{len(self.channels)}"
+                )
+                message = f"Progress: {id_p + 1}/{len(all_playlist_ids)}"
+                mess_dict = {
+                    "status": "message:download",
+                    "level": "info",
+                    "title": title,
+                    "message": message,
+                }
+                if id_p + 1 == len(all_playlist_ids):
+                    RedisArchivist().set_message(
+                        "message:download", mess_dict, expire=4
+                    )
+                else:
+                    RedisArchivist().set_message("message:download", mess_dict)
+
+    @staticmethod
+    def auto_delete_watched(autodelete_days):
+        """delete watched videos after x days"""
+        now = int(datetime.now().strftime("%s"))
+        now_lte = now - autodelete_days * 24 * 60 * 60
+        data = {
+            "query": {"range": {"player.watched_date": {"lte": now_lte}}},
+            "sort": [{"player.watched_date": {"order": "asc"}}],
+        }
+        all_to_delete = IndexPaginate("ta_video", data).get_results()
+        all_youtube_ids = [i["youtube_id"] for i in all_to_delete]
+        if not all_youtube_ids:
+            return
+
+        for youtube_id in all_youtube_ids:
+            print(f"autodelete {youtube_id}")
+            YoutubeVideo(youtube_id).delete_media_file()
+
+        print("add deleted to ignore list")
+        pending_handler = PendingList()
+        pending_handler.add_to_pending(all_youtube_ids, ignore=True)

tubearchivist/home/src/frontend.py

@@ -0,0 +1,321 @@
+"""
+Functionality:
+- collection of functions and tasks from frontend
+- called via user input
+"""
+
+from home.src.download import (
+    ChannelSubscription,
+    PendingList,
+    PlaylistSubscription,
+)
+from home.src.helper import RedisArchivist, RedisQueue, UrlListParser
+from home.src.index import (
+    WatchState,
+    YoutubeChannel,
+    YoutubePlaylist,
+    YoutubeVideo,
+)
+from home.src.searching import SearchForm
+from home.tasks import (
+    download_pending,
+    download_single,
+    extrac_dl,
+    index_channel_playlists,
+    kill_dl,
+    re_sync_thumbs,
+    rescan_filesystem,
+    run_backup,
+    run_manual_import,
+    run_restore_backup,
+    subscribe_to,
+    update_subscribed,
+)
+
+
+class PostData:
+    """
+    map frontend http post values to backend funcs
+    handover long running tasks to celery
+    """
+
+    def __init__(self, post_dict, current_user):
+        self.post_dict = post_dict
+        self.to_exec, self.exec_val = list(post_dict.items())[0]
+        self.current_user = current_user
+
+    def run_task(self):
+        """execute and return task result"""
+        to_exec = self.exec_map()
+        task_result = to_exec()
+        return task_result
+
+    def exec_map(self):
+        """map dict key and return function to execute"""
+        exec_map = {
+            "watched": self._watched,
+            "un_watched": self._un_watched,
+            "change_view": self._change_view,
+            "rescan_pending": self._rescan_pending,
+            "ignore": self._ignore,
+            "dl_pending": self._dl_pending,
+            "queue": self._queue_handler,
+            "unsubscribe": self._unsubscribe,
+            "subscribe": self._subscribe,
+            "sort_order": self._sort_order,
+            "hide_watched": self._hide_watched,
+            "show_subed_only": self._show_subed_only,
+            "dlnow": self._dlnow,
+            "show_ignored_only": self._show_ignored_only,
+            "forgetIgnore": self._forget_ignore,
+            "addSingle": self._add_single,
+            "deleteQueue": self._delete_queue,
+            "manual-import": self._manual_import,
+            "re-embed": self._re_embed,
+            "db-backup": self._db_backup,
+            "db-restore": self._db_restore,
+            "fs-rescan": self._fs_rescan,
+            "multi_search": self._multi_search,
+            "delete-video": self._delete_video,
+            "delete-channel": self._delete_channel,
+            "delete-playlist": self._delete_playlist,
+            "find-playlists": self._find_playlists,
+        }
+
+        return exec_map[self.to_exec]
+
+    def _watched(self):
+        """mark as watched"""
+        WatchState(self.exec_val).mark_as_watched()
+        return {"success": True}
+
+    def _un_watched(self):
+        """mark as unwatched"""
+        WatchState(self.exec_val).mark_as_unwatched()
+        return {"success": True}
+
+    def _change_view(self):
+        """process view changes in home, channel, and downloads"""
+        origin, new_view = self.exec_val.split(":")
+        key = f"{self.current_user}:view:{origin}"
+        print(f"change view: {key} to {new_view}")
+        RedisArchivist().set_message(key, {"status": new_view}, expire=False)
+        return {"success": True}
+
+    @staticmethod
+    def _rescan_pending():
+        """look for new items in subscribed channels"""
+        print("rescan subscribed channels")
+        update_subscribed.delay()
+        return {"success": True}
+
+    def _ignore(self):
+        """ignore from download queue"""
+        id_to_ignore = self.exec_val
+        print("ignore video " + id_to_ignore)
+        handler = PendingList()
+        handler.ignore_from_pending([id_to_ignore])
+        # also clear from redis queue
+        RedisQueue("dl_queue").clear_item(id_to_ignore)
+        return {"success": True}
+
+    @staticmethod
+    def _dl_pending():
+        """start the download queue"""
+        print("download pending")
+        running = download_pending.delay()
+        task_id = running.id
+        print("set task id: " + task_id)
+        RedisArchivist().set_message("dl_queue_id", task_id, expire=False)
+        return {"success": True}
+
+    def _queue_handler(self):
+        """queue controls from frontend"""
+        to_execute = self.exec_val
+        if to_execute == "stop":
+            print("stopping download queue")
+            RedisQueue("dl_queue").clear()
+        elif to_execute == "kill":
+            task_id = RedisArchivist().get_message("dl_queue_id")
+            if not isinstance(task_id, str):
+                task_id = False
+            else:
+                print("brutally killing " + task_id)
+            kill_dl(task_id)
+
+        return {"success": True}
+
+    def _unsubscribe(self):
+        """unsubscribe from channels or playlists"""
+        id_unsub = self.exec_val
+        print("unsubscribe from " + id_unsub)
+        to_unsub_list = UrlListParser(id_unsub).process_list()
+        for to_unsub in to_unsub_list:
+            unsub_type = to_unsub["type"]
+            unsub_id = to_unsub["url"]
+            if unsub_type == "playlist":
+                PlaylistSubscription().change_subscribe(
+                    unsub_id, subscribe_status=False
+                )
+            elif unsub_type == "channel":
+                ChannelSubscription().change_subscribe(
+                    unsub_id, channel_subscribed=False
+                )
+            else:
+                raise ValueError("failed to process " + id_unsub)
+
+        return {"success": True}
+
+    def _subscribe(self):
+        """subscribe to channel or playlist, called from js buttons"""
+        id_sub = self.exec_val
+        print("subscribe to " + id_sub)
+        subscribe_to.delay(id_sub)
+        return {"success": True}
+
+    def _sort_order(self):
+        """change the sort between published to downloaded"""
+        sort_order = {"status": self.exec_val}
+        if self.exec_val in ["asc", "desc"]:
+            RedisArchivist().set_message(
+                f"{self.current_user}:sort_order", sort_order, expire=False
+            )
+        else:
+            RedisArchivist().set_message(
+                f"{self.current_user}:sort_by", sort_order, expire=False
+            )
+        return {"success": True}
+
+    def _hide_watched(self):
+        """toggle if to show watched vids or not"""
+        key = f"{self.current_user}:hide_watched"
+        message = {"status": bool(int(self.exec_val))}
+        print(f"toggle {key}: {message}")
+        RedisArchivist().set_message(key, message, expire=False)
+        return {"success": True}
+
+    def _show_subed_only(self):
+        """show or hide subscribed channels only on channels page"""
+        key = f"{self.current_user}:show_subed_only"
+        message = {"status": bool(int(self.exec_val))}
+        print(f"toggle {key}: {message}")
+        RedisArchivist().set_message(key, message, expire=False)
+        return {"success": True}
+
+    def _dlnow(self):
+        """start downloading single vid now"""
+        youtube_id = self.exec_val
+        print("downloading: " + youtube_id)
+        running = download_single.delay(youtube_id=youtube_id)
+        task_id = running.id
+        print("set task id: " + task_id)
+        RedisArchivist().set_message("dl_queue_id", task_id, expire=False)
+        return {"success": True}
+
+    def _show_ignored_only(self):
+        """switch view on /downloads/ to show ignored only"""
+        show_value = self.exec_val
+        key = f"{self.current_user}:show_ignored_only"
+        value = {"status": show_value}
+        print(f"Filter download view ignored only: {show_value}")
+        RedisArchivist().set_message(key, value, expire=False)
+        return {"success": True}
+
+    def _forget_ignore(self):
+        """delete from ta_download index"""
+        youtube_id = self.exec_val
+        print("forgetting from download index: " + youtube_id)
+        PendingList().delete_from_pending(youtube_id)
+        return {"success": True}
+
+    def _add_single(self):
+        """add single youtube_id to download queue"""
+        youtube_id = self.exec_val
+        print("add vid to dl queue: " + youtube_id)
+        PendingList().delete_from_pending(youtube_id)
+        youtube_ids = UrlListParser(youtube_id).process_list()
+        extrac_dl.delay(youtube_ids)
+        return {"success": True}
+
+    def _delete_queue(self):
+        """delete download queue"""
+        status = self.exec_val
+        print("deleting from download queue: " + status)
+        if status == "pending":
+            PendingList().delete_pending("pending")
+        elif status == "ignore":
+            PendingList().delete_pending("ignore")
+        return {"success": True}
+
+    @staticmethod
+    def _manual_import():
+        """run manual import from settings page"""
+        print("starting manual import")
+        run_manual_import.delay()
+        return {"success": True}
+
+    @staticmethod
+    def _re_embed():
+        """rewrite thumbnails into media files"""
+        print("start video thumbnail embed process")
+        re_sync_thumbs.delay()
+        return {"success": True}
+
+    @staticmethod
+    def _db_backup():
+        """backup es to zip from settings page"""
+        print("backing up database")
+        run_backup.delay("manual")
+        return {"success": True}
+
+    def _db_restore(self):
+        """restore es zip from settings page"""
+        print("restoring index from backup zip")
+        filename = self.exec_val
+        run_restore_backup.delay(filename)
+        return {"success": True}
+
+    @staticmethod
+    def _fs_rescan():
+        """start file system rescan task"""
+        print("start filesystem scan")
+        rescan_filesystem.delay()
+        return {"success": True}
+
+    def _multi_search(self):
+        """search through all indexes"""
+        search_query = self.exec_val
+        print("searching for: " + search_query)
+        search_results = SearchForm().multi_search(search_query)
+        return search_results
+
+    def _delete_video(self):
+        """delete media file, metadata and thumb"""
+        youtube_id = self.exec_val
+        YoutubeVideo(youtube_id).delete_media_file()
+        return {"success": True}
+
+    def _delete_channel(self):
+        """delete channel and all matching videos"""
+        channel_id = self.exec_val
+        YoutubeChannel(channel_id).delete_channel()
+        return {"success": True}
+
+    def _delete_playlist(self):
+        """delete playlist, only metadata or incl all videos"""
+        playlist_dict = self.exec_val
+        playlist_id = playlist_dict["playlist-id"]
+        playlist_action = playlist_dict["playlist-action"]
+        print(f"delete {playlist_action} from playlist {playlist_id}")
+        if playlist_action == "metadata":
+            YoutubePlaylist(playlist_id).delete_metadata()
+        elif playlist_action == "all":
+            YoutubePlaylist(playlist_id).delete_videos_playlist()
+
+        return {"success": True}
+
+    def _find_playlists(self):
+        """add all playlists of a channel"""
+        channel_id = self.exec_val
+        index_channel_playlists.delay(channel_id)
+        return {"success": True}

tubearchivist/home/src/helper.py

@@ -9,18 +9,20 @@ import re
 import string
 import subprocess
 import unicodedata
+from urllib.parse import parse_qs, urlparse
 
 import redis
 import requests
+import yt_dlp
 
 
-def get_total_hits(index, es_url, match_field):
+def get_total_hits(index, es_url, es_auth, match_field):
     """get total hits from index"""
     headers = {"Content-type": "application/json"}
     data = {"query": {"match": {match_field: True}}}
     payload = json.dumps(data)
     url = f"{es_url}/{index}/_search?filter_path=hits.total"
-    request = requests.post(url, data=payload, headers=headers)
+    request = requests.post(url, data=payload, headers=headers, auth=es_auth)
     if not request.ok:
         print(request.text)
     total_json = json.loads(request.text)
@@ -51,43 +53,117 @@ def ignore_filelist(filelist):
     return cleaned
 
 
-def process_url_list(url_str):
-    """parse url_list to find valid youtube video or channel ids"""
-    to_replace = ["watch?v=", "playlist?list="]
-    url_list = re.split("\n+", url_str[0])
-    youtube_ids = []
-    for url in url_list:
-        if "/c/" in url or "/user/" in url:
-            raise ValueError("user name is not unique, use channel ID")
+class UrlListParser:
+    """take a multi line string and detect valid youtube ids"""
 
-        url_clean = url.strip().strip("/").split("/")[-1]
-        for i in to_replace:
-            url_clean = url_clean.replace(i, "")
-        url_no_param = url_clean.split("&")[0]
-        str_len = len(url_no_param)
+    def __init__(self, url_str):
+        self.url_list = [i.strip() for i in url_str.split()]
+
+    def process_list(self):
+        """loop through the list"""
+        youtube_ids = []
+        for url in self.url_list:
+            parsed = urlparse(url)
+            print(f"processing: {url}")
+            print(parsed)
+            if not parsed.netloc:
+                # is not a url
+                id_type = self.find_valid_id(url)
+                youtube_id = url
+            elif "youtube.com" not in url and "youtu.be" not in url:
+                raise ValueError(f"{url} is not a youtube link")
+            elif parsed.path:
+                # is a url
+                youtube_id, id_type = self.detect_from_url(parsed)
+            else:
+                # not detected
+                raise ValueError(f"failed to detect {url}")
+
+            youtube_ids.append({"url": youtube_id, "type": id_type})
+
+        return youtube_ids
+
+    def detect_from_url(self, parsed):
+        """detect from parsed url"""
+        if parsed.netloc == "youtu.be":
+            # shortened
+            youtube_id = parsed.path.strip("/")
+            _ = self.find_valid_id(youtube_id)
+            return youtube_id, "video"
+
+        if parsed.query:
+            # detect from query string
+            query_parsed = parse_qs(parsed.query)
+            if "v" in query_parsed.keys():
+                youtube_id = query_parsed["v"][0]
+                _ = self.find_valid_id(youtube_id)
+                return youtube_id, "video"
+
+            if "list" in query_parsed.keys():
+                youtube_id = query_parsed["list"][0]
+                return youtube_id, "playlist"
+
+        if parsed.path.startswith("/channel/"):
+            # channel id in url
+            youtube_id = parsed.path.split("/")[2]
+            _ = self.find_valid_id(youtube_id)
+            return youtube_id, "channel"
+
+        # dedect channel with yt_dlp
+        youtube_id = self.extract_channel_name(parsed.geturl())
+        return youtube_id, "channel"
+
+    @staticmethod
+    def find_valid_id(id_str):
+        """dedect valid id from length of string"""
+        str_len = len(id_str)
         if str_len == 11:
-            link_type = "video"
+            id_type = "video"
         elif str_len == 24:
-            link_type = "channel"
-        elif str_len == 34:
-            link_type = "playlist"
+            id_type = "channel"
+        elif str_len in [34, 18]:
+            id_type = "playlist"
         else:
             # unable to parse
-            raise ValueError("not a valid url: " + url)
+            raise ValueError("not a valid id_str: " + id_str)
 
-        youtube_ids.append({"url": url_no_param, "type": link_type})
+        return id_type
 
-    return youtube_ids
+    @staticmethod
+    def extract_channel_name(url):
+        """find channel id from channel name with yt-dlp help"""
+        obs = {
+            "default_search": "ytsearch",
+            "quiet": True,
+            "skip_download": True,
+            "extract_flat": True,
+            "playlistend": 0,
+        }
+        url_info = yt_dlp.YoutubeDL(obs).extract_info(url, download=False)
+        try:
+            channel_id = url_info["channel_id"]
+        except KeyError as error:
+            print(f"failed to extract channel id from {url}")
+            raise ValueError from error
+
+        return channel_id
 
 
 class RedisArchivist:
     """collection of methods to interact with redis"""
 
     REDIS_HOST = os.environ.get("REDIS_HOST")
-    REDIS_PORT = os.environ.get("REDIS_PORT")
-
-    if not REDIS_PORT:
-        REDIS_PORT = 6379
+    REDIS_PORT = os.environ.get("REDIS_PORT") or 6379
+    NAME_SPACE = "ta:"
+    CHANNELS = [
+        "download",
+        "add",
+        "rescan",
+        "subchannel",
+        "subplaylist",
+        "playlistscan",
+        "setting",
+    ]
 
     def __init__(self):
         self.redis_connection = redis.Redis(
@@ -97,15 +173,23 @@ class RedisArchivist:
     def set_message(self, key, message, expire=True):
         """write new message to redis"""
         self.redis_connection.execute_command(
-            "JSON.SET", key, ".", json.dumps(message)
+            "JSON.SET", self.NAME_SPACE + key, ".", json.dumps(message)
         )
 
         if expire:
-            self.redis_connection.execute_command("EXPIRE", key, 20)
+            if isinstance(expire, bool):
+                secs = 20
+            else:
+                secs = expire
+            self.redis_connection.execute_command(
+                "EXPIRE", self.NAME_SPACE + key, secs
+            )
 
     def get_message(self, key):
         """get message dict from redis"""
-        reply = self.redis_connection.execute_command("JSON.GET", key)
+        reply = self.redis_connection.execute_command(
+            "JSON.GET", self.NAME_SPACE + key
+        )
         if reply:
             json_str = json.loads(reply)
         else:
@@ -115,27 +199,29 @@ class RedisArchivist:
 
     def del_message(self, key):
         """delete key from redis"""
-        response = self.redis_connection.execute_command("DEL", key)
+        response = self.redis_connection.execute_command(
+            "DEL", self.NAME_SPACE + key
+        )
         return response
 
     def get_lock(self, lock_key):
         """handle lock for task management"""
-        redis_lock = self.redis_connection.lock(lock_key)
+        redis_lock = self.redis_connection.lock(self.NAME_SPACE + lock_key)
         return redis_lock
 
-    def get_dl_message(self, cache_dir):
-        """get latest download progress message if available"""
-        reply = self.redis_connection.execute_command(
-            "JSON.GET", "progress:download"
-        )
-        if reply:
-            json_str = json.loads(reply)
-        elif json_str := self.monitor_cache_dir(cache_dir):
-            json_str = self.monitor_cache_dir(cache_dir)
-        else:
-            json_str = {"status": False}
+    def get_progress(self):
+        """get a list of all progress messages"""
+        all_messages = []
+        for channel in self.CHANNELS:
+            key = "message:" + channel
+            reply = self.redis_connection.execute_command(
+                "JSON.GET", self.NAME_SPACE + key
+            )
+            if reply:
+                json_str = json.loads(reply)
+                all_messages.append(json_str)
 
-        return json_str
+        return all_messages
 
     @staticmethod
     def monitor_cache_dir(cache_dir):
@@ -148,7 +234,7 @@ class RedisArchivist:
         if cache_file:
             filename = cache_file[0][12:].replace("_", " ").split(".")[0]
             mess_dict = {
-                "status": "downloading",
+                "status": "message:download",
                 "level": "info",
                 "title": "Downloading: " + filename,
                 "message": "",
@@ -164,12 +250,13 @@ class RedisQueue:
 
     REDIS_HOST = os.environ.get("REDIS_HOST")
     REDIS_PORT = os.environ.get("REDIS_PORT")
+    NAME_SPACE = "ta:"
 
     if not REDIS_PORT:
         REDIS_PORT = 6379
 
     def __init__(self, key):
-        self.key = key
+        self.key = self.NAME_SPACE + key
         self.conn = redis.Redis(host=self.REDIS_HOST, port=self.REDIS_PORT)
 
     def get_all(self):

tubearchivist/home/src/index.py

@@ -12,11 +12,12 @@ from datetime import datetime
 from time import sleep
 
 import requests
-import yt_dlp as youtube_dl
+import yt_dlp
 from bs4 import BeautifulSoup
 from home.src.config import AppConfig
-from home.src.helper import DurationConverter, clean_string, process_url_list
+from home.src.helper import DurationConverter, UrlListParser, clean_string
 from home.src.thumbnails import ThumbManager
+from ryd_client import ryd_client
 
 
 class YoutubeChannel:
@@ -24,6 +25,7 @@ class YoutubeChannel:
 
     CONFIG = AppConfig().config
     ES_URL = CONFIG["application"]["es_url"]
+    ES_AUTH = CONFIG["application"]["es_auth"]
     CACHE_DIR = CONFIG["application"]["cache_dir"]
     VIDEOS = CONFIG["application"]["videos"]
 
@@ -51,7 +53,7 @@ class YoutubeChannel:
         """get from elastic search first if possible"""
         channel_id = self.channel_id
         url = f"{self.ES_URL}/ta_channel/_doc/{channel_id}"
-        response = requests.get(url)
+        response = requests.get(url, auth=self.ES_AUTH)
         if response.ok:
             channel_source = response.json()["_source"]
             self.source = "elastic"
@@ -63,7 +65,7 @@ class YoutubeChannel:
         channel_id = self.channel_id
         url = f"https://www.youtube.com/channel/{channel_id}/about?hl=en"
         cookies = {"CONSENT": "YES+xxxxxxxxxxxxxxxxxxxxxxxxxxx"}
-        response = requests.get(url, cookies=cookies)
+        response = requests.get(url, cookies=cookies, auth=self.ES_AUTH)
         if response.ok:
             channel_page = response.text
         else:
@@ -166,10 +168,11 @@ class YoutubeChannel:
     def upload_to_es(self):
         """upload channel data to elastic search"""
         url = f"{self.ES_URL}/ta_channel/_doc/{self.channel_id}"
-        response = requests.put(url, json=self.channel_dict)
+        response = requests.put(url, json=self.channel_dict, auth=self.ES_AUTH)
         print(f"added {self.channel_id} to es")
         if not response.ok:
             print(response.text)
+            raise ValueError("failed to add channel to index")
 
     def sync_to_videos(self):
         """sync new channel_dict to all videos of channel"""
@@ -183,14 +186,18 @@ class YoutubeChannel:
         data = {"description": channel_id, "processors": processors}
         payload = json.dumps(data)
         url = self.ES_URL + "/_ingest/pipeline/" + channel_id
-        request = requests.put(url, data=payload, headers=headers)
+        request = requests.put(
+            url, data=payload, headers=headers, auth=self.ES_AUTH
+        )
         if not request.ok:
             print(request.text)
         # apply pipeline
         data = {"query": {"match": {"channel.channel_id": channel_id}}}
         payload = json.dumps(data)
         url = self.ES_URL + "/ta_video/_update_by_query?pipeline=" + channel_id
-        request = requests.post(url, data=payload, headers=headers)
+        request = requests.post(
+            url, data=payload, headers=headers, auth=self.ES_AUTH
+        )
         if not request.ok:
             print(request.text)
 
@@ -211,35 +218,77 @@ class YoutubeChannel:
         }
         payload = json.dumps(data)
         url = self.ES_URL + "/ta_video/_delete_by_query"
-        response = requests.post(url, data=payload, headers=headers)
+        response = requests.post(
+            url, data=payload, headers=headers, auth=self.ES_AUTH
+        )
         if not response.ok:
             print(response.text)
 
+    def delete_playlists(self):
+        """delete all indexed playlist from es"""
+        all_playlists = self.get_indexed_playlists()
+        for playlist in all_playlists:
+            playlist_id = playlist["playlist_id"]
+            YoutubePlaylist(playlist_id).delete_metadata()
+
     def delete_channel(self):
         """delete channel and all videos"""
         print(f"deleting {self.channel_id} and all matching media files")
         folder_path = self.get_folder_path()
         print("delete all media files")
-        all_videos = os.listdir(folder_path)
-        for video in all_videos:
-            video_path = os.path.join(folder_path, video)
-            os.remove(video_path)
-        os.rmdir(folder_path)
-        ThumbManager().delete_chan_thumb(self.channel_id)
+        try:
+            all_videos = os.listdir(folder_path)
+            for video in all_videos:
+                video_path = os.path.join(folder_path, video)
+                os.remove(video_path)
+            os.rmdir(folder_path)
+        except FileNotFoundError:
+            print(f"no videos found for {folder_path}")
 
+        ThumbManager().delete_chan_thumb(self.channel_id)
+        print("delete indexed playlists")
+        self.delete_playlists()
         print("delete indexed videos")
         self.delete_es_videos()
         url = self.ES_URL + "/ta_channel/_doc/" + self.channel_id
-        response = requests.delete(url)
+        response = requests.delete(url, auth=self.ES_AUTH)
         if not response.ok:
             print(response.text)
 
+    def get_all_playlists(self):
+        """get all playlists owned by this channel"""
+        url = (
+            f"https://www.youtube.com/channel/{self.channel_id}"
+            + "/playlists?view=1&sort=dd&shelf_id=0"
+        )
+        obs = {
+            "quiet": True,
+            "skip_download": True,
+            "extract_flat": True,
+        }
+        playlists = yt_dlp.YoutubeDL(obs).extract_info(url)
+        all_entries = [(i["id"], i["title"]) for i in playlists["entries"]]
+
+        return all_entries
+
+    def get_indexed_playlists(self):
+        """get all indexed playlists from channel"""
+        data = {
+            "query": {
+                "term": {"playlist_channel_id": {"value": self.channel_id}}
+            },
+            "sort": [{"playlist_channel.keyword": {"order": "desc"}}],
+        }
+        all_playlists = IndexPaginate("ta_playlist", data).get_results()
+        return all_playlists
+
 
 class YoutubeVideo:
     """represents a single youtube video"""
 
     CONFIG = AppConfig().config
     ES_URL = CONFIG["application"]["es_url"]
+    ES_AUTH = CONFIG["application"]["es_auth"]
     CACHE_DIR = CONFIG["application"]["cache_dir"]
     VIDEOS = CONFIG["application"]["videos"]
 
@@ -249,7 +298,7 @@ class YoutubeVideo:
         self.vid_dict = None
 
     def get_vid_dict(self):
-        """wrapper to loop around youtube_dl to retry on failure"""
+        """wrapper to loop around yt_dlp to retry on failure"""
         print(f"get video data for {self.youtube_id}")
         vid_dict = False
         for i in range(3):
@@ -263,6 +312,8 @@ class YoutubeVideo:
                 break
 
         self.vid_dict = vid_dict
+        if self.CONFIG["downloads"]["integrate_ryd"]:
+            self.get_ryd_stats()
 
     def get_youtubedl_vid_data(self):
         """parse youtubedl extract info"""
@@ -271,12 +322,14 @@ class YoutubeVideo:
             "quiet": True,
             "default_search": "ytsearch",
             "skip_download": True,
+            "check_formats": "selected",
+            "noplaylist": True,
         }
         try:
-            vid = youtube_dl.YoutubeDL(obs).extract_info(youtube_id)
+            vid = yt_dlp.YoutubeDL(obs).extract_info(youtube_id)
         except (
-            youtube_dl.utils.ExtractorError,
-            youtube_dl.utils.DownloadError,
+            yt_dlp.utils.ExtractorError,
+            yt_dlp.utils.DownloadError,
         ):
             print("failed to get info for " + youtube_id)
             return False
@@ -360,18 +413,19 @@ class YoutubeVideo:
     def get_es_data(self):
         """get current data from elastic search"""
         url = self.ES_URL + "/ta_video/_doc/" + self.youtube_id
-        response = requests.get(url)
+        response = requests.get(url, auth=self.ES_AUTH)
         if not response.ok:
             print(response.text)
         es_vid_dict = json.loads(response.text)
         return es_vid_dict
 
     def upload_to_es(self):
-        """upload channel data to elastic search"""
-        url = f"{self.ES_URL}/ta_video/_doc/{self.youtube_id}"
-        response = requests.put(url, json=self.vid_dict)
+        """upload video data to elastic search"""
+        url = f"{self.ES_URL}/ta_video/_doc/{self.youtube_id}/?refresh=true"
+        response = requests.put(url, json=self.vid_dict, auth=self.ES_AUTH)
         if not response.ok:
             print(response.text)
+            raise ValueError("failed to add video to index")
 
     def deactivate(self):
         """deactivate document on extractor error"""
@@ -380,7 +434,9 @@ class YoutubeVideo:
         url = f"{self.ES_URL}/ta_video/_update/{youtube_id}"
         data = {"script": "ctx._source.active = false"}
         json_str = json.dumps(data)
-        response = requests.post(url, data=json_str, headers=headers)
+        response = requests.post(
+            url, data=json_str, headers=headers, auth=self.ES_AUTH
+        )
         print(f"deactivated {youtube_id}")
         if not response.ok:
             print(response.text)
@@ -395,18 +451,308 @@ class YoutubeVideo:
         os.remove(to_delete)
         # delete from index
         url = f"{self.ES_URL}/ta_video/_doc/{self.youtube_id}"
-        response = requests.delete(url)
+        response = requests.delete(url, auth=self.ES_AUTH)
         if not response.ok:
             print(response.text)
         # delete thumbs from cache
         ThumbManager().delete_vid_thumb(self.youtube_id)
 
+    def get_ryd_stats(self):
+        """get optional stats from returnyoutubedislikeapi.com"""
+        try:
+            print(f"get ryd stats for: {self.youtube_id}")
+            result = ryd_client.get(self.youtube_id)
+        except requests.exceptions.ConnectionError:
+            print(f"failed to query ryd api, skipping {self.youtube_id}")
+            return False
+
+        dislikes = {
+            "dislike_count": result["dislikes"],
+            "average_rating": result["rating"],
+        }
+        self.vid_dict["stats"].update(dislikes)
+
+        return True
+
+
+class YoutubePlaylist:
+    """represent a single playlist on YouTube"""
+
+    CONFIG = AppConfig().config
+    ES_URL = CONFIG["application"]["es_url"]
+    ES_AUTH = CONFIG["application"]["es_auth"]
+
+    def __init__(self, playlist_id, all_youtube_ids=False):
+        self.playlist_id = playlist_id
+        self.stamp = int(datetime.now().strftime("%s"))
+        self.all_youtube_ids = all_youtube_ids
+        self.playlist_dict = False
+
+    def get_playlist_dict(self, scrape=False):
+        """get data from es or youtube"""
+        print(f"get playlist with id {self.playlist_id}")
+
+        if scrape:
+            playlist_dict = self.get_youtube_playlist()
+            if not playlist_dict:
+                return False
+            playlist_dict["playlist_entries"] = self.get_entries()
+        else:
+            playlist_dict = self.get_es_playlist()
+            if not playlist_dict:
+                playlist_dict = self.get_youtube_playlist()
+                playlist_dict["playlist_entries"] = self.get_entries()
+
+        self.playlist_dict = playlist_dict
+        return True
+
+    def get_youtube_playlist(self):
+        """get meta data dict from youtube"""
+        url = "https://www.youtube.com/playlist?list=" + self.playlist_id
+        obs = {
+            "default_search": "ytsearch",
+            "quiet": True,
+            "skip_download": True,
+            "extract_flat": True,
+            "playlistend": 0,
+        }
+        try:
+            playlist = yt_dlp.YoutubeDL(obs).extract_info(url, download=False)
+        except (
+            yt_dlp.utils.ExtractorError,
+            yt_dlp.utils.DownloadError,
+        ):
+            print("failed to get info for " + self.playlist_id)
+            return False
+
+        playlist_es = {
+            "playlist_id": self.playlist_id,
+            "playlist_active": True,
+            "playlist_subscribed": False,
+            "playlist_name": playlist["title"],
+            "playlist_channel": playlist["channel"],
+            "playlist_channel_id": playlist["channel_id"],
+            "playlist_thumbnail": playlist["thumbnails"][-1]["url"],
+            "playlist_description": playlist["description"] or False,
+            "playlist_last_refresh": self.stamp,
+        }
+
+        return playlist_es
+
+    def get_es_playlist(self):
+        """get indexed data from es"""
+        url = f"{self.ES_URL}/ta_playlist/_doc/{self.playlist_id}"
+        response = requests.get(url, auth=self.ES_AUTH)
+        if response.ok:
+            return json.loads(response.text)["_source"]
+
+        return False
+
+    def get_entries(self, playlistend=False):
+        """get all videos in playlist"""
+        url = "https://www.youtube.com/playlist?list=" + self.playlist_id
+        obs = {
+            "default_search": "ytsearch",
+            "quiet": True,
+            "skip_download": True,
+            "extract_flat": True,
+        }
+        if playlistend:
+            obs["playlistend"] = playlistend
+
+        try:
+            playlist = yt_dlp.YoutubeDL(obs).extract_info(url, download=False)
+        except (
+            yt_dlp.utils.ExtractorError,
+            yt_dlp.utils.DownloadError,
+        ):
+            print("failed to get plealist entries for " + self.playlist_id)
+            return False
+
+        all_members = []
+        for idx, entry in enumerate(playlist["entries"]):
+            uploader = entry["uploader"]
+            youtube_id = entry["id"]
+            if self.all_youtube_ids:
+                downloaded = youtube_id in self.all_youtube_ids
+            else:
+                downloaded = False
+            if not uploader:
+                continue
+            to_append = {
+                "youtube_id": youtube_id,
+                "title": entry["title"],
+                "uploader": uploader,
+                "idx": idx,
+                "downloaded": downloaded,
+            }
+            all_members.append(to_append)
+
+        return all_members
+
+    def upload_to_es(self):
+        """add playlist to es with its entries"""
+        playlist = self.playlist_dict
+        url = f"{self.ES_URL}/ta_playlist/_doc/{self.playlist_id}"
+        response = requests.put(url, json=playlist, auth=self.ES_AUTH)
+        if not response.ok:
+            print(response.text)
+            raise ValueError("failed to add playlist to index")
+
+    def add_vids_to_playlist(self):
+        """sync the playlist id to videos"""
+        playlist_dict = self.playlist_dict
+        script = (
+            'if (!ctx._source.containsKey("playlist")) '
+            + "{ctx._source.playlist = [params.playlist]} "
+            + "else if (!ctx._source.playlist.contains(params.playlist)) "
+            + "{ctx._source.playlist.add(params.playlist)} "
+            + "else {ctx.op = 'none'}"
+        )
+
+        bulk_list = []
+        for entry in playlist_dict["playlist_entries"]:
+            youtube_id = entry["youtube_id"]
+            action = {"update": {"_id": youtube_id, "_index": "ta_video"}}
+            source = {
+                "script": {
+                    "source": script,
+                    "lang": "painless",
+                    "params": {"playlist": self.playlist_id},
+                }
+            }
+            bulk_list.append(json.dumps(action))
+            bulk_list.append(json.dumps(source))
+
+        # add last newline
+        bulk_list.append("\n")
+        query_str = "\n".join(bulk_list)
+        headers = {"Content-type": "application/x-ndjson"}
+        url = self.ES_URL + "/_bulk"
+        response = requests.post(
+            url, data=query_str, headers=headers, auth=self.ES_AUTH
+        )
+        if not response.ok:
+            print(response.text)
+
+    def update_playlist(self):
+        """update metadata for playlist with data from YouTube"""
+        subscribed = self.get_es_playlist()["playlist_subscribed"]
+        self.get_playlist_dict(scrape=True)
+        if not self.playlist_dict:
+            # return false to deactivate
+            return False
+
+        self.playlist_dict["playlist_subscribed"] = subscribed
+        self.upload_to_es()
+        return self.playlist_dict
+
+    def build_nav(self, youtube_id):
+        """find next and previous in playlist of a given youtube_id"""
+        all_entries_available = self.playlist_dict["playlist_entries"]
+        all_entries = [i for i in all_entries_available if i["downloaded"]]
+        current = [i for i in all_entries if i["youtube_id"] == youtube_id]
+        # stop if not found or playlist of 1
+        if not current or not len(all_entries) > 1:
+            return False
+
+        current_idx = all_entries.index(current[0])
+        if current_idx == 0:
+            previous_item = False
+        else:
+            previous_item = all_entries[current_idx - 1]
+            prev_thumb = ThumbManager().vid_thumb_path(
+                previous_item["youtube_id"]
+            )
+            previous_item["vid_thumb"] = prev_thumb
+
+        if current_idx == len(all_entries) - 1:
+            next_item = False
+        else:
+            next_item = all_entries[current_idx + 1]
+            next_thumb = ThumbManager().vid_thumb_path(next_item["youtube_id"])
+            next_item["vid_thumb"] = next_thumb
+
+        nav = {
+            "playlist_meta": {
+                "current_idx": current[0]["idx"],
+                "playlist_id": self.playlist_id,
+                "playlist_name": self.playlist_dict["playlist_name"],
+                "playlist_channel": self.playlist_dict["playlist_channel"],
+            },
+            "playlist_previous": previous_item,
+            "playlist_next": next_item,
+        }
+        return nav
+
+    def delete_metadata(self):
+        """delete metadata for playlist"""
+        script = (
+            "ctx._source.playlist.removeAll("
+            + "Collections.singleton(params.playlist)) "
+        )
+        data = {
+            "query": {
+                "term": {"playlist.keyword": {"value": self.playlist_id}}
+            },
+            "script": {
+                "source": script,
+                "lang": "painless",
+                "params": {"playlist": self.playlist_id},
+            },
+        }
+        payload = json.dumps(data)
+        url = f"{self.ES_URL}/ta_video/_update_by_query"
+        headers = {"Content-type": "application/json"}
+        response = requests.post(
+            url, data=payload, headers=headers, auth=self.ES_AUTH
+        )
+        if not response.ok:
+            print(response.text)
+
+        self.delete_playlist()
+
+    def delete_videos_playlist(self):
+        """delete playlist with all videos"""
+        print(f"delete playlist {self.playlist_id} with all videos")
+        self.get_playlist_dict()
+        all_youtube_id = [
+            i["youtube_id"]
+            for i in self.playlist_dict["playlist_entries"]
+            if i["downloaded"]
+        ]
+        for youtube_id in all_youtube_id:
+            YoutubeVideo(youtube_id).delete_media_file()
+
+        self.delete_playlist()
+
+    def delete_playlist(self):
+        """delete only playlist document"""
+        url = f"{self.ES_URL}/ta_playlist/_doc/{self.playlist_id}"
+        response = requests.delete(url, auth=self.ES_AUTH)
+        if not response.ok:
+            print(response.text)
+
+    def deactivate(self):
+        """deactivate document on extractor error"""
+        headers = {"Content-type": "application/json"}
+        url = f"{self.ES_URL}/ta_playlist/_update/{self.playlist_id}"
+        data = {"script": "ctx._source.playlist_active = false"}
+        json_str = json.dumps(data)
+        response = requests.post(
+            url, data=json_str, headers=headers, auth=self.ES_AUTH
+        )
+        print(f"deactivated {self.playlist_id}")
+        if not response.ok:
+            print(response.text)
+
 
 class WatchState:
     """handle watched checkbox for videos and channels"""
 
     CONFIG = AppConfig().config
     ES_URL = CONFIG["application"]["es_url"]
+    ES_AUTH = CONFIG["application"]["es_auth"]
     HEADERS = {"Content-type": "application/json"}
 
     def __init__(self, youtube_id):
@@ -420,6 +766,8 @@ class WatchState:
             self.mark_vid_watched()
         elif url_type == "channel":
             self.mark_channel_watched()
+        elif url_type == "playlist":
+            self.mark_playlist_watched()
 
         print(f"marked {self.youtube_id} as watched")
 
@@ -428,16 +776,14 @@ class WatchState:
         url_type = self.dedect_type()
         if url_type == "video":
             self.mark_vid_watched(revert=True)
-        elif url_type == "channel":
-            self.mark_channel_watched(revert=True)
 
         print(f"revert {self.youtube_id} as unwatched")
 
     def dedect_type(self):
         """find youtube id type"""
-        url_process = process_url_list([self.youtube_id])
+        print(self.youtube_id)
+        url_process = UrlListParser(self.youtube_id).process_list()
         url_type = url_process[0]["type"]
-
         return url_type
 
     def mark_vid_watched(self, revert=False):
@@ -450,44 +796,150 @@ class WatchState:
             data["doc"]["player"]["watched"] = False
 
         payload = json.dumps(data)
-        request = requests.post(url, data=payload, headers=self.HEADERS)
+        request = requests.post(
+            url, data=payload, headers=self.HEADERS, auth=self.ES_AUTH
+        )
         if not request.ok:
             print(request.text)
+            raise ValueError("failed to mark video as watched")
 
-    def mark_channel_watched(self, revert=False):
+    def mark_channel_watched(self):
         """change watched status of every video in channel"""
-        es_url = self.ES_URL
-        headers = self.HEADERS
-        youtube_id = self.youtube_id
-        # create pipeline
         data = {
-            "description": youtube_id,
-            "processors": [
-                {"set": {"field": "player.watched", "value": True}},
-                {"set": {"field": "player.watched_date", "value": self.stamp}},
-            ],
+            "query": {
+                "bool": {
+                    "must": [
+                        {
+                            "term": {
+                                "channel.channel_id": {
+                                    "value": self.youtube_id
+                                }
+                            }
+                        },
+                        {"term": {"player.watched": {"value": False}}},
+                    ]
+                }
+            },
+            "script": {
+                "source": "ctx._source.player['watched'] = true",
+                "lang": "painless",
+            },
         }
-        if revert:
-            data["processors"][0]["set"]["value"] = False
-
         payload = json.dumps(data)
-        url = f"{es_url}/_ingest/pipeline/{youtube_id}"
-        request = requests.put(url, data=payload, headers=headers)
+        url = f"{self.ES_URL}/ta_video/_update_by_query"
+        request = requests.post(
+            url, data=payload, headers=self.HEADERS, auth=self.ES_AUTH
+        )
         if not request.ok:
             print(request.text)
-            raise ValueError("failed to post ingest pipeline")
+            raise ValueError("failed mark channel as watched")
 
-        # apply pipeline
-        must_list = [
-            {"term": {"channel.channel_id": {"value": youtube_id}}},
-            {"term": {"player.watched": {"value": False}}},
-        ]
-        data = {"query": {"bool": {"must": must_list}}}
+    def mark_playlist_watched(self):
+        """change watched state of all videos in playlist"""
+        data = {
+            "query": {
+                "bool": {
+                    "must": [
+                        {
+                            "term": {
+                                "playlist.keyword": {"value": self.youtube_id}
+                            }
+                        },
+                        {"term": {"player.watched": {"value": False}}},
+                    ]
+                }
+            },
+            "script": {
+                "source": "ctx._source.player['watched'] = true",
+                "lang": "painless",
+            },
+        }
         payload = json.dumps(data)
-        url = f"{es_url}/ta_video/_update_by_query?pipeline={youtube_id}"
-        request = requests.post(url, data=payload, headers=headers)
+        url = f"{self.ES_URL}/ta_video/_update_by_query"
+        request = requests.post(
+            url, data=payload, headers=self.HEADERS, auth=self.ES_AUTH
+        )
         if not request.ok:
             print(request.text)
+            raise ValueError("failed mark playlist as watched")
+
+
+class IndexPaginate:
+    """use search_after to go through whole index"""
+
+    CONFIG = AppConfig().config
+    ES_URL = CONFIG["application"]["es_url"]
+    ES_AUTH = CONFIG["application"]["es_auth"]
+    HEADERS = {"Content-type": "application/json"}
+    DEFAULT_SIZE = 500
+
+    def __init__(self, index_name, data, size=False):
+        self.index_name = index_name
+        self.data = data
+        self.pit_id = False
+        self.size = size
+
+    def get_results(self):
+        """get all results"""
+        self.get_pit()
+        self.validate_data()
+        all_results = self.run_loop()
+        self.clean_pit()
+        return all_results
+
+    def get_pit(self):
+        """get pit for index"""
+        url = f"{self.ES_URL}/{self.index_name}/_pit?keep_alive=10m"
+        response = requests.post(url, auth=self.ES_AUTH)
+        json_data = json.loads(response.text)
+
+        self.pit_id = json_data["id"]
+
+    def validate_data(self):
+        """add pit and size to data"""
+        if "sort" not in self.data.keys():
+            print(self.data)
+            raise ValueError("missing sort key in data")
+
+        size = self.size or self.DEFAULT_SIZE
+
+        self.data["size"] = size
+        self.data["pit"] = {"id": self.pit_id, "keep_alive": "10m"}
+
+    def run_loop(self):
+        """loop through results until last hit"""
+        query_str = json.dumps(self.data)
+        url = self.ES_URL + "/_search"
+
+        all_results = []
+        while True:
+            response = requests.get(
+                url, data=query_str, headers=self.HEADERS, auth=self.ES_AUTH
+            )
+            json_data = json.loads(response.text)
+            all_hits = json_data["hits"]["hits"]
+            if all_hits:
+                for hit in all_hits:
+                    source = hit["_source"]
+                    search_after = hit["sort"]
+                    all_results.append(source)
+                # update search_after with last hit data
+                self.data["search_after"] = search_after
+                query_str = json.dumps(self.data)
+            else:
+                break
+
+        return all_results
+
+    def clean_pit(self):
+        """delete pit from elastic search"""
+        query_str = json.dumps({"id": self.pit_id})
+        requests.delete(
+            self.ES_URL + "/_pit",
+            data=query_str,
+            headers=self.HEADERS,
+            auth=self.ES_AUTH,
+        )
 
 
 def index_new_video(youtube_id, missing_vid=False):

tubearchivist/home/src/index_management.py

@@ -25,6 +25,7 @@ INDEX_CONFIG = [
             },
             "channel_name": {
                 "type": "text",
+                "analyzer": "english",
                 "fields": {
                     "keyword": {
                         "type": "keyword",
@@ -64,6 +65,7 @@ INDEX_CONFIG = [
                     },
                     "channel_name": {
                         "type": "text",
+                        "analyzer": "english",
                         "fields": {
                             "keyword": {
                                 "type": "keyword",
@@ -88,8 +90,16 @@ INDEX_CONFIG = [
             },
             "description": {"type": "text"},
             "media_url": {"type": "keyword", "index": False},
+            "tags": {
+                "type": "text",
+                "analyzer": "english",
+                "fields": {
+                    "keyword": {"type": "keyword", "ignore_above": 256}
+                },
+            },
             "title": {
                 "type": "text",
+                "analyzer": "english",
                 "fields": {
                     "keyword": {
                         "type": "keyword",
@@ -106,6 +116,16 @@ INDEX_CONFIG = [
             "vid_last_refresh": {"type": "date"},
             "youtube_id": {"type": "keyword"},
             "published": {"type": "date"},
+            "playlist": {
+                "type": "text",
+                "fields": {
+                    "keyword": {
+                        "type": "keyword",
+                        "ignore_above": 256,
+                        "normalizer": "to_lower",
+                    }
+                },
+            },
         },
         "expected_set": {
             "analysis": {
@@ -154,6 +174,50 @@ INDEX_CONFIG = [
             "number_of_replicas": "0",
         },
     },
+    {
+        "index_name": "playlist",
+        "expected_map": {
+            "playlist_id": {"type": "keyword"},
+            "playlist_description": {"type": "text"},
+            "playlist_name": {
+                "type": "text",
+                "analyzer": "english",
+                "fields": {
+                    "keyword": {
+                        "type": "keyword",
+                        "ignore_above": 256,
+                        "normalizer": "to_lower",
+                    },
+                    "search_as_you_type": {
+                        "type": "search_as_you_type",
+                        "doc_values": False,
+                        "max_shingle_size": 3,
+                    },
+                },
+            },
+            "playlist_channel": {
+                "type": "text",
+                "fields": {
+                    "keyword": {
+                        "type": "keyword",
+                        "ignore_above": 256,
+                        "normalizer": "to_lower",
+                    }
+                },
+            },
+            "playlist_channel_id": {"type": "keyword"},
+            "playlist_thumbnail": {"type": "keyword"},
+            "playlist_last_refresh": {"type": "date"},
+        },
+        "expected_set": {
+            "analysis": {
+                "normalizer": {
+                    "to_lower": {"type": "custom", "filter": ["lowercase"]}
+                }
+            },
+            "number_of_replicas": "0",
+        },
+    },
 ]
 
 
@@ -164,6 +228,7 @@ class ElasticIndex:
 
     CONFIG = AppConfig().config
     ES_URL = CONFIG["application"]["es_url"]
+    ES_AUTH = CONFIG["application"]["es_auth"]
     HEADERS = {"Content-type": "application/json"}
 
     def __init__(self, index_name, expected_map, expected_set):
@@ -176,7 +241,7 @@ class ElasticIndex:
         """check if index already exists and return mapping if it does"""
         index_name = self.index_name
         url = f"{self.ES_URL}/ta_{index_name}"
-        response = requests.get(url)
+        response = requests.get(url, auth=self.ES_AUTH)
         exists = response.ok
 
         if exists:
@@ -274,7 +339,9 @@ class ElasticIndex:
         query = {"source": {"index": source}, "dest": {"index": destination}}
         data = json.dumps(query)
         url = self.ES_URL + "/_reindex?refresh=true"
-        response = requests.post(url=url, data=data, headers=self.HEADERS)
+        response = requests.post(
+            url=url, data=data, headers=self.HEADERS, auth=self.ES_AUTH
+        )
         if not response.ok:
             print(response.text)
 
@@ -284,7 +351,7 @@ class ElasticIndex:
             url = f"{self.ES_URL}/ta_{self.index_name}_backup"
         else:
             url = f"{self.ES_URL}/ta_{self.index_name}"
-        response = requests.delete(url)
+        response = requests.delete(url, auth=self.ES_AUTH)
         if not response.ok:
             print(response.text)
 
@@ -301,7 +368,9 @@ class ElasticIndex:
         # create
         url = f"{self.ES_URL}/ta_{self.index_name}"
         data = json.dumps(payload)
-        response = requests.put(url=url, data=data, headers=self.HEADERS)
+        response = requests.put(
+            url=url, data=data, headers=self.HEADERS, auth=self.ES_AUTH
+        )
         if not response.ok:
             print(response.text)
 
@@ -309,9 +378,10 @@ class ElasticIndex:
 class ElasticBackup:
     """dump index to nd-json files for later bulk import"""
 
-    def __init__(self, index_config):
+    def __init__(self, index_config, reason):
         self.config = AppConfig().config
         self.index_config = index_config
+        self.reason = reason
         self.timestamp = datetime.now().strftime("%Y%m%d")
         self.backup_files = []
 
@@ -319,9 +389,10 @@ class ElasticBackup:
         """export all documents of a single index"""
         headers = {"Content-type": "application/json"}
         es_url = self.config["application"]["es_url"]
+        es_auth = self.config["application"]["es_auth"]
         # get PIT ID
         url = f"{es_url}/ta_{index_name}/_pit?keep_alive=1m"
-        response = requests.post(url)
+        response = requests.post(url, auth=es_auth)
         json_data = json.loads(response.text)
         pit_id = json_data["id"]
         # build query
@@ -336,7 +407,9 @@ class ElasticBackup:
         # loop until nothing left
         all_results = []
         while True:
-            response = requests.get(url, data=query_str, headers=headers)
+            response = requests.get(
+                url, data=query_str, headers=headers, auth=es_auth
+            )
             json_data = json.loads(response.text)
             all_hits = json_data["hits"]["hits"]
             if all_hits:
@@ -350,7 +423,9 @@ class ElasticBackup:
                 break
         # clean up PIT
         query_str = json.dumps({"id": pit_id})
-        requests.delete(es_url + "/_pit", data=query_str, headers=headers)
+        requests.delete(
+            es_url + "/_pit", data=query_str, headers=headers, auth=es_auth
+        )
 
         return all_results
 
@@ -398,7 +473,7 @@ class ElasticBackup:
     def zip_it(self):
         """pack it up into single zip file"""
         cache_dir = self.config["application"]["cache_dir"]
-        file_name = f"ta_backup-{self.timestamp}.zip"
+        file_name = f"ta_backup-{self.timestamp}-{self.reason}.zip"
         backup_folder = os.path.join(cache_dir, "backup")
         backup_file = os.path.join(backup_folder, file_name)
 
@@ -416,6 +491,7 @@ class ElasticBackup:
         """send bulk to es"""
         cache_dir = self.config["application"]["cache_dir"]
         es_url = self.config["application"]["es_url"]
+        es_auth = self.config["application"]["es_auth"]
         headers = {"Content-type": "application/x-ndjson"}
         file_path = os.path.join(cache_dir, file_name)
 
@@ -426,12 +502,14 @@ class ElasticBackup:
             return
 
         url = es_url + "/_bulk"
-        request = requests.post(url, data=query_str, headers=headers)
+        request = requests.post(
+            url, data=query_str, headers=headers, auth=es_auth
+        )
         if not request.ok:
             print(request.text)
 
-    def unpack_zip_backup(self):
-        """extract backup zip and return filelist"""
+    def get_all_backup_files(self):
+        """build all available backup files for view"""
         cache_dir = self.config["application"]["cache_dir"]
         backup_dir = os.path.join(cache_dir, "backup")
         backup_files = os.listdir(backup_dir)
@@ -441,9 +519,32 @@ class ElasticBackup:
             for i in all_backup_files
             if i.startswith("ta_") and i.endswith(".zip")
         ]
-        all_available_backups.sort()
-        newest_backup = all_available_backups[-1]
-        file_path = os.path.join(backup_dir, newest_backup)
+        all_available_backups.sort(reverse=True)
+
+        backup_dicts = []
+        for backup_file in all_available_backups:
+            file_split = backup_file.split("-")
+            if len(file_split) == 2:
+                timestamp = file_split[1].strip(".zip")
+                reason = False
+            elif len(file_split) == 3:
+                timestamp = file_split[1]
+                reason = file_split[2].strip(".zip")
+
+            to_add = {
+                "filename": backup_file,
+                "timestamp": timestamp,
+                "reason": reason,
+            }
+            backup_dicts.append(to_add)
+
+        return backup_dicts
+
+    def unpack_zip_backup(self, filename):
+        """extract backup zip and return filelist"""
+        cache_dir = self.config["application"]["cache_dir"]
+        backup_dir = os.path.join(cache_dir, "backup")
+        file_path = os.path.join(backup_dir, filename)
 
         with zipfile.ZipFile(file_path, "r") as z:
             zip_content = z.namelist()
@@ -469,13 +570,53 @@ class ElasticBackup:
             self.post_bulk_restore(file_name)
             os.remove(file_name)
 
+    def index_exists(self, index_name):
+        """check if index already exists to skip"""
+        es_url = self.config["application"]["es_url"]
+        es_auth = self.config["application"]["es_auth"]
+        url = f"{es_url}/ta_{index_name}"
+        response = requests.get(url, auth=es_auth)
 
-def backup_all_indexes():
+        return response.ok
+
+    def rotate_backup(self):
+        """delete old backups if needed"""
+        rotate = self.config["scheduler"]["run_backup_rotate"]
+        if not rotate:
+            return
+
+        all_backup_files = self.get_all_backup_files()
+        auto = [i for i in all_backup_files if i["reason"] == "auto"]
+
+        if len(auto) <= rotate:
+            print("no backup files to rotate")
+            return
+
+        cache_dir = self.config["application"]["cache_dir"]
+        backup_dir = os.path.join(cache_dir, "backup")
+
+        all_to_delete = auto[rotate:]
+        for to_delete in all_to_delete:
+            file_path = os.path.join(backup_dir, to_delete["filename"])
+            print(f"remove old backup file: {file_path}")
+            os.remove(file_path)
+
+
+def get_available_backups():
+    """return dict of available backups for settings view"""
+    backup_handler = ElasticBackup(INDEX_CONFIG, reason=False)
+    all_backup_files = backup_handler.get_all_backup_files()
+    return all_backup_files
+
+
+def backup_all_indexes(reason):
     """backup all es indexes to disk"""
-    backup_handler = ElasticBackup(INDEX_CONFIG)
+    backup_handler = ElasticBackup(INDEX_CONFIG, reason)
 
     for index in backup_handler.index_config:
         index_name = index["index_name"]
+        if not backup_handler.index_exists(index_name):
+            continue
         all_results = backup_handler.get_all_documents(index_name)
         file_content = backup_handler.build_bulk(all_results)
         backup_handler.write_es_json(file_content, index_name)
@@ -483,14 +624,17 @@ def backup_all_indexes():
 
     backup_handler.zip_it()
 
+    if reason == "auto":
+        backup_handler.rotate_backup()
 
-def restore_from_backup():
+
+def restore_from_backup(filename):
     """restore indexes from backup file"""
     # delete
     index_check(force_restore=True)
     # recreate
-    backup_handler = ElasticBackup(INDEX_CONFIG)
-    zip_content = backup_handler.unpack_zip_backup()
+    backup_handler = ElasticBackup(INDEX_CONFIG, reason=False)
+    zip_content = backup_handler.unpack_zip_backup(filename)
     backup_handler.restore_json_files(zip_content)
 
 
@@ -522,7 +666,7 @@ def index_check(force_restore=False):
             # make backup before rebuild
             if not backed_up:
                 print("running backup first")
-                backup_all_indexes()
+                backup_all_indexes(reason="update")
                 backed_up = True
 
             print(f"applying new mappings to index ta_{index_name}...")

tubearchivist/home/src/reindex.py

@@ -23,7 +23,12 @@ from home.src.helper import (
     get_total_hits,
     ignore_filelist,
 )
-from home.src.index import YoutubeChannel, YoutubeVideo, index_new_video
+from home.src.index import (
+    YoutubeChannel,
+    YoutubePlaylist,
+    YoutubeVideo,
+    index_new_video,
+)
 from home.src.thumbnails import ThumbManager
 
 
@@ -35,35 +40,42 @@ class Reindex:
         config = AppConfig().config
         self.sleep_interval = config["downloads"]["sleep_interval"]
         self.es_url = config["application"]["es_url"]
-        self.refresh_interval = 90
+        self.es_auth = config["application"]["es_auth"]
+        self.refresh_interval = config["scheduler"]["check_reindex_days"]
+        self.integrate_ryd = config["downloads"]["integrate_ryd"]
         # scan
-        self.video_daily, self.channel_daily = self.get_daily()
         self.all_youtube_ids = False
         self.all_channel_ids = False
+        self.all_playlist_ids = False
 
     def get_daily(self):
         """get daily refresh values"""
-        total_videos = get_total_hits("ta_video", self.es_url, "active")
+        total_videos = get_total_hits(
+            "ta_video", self.es_url, self.es_auth, "active"
+        )
         video_daily = ceil(total_videos / self.refresh_interval * 1.2)
         total_channels = get_total_hits(
-            "ta_channel", self.es_url, "channel_active"
+            "ta_channel", self.es_url, self.es_auth, "channel_active"
         )
         channel_daily = ceil(total_channels / self.refresh_interval * 1.2)
-        return (video_daily, channel_daily)
+        total_playlists = get_total_hits(
+            "ta_playlist", self.es_url, self.es_auth, "playlist_active"
+        )
+        playlist_daily = ceil(total_playlists / self.refresh_interval * 1.2)
+        return (video_daily, channel_daily, playlist_daily)
 
-    def get_outdated_vids(self):
+    def get_outdated_vids(self, size):
         """get daily videos to refresh"""
         headers = {"Content-type": "application/json"}
         now = int(datetime.now().strftime("%s"))
-        now_3m = now - 3 * 30 * 24 * 60 * 60
-        size = self.video_daily
+        now_lte = now - self.refresh_interval * 24 * 60 * 60
         data = {
             "size": size,
             "query": {
                 "bool": {
                     "must": [
                         {"match": {"active": True}},
-                        {"range": {"vid_last_refresh": {"lte": now_3m}}},
+                        {"range": {"vid_last_refresh": {"lte": now_lte}}},
                     ]
                 }
             },
@@ -72,26 +84,49 @@ class Reindex:
         }
         query_str = json.dumps(data)
         url = self.es_url + "/ta_video/_search"
-        response = requests.get(url, data=query_str, headers=headers)
+        response = requests.get(
+            url, data=query_str, headers=headers, auth=self.es_auth
+        )
         if not response.ok:
             print(response.text)
         response_dict = json.loads(response.text)
         all_youtube_ids = [i["_id"] for i in response_dict["hits"]["hits"]]
         return all_youtube_ids
 
-    def get_outdated_channels(self):
+    def get_unrated_vids(self):
+        """get all videos without rating if ryd integration is enabled"""
+        headers = {"Content-type": "application/json"}
+        data = {
+            "size": 200,
+            "query": {
+                "bool": {
+                    "must_not": [{"exists": {"field": "stats.average_rating"}}]
+                }
+            },
+        }
+        query_str = json.dumps(data)
+        url = self.es_url + "/ta_video/_search"
+        response = requests.get(
+            url, data=query_str, headers=headers, auth=self.es_auth
+        )
+        if not response.ok:
+            print(response.text)
+        response_dict = json.loads(response.text)
+        missing_rating = [i["_id"] for i in response_dict["hits"]["hits"]]
+        self.all_youtube_ids = self.all_youtube_ids + missing_rating
+
+    def get_outdated_channels(self, size):
         """get daily channels to refresh"""
         headers = {"Content-type": "application/json"}
         now = int(datetime.now().strftime("%s"))
-        now_3m = now - 3 * 30 * 24 * 60 * 60
-        size = self.channel_daily
+        now_lte = now - self.refresh_interval * 24 * 60 * 60
         data = {
             "size": size,
             "query": {
                 "bool": {
                     "must": [
                         {"match": {"channel_active": True}},
-                        {"range": {"channel_last_refresh": {"lte": now_3m}}},
+                        {"range": {"channel_last_refresh": {"lte": now_lte}}},
                     ]
                 }
             },
@@ -100,17 +135,52 @@ class Reindex:
         }
         query_str = json.dumps(data)
         url = self.es_url + "/ta_channel/_search"
-        response = requests.get(url, data=query_str, headers=headers)
+        response = requests.get(
+            url, data=query_str, headers=headers, auth=self.es_auth
+        )
         if not response.ok:
             print(response.text)
         response_dict = json.loads(response.text)
         all_channel_ids = [i["_id"] for i in response_dict["hits"]["hits"]]
         return all_channel_ids
 
+    def get_outdated_playlists(self, size):
+        """get daily outdated playlists to refresh"""
+        headers = {"Content-type": "application/json"}
+        now = int(datetime.now().strftime("%s"))
+        now_lte = now - self.refresh_interval * 24 * 60 * 60
+        data = {
+            "size": size,
+            "query": {
+                "bool": {
+                    "must": [
+                        {"match": {"playlist_active": True}},
+                        {"range": {"playlist_last_refresh": {"lte": now_lte}}},
+                    ]
+                }
+            },
+            "sort": [{"playlist_last_refresh": {"order": "asc"}}],
+            "_source": False,
+        }
+        query_str = json.dumps(data)
+        url = self.es_url + "/ta_playlist/_search"
+        response = requests.get(
+            url, data=query_str, headers=headers, auth=self.es_auth
+        )
+        if not response.ok:
+            print(response.text)
+        response_dict = json.loads(response.text)
+        all_playlist_ids = [i["_id"] for i in response_dict["hits"]["hits"]]
+        return all_playlist_ids
+
     def check_outdated(self):
         """add missing vids and channels"""
-        self.all_youtube_ids = self.get_outdated_vids()
-        self.all_channel_ids = self.get_outdated_channels()
+        video_daily, channel_daily, playlist_daily = self.get_daily()
+        self.all_youtube_ids = self.get_outdated_vids(video_daily)
+        self.all_channel_ids = self.get_outdated_channels(channel_daily)
+        self.all_playlist_ids = self.get_outdated_playlists(playlist_daily)
+        if self.integrate_ryd:
+            self.get_unrated_vids()
 
     def rescrape_all_channels(self):
         """sync new data from channel to all matching videos"""
@@ -121,14 +191,6 @@ class Reindex:
 
         counter = 1
         for channel_id in all_channel_ids:
-            message = f"Progress: {counter}/{len(all_channels)}"
-            mess_dict = {
-                "status": "scraping",
-                "level": "info",
-                "title": "Scraping all youtube channels",
-                "message": message,
-            }
-            RedisArchivist().set_message("progress:download", mess_dict)
             channel_index = YoutubeChannel(channel_id)
             subscribed = channel_index.channel_dict["channel_subscribed"]
             channel_index.channel_dict = channel_index.build_channel_dict(
@@ -156,11 +218,18 @@ class Reindex:
         date_downloaded = es_vid_dict["_source"]["date_downloaded"]
         channel_dict = es_vid_dict["_source"]["channel"]
         channel_name = channel_dict["channel_name"]
+        try:
+            playlist = es_vid_dict["_source"]["playlist"]
+        except KeyError:
+            playlist = False
+
         vid_handler.build_file_path(channel_name)
         # add to vid_dict
         vid_handler.vid_dict["player"] = player
         vid_handler.vid_dict["date_downloaded"] = date_downloaded
         vid_handler.vid_dict["channel"] = channel_dict
+        if playlist:
+            vid_handler.vid_dict["playlist"] = playlist
         # update
         vid_handler.upload_to_es()
         thumb_handler = ThumbManager()
@@ -187,6 +256,22 @@ class Reindex:
         to_download = (channel_id, channel_thumb, channel_banner)
         thumb_handler.download_chan([to_download])
 
+    @staticmethod
+    def reindex_single_playlist(playlist_id, all_indexed_ids):
+        """refresh playlist data"""
+        playlist_handler = YoutubePlaylist(
+            playlist_id, all_youtube_ids=all_indexed_ids
+        )
+        playlist = playlist_handler.update_playlist()
+        if not playlist:
+            playlist_handler.deactivate()
+            return
+
+        playlist_thumbnail = (playlist_id, playlist["playlist_thumbnail"])
+        thumb_handler = ThumbManager()
+        thumb_handler.download_playlist([playlist_thumbnail])
+        return
+
     def reindex(self):
         """reindex what's needed"""
         # videos
@@ -201,6 +286,15 @@ class Reindex:
             self.reindex_single_channel(channel_id)
             if self.sleep_interval:
                 sleep(self.sleep_interval)
+        # playlist
+        print(f"reindexing {len(self.all_playlist_ids)} playlists")
+        if self.all_playlist_ids:
+            all_indexed = PendingList().get_all_indexed()
+            all_indexed_ids = [i["youtube_id"] for i in all_indexed]
+            for playlist_id in self.all_playlist_ids:
+                self.reindex_single_playlist(playlist_id, all_indexed_ids)
+                if self.sleep_interval:
+                    sleep(self.sleep_interval)
 
 
 class FilesystemScanner:
@@ -208,6 +302,7 @@ class FilesystemScanner:
 
     CONFIG = AppConfig().config
     ES_URL = CONFIG["application"]["es_url"]
+    ES_AUTH = CONFIG["application"]["es_auth"]
     VIDEOS = CONFIG["application"]["videos"]
 
     def __init__(self):
@@ -241,10 +336,10 @@ class FilesystemScanner:
         all_indexed_raw = index_handler.get_all_indexed()
         all_indexed = []
         for video in all_indexed_raw:
-            youtube_id = video["_id"]
-            media_url = video["_source"]["media_url"]
-            published = video["_source"]["published"]
-            title = video["_source"]["title"]
+            youtube_id = video["youtube_id"]
+            media_url = video["media_url"]
+            published = video["published"]
+            title = video["title"]
             all_indexed.append((youtube_id, media_url, published, title))
         return all_indexed
 
@@ -329,7 +424,9 @@ class FilesystemScanner:
         # make the call
         headers = {"Content-type": "application/x-ndjson"}
         url = self.ES_URL + "/_bulk"
-        request = requests.post(url, data=query_str, headers=headers)
+        request = requests.post(
+            url, data=query_str, headers=headers, auth=self.ES_AUTH
+        )
         if not request.ok:
             print(request.text)
 
@@ -339,7 +436,7 @@ class FilesystemScanner:
             youtube_id = indexed[0]
             print(f"deleting {youtube_id} from index")
             url = self.ES_URL + "/ta_video/_doc/" + youtube_id
-            request = requests.delete(url)
+            request = requests.delete(url, auth=self.ES_AUTH)
             if not request.ok:
                 print(request.text)
 
@@ -494,14 +591,10 @@ def scan_filesystem():
 
 def reindex_old_documents():
     """daily refresh of old documents"""
-    # check needed last run
-    now = int(datetime.now().strftime("%s"))
-    last_reindex = RedisArchivist().get_message("last_reindex")
-    if isinstance(last_reindex, int) and now - last_reindex < 60 * 60 * 24:
-        return
     # continue if needed
     reindex_handler = Reindex()
     reindex_handler.check_outdated()
     reindex_handler.reindex()
     # set timestamp
+    now = int(datetime.now().strftime("%s"))
     RedisArchivist().set_message("last_reindex", now, expire=False)

tubearchivist/home/src/searching.py

@@ -12,6 +12,7 @@ from datetime import datetime
 
 import requests
 from home.src.config import AppConfig
+from home.src.helper import RedisArchivist
 from home.src.thumbnails import ThumbManager
 
 
@@ -20,6 +21,7 @@ class SearchHandler:
 
     CONFIG = AppConfig().config
     CACHE_DIR = CONFIG["application"]["cache_dir"]
+    ES_AUTH = CONFIG["application"]["es_auth"]
 
     def __init__(self, url, data):
         self.max_hits = None
@@ -29,9 +31,11 @@ class SearchHandler:
     def get_data(self):
         """get the data"""
         if self.data:
-            response = requests.get(self.url, json=self.data).json()
+            response = requests.get(
+                self.url, json=self.data, auth=self.ES_AUTH
+            ).json()
         else:
-            response = requests.get(self.url).json()
+            response = requests.get(self.url, auth=self.ES_AUTH).json()
 
         if "hits" in response.keys():
             self.max_hits = response["hits"]["total"]["value"]
@@ -117,6 +121,12 @@ class SearchHandler:
             date_str = datetime.strftime(date_refresh, "%d %b, %Y")
             hit["source"]["vid_last_refresh"] = date_str
 
+        if "playlist_last_refresh" in hit_keys:
+            playlist_last_refresh = hit["source"]["playlist_last_refresh"]
+            date_refresh = datetime.fromtimestamp(playlist_last_refresh)
+            date_str = datetime.strftime(date_refresh, "%d %b, %Y")
+            hit["source"]["playlist_last_refresh"] = date_str
+
         if "vid_thumb_url" in hit_keys:
             youtube_id = hit["source"]["youtube_id"]
             thumb_path = ThumbManager().vid_thumb_path(youtube_id)
@@ -145,32 +155,63 @@ class SearchForm:
     CONFIG = AppConfig().config
     ES_URL = CONFIG["application"]["es_url"]
 
-    def search_channels(self, search_query):
-        """fancy searching channels as you type"""
-        url = self.ES_URL + "/ta_channel/_search"
+    def multi_search(self, search_query):
+        """searching through index"""
+        url = self.ES_URL + "/ta_video,ta_channel,ta_playlist/_search"
         data = {
-            "size": 10,
+            "size": 30,
             "query": {
                 "multi_match": {
                     "query": search_query,
                     "type": "bool_prefix",
+                    "operator": "and",
+                    "fuzziness": "auto",
                     "fields": [
-                        "channel_name.search_as_you_type",
+                        "category",
+                        "channel_description",
                         "channel_name._2gram",
                         "channel_name._3gram",
+                        "channel_name.search_as_you_type",
+                        "playlist_description",
+                        "playlist_name._2gram",
+                        "playlist_name._3gram",
+                        "playlist_name.search_as_you_type",
+                        "tags",
+                        "title._2gram",
+                        "title._3gram",
+                        "title.search_as_you_type",
                     ],
                 }
             },
         }
         look_up = SearchHandler(url, data)
         search_results = look_up.get_data()
-        return {"results": search_results}
+        all_results = self.build_results(search_results)
+
+        return {"results": all_results}
 
     @staticmethod
-    def search_videos():
-        """searching for videos"""
-        # TBD palceholder for now
-        return False
+    def build_results(search_results):
+        """build the all_results dict"""
+        video_results = []
+        channel_results = []
+        playlist_results = []
+        if search_results:
+            for result in search_results:
+                if result["_index"] == "ta_video":
+                    video_results.append(result)
+                elif result["_index"] == "ta_channel":
+                    channel_results.append(result)
+                elif result["_index"] == "ta_playlist":
+                    playlist_results.append(result)
+
+        all_results = {
+            "video_results": video_results,
+            "channel_results": channel_results,
+            "playlist_results": playlist_results,
+        }
+
+        return all_results
 
 
 class Pagination:
@@ -178,13 +219,23 @@ class Pagination:
     figure out the pagination based on page size and total_hits
     """
 
-    def __init__(self, page_get, search_get=False):
-        config = AppConfig().config
-        self.page_size = config["archive"]["page_size"]
+    def __init__(self, page_get, user_id, search_get=False):
+        self.user_id = user_id
+        self.page_size = self.get_page_size()
         self.page_get = page_get
         self.search_get = search_get
         self.pagination = self.first_guess()
 
+    def get_page_size(self):
+        """get default or user modified page_size"""
+        key = f"{self.user_id}:page_size"
+        page_size = RedisArchivist().get_message(key)["status"]
+        if not page_size:
+            config = AppConfig().config
+            page_size = config["archive"]["page_size"]
+
+        return page_size
+
     def first_guess(self):
         """build first guess before api call"""
         page_get = self.page_get

tubearchivist/home/src/thumbnails.py

@@ -11,6 +11,7 @@ import home.src.download as download
 import requests
 from home.src.config import AppConfig
 from home.src.helper import RedisArchivist, ignore_filelist
+from mutagen.mp4 import MP4, MP4Cover
 from PIL import Image
 
 
@@ -18,9 +19,11 @@ class ThumbManager:
     """handle thumbnails related functions"""
 
     CONFIG = AppConfig().config
+    MEDIA_DIR = CONFIG["application"]["videos"]
     CACHE_DIR = CONFIG["application"]["cache_dir"]
     VIDEO_DIR = os.path.join(CACHE_DIR, "videos")
     CHANNEL_DIR = os.path.join(CACHE_DIR, "channels")
+    PLAYLIST_DIR = os.path.join(CACHE_DIR, "playlists")
 
     def get_all_thumbs(self):
         """get all video artwork already downloaded"""
@@ -57,8 +60,8 @@ class ThumbManager:
 
         needed_thumbs = []
         for video in all_indexed:
-            youtube_id = video["_source"]["youtube_id"]
-            thumb_url = video["_source"]["vid_thumb_url"]
+            youtube_id = video["youtube_id"]
+            thumb_url = video["vid_thumb_url"]
             if missing_only:
                 if youtube_id + ".jpg" not in all_thumbs:
                     needed_thumbs.append((youtube_id, thumb_url))
@@ -97,6 +100,24 @@ class ThumbManager:
 
         return missing_channels
 
+    def get_missing_playlists(self):
+        """get all missing playlist artwork"""
+        all_downloaded = ignore_filelist(os.listdir(self.PLAYLIST_DIR))
+        all_ids_downloaded = [i.replace(".jpg", "") for i in all_downloaded]
+
+        playlists = download.PlaylistSubscription().get_playlists(
+            subscribed_only=False
+        )
+
+        missing_playlists = []
+        for playlist in playlists:
+            playlist_id = playlist["playlist_id"]
+            if playlist_id not in all_ids_downloaded:
+                playlist_thumb = playlist["playlist_thumbnail"]
+                missing_playlists.append((playlist_id, playlist_thumb))
+
+        return missing_playlists
+
     def get_raw_img(self, img_url, thumb_type):
         """get raw image from youtube and handle 404"""
         try:
@@ -117,7 +138,11 @@ class ThumbManager:
             ),
         }
         if img_url:
-            response = requests.get(img_url, stream=True)
+            try:
+                response = requests.get(img_url, stream=True)
+            except ConnectionError:
+                sleep(5)
+                response = requests.get(img_url, stream=True)
             if not response.ok and not response.status_code == 404:
                 print("retry thumbnail download for " + img_url)
                 sleep(5)
@@ -136,16 +161,12 @@ class ThumbManager:
 
     def download_vid(self, missing_thumbs, notify=True):
         """download all missing thumbnails from list"""
-        total = len(missing_thumbs)
-        print(f"downloading {total} thumbnails")
-        # videos
-        # counter as update path from v0.0.5, remove in future version
-        counter = 0
-        for youtube_id, thumb_url in missing_thumbs:
-            folder_name = youtube_id[0].lower()
-            folder_path = os.path.join(self.VIDEO_DIR, folder_name)
-            thumb_path_part = self.vid_thumb_path(youtube_id)
-            thumb_path = os.path.join(self.CACHE_DIR, thumb_path_part)
+        print(f"downloading {len(missing_thumbs)} thumbnails")
+        for idx, (youtube_id, thumb_url) in enumerate(missing_thumbs):
+            folder_path = os.path.join(self.VIDEO_DIR, youtube_id[0].lower())
+            thumb_path = os.path.join(
+                self.CACHE_DIR, self.vid_thumb_path(youtube_id)
+            )
 
             os.makedirs(folder_path, exist_ok=True)
             img_raw = self.get_raw_img(thumb_url, "video")
@@ -155,21 +176,25 @@ class ThumbManager:
                 new_height = width / 16 * 9
                 offset = (height - new_height) / 2
                 img_raw = img_raw.crop((0, offset, width, height - offset))
-
             img_raw.convert("RGB").save(thumb_path)
 
+            progress = f"{idx + 1}/{len(missing_thumbs)}"
             if notify:
                 mess_dict = {
-                    "status": "pending",
+                    "status": "message:add",
                     "level": "info",
-                    "title": "Adding to download queue.",
-                    "message": "Downloading Thumbnails...",
+                    "title": "Processing Videos",
+                    "message": "Downloading Thumbnails, Progress: " + progress,
                 }
-                RedisArchivist().set_message("progress:download", mess_dict)
+                if idx + 1 == len(missing_thumbs):
+                    RedisArchivist().set_message(
+                        "message:add", mess_dict, expire=4
+                    )
+                else:
+                    RedisArchivist().set_message("message:add", mess_dict)
 
-            if counter % 50 == 0 and counter != 0:
-                print(f"thumbnail progress: {counter}/{total}")
-            counter = counter + 1
+            if idx + 1 % 25 == 0:
+                print("thumbnail progress: " + progress)
 
     def download_chan(self, missing_channels):
         """download needed artwork for channels"""
@@ -190,12 +215,29 @@ class ThumbManager:
             img_raw.convert("RGB").save(banner_path)
 
             mess_dict = {
-                "status": "pending",
+                "status": "message:download",
                 "level": "info",
-                "title": "Adding to download queue.",
-                "message": "Downloading Channel Art...",
+                "title": "Processing Channels",
+                "message": "Downloading Channel Art.",
             }
-            RedisArchivist().set_message("progress:download", mess_dict)
+            RedisArchivist().set_message("message:download", mess_dict)
+
+    def download_playlist(self, missing_playlists):
+        """download needed artwork for playlists"""
+        print(f"downloading {len(missing_playlists)} playlist artwork")
+        for playlist in missing_playlists:
+            playlist_id, playlist_thumb_url = playlist
+            thumb_path = os.path.join(self.PLAYLIST_DIR, playlist_id + ".jpg")
+            img_raw = self.get_raw_img(playlist_thumb_url, "video")
+            img_raw.convert("RGB").save(thumb_path)
+
+            mess_dict = {
+                "status": "message:download",
+                "level": "info",
+                "title": "Processing Playlists",
+                "message": "Downloading Playlist Art.",
+            }
+            RedisArchivist().set_message("message:download", mess_dict)
 
     @staticmethod
     def vid_thumb_path(youtube_id):
@@ -232,6 +274,45 @@ class ThumbManager:
                 youtube_id = thumb.rstrip(".jpg")
                 self.delete_vid_thumb(youtube_id)
 
+    def get_thumb_list(self):
+        """get list of mediafiles and matching thumbnails"""
+        all_indexed = download.PendingList().get_all_indexed()
+        video_list = []
+        for video in all_indexed:
+            youtube_id = video["youtube_id"]
+            media_url = os.path.join(self.MEDIA_DIR, video["media_url"])
+            thumb_path = os.path.join(
+                self.CACHE_DIR, self.vid_thumb_path(youtube_id)
+            )
+            video_list.append(
+                {
+                    "media_url": media_url,
+                    "thumb_path": thumb_path,
+                }
+            )
+
+        return video_list
+
+    @staticmethod
+    def write_all_thumbs(video_list):
+        """rewrite the thumbnail into media file"""
+
+        counter = 1
+        for video in video_list:
+            # loop through all videos
+            media_url = video["media_url"]
+            thumb_path = video["thumb_path"]
+
+            mutagen_vid = MP4(media_url)
+            with open(thumb_path, "rb") as f:
+                mutagen_vid["covr"] = [
+                    MP4Cover(f.read(), imageformat=MP4Cover.FORMAT_JPEG)
+                ]
+            mutagen_vid.save()
+            if counter % 50 == 0:
+                print(f"thumbnail write progress {counter}/{len(video_list)}")
+            counter = counter + 1
+
 
 def validate_thumbnails():
     """check if all thumbnails are there and organized correctly"""
@@ -240,4 +321,6 @@ def validate_thumbnails():
     handler.download_vid(thumbs_to_download)
     missing_channels = handler.get_missing_channels()
     handler.download_chan(missing_channels)
+    missing_playlists = handler.get_missing_playlists()
+    handler.download_playlist(missing_playlists)
     handler.cleanup_downloaded()

tubearchivist/home/tasks.py

@@ -2,49 +2,77 @@
 Functionality:
 - initiate celery app
 - collect tasks
+- user config changes won't get applied here
+  because tasks are initiated at application start
 """
 
 import os
 
+import home.apps as startup_apps
 from celery import Celery, shared_task
-from home.src.config import AppConfig
-from home.src.download import ChannelSubscription, PendingList, VideoDownloader
-from home.src.helper import RedisArchivist, RedisQueue
+from home.src.config import AppConfig, ScheduleBuilder
+from home.src.download import (
+    ChannelSubscription,
+    PendingList,
+    PlaylistSubscription,
+    VideoDownloader,
+)
+from home.src.helper import RedisArchivist, RedisQueue, UrlListParser
+from home.src.index import YoutubeChannel, YoutubePlaylist
 from home.src.index_management import backup_all_indexes, restore_from_backup
 from home.src.reindex import (
     ManualImport,
     reindex_old_documents,
     scan_filesystem,
 )
-from home.src.thumbnails import ThumbManager
+from home.src.thumbnails import ThumbManager, validate_thumbnails
 
 CONFIG = AppConfig().config
 REDIS_HOST = os.environ.get("REDIS_HOST")
-REDIS_PORT = os.environ.get("REDIS_PORT")
+REDIS_PORT = os.environ.get("REDIS_PORT") or 6379
 
-if not REDIS_PORT:
-    REDIS_PORT = 6379
-
-os.environ.setdefault("DJANGO_SETTINGS_MODULE", "home.settings")
+os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings")
 app = Celery("tasks", broker=f"redis://{REDIS_HOST}:{REDIS_PORT}")
-app.config_from_object("django.conf:settings", namespace="CELERY")
+app.config_from_object("django.conf:settings", namespace="ta:")
 app.autodiscover_tasks()
+app.conf.timezone = os.environ.get("TZ") or "UTC"
 
 
-@shared_task
+@shared_task(name="update_subscribed")
 def update_subscribed():
     """look for missing videos and add to pending"""
-    channel_handler = ChannelSubscription()
-    missing_videos = channel_handler.find_missing()
-    if missing_videos:
-        pending_handler = PendingList()
-        all_videos_added = pending_handler.add_to_pending(missing_videos)
-        ThumbManager().download_vid(all_videos_added)
-    # check if reindex is needed
-    check_reindex.delay()
+    message = {
+        "status": "message:rescan",
+        "level": "info",
+        "title": "Rescanning channels and playlists.",
+        "message": "Looking for new videos.",
+    }
+    RedisArchivist().set_message("message:rescan", message)
+
+    have_lock = False
+    my_lock = RedisArchivist().get_lock("rescan")
+
+    try:
+        have_lock = my_lock.acquire(blocking=False)
+        if have_lock:
+            channel_handler = ChannelSubscription()
+            missing_from_channels = channel_handler.find_missing()
+            playlist_handler = PlaylistSubscription()
+            missing_from_playlists = playlist_handler.find_missing()
+            missing = missing_from_channels + missing_from_playlists
+            if missing:
+                pending_handler = PendingList()
+                all_videos_added = pending_handler.add_to_pending(missing)
+                ThumbManager().download_vid(all_videos_added)
+        else:
+            print("Did not acquire rescan lock.")
+
+    finally:
+        if have_lock:
+            my_lock.release()
 
 
-@shared_task
+@shared_task(name="download_pending")
 def download_pending():
     """download latest pending videos"""
     have_lock = False
@@ -56,6 +84,7 @@ def download_pending():
             downloader = VideoDownloader()
             downloader.add_pending()
             downloader.run_queue()
+            downloader.validate_playlists()
         else:
             print("Did not acquire download lock.")
 
@@ -77,6 +106,13 @@ def download_single(youtube_id):
     try:
         have_lock = my_lock.acquire(blocking=False)
         if have_lock:
+            mess_dict = {
+                "status": "message:download",
+                "level": "info",
+                "title": "Download single video",
+                "message": "processing",
+            }
+            RedisArchivist().set_message("message:download", mess_dict)
             VideoDownloader().run_queue()
         else:
             print("Download queue already running.")
@@ -93,10 +129,19 @@ def extrac_dl(youtube_ids):
     pending_handler = PendingList()
     missing_videos = pending_handler.parse_url_list(youtube_ids)
     all_videos_added = pending_handler.add_to_pending(missing_videos)
-    ThumbManager().download_vid(all_videos_added)
+    missing_playlists = pending_handler.missing_from_playlists
+
+    thumb_handler = ThumbManager()
+    if missing_playlists:
+        new_thumbs = PlaylistSubscription().process_url_str(
+            missing_playlists, subscribed=False
+        )
+        thumb_handler.download_playlist(new_thumbs)
+
+    thumb_handler.download_vid(all_videos_added)
 
 
-@shared_task
+@shared_task(name="check_reindex")
 def check_reindex():
     """run the reindex main command"""
     reindex_old_documents()
@@ -124,17 +169,17 @@ def run_manual_import():
             my_lock.release()
 
 
-@shared_task
-def run_backup():
+@shared_task(name="run_backup")
+def run_backup(reason="auto"):
     """called from settings page, dump backup to zip file"""
-    backup_all_indexes()
+    backup_all_indexes(reason)
     print("backup finished")
 
 
 @shared_task
-def run_restore_backup():
+def run_restore_backup(filename):
     """called from settings page, dump backup to zip file"""
-    restore_from_backup()
+    restore_from_backup(filename)
     print("index restore finished")
 
 
@@ -154,15 +199,131 @@ def kill_dl(task_id):
 
     # notify
     mess_dict = {
-        "status": "downloading",
+        "status": "message:download",
         "level": "error",
-        "title": "Brutally killing download queue",
-        "message": "",
+        "title": "Canceling download process",
+        "message": "Canceling download queue now.",
     }
-    RedisArchivist().set_message("progress:download", mess_dict)
+    RedisArchivist().set_message("message:download", mess_dict)
 
 
 @shared_task
 def rescan_filesystem():
     """check the media folder for mismatches"""
     scan_filesystem()
+    validate_thumbnails()
+
+
+@shared_task(name="thumbnail_check")
+def thumbnail_check():
+    """validate thumbnails"""
+    validate_thumbnails()
+
+
+@shared_task
+def re_sync_thumbs():
+    """sync thumbnails to mediafiles"""
+    handler = ThumbManager()
+    video_list = handler.get_thumb_list()
+    handler.write_all_thumbs(video_list)
+
+
+@shared_task
+def subscribe_to(url_str):
+    """take a list of urls to subscribe to"""
+    to_subscribe_list = UrlListParser(url_str).process_list()
+    counter = 1
+    for item in to_subscribe_list:
+        to_sub_id = item["url"]
+        if item["type"] == "playlist":
+            new_thumbs = PlaylistSubscription().process_url_str([item])
+            if new_thumbs:
+                ThumbManager().download_playlist(new_thumbs)
+            continue
+
+        if item["type"] == "video":
+            vid_details = PendingList().get_youtube_details(to_sub_id)
+            channel_id_sub = vid_details["channel_id"]
+        elif item["type"] == "channel":
+            channel_id_sub = to_sub_id
+        else:
+            raise ValueError("failed to subscribe to: " + to_sub_id)
+
+        ChannelSubscription().change_subscribe(
+            channel_id_sub, channel_subscribed=True
+        )
+        # notify for channels
+        message = {
+            "status": "message:subchannel",
+            "level": "info",
+            "title": "Subscribing to Channels",
+            "message": f"Processing {counter} of {len(to_subscribe_list)}",
+        }
+        RedisArchivist().set_message("message:subchannel", message=message)
+        counter = counter + 1
+
+
+@shared_task
+def index_channel_playlists(channel_id):
+    """add all playlists of channel to index"""
+    channel_handler = YoutubeChannel(channel_id)
+    channel_name = channel_handler.channel_dict["channel_name"]
+    # notify
+    mess_dict = {
+        "status": "message:playlistscan",
+        "level": "info",
+        "title": "Looking for playlists",
+        "message": f'Scanning channel "{channel_name}" in progress',
+    }
+    RedisArchivist().set_message("message:playlistscan", mess_dict)
+    all_playlists = channel_handler.get_all_playlists()
+
+    if not all_playlists:
+        print(f"no playlists found for channel {channel_id}")
+        return
+
+    all_indexed = PendingList().get_all_indexed()
+    all_youtube_ids = [i["youtube_id"] for i in all_indexed]
+
+    for idx, (playlist_id, playlist_title) in enumerate(all_playlists):
+        # notify
+        mess_dict = {
+            "status": "message:playlistscan",
+            "level": "info",
+            "title": "Scanning channel for playlists",
+            "message": f"Progress: {idx + 1}/{len(all_playlists)}",
+        }
+        RedisArchivist().set_message("message:playlistscan", mess_dict)
+        print("add playlist: " + playlist_title)
+        playlist_handler = YoutubePlaylist(
+            playlist_id, all_youtube_ids=all_youtube_ids
+        )
+        playlist_handler.get_playlist_dict()
+        if not playlist_handler.playlist_dict:
+            # skip if not available
+            continue
+        # don't add if no videos downloaded
+        downloaded = [
+            i
+            for i in playlist_handler.playlist_dict["playlist_entries"]
+            if i["downloaded"]
+        ]
+        if not downloaded:
+            continue
+        playlist_handler.upload_to_es()
+        playlist_handler.add_vids_to_playlist()
+
+    if all_playlists:
+        handler = ThumbManager()
+        missing_playlists = handler.get_missing_playlists()
+        handler.download_playlist(missing_playlists)
+
+    return
+
+
+try:
+    app.conf.beat_schedule = ScheduleBuilder().build_schedule()
+except KeyError:
+    # update path from v0.0.8 to v0.0.9 to load new defaults
+    startup_apps.sync_redis_state()
+    app.conf.beat_schedule = ScheduleBuilder().build_schedule()

tubearchivist/home/templates/home/about.html

@@ -1,17 +1,19 @@
 {% extends "home/base.html" %}
 {% load static %}
 {% block content %}
-<div class="title-bar">
-    <h1>About The Tube Archivist</h1>
-</div>
-<div class="about-section">
-    <h2>Useful Links</h2>
-    <p>This project is in active and constant development, take a look at the <a href="https://github.com/bbilly1/tubearchivist#roadmap" target="_blank">roadmap</a> for a overview.</p>
-    <p>For any questions on what a button or a function does, You can find the up-to-date user documentation on <a href="https://github.com/bbilly1/tubearchivist/wiki" target="_blank">Github</a>.</p>
-    <p>All contributions are welcome: Open an <a href="https://github.com/bbilly1/tubearchivist/issues" target="_blank">issue</a> for any bugs and errors, start a <a href="https://github.com/bbilly1/tubearchivist/discussions" target="_blank">discussion</a> for anything that will require a more indepth look. The <a href="https://github.com/bbilly1/tubearchivist/blob/master/CONTRIBUTING.md" target="_blank">contributing</a> page is a good place to get started.</p>
-</div>
-<div class="about-section">
-    <h2>Donate</h2>
-    <p>Here are <a href="https://github.com/bbilly1/tubearchivist#donate" target="_blank">some links</a>, if you want to buy the developer a coffee. Thank you for your support!</p>
+<div class="boxed-content">
+    <div class="title-bar">
+        <h1>About The Tube Archivist</h1>
+    </div>
+    <div class="about-section">
+        <h2>Useful Links</h2>
+        <p>This project is in active and constant development, take a look at the <a href="https://github.com/bbilly1/tubearchivist#roadmap" target="_blank">roadmap</a> for a overview.</p>
+        <p>For any questions on what a button or a function does, You can find the up-to-date user documentation on <a href="https://github.com/bbilly1/tubearchivist/wiki" target="_blank">Github</a>.</p>
+        <p>All contributions are welcome: Open an <a href="https://github.com/bbilly1/tubearchivist/issues" target="_blank">issue</a> for any bugs and errors, start a <a href="https://github.com/bbilly1/tubearchivist/discussions" target="_blank">discussion</a> for anything that will require a more indepth look. The <a href="https://github.com/bbilly1/tubearchivist/blob/master/CONTRIBUTING.md" target="_blank">contributing</a> page is a good place to get started.</p>
+    </div>
+    <div class="about-section">
+        <h2>Donate</h2>
+        <p>Here are <a href="https://github.com/bbilly1/tubearchivist#donate" target="_blank">some links</a>, if you want to buy the developer a coffee. Thank you for your support!</p>
+    </div>
 </div>
 {% endblock content %}

tubearchivist/home/templates/home/base.html

@@ -6,7 +6,17 @@
     <meta http-equiv="X-UA-Compatible" content="IE=edge">
     <meta name="viewport" content="width=device-width, initial-scale=1.0">
     <link rel="stylesheet" type="text/css" href="{% static 'css/style.css' %}">
-    <link rel="shortcut icon" type="image/png" href="{% static 'favicon.ico' %}"/>
+    <link rel="apple-touch-icon" sizes="180x180" href="{% static 'favicon/apple-touch-icon.png' %}">
+    <link rel="icon" type="image/png" sizes="32x32" href="{% static 'favicon/favicon-32x32.png' %}">
+    <link rel="icon" type="image/png" sizes="16x16" href="{% static 'favicon/favicon-16x16.png' %}">
+    <link rel="manifest" href="{% static 'favicon/site.webmanifest' %}">
+    <link rel="mask-icon" href="{% static 'favicon/safari-pinned-tab.svg' %}" color="#01202e">
+    <link rel="shortcut icon" href="{% static 'favicon/favicon.ico' %}">
+    <meta name="apple-mobile-web-app-title" content="TubeArchivist">
+    <meta name="application-name" content="TubeArchivist">
+    <meta name="msapplication-TileColor" content="#01202e">
+    <meta name="msapplication-config" content="{% static 'favicon/browserconfig.xml' %}">
+    <meta name="theme-color" content="#01202e">
     {% if title %}
         <title>TA | {{ title }}</title>
     {% else %}
@@ -20,39 +30,48 @@
     <script type="text/javascript" src="{% static 'script.js' %}"></script>
 </head>
 <body>
-    <div class="boxed-content">
-        <div class="top-banner">
-            <a href="{% url 'home' %}">
-                {% if colors == 'dark' %}
-                    <img src="{% static 'img/banner-tube-archivist-dark.png' %}" alt="tube-archivist-banner">
-                {% endif %}
-                {% if colors == 'light' %}
-                    <img src="{% static 'img/banner-tube-archivist-light.png' %}" alt="tube-archivist-banner">
-                {% endif %}
-            </a>
-        </div>
-        <div class="top-nav">
-            <div class="nav-items">
+    <div class="main-content">
+        <div class="boxed-content">
+            <div class="top-banner">
                 <a href="{% url 'home' %}">
-                    <div class="nav-item">home</div>
-                </a>
-                <a href="{% url 'channel' %}">
-                    <div class="nav-item">channels</div>
-                </a>
-                <a href="{% url 'downloads' %}">
-                    <div class="nav-item">downloads</div>
+                    {% if colors == 'dark' %}
+                        <img src="{% static 'img/banner-tube-archivist-dark.png' %}" alt="tube-archivist-banner">
+                    {% endif %}
+                    {% if colors == 'light' %}
+                        <img src="{% static 'img/banner-tube-archivist-light.png' %}" alt="tube-archivist-banner">
+                    {% endif %}
                 </a>
             </div>
-            <div class="nav-icons">
-                <a href="{% url 'about' %}">
-                    <img src="{% static 'img/icon-help.svg' %}" alt="help-icon">
-                </a>
-                <a href="{% url 'settings' %}">
-                    <img src="{% static 'img/icon-gear.svg' %}" alt="gear-icon">
-                </a>
+            <div class="top-nav">
+                <div class="nav-items">
+                    <a href="{% url 'home' %}">
+                        <div class="nav-item">home</div>
+                    </a>
+                    <a href="{% url 'channel' %}">
+                        <div class="nav-item">channels</div>
+                    </a>
+                    <a href="{% url 'playlist' %}">
+                        <div class="nav-item">playlists</div>
+                    </a>
+                    <a href="{% url 'downloads' %}">
+                        <div class="nav-item">downloads</div>
+                    </a>
+                </div>
+                <div class="nav-icons">
+                    <a href="{% url 'search' %}">
+                        <img src="{% static 'img/icon-search.svg' %}" alt="search-icon" title="Search">
+                    </a>
+                    <a href="{% url 'settings' %}">
+                        <img src="{% static 'img/icon-gear.svg' %}" alt="gear-icon" title="Settings">
+                    </a>
+                    <a href="{% url 'logout' %}">
+                        <img src="{% static 'img/icon-exit.svg' %}" alt="exit-icon" title="Logout">
+                    </a>
+                </div>
             </div>
         </div>
         {% block content %}{% endblock %}
+        <div class="boxed-content">
             <div class="pagination">
                 {% if pagination %}
                     {% if pagination.current_page > 1 %}
@@ -93,11 +112,12 @@
                     {% endif %}
                 {% endif %}
             </div>
+        </div>
     </div>
     <div class="footer">
         <div class="boxed-content">
-            <span>© 2021 The Tube Archivist v0.0.6 | <a href="https://github.com/bbilly1/tubearchivist" target="_blank">Github</a> | <a href="https://hub.docker.com/r/bbilly1/tubearchivist" target="_blank">Docker Hub</a></span>
+            <span>© 2021 - <script type="text/javascript">document.write(new Date().getFullYear());</script> TubeArchivist v0.1.0 </span><span><a href="{% url 'about' %}">About</a> | <a href="https://github.com/bbilly1/tubearchivist" target="_blank">GitHub</a> | <a href="https://hub.docker.com/r/bbilly1/tubearchivist" target="_blank">Docker Hub</a> | <a href="https://discord.gg/AFwz8nE7BK" target="_blank">Discord</a> | <a href="https://www.reddit.com/r/TubeArchivist/">Reddit</a></span>
         </div>
     </div>
 </body>
-</html>
+</html>

tubearchivist/home/templates/home/channel.html

@@ -2,96 +2,83 @@
 {% load static %}
 {% load humanize %}
 {% block content %}
-<div class="title-bar">
-    <h1>Channels</h1>
-</div>
-<div class="info-box info-box-2">
-    <div class="icon-text">
-        <img id="add-icon" onclick="showForm()" src="{% static 'img/icon-add.svg' %}" alt="add-icon">
-        <p>Subscribe to Channels</p>
-        <div class="show-form">
-            <form id="hidden-form" action="/channel/" method="post">
-                {% csrf_token %}
-                <textarea rows="3" placeholder="Input channel ID, channel URL or Video of a channel" id="subscribe" name="subscribe"></textarea>
-                <button type="submit">Subscribe</button>
-            </form>
+<div class="boxed-content">
+    <div class="title-split">
+        <div class="title-bar">
+            <h1>Channels</h1>
+        </div>
+        <div class="title-split-form">
+            <img id="animate-icon" onclick="showForm()" src="{% static 'img/icon-add.svg' %}" alt="add-icon" title="Subscribe to Channels">
+            <div class="show-form">
+                <form id="hidden-form" action="/channel/" method="post">
+                    {% csrf_token %}
+                    {{ subscribe_form }}
+                    <button type="submit">Subscribe</button>
+                </form>
+            </div>
         </div>
     </div>
-    <div class="search-form icon-text">
-        <div class="search-icon">
-            <img src="{% static 'img/icon-search.svg' %}" alt="search-icon" onclick="showSearch()">
-            <p>Search Channels</p>
-        </div>
-        <form onSubmit="return channelRedirect();" id="search-box">
-            {% csrf_token %}
-            <input name="videoSearch" list="resultBox" type="text" id="searchInput" autocomplete="off" oninput="searchChannels(this.value)">
-            <datalist id="resultBox">
-            </datalist>
-        </form>
-    </div>
-</div>
-<div class="view-controls">
-    <div class="toggle">
-        <span>Show only subscribed channels:</span>
-        <div class="toggleBox">
-            <input 
-                id="show_subed_only" onclick="toggleCheckbox(this)" type="checkbox"
+    <div id="notifications" data="channel"></div>
+    <div class="view-controls">
+        <div class="toggle">
+            <span>Show subscribed only:</span>
+            <div class="toggleBox">
+                <input id="show_subed_only" onclick="toggleCheckbox(this)" type="checkbox" {% if show_subed_only %}checked{% endif %}>
                 {% if show_subed_only %}
-                    checked
+                    <label for="" class="ofbtn">Off</label>
+                {% else %}
+                    <label for="" class="onbtn">On</label>
                 {% endif %}
-                >
-            <label for="" class="onbtn">On</label>
-            <label for="" class="ofbtn">Off</label>
+            </div>
+        </div>
+        <div class="view-icons">
+            <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="channel" data-value="grid" alt="grid view">
+            <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="channel" data-value="list" alt="list view">
         </div>
     </div>
-    <div class="view-icons">
-        <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="channel" data-value="grid" alt="grid view">
-        <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="channel" data-value="list" alt="list view">
-    </div>
-</div>
-<h2>Total matching channels: {{ max_hits }}</h2>
-<div class="channel-list {{ view_style }}">
-    {% if channels %}
-        {% for channel in channels %}
-        <div class="channel-item {{ view_style }}">
-            <div class="channel-banner {{ view_style }}">
-                <a href="{% url 'channel_id' channel.source.channel_id %}">
-                    <img src="/cache/channels/{{ channel.source.channel_id }}_banner.jpg" alt="{{ channel.source.channel_id }}-banner">
-                </a>
-            </div>
-            <div class="info-box info-box-2 {{ view_style }}">
-                <div class="info-box-item">
-                    <div class="round-img">
-                        <a href="{% url 'channel_id' channel.source.channel_id %}">
-                            <img src="/cache/channels/{{ channel.source.channel_id }}_thumb.jpg" alt="channel-thumb">
-                        </a>
-                    </div>
-                    <div>
-                        <h3><a href="{% url 'channel_id' channel.source.channel_id %}">{{ channel.source.channel_name }}</a></h3>
-                        {% if channel.source.channel_subs >= 1000000 %}
-                            <p>Subscribers: {{ channel.source.channel_subs|intword }}</p>
-                        {% else %}
-                            <p>Subscribers: {{ channel.source.channel_subs|intcomma }}</p>
-                        {% endif %}
-                    </div>
+    <h2>Total matching channels: {{ max_hits }}</h2>
+    <div class="channel-list {{ view_style }}">
+        {% if results %}
+            {% for channel in results %}
+            <div class="channel-item {{ view_style }}">
+                <div class="channel-banner {{ view_style }}">
+                    <a href="{% url 'channel_id' channel.source.channel_id %}">
+                        <img src="/cache/channels/{{ channel.source.channel_id }}_banner.jpg" alt="{{ channel.source.channel_id }}-banner">
+                    </a>
                 </div>
-                <div class="info-box-item">
-                    <div>
-                        <p>Last refreshed: {{ channel.source.channel_last_refresh }}</p>
-                        <p>Subscribed: 
-                        {% if channel.source.channel_subscribed %}
-                            <button type="button" id="{{ channel.source.channel_id }}" onclick="unsubscribe(this.id)" title="Unsubscribe from {{ channel.source.channel_name }}">Unsubscribe</button>
-                        {% else %}
-                            false
-                        {% endif %}
-                        </p>
+                <div class="info-box info-box-2 {{ view_style }}">
+                    <div class="info-box-item">
+                        <div class="round-img">
+                            <a href="{% url 'channel_id' channel.source.channel_id %}">
+                                <img src="/cache/channels/{{ channel.source.channel_id }}_thumb.jpg" alt="channel-thumb">
+                            </a>
+                        </div>
+                        <div>
+                            <h3><a href="{% url 'channel_id' channel.source.channel_id %}">{{ channel.source.channel_name }}</a></h3>
+                            {% if channel.source.channel_subs >= 1000000 %}
+                                <p>Subscribers: {{ channel.source.channel_subs|intword }}</p>
+                            {% else %}
+                                <p>Subscribers: {{ channel.source.channel_subs|intcomma }}</p>
+                            {% endif %}
+                        </div>
+                    </div>
+                    <div class="info-box-item">
+                        <div>
+                            <p>Last refreshed: {{ channel.source.channel_last_refresh }}</p>
+                            {% if channel.source.channel_subscribed %}
+                                <button class="unsubscribe" type="button" id="{{ channel.source.channel_id }}" onclick="unsubscribe(this.id)" title="Unsubscribe from {{ channel.source.channel_name }}">Unsubscribe</button>
+                            {% else %}
+                                <button type="button" id="{{ channel.source.channel_id }}" onclick="subscribe(this.id)" title="Subscribe to {{ channel.source.channel_name }}">Subscribe</button>
+                            {% endif %}
+                        </div>
                     </div>
                 </div>
             </div>
-        </div>
-        {% endfor %}
-    {% else %}
-        <h2>No channels found...</h2>
-    {% endif %}
+            {% endfor %}
+        {% else %}
+            <h2>No channels found...</h2>
+        {% endif %}
+    </div>
 </div>
+<script type="text/javascript" src="{% static 'progress.js' %}"></script>
 {% endblock content %}

tubearchivist/home/templates/home/channel_id.html

@@ -2,104 +2,139 @@
 {% block content %}
 {% load static %}
 {% load humanize %}
-<div class="title-bar">
-    <h1>Channel: {{ channel_info.channel_name }}</h1>
-</div>
-<div class="channel-banner">
-    <a href="/channel/{{ channel_info.channel_id }}/"><img src="/cache/channels/{{ channel_info.channel_id }}_banner.jpg" alt="channel_banner"></a>
-</div>
-<div class="info-box info-box-3">
-    <div class="info-box-item">
-        <div class="round-img">
-            <a href="{% url 'channel_id' channel_info.channel_id %}">
-                <img src="/cache/channels/{{ channel_info.channel_id }}_thumb.jpg" alt="channel-thumb">
-            </a>
-        </div>
-        <div>
-            <h3><a href="{% url 'channel_id' channel_info.channel_id %}">{{ channel_info.channel_name }}</a></h3>
-            {% if channel_info.channel_subs >= 1000000 %}
-                <span>Subscribers: {{ channel_info.channel_subs|intword }}</span>
-            {% else %}
-                <span>Subscribers: {{ channel_info.channel_subs|intcomma }}</span>
-            {% endif %}
-        </div>
+<div class="boxed-content">
+    <div class="channel-banner">
+        <a href="/channel/{{ channel_info.channel_id }}/"><img src="/cache/channels/{{ channel_info.channel_id }}_banner.jpg" alt="channel_banner"></a>
     </div>
-    <div class="info-box-item">
-        <div>
-            <p>Last refreshed: {{ channel_info.channel_last_refresh }}</p>
-            <p>Subscribed: 
-            {% if channel_info.channel_subscribed %}
-                <button type="button" id="{{ channel_info.channel_id }}" onclick="unsubscribe(this.id)" title="Unsubscribe from {{ channel_info.channel_name }}">Unsubscribe</button>
-            {% else %}
-                false
-            {% endif %}
-            </p>
-        </div>
-    </div>
-    <div class="info-box-item">
-        <div>
-            {% if channel_info.channel_views >= 1000000 %}
-                <p>Channel views: {{ channel_info.channel_views|intword }}</p>
-            {% elif channel_info.channel_views > 0 %}
-                <p>Channel views: {{ channel_info.channel_views|intcomma }}</p>
-            {% endif %}
-            {% if max_hits %}
-                <p>Total Videos archived: {{ max_hits }}</p>
-                <p>Watched: <button title="Mark all videos from {{ channel_info.channel_name }} as watched" type="button" id="{{ channel_info.channel_id }}" onclick="isWatched(this.id)">Mark as watched</button></p>
-            {% endif %}
-            <p>Channel id: {{ channel_info.channel_id }}</p>
-            <button onclick="deleteConfirm()" id="delete-item">Delete Channel</button>
-            <div class="delete-confirm" id="delete-button">
-                <span>Delete {{ channel_info.channel_name }} including all videos? </span><button class="danger-button" onclick="deleteChannel(this)" data-id="{{ channel_info.channel_id }}">Delete</button> <button onclick="cancelDelete()">Cancel</button>
+    <div class="info-box info-box-3">
+        <div class="info-box-item">
+            <div class="round-img">
+                <a href="{% url 'channel_id' channel_info.channel_id %}">
+                    <img src="/cache/channels/{{ channel_info.channel_id }}_thumb.jpg" alt="channel-thumb">
+                </a>
+            </div>
+            <div>
+                <h3><a href="{% url 'channel_id' channel_info.channel_id %}">{{ channel_info.channel_name }}</a></h3>
+                {% if channel_info.channel_subs >= 1000000 %}
+                    <p>Subscribers: {{ channel_info.channel_subs|intword }}</p>
+                {% else %}
+                    <p>Subscribers: {{ channel_info.channel_subs|intcomma }}</p>
+                {% endif %}
+                {% if channel_info.channel_subscribed %}
+                    <button class="unsubscribe" type="button" id="{{ channel_info.channel_id }}" onclick="unsubscribe(this.id)" title="Unsubscribe from {{ channel_info.channel_name }}">Unsubscribe</button>
+                {% else %}
+                    <button type="button" id="{{ channel_info.channel_id }}" onclick="subscribe(this.id)" title="Subscribe to {{ channel_info.channel_name }}">Subscribe</button>
+                {% endif %}
             </div>
         </div>
-    </div>
-</div>
-{% if channel_info.channel_description %}
-    <div class="info-box-item description-box">
-        <p>Description: <button onclick="textReveal()" id="text-reveal-button">Show</button></p>
-        <div id="text-reveal" class="description-text">
-            {{ channel_info.channel_description|linebreaks }}
-        </div>
-    </div>
-{% endif %}
-<div class="view-icons">
-    <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="grid" alt="grid view">
-    <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="list" alt="list view">
-</div>
-<div id="player" class="video-player"></div>
-<h2>Videos</h2>
-<div class="video-list {{ view_style }}">
-    {% if videos %}
-        {% for video in videos %}
-            <div class="video-item {{ view_style }}">
-                <a href="#player" data-src="/media/{{ video.source.media_url }}" data-thumb="/cache/{{ video.source.vid_thumb_url }}" data-title="{{ video.source.title }}" data-channel="{{ video.source.channel.channel_name }}" data-id="{{ video.source.youtube_id }}" onclick="createPlayer(this)">
-                    <div class="video-thumb-wrap {{ view_style }}">
-                        <div class="video-thumb">
-                            <img src="/cache/{{ video.source.vid_thumb_url }}" alt="video-thumb">
-                        </div>
-                        <div class="video-play">
-                            <img src="{% static 'img/icon-play.svg' %}" alt="play-icon">
-                        </div>
-                    </div>
-                </a>
-                <div class="video-desc {{ view_style }}">
-                    <div class="video-desc-player" id="video-info-{{ video.source.youtube_id }}">
-                        {% if video.source.player.watched %}
-                            <img src="{% static 'img/icon-seen.svg' %}" alt="seen-icon" id="{{ video.source.youtube_id }}" onclick="isUnwatched(this.id)" class="seen-icon" title="Mark as unwatched">
-                        {% else %}
-                            <img src="{% static 'img/icon-unseen.svg' %}" alt="unseen-icon" id="{{ video.source.youtube_id }}" onclick="isWatched(this.id)" class="unseen-icon" title="Mark as watched.">
-                        {% endif %}
-                        <span>{{ video.source.published }} | {{ video.source.player.duration_str }}</span>
-                    </div>
-                    <div>
-                        <a class="video-more" href="{% url 'video' video.source.youtube_id %}"><h2>{{ video.source.title }}</h2></a>
-                    </div>
+        <div class="info-box-item">
+            <div>
+                <p>Last refreshed: {{ channel_info.channel_last_refresh }}</p>
+                {% if channel_info.channel_active %}
+                    <p>Youtube: <a href="https://www.youtube.com/channel/{{ channel_info.channel_id }}" target="_blank">Active</a></p>
+                {% else %}
+                    <p>Youtube: Deactivated</p>
+                {% endif %}
+                <button onclick="deleteConfirm()" id="delete-item">Delete Channel</button>
+                <div class="delete-confirm" id="delete-button">
+                    <span>Delete {{ channel_info.channel_name }} including all videos? </span><button class="danger-button" onclick="deleteChannel(this)" data-id="{{ channel_info.channel_id }}">Delete</button> <button onclick="cancelDelete()">Cancel</button>
                 </div>
             </div>
-        {% endfor %}
-    {% else %}
-        <h2>No videos found...</h2>
+        </div>
+        <div class="info-box-item">
+            <div>
+                {% if channel_info.channel_views >= 1000000 %}
+                    <p>Channel views: {{ channel_info.channel_views|intword }}</p>
+                {% elif channel_info.channel_views > 0 %}
+                    <p>Channel views: {{ channel_info.channel_views|intcomma }}</p>
+                {% endif %}
+                {% if max_hits %}
+                    <p>Total Videos archived: {{ max_hits }}</p>
+                    <p>Watched: <button title="Mark all videos from {{ channel_info.channel_name }} as watched" type="button" id="watched-button" data-id="{{ channel_info.channel_id }}" onclick="isWatchedButton(this)">Mark as watched</button></p>
+                {% endif %}
+                <button title="Search for playlists on YouTube for {{ channel_info.channel_name }}" type="button" id="find-playlists-button" data-id="{{ channel_info.channel_id }}" onclick="findPlaylists(this)">Find Playlists</button>
+                <a href="/playlist/?search={{ channel_info.channel_id }}" title="Show all playlists belonging to {{ channel_info.channel_name }}"><button>Show Playlists</button></a>
+            </div>
+        </div>
+    </div>
+    {% if channel_info.channel_description %}
+        <div class="info-box-item description-box">
+            <p>Description: <button onclick="textReveal()" id="text-reveal-button">Show</button></p>
+            <div id="text-reveal" class="description-text">
+                {{ channel_info.channel_description|linebreaks }}
+            </div>
+        </div>
     {% endif %}
+    <div id="notifications" data="channel_id"></div>
+    <div class="view-controls">
+        <div class="toggle">
+            <span>Hide watched videos:</span>
+            <div class="toggleBox">
+                <input id="hide_watched" onclick="toggleCheckbox(this)" type="checkbox" {% if hide_watched %}checked{% endif %}>
+                {% if hide_watched %}
+                    <label for="" class="ofbtn">Off</label>
+                {% else %}
+                    <label for="" class="onbtn">On</label>
+                {% endif %}
+            </div>
+        </div>
+        <div class="sort">
+            <div id="hidden-form">
+                <span>Sort by:</span>
+                <select name="sort" id="sort" onchange="sortChange(this.value)">
+                    <option value="published" {% if sort_by == "published" %}selected{% endif %}>date published</option>
+                    <option value="downloaded" {% if sort_by == "downloaded" %}selected{% endif %}>date downloaded</option>
+                    <option value="views" {% if sort_by == "views" %}selected{% endif %}>views</option>
+                    <option value="likes" {% if sort_by == "likes" %}selected{% endif %}>likes</option>
+                </select>
+                <select name="sord-order" id="sort-order" onchange="sortChange(this.value)">
+                    <option value="asc" {% if sort_order == "asc" %}selected{% endif %}>asc</option>
+                    <option value="desc" {% if sort_order == "desc" %}selected{% endif %}>desc</option>
+                </select>
+            </div>
+        </div>
+        <div class="view-icons">
+            <img src="{% static 'img/icon-sort.svg' %}" alt="sort-icon" onclick="showForm()" id="animate-icon">
+            <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="grid" alt="grid view">
+            <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="list" alt="list view">
+        </div>
+    </div>
 </div>
+<div id="player" class="player-wrapper"></div>
+<div class="boxed-content">
+    <div class="video-list {{ view_style }}">
+        {% if results %}
+            {% for video in results %}
+                <div class="video-item {{ view_style }}">
+                    <a href="#player" data-src="/media/{{ video.source.media_url }}" data-thumb="/cache/{{ video.source.vid_thumb_url }}" data-title="{{ video.source.title }}" data-channel="{{ video.source.channel.channel_name }}" data-channel-id="{{ video.source.channel.channel_id }}" data-id="{{ video.source.youtube_id }}" onclick="createPlayer(this)">
+                        <div class="video-thumb-wrap {{ view_style }}">
+                            <div class="video-thumb">
+                                <img src="/cache/{{ video.source.vid_thumb_url }}" alt="video-thumb">
+                            </div>
+                            <div class="video-play">
+                                <img src="{% static 'img/icon-play.svg' %}" alt="play-icon">
+                            </div>
+                        </div>
+                    </a>
+                    <div class="video-desc {{ view_style }}">
+                        <div class="video-desc-player" id="video-info-{{ video.source.youtube_id }}">
+                            {% if video.source.player.watched %}
+                                <img src="{% static 'img/icon-seen.svg' %}" alt="seen-icon" id="{{ video.source.youtube_id }}" onclick="isUnwatched(this.id)" class="seen-icon" title="Mark as unwatched">
+                            {% else %}
+                                <img src="{% static 'img/icon-unseen.svg' %}" alt="unseen-icon" id="{{ video.source.youtube_id }}" onclick="isWatched(this.id)" class="unseen-icon" title="Mark as watched.">
+                            {% endif %}
+                            <span>{{ video.source.published }} | {{ video.source.player.duration_str }}</span>
+                        </div>
+                        <div>
+                            <a class="video-more" href="{% url 'video' video.source.youtube_id %}"><h2>{{ video.source.title }}</h2></a>
+                        </div>
+                    </div>
+                </div>
+            {% endfor %}
+        {% else %}
+            <h2>No videos found...</h2>
+        {% endif %}
+    </div>
+</div>
+<script type="text/javascript" src="{% static 'progress.js' %}"></script>
 {% endblock content %}

tubearchivist/home/templates/home/downloads.html

@@ -1,87 +1,92 @@
 {% extends "home/base.html" %}
 {% load static %}
 {% block content %}
-<div class="title-bar">
-    <h1>Downloads</h1>
-</div>
-<div id="downloadMessage"></div>
-<div id="downloadControl"></div>
-<div class="info-box info-box-3">
-    <div class="icon-text">
-        <img id="rescan-icon" onclick="rescanPending()" src="{% static 'img/icon-rescan.svg' %}" alt="rescan-icon">
-        <p>Rescan subscriptions</p>
+<div class="boxed-content">
+    <div class="title-bar">
+        <h1>Downloads</h1>
     </div>
-    <div class="icon-text">
-        <img id="download-icon" onclick="dlPending()" src="{% static 'img/icon-download.svg' %}" alt="download-icon">
-        <p>Start download</p>
-    </div>
-    <div class="icon-text">
-        <img id="add-icon" onclick="showForm()" src="{% static 'img/icon-add.svg' %}" alt="add-icon">
-        <p>Add to download queue</p>
-        <div class="show-form">
-            <form id='hidden-form' action="/downloads/" method="post">
-                {% csrf_token %}
-                <textarea rows="4" placeholder="Enter Video Urls or IDs here..." id="vid-url" name="vid-url"></textarea>
-                <button type="submit">Add to download queue</button>
-            </form>
+    <div id="notifications" data="download"></div>
+    <div id="downloadControl"></div>
+    <div class="info-box info-box-3">
+        <div class="icon-text">
+            <img id="rescan-icon" onclick="rescanPending()" src="{% static 'img/icon-rescan.svg' %}" alt="rescan-icon">
+            <p>Rescan subscriptions</p>
         </div>
-    </div>
-</div>
-<div class="view-controls">
-    <div class="toggle">
-        <span>Show only ignored videos:</span>
-        <div class="toggleBox">
-            <input 
-                id="show_ignored_only" onclick="toggleCheckbox(this)" type="checkbox"
-                {% if show_ignored_only %}
-                    checked
-                {% endif %}
-                >
-            <label for="" class="onbtn">On</label>
-            <label for="" class="ofbtn">Off</label>
+        <div class="icon-text">
+            <img id="download-icon" onclick="dlPending()" src="{% static 'img/icon-download.svg' %}" alt="download-icon">
+            <p>Start download</p>
         </div>
-    </div>
-    <div class="view-icons">
-        <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="downloads" data-value="grid" alt="grid view">
-        <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="downloads" data-value="list" alt="list view">
-    </div>
-</div>
-{% if show_ignored_only %}
-    <h2>Ignored from download</h2>
-{% else %}
-    <h2>Download queue</h2>
-{% endif %}
-<h3>Total videos: {{ max_hits }}</h3>
-<div class="dl-list {{ view_style }}">
-    {% if all_video_hits %}
-        {% for video in all_video_hits %}
-            <div class="dl-item {{ view_style }}" id="dl-{{ video.youtube_id }}">
-                <div class="dl-thumb {{ view_style }}">
-                    <img src="/cache/{{ video.vid_thumb_url }}" alt="video_thumb">
-                </div>
-                <div class="dl-desc {{ view_style }}">
-                    {% if show_ignored_only %}
-                        <h3>Ignore: {{ video.title }}</h3>
-                    {% else %}
-                        <h3>Download: {{ video.title }}</h3>
-                    {% endif %}
-                    {% if video.channel_indexed %}
-                        <a href="{% url 'channel_id' video.channel_id %}">{{ video.channel_name }}</a>
-                    {% else %}
-                        <span>{{ video.channel_name }}</span>
-                    {% endif %}
-                    <p>Published: {{ video.published }} | Duration: {{ video.duration }} | {{ video.youtube_id }}</p>
-                    {% if show_ignored_only %}
-                        <button data-id="{{ video.youtube_id }}" onclick="forgetIgnore(this)">Forget</button>
-                        <button data-id="{{ video.youtube_id }}" onclick="addSingle(this)">Add to queue</button>
-                    {% else %}
-                        <button data-id="{{ video.youtube_id }}" onclick="toIgnore(this)">Ignore</button>
-                        <button id="{{ video.youtube_id }}" data-id="{{ video.youtube_id }}" onclick="downloadNow(this)">Download now</button>
-                    {% endif %}
-                </div>
+        <div class="icon-text">
+            <img id="animate-icon" onclick="showForm()" src="{% static 'img/icon-add.svg' %}" alt="add-icon">
+            <p>Add to download queue</p>
+            <div class="show-form">
+                <form id='hidden-form' action="/downloads/" method="post">
+                    {% csrf_token %}
+                    {{ add_form }}
+                    <button type="submit">Add to download queue</button>
+                </form>
             </div>
-        {% endfor %}
-    {% endif %}
+        </div>
+    </div>
+    <div class="view-controls">
+        <div class="toggle">
+            <span>Show only ignored videos:</span>
+            <div class="toggleBox">
+                <input id="show_ignored_only" onclick="toggleCheckbox(this)" type="checkbox" {% if show_ignored_only %}checked{% endif %}>
+                {% if show_ignored_only %}
+                    <label for="" class="ofbtn">Off</label>
+                {% else %}
+                    <label for="" class="onbtn">On</label>
+                {% endif %}
+            </div>
+        </div>
+        <div class="view-icons">
+            <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="downloads" data-value="grid" alt="grid view">
+            <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="downloads" data-value="list" alt="list view">
+        </div>
+    </div>
+    <div class="title-split">
+        {% if show_ignored_only %}
+            <h2>Ignored from download</h2>
+            <button onclick="deleteQueue(this)" data-id="ignore" title="Delete all previously ignored videos from the queue">Delete all ignored</button>
+        {% else %}
+            <h2>Download queue</h2>
+            <button onclick="deleteQueue(this)" data-id="pending" title="Delete all pending videos from the queue">Delete all queued</button>
+        {% endif %}
+    </div>
+    <h3>Total videos: {{ max_hits }}</h3>
+    <div class="dl-list {{ view_style }}">
+        {% if results %}
+            {% for video in results %}
+                <div class="dl-item {{ view_style }}" id="dl-{{ video.source.youtube_id }}">
+                    <div class="dl-thumb {{ view_style }}">
+                        <img src="/cache/{{ video.source.vid_thumb_url }}" alt="video_thumb">
+                        {% if show_ignored_only %}
+                            <span>ignored</span>
+                        {% else %}
+                            <span>queued</span>
+                        {% endif %}
+                    </div>
+                    <div class="dl-desc {{ view_style }}">
+                        <h3>{{ video.source.title }}</h3>
+                        {% if video.source.channel_indexed %}
+                            <a href="{% url 'channel_id' video.source.channel_id %}">{{ video.source.channel_name }}</a>
+                        {% else %}
+                            <span>{{ video.source.channel_name }}</span>
+                        {% endif %}
+                        <p>Published: {{ video.source.published }} | Duration: {{ video.source.duration }} | {{ video.source.youtube_id }}</p>
+                        {% if show_ignored_only %}
+                            <button data-id="{{ video.source.youtube_id }}" onclick="forgetIgnore(this)">Forget</button>
+                            <button data-id="{{ video.source.youtube_id }}" onclick="addSingle(this)">Add to queue</button>
+                        {% else %}
+                            <button data-id="{{ video.source.youtube_id }}" onclick="toIgnore(this)">Ignore</button>
+                            <button id="{{ video.source.youtube_id }}" data-id="{{ video.source.youtube_id }}" onclick="downloadNow(this)">Download now</button>
+                        {% endif %}
+                    </div>
+                </div>
+            {% endfor %}
+        {% endif %}
+    </div>
 </div>
 <script type="text/javascript" src="{% static 'progress.js' %}"></script>
 {% endblock content %}

tubearchivist/home/templates/home/home.html

@@ -1,81 +1,79 @@
 {% extends "home/base.html" %}
 {% block content %}
 {% load static %}
-<div class="title-bar">
-    <h1>Recent Videos</h1>
-</div>
-<div class="info-box info-box-2">
-    <div class="sort">
-        <p>Sort order from <span class="settings-current">{{ sortorder }}</span>
-            <select name="sort" id="sort" onchange="sortChange(this.value)">
-                <option value="" disabled selected> -- change sort order -- </option>
-                <option value="published">date published</option>
-                <option value="downloaded">date downloaded</option>
-            </select>
-        </p>
+<div class="boxed-content">
+    <div class="title-bar">
+        <h1>Recent Videos</h1>
     </div>
-    <div class="search-form icon-text">
-        <div class="search-icon">
-            <img src="{% static 'img/icon-search.svg' %}" alt="search-icon" onclick="showSearch()">
-        </div>
-        <form action="/" method="POST" id="search-box">
-            {% csrf_token %}
-            <input name="videoSearch" list="resultBox" type="text" id="searchInput" autocomplete="off">
-        </form>
-    </div>
-</div>
-<div class="view-controls">
-    <div class="toggle">
-        <span>Hide watched videos:</span>
-        <div class="toggleBox">
-            <input 
-                id="hide_watched" onclick="toggleCheckbox(this)" type="checkbox"
+    <div class="view-controls">
+        <div class="toggle">
+            <span>Hide watched:</span>
+            <div class="toggleBox">
+                <input id="hide_watched" onclick="toggleCheckbox(this)" type="checkbox" {% if hide_watched %}checked{% endif %}>
                 {% if hide_watched %}
-                    checked
+                    <label for="" class="ofbtn">Off</label>
+                {% else %}
+                    <label for="" class="onbtn">On</label>
                 {% endif %}
-                >
-            <label for="" class="onbtn">On</label>
-            <label for="" class="ofbtn">Off</label>
+            </div>
+        </div>
+        <div class="sort">
+            <div id="hidden-form">
+                <span>Sort by:</span>
+                <select name="sort" id="sort" onchange="sortChange(this.value)">
+                    <option value="published" {% if sort_by == "published" %}selected{% endif %}>date published</option>
+                    <option value="downloaded" {% if sort_by == "downloaded" %}selected{% endif %}>date downloaded</option>
+                    <option value="views" {% if sort_by == "views" %}selected{% endif %}>views</option>
+                    <option value="likes" {% if sort_by == "likes" %}selected{% endif %}>likes</option>
+                </select>
+                <select name="sord-order" id="sort-order" onchange="sortChange(this.value)">
+                    <option value="asc" {% if sort_order == "asc" %}selected{% endif %}>asc</option>
+                    <option value="desc" {% if sort_order == "desc" %}selected{% endif %}>desc</option>
+                </select>
+            </div>
+        </div>
+        <div class="view-icons">
+            <img src="{% static 'img/icon-sort.svg' %}" alt="sort-icon" onclick="showForm()" id="animate-icon">
+            <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="grid" alt="grid view">
+            <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="list" alt="list view">
         </div>
     </div>
-    <div class="view-icons">
-        <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="grid" alt="grid view">
-        <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="list" alt="list view">
-    </div>
 </div>
-<div id="player" class="video-player"></div>
-<div class="video-list {{ view_style }}">
-    {% if videos %}
-        {% for video in videos %}
-            <div class="video-item {{ view_style }}">
-                <a href="#player" data-src="/media/{{ video.source.media_url }}" data-thumb="/cache/{{ video.source.vid_thumb_url }}" data-title="{{ video.source.title }}" data-channel="{{ video.source.channel.channel_name }}" data-id="{{ video.source.youtube_id }}" onclick="createPlayer(this)">
-                    <div class="video-thumb-wrap {{ view_style }}">
-                        <div class="video-thumb">
-                            <img src="/cache/{{ video.source.vid_thumb_url }}" alt="video-thumb">
+<div id="player" class="player-wrapper"></div>
+<div class="boxed-content">
+    <div class="video-list {{ view_style }}">
+        {% if results %}
+            {% for video in results %}
+                <div class="video-item {{ view_style }}">
+                    <a href="#player" data-src="/media/{{ video.source.media_url }}" data-thumb="/cache/{{ video.source.vid_thumb_url }}" data-title="{{ video.source.title }}" data-channel="{{ video.source.channel.channel_name }}" data-channel-id="{{ video.source.channel.channel_id }}" data-id="{{ video.source.youtube_id }}" onclick="createPlayer(this)">
+                        <div class="video-thumb-wrap {{ view_style }}">
+                            <div class="video-thumb">
+                                <img src="/cache/{{ video.source.vid_thumb_url }}" alt="video-thumb">
+                            </div>
+                            <div class="video-play">
+                                <img src="{% static 'img/icon-play.svg' %}" alt="play-icon">
+                            </div>
                         </div>
-                        <div class="video-play">
-                            <img src="{% static 'img/icon-play.svg' %}" alt="play-icon">
+                    </a>
+                    <div class="video-desc {{ view_style }}">
+                        <div class="video-desc-player" id="video-info-{{ video.source.youtube_id }}">
+                            {% if video.source.player.watched %}
+                                <img src="{% static 'img/icon-seen.svg' %}" alt="seen-icon" id="{{ video.source.youtube_id }}" onclick="isUnwatched(this.id)" class="seen-icon" title="Mark as unwatched">
+                            {% else %}
+                                <img src="{% static 'img/icon-unseen.svg' %}" alt="unseen-icon" id="{{ video.source.youtube_id }}" onclick="isWatched(this.id)" class="unseen-icon" title="Mark as watched.">
+                            {% endif %}
+                            <span>{{ video.source.published }} | {{ video.source.player.duration_str }}</span>
+                        </div>
+                        <div>
+                            <a href="{% url 'channel_id' video.source.channel.channel_id %}"><h3>{{ video.source.channel.channel_name }}</h3></a>
+                            <a class="video-more" href="{% url 'video' video.source.youtube_id %}"><h2>{{ video.source.title }}</h2></a>
                         </div>
-                    </div>
-                </a>
-                <div class="video-desc {{ view_style }}">
-                    <div class="video-desc-player" id="video-info-{{ video.source.youtube_id }}">
-                        {% if video.source.player.watched %}
-                            <img src="{% static 'img/icon-seen.svg' %}" alt="seen-icon" id="{{ video.source.youtube_id }}" onclick="isUnwatched(this.id)" class="seen-icon" title="Mark as unwatched">
-                        {% else %}
-                            <img src="{% static 'img/icon-unseen.svg' %}" alt="unseen-icon" id="{{ video.source.youtube_id }}" onclick="isWatched(this.id)" class="unseen-icon" title="Mark as watched.">
-                        {% endif %}
-                        <span>{{ video.source.published }} | {{ video.source.player.duration_str }}</span>
-                    </div>
-                    <div>
-                        <a href="{% url 'channel_id' video.source.channel.channel_id %}"><h3>{{ video.source.channel.channel_name }}</h3></a>
-                        <a class="video-more" href="{% url 'video' video.source.youtube_id %}"><h2>{{ video.source.title }}</h2></a>
                     </div>
                 </div>
-            </div>
-        {% endfor %}
-    {% else %}
-        <h2>No videos found...</h2>
-    {% endif %}
+            {% endfor %}
+        {% else %}
+            <h2>No videos found...</h2>
+        {% endif %}
+    </div>
 </div>
 {% endblock content %}

tubearchivist/home/templates/home/login.html

@@ -0,0 +1,56 @@
+{% load static %}
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta http-equiv="X-UA-Compatible" content="IE=edge">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>TA | Welcome</title>
+    <link rel="stylesheet" type="text/css" href="{% static 'css/style.css' %}">
+    <link rel="apple-touch-icon" sizes="180x180" href="{% static 'favicon/apple-touch-icon.png' %}">
+    <link rel="icon" type="image/png" sizes="32x32" href="{% static 'favicon/favicon-32x32.png' %}">
+    <link rel="icon" type="image/png" sizes="16x16" href="{% static 'favicon/favicon-16x16.png' %}">
+    <link rel="manifest" href="{% static 'favicon/site.webmanifest' %}">
+    <link rel="mask-icon" href="{% static 'favicon/safari-pinned-tab.svg' %}" color="#01202e">
+    <link rel="shortcut icon" href="{% static 'favicon/favicon.ico' %}">
+    <meta name="apple-mobile-web-app-title" content="TubeArchivist">
+    <meta name="application-name" content="TubeArchivist">
+    <meta name="msapplication-TileColor" content="#01202e">
+    <meta name="msapplication-config" content="{% static 'favicon/browserconfig.xml' %}">
+    <meta name="theme-color" content="#01202e">
+    {% if colors == "dark" %}
+        <link rel="stylesheet" href="{% static 'css/dark.css' %}">
+    {% else %}
+        <link rel="stylesheet" href="{% static 'css/light.css' %}">
+    {% endif %}
+</head>
+<body>
+    <div class="boxed-content login-page">
+        {% if colors == 'dark' %}
+            <img src="{% static 'img/logo-tube-archivist-dark.png' %}" alt="tube-archivist-logo">
+        {% endif %}
+        {% if colors == 'light' %}
+            <img src="{% static 'img/logo-tube-archivist-light.png' %}" alt="tube-archivist-banner">
+        {% endif %}
+        <h1>Tube Archivist</h1>
+        <h2>Your Self Hosted YouTube Media Server</h2>
+        {% if form_error %}
+            <p class="danger-zone">Failed to login.</p>
+        {% endif %}
+        <form action="/login/" method="POST" name="login">
+            {% csrf_token %}
+            {{ form.username }}<br>
+            {{ form.password }}<br>
+            <p>Remember me: {{ form.remember_me }}</p>
+            <input type="hidden" name="next" value="{{ request.GET.next }}" />
+            <button type="submit">Login</button>
+        </form>
+        <p class="login-links"><span><a href="https://github.com/bbilly1/tubearchivist" target="_blank">Github</a></span> <span><a href="https://github.com/bbilly1/tubearchivist#donate" target="_blank">Donate</a></span></p>
+    </div>
+    <div class="footer-colors">
+        <div class="col-1"></div>
+        <div class="col-2"></div>
+        <div class="col-3"></div>
+    </div>
+</body>
+</html>

tubearchivist/home/templates/home/playlist.html

@@ -0,0 +1,65 @@
+{% extends "home/base.html" %}
+{% load static %}
+{% block content %}
+<div class="boxed-content">
+    <div class="title-split">
+        <div class="title-bar">
+            <h1>Playlists</h1>
+        </div>
+        <div class="title-split-form">
+            <img id="animate-icon" onclick="showForm()" src="{% static 'img/icon-add.svg' %}" alt="add-icon" title="Subscribe to Playlists">
+            <div class="show-form">
+                <form id="hidden-form" action="/playlist/" method="post">
+                    {% csrf_token %}
+                    {{ subscribe_form }}
+                    <button type="submit">Subscribe</button>
+                </form>
+            </div>
+        </div>
+    </div>
+    <div id="notifications" data="playlist"></div>
+    <div class="view-controls">
+        <div class="toggle">
+            <span>Show subscribed only:</span>
+            <div class="toggleBox">
+                <input id="show_subed_only" onclick="toggleCheckbox(this)" type="checkbox" {% if show_subed_only %}checked{% endif %}>
+                {% if show_subed_only %}
+                    <label for="" class="ofbtn">Off</label>
+                {% else %}
+                    <label for="" class="onbtn">On</label>
+                {% endif %}
+            </div>
+        </div>
+        <div class="view-icons">
+            <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="playlist" data-value="grid" alt="grid view">
+            <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="playlist" data-value="list" alt="list view">
+        </div>
+    </div>
+    <div class="playlist-list {{ view_style }}">
+        {% if results %}
+            {% for playlist in results %}
+            <div class="playlist-item {{ view_style }}">
+                <div class="playlist-thumbnail">
+                    <a href="{% url 'playlist_id' playlist.source.playlist_id %}">
+                        <img src="/cache/playlists/{{ playlist.source.playlist_id }}.jpg" alt="{{ playlist.source.playlist_id }}-thumbnail">
+                    </a>
+                </div>
+                <div class="playlist-desc {{ view_style }}">
+                    <a href="{% url 'channel_id' playlist.source.playlist_channel_id %}"><h3>{{ playlist.source.playlist_channel }}</h3></a>
+                    <a href="{% url 'playlist_id' playlist.source.playlist_id %}"><h2>{{ playlist.source.playlist_name }}</h2></a>
+                    <p>Last refreshed: {{ playlist.source.playlist_last_refresh }}</p>
+                    {% if playlist.source.playlist_subscribed %}
+                        <button class="unsubscribe" type="button" id="{{ playlist.source.playlist_id }}" onclick="unsubscribe(this.id)" title="Unsubscribe from {{ playlist.source.playlist_name }}">Unsubscribe</button>
+                    {% else %}
+                        <button type="button" id="{{ playlist.source.playlist_id }}" onclick="subscribe(this.id)" title="Subscribe to {{ playlist.source.playlist_name }}">Subscribe</button>
+                    {% endif %}
+                </div>
+            </div>
+            {% endfor %}
+        {% else %}
+            <h2>No playlists found...</h2>
+        {% endif %}
+    </div>
+</div>
+<script type="text/javascript" src="{% static 'progress.js' %}"></script>
+{% endblock content %}

tubearchivist/home/templates/home/playlist_id.html

@@ -0,0 +1,120 @@
+{% extends "home/base.html" %}
+{% load static %}
+{% load humanize %}
+{% block content %}
+<div class="boxed-content">
+    <div class="title-bar">
+        <h1>{{ playlist_info.playlist_name }}</h1>
+    </div>
+    <div class="info-box info-box-3">
+        <div class="info-box-item">
+            <div class="round-img">
+                <a href="{% url 'channel_id' channel_info.channel_id %}">
+                    <img src="/cache/channels/{{ channel_info.channel_id }}_thumb.jpg" alt="channel-thumb">
+                </a>
+            </div>
+            <div>
+                <h3><a href="{% url 'channel_id' channel_info.channel_id %}">{{ channel_info.channel_name }}</a></h3>
+                {% if channel_info.channel_subs >= 1000000 %}
+                    <span>Subscribers: {{ channel_info.channel_subs|intword }}</span>
+                {% else %}
+                    <span>Subscribers: {{ channel_info.channel_subs|intcomma }}</span>
+                {% endif %}
+            </div>
+        </div>
+        <div class="info-box-item">
+            <div>
+                <p>Last refreshed: {{ playlist_info.playlist_last_refresh }}</p>
+                <p>Playlist:
+                    {% if playlist_info.playlist_subscribed %}
+                        <button class="unsubscribe" type="button" id="{{ playlist_info.playlist_id }}" onclick="unsubscribe(this.id)" title="Unsubscribe from {{ playlist_info.playlist_name }}">Unsubscribe</button>
+                    {% else %}
+                        <button type="button" id="{{ playlist_info.playlist_id }}" onclick="subscribe(this.id)" title="Subscribe to {{ playlist_info.playlist_name }}">Subscribe</button>
+                    {% endif %}
+                </p>
+                {% if playlist_info.playlist_active %}
+                    <p>Youtube: <a href="https://www.youtube.com/playlist?list={{ playlist_info.playlist_id }}" target="_blank">Active</a></p>
+                {% else %}
+                    <p>Youtube: Deactivated</p>
+                {% endif %}
+                <button onclick="deleteConfirm()" id="delete-item">Delete Playlist</button>
+                <div class="delete-confirm" id="delete-button">
+                    <span>Delete {{ playlist_info.playlist_name }}?</span>
+                    <button onclick="deletePlaylist(this)" data-action="metadata" data-id="{{ playlist_info.playlist_id }}">Delete metadata</button>
+                    <button onclick="deletePlaylist(this)" data-action="all" class="danger-button" data-id="{{ playlist_info.playlist_id }}">Delete all</button><br>
+                    <button onclick="cancelDelete()">Cancel</button>
+                </div>
+            </div>
+        </div>
+        <div class="info-box-item">
+            <div>
+                {% if max_hits %}
+                <p>Total Videos archived: {{ max_hits }}/{{ playlist_info.playlist_entries|length }}</p>
+                <p>Watched: <button title="Mark all videos from {{ playlist_info.playlist_name }} as watched" type="button" id="watched-button" data-id="{{ playlist_info.playlist_id }}" onclick="isWatchedButton(this)">Mark as watched</button></p>
+                {% endif %}
+            </div>
+        </div>
+    </div>
+    {% if playlist_info.playlist_description %}
+        <div class="info-box-item description-box">
+            <p>Description: <button onclick="textReveal()" id="text-reveal-button">Show</button></p>
+            <div id="text-reveal" class="description-text">
+                {{ playlist_info.playlist_description|linebreaks }}
+            </div>
+        </div>
+    {% endif %}
+    <div class="view-controls">
+        <div class="toggle">
+            <span>Hide watched videos:</span>
+            <div class="toggleBox">
+                <input id="hide_watched" onclick="toggleCheckbox(this)" type="checkbox" {% if hide_watched %}checked{% endif %}>
+                {% if hide_watched %}
+                    <label for="" class="ofbtn">Off</label>
+                {% else %}
+                    <label for="" class="onbtn">On</label>
+                {% endif %}
+            </div>
+        </div>
+        <div class="view-icons">
+            <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="grid" alt="grid view">
+            <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="list" alt="list view">
+        </div>
+    </div>
+</div>
+<div id="player" class="player-wrapper"></div>
+<div class="boxed-content">
+    <div class="video-list {{ view_style }}">
+        {% if results %}
+            {% for video in results %}
+                <div class="video-item {{ view_style }}">
+                    <a href="#player" data-src="/media/{{ video.source.media_url }}" data-thumb="/cache/{{ video.source.vid_thumb_url }}" data-title="{{ video.source.title }}" data-channel="{{ video.source.channel.channel_name }}" data-channel-id="{{ video.source.channel.channel_id }}" data-id="{{ video.source.youtube_id }}" onclick="createPlayer(this)">
+                        <div class="video-thumb-wrap {{ view_style }}">
+                            <div class="video-thumb">
+                                <img src="/cache/{{ video.source.vid_thumb_url }}" alt="video-thumb">
+                            </div>
+                            <div class="video-play">
+                                <img src="{% static 'img/icon-play.svg' %}" alt="play-icon">
+                            </div>
+                        </div>
+                    </a>
+                    <div class="video-desc {{ view_style }}">
+                        <div class="video-desc-player" id="video-info-{{ video.source.youtube_id }}">
+                            {% if video.source.player.watched %}
+                                <img src="{% static 'img/icon-seen.svg' %}" alt="seen-icon" id="{{ video.source.youtube_id }}" onclick="isUnwatched(this.id)" class="seen-icon" title="Mark as unwatched">
+                            {% else %}
+                                <img src="{% static 'img/icon-unseen.svg' %}" alt="unseen-icon" id="{{ video.source.youtube_id }}" onclick="isWatched(this.id)" class="unseen-icon" title="Mark as watched.">
+                            {% endif %}
+                            <span>{{ video.source.published }} | {{ video.source.player.duration_str }}</span>
+                        </div>
+                        <div>
+                            <a class="video-more" href="{% url 'video' video.source.youtube_id %}"><h2>{{ video.source.title }}</h2></a>
+                        </div>
+                    </div>
+                </div>
+            {% endfor %}
+        {% else %}
+            <h2>No videos found...</h2>
+        {% endif %}
+    </div>
+</div>
+{% endblock content %}

tubearchivist/home/templates/home/search.html

@@ -0,0 +1,30 @@
+{% extends "home/base.html" %}
+{% block content %}
+<div id="player" class="player-wrapper"></div>
+<div class="boxed-content">
+    <div class="title-bar">
+        <h1>Search</h1>
+    </div>
+    <div class="multi-search-box">
+        {{ search_form }}
+    </div>
+    <div class="multi-search-result">
+        <h2>Video Results</h2>
+        <div id="video-results" class="video-list {{ all_styles.home }}">
+            <p>No videos found.</p>
+        </div>
+    </div>
+    <div class="multi-search-result">
+        <h2>Channel Results</h2>
+        <div id="channel-results" class="channel-list {{ all_styles.channel }}">
+            <p>No channels found.</p>
+        </div>
+    </div>
+    <div class="multi-search-result">
+        <h2>Playlist Results</h2>
+        <div id="playlist-results" class="playlist-list {{ all_styles.playlist }}">
+            <p>No playlists found.</p>
+        </div>
+    </div>
+</div>
+{% endblock content %}

tubearchivist/home/templates/home/settings.html

@@ -1,136 +1,287 @@
 {% extends "home/base.html" %}
+{% load static %}
 {% block content %}
-<div class="title-bar">
-    <h1>Settings</h1>
-</div>
-<form action="/settings/" method="POST" name="settings-update">
-    {% csrf_token %}
-    <div class="settings-group">
-        <h2>Color scheme</h2>
-        <div class="settings-item">
-            <p>Current color scheme: <span class="settings-current">{{ config.application.colors }}</span></p>
-            <i>Select your preferred color scheme between dark and light mode.</i><br>
-            <select name="application.colors" id="application.colors">
-                <option value="" disabled selected> -- change color scheme -- </option>
-                <option value="dark">dark mode</option>
-                <option value="light">light mode</option>
-            </select>
-        </div>
+<div class="boxed-content">
+    <div id="notifications" data="setting"></div>
+    <div class="title-bar">
+        <h1>User Configurations</h1>
     </div>
-    <div class="settings-group">
-        <h2>Archive View</h2>
-        <div class="settings-item">
-            <p>Current page size: <span class="settings-current">{{ config.archive.page_size }}</span></p>
-            <i>Result of videos showing in archive page</i><br>
-            <input type="number" name="archive.page_size" id="archive.page_size">
+    <form action="/settings/" method="POST" name="user-update">
+        {% csrf_token %}
+        <div class="settings-group">
+            <h2>Color scheme</h2>
+            <div class="settings-item">
+                <p>Current color scheme: <span class="settings-current">{{ config.application.colors }}</span></p>
+                <i>Select your preferred color scheme between dark and light mode.</i><br>
+                {{ user_form.colors }}
+            </div>
         </div>
+        <div class="settings-group">
+            <h2>Archive View</h2>
+            <div class="settings-item">
+                <p>Current page size: <span class="settings-current">{{ config.archive.page_size }}</span></p>
+                <i>Result of videos showing in archive page</i><br>
+                {{ user_form.page_size }}
+            </div>
+        </div>
+        <button type="submit" name="user-settings">Update User Configurations</button>
+    </form>
+    <div class="title-bar">
+        <h1>Application Configurations</h1>
     </div>
-    <div class="settings-group">
-        <h2 id="subscriptions">Subscriptions</h2>
-        <div class="settings-item">
-            <p>Current channel page size: <span class="settings-current">{{ config.subscriptions.channel_size }}</span></p>
-            <i>Recent videos to check on check pending, max recommended 50.</i><br>
-            <input type="number" name="subscriptions.channel_size" id="subscriptions.channel_size">
+    <form action="/settings/" method="POST" name="application-update">
+        {% csrf_token %}
+        <div class="settings-group">
+            <h2 id="subscriptions">Subscriptions</h2>
+            <div class="settings-item">
+                <p>Current page size: <span class="settings-current">{{ config.subscriptions.channel_size }}</span></p>
+                <i>Recent videos for channels and playlist to check when running <b>Rescan subscriptions</b>, max recommended 50.</i><br>
+                {{ app_form.subscriptions_channel_size }}
+            </div>
         </div>
-        <div class="settings-item">
-            <p>Auto scan subscribed channels:</p>
-            <i>Coming soon</i>
+        <div class="settings-group">
+            <h2 id="downloads">Downloads</h2>
+            <div class="settings-item">
+                <p>Current download limit: <span class="settings-current">{{ config.downloads.limit_count }}</span></p>
+                <i>Limit the number of videos getting downloaded on every run. 0 (zero) to deactivate.</i><br>
+                {{ app_form.downloads_limit_count }}
+            </div>
+            <div class="settings-item">
+                <p>Current download speed limit in KB/s: <span class="settings-current">{{ config.downloads.limit_speed }}</span></p>
+                <i>Limit download speed. 0 (zero) to deactivate, e.g. 1000 (1MB/s). Speeds are in KB/s.</i><br>
+                {{ app_form.downloads_limit_speed }}
+            </div>
+            <div class="settings-item">
+                <p>Current throttled rate limit in KB/s: <span class="settings-current">{{ config.downloads.throttledratelimit }}</span></p>
+                <i>Download will restart if speeds drop below specified amount. 0 (zero) to deactivate, e.g. 100. Speeds are in KB/s.</i><br>
+                {{ app_form.downloads_throttledratelimit }}
+            </div>
+            <div class="settings-item">
+                <p>Current scraping sleep interval: <span class="settings-current">{{ config.downloads.sleep_interval }}</p>
+                <i>Seconds to sleep between calls to YouTube. Might be necessary to avoid throttling. Recommended 3.</i><br>
+                {{ app_form.downloads_sleep_interval }}
+            </div>
+            <div class="settings-item">
+                <p><span class="danger-zone">Danger Zone</span>: Current auto delete watched videos: <span class="settings-current">{{ config.downloads.autodelete_days }}</span></p>
+                <i>Auto delete watched videos after x days, 0 (zero) to deactivate:</i><br>
+                {{ app_form.downloads_autodelete_days }}
+            </div>
         </div>
-        <div class="settings-item">
-            <p>Auto download pending:</p>
-            <i>Coming soon</i>
+        <div class="settings-group">
+            <h2 id="format">Download Format</h2>
+            <div class="settings-item">
+                <p>Limit video and audio quality format for yt-dlp.<br>
+                    Currently: <span class="settings-current">{{ config.downloads.format }}</span>
+                </p>
+                <p>Example configurations:</p>
+                <ul>
+                    <li><span class="settings-current">bestvideo[height<=720]+bestaudio/best[height<=720]</span>: best audio and max video height of 720p.</li>
+                    <li><span class="settings-current">bestvideo[height<=1080]+bestaudio/best[height<=1080]</span>: best audio and max video height of 1080p.</li>
+                    <li><span class="settings-current">bestvideo[height<=1080][VCODEC=avc1]+bestaudio[ACODEC=mp4a]/mp4</span>: Max 1080p video height with iOS compatible video and audio codecs.</li>
+                    <li><span class="settings-current">0</span>: deactivate and download the best quality possible as decided by yt-dlp.</li>
+                </ul>
+                <i>Make sure your custom format gets merged into a single file. Check out the <a href="https://github.com/yt-dlp/yt-dlp#format-selection" target="_blank">documentation</a> for valid configurations.</i><br>
+                {{ app_form.downloads_format }}
+                <br>
+            </div>
+            <div class="settings-item">
+                <p>Current metadata embed setting: <span class="settings-current">{{ config.downloads.add_metadata }}</span></p>
+                <i>Metadata is not embedded into the downloaded files by default.</i><br>
+                {{ app_form.downloads_add_metadata }}
+            </div>
+            <div class="settings-item">
+                <p>Current thumbnail embed setting: <span class="settings-current">{{ config.downloads.add_thumbnail }}</span></p>
+                <i>Embed thumbnail into the mediafile.</i><br>
+                {{ app_form.downloads_add_thumbnail }}
+            </div>
         </div>
-    </div>
-    <div class="settings-group">
-        <h2 id="downloads">Downloads</h2>
-        <div class="settings-item">
-            <p>Current download limit: <span class="settings-current">{{ config.downloads.limit_count }}</span></p>
-            <i>Limit the number of videos getting downloaded on every run. 0 (zero) to deactivate.</i><br>
-            <input type="number" name="downloads.limit_count" id="downloads.limit_count">
+        <div class="settings-group">
+            <h2 id="integrations">Integrations</h2>
+            <div class="settings-item">
+                <p>Integrate with <a href="https://returnyoutubedislike.com/">returnyoutubedislike.com</a> to get dislikes and average ratings back: <span class="settings-current">{{ config.downloads.integrate_ryd }}</span></p>
+                <i>Before activating that, make sure you have a scraping sleep interval of at least 3 secs set to avoid ratelimiting issues.</i><br>
+                {{ app_form.downloads_integrate_ryd }}
+            </div>
         </div>
-        <div class="settings-item">
-            <p>Current download speed limit: <span class="settings-current">{{ config.downloads.limit_speed }}</span></p>
-            <i>Limit download speed. 0 (zero) to deactivate.</i><br>
-            <input type="number" name="downloads.limit_speed" id="downloads.limit_speed"><span>KB/sec</span>
-        </div>
-        <div class="settings-item">
-            <p>Current scraping sleep interval: <span class="settings-current">{{ config.downloads.sleep_interval }}</p>
-            <i>Seconds to sleep between calls to YouTube. Might be necessary to avoid throttling. Recommended 3.</i><br>
-            <input type="number" name="downloads.sleep_interval" id="downloads.sleep_interval">
-        </div>
-        <div class="settings-item">
-            <p>External downloader:</p>
-            <i>Coming soon</i>
-        </div>
-    </div>
-    <div class="settings-group">
-        <h2 id="format">Download Format</h2>
-        <div class="settings-item">
-            <p>Limit video and audio quality format for yt-dlp.<br>
-                Currently: <span class="settings-current">{{ config.downloads.format }}</span>
-            </p>
-            <p>Example configurations:</p>
+        <button type="submit" name="application-settings">Update Application Configurations</button>
+    </form>
+    <div class="title-bar">
+        <h1>Scheduler Setup</h1>
+        <div class="settings-group">
+            <p>Schedule settings expect a cron like format, where the first value is minute, second is hour and third is day of the week.</p>
+            <p>Examples:</p>
             <ul>
-                <li><span class="settings-current">bestvideo[height<=720]+bestaudio/best[height<=720]</span>: best audio and max video height of 720p.</li>
-                <li><span class="settings-current">bestvideo[height<=1080]+bestaudio/best[height<=1080]</span>: best audio and max video height of 1080p.</li>
-                <li><span class="settings-current">0</span>: deactivate and download the best quality possible as decided by yt-dlp.</li>
+                <li><span class="settings-current">0 15 *</span>: Run task every day at 15:00 in the afternoon.</li>
+                <li><span class="settings-current">30 8 */2</span>: Run task every second day of the week (Sun, Tue, Thu, Sat) at 08:30 in the morning.</li>
+                <li><span class="settings-current">auto</span>: Sensible default.</li>
+                <li><span class="settings-current">0</span>: (zero), deactivate that task.</li>
+            </ul>
+            <p>Note:</p>
+            <ul>
+                <li>Changes in the scheduler settings require a container restart to take effect.</li>
+                <li>Avoid an unnecessary frequent schedule to not get blocked by YouTube. For that reason <i>*</i> or <i>wildcard</i> for minutes is not supported.</li>
             </ul>
-            <i>Make sure your custom format gets merged into a single file. Check out the <a href="https://github.com/yt-dlp/yt-dlp#format-selection" target="_blank">documentation</a> for valid configurations.</i><br>
-            <input type="text" name="downloads.format" id="downloads.format">
-            <br>
-        </div>
-        <div class="settings-item">
-            <p>Current metadata embed setting: <span class="settings-current">{{ config.downloads.add_metadata }}</span></p>
-            <i>Metadata is not embedded into the downloaded files by default.</i><br>
-            <select name="downloads.add_metadata" id="downloads.add_metadata"">
-                <option value="" disabled selected> -- change metadata embed -- </option>
-                <option value="0">don't embed metadata</option>
-                <option value="1">embed metadata</option>
-            </select>
-        </div>
-        <div class="settings-item">
-            <p>Current thumbnail embed setting: <span class="settings-current">{{ config.downloads.add_thumbnail }}</span></p>
-            <i>Embed thumbnail into the mediafile.</i><br>
-            <select name="downloads.add_thumbnail" id="downloads.add_thumbnail"">
-                <option value="" disabled selected> -- change thumbnail embed -- </option>
-                <option value="0">don't embed thumbnail</option>
-                <option value="1">embed thumbnail</option>
-            </select>
         </div>
     </div>
-    <button type="submit">Update Settings</button>
-</form>
-<div class="title-bar">
-    <h1>Actions</h1>
-</div>
-<div class="settings-group">
-    <h2>Manual media files import.</h2>
-    <p>Add files to the <span class="settings-current">cache/import</span> folder. Make sure to follow the instructions in the Github <a href="https://github.com/bbilly1/tubearchivist/wiki/Settings" target="_blank">Wiki</a>.</p>
-    <div id="manual-import">
-        <button onclick="manualImport()">Start import</button>
+    <form action="/settings/" method="POST" name="scheduler-update">
+        {% csrf_token %}
+        <div class="settings-group">
+            <h2>Rescan Subscriptions</h2>
+            <div class="settings-item">
+                <p>Current rescan schedule: <span class="settings-current">
+                    {% if config.scheduler.update_subscribed %}
+                        {% for key, value in config.scheduler.update_subscribed.items %}
+                            {{ value }}
+                        {% endfor %}
+                    {% else %}
+                        False
+                    {% endif %}
+                </span></p>
+                <p>Periodically rescan your subscriptions:</p>
+                {{ scheduler_form.update_subscribed }}
+            </div>
+        </div>
+        <div class="settings-group">
+            <h2>Start download</h2>
+            <div class="settings-item">
+                <p>Current Download schedule: <span class="settings-current">
+                    {% if config.scheduler.download_pending %}
+                        {% for key, value in config.scheduler.download_pending.items %}
+                            {{ value }}
+                        {% endfor %}
+                    {% else %}
+                        False
+                    {% endif %}
+                </span></p>
+                <p>Automatic video download schedule:</p>
+                {{ scheduler_form.download_pending }}
+            </div>
+        </div>
+        <div class="settings-group">
+            <h2>Refresh Metadata</h2>
+            <div class="settings-item">
+                <p>Current Metadata refresh schedule: <span class="settings-current">
+                    {% if config.scheduler.check_reindex %}
+                        {% for key, value in config.scheduler.check_reindex.items %}
+                            {{ value }}
+                        {% endfor %}
+                    {% else %}
+                        False
+                    {% endif %}
+                </span></p>
+                <p>Daily schedule to refresh metadata from YouTube:</p>
+                {{ scheduler_form.check_reindex }}
+            </div>
+            <div class="settings-item">
+                <p>Current refresh for metadata older than x days: <span class="settings-current">{{ config.scheduler.check_reindex_days }}</span></p>
+                <p>Refresh older than x days, recommended 90:</p>
+                {{ scheduler_form.check_reindex_days }}
+            </div>
+        </div>
+        <div class="settings-group">
+            <h2>Thumbnail check</h2>
+            <div class="settings-item">
+                <p>Current thumbnail check schedule: <span class="settings-current">
+                    {% if config.scheduler.thumbnail_check %}
+                        {% for key, value in config.scheduler.thumbnail_check.items %}
+                            {{ value }}
+                        {% endfor %}
+                    {% else %}
+                        False
+                    {% endif %}
+                </span></p>
+                <p>Periodically check and cleanup thumbnails:</p>
+                {{ scheduler_form.thumbnail_check }}
+            </div>
+        </div>
+        <div class="settings-group">
+            <h2>Index backup</h2>
+            <div class="settings-item">
+                <p>Current index backup schedule: <span class="settings-current">
+                    {% if config.scheduler.run_backup %}
+                        {% for key, value in config.scheduler.run_backup.items %}
+                            {{ value }}
+                        {% endfor %}
+                    {% else %}
+                        False
+                    {% endif %}
+                </span></p>
+                <p>Automatically backup metadata to a zip file:</p>
+                {{ scheduler_form.run_backup }}
+            </div>
+            <div class="settings-item">
+                <p>Current backup files to keep: <span class="settings-current">{{ config.scheduler.run_backup_rotate }}</span></p>
+                <p>Max auto backups to keep:</p>
+                {{ scheduler_form.run_backup_rotate }}
+            </div>
+        </div>
+        <button type="submit" name="scheduler-settings">Update Scheduler Settings</button>
+    </form>
+    <div class="title-bar">
+        <h1>Actions</h1>
     </div>
-</div>
-<div class="settings-group">
-    <h2>Backup database</h2>
-    <p>Export your database to a zip file stored at <span class="settings-current">cache/backup</span>.</p>
-    <div id="db-backup">
-        <button onclick="dbBackup()">Start backup</button>
+    <div class="settings-group">
+        <h2>Manual media files import.</h2>
+        <p>Add files to the <span class="settings-current">cache/import</span> folder. Make sure to follow the instructions in the Github <a href="https://github.com/bbilly1/tubearchivist/wiki/Settings" target="_blank">Wiki</a>.</p>
+        <div id="manual-import">
+            <button onclick="manualImport()">Start import</button>
+        </div>
     </div>
-</div>
-<div class="settings-group">
-    <h2>Restore from backup</h2>
-    <p><span class="danger-zone">Danger Zone</span>: This will replace your existing index with the backup.</p>
-    <p>Add the backup zip file to the <span class="settings-current">cache/backup</span> folder.</p>
-    <div id="db-restore">
-        <button onclick="dbRestore()">Restore backup</button>
+    <div class="settings-group">
+        <h2>Embed thumbnails into media file.</h2>
+        <p>Set extracted youtube thumbnail as cover art of the media file.</p>
+        <div id="re-embed">
+            <button onclick="reEmbed()">Start process</button>
+        </div>
     </div>
-</div>
-<div class="settings-group">
-    <h2>Rescan filesystem</h2>
-    <p><span class="danger-zone">Danger Zone</span>: This will delete the metadata of deleted videos from the filesystem.</p>
-    <p>Rescan your media folder looking for missing videos and clean up index. More infos on the Github <a href="https://github.com/bbilly1/tubearchivist/wiki/Settings" target="_blank">Wiki</a>.</p>
-    <div id="fs-rescan">
-        <button onclick="fsRescan()">Rescan filesystem</button>
+    <div class="settings-group">
+        <h2>Backup database</h2>
+        <p>Export your database to a zip file stored at <span class="settings-current">cache/backup</span>.</p>
+        <div id="db-backup">
+            <button onclick="dbBackup()">Start backup</button>
+        </div>
     </div>
+    <div class="settings-group">
+        <h2>Restore from backup</h2>
+        <p><span class="danger-zone">Danger Zone</span>: This will replace your existing index with the backup.</p>
+        <p>Restore from available backup files from <span class="settings-current">cache/backup</span>.</p>
+        {% if available_backups %}
+            <div class="backup-grid-row">
+                <span></span>
+                <span>Timestamp</span>
+                <span>Source</span>
+                <span>Filename</span>
+            </div>
+            {% for backup in available_backups %}
+                <div class="backup-grid-row" id="{{ backup.filename }}">
+                    <button onclick="dbRestore(this)" data-id="{{ backup.filename }}">Restore</button>
+                    <span>{{ backup.timestamp }}</span>
+                    <span>{{ backup.reason }}</span>
+                    <span>{{ backup.filename }}</span>
+                </div>
+            {% endfor %}
+        {% else %}
+            <p>No backups found.</p>
+        {% endif %}
+    </div>
+    <div class="settings-group">
+        <h2>Rescan filesystem</h2>
+        <p><span class="danger-zone">Danger Zone</span>: This will delete the metadata of deleted videos from the filesystem.</p>
+        <p>Rescan your media folder looking for missing videos and clean up index. More infos on the Github <a href="https://github.com/bbilly1/tubearchivist/wiki/Settings" target="_blank">Wiki</a>.</p>
+        <div id="fs-rescan">
+            <button onclick="fsRescan()">Rescan filesystem</button>
+        </div>
+    </div>
+    {% if request.user.is_superuser %}
+        <div class="title-bar">
+            <h1>Users</h1>
+        </div>
+        <div class="settings-group">
+            <h2>User Management</h2>
+            <p>Access the admin interface for basic user management functionality like adding and deleting users, changing passwords and more.</p>
+            <a href="/admin/"><button>Admin Interface</button></a>
+        </div>
+    {% endif %}
 </div>
-{% endblock content %}
+<script type="text/javascript" src="{% static 'progress.js' %}"></script>
+{% endblock content %}

tubearchivist/home/templates/home/video.html

@@ -2,76 +2,116 @@
 {% block content %}
 {% load static %}
 {% load humanize %}
-<div class="title-bar">
-    <h1>Video: {{ video.title }}</h1>
-</div>
 <div class="video-main">
     <video 
         src="/media/{{ video.media_url }}" 
         poster="/cache/{{ video.vid_thumb_url }}" controls preload="false" 
-        type='video/mp4' width="100%">
+        type='video/mp4' width="100%" playsinline>
     </video>
 </div>
-<div class="info-box info-box-3 padding-box">
-    <div class="info-box-item">
-        <div class="round-img">
-            <a href="{% url 'channel_id' video.channel.channel_id %}">
-                <img src="/cache/channels/{{ video.channel.channel_id }}_thumb.jpg" alt="channel-thumb">
-            </a>
-        </div>
-        <div>
-            <h3><a href="{% url 'channel_id' video.channel.channel_id %}">{{ video.channel.channel_name }}</a></h3>
-            {% if video.channel.channel_subs >= 1000000 %}
-                <p>Subscribers: {{ video.channel.channel_subs|intword }}</p>
-            {% else %}
-                <p>Subscribers: {{ video.channel.channel_subs|intcomma }}</p>
-            {% endif %}
-        </div>
+<div class="boxed-content">
+    <div class="title-bar">
+        <h1>{{ video.title }}</h1>
     </div>
-    <div class="info-box-item">
-        <div>
-            <p>Published: {{ video.published }}</p>
-            <p>Last refreshed: {{ video.vid_last_refresh }}</p>
-            <p class="video-info-watched">Watched:
-                {% if video.player.watched %}
-                    <img src="{% static 'img/icon-seen.svg' %}" alt="seen-icon" id="{{ video.youtube_id }}" onclick="isUnwatched(this.id)" class="seen-icon" title="Mark as unwatched">
+    <div class="info-box info-box-3">
+        <div class="info-box-item">
+            <div class="round-img">
+                <a href="{% url 'channel_id' video.channel.channel_id %}">
+                    <img src="/cache/channels/{{ video.channel.channel_id }}_thumb.jpg" alt="channel-thumb">
+                </a>
+            </div>
+            <div>
+                <h3><a href="{% url 'channel_id' video.channel.channel_id %}">{{ video.channel.channel_name }}</a></h3>
+                {% if video.channel.channel_subs >= 1000000 %}
+                    <p>Subscribers: {{ video.channel.channel_subs|intword }}</p>
                 {% else %}
-                    <img src="{% static 'img/icon-unseen.svg' %}" alt="unseen-icon" id="{{ video.youtube_id }}" onclick="isWatched(this.id)" class="unseen-icon" title="Mark as watched.">
+                    <p>Subscribers: {{ video.channel.channel_subs|intcomma }}</p>
+                {% endif %}
+            </div>
+        </div>
+        <div class="info-box-item">
+            <div>
+                <p>Published: {{ video.published }}</p>
+                <p>Last refreshed: {{ video.vid_last_refresh }}</p>
+                <p class="video-info-watched">Watched:
+                    {% if video.player.watched %}
+                        <img src="{% static 'img/icon-seen.svg' %}" alt="seen-icon" id="{{ video.youtube_id }}" onclick="isUnwatched(this.id)" class="seen-icon" title="Mark as unwatched">
+                    {% else %}
+                        <img src="{% static 'img/icon-unseen.svg' %}" alt="unseen-icon" id="{{ video.youtube_id }}" onclick="isWatched(this.id)" class="unseen-icon" title="Mark as watched.">
+                    {% endif %}
+                </p>
+                {% if video.active %}
+                    <p>Youtube: <a href="https://www.youtube.com/watch?v={{ video.youtube_id }}" target="_blank">Active</a></p>
+                {% else %}
+                    <p>Youtube: Deactivated</p>
+                {% endif %}
+                <button onclick="deleteConfirm()" id="delete-item">Delete Video</button>
+                <div class="delete-confirm" id="delete-button">
+                    <span>Are you sure? </span><button class="danger-button" onclick="deleteVideo(this)" data-id="{{ video.youtube_id }}" data-redirect = "{{ video.channel.channel_id }}">Delete</button> <button onclick="cancelDelete()">Cancel</button>
+                </div>
+            </div>
+        </div>
+        <div class="info-box-item">
+            <div>
+                <p>Views: {{ video.stats.view_count|intcomma }}</p>
+                <p class="thumb-icon like"><img src="{% static 'img/icon-thumb.svg' %}" alt="thumbs-up">: {{ video.stats.like_count|intcomma }}</p>
+                {% if video.stats.dislike_count %}
+                    <p class="thumb-icon dislike"><img src="{% static 'img/icon-thumb.svg' %}" alt="thumbs-down">: {{ video.stats.dislike_count|intcomma }}</p>
+                {% endif %}
+                {% if video.stats.average_rating %}
+                    <p class="rating-stars">Rating: 
+                        {% for star in video.stats.average_rating %}
+                            <img src="/static/img/icon-star-{{ star }}.svg" alt="{{ star }}">
+                        {% endfor %}
+                    </p>
                 {% endif %}
-            </p>
-            {% if video.active %}
-                <p>Youtube: Active</p>
-            {% else %}
-                <p>Youtube: Deactivated</p>
-            {% endif %}
-            <p>Video ID: {{ video.youtube_id }}</p>
-            <button onclick="deleteConfirm()" id="delete-item">Delete Video</button>
-            <div class="delete-confirm" id="delete-button">
-                <span>Are you sure? </span><button class="danger-button" onclick="deleteVideo(this)" data-id="{{ video.youtube_id }}" data-redirect = "{{ video.channel.channel_id }}">Delete</button> <button onclick="cancelDelete()">Cancel</button>
             </div>
         </div>
     </div>
-    <div class="info-box-item">
-        <div>
-            <p>Views: {{ video.stats.view_count|intcomma }}</p>
-            <p class="thumb-icon like"><img src="{% static 'img/icon-thumb.svg' %}" alt="thumbs-up">: {{ video.stats.like_count|intcomma }}</p>
-            <p class="thumb-icon dislike"><img src="{% static 'img/icon-thumb.svg' %}" alt="thumbs-down">: {{ video.stats.dislike_count|intcomma }}</p>
-            {% if video.stats.average_rating %}
-                <p class="rating-stars">Rating: 
-                    {% for star in video.stats.average_rating %}
-                        <img src="/static/img/icon-star-{{ star }}.svg" alt="{{ star }}">
-                    {% endfor %}
-                </p>
-            {% endif %}
+    {% if video.description %}
+        <div class="info-box-item description-box">
+            <p>Description: <button onclick="textReveal()" id="text-reveal-button">Show</button></p>
+            <div id="text-reveal" class="description-text">
+                {{ video.description|linebreaks }}
+            </div>
         </div>
-    </div>
+    {% endif %}
+    {% if playlist_nav %}
+        {% for playlist_item in playlist_nav %}
+            <div class="playlist-wrap">
+                <a href="{% url 'playlist_id' playlist_item.playlist_meta.playlist_id %}">
+                    <h3>Playlist [{{ playlist_item.playlist_meta.current_idx|add:"1" }}]: {{ playlist_item.playlist_meta.playlist_name }}</h3>
+                </a>
+                <div class="playlist-nav">    
+                    <div class="playlist-nav-item">
+                        {% if playlist_item.playlist_previous %}
+                            <a href="{% url 'video' playlist_item.playlist_previous.youtube_id %}">
+                                <img src="/cache/{{ playlist_item.playlist_previous.vid_thumb }}" alt="previous thumbnail">
+                            </a>
+                            <div class="playlist-desc">
+                                <p>Previous:</p>
+                                <a href="{% url 'video' playlist_item.playlist_previous.youtube_id %}">
+                                    <h3>[{{ playlist_item.playlist_previous.idx|add:"1" }}] {{ playlist_item.playlist_previous.title }}</h3>
+                                </a>
+                            </div>
+                        {% endif %}
+                    </div>
+                    <div class="playlist-nav-item">
+                        {% if playlist_item.playlist_next %}
+                            <div class="playlist-desc">
+                                <p>Next:</p>
+                                <a href="{% url 'video' playlist_item.playlist_next.youtube_id %}">
+                                    <h3>[{{ playlist_item.playlist_next.idx|add:"1" }}] {{ playlist_item.playlist_next.title }}</h3>
+                                </a>
+                            </div>
+                            <a href="{% url 'video' playlist_item.playlist_next.youtube_id %}">
+                                <img src="/cache/{{ playlist_item.playlist_next.vid_thumb }}" alt="previous thumbnail">
+                            </a>
+                        {% endif %}
+                    </div>
+                </div>
+            </div>
+        {% endfor %}
+    {% endif %}
 </div>
-{% if video.description %}
-    <div class="info-box-item description-box">
-        <p>Description: <button onclick="textReveal()" id="text-reveal-button">Show</button></p>
-        <div id="text-reveal" class="description-text">
-            {{ video.description|linebreaks }}
-        </div>
-    </div>
-{% endif %}
 {% endblock content %}

tubearchivist/home/urls.py

@@ -1,5 +1,8 @@
 """ all home app urls """
 
+from django.conf import settings
+from django.contrib.auth.decorators import login_required
+from django.contrib.auth.views import LogoutView
 from django.urls import path
 from home.views import (
     AboutView,
@@ -7,24 +10,48 @@ from home.views import (
     ChannelView,
     DownloadView,
     HomeView,
+    LoginView,
+    PlaylistIdView,
+    PlaylistView,
+    SearchView,
     SettingsView,
     VideoView,
+    process,
+    progress,
 )
 
-from . import views
-
 urlpatterns = [
-    path("", HomeView.as_view(), name="home"),
-    path("about/", AboutView.as_view(), name="about"),
-    path("downloads/", DownloadView.as_view(), name="downloads"),
-    path("settings/", SettingsView.as_view(), name="settings"),
-    path("process/", views.process, name="process"),
-    path("downloads/progress/", views.progress, name="progress"),
-    path("channel/", ChannelView.as_view(), name="channel"),
+    path("", login_required(HomeView.as_view()), name="home"),
+    path("login/", LoginView.as_view(), name="login"),
     path(
-        "channel/<slug:channel_id_detail>/",
-        ChannelIdView.as_view(),
+        "logout/",
+        LogoutView.as_view(),
+        {"next_page": settings.LOGOUT_REDIRECT_URL},
+        name="logout",
+    ),
+    path("about/", AboutView.as_view(), name="about"),
+    path(
+        "downloads/", login_required(DownloadView.as_view()), name="downloads"
+    ),
+    path("settings/", login_required(SettingsView.as_view()), name="settings"),
+    path("process/", login_required(process), name="process"),
+    path("progress/", login_required(progress), name="progress"),
+    path("channel/", login_required(ChannelView.as_view()), name="channel"),
+    path(
+        "channel/<slug:channel_id>/",
+        login_required(ChannelIdView.as_view()),
         name="channel_id",
     ),
-    path("video/<slug:video_id>/", VideoView.as_view(), name="video"),
+    path(
+        "video/<slug:video_id>/",
+        login_required(VideoView.as_view()),
+        name="video",
+    ),
+    path("playlist/", login_required(PlaylistView.as_view()), name="playlist"),
+    path(
+        "playlist/<slug:playlist_id>/",
+        login_required(PlaylistIdView.as_view()),
+        name="playlist_id",
+    ),
+    path("search/", login_required(SearchView.as_view()), name="search"),
 ]

tubearchivist/requirements.txt

@@ -1,9 +1,10 @@
 beautifulsoup4==4.10.0
-celery==5.1.2
-Django==3.2.8
-Pillow==8.4.0
-redis==3.5.3
-requests==2.26.0
+celery==5.2.3
+Django==4.0.1
+Pillow==9.0.0
+redis==4.1.0
+requests==2.27.1
+ryd-client==0.0.3
 uWSGI==2.0.20
 whitenoise==5.3.0
-yt_dlp==2021.10.10
+yt_dlp==2021.12.27

tubearchivist/static/css/style.css

@@ -62,7 +62,7 @@ ul {
     margin-left: 20px;
 }
 
-td, span {
+td, th, span, label {
     font-family: Sen-Regular, sans-serif;
     color: var(--main-font);
 }
@@ -102,6 +102,14 @@ button:hover {
     color: var(--main-bg);
 }
 
+.unsubscribe {
+    background-color: var(--accent-font-light);
+}
+
+.unsubscribe:hover {
+    background-color: var(--accent-font-dark);
+}
+
 .boxed-content {
     max-width: 1000px;
     width: 80%;
@@ -118,7 +126,6 @@ button:hover {
 
 .top-banner {
     text-align: center;
-    margin-top: 10px;
 }
 
 .top-banner img {
@@ -193,14 +200,16 @@ button:hover {
 }
 
 .toggleBox > .onbtn {
-    bottom: 15px;
-    left: 15px;
+    right: 35%;
+    top: 45%;
+    transform: translate(50%,-50%);
     font-family: Sen-Regular, sans-serif;
 }
 
 .toggleBox > .ofbtn {
-    bottom: 15px;
-    right: 15px;
+    left: 0;
+    top: 45%;
+    transform: translate(50%,-50%);
     font-family: Sen-Regular, sans-serif;
     color: var(--main-font);
 }
@@ -210,6 +219,10 @@ button:hover {
     display: none;
 }
 
+.delete-confirm button {
+    margin: 3px 0;
+}
+
 .danger-button {
     background-color: var(--highlight-error);
 }
@@ -221,7 +234,7 @@ button:hover {
 /* navigation */
 .top-nav {
     display: block;
-    padding: 15px 0;
+    padding: 5px 0;
     position: relative;
 }
 
@@ -250,19 +263,19 @@ button:hover {
 
 .nav-icons img {
     width: 40px;
-    padding: 0 10px;
+    padding: 0 5px;
     filter: var(--img-filter);
 }
 
 /* top of page */
 .title-bar {
-    padding: 20px 0;
+    padding-top: 30px;
 }
 
 .sort {
     display: flex;
     flex-wrap: wrap;
-    align-content: center;
+    align-items: center;
 }
 
 .padding-box {
@@ -277,23 +290,9 @@ button:hover {
     width: 50%;
 }
 
-.search-form input {
-    width: 90%;
-}
-
-.search-icon {
-    flex: 0 0 40px;
-}
-
-.search-icon img {
-    width: 100%;
-    cursor: pointer;
-    filter: var(--img-filter);
-}
-
 .view-controls {
     display: grid;
-    grid-template-columns: 1fr 1fr;
+    grid-template-columns: 1fr auto auto;
     border-bottom: 2px solid;
     border-color: var(--accent-font-dark);
     margin: 15px 0;
@@ -311,12 +310,6 @@ button:hover {
     filter: var(--img-filter);
 }
 
-#search-box {
-    display: none;
-    flex: auto;
-    padding: 10px;
-}
-
 #hidden-form {
     display: none;
 }
@@ -328,38 +321,31 @@ button:hover {
 
 
 /* video player */
-.video-player {
-    position: relative;
+.player-wrapper {
+    background-color: var(--highlight-bg);
     margin: 20px 0;
 }
 
-.video-player video {
-    max-height: 70vh;
+.video-player {
+    display: grid;
+    align-content: space-evenly;
+    height: 100vh;
+}
+
+.video-player video,
+.video-main video {
+    max-height: 80vh;
+    width: 90%;
+    max-width: 1500px;
+    margin: 0 auto;
+    display: block;
 }
 
 .player-title img {
     width: 30px;
+    margin: 10px 10px 10px 0;
 }
 
-.player-title:hover {
-    opacity: 1;
-}
-
-.player-title {
-    opacity: 0;
-    background-color: var(--highlight-bg-transparent);
-    position: absolute;
-    top: 0;
-    z-index: 1;
-    width: 100%;
-    padding: 20px 0;
-}
-
-.player-title > * {
-    margin-left: 20px;
-}
-
-
 /* video list */
 .video-list.grid {
     display: grid;
@@ -441,7 +427,8 @@ button:hover {
     text-align: left;
 }
 
-.video-desc h3 {
+.video-desc h3,
+.player-title h3 {
     font-size: 0.9em;
     text-transform: uppercase;
 }
@@ -480,10 +467,31 @@ button:hover {
 
 
 /* info box */
+.title-split {
+    display: grid;
+    grid-template-columns: 1fr 1fr;
+}
+
+.title-split img {
+    width: 40px;
+    filter: var(--img-filter);
+    cursor: pointer;
+    margin: 0 10px;
+}
+
+.title-split-form {
+    padding-top: 30px;
+    display: flex;
+}
+
 .info-box {
     display: grid;
     grid-gap: 1rem;
-    margin: 1rem 0;
+    margin-top: 1rem;
+}
+
+.description-box {
+    margin-top: 1rem;
 }
 
 .info-box-3 {
@@ -514,10 +522,83 @@ button:hover {
     margin-bottom: 10px;
 }
 
+/* login */
+.login-page {
+    display: flex;
+    flex-wrap: wrap;
+    justify-content: center;
+    text-align: center;
+    align-content: center;
+}
+
+.login-page > * {
+    width: 100%;
+}
+
+.login-page img {
+    width: 100%;
+    max-width: 200px;
+    max-height: 200px;
+    margin-bottom: 40px;
+}
+
+.login-page form {
+    margin: 30px 0;
+}
+
+.login-page input {
+    min-width: 200px;
+}
+
+#id_remember_me {
+    min-width: unset;
+}
+
+.login-page button,
+.login-page .danger-zone {
+    width: 210px;
+    margin-top: 5px;
+}
+
+.login-links a {
+    text-decoration: underline;
+    margin: 30px 0;
+    padding: 20px;
+}
+
+.footer-colors {
+    grid-row-start: 2;
+    grid-row-end: 3;
+    display: flex;
+}
+
+.footer-colors div {
+    padding: 20px 0;
+    width: 33.33%;
+}
+
+.col-1 {
+    background-color: var(--highlight-bg);
+}
+
+.col-2 {
+    background-color: var(--accent-font-dark);
+}
+
+.col-3 {
+    background-color: var(--accent-font-light);
+}
+
 /* video page */
+.video-main {
+    margin: 1rem 0;
+}
+
 .video-main video {
     max-height: 70vh;
+    margin-bottom: 1rem;
 }
+
 .video-info-watched {
     display: flex;
     align-items: center;
@@ -539,6 +620,45 @@ button:hover {
     transform: rotate(180deg);
 }
 
+.playlist-wrap {
+    background-color: var(--highlight-bg);
+    margin: 1rem 0;
+    padding: 1rem;
+}
+
+.playlist-wrap > a > h3 {
+    text-align: center;
+}
+
+.playlist-nav {
+    display: grid;
+    grid-template-columns: 1fr 1fr;
+    margin-bottom: 10px;
+}
+
+.playlist-nav-item {
+    display: flex;
+    justify-content: space-between;
+}
+
+.playlist-nav-item img {
+    width: 200px;
+}
+
+.playlist-desc {
+    padding: 5px;
+    width: 100%;
+}
+
+/* multi search page */
+.multi-search-box input {
+    width: 100%;
+}
+
+.multi-search-result {
+    padding: 1rem 0;
+}
+
 /* channel overview page */
 .channel-list.list {
     display: block;
@@ -551,7 +671,7 @@ button:hover {
 }
 
 .channel-item.list {
-    padding: 20px 0;
+    padding-bottom: 1rem;
 }
 
 .channel-item.grid > .info-box {
@@ -559,6 +679,7 @@ button:hover {
 }
 
 .channel-banner img {
+    margin-top: 1rem;
     width: 100%;
 }
 
@@ -575,6 +696,51 @@ button:hover {
     transform: translateX(-30%);
 }
 
+/* playlist overview page */
+.playlist-list.list {
+    display: grid;
+    grid-template-columns: unset;
+    gap: 1rem;
+}
+
+.playlist-list.grid {
+    display: grid;
+    grid-template-columns: 1fr 1fr 1fr;
+    gap: 1rem;
+}
+
+.playlist-item {
+    overflow: hidden;
+}
+
+.playlist-item.list {
+    display: flex;
+}
+
+.playlist-thumbnail img {
+    width: 100%;
+}
+
+.playlist-desc.grid {
+    padding: 10px;
+    height: 100%;
+    background-color: var(--highlight-bg);
+}
+
+.playlist-desc.list {
+    width: 100%;
+    padding: 10px;
+    height: unset;
+    background-color: var(--highlight-bg);
+    display: flex;
+    flex-wrap: wrap;
+    align-content: center;
+}
+
+.playlist-desc.list > a,
+.playlist-desc.list > p {
+    width: 100%;
+}
 
 /* download page */
 .icon-text {
@@ -615,6 +781,24 @@ button:hover {
     width: 30px;
 }
 
+.dl-item:hover > .dl-thumb span {
+    opacity: 1;
+}
+
+.dl-thumb {
+    position: relative;
+}
+
+.dl-thumb span {
+    position: absolute;
+    top: 5px;
+    left: 5px;
+    background-color: var(--accent-font-light);
+    padding: 5px;
+    opacity: 0;
+    transition: 300ms ease-in-out;
+}
+
 .dl-thumb.list {
     width: 25%;
 }
@@ -640,7 +824,7 @@ button:hover {
 .dl-control-icons {
     display: flex;
     justify-content: center;
-    padding: 10px 0;
+    padding-bottom: 10px;
 }
 
 .dl-control-icons img {
@@ -657,22 +841,28 @@ button:hover {
     filter: var(--img-filter-error);
 }
 
+.title-split {
+    display: flex;
+    justify-content: space-between;
+}
+
 /* status message */
-.download-progress {
+.notification {
     background-color: var(--highlight-bg);
     text-align: center;
     padding: 30px 0 15px 0;
+    margin: 1rem 0;
 }
 
-.download-progress.info {
+.notification.info {
     background-color: var(--highlight-bg);
 }
 
-.download-progress.error {
+.notification.error {
     background-color: var(--highlight-error);
 }
 
-.download-progress.error h3 {
+.notification.error h3 {
     color: #fff;
 }
 
@@ -697,6 +887,19 @@ button:hover {
     padding: 3px;
 }
 
+.backup-grid-row {
+    display: grid;
+    grid-template-columns: 10% 10% 10% auto;
+    align-items: center;
+    padding: 5px 10px;
+    border-bottom: solid 1px;
+    border-color: var(--main-font);
+}
+
+.backup-grid-row > span {
+    margin-left: 10px;
+}
+
 /* about */
 .about-section {
     padding: 20px 0;
@@ -768,13 +971,15 @@ button:hover {
     }
     .video-list.grid,
     .dl-list.grid,
-    .channel-list.grid {
+    .channel-list.grid,
+    .playlist-list.grid {
         grid-template-columns: 1fr 1fr;
     }
     .dl-thumb.list {
         width: 35%;
     }
-    .video-item.list {
+    .video-item.list,
+    .playlist-item.list {
         display: grid;
         grid-template-columns: 35% auto;
     }
@@ -784,6 +989,23 @@ button:hover {
     .two-col > div {
         width: 100%;
     }
+    .top-nav {
+        flex-wrap: wrap-reverse;
+        display: flex;
+    }
+    .nav-icons {
+        width: 100%;
+        justify-content: center;
+        position: unset;
+        transform: unset;
+    }
+    .video-player {
+        height: unset;
+        padding: 20px 0
+    }
+    .video-player video {
+        width: 90%;
+    }
 }
 
 /* phone */
@@ -794,9 +1016,14 @@ button:hover {
     .video-list.grid,
     .dl-list.grid,
     .channel-list.grid,
-    .video-item.list {
+    .video-item.list,
+    .playlist-list.list,
+    .playlist-list.grid {
         grid-template-columns: 1fr;
     }
+    .playlist-item.list {
+        display: block;
+    }
     .video-desc.grid {
         height: unset;
         display: flex;
@@ -808,28 +1035,26 @@ button:hover {
     .footer {
         text-align: center;
     }
+    .footer .boxed-content span {
+        width: 100%;
+        display: block;
+    }
     .toggle {
         flex-wrap: wrap;
     }
-    .top-nav {
-        flex-wrap: wrap-reverse;
-        display: flex;
-    }
-    .nav-icons {
-        width: 100%;
-        justify-content: center;
-        position: unset;
-        transform: unset;
+    .nav-items {
+        display: grid;
+        grid-template-columns: 1fr 1fr;
     }
     .nav-item {
-        padding: unset;
+        padding: 5px 0;
         margin: 15px;
+        text-align: center;
     }
     .sort {
         display: block;
     }
     .sort select {
-        width: 100%;
         margin: unset;
     }
     .info-box-2,
@@ -848,4 +1073,24 @@ button:hover {
     .dl-desc {
         padding: 15px;
     }
+    .backup-grid-row {
+        display: flex;
+        flex-wrap: wrap;
+        padding: 10px 0;
+        justify-content: center;
+    }
+    .backup-grid-row span {
+        padding: 5px 0;
+    }
+    .playlist-nav {
+        display: block;
+        grid-template-columns: unset;
+    }
+    .playlist-nav-item {
+        display: block;
+        justify-content: unset;
+    }
+    .playlist-nav-item img {
+        width: 100%;
+    }
 }

tubearchivist/static/favicon/browserconfig.xml

@@ -0,0 +1,9 @@
+<?xml version="1.0" encoding="utf-8"?>
+<browserconfig>
+    <msapplication>
+        <tile>
+            <square150x150logo src="/static/favicon/mstile-150x150.png"/>
+            <TileColor>#01202e</TileColor>
+        </tile>
+    </msapplication>
+</browserconfig>

tubearchivist/static/favicon/safari-pinned-tab.svg

@@ -0,0 +1,100 @@
+<?xml version="1.0" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 20010904//EN"
+ "http://www.w3.org/TR/2001/REC-SVG-20010904/DTD/svg10.dtd">
+<svg version="1.0" xmlns="http://www.w3.org/2000/svg"
+ width="1000.000000pt" height="1000.000000pt" viewBox="0 0 1000.000000 1000.000000"
+ preserveAspectRatio="xMidYMid meet">
+<metadata>
+Created by potrace 1.14, written by Peter Selinger 2001-2017
+</metadata>
+<g transform="translate(0.000000,1000.000000) scale(0.100000,-0.100000)"
+fill="#000000" stroke="none">
+<path d="M4521 9370 c-4 -3 -7 -31 -6 -63 1 -32 -3 -61 -9 -66 -6 -5 -36 -12
+-66 -16 -55 -7 -68 -9 -125 -19 -16 -3 -48 -8 -70 -12 -161 -24 -504 -118
+-709 -193 -208 -77 -507 -215 -636 -294 -19 -12 -47 -29 -62 -37 -14 -8 -41
+-24 -60 -34 -84 -48 -288 -189 -423 -293 -309 -239 -654 -606 -886 -943 -53
+-78 -169 -258 -169 -264 0 -3 -21 -40 -47 -83 -97 -164 -313 -646 -294 -658
+14 -8 670 -195 686 -195 8 0 15 9 15 19 0 11 5 23 10 26 6 3 10 13 10 21 0 8
+14 44 30 79 17 35 30 67 30 70 0 3 21 48 47 100 l47 95 148 0 c81 0 276 1 433
+1 157 0 310 -1 340 -1 l55 0 0 -615 0 -615 415 1 415 0 1 142 c0 78 0 355 1
+615 l1 472 46 0 c25 0 322 0 659 0 l612 1 0 32 c1 44 -1 538 -2 660 0 63 -5
+97 -12 98 -6 1 -581 1 -1278 0 -698 0 -1268 2 -1268 5 0 7 143 147 241 235 94
+85 232 193 329 260 36 24 67 47 68 52 2 4 7 7 12 7 4 0 52 28 106 62 126 79
+371 202 510 257 60 23 118 46 129 51 131 56 685 193 713 176 4 -2 7 -34 6 -70
+l0 -66 162 0 162 0 0 503 0 502 -155 0 c-85 0 -158 -2 -162 -5z"/>
+<path d="M5210 8905 l0 -365 48 -1 c26 0 54 -2 62 -4 8 -1 45 -5 82 -9 72 -6
+198 -23 232 -31 12 -3 32 -7 46 -9 284 -48 643 -169 935 -314 527 -262 984
+-651 1324 -1127 39 -55 75 -105 81 -112 5 -7 10 -17 10 -23 0 -5 4 -10 9 -10
+5 0 14 -12 20 -27 6 -16 24 -48 40 -73 31 -47 61 -101 117 -208 30 -57 32 -65
+16 -69 -75 -20 -117 -40 -113 -53 9 -35 83 -285 85 -287 1 -2 18 2 37 8 19 5
+41 12 49 14 49 12 856 241 876 249 11 4 10 14 -2 57 -9 29 -17 60 -19 68 -7
+32 -53 183 -57 186 -2 2 -22 -2 -43 -9 -93 -31 -88 -33 -126 46 -232 488 -557
+933 -950 1303 -285 269 -647 520 -1014 705 -533 268 -1067 412 -1682 455 l-63
+4 0 -364z"/>
+<path d="M6122 7336 c-4 -6 -14 -31 -23 -56 -17 -47 -238 -630 -288 -760 -16
+-41 -61 -160 -100 -265 -40 -104 -76 -199 -80 -210 -10 -23 -54 -138 -104
+-275 -20 -52 -41 -108 -48 -125 -7 -16 -13 -32 -13 -35 -1 -3 -7 -18 -13 -35
+-6 -16 -55 -145 -108 -285 -53 -140 -102 -271 -110 -290 -8 -19 -48 -125 -89
+-235 -41 -110 -90 -240 -109 -290 -19 -49 -76 -200 -127 -335 -133 -356 -151
+-402 -166 -437 -8 -17 -14 -34 -14 -37 0 -3 -15 -45 -34 -94 -19 -48 -61 -160
+-95 -248 -33 -89 -65 -174 -72 -190 -22 -55 -28 -70 -98 -258 -39 -103 -71
+-192 -71 -197 0 -5 182 -8 441 -7 l441 3 114 320 c63 176 126 352 140 390 14
+39 46 129 70 200 25 72 50 139 55 150 5 11 22 58 38 105 29 84 89 254 171 480
+23 63 47 133 54 155 12 36 70 199 131 365 13 36 60 169 105 295 44 127 85 239
+90 250 4 11 62 173 129 360 197 557 186 530 195 504 3 -8 18 -52 35 -99 16
+-47 54 -155 84 -240 30 -85 96 -272 146 -415 51 -143 96 -269 100 -280 5 -11
+27 -72 49 -135 120 -343 143 -407 152 -422 6 -10 10 -24 10 -33 0 -8 6 -29 14
+-47 8 -18 44 -116 80 -218 182 -521 327 -927 336 -942 6 -10 10 -23 10 -29 0
+-9 134 -393 176 -502 8 -21 28 -78 45 -128 l31 -90 -49 -56 c-26 -31 -78 -88
+-115 -127 -64 -68 -68 -70 -81 -51 -8 11 -28 37 -44 57 l-30 37 -124 -91 c-68
+-50 -123 -96 -123 -104 1 -9 281 -404 327 -459 4 -5 62 -84 128 -175 65 -91
+124 -166 129 -168 8 -3 73 39 90 58 3 3 42 32 88 64 l82 58 -40 56 -39 56 94
+96 c558 566 945 1258 1130 2025 18 77 38 167 44 200 6 33 13 69 15 80 6 33 19
+119 21 140 2 11 6 43 9 70 35 253 40 715 11 930 -2 17 -9 75 -15 129 -7 55
+-16 109 -20 122 -5 12 -7 24 -5 26 11 11 -85 428 -98 428 -11 0 -680 -190
+-685 -195 -2 -1 8 -47 21 -101 14 -55 28 -115 31 -134 3 -19 8 -46 11 -60 10
+-53 19 -109 29 -195 4 -33 9 -70 11 -82 8 -53 18 -259 18 -369 -1 -400 -74
+-812 -213 -1194 -23 -63 -44 -123 -47 -132 -9 -29 -20 -21 -35 25 -22 62 -198
+529 -208 552 -5 11 -69 178 -141 370 -72 193 -187 499 -256 680 -68 182 -164
+436 -213 565 -48 129 -93 246 -98 260 -6 14 -27 70 -47 125 -19 55 -40 109
+-45 120 -8 18 -191 503 -224 595 -15 42 -95 253 -139 366 l-31 80 -398 0
+c-270 0 -401 -3 -406 -10z"/>
+<path d="M720 6002 c-47 -160 -68 -237 -64 -243 2 -4 28 -13 56 -19 29 -7 56
+-14 59 -16 4 -2 3 -23 -1 -46 -5 -24 -11 -61 -14 -83 -4 -23 -8 -50 -10 -60
+-2 -11 -7 -47 -10 -80 -4 -33 -9 -73 -11 -90 -31 -231 -26 -701 11 -955 3 -25
+8 -61 10 -80 2 -19 5 -44 8 -55 3 -11 8 -42 12 -70 3 -27 8 -53 10 -56 2 -4 6
+-24 10 -45 20 -134 154 -594 209 -719 7 -16 27 -66 45 -110 59 -147 189 -400
+297 -579 89 -147 270 -400 359 -502 11 -12 39 -46 64 -75 74 -89 358 -369 465
+-459 55 -46 110 -92 121 -102 22 -19 22 -19 107 99 48 65 90 123 94 128 35 42
+243 336 242 342 0 5 -38 40 -85 78 -313 258 -611 618 -818 988 -63 114 -176
+341 -176 355 0 5 -9 26 -19 48 -54 113 -171 502 -200 669 -6 33 -13 69 -15 80
+-8 43 -21 134 -31 210 -19 156 -24 420 -12 605 8 119 13 166 33 295 5 28 7 53
+6 58 -2 8 24 3 99 -18 l45 -13 23 82 c12 44 29 106 38 136 30 104 34 95 -54
+118 -43 11 -109 29 -148 40 -38 11 -104 30 -145 41 -41 12 -79 23 -85 25 -5 1
+-28 8 -50 14 -22 7 -112 33 -200 58 -88 25 -181 52 -207 59 l-48 14 -20 -67z"/>
+<path d="M2693 5210 c-48 -11 -119 -56 -148 -95 -112 -146 -57 -352 115 -428
+43 -19 68 -20 560 -21 449 -1 521 1 563 15 65 22 114 64 151 129 27 48 31 64
+31 130 0 116 -49 200 -149 253 -41 22 -44 22 -566 23 -289 1 -539 -2 -557 -6z"/>
+<path d="M6425 5138 c-50 -19 -102 -55 -128 -90 -56 -73 -55 -54 -55 -901 -1
+-430 -1 -793 -2 -807 0 -21 -26 1 -169 148 -93 94 -173 172 -178 172 -4 0 -58
+-50 -118 -110 l-110 -110 25 -24 c14 -12 202 -204 419 -424 217 -221 397 -402
+401 -402 3 0 199 192 435 426 l429 425 -114 114 -115 114 -177 -178 -178 -178
+-1 381 c-2 1238 -2 1258 -24 1301 -26 52 -71 98 -126 126 -51 26 -166 35 -214
+17z"/>
+<path d="M2814 4501 c-2 -2 -4 -415 -4 -918 l0 -913 415 0 415 0 0 28 c4 223
+0 1795 -4 1799 -6 7 -816 10 -822 4z"/>
+<path d="M3181 2038 c-8 -13 -44 -63 -81 -113 -36 -49 -101 -137 -143 -195
+-42 -58 -80 -109 -84 -115 -16 -20 -88 -119 -103 -141 -8 -13 -48 -68 -89
+-124 -40 -55 -72 -103 -70 -105 2 -2 60 -44 129 -94 l125 -91 25 32 c62 79 35
+81 231 -15 382 -187 717 -298 1149 -382 8 -1 36 -5 61 -9 25 -4 57 -9 70 -11
+45 -7 183 -24 264 -32 105 -10 629 -10 725 0 116 13 131 14 195 22 58 7 70 9
+125 19 14 3 42 8 63 11 48 7 174 32 252 51 33 8 67 16 76 19 145 33 419 126
+599 202 122 52 413 195 418 206 2 4 8 7 12 7 11 0 193 108 273 162 53 36 57
+41 45 59 -107 155 -406 558 -415 558 -7 1 -44 -19 -82 -44 -236 -153 -537
+-295 -814 -386 -97 -31 -291 -85 -342 -94 -16 -2 -41 -7 -55 -10 -39 -9 -240
+-43 -288 -49 -212 -28 -609 -29 -822 -2 -14 2 -52 7 -85 11 -33 4 -69 9 -80
+12 -11 2 -33 6 -50 9 -211 34 -520 123 -735 211 -86 36 -291 130 -299 138 -2
+3 15 30 38 60 24 31 41 58 39 60 -11 11 -243 178 -252 181 -6 2 -17 -6 -25
+-18z"/>
+</g>
+</svg>

tubearchivist/static/favicon/site.webmanifest

@@ -0,0 +1,19 @@
+{
+    "name": "TubeArchivist",
+    "short_name": "TubeArchivist",
+    "icons": [
+        {
+            "src": "/static/favicon/android-chrome-192x192.png",
+            "sizes": "192x192",
+            "type": "image/png"
+        },
+        {
+            "src": "/static/favicon/android-chrome-512x512.png",
+            "sizes": "512x512",
+            "type": "image/png"
+        }
+    ],
+    "theme_color": "#01202e",
+    "background_color": "#01202e",
+    "display": "standalone"
+}

tubearchivist/static/img/icon-exit.svg

@@ -0,0 +1,67 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   xmlns:dc="http://purl.org/dc/elements/1.1/"
+   xmlns:cc="http://creativecommons.org/ns#"
+   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+   xmlns:svg="http://www.w3.org/2000/svg"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   width="500"
+   height="500"
+   viewBox="0 0 132.29197 132.29167"
+   version="1.1"
+   id="svg1303"
+   inkscape:version="0.92.4 (5da689c313, 2019-01-14)"
+   sodipodi:docname="Icons_exit.svg">
+  <defs
+     id="defs1297" />
+  <sodipodi:namedview
+     id="base"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageopacity="0.0"
+     inkscape:pageshadow="2"
+     inkscape:zoom="0.66442107"
+     inkscape:cx="161.45413"
+     inkscape:cy="207.61753"
+     inkscape:document-units="mm"
+     inkscape:current-layer="layer1"
+     showgrid="false"
+     units="px"
+     inkscape:window-width="1169"
+     inkscape:window-height="893"
+     inkscape:window-x="729"
+     inkscape:window-y="13"
+     inkscape:window-maximized="0" />
+  <metadata
+     id="metadata1300">
+    <rdf:RDF>
+      <cc:Work
+         rdf:about="">
+        <dc:format>image/svg+xml</dc:format>
+        <dc:type
+           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+        <dc:title></dc:title>
+      </cc:Work>
+    </rdf:RDF>
+  </metadata>
+  <g
+     inkscape:label="Ebene 1"
+     inkscape:groupmode="layer"
+     id="layer1"
+     transform="translate(0,-164.70764)">
+    <g
+       id="g855"
+       transform="matrix(1.9016362,0,0,1.9016362,-197.93838,-58.9418)">
+      <path
+         inkscape:connector-curvature="0"
+         id="rect1208"
+         style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;paint-order:markers fill stroke"
+         d="m 124.57603,151.92962 c -0.0433,2.30016 2.0751,4.19245 4.75007,4.24278 l 30.26401,0.43007 -6.00195,5.78023 3.43246,3.56154 10.2778,-9.9006 0.002,0.002 3.5183,-3.3908 -3.42991,-3.564 -9.8737,-10.24989 -3.51834,3.39083 5.84388,6.06803 -30.35875,-0.43185 c -2.67494,-0.0503 -4.86301,1.76094 -4.90629,4.06112 z m -17.65039,-32.01644 v 64.95883 h 7.44347 v -58.27707 h 26.3896 v 18.5229 h 7.44296 v -25.20466 z m 33.83307,39.75416 v 25.20467 h 7.44296 v -25.20467 z" />
+    </g>
+  </g>
+</svg>

tubearchivist/static/img/icon-help.svg

@@ -1,68 +0,0 @@
-<?xml version="1.0" encoding="UTF-8" standalone="no"?>
-<!-- Created with Inkscape (http://www.inkscape.org/) -->
-
-<svg
-   xmlns:dc="http://purl.org/dc/elements/1.1/"
-   xmlns:cc="http://creativecommons.org/ns#"
-   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
-   xmlns:svg="http://www.w3.org/2000/svg"
-   xmlns="http://www.w3.org/2000/svg"
-   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
-   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
-   width="500"
-   height="500"
-   viewBox="0 0 132.29197 132.29167"
-   version="1.1"
-   id="svg1303"
-   inkscape:version="0.92.4 (5da689c313, 2019-01-14)"
-   sodipodi:docname="Icons_Help.svg">
-  <defs
-     id="defs1297" />
-  <sodipodi:namedview
-     id="base"
-     pagecolor="#ffffff"
-     bordercolor="#666666"
-     borderopacity="1.0"
-     inkscape:pageopacity="0.0"
-     inkscape:pageshadow="2"
-     inkscape:zoom="1.0105705"
-     inkscape:cx="58.993585"
-     inkscape:cy="143.32456"
-     inkscape:document-units="mm"
-     inkscape:current-layer="layer1"
-     showgrid="false"
-     units="px"
-     inkscape:window-width="1920"
-     inkscape:window-height="1017"
-     inkscape:window-x="-8"
-     inkscape:window-y="-8"
-     inkscape:window-maximized="1" />
-  <metadata
-     id="metadata1300">
-    <rdf:RDF>
-      <cc:Work
-         rdf:about="">
-        <dc:format>image/svg+xml</dc:format>
-        <dc:type
-           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
-        <dc:title />
-      </cc:Work>
-    </rdf:RDF>
-  </metadata>
-  <g
-     inkscape:label="Ebene 1"
-     inkscape:groupmode="layer"
-     id="layer1"
-     transform="translate(0,-164.70764)">
-    <g
-       transform="matrix(6.2585232,0,0,6.2585232,-555.66096,-762.59652)"
-       id="g1069"
-       style="fill:#000000">
-      <path
-         style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;paint-order:markers fill stroke"
-         d="M 99.35362,148.32944 A 10.406119,10.406119 0 0 0 88.947542,158.73553 10.406119,10.406119 0 0 0 99.35362,169.14162 10.406119,10.406119 0 0 0 109.7597,158.73553 10.406119,10.406119 0 0 0 99.35362,148.32944 Z m 0.022,2.79535 c 0.58489,0 1.12588,0.073 1.62304,0.2192 0.51178,0.14622 0.95018,0.37308 1.31574,0.68016 0.38018,0.30706 0.67262,0.70157 0.87734,1.18411 0.21933,0.46791 0.32933,1.02372 0.32933,1.6671 0,0.4533 -0.0809,0.86261 -0.24176,1.22816 -0.16084,0.35094 -0.3655,0.67987 -0.61408,0.98694 -0.23396,0.30706 -0.48991,0.60692 -0.76774,0.89936 -0.26321,0.27783 -0.51176,0.57027 -0.74571,0.87734 -0.21932,0.30707 -0.39436,0.63601 -0.52597,0.98694 -0.1316,0.33631 -0.17549,0.72394 -0.13163,1.16261 h -2.829714 c -0.07311,-0.49715 -0.0583,-0.95035 0.04406,-1.35979 0.116978,-0.40941 0.277739,-0.78965 0.482453,-1.14059 0.204712,-0.35093 0.431586,-0.67246 0.680164,-0.96491 0.263203,-0.29244 0.511747,-0.57062 0.745707,-0.83381 0.23395,-0.27783 0.4238,-0.54119 0.57002,-0.78977 0.16086,-0.26321 0.24123,-0.53344 0.24123,-0.81125 0,-0.4533 -0.13908,-0.79704 -0.41691,-1.03099 -0.2632,-0.23397 -0.61381,-0.35083 -1.052474,-0.35083 -0.409426,0 -0.774845,0.0952 -1.096536,0.28528 -0.307068,0.19009 -0.606919,0.43123 -0.899364,0.72368 l -1.7767,-1.62304 c 0.511782,-0.59952 1.118345,-1.08235 1.820217,-1.4479 0.716494,-0.36557 1.506571,-0.548 2.369287,-0.548 z m -0.307306,11.16414 c 0.555636,0 1.016256,0.19727 1.381816,0.59205 0.38018,0.38019 0.57003,0.86249 0.57003,1.44737 0,0.57028 -0.18985,1.05311 -0.57003,1.4479 -0.36556,0.38018 -0.82618,0.57002 -1.381816,0.57002 -0.555651,0 -1.016262,-0.18984 -1.38182,-0.57002 -0.36556,-0.39479 -0.548536,-0.87762 -0.548536,-1.4479 0,-0.58488 0.182976,-1.06718 0.548536,-1.44737 0.365558,-0.39478 0.826169,-0.59205 1.38182,-0.59205 z"
-         id="path1038"
-         inkscape:connector-curvature="0" />
-    </g>
-  </g>
-</svg>

tubearchivist/static/img/icon-sort.svg

@@ -0,0 +1,67 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   xmlns:dc="http://purl.org/dc/elements/1.1/"
+   xmlns:cc="http://creativecommons.org/ns#"
+   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+   xmlns:svg="http://www.w3.org/2000/svg"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   width="500"
+   height="500"
+   viewBox="0 0 132.29197 132.29167"
+   version="1.1"
+   id="svg1303"
+   inkscape:version="0.92.4 (5da689c313, 2019-01-14)"
+   sodipodi:docname="Icons_sort.svg">
+  <defs
+     id="defs1297" />
+  <sodipodi:namedview
+     id="base"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageopacity="0.0"
+     inkscape:pageshadow="2"
+     inkscape:zoom="0.6482696"
+     inkscape:cx="214.8721"
+     inkscape:cy="136.02434"
+     inkscape:document-units="mm"
+     inkscape:current-layer="g855"
+     showgrid="false"
+     units="px"
+     inkscape:window-width="957"
+     inkscape:window-height="893"
+     inkscape:window-x="941"
+     inkscape:window-y="13"
+     inkscape:window-maximized="0" />
+  <metadata
+     id="metadata1300">
+    <rdf:RDF>
+      <cc:Work
+         rdf:about="">
+        <dc:format>image/svg+xml</dc:format>
+        <dc:type
+           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+        <dc:title></dc:title>
+      </cc:Work>
+    </rdf:RDF>
+  </metadata>
+  <g
+     inkscape:label="Ebene 1"
+     inkscape:groupmode="layer"
+     id="layer1"
+     transform="translate(0,-164.70764)">
+    <g
+       id="g855"
+       transform="matrix(1.9016362,0,0,1.9016362,-197.93838,-58.9418)">
+      <path
+         style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:2.55118108;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;paint-order:markers fill stroke"
+         d="M 341.125 82.275391 L 315.50977 106.92773 L 241.84375 177.89258 L 266.21289 203.17969 L 309.82617 161.17969 L 306.72266 379.375 C 306.36114 398.60059 319.37807 414.32761 335.91016 414.63867 L 335.91406 414.63477 C 352.44586 414.94597 366.04647 399.71987 366.4082 380.49414 L 369.5 162.97852 L 411.04297 206.11719 L 436.64062 181.44727 L 365.48242 107.57812 L 365.49609 107.5625 L 341.125 82.275391 z M 175.93359 82.277344 L 175.92969 82.28125 C 159.39782 81.970041 145.79728 97.196144 145.43555 116.42188 L 142.34375 333.9375 L 100.80078 290.79883 L 75.203125 315.46875 L 146.36133 389.33789 L 146.3457 389.35156 L 170.7168 414.63867 L 196.33203 389.98633 L 270 319.02344 L 245.63086 293.73633 L 202.01758 335.73633 L 205.12109 117.54102 C 205.48261 98.315428 192.46568 82.588409 175.93359 82.277344 z "
+         transform="matrix(0.13913489,0,0,0.13913489,104.08846,117.60887)"
+         id="path814" />
+    </g>
+  </g>
+</svg>

tubearchivist/static/progress.js

@@ -1,89 +1,90 @@
 /**
- * Handle progress bar on /download
+ * Handle multi channel notifications
  * 
  */
 
+checkMessages()
 
-setTimeout(function(){
-    checkMessage();
-}, 1000);
-
-// initial check for messages
-function checkMessage() {
-    var req = new XMLHttpRequest();
-    req.responseType = 'json';
-    req.open('GET', '/downloads/progress', true);
-    req.onload = function() {
-        var dlProgress = req.response;
-        // var dlStatus = dlProgress['status'];
-        if (dlProgress['status']) {
-            buildDownloadMessage(dlProgress);
-            handleInterval();
-            // if (dlStatus == 'downloading') {
-            //     buildDownloadIcons();
-            // };
-        };
-    };
-    req.send();
+// page map to notification status
+const messageTypes = {
+    "download": ["message:download", "message:add", "message:rescan"],
+    "channel": ["message:subchannel"],
+    "channel_id": ["message:playlistscan"],
+    "playlist": ["message:subplaylist"],
+    "setting": ["message:setting"]
 }
 
-// set interval until no more messages
-function handleInterval() {
-    var watchDownload = setInterval(function() {
-        var req = new XMLHttpRequest();
-        req.responseType = 'json';
-        req.open('GET', '/downloads/progress/', true);
-        req.onload = function() {
-            var dlProgress = req.response;
-            if (dlProgress['status']) {
-                buildDownloadMessage(dlProgress);
-            } else {
-                clearInterval(watchDownload);
-                location.reload();
-            };
+// start to look for messages
+function checkMessages() {
+    var notifications = document.getElementById("notifications");
+    if (notifications) {
+        var dataOrigin = notifications.getAttribute("data");
+        getMessages(dataOrigin);
+    }
+}
+
+// get messages for page on timer
+function getMessages(dataOrigin) {
+    fetch('/progress/').then(response => {
+        return response.json();
+    }).then(responseData => {
+        var messages = buildMessage(responseData, dataOrigin);
+        if (messages.length > 0) {
+            // restart itself
+            setTimeout(function() {
+                getMessages(dataOrigin);
+            }, 3000);
         };
-        req.send();
-    }, 3000);
-};
+    });
+}
 
-// remove and set message
-function buildDownloadMessage(dlProgress) {
-    var box = document.getElementById('downloadMessage');
-    box.innerHTML = '';
-    var dlStatus = dlProgress['status'];
-    var dlTitle = dlProgress['title'];
-    var dlMessage = dlProgress['message'];
-    var dlLevel = dlProgress['level'];
-    // animate
-    if (dlStatus === 'rescan') {
-        animate('rescan-icon', 'rotate-img');
-    } else if (dlStatus === 'downloading') {
-        animate('download-icon', 'bounce-img');
+// make div for all messages, return relevant
+function buildMessage(responseData, dataOrigin) {
+    // filter relevan messages
+    var allMessages = responseData["messages"];
+    var messages = allMessages.filter(function(value) {
+        return messageTypes[dataOrigin].includes(value["status"])
+    }, dataOrigin);
+    // build divs
+    var notificationDiv = document.getElementById("notifications");
+    var nots = notificationDiv.childElementCount;
+    notificationDiv.innerHTML = "";
+    for (let i = 0; i < messages.length; i++) {
+        var messageData = messages[i];
+        var messageStatus = messageData["status"];
+        var messageBox = document.createElement("div");
+        var title = document.createElement("h3");
+        title.innerHTML = messageData["title"];
+        var message = document.createElement("p");
+        message.innerHTML = messageData["message"];
+        messageBox.appendChild(title);
+        messageBox.appendChild(message);
+        messageBox.classList.add(messageData["level"], "notification");
+        notificationDiv.appendChild(messageBox);
+        if (messageStatus === "message:download") {
+            checkDownloadIcons();
+        };
     };
-    // div box
-    var box = document.getElementById('downloadMessage');
-    var message = document.createElement('div');
-    message.classList.add('download-progress');
-    message.classList.add(dlLevel);
-    message.id = 'progress';
-    message.setAttribute('data', dlStatus);
-    var title = document.createElement('h3');
-    title.innerHTML = dlTitle;
-    var messageText = document.createElement('p');
-    messageText.innerHTML = dlMessage;
-    message.appendChild(title);
-    message.appendChild(messageText);
-    box.appendChild(message);
-    if (dlStatus == 'downloading' && dlLevel != 'error') {
-        box.appendChild(buildDownloadIcons());
+    // reload page when no more notifications
+    if (nots > 0 && messages.length === 0) {
+        location.reload();
     };
-};
+    return messages
+}
 
+// check if download icons are needed
+function checkDownloadIcons() {
+    var iconBox = document.getElementById("downloadControl");
+    if (iconBox.childElementCount === 0) {
+        var downloadIcons = buildDownloadIcons();
+        iconBox.appendChild(downloadIcons);
+    };
+}
 
 // add dl control icons
 function buildDownloadIcons() {
-    var iconBox = document.createElement('div');
-    iconBox.classList = 'dl-control-icons';
+    var downloadIcons = document.createElement('div');
+    downloadIcons.classList = 'dl-control-icons';
     // stop icon
     var stopIcon = document.createElement('img');
     stopIcon.setAttribute('id', "stop-icon");
@@ -99,8 +100,7 @@ function buildDownloadIcons() {
     killIcon.setAttribute('alt', "kill icon");
     killIcon.setAttribute('onclick', 'killQueue()');
     // stich together
-    iconBox.appendChild(stopIcon);
-    iconBox.appendChild(killIcon);
-    
-    return iconBox
+    downloadIcons.appendChild(stopIcon);
+    downloadIcons.appendChild(killIcon);
+    return downloadIcons
 }

tubearchivist/static/script.js

@@ -21,6 +21,17 @@ function isWatched(youtube_id) {
     document.getElementById(youtube_id).replaceWith(seenIcon);
 }
 
+function isWatchedButton(button) {
+    youtube_id = button.getAttribute("data-id");
+    var payload = JSON.stringify({'watched': youtube_id});
+    button.remove();
+    sendPost(payload);
+    setTimeout(function(){
+        location.reload();
+        return false;
+    }, 1000);
+}
+
 function isUnwatched(youtube_id) {
     var payload = JSON.stringify({'un_watched': youtube_id});
     sendPost(payload);
@@ -34,10 +45,20 @@ function isUnwatched(youtube_id) {
     document.getElementById(youtube_id).replaceWith(unseenIcon);
 }
 
-function unsubscribe(channel_id) {
-    var payload = JSON.stringify({'unsubscribe': channel_id});
+function unsubscribe(id_unsub) {
+    var payload = JSON.stringify({'unsubscribe': id_unsub});
     sendPost(payload);
-    document.getElementById(channel_id).remove();
+    var message = document.createElement('span');
+    message.innerText = "You are unsubscribed.";
+    document.getElementById(id_unsub).replaceWith(message);
+}
+
+function subscribe(id_sub) {
+    var payload = JSON.stringify({'subscribe': id_sub});
+    sendPost(payload);
+    var message = document.createElement('span');
+    message.innerText = "You are subscribed.";
+    document.getElementById(id_sub).replaceWith(message);
 }
 
 function changeView(image) {
@@ -72,7 +93,7 @@ function rescanPending() {
     animate('rescan-icon', 'rotate-img');
     sendPost(payload);
     setTimeout(function(){
-        handleInterval();
+        checkMessages();
     }, 500);
 }
 
@@ -81,7 +102,7 @@ function dlPending() {
     animate('download-icon', 'bounce-img');
     sendPost(payload);
     setTimeout(function(){
-        handleInterval();
+        checkMessages();
     }, 500);
 }
 
@@ -98,7 +119,7 @@ function downloadNow(button) {
     sendPost(payload);
     document.getElementById(youtube_id).remove();
     setTimeout(function(){
-        handleInterval();
+        checkMessages();
     }, 500);
 }
 
@@ -115,10 +136,21 @@ function addSingle(button) {
     sendPost(payload);
     document.getElementById("dl-" + youtube_id).remove();
     setTimeout(function(){
-        handleInterval();
+        checkMessages();
     }, 500);
 }
 
+function deleteQueue(button) {
+    var to_delete = button.getAttribute('data-id');
+    var payload = JSON.stringify({'deleteQueue': to_delete});
+    console.log(payload);
+    sendPost(payload);
+    setTimeout(function(){
+        location.reload();
+        return false;
+    }, 1000);
+}
+
 function stopQueue() {
     var payload = JSON.stringify({'queue': 'stop'});
     sendPost(payload);
@@ -143,6 +175,17 @@ function manualImport() {
     toReplace.appendChild(message);
 }
 
+function reEmbed() {
+    var payload = JSON.stringify({'re-embed': true});
+    sendPost(payload);
+    // clear button
+    var message = document.createElement('p');
+    message.innerText = 'processing thumbnails';
+    var toReplace = document.getElementById('re-embed');
+    toReplace.innerHTML = '';
+    toReplace.appendChild(message);
+}
+
 function dbBackup() {
     var payload = JSON.stringify({'db-backup': true});
     sendPost(payload)
@@ -154,13 +197,14 @@ function dbBackup() {
     toReplace.appendChild(message);
 }
 
-function dbRestore() {
-    var payload = JSON.stringify({'db-restore': true});
+function dbRestore(button) {
+    var fileName = button.getAttribute("data-id");
+    var payload = JSON.stringify({'db-restore': fileName});
     sendPost(payload);
-    // clear button
+    // clear backup row
     var message = document.createElement('p');
     message.innerText = 'restoring from backup';
-    var toReplace = document.getElementById('db-restore');
+    var toReplace = document.getElementById(fileName);
     toReplace.innerHTML = '';
     toReplace.appendChild(message);
 }
@@ -176,6 +220,19 @@ function fsRescan() {
     toReplace.appendChild(message);
 }
 
+function findPlaylists(button) {
+    var channel_id = button.getAttribute("data-id");
+    var payload = JSON.stringify({'find-playlists': channel_id});
+    sendPost(payload);
+    // clear button
+    var message = document.createElement('p');
+    message.innerText = 'Scraping for playlists in progress';
+    document.getElementById("find-playlists-button").replaceWith(message);
+    setTimeout(function(){
+        checkMessages();
+    }, 500);
+}
+
 // delete from file system
 function deleteConfirm() {
     to_show = document.getElementById("delete-button");
@@ -205,6 +262,23 @@ function deleteChannel(button) {
     }, 1000);
 }
 
+function deletePlaylist(button) {
+    var playlist_id = button.getAttribute("data-id");
+    var playlist_action = button.getAttribute("data-action");
+    var payload = JSON.stringify({
+        "delete-playlist": {
+            "playlist-id": playlist_id,
+            "playlist-action": playlist_action
+        }
+    });
+    console.log(payload);
+    sendPost(payload);
+    setTimeout(function(){
+        window.location.replace("/playlist/");
+        return false;
+    }, 1000);
+}
+
 function cancelDelete() {
     document.getElementById("delete-button").style.display = 'none';
     document.getElementById("delete-item").style.display = 'block';
@@ -216,13 +290,16 @@ function createPlayer(button) {
     var mediaThumb = button.getAttribute('data-thumb');
     var mediaTitle = button.getAttribute('data-title');
     var mediaChannel = button.getAttribute('data-channel');
+    var mediaChannelId = button.getAttribute('data-channel-id');
     var dataId = button.getAttribute('data-id');
     // get watched status
     var playedStatus = document.createDocumentFragment();
     playedStatus.appendChild(document.getElementById(dataId));
     // create player
     removePlayer();
-    var playerElement = document.getElementById('player');
+    var playerElement = document.createElement('div');
+    playerElement.classList.add("video-player");
+    // var playerElement = document.getElementById('player');
     playerElement.setAttribute('data-id', dataId);
     // playerElement.innerHTML = '';
     var videoPlayer = document.createElement('video');
@@ -230,11 +307,12 @@ function createPlayer(button) {
     videoPlayer.setAttribute('controls', true);
     videoPlayer.setAttribute('autoplay', true);
     videoPlayer.setAttribute('width', '100%');
+    videoPlayer.setAttribute('playsinline', true);
     videoPlayer.setAttribute('poster', mediaThumb);
     playerElement.appendChild(videoPlayer);
     // title bar
     var titleBar = document.createElement('div');
-    titleBar.className = 'player-title';
+    titleBar.classList.add('player-title', 'boxed-content');
     // close
     var closeButton = document.createElement('img');
     closeButton.className = 'close-button';
@@ -246,27 +324,35 @@ function createPlayer(button) {
     titleBar.appendChild(closeButton);
     // played
     titleBar.appendChild(playedStatus);
-    playerElement.appendChild(titleBar);
-    // title
-    var videoTitle = document.createElement('p');
+    // channel title
+    var channelTitleLink = document.createElement('a');
+    channelTitleLink.setAttribute('href', '/channel/' + mediaChannelId + '/');
+    var channelTitle = document.createElement('h3');
+    channelTitle.innerText = mediaChannel;
+    channelTitleLink.appendChild(channelTitle);
+    titleBar.appendChild(channelTitleLink);
+    // video title
+    var videoTitleLink = document.createElement('a');
+    videoTitleLink.setAttribute('href', '/video/' + dataId + '/');
+    var videoTitle = document.createElement('h2');
     videoTitle.innerText = mediaTitle;
-    titleBar.appendChild(videoTitle);
-    var videoChannel = document.createElement('p');
-    videoChannel.innerText = mediaChannel
-    titleBar.appendChild(videoChannel);
-    // button
-    var videoButton = document.createElement('button');
-    videoButton.innerText = 'Details';
-    videoButton.setAttribute('onclick', "window.location.href='/video/" + dataId + "/';");
-    titleBar.appendChild(videoButton);
+    videoTitleLink.appendChild(videoTitle);
+    titleBar.appendChild(videoTitleLink);
+    // add titlebar
+    playerElement.appendChild(titleBar);
+    // add whole
+    document.getElementById("player").appendChild(playerElement);
 }
 
 function removePlayer() {
     var playerElement = document.getElementById('player');
     if (playerElement.hasChildNodes()) {
-        var youtubeId = playerElement.getAttribute('data-id');
+        var youtubeId = playerElement.childNodes[0].getAttribute("data-id");
         var playedStatus = document.createDocumentFragment();
-        playedStatus.appendChild(document.getElementById(youtubeId));
+        var playedBox = document.getElementById(youtubeId);
+        if (playedBox) {
+            playedStatus.appendChild(playedBox);
+        };
         playerElement.innerHTML = '';
         // append played status
         var videoInfo = document.getElementById('video-info-' + youtubeId);
@@ -274,54 +360,182 @@ function removePlayer() {
     };
 }
 
-// searching channels
-function searchChannels(query) {
-    var searchResultBox = document.getElementById('resultBox');
-    searchResultBox.innerHTML = '';
+
+// multi search form
+function searchMulti(query) {
     if (query.length > 1) {
-        var payload = JSON.stringify({'channel-search': query})
-        sendSearchAsYouType(payload);
-    };
-}
-
-
-function populateChannelResults(allResults) {
-    var searchResultBox = document.getElementById('resultBox');
-    for (let i = 0; i < allResults.length; i++) {
-        var singleResult = allResults[i];
-        var source = singleResult['source'];
-        var channelName = source['channel_name'];
-        var channelId = source['channel_id'];
-        var optionElement = document.createElement('option');
-        optionElement.value = channelName;
-        optionElement.setAttribute('data', channelId);
-        searchResultBox.appendChild(optionElement);
-    };
-}
-
-function channelRedirect(){
-    var response = document.getElementById('resultBox');
-    var firstChild = response.firstChild
-    if (firstChild) {
-        var redirectId = firstChild.getAttribute('data');
-        location = '/channel/' + redirectId;
-    };
-    return false;
-}
-
-
-function sendSearchAsYouType(payload) {
-    var http = new XMLHttpRequest();
-    http.onreadystatechange = function() {
-        if (http.readyState === 4) {
-            allResults = JSON.parse(http.response)['results'];
-            populateChannelResults(allResults);
+        var payload = JSON.stringify({'multi_search': query})
+        var http = new XMLHttpRequest();
+        http.onreadystatechange = function() {
+            if (http.readyState === 4) {
+                allResults = JSON.parse(http.response)['results'];
+                populateMultiSearchResults(allResults);
+            };
         };
+        http.open("POST", "/process/", true);
+        http.setRequestHeader("X-CSRFToken", getCookie("csrftoken"));
+        http.setRequestHeader("Content-type", "application/json");
+        http.send(payload);
     };
-    http.open("POST", "/process/", true);
-    http.setRequestHeader("X-CSRFToken", getCookie("csrftoken"));
-    http.setRequestHeader("Content-type", "application/json");
-    http.send(payload);
+}
+
+function getViewDefaults(view) {
+    var defaultView = document.getElementById("id_" + view).value;
+    return defaultView
+}
+
+function populateMultiSearchResults(allResults) {
+    // videos
+    var defaultVideo = getViewDefaults("home");
+    var allVideos = allResults["video_results"];
+    var videoBox = document.getElementById("video-results");
+    videoBox.innerHTML = "";
+    for (let index = 0; index < allVideos.length; index++) {
+        const video = allVideos[index]["source"];
+        const videoDiv = createVideo(video, defaultVideo);
+        videoBox.appendChild(videoDiv);
+    };
+    // channels
+    var defaultChannel = getViewDefaults("channel");
+    var allChannels = allResults["channel_results"];
+    var channelBox = document.getElementById("channel-results");
+    channelBox.innerHTML = "";
+    for (let index = 0; index < allChannels.length; index++) {
+        const channel = allChannels[index]["source"];
+        const channelDiv = createChannel(channel, defaultChannel);
+        channelBox.appendChild(channelDiv);
+    };
+    // playlists
+    var defaultPlaylist = getViewDefaults("playlist");
+    var allPlaylists = allResults["playlist_results"];
+    var playlistBox = document.getElementById("playlist-results");
+    playlistBox.innerHTML = "";
+    for (let index = 0; index < allPlaylists.length; index++) {
+        const playlist = allPlaylists[index]["source"];
+        const playlistDiv = createPlaylist(playlist, defaultPlaylist);
+        playlistBox.appendChild(playlistDiv);
+    };
+}
+
+
+function createVideo(video, viewStyle) {
+    // create video item div from template
+    const videoId = video["youtube_id"];
+    const mediaUrl = video["media_url"];
+    const thumbUrl = "/cache/" + video["vid_thumb_url"];
+    const videoTitle = video["title"];
+    const videoPublished = video["published"];
+    const videoDuration = video["player"]["duration_str"];
+    if (video["player"]["watched"]) {
+        var playerState = "seen";
+    } else {
+        var playerState = "unseen";
+    };
+    const channelId = video["channel"]["channel_id"];
+    const channelName = video["channel"]["channel_name"];
+    // build markup
+    const markup = `
+    <a href="#player" data-src="/media/${mediaUrl}" data-thumb="${thumbUrl}" data-title="${videoTitle}" data-channel="${channelName}" data-channel-id="${channelId}" data-id="${videoId}" onclick="createPlayer(this)">
+        <div class="video-thumb-wrap ${viewStyle}">
+            <div class="video-thumb">
+                <img src="${thumbUrl}" alt="video-thumb">
+            </div>
+            <div class="video-play">
+                <img src="/static/img/icon-play.svg" alt="play-icon">
+            </div>
+        </div>
+    </a>
+    <div class="video-desc ${viewStyle}">
+        <div class="video-desc-player" id="video-info-${videoId}">
+                <img src="/static/img/icon-${playerState}.svg" alt="${playerState}-icon" id="${videoId}" onclick="isWatched(this.id)" class="${playerState}-icon">
+            <span>${videoPublished} | ${videoDuration}</span>
+        </div>
+        <div>
+            <a href="/channel/${channelId}/"><h3>${channelName}</h3></a>
+            <a class="video-more" href="/video/${videoId}/"><h2>${videoTitle}</h2></a>
+        </div>
+    </div>
+    `
+    const videoDiv = document.createElement("div");
+    videoDiv.setAttribute("class", "video-item " + viewStyle);
+    videoDiv.innerHTML = markup
+    return videoDiv
+}
+
+
+function createChannel(channel, viewStyle) {
+    // create channel item div from template
+    const channelId = channel["channel_id"];
+    const channelName = channel["channel_name"];
+    const channelSubs = channel["channel_subs"];
+    const channelLastRefresh = channel["channel_last_refresh"];
+    if (channel["channel_subscribed"]) {
+        var button = `<button class="unsubscribe" type="button" id="${channelId}" onclick="unsubscribe(this.id)" title="Unsubscribe from ${channelName}">Unsubscribe</button>`
+    } else {
+        var button = `<button type="button" id="${channelId}" onclick="subscribe(this.id)" title="Subscribe to ${channelName}">Subscribe</button>`
+    };
+    // build markup
+    const markup = `
+    <div class="channel-banner ${viewStyle}">
+        <a href="/channel/${channelId}/">
+            <img src="/cache/channels/${channelId}_banner.jpg" alt="${channelId}-banner">
+        </a>
+    </div>
+    <div class="info-box info-box-2 ${viewStyle}">
+        <div class="info-box-item">
+            <div class="round-img">
+                <a href="/channel/${channelId}/">
+                    <img src="/cache/channels/${channelId}_thumb.jpg" alt="channel-thumb">
+                </a>
+            </div>
+            <div>
+                <h3><a href="/channel/${channelId}/">${channelName}</a></h3>
+                <p>Subscribers: ${channelSubs}</p>
+            </div>
+        </div>
+        <div class="info-box-item">
+            <div>
+                <p>Last refreshed: ${channelLastRefresh}</p>
+                ${button}
+            </div>
+        </div>
+    </div>
+    `
+    const channelDiv = document.createElement("div");
+    channelDiv.setAttribute("class", "channel-item " + viewStyle);
+    channelDiv.innerHTML = markup;
+    return channelDiv
+}
+
+function createPlaylist(playlist, viewStyle) {
+    // create playlist item div from template
+    const playlistId = playlist["playlist_id"];
+    const playlistName = playlist["playlist_name"];
+    const playlistChannelId = playlist["playlist_channel_id"];
+    const playlistChannel = playlist["playlist_channel"];
+    const playlistLastRefresh = playlist["playlist_last_refresh"];
+    if (playlist["playlist_subscribed"]) {
+        var button = `<button class="unsubscribe" type="button" id="${playlistId}" onclick="unsubscribe(this.id)" title="Unsubscribe from ${playlistName}">Unsubscribe</button>`
+    } else {
+        var button = `<button type="button" id="${playlistId}" onclick="subscribe(this.id)" title="Subscribe to ${playlistName}">Subscribe</button>`
+    };
+    const markup = `
+    <div class="playlist-thumbnail">
+        <a href="/playlist/${playlistId}/">
+            <img src="/cache/playlists/${playlistId}.jpg" alt="${playlistId}-thumbnail">
+        </a>
+    </div>
+    <div class="playlist-desc ${viewStyle}">
+        <a href="/channel/${playlistChannelId}/"><h3>${playlistChannel}</h3></a>
+        <a href="/playlist/${playlistId}/"><h2>${playlistName}</h2></a>
+        <p>Last refreshed: ${playlistLastRefresh}</p>
+        ${button}
+    </div>
+    `
+    const playlistDiv = document.createElement("div");
+    playlistDiv.setAttribute("class", "playlist-item " + viewStyle);
+    playlistDiv.innerHTML = markup;
+    return playlistDiv
 }
 
 
@@ -363,32 +577,22 @@ function textReveal() {
     };
 }
 
-function showSearch() {
-    var searchBox = document.getElementById('search-box');
-    var displayStyle = searchBox.style.display
-    if (displayStyle === "") {
-        searchBox.style.display = 'block';
-    } else {
-        searchBox.style.display = "";
-    }
-    var inputBox = document.getElementById('searchInput');
-    inputBox.focus();
-}
-
 function showForm() {
     var formElement = document.getElementById('hidden-form');
     var displayStyle = formElement.style.display
     if (displayStyle === "") {
         formElement.style.display = 'block';
-        animate('add-icon', 'pulse-img');
     } else {
         formElement.style.display = "";
-    }
+    };
+    animate('animate-icon', 'pulse-img');
 }
 
 function animate(elementId, animationClass) {
     var toAnimate = document.getElementById(elementId);
     if (toAnimate.className !== animationClass) {
         toAnimate.className = animationClass;
-    }
+    } else {
+        toAnimate.classList.remove(animationClass);
+    };
 }

uwsgi.ini

@@ -1,8 +1,8 @@
 [uwsgi]
-; chdir = /app
-module=config.wsgi:application
-master=True
-pidfile=/tmp/project-master.pid
-vacuum=True
-max-requests=5000
-socket = :8080
+module = config.wsgi:application
+master = True
+pidfile = /tmp/project-master.pid
+vacuum = True
+max-requests = 5000
+socket = :8080
+buffer-size = 8192

version_check.py

@@ -2,7 +2,6 @@
 """ check requirements.txt for outdated packages """
 
 import pathlib
-import sys
 
 import requests
 
@@ -109,8 +108,7 @@ def main():
         if input_response == "y":
             handler.apply_updates()
         else:
-            print("cancel update...")
-            sys.exit(1)
+            print("skip update...")
 
 
 if __name__ == "__main__":