update TA_VERSION

Changed:
- Match exiting videos with new video type
- Randomize version check schedule per installation

.github/ISSUE_TEMPLATE/BUG-REPORT.yml

@@ -11,10 +11,12 @@ body:
   - type: checkboxes
     id: latest
     attributes:
-      label: Latest and Greatest
+      label: "I've read the documentation"
       options:
         - label: I'm running the latest version of Tube Archivist and have read the [release notes](https://github.com/tubearchivist/tubearchivist/releases/latest).
           required: true
+        - label: I have read through the [wiki](https://github.com/tubearchivist/tubearchivist/wiki) and the [readme](https://github.com/tubearchivist/tubearchivist#installing-and-updating), particularly the [common errors](https://github.com/tubearchivist/tubearchivist#common-errors) section.
+          required: true
 
   - type: input
     id: os

.github/ISSUE_TEMPLATE/INSTALLATION-HELP.yml

@@ -15,7 +15,7 @@ body:
       options:
         - label: I have read and understand the [installation instructions](https://github.com/tubearchivist/tubearchivist#installing-and-updating).
           required: true
-        - label: My issue is not described in the [potential pitfalls](https://github.com/tubearchivist/tubearchivist#potential-pitfalls) section.
+        - label: My issue is not described in the [common errors](https://github.com/tubearchivist/tubearchivist#common-errors) section.
           required: true
 
   - type: input

ARCHITECTURE.md

@@ -0,0 +1,48 @@
+# The Inner Workings of Tube Archivist
+This is a high level overview of the architecture of Tube Archivist, intended for interested contributors to find your way around quickly.
+
+```
+                        Tube Archivist
+        ______________________|_____________________
+        |                     |                    |
+-------------------    ---------------    -------------------
+|                 |    |             |    |                 |
+|  DjangoProject  |    |  RedisJson  |    |  ElasticSearch  |
+|                 |    |             |    |                 |
+-------------------    ---------------    -------------------
+```
+
+## DjangoProject
+This is the main Python application. Django serves its data container internally with **Uwsgi** on port 8080, the interface is served with **Nginx** on the public port 8000.
+
+Users created static files like media files and artwork as well as application artwork like logos and fonts are served directly from Nginx, while the rest of the application uses uwsgi_pass to proxy the requests to uwsgi.
+
+Config files are located in the `docker_assets` folder. The script `run.sh` is the container `CMD` command and entry point, validating env vars, connection to ElasticSearch (ES) and will start the application.
+
+Compared to other Django projects, this application doesn't make use of the database models, due to a lack of integration with ES. This project has its own abstractions and integrations, treating ES as a REST API.
+
+Long running application tasks are handed off to **Celery** - using **Redis** as a broker - to run asynchronously from the main threads. 
+- All tasks are defined in the `home.tasks.py` module.
+
+There are three Django apps:
+- **config**: The root app, routing the main endpoints and the main `settings.py` file
+- **api**: The API app with its views and functionality
+- **home**: Most of the application logic, templates and views, will probably get split up further in the future.
+
+The *home* app is split up into packages in the `src` directory:
+- **download**: All download related classes, interact with yt-dlp, download artwork, handle the download queue and post processing tasks.
+- **es**: All index setup and validation classes, handles mapping validations and makes mapping changes, wrapper functions to simplify interactions with Elasticsearch, backup and restore.
+- **frontend**: All direct interactions with the frontend, like Django forms, searching, watched state changes, and legacy api_calls in the process of moving to the api app.
+- **index**: Contains all functionality for scraping and indexing videos, channels, playlists, comments, subtitles, etc...
+- **ta**: Loose collection of functions and classes, handle application config and contains redis wrapper classes.
+
+## RedisJson
+Holds the main application config json object that gets dynamically edited from the frontend, serves as a message broker for **Celery**. Redis serves as a temporary and thread safe link between Django and the frontend, storing progress messages and temporary queues for processing. Used to store locking keys for threads and execution details for tasks.
+
+- Wrapper classes to interact with Redis are located in the `home.src.ta.ta_redis.py` module.
+
+## ElasticSearch (ES)
+Is used to store and index all metadata, functions as an application database and makes it all searchable. The mapping defines which fields are indexed as searchable text fields and which fields are used for match filtering.
+
+- The index setup and validation is handled in the `home.src.es.index_setup.py` module.
+- Wrapper classes for making requests to ES are located in the `home.src.es.connect.py` module.

Dockerfile

@@ -3,7 +3,7 @@
 
 
 # First stage to build python wheel
-FROM python:3.10.8-slim-bullseye AS builder
+FROM python:3.10.9-slim-bullseye AS builder
 ARG TARGETPLATFORM
 
 RUN apt-get update && apt-get install -y --no-install-recommends \
@@ -14,7 +14,7 @@ COPY ./tubearchivist/requirements.txt /requirements.txt
 RUN pip install --user -r requirements.txt
 
 # build final image
-FROM python:3.10.8-slim-bullseye as tubearchivist
+FROM python:3.10.9-slim-bullseye as tubearchivist
 
 ARG TARGETPLATFORM
 ARG INSTALL_DEBUG

README.md

@@ -2,9 +2,10 @@
 
 <h1 align="center">Your self hosted YouTube media server</h1>
 <div align="center">
-<a href="https://github.com/bbilly1/tilefy" target="_blank"><img src="https://tiles.tilefy.me/t/tubearchivist-docker.png" alt="tubearchivist-docker" title="Tube Archivist Docker Pulls" height="50" width="200"/></a>
-<a href="https://github.com/bbilly1/tilefy" target="_blank"><img src="https://tiles.tilefy.me/t/tubearchivist-github-star.png" alt="tubearchivist-github-star" title="Tube Archivist GitHub Stars" height="50" width="200"/></a>
-<a href="https://github.com/bbilly1/tilefy" target="_blank"><img src="https://tiles.tilefy.me/t/tubearchivist-github-forks.png" alt="tubearchivist-github-forks" title="Tube Archivist GitHub Forks" height="50" width="200"/></a>
+<a href="https://github.com/bbilly1/tilefy" target="_blank"><img src="https://tiles.tilefy.me/t/tubearchivist-docker.png" alt="tubearchivist-docker" title="Tube Archivist Docker Pulls" height="50" width="190"/></a>
+<a href="https://github.com/bbilly1/tilefy" target="_blank"><img src="https://tiles.tilefy.me/t/tubearchivist-github-star.png" alt="tubearchivist-github-star" title="Tube Archivist GitHub Stars" height="50" width="190"/></a>
+<a href="https://github.com/bbilly1/tilefy" target="_blank"><img src="https://tiles.tilefy.me/t/tubearchivist-github-forks.png" alt="tubearchivist-github-forks" title="Tube Archivist GitHub Forks" height="50" width="190"/></a>
+<a href="https://www.tubearchivist.com/discord" target="_blank"><img src="https://tiles.tilefy.me/t/tubearchivist-discord.png" alt="tubearchivist-discord" title="TA Discord Server Members" height="50" width="190"/></a>
 </div>
 
 ## Table of contents:
@@ -16,7 +17,7 @@
 * [Extended Universe](#extended-universe)
 * [Installing and updating](#installing-and-updating)
 * [Getting Started](#getting-started)
-* [Potential pitfalls](#potential-pitfalls)
+* [Common Errors](#common-errors)
 * [Roadmap](#roadmap)
 * [Known limitations](#known-limitations)
 * [Donate](#donate)
@@ -53,7 +54,7 @@
 Once your YouTube video collection grows, it becomes hard to search and find a specific video. That's where Tube Archivist comes in: By indexing your video collection with metadata from YouTube, you can organize, search and enjoy your archived YouTube videos without hassle offline through a convenient web interface.
 
 ## Connect
-- [Discord](https://discord.gg/AFwz8nE7BK): Connect with us on our Discord server.
+- [Discord](https://www.tubearchivist.com/discord): Connect with us on our Discord server.
 - [r/TubeArchivist](https://www.reddit.com/r/TubeArchivist/): Join our Subreddit.
 
 ## Extended Universe
@@ -61,11 +62,11 @@ Once your YouTube video collection grows, it becomes hard to search and find a s
 - [Tube Archivist Metrics](https://github.com/tubearchivist/tubearchivist-metrics) to create statistics in Prometheus/OpenMetrics format.
 
 ## Installing and updating
-There's dedicated user-contributed install steps under [docs/Installation.md](./docs/Installation.md) for Unraid, Truenas and Synology which you can use instead of this section if you happen to be using one of those. Otherwise, continue on.
+There's dedicated user-contributed install steps under [docs/Installation.md](./docs/Installation.md) for podman, Unraid, Truenas and Synology which you can use instead of this section if you happen to be using one of those. Otherwise, continue on.
 
-For minimal system requirements, the Tube Archivist stack needs around 2GB of available memory for a small testing setup and around 4GB of available memory for a mid to large sized installation.  
+For minimal system requirements, the Tube Archivist stack needs around 2GB of available memory for a small testing setup and around 4GB of available memory for a mid to large sized installation. Minimal with dual core with 4 threads, better quad core plus.  
 
-Note for arm64 hosts: The Tube Archivist container is multi arch, so is Elasticsearch. RedisJSON doesn't offer arm builds, but you can use the image `bbilly1/rejson` (an unofficial rebuild for arm64) instead of [the official one](https://github.com/tubearchivist/tubearchivist/blob/4af12aee15620e330adf3624c984c3acf6d0ac8b/docker-compose.yml#L27).
+Note for arm64 hosts: The Tube Archivist container is multi arch, so is Elasticsearch. RedisJSON doesn't offer arm builds, but you can use the image `bbilly1/rejson`, an unofficial rebuild for arm64.
 
 This project requires docker. Ensure it is installed and running on your system.
 
@@ -107,6 +108,7 @@ The main Python application that displays and serves your video collection, buil
   - Change the environment variables `TA_USERNAME` and `TA_PASSWORD` to create the initial credentials. 
   - `ELASTIC_PASSWORD` is for the password for Elasticsearch. The environment variable `ELASTIC_USER` is optional, should you want to change the username from the default *elastic*.
   - For the scheduler to know what time it is, set your timezone with the `TZ` environment variable, defaults to *UTC*.
+  - Set the environment variable `ENABLE_CAST=True` to send videos to your cast device, [read more](#enable-cast).
 
 ### Port collisions
 If you have a collision on port `8000`, best solution is to use dockers *HOST_PORT* and *CONTAINER_PORT* distinction: To for example change the interface to port 9000 use `9000:8000` in your docker-compose file.  
@@ -134,6 +136,18 @@ You can configure LDAP with the following environment variables:
 
 When LDAP authentication is enabled, django passwords (e.g. the password defined in TA_PASSWORD), will not allow you to login, only the LDAP server is used.
 
+### Enable Cast
+As Cast doesn't support authentication, enabling this functionality will make your static files like artwork and media files accessible by guessing the links. That's read only access, the application itself is still protected.
+
+Enabling this integration will embed an additional third party JS library from **Google**.  
+
+**Requirements**:
+- HTTPS: To use the cast integration HTTPS needs to be enabled, which can be done using a reverse proxy. This is a requirement by Google as communication to the cast device is required to be encrypted, but the content itself is not.
+- Supported Browser: A supported browser is required for this integration such as Google Chrome. Other browsers, especially Chromium-based browsers, may support casting by enabling it in the settings.
+- Subtitles: Subtitles are supported however they do not work out of the box and require additional configuration. Due to requirements by Google, to use subtitles you need additional headers which will need to be configured in your reverse proxy. See this [page](https://developers.google.com/cast/docs/web_sender/advanced#cors_requirements) for the specific requirements.  
+You need the following headers: Content-Type, Accept-Encoding, and Range. Note that the last two headers, Accept-Encoding and Range, are additional headers that you may not have needed previously.  
+Wildcards "*" can not be used for the Access-Control-Allow-Origin header. If the page has protected media content, it must use a domain instead of a wildcard.
+
 ### Elasticsearch
 **Note**: Tube Archivist depends on Elasticsearch 8. 
 
@@ -157,7 +171,7 @@ For some architectures it might be required to run Redis JSON on a nonstandard p
 - Additionally set the following value to the *archivist-redis* service: `command: --port 6380 --loadmodule /usr/lib/redis/modules/rejson.so`
 
 ### Updating Tube Archivist
-You will see the current version number of **Tube Archivist** in the footer of the interface so you can compare it with the latest release to make sure you are running the *latest and greatest*.  
+You will see the current version number of **Tube Archivist** in the footer of the interface. There is a daily version check task querying tubearchivist.com, notifying you of any new releases in the footer. To take advantage of the latest fixes and improvements, make sure you are running the *latest and greatest*.  
 * There can be breaking changes between updates, particularly as the application grows, new environment variables or settings might be required for you to set in the your docker-compose file. *Always* check the **release notes**: Any breaking changes will be marked there.  
 * All testing and development is done with the Elasticsearch version number as mentioned in the provided *docker-compose.yml* file. This will be updated when a new release of Elasticsearch is available. Running an older version of Elasticsearch is most likely not going to result in any issues, but it's still recommended to run the same version as mentioned. Use `bbilly1/tubearchivist-es` to automatically get the recommended version.
 
@@ -165,7 +179,7 @@ You will see the current version number of **Tube Archivist** in the footer of t
 
 There is a Helm Chart available at https://github.com/insuusvenerati/helm-charts. Mostly self-explanatory but feel free to ask questions in the discord / subreddit.
 
-## Potential pitfalls
+## Common Errors
 ### vm.max_map_count
 **Elastic Search** in Docker requires the kernel setting of the host machine `vm.max_map_count` to be set to at least 262144.
 
@@ -180,7 +194,7 @@ To apply the change permanently depends on your host operating system:
 - On any other platform look up in the documentation on how to pass kernel parameters.
 
 ### Permissions for elasticsearch
-If you see a message similar to `failed to obtain node locks, tried [/usr/share/elasticsearch/data]` and `maybe these locations are not writable` when initially starting elasticsearch, that probably means the container is not allowed to write files to the volume.  
+If you see a message similar to `Unable to access 'path.repo' (/usr/share/elasticsearch/data/snapshot)` or `failed to obtain node locks, tried [/usr/share/elasticsearch/data]` and `maybe these locations are not writable` when initially starting elasticsearch, that probably means the container is not allowed to write files to the volume.  
 To fix that issue, shutdown the container and on your host machine run:
 ```
 chown 1000:0 -R /path/to/mount/point
@@ -211,16 +225,16 @@ We have come far, nonetheless we are not short of ideas on how to improve and ex
 - [ ] Implement [Apprise](https://github.com/caronc/apprise) for notifications ([#97](https://github.com/tubearchivist/tubearchivist/issues/97))
 - [ ] User created playlists, random and repeat controls ([#108](https://github.com/tubearchivist/tubearchivist/issues/108), [#220](https://github.com/tubearchivist/tubearchivist/issues/220))
 - [ ] Auto play or play next link ([#226](https://github.com/tubearchivist/tubearchivist/issues/226))
-- [ ] Show similar videos on video page
 - [ ] Multi language support
 - [ ] Show total video downloaded vs total videos available in channel
 - [ ] Add statistics of index
 - [ ] Download speed schedule ([#198](https://github.com/tubearchivist/tubearchivist/issues/198))
 - [ ] Auto ignore videos by keyword ([#163](https://github.com/tubearchivist/tubearchivist/issues/163))
 - [ ] Custom searchable notes to videos, channels, playlists ([#144](https://github.com/tubearchivist/tubearchivist/issues/144))
-- [ ] Download video comments
 
 Implemented:
+- [X] Download video comments [2022-11-30]
+- [X] Show similar videos on video page [2022-11-30]
 - [X] Implement complete offline media file import from json file [2022-08-20]
 - [X] Filter and query in search form, search by url query [2022-07-23]
 - [X] Make items in grid row configurable to use more of the screen [2022-06-04]

docker-compose.yml

@@ -34,7 +34,7 @@ services:
     depends_on:
       - archivist-es
   archivist-es:
-    image: bbilly1/tubearchivist-es         # only for amd64, or use official es 8.4.3
+    image: bbilly1/tubearchivist-es         # only for amd64, or use official es 8.6.0
     container_name: archivist-es
     restart: unless-stopped
     environment:

docker_assets/run.sh

@@ -25,6 +25,11 @@ if [[ -n "$TA_UWSGI_PORT" ]]; then
     sed -i "s/8080/$TA_UWSGI_PORT/g" /app/uwsgi.ini
 fi
 
+# disable auth on static files for cast support
+if [[ -n "$ENABLE_CAST" ]]; then
+    sed -i "/auth_request/d" /etc/nginx/sites-available/default
+fi
+
 # wait for elasticsearch
 counter=0
 until curl -u "$ELASTIC_USER":"$ELASTIC_PASSWORD" "$ES_URL" -fs; do

docs/Channels.md

@@ -9,23 +9,35 @@ Accessible at `/channel/` of your Tube Archivist, the **Overview Page** shows a
 
 The **Subscribe to Channels** button <img src="assets/icon-add.png?raw=true" alt="add icon" width="20px" style="margin:0 5px;"> opens a text field to subscribe to a channel. You have a few options:
 - Enter the YouTube channel ID, a 25 character alphanumeric string. For example *UCBa659QWEk1AI4Tg--mrJ2A*
-- Enter the URL to the channel page on YouTube. For example *https://www.youtube.com/channel/UCBa659QWEk1AI4Tg--mrJ2A*
-- Enter the channel name for example: *https://www.youtube.com/c/TomScottGo*.
-- Enter the video URL for any video and let Tube Archivist extract the channel ID for you. For example *https://www.youtube.com/watch?v=2tdiKTSdE9Y*
+- Enter the URL to the channel page on YouTube. For example *https://www.youtube.com/channel/UCBa659QWEk1AI4Tg--mrJ2A* or alias url *https://www.youtube.com/@TomScottGo*
+- Enter a channel alias starting with *@*, for example: *@TomScottGo*
+- Enter the video URL for any video and let Tube Archivist extract the channel ID for you, for example *https://www.youtube.com/watch?v=2tdiKTSdE9Y*
 - Add one per line.
 
-You can search your indexed channels by clicking on the search icon <img src="assets/icon-search.png?raw=true" alt="search icon" width="20px" style="margin:0 5px;">. This will open a dedicated page.
+To search your channels, click on the search icon <img src="assets/icon-search.png?raw=true" alt="search icon" width="20px" style="margin:0 5px;"> to reach the search page. Start your query with `channel:`, learn more on the [search](Search) page.
 
 ## Channel Detail
-Each channel will get a dedicated channel detail page accessible at `/channel/<channel-id>/` of your Tube Archivist. This page shows all the videos you have downloaded from this channel.
-
+Each channel gets a set of channel detail pages.
 - If you are subscribed to the channel, an *Unsubscribe* button will show, else the *Subscribe* button will show.
 - The **Mark as Watched** button will mark all videos of this channel as watched.
 
-Additionally there is a *Channel Playlist* page, accessible at `/channel/<channel-id>/playlist/` to show all indexed playlists from this channel. 
+### Videos
+Accessible at `/channel/<channel-id>/`, this page shows all the videos you have downloaded from this channel.
 
+### Streams
+If you have any streams indexed, this page will become accessible at `/channel/<channel-id>/streams/`, this page shows all available live streams of that channel. 
+
+### Shorts
+If you have any shorts videos indexed, this page will become accessible at `/channel/<channel-id>/shorts/`, this page shows all the shorts videos of that channel.
+
+### Playlists
+If you have playlists from this channel indexed, this page will become accessible at `/channel/<channel-id>/playlist/`. Activate channel playlist indexing on the about page.
+
+### About
 On the *Channel About* page, accessible at `/channel/<channel-id>/about/`, you can see additional metadata.
 - The button **Delete Channel** will delete the channel plus all videos of this channel, both media files and metadata additionally this will also delete playlists metadata belonging to that channel.
+- The button **Reindex** will reindex all channel metadata. This will also categorize existing videos as shorts or streams.
+- The button **Reindex Videos** will reindex metadata for all videos in this channel.
 
 The channel customize form gives options to change settings on a per channel basis. Any configurations here will overwrite your configurations from the [settings](Settings) page.
 - **Download Format**: Overwrite the download quality for videos from this channel.
@@ -33,4 +45,5 @@ The channel customize form gives options to change settings on a per channel bas
 - **Index Playlists**: Automatically add all Playlists with at least a video downloaded to your index. Only do this for channels where you care about playlists as this will slow down indexing new videos for having to check which playlist this belongs to.
 - **SponsorBlock**: Using [SponsorBlock](https://sponsor.ajay.app/) to get and skip sponsored content. Customize per channel: You can *disable* or *enable* SponsorBlock for certain channels only to overwrite the behavior set on the [Settings](settings) page. Selecting *unset* will remove the overwrite and your setting will fall back to the default on the settings page.
 
-If you have any videos pending in the download queue, a *Downloads* link will show, bringing you directly to the [downloads](Downloads) page, filtering the list by the selected channel.
+### Downloads
+If you have any videos from this channel pending in the download queue, a *Downloads* link will show, bringing you directly to the [downloads](Downloads) page, filtering the list by the selected channel.

docs/Downloads.md

@@ -3,28 +3,44 @@ Accessible at `/downloads/` of your Tube Archivist, this page handles all the do
 
 
 ## Rescan Subscriptions
-The **Rescan Subscriptions** icon <img src="assets/icon-rescan.png?raw=true" alt="rescan icon" width="20px" style="margin:0 5px;"> will start a background task to look for new videos from the channels and playlists you are subscribed to. You can define the channel and playlist page size on the [settings page](Settings#subscriptions). With the default page size, expect this process to take around 2-3 seconds for each channel or playlist you are subscribed to. A status message will show the progress.
+The **Rescan Subscriptions** icon <img src="assets/icon-rescan.png?raw=true" alt="rescan icon" width="20px" style="margin:0 5px;"> will start a background task to look for new videos from the channels and playlists you are subscribed to.  
+Tube Archivist will get available *videos*, *shorts* and *streams* from each channel, you can define the channel and playlist page size on the [settings page](Settings#subscriptions). With the default page size, expect this process to take around 2-3 seconds for each channel or playlist you are subscribed to. A status message will show the progress.
 
-Then for every video found, **Tube Archivist** will skip the video if it has already been downloaded or if you added it to the *ignored* list before. All the other videos will get added to the download queue. Expect this to take around 2 seconds for each video as **Tube Archivist** needs to grab some additional metadata. New videos will get added at the bottom of the download queue.
+Then for every video found, **Tube Archivist** will skip the video if it has already been downloaded or if you added it to the *ignored* list before. All the other videos will get added to the download queue. Expect this to take around 2 seconds for each video as **Tube Archivist** needs to grab some additional metadata and artwork. New videos will get added at the bottom of the download queue.
 
 ## Download Queue
 The **Start Download** icon <img src="assets/icon-download.png?raw=true" alt="download icon" width="20px" style="margin:0 5px;"> will start the download process starting from the top of the queue. Take a look at the relevant settings on the [Settings Page](Settings#downloads). Once the process started, a progress message will show with additional details and controls: 
 - The stop icon <img src="assets/icon-stop.png?raw=true" alt="stop icon" width="20px" style="margin:0 5px;"> will gracefully stop the download process, once the current video has been finished successfully.
 - The cancel icon <img src="assets/icon-close-red.png?raw=true" alt="close icon" width="20px" style="margin:0 5px;"> is equivalent to killing the process and will stop the download immediately. Any leftover files will get deleted, the canceled video will still be available in the download queue.
 
-After downloading, Tube Archivist tries to add new videos to already indexed playlists.
+After downloading, Tube Archivist tries to add new videos to already indexed playlists and if activated on the settings page, get comments for the new videos.
 
 ## Add to Download Queue
-The **Add to Download Queue** icon <img src="assets/icon-add.png?raw=true" alt="add icon" width="20px" style="margin:0 5px;"> opens a text field to manually add videos to the download queue. You have a few options:
-- Add a link to a YouTube video. For example *https://www.youtube.com/watch?v=2tdiKTSdE9Y*.
-- Add a YouTube video ID. For example *2tdiKTSdE9Y*.
-- Add a link to a YouTube video by providing the shortened URL, for example *https://youtu.be/2tdiKTSdE9Y*.
-- Add a Channel ID or Channel URL to add every available video to the download queue. This will ignore the channel page size as described before and is meant for an initial download of the whole channel. You can still ignore selected videos before starting the download.
-- Add a channel name like for example *https://www.youtube.com/c/TomScottGo*.
+The **Add to Download Queue** icon <img src="assets/icon-add.png?raw=true" alt="add icon" width="20px" style="margin:0 5px;"> opens a text field to manually add videos to the download queue. Add one item per line. You have a few options:
+
+### Videos
+- Add a YouTube video ID, for example *2tdiKTSdE9Y*
+- Add a link to a YouTube video, for example *https://www.youtube.com/watch?v=2tdiKTSdE9Y*
+- Add a link to a YouTube video by providing the shortened URL, for example *https://youtu.be/2tdiKTSdE9Y*
+- Add a link to a shorts video, for example *https://www.youtube.com/shorts/UOfe6e0k7cQ*
+
+### Channels
+- When adding a channel, Tube Archivist will ignore the channel page size as described above, this is meant for an initial download of the whole channel. You can still ignore selected videos from the queue before starting the download.
+- Download a complete channel including shorts and streams by entering:
+  - Channel ID: *UCBa659QWEk1AI4Tg--mrJ2A*
+  - Channel URL: *https://www.youtube.com/channel/UCBa659QWEk1AI4Tg--mrJ2A*
+  - Channel *@* alias handler: For example *@TomScottGo*
+  - Channel alias URL: *https://www.youtube.com/@TomScottGo*
+- Download videos, live streams or shorts only, by providing a partial channel URL:
+  - Videos only: *https://www.youtube.com/@IBRACORP/videos*
+  - Shorts only: *https://www.youtube.com/@IBRACORP/shorts*
+  - Streams only: *https://www.youtube.com/@IBRACORP/streams*
+  - Every other channel sub page will default to download all, for example *https://www.youtube.com/@IBRACORP/featured* will download videos and shorts and streams.
+
+### Playlist
 - Add a playlist ID or URL to add every available video in the list to the download queue, for example *https://www.youtube.com/playlist?list=PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha* or *PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha*. 
-  - Note: When adding a playlist to the queue, this playlist will automatically get [indexed](Playlists#playlist-detail).
-  - Note: When you add a link to a video in a playlist, Tube Archivist assumes you want to download only the specific video and not the whole playlist, for example *https://www.youtube.com/watch?v=CINVwWHlzTY&list=PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha* will only add one video *CINVwWHlzTY* to the queue.
-- Add one link per line.
+- When adding a playlist to the queue, this playlist will automatically get [indexed](Playlists#playlist-detail).
+- When you add a link to a video in a playlist, Tube Archivist assumes you want to download only the specific video and not the whole playlist, for example *https://www.youtube.com/watch?v=CINVwWHlzTY&list=PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha* will only add one video *CINVwWHlzTY* to the queue.
 
 ## The Download Queue
 Below the three buttons you find the download queue. New items will get added at the bottom of the queue, the next video to download once you click on **Start Download** will be the first in the list.
@@ -37,7 +53,7 @@ Every video in the download queue has two buttons:
 
 You can flip the view by activating **Show Only Ignored Videos**. This will show all videos you have previously *ignored*.  
 Every video in the ignored list has two buttons:
-- **Forget**: This will delete the item form the ignored list.
+- **Forget**: This will delete the item from the ignored list.
 - **Add to Queue**: This will add the ignored video back to the download queue.  
 
 You can delete your download queue from the [Settings](Settings#actions) page.

docs/Installation.md

@@ -1,40 +1,71 @@
 # Detailed Installation Instructions for Various Platforms
 
 ## Table of Contents
+- [Podman](#podman)
 - [Unraid](#unraid)
 - [Truenas Scale](#truenas-scale)
 - [Synology](#synology)
 
 These are beginners guides installation instructions for additional platforms generously provided by users of these platforms. When in doubt, verify the details with the [Readme](https://github.com/tubearchivist/tubearchivist#installing-and-updating). If you see any issues here while using these instructions, please contribute. 
 
+## Podman
+Podman handles container hostname resolving slightly differently than docker, so you need to make a few changes to the `docker-compose.yml` to get up and running.
+
+### Step 1: Follow the installation instructions from the [README](https://github.com/tubearchivist/tubearchivist#installing-and-updating), with a few additional changes to the `docker-compose.yml`.
+
+Edit these additional changes to the `docker-compose.yml`:
+ - under `tubearchivist`->`image`:
+ 	- prefix the container name with `docker.io/` (or the url of your repo of choice).
+ - under `tubearchivist`->`environment`:
+  - `ES_URL`: change `archivist-es` to the internal IP of the computer that will be running the containers.
+  - `REDIS_HOST`: change `archivist-redis` to the internal IP of the computer that will be running the containers (should be the same as above).
+ - under `archivist-redis`->`image`: 
+ 	- prefix the container name with `docker.io/` again.
+ - under `archivist-redis`->`expose`: 
+  - change the whole entry from `expose: ["<PORT>"]` into `ports: ["<PORT>:<PORT>"].
+ - under `archivist-es`->`image`: 
+ 	- prefix the container name with `docker.io/` again.
+ - under `archivist-es`->`expose`: 
+  - change the whole entry from `expose: ["<PORT>"]` into `ports: ["<PORT>:<PORT>"].
+
+### Step 2: Create service files (optional)
+
+Since podman doesn't run as a service, it can't start containers after reboots, at least not without some help.
+
+If you want to enable this behavior, you can follow [this example](https://techblog.jeppson.org/2020/04/create-podman-services-with-podman-compose/) to have `systemd` start up the containers with `podman-compose` when the computer boots up.
+
+
 ## Unraid
 
-Tube Archivist, and all if it's dependencies are located in the [community applications](https://forums.unraid.net/topic/38582-plug-in-community-applications/) store. The three containers you will need are as follows:
+Tube Archivist, and all if it's dependencies are located in the [community applications](https://unraid.net/community/apps?q=tubearchivist) store. The three containers you will need are as follows:
 
 -   **TubeArchivist-RedisJSON**: This container acts as a cache and temporary link between the application and the file system. Used to store and display messages and configuration variables.
--   **TubeArchivist-ES**: ElasticSearch stores video meta data and makes everything searchable. Also keeps track of the download queue.
+-   **TubeArchivist-ES**: ElasticSearch stores video metadata and makes everything searchable. Also keeps track of the download queue.
 -   **TubeArchivist**: Once your YouTube video collection grows, it becomes hard to search and find a specific video. That's where Tube Archivist comes in: By indexing your video collection with metadata from YouTube, you can organize, search and enjoy your archived YouTube videos without hassle offline through a convenient web interface.
 
 ### Step 1: Install  `TubeArchivist-RedisJSON`
 
-![enter image description here](https://i.imgur.com/ycAqFRU.png)
-This is the easiest container to setup of the thee, just make sure that you do not have any port conflicts, and that your `/data` is mounted to the correct path. The other containers will map to the same directory.
+![TubeArchivist-RedisJSON](https://i.imgur.com/iGJZKYd.png)
+This is the easiest container to setup of the thee, just make sure that you do not have any port conflicts, and that your `/data` is mounted to the correct path. The other containers will map to the same root directory (/mnt/user/appdata/TubeArchivist).
 
 If you need to install `TubeArchivist-RedisJSON`on a different port, you'll have to follow [these steps](https://github.com/tubearchivist/tubearchivist#redis-on-a-custom-port) later on when installing the `TubeArchivist` container
 
 
 ### Step 2: Install  `TubeArchivist-ES`
-![enter image description here](https://i.imgur.com/o6tsTdt.png)
-ElasticSeach is also pretty easy to setup. Again, make sure you have no port conflicts, make sure that you mapped `/usr/share/elasticsearch/data` to the same directory as `RedisJSON`, and make sure to change the default password to something more secure. 
+![TubeArchivist-ES](https://i.imgur.com/RC38b3u.png)
+ElasticSeach is also pretty easy to setup. Again, make sure you have no port conflicts, make sure that you mapped the ElasticSearch Data to the same root directory as `RedisJSON` (/mnt/user/appdata/TubeArchivist), and make sure to change the default password to something more secure. 
 
-There is three additional settings in the "show more settings" area, but leave those as they are.
+There is four additional settings in the "show more settings" area, but don't require any changes.
 
 
 ### Step 3: Install  `TubeArchivist`
 
-![enter image description here](https://i.imgur.com/dwSCfgO.png)
+![TubeArchivist](https://i.imgur.com/oLsmwBZ.png)
 It's finally time to set up TubeArchivist!
 
+ - `HOST:`This is a list of IP addresses that you will host TA from. Example, 192.168.0.14 is the IP address of my Unraid server. If I was going to access TA from a VPN, or domain name, I'd put those next to my host IP with just a space separating the different addresses. More information [here.](https://github.com/tubearchivist/tubearchivist#installing-and-updating)
+
+
  - `Port:`Again, make sure that you have no port conflicts on 8000.
    
  - `Youtube Media Path:` is where you'll download all of your videos to.
@@ -44,23 +75,23 @@ It's finally time to set up TubeArchivist!
    wiki.](https://github.com/tubearchivist/tubearchivist/wiki/Settings#manual-media-files-import)
    
    
-- `Appdata:` This should be the same base path as the other two containers.
+- `Appdata:` This should be the same base path as the other two containers (/mnt/user/appdata/TubeArchivist).
    
  - `TA Username:`This will be your username for TubeArchivist.
    
  - `TA Password:`This will be your password for TubeArchivist.
    
- - `Redis` This will be JUST the ip address of your redis container
+ - `Redis` This will be JUST the ip address of your redis container.
 
  - `ElasticSearch Password:`This is the password you defined in the `TubeArchivist-ES` container.
  - `ElasticSearch:` This seems to cause some confusion, but it's a pretty simple step, just replace the IP and Port to match you `TubeArchivist-ES` container.
 
- (example: if your IP is 192.168.1.15, the value should be http://192.168.1.15:9200)
+ (example: if your IP is 192.168.0.14, the value should be http://192.168.0.14:9200)
 
  - `Time Zone:` This is an important step for your scheduler, to find your timezone, use a site like [TimeZoneConverter](http://www.timezoneconverter.com/cgi-bin/findzone.tzc) 
 
 ### From there, you should be able to start up your containers and you're good to go!
-If you're still having trouble, join us on [discord](https://discord.gg/AFwz8nE7BK) and come to the #unraid channel.
+If you're still having trouble, join us on [discord](https://www.tubearchivist.com/discord) and come to the #support channel.
 
 <br />
 <br />
@@ -195,7 +226,9 @@ Once all of the folders have been created, it should have a folder structure wit
         - "GID=0"
         - "xpack.security.enabled=true"
         - "ELASTIC_PASSWORD=verysecret" 
+        - "path.repo=/usr/share/elasticsearch/data/snapshot"
           > Do not use the default password as it is very insecure.
+          > Activating snapshots for backups should only be done *after* setting the `path.repo` setting.
         ![Synology - ElasticSearch Environment Configurations](assets/Synology_0.2.0_Docker-ES-Env-Conf.png)
     12. Click on the **Apply** button.
     13. Back on the **Create Container** screen, click the **Next** button.
@@ -252,4 +285,4 @@ Once all of the folders have been created, it should have a folder structure wit
 <!-- This section is a Work In Progress -->
 
 ### From there, you should be able to start up your containers and you're good to go!
-If you're still having trouble, join us on [discord](https://discord.gg/AFwz8nE7BK) and come to the #synology channel.
+If you're still having trouble, join us on [discord](https://www.tubearchivist.com/discord) and come to the #support channel.

docs/Playlists.md

@@ -19,5 +19,7 @@ You can search your indexed playlists by clicking on the search icon <img src="a
 Each playlist will get a dedicated playlist detail page accessible at `/playlist/<playlist-id>/` of your Tube Archivist. This page shows all the videos you have downloaded from this playlist.
 
 - If you are subscribed to the playlist, an Unsubscribe button will show, else the Subscribe button will show.
-- The Mark as Watched button will mark all videos of this playlist as watched.
+- The **Mark as Watched** button will mark all videos of this playlist as watched.
+- The button **Reindex** will reindex the playlist metadata.
+- The button **Reindex Videos** will reindex all videos from this playlist.
 - The **Delete Playlist** button will give you the option to delete just the *metadata* which won't delete any media files or *delete all* which will delete metadata plus all videos belonging to this playlist.

docs/Settings.md

@@ -12,6 +12,8 @@ Switch between the easy on the eyes dark theme and the burning bright theme.
 ## Subscriptions
 Settings related to the channel management.
 - **Channel Page Size**: Defines how many pages will get analyzed by **Tube Archivist** each time you click on *Rescan Subscriptions*. The default page size used by yt-dlp is **50**, that's also the recommended value to set here. Any value higher will slow down the rescan process, for example if you set the value to 51, that means yt-dlp will have to go through 2 pages of results instead of 1 and by that doubling the time that process takes.
+- **Live Page Size**: Same as above, but for channel live streams.
+- **Shorts page Size**: Same as above, but for shorts videos.
 
 ## Downloads
 Settings related to the download process.
@@ -32,6 +34,15 @@ Additional settings passed to yt-dlp.
 - **Source Settings**: User created subtitles are provided from the uploader and are usually the video script. Auto generated is from YouTube, quality varies, particularly for auto translated tracks.
 - **Index Settings**: Enabling subtitle indexing will add the lines to Elasticsearch and will make subtitles searchable. This will increase the index size and is not recommended on low-end hardware.
 
+## Comments
+- **Download and index comments**: Set your configuration for downloading and indexing comments. This takes the same values as documented in the `max_comments` section for the youtube extractor of [yt-dlp](https://github.com/yt-dlp/yt-dlp#youtube). Add without space between the four different fields: *max-comments,max-parents,max-replies,max-replies-per-thread*. Example:
+  - `all,100,all,30`: Get 100 max-parents and 30 max-replies-per-thread.
+  - `1000,all,all,50`: Get a total of 1000 comments over all, 50 replies per thread.
+- **Comment sort method**: Change sort method between *top* or *new*. The default is *top*, as decided by YouTube.
+- The [Refresh Metadata](#refresh-metadata) background task will get comments from your already archived videos, spreading the requests out over time.  
+
+Archiving comments is slow as only very few comments get returned per request with yt-dlp. Choose your configuration above wisely. Tube Archivist will download comments after the download queue finishes, your videos will be already available while the comments are getting downloaded.
+
 ## Cookie
 Importing your YouTube Cookie into Tube Archivist allows yt-dlp to bypass age restrictions, gives access to private videos and your *watch later* or *liked videos*.
 
@@ -63,14 +74,6 @@ All third party integrations of TubeArchivist will **always** be *opt in*.
 - **API**: Your access token for the Tube Archivist API.
 - **returnyoutubedislike.com**: This will get return dislikes and average ratings for each video by integrating with the API from [returnyoutubedislike.com](https://www.returnyoutubedislike.com/).
 - **SponsorBlock**: Using [SponsorBlock](https://sponsor.ajay.app/) to get and skip sponsored content. If a video doesn't have timestamps, or has unlocked timestamps, use the browser addon to contribute to this excellent project. Can also be activated and deactivated as a per [channel overwrite](Settings#channel-customize).
-- **Cast**: Enabling the cast integration in the settings page will load an additional JS library from **Google**.  
-*NOTE*: This feature is currently broken due to an authentication issue, see #331.
-Requirements:
-    - HTTPS: To use the cast integration HTTPS needs to be enabled, which can be done using a reverse proxy. This is a requirement by Google as communication to the cast device is required to be encrypted, but the content itself is not.
-    - Supported Browser:A supported browser is required for this integration such as Google Chrome. Other browsers, especially Chromium-based browsers, may support casting by enabling it in the settings.
-    - Subtitles: Subtitles are supported however they do not work out of the box and require additional configuration. Due to requirements by Google, to use subtitles you need additional headers which will need to be configured in your reverse proxy. See this [page](https://developers.google.com/cast/docs/web_sender/advanced#cors_requirements) for the specific requirements.  
-    You need the following headers: Content-Type, Accept-Encoding, and Range. Note that the last two headers, Accept-Encoding and Range, are additional headers that you may not have needed previously.  
-    Wildcards "*" cannot be used for the Access-Control-Allow-Origin header. If the page has protected media content, it must use a domain instead of a wildcard.
 
 ## Snapshots
 System snapshots will automatically make daily snapshots of the Elasticsearch index. The task will start at 12pm your local time. Snapshots are deduplicated, meaning that each snapshot will only have to backup changes since the last snapshot. The initial snapshot may be slow, but subsequent runs will be much faster. There is also a cleanup function implemented, that will remove snapshots older than 30 days.
@@ -102,7 +105,7 @@ Examples:
 NOTE:
 - Changes in the scheduler settings require a container restart to take effect.
 - Cron format as *number*/*number* are none standard cron and are not supported by the scheduler, for example **0 0/12 \*** is invalid, use **0 \*/12 \*** instead.
-- Avoid an unnecessary frequent schedule to not get blocked by YouTube. For that reason * or wildcards for minutes are not supported.
+- Avoid an unnecessary frequent schedule to not get blocked by YouTube. For that reason, the scheduler doesn't support schedules that trigger more than once per hour.
 
 ## Rescan Subscriptions
 That's the equivalent task as run from the downloads page looking through your channel and playlist and add missing videos to the download queue.

docs/Video.md

@@ -1,11 +1,25 @@
 # Video Page
-
-Every video downloaded gets a dedicated page accessible at `/video/<video-id>` of your Tube Archivist.
+Every video downloaded gets a dedicated page accessible at `/video/<video-id>/` of your Tube Archivist.
 
 Clicking on the channel name or the channel icon will bring you to the dedicated channel detail [page](Channels#channel-detail).
 
-The button **Delete Video** will delete that video including the media file.
+- The button **Reindex** will reindex the metadata of this video.
+- The button **Download File** will download the media file in the browser.
+- The button **Delete Video** will delete that video including the media file.
 
+If available, a tag cloud will show, representing the tags set by the uploader.
+
+The video description is truncated to the first few lines, click on *show more* to expand the whole description.
+
+## Playlist
 When available, a playlist navigation will show at the bottom. Clicking on the playlist name will bring you to the dedicated [Playlist Detail](Playlists#playlist-detail) page showing all videos downloaded from that playlist. The number in square brackets indicates the position of the current video in that playlist.
 
-Clicking on the next or previous video name or thumbnail will bring you to that dedicated video page.
+Clicking on the next or previous video name or thumbnail will bring you to that dedicated video page.
+
+## Similar Videos
+Tube Archivist will show up to six similar videos in a grid. Similarity is detected from the **video title** and the **video tags**. This naturally will show some videos from the same channel, but can also return videos about the same topic from other channels.
+
+When playing a video from the similar section with the inline player, the current video will get replaced, refresh the page to reset that or click on the video title to avoid that behavior. 
+
+## Comments
+If activated on the settings page, this will show the indexed comments. Reveal the threads by clicking the *+ Replies* button. Comments with a heart symbol are favorited by the uploader, comments by the uploader are highlighted in a different color.

tubearchivist/api/README.md

@@ -12,6 +12,8 @@ Note:
 **Video**
 - [Video List](#video-list-view)
 - [Video Single](#video-item-view)
+- [Video Comments](#video-comment-view)
+- [Video Similar](#video-similar-view)
 - [Video Single Progress](#video-progress-view)
 - [Video Single Sponsorblock](#sponsor-block-view) WIP
 
@@ -36,8 +38,10 @@ Note:
 **Additional**
 - [Login](#login-view)
 - [Task](#task-view) WIP
+- [Refresh](#refresh-view)
 - [Cookie](#cookie-view)
 - [Search](#search-view)
+- [Watched](#watched-view)
 - [Ping](#ping-view)
 
 ## Authentication
@@ -76,15 +80,22 @@ Pass page number as a query parameter: `page=2`. Defaults to *0*, `page=1` is re
 /api/video/
 
 ## Video Item View
-/api/video/\<video_id>/
+GET: /api/video/\<video_id>/  
+DELETE: /api/video/\<video_id>/
+
+## Video Comment View
+/api/video/\<video_id>/comment/  
+
+## Video Similar View
+/api/video/\<video_id>/similar/  
 
 ## Video Progress View
-/api/video/\<video_id>/progress  
+/api/video/\<video_id>/progress/  
 
 Progress is stored for each user.
 
 ### Get last player position of a video
-GET /api/video/\<video_id>/progress
+GET /api/video/\<video_id>/progress/
 ```json
 {
     "youtube_id": "<video_id>",
@@ -94,7 +105,7 @@ GET /api/video/\<video_id>/progress
 ```
 
 ### Post player position of video
-POST /api/video/\<video_id>/progress
+POST /api/video/\<video_id>/progress/
 ```json
 {
     "position": 100
@@ -102,7 +113,7 @@ POST /api/video/\<video_id>/progress
 ```
 
 ### Delete player position of video
-DELETE /api/video/\<video_id>/progress  
+DELETE /api/video/\<video_id>/progress/  
 
 
 ## Sponsor Block View
@@ -155,7 +166,9 @@ POST /api/channel/
 ```
 
 ## Channel Item View
-/api/channel/\<channel_id>/
+GET: /api/channel/\<channel_id>/  
+DELETE: /api/channel/\<channel_id>/
+- Will delete channel with all it's videos
 
 ## Channel Videos View
 /api/channel/\<channel_id>/video/
@@ -208,6 +221,13 @@ Add to queue previously ignored video:
 }
 ```
 
+Download existing video now:
+```json
+{
+    "status": "priority"
+}
+```
+
 DELETE /api/download/\<video_id>/  
 Forget or delete from download queue
 
@@ -255,7 +275,7 @@ Remove this snapshot from index
 
 ## Login View
 Return token and user ID for username and password:  
-POST /api/login
+POST /api/login/
 ```json
 {
     "username": "tubearchivist",
@@ -298,6 +318,52 @@ List of valid task names:
 - **download_pending**: Start the download queue
 - **rescan_pending**: Rescan your subscriptions
 
+## Refresh View
+GET /api/refresh/  
+parameters:
+- **type**: one of *video*, *channel*, *playlist*, optional
+- **id**: item id, optional
+
+without specifying type: return total for all queued items:
+```json
+{
+    "total_queued": 2,
+    "type": "all",
+    "state": "running"
+}
+```
+
+specify type: return total items queue of this type:
+```json
+{
+    "total_queued": 2,
+    "type": "video",
+    "state": "running"
+}
+```
+
+specify type *and* id to get state of item in queue:
+```json
+{
+    "total_queued": 2,
+    "type": "video",
+    "state": "in_queue",
+    "id": "video-id"
+}
+```
+
+POST /api/refresh/  
+Parameter:
+- extract_videos: to refresh all videos for channels/playlists, default False
+
+Manually start a refresh task: post list of *video*, *channel*, *playlist* IDs.
+```json
+{
+    "video": ["video1", "video2", "video3"],
+    "channel": ["channel1", "channel2", "channel3"],
+    "playlist": ["playlist1", "playlist2"]
+}
+```
 
 ## Cookie View
 Check your youtube cookie settings, *status* turns to `true` if cookie has been validated.  
@@ -346,9 +412,22 @@ GET /api/search/?query=\<query>
 
 Returns search results from your query.
 
+## Watched View
+POST /api/watched/  
+
+Change watched state, where the `id` can be a single video, or channel/playlist to change all videos belonging to that channel/playlist.
+
+```json
+{
+    "id": "xxxxxxx",
+    "is_watched": True
+}
+```
+
+
 ## Ping View
 Validate your connection with the API  
-GET /api/ping
+GET /api/ping/
 
 When valid returns message with user id: 
 ```json

tubearchivist/api/src/search_processor.py

@@ -48,6 +48,8 @@ class SearchProcess:
             processed = self._process_playlist(result["_source"])
         if index == "ta_download":
             processed = self._process_download(result["_source"])
+        if index == "ta_comment":
+            processed = self._process_comment(result["_source"])
 
         return processed
 
@@ -123,3 +125,17 @@ class SearchProcess:
             }
         )
         return dict(sorted(download_dict.items()))
+
+    def _process_comment(self, comment_dict):
+        """run on all comments, create reply thread"""
+        all_comments = comment_dict["comment_comments"]
+        processed_comments = []
+
+        for comment in all_comments:
+            if comment["comment_parent"] == "root":
+                comment.update({"comment_replies": []})
+                processed_comments.append(comment)
+            else:
+                processed_comments[-1]["comment_replies"].append(comment)
+
+        return processed_comments

tubearchivist/api/urls.py

@@ -12,14 +12,18 @@ from api.views import (
     PlaylistApiListView,
     PlaylistApiVideoView,
     PlaylistApiView,
+    RefreshView,
     SearchView,
     SnapshotApiListView,
     SnapshotApiView,
     TaskApiView,
     VideoApiListView,
     VideoApiView,
+    VideoCommentView,
     VideoProgressView,
+    VideoSimilarView,
     VideoSponsorView,
+    WatchedView,
 )
 from django.urls import path
 
@@ -41,6 +45,16 @@ urlpatterns = [
         VideoProgressView.as_view(),
         name="api-video-progress",
     ),
+    path(
+        "video/<slug:video_id>/comment/",
+        VideoCommentView.as_view(),
+        name="api-video-comment",
+    ),
+    path(
+        "video/<slug:video_id>/similar/",
+        VideoSimilarView.as_view(),
+        name="api-video-similar",
+    ),
     path(
         "video/<slug:video_id>/sponsor/",
         VideoSponsorView.as_view(),
@@ -86,6 +100,11 @@ urlpatterns = [
         DownloadApiView.as_view(),
         name="api-download",
     ),
+    path(
+        "refresh/",
+        RefreshView.as_view(),
+        name="api-refresh",
+    ),
     path(
         "task/",
         TaskApiView.as_view(),
@@ -106,6 +125,11 @@ urlpatterns = [
         CookieView.as_view(),
         name="api-cookie",
     ),
+    path(
+        "watched/",
+        WatchedView.as_view(),
+        name="api-watched",
+    ),
     path(
         "search/",
         SearchView.as_view(),

tubearchivist/api/views.py

@@ -7,12 +7,15 @@ from home.src.download.yt_dlp_base import CookieHandler
 from home.src.es.connect import ElasticWrap
 from home.src.es.snapshot import ElasticSnapshot
 from home.src.frontend.searching import SearchForm
+from home.src.frontend.watched import WatchState
+from home.src.index.channel import YoutubeChannel
 from home.src.index.generic import Pagination
-from home.src.index.video import SponsorBlock
+from home.src.index.reindex import ReindexProgress
+from home.src.index.video import SponsorBlock, YoutubeVideo
 from home.src.ta.config import AppConfig
-from home.src.ta.helper import UrlListParser
 from home.src.ta.ta_redis import RedisArchivist, RedisQueue
-from home.tasks import extrac_dl, subscribe_to
+from home.src.ta.urlparser import Parser
+from home.tasks import check_reindex, download_single, extrac_dl, subscribe_to
 from rest_framework.authentication import (
     SessionAuthentication,
     TokenAuthentication,
@@ -43,7 +46,6 @@ class ApiBaseView(APIView):
     def get_document(self, document_id):
         """get single document from es"""
         path = f"{self.search_base}{document_id}"
-        print(path)
         response, status_code = ElasticWrap(path).get()
         try:
             self.response["data"] = SearchProcess(response).process()
@@ -62,10 +64,11 @@ class ApiBaseView(APIView):
             }
         )
 
-    def get_document_list(self, request):
+    def get_document_list(self, request, pagination=True):
         """get a list of results"""
-        print(self.search_base)
-        self.initiate_pagination(request)
+        if pagination:
+            self.initiate_pagination(request)
+
         es_handler = ElasticWrap(self.search_base)
         response, status_code = es_handler.get(data=self.data)
         self.response["data"] = SearchProcess(response).process()
@@ -74,8 +77,11 @@ class ApiBaseView(APIView):
         else:
             self.status_code = 404
 
-        self.pagination_handler.validate(response["hits"]["total"]["value"])
-        self.response["paginate"] = self.pagination_handler.pagination
+        if pagination:
+            self.pagination_handler.validate(
+                response["hits"]["total"]["value"]
+            )
+            self.response["paginate"] = self.pagination_handler.pagination
 
 
 class VideoApiView(ApiBaseView):
@@ -91,6 +97,20 @@ class VideoApiView(ApiBaseView):
         self.get_document(video_id)
         return Response(self.response, status=self.status_code)
 
+    def delete(self, request, video_id):
+        # pylint: disable=unused-argument
+        """delete single video"""
+        message = {"video": video_id}
+        try:
+            YoutubeVideo(video_id).delete_media_file()
+            status_code = 200
+            message.update({"state": "delete"})
+        except FileNotFoundError:
+            status_code = 404
+            message.update({"state": "not found"})
+
+        return Response(message, status=status_code)
+
 
 class VideoApiListView(ApiBaseView):
     """resolves to /api/video/
@@ -145,6 +165,46 @@ class VideoProgressView(ApiBaseView):
         return Response(self.response)
 
 
+class VideoCommentView(ApiBaseView):
+    """resolves to /api/video/<video_id>/comment/
+    handle video comments
+    GET: return all comments from video with reply threads
+    """
+
+    search_base = "ta_comment/_doc/"
+
+    def get(self, request, video_id):
+        """get video comments"""
+        # pylint: disable=unused-argument
+        self.get_document(video_id)
+
+        return Response(self.response, status=self.status_code)
+
+
+class VideoSimilarView(ApiBaseView):
+    """resolves to /api/video/<video-id>/similar/
+    GET: return max 3 videos similar to this
+    """
+
+    search_base = "ta_video/_search/"
+
+    def get(self, request, video_id):
+        """get similar videos"""
+        self.data = {
+            "size": 6,
+            "query": {
+                "more_like_this": {
+                    "fields": ["tags", "title"],
+                    "like": {"_id": video_id},
+                    "min_term_freq": 1,
+                    "max_query_terms": 25,
+                }
+            },
+        }
+        self.get_document_list(request, pagination=False)
+        return Response(self.response, status=self.status_code)
+
+
 class VideoSponsorView(ApiBaseView):
     """resolves to /api/video/<video_id>/sponsor/
     handle sponsor block integration
@@ -207,6 +267,20 @@ class ChannelApiView(ApiBaseView):
         self.get_document(channel_id)
         return Response(self.response, status=self.status_code)
 
+    def delete(self, request, channel_id):
+        # pylint: disable=unused-argument
+        """delete channel"""
+        message = {"channel": channel_id}
+        try:
+            YoutubeChannel(channel_id).delete_channel()
+            status_code = 200
+            message.update({"state": "delete"})
+        except FileNotFoundError:
+            status_code = 404
+            message.update({"state": "not found"})
+
+        return Response(message, status=status_code)
+
 
 class ChannelApiListView(ApiBaseView):
     """resolves to /api/channel/
@@ -321,7 +395,7 @@ class DownloadApiView(ApiBaseView):
     """
 
     search_base = "ta_download/_doc/"
-    valid_status = ["pending", "ignore"]
+    valid_status = ["pending", "ignore", "priority"]
 
     def get(self, request, video_id):
         # pylint: disable=unused-argument
@@ -331,15 +405,23 @@ class DownloadApiView(ApiBaseView):
 
     def post(self, request, video_id):
         """post to video to change status"""
-        item_status = request.data["status"]
+        item_status = request.data.get("status")
         if item_status not in self.valid_status:
             message = f"{video_id}: invalid status {item_status}"
             print(message)
             return Response({"message": message}, status=400)
 
+        pending_video, status_code = PendingInteract(video_id).get_item()
+        if status_code == 404:
+            message = f"{video_id}: item not found {status_code}"
+            return Response({"message": message}, status=404)
+
         print(f"{video_id}: change status to {item_status}")
-        PendingInteract(video_id=video_id, status=item_status).update_status()
-        RedisQueue().clear_item(video_id)
+        if item_status == "priority":
+            download_single.delay(pending_video)
+        else:
+            PendingInteract(video_id, item_status).update_status()
+            RedisQueue(queue_name="dl_queue").clear_item(video_id)
 
         return Response(request.data)
 
@@ -391,7 +473,6 @@ class DownloadApiListView(ApiBaseView):
     @staticmethod
     def post(request):
         """add list of videos to download queue"""
-        print(f"request meta data: {request.META}")
         data = request.data
         try:
             to_add = data["data"]
@@ -403,7 +484,7 @@ class DownloadApiListView(ApiBaseView):
         pending = [i["youtube_id"] for i in to_add if i["status"] == "pending"]
         url_str = " ".join(pending)
         try:
-            youtube_ids = UrlListParser(url_str).process_list()
+            youtube_ids = Parser(url_str).parse()
         except ValueError:
             message = f"failed to parse: {url_str}"
             print(message)
@@ -478,10 +559,7 @@ class TaskApiView(ApiBaseView):
 
     def post(self, request):
         """handle post request"""
-
-        data = request.data
-        print(data)
-        response = TaskHandler(data).run_task()
+        response = TaskHandler(request.data).run_task()
 
         return Response(response)
 
@@ -550,6 +628,38 @@ class SnapshotApiView(ApiBaseView):
         return Response(response)
 
 
+class RefreshView(ApiBaseView):
+    """resolves to /api/refresh/
+    GET: get refresh progress
+    POST: start a manual refresh task
+    """
+
+    def get(self, request):
+        """handle get request"""
+        request_type = request.GET.get("type")
+        request_id = request.GET.get("id")
+
+        if request_id and not request_type:
+            return Response({"status": "Bad Request"}, status=400)
+
+        try:
+            progress = ReindexProgress(
+                request_type=request_type, request_id=request_id
+            ).get_progress()
+        except ValueError:
+            return Response({"status": "Bad Request"}, status=400)
+
+        return Response(progress)
+
+    def post(self, request):
+        """handle post request"""
+        data = request.data
+        extract_videos = bool(request.GET.get("extract_videos", False))
+        check_reindex.delay(data=data, extract_videos=extract_videos)
+
+        return Response(data)
+
+
 class CookieView(ApiBaseView):
     """resolves to /api/cookie/
     GET: check if cookie is enabled
@@ -602,6 +712,24 @@ class CookieView(ApiBaseView):
         return Response(message)
 
 
+class WatchedView(ApiBaseView):
+    """resolves to /api/watched/
+    POST: change watched state of video, channel or playlist
+    """
+
+    def post(self, request):
+        """change watched state"""
+        youtube_id = request.data.get("id")
+        is_watched = request.data.get("is_watched")
+
+        if not youtube_id or is_watched is None:
+            message = {"message": "missing id or is_watched"}
+            return Response(message, status=400)
+
+        WatchState(youtube_id, is_watched).change()
+        return Response({"message": "success"}, status=200)
+
+
 class SearchView(ApiBaseView):
     """resolves to /api/search/
     GET: run a search with the string in the ?query parameter
@@ -616,6 +744,6 @@ class SearchView(ApiBaseView):
             return Response(
                 {"message": "no search query specified"}, status=400
             )
-        print("searching for: " + search_query)
+
         search_results = SearchForm().multi_search(search_query)
         return Response(search_results)

tubearchivist/config/settings.py

@@ -262,4 +262,4 @@ CORS_ALLOW_HEADERS = list(default_headers) + [
 
 # TA application settings
 TA_UPSTREAM = "https://github.com/tubearchivist/tubearchivist"
-TA_VERSION = "v0.2.4"
+TA_VERSION = "v0.3.1"

tubearchivist/home/apps.py

@@ -5,9 +5,11 @@ import sys
 
 from django.apps import AppConfig
 from home.src.es.connect import ElasticWrap
-from home.src.es.index_setup import index_check
+from home.src.es.index_setup import ElasitIndexWrap
 from home.src.es.snapshot import ElasticSnapshot
 from home.src.ta.config import AppConfig as ArchivistConfig
+from home.src.ta.config import ReleaseVersion
+from home.src.ta.helper import clear_dl_cache
 from home.src.ta.ta_redis import RedisArchivist
 
 
@@ -27,11 +29,14 @@ class StartupCheck:
         print("run startup checks")
         self.es_version_check()
         self.release_lock()
-        index_check()
         self.sync_redis_state()
         self.set_redis_conf()
+        ElasitIndexWrap().setup()
         self.make_folders()
+        clear_dl_cache(self.config_handler.config)
         self.snapshot_check()
+        self.ta_version_check()
+        self.es_set_vid_type()
         self.set_has_run()
 
     def get_has_run(self):
@@ -75,6 +80,7 @@ class StartupCheck:
             "downloading",
             "dl_queue",
             "dl_queue_id",
+            "reindex",
             "rescan",
             "run_backup",
         ]
@@ -117,6 +123,37 @@ class StartupCheck:
 
         print("elasticsearch version check passed")
 
+    def ta_version_check(self):
+        """remove key if updated now"""
+        ReleaseVersion().is_updated()
+
+    def es_set_vid_type(self):
+        """
+        update path 0.3.0 to 0.3.1, set default vid_type to video
+        fix unidentified vids in unstable
+        """
+        index_list = ["ta_video", "ta_download"]
+        data = {
+            "query": {
+                "bool": {
+                    "should": [
+                        {
+                            "bool": {
+                                "must_not": [{"exists": {"field": "vid_type"}}]
+                            }
+                        },
+                        {"term": {"vid_type": {"value": "unknown"}}},
+                    ]
+                }
+            },
+            "script": {"source": "ctx._source['vid_type'] = 'videos'"},
+        }
+
+        for index_name in index_list:
+            path = f"{index_name}/_update_by_query"
+            response, _ = ElasticWrap(path).post(data=data)
+            print(f"{index_name} vid_type index update ran: {response}")
+
 
 class HomeConfig(AppConfig):
     """call startup funcs"""

tubearchivist/home/config.json

@@ -14,7 +14,9 @@
     "subscriptions": {
         "auto_search": false,
         "auto_download": false,
-        "channel_size": 50
+        "channel_size": 50,
+        "live_channel_size": 50,
+        "shorts_channel_size": 50
     },
     "downloads": {
         "limit_count": false,
@@ -27,6 +29,8 @@
         "subtitle": false,
         "subtitle_source": false,
         "subtitle_index": false,
+        "comment_max": false,
+        "comment_sort": "top",
         "cookie_import": false,
         "throttledratelimit": false,
         "integrate_ryd": false,
@@ -47,6 +51,7 @@
         "check_reindex_days": 90,
         "thumbnail_check": {"minute": "0", "hour": "17", "day_of_week": "*"},
         "run_backup": {"minute": "0", "hour": "8", "day_of_week": "0"},
-        "run_backup_rotate": 5
+        "run_backup_rotate": 5,
+        "version_check": "rand-d"
     }
 }

tubearchivist/home/src/download/queue.py

@@ -15,8 +15,9 @@ from home.src.download.thumbnails import ThumbManager
 from home.src.download.yt_dlp_base import YtWrap
 from home.src.es.connect import ElasticWrap, IndexPaginate
 from home.src.index.playlist import YoutubePlaylist
+from home.src.index.video_constants import VideoTypeEnum
 from home.src.ta.config import AppConfig
-from home.src.ta.helper import DurationConverter
+from home.src.ta.helper import DurationConverter, is_shorts
 from home.src.ta.ta_redis import RedisArchivist
 
 
@@ -116,6 +117,12 @@ class PendingInteract:
         path = f"ta_download/_update/{self.video_id}"
         _, _ = ElasticWrap(path).post(data=data)
 
+    def get_item(self):
+        """return pending item dict"""
+        path = f"ta_download/_doc/{self.video_id}"
+        response, status_code = ElasticWrap(path).get()
+        return response["_source"], status_code
+
 
 class PendingList(PendingIndex):
     """manage the pending videos list"""
@@ -155,31 +162,40 @@ class PendingList(PendingIndex):
 
     def _process_entry(self, entry):
         """process single entry from url list"""
+        vid_type = self._get_vid_type(entry)
         if entry["type"] == "video":
-            self._add_video(entry["url"])
+            self._add_video(entry["url"], vid_type)
         elif entry["type"] == "channel":
-            self._parse_channel(entry["url"])
+            self._parse_channel(entry["url"], vid_type)
         elif entry["type"] == "playlist":
             self._parse_playlist(entry["url"])
             PlaylistSubscription().process_url_str([entry], subscribed=False)
         else:
             raise ValueError(f"invalid url_type: {entry}")
 
-    def _add_video(self, url):
+    @staticmethod
+    def _get_vid_type(entry):
+        """add vid type enum if available"""
+        vid_type_str = entry.get("vid_type")
+        if not vid_type_str:
+            return VideoTypeEnum.UNKNOWN
+
+        return VideoTypeEnum(vid_type_str)
+
+    def _add_video(self, url, vid_type):
         """add video to list"""
         if url not in self.missing_videos and url not in self.to_skip:
-            self.missing_videos.append(url)
+            self.missing_videos.append((url, vid_type))
         else:
             print(f"{url}: skipped adding already indexed video to download.")
 
-    def _parse_channel(self, url):
+    def _parse_channel(self, url, vid_type):
         """add all videos of channel to list"""
         video_results = ChannelSubscription().get_last_youtube_videos(
-            url, limit=False
+            url, limit=False, query_filter=vid_type
         )
-        youtube_ids = [i[0] for i in video_results]
-        for video_id in youtube_ids:
-            self._add_video(video_id)
+        for video_id, _, vid_type in video_results:
+            self._add_video(video_id, vid_type)
 
     def _parse_playlist(self, url):
         """add all videos of playlist to list"""
@@ -188,16 +204,17 @@ class PendingList(PendingIndex):
         video_results = playlist.json_data.get("playlist_entries")
         youtube_ids = [i["youtube_id"] for i in video_results]
         for video_id in youtube_ids:
-            self._add_video(video_id)
+            # match vid_type later
+            self._add_video(video_id, VideoTypeEnum.UNKNOWN)
 
     def add_to_pending(self, status="pending"):
         """add missing videos to pending list"""
         self.get_channels()
         bulk_list = []
 
-        for idx, youtube_id in enumerate(self.missing_videos):
-            print(f"{youtube_id}: add to download queue")
-            video_details = self.get_youtube_details(youtube_id)
+        for idx, (youtube_id, vid_type) in enumerate(self.missing_videos):
+            print(f"{youtube_id} ({vid_type}): add to download queue")
+            video_details = self.get_youtube_details(youtube_id, vid_type)
             if not video_details:
                 continue
 
@@ -235,7 +252,7 @@ class PendingList(PendingIndex):
         if idx + 1 % 25 == 0:
             print("adding to queue progress: " + progress)
 
-    def get_youtube_details(self, youtube_id):
+    def get_youtube_details(self, youtube_id, vid_type=VideoTypeEnum.VIDEOS):
         """get details from youtubedl for single pending video"""
         vid = YtWrap(self.yt_obs, self.config).extract(youtube_id)
         if not vid:
@@ -249,9 +266,30 @@ class PendingList(PendingIndex):
         if vid["live_status"] in ["is_upcoming", "is_live"]:
             return False
 
-        return self._parse_youtube_details(vid)
+        if vid["live_status"] == "was_live":
+            vid_type = VideoTypeEnum.STREAMS
+        else:
+            if self._check_shorts(vid):
+                vid_type = VideoTypeEnum.SHORTS
+            else:
+                vid_type = VideoTypeEnum.VIDEOS
 
-    def _parse_youtube_details(self, vid):
+        return self._parse_youtube_details(vid, vid_type)
+
+    @staticmethod
+    def _check_shorts(vid):
+        """check if vid is shorts video"""
+        if vid["width"] > vid["height"]:
+            return False
+
+        duration = vid.get("duration")
+        if duration and isinstance(duration, int):
+            if duration > 60:
+                return False
+
+        return is_shorts(vid["id"])
+
+    def _parse_youtube_details(self, vid, vid_type=VideoTypeEnum.VIDEOS):
         """parse response"""
         vid_id = vid.get("id")
         duration_str = DurationConverter.get_str(vid["duration"])
@@ -270,7 +308,9 @@ class PendingList(PendingIndex):
             "channel_id": vid["channel_id"],
             "duration": duration_str,
             "published": published,
-            "timestamp": int(datetime.now().strftime("%s")),
+            "timestamp": int(datetime.now().timestamp()),
+            # Pulling enum value out so it is serializable
+            "vid_type": vid_type.value,
         }
         if self.all_channels:
             youtube_details.update(

tubearchivist/home/src/download/subscriptions.py

@@ -10,6 +10,7 @@ from home.src.download.yt_dlp_base import YtWrap
 from home.src.es.connect import IndexPaginate
 from home.src.index.channel import YoutubeChannel
 from home.src.index.playlist import YoutubePlaylist
+from home.src.index.video_constants import VideoTypeEnum
 from home.src.ta.config import AppConfig
 from home.src.ta.ta_redis import RedisArchivist
 
@@ -35,23 +36,64 @@ class ChannelSubscription:
 
         return all_channels
 
-    def get_last_youtube_videos(self, channel_id, limit=True):
+    def get_last_youtube_videos(
+        self, channel_id, limit=True, query_filter=VideoTypeEnum.UNKNOWN
+    ):
         """get a list of last videos from channel"""
-        obs = {
-            "skip_download": True,
-            "extract_flat": True,
-        }
-        if limit:
-            obs["playlistend"] = self.config["subscriptions"]["channel_size"]
+        queries = self._build_queries(query_filter, limit)
 
-        url = f"https://www.youtube.com/channel/{channel_id}/videos"
-        channel = YtWrap(obs, self.config).extract(url)
-        if not channel:
-            return False
+        last_videos = []
+
+        for vid_type, limit_amount in queries:
+            obs = {
+                "skip_download": True,
+                "extract_flat": True,
+            }
+            if limit:
+                obs["playlistend"] = limit_amount
+
+            path = vid_type.value
+            channel = YtWrap(obs, self.config).extract(
+                f"https://www.youtube.com/channel/{channel_id}/{path}"
+            )
+            if not channel:
+                continue
+            last_videos.extend(
+                [(i["id"], i["title"], vid_type) for i in channel["entries"]]
+            )
 
-        last_videos = [(i["id"], i["title"]) for i in channel["entries"]]
         return last_videos
 
+    def _build_queries(self, query_filter, limit):
+        """build query list for vid_type"""
+        limit_map = {
+            "videos": self.config["subscriptions"]["channel_size"],
+            "streams": self.config["subscriptions"]["live_channel_size"],
+            "shorts": self.config["subscriptions"]["shorts_channel_size"],
+        }
+
+        queries = []
+
+        if query_filter and query_filter.value != "unknown":
+            if limit:
+                query_limit = limit_map.get(query_filter.value)
+            else:
+                query_limit = False
+
+            queries.append((query_filter, query_limit))
+
+            return queries
+
+        for query_item, default_limit in limit_map.items():
+            if limit:
+                query_limit = default_limit
+            else:
+                query_limit = False
+
+            queries.append((VideoTypeEnum(query_item), query_limit))
+
+        return queries
+
     def find_missing(self):
         """add missing videos from subscribed channels to pending"""
         all_channels = self.get_channels()
@@ -67,9 +109,9 @@ class ChannelSubscription:
             last_videos = self.get_last_youtube_videos(channel_id)
 
             if last_videos:
-                for video in last_videos:
-                    if video[0] not in pending.to_skip:
-                        missing_videos.append(video[0])
+                for video_id, _, vid_type in last_videos:
+                    if video_id not in pending.to_skip:
+                        missing_videos.append((video_id, vid_type))
             # notify
             message = {
                 "status": "message:rescan",

tubearchivist/home/src/download/yt_dlp_base.py

@@ -151,7 +151,7 @@ class CookieHandler:
         now = datetime.now()
         message = {
             "status": response,
-            "validated": int(now.strftime("%s")),
+            "validated": int(now.timestamp()),
             "validated_str": now.strftime("%Y-%m-%d %H:%M"),
         }
         RedisArchivist().set_message("cookie:valid", message)

tubearchivist/home/src/download/yt_dlp_handler.py

@@ -6,6 +6,7 @@ functionality:
 - move to archive
 """
 
+import json
 import os
 import shutil
 from datetime import datetime
@@ -15,8 +16,10 @@ from home.src.download.subscriptions import PlaylistSubscription
 from home.src.download.yt_dlp_base import CookieHandler, YtWrap
 from home.src.es.connect import ElasticWrap, IndexPaginate
 from home.src.index.channel import YoutubeChannel
+from home.src.index.comments import CommentList
 from home.src.index.playlist import YoutubePlaylist
 from home.src.index.video import YoutubeVideo, index_new_video
+from home.src.index.video_constants import VideoTypeEnum
 from home.src.ta.config import AppConfig
 from home.src.ta.helper import clean_string, ignore_filelist
 from home.src.ta.ta_redis import RedisArchivist, RedisQueue
@@ -27,7 +30,7 @@ class DownloadPostProcess:
 
     def __init__(self, download):
         self.download = download
-        self.now = int(datetime.now().strftime("%s"))
+        self.now = int(datetime.now().timestamp())
         self.pending = False
 
     def run(self):
@@ -39,6 +42,7 @@ class DownloadPostProcess:
         self.auto_delete_all()
         self.auto_delete_overwrites()
         self.validate_playlists()
+        self.get_comments()
 
     def auto_delete_all(self):
         """handle auto delete"""
@@ -100,7 +104,7 @@ class DownloadPostProcess:
                 continue
 
             # validate from local
-            playlists = channel.get_indexed_playlists()
+            playlists = channel.get_indexed_playlists(active_only=True)
             all_channel_playlist = [i["playlist_id"] for i in playlists]
             self._validate_channel_playlist(all_channel_playlist, id_c)
 
@@ -113,6 +117,7 @@ class DownloadPostProcess:
             playlist.build_json(scrape=True)
             if not playlist.json_data:
                 playlist.deactivate()
+                continue
 
             playlist.add_vids_to_playlist()
             playlist.upload_to_es()
@@ -139,6 +144,10 @@ class DownloadPostProcess:
 
         RedisArchivist().set_message(key, mess_dict, expire=expire)
 
+    def get_comments(self):
+        """get comments from youtube"""
+        CommentList(self.download.videos).index(notify=True)
+
 
 class VideoDownloader:
     """
@@ -155,22 +164,36 @@ class VideoDownloader:
         self.config = AppConfig().config
         self._build_obs()
         self.channels = set()
+        self.videos = set()
 
     def run_queue(self):
         """setup download queue in redis loop until no more items"""
         self._setup_queue()
 
-        queue = RedisQueue()
+        queue = RedisQueue(queue_name="dl_queue")
 
         limit_queue = self.config["downloads"]["limit_count"]
         if limit_queue:
             queue.trim(limit_queue - 1)
 
         while True:
-            youtube_id = queue.get_next()
-            if not youtube_id:
+            youtube_data = queue.get_next()
+            if not youtube_data:
                 break
 
+            try:
+                youtube_data = json.loads(youtube_data)
+            except json.JSONDecodeError:  # This many not be necessary
+                continue
+
+            youtube_id = youtube_data.get("youtube_id")
+
+            tmp_vid_type = youtube_data.get(
+                "vid_type", VideoTypeEnum.VIDEOS.value
+            )
+            video_type = VideoTypeEnum(tmp_vid_type)
+            print(f"Downloading type: {video_type}")
+
             success = self._dl_single_vid(youtube_id)
             if not success:
                 continue
@@ -184,9 +207,12 @@ class VideoDownloader:
             RedisArchivist().set_message(self.MSG, mess_dict, expire=60)
 
             vid_dict = index_new_video(
-                youtube_id, video_overwrites=self.video_overwrites
+                youtube_id,
+                video_overwrites=self.video_overwrites,
+                video_type=video_type,
             )
             self.channels.add(vid_dict["channel"]["channel_id"])
+            self.videos.add(vid_dict["youtube_id"])
             mess_dict = {
                 "status": self.MSG,
                 "level": "info",
@@ -237,7 +263,17 @@ class VideoDownloader:
         RedisArchivist().set_message(self.MSG, mess_dict, expire=True)
         pending = PendingList()
         pending.get_download()
-        to_add = [i["youtube_id"] for i in pending.all_pending]
+        to_add = [
+            json.dumps(
+                {
+                    "youtube_id": i["youtube_id"],
+                    # Using .value in default val to match what would be
+                    # decoded when parsing json if not set
+                    "vid_type": i.get("vid_type", VideoTypeEnum.VIDEOS.value),
+                }
+            )
+            for i in pending.all_pending
+        ]
         if not to_add:
             # there is nothing pending
             print("download queue is empty")
@@ -250,7 +286,7 @@ class VideoDownloader:
             RedisArchivist().set_message(self.MSG, mess_dict, expire=True)
             return
 
-        RedisQueue().add_list(to_add)
+        RedisQueue(queue_name="dl_queue").add_list(to_add)
 
     def _progress_hook(self, response):
         """process the progress_hooks from yt_dlp"""

tubearchivist/home/src/es/backup.py

@@ -0,0 +1,224 @@
+"""
+Functionality:
+- Handle json zip file based backup
+- create backup
+- restore backup
+"""
+
+import json
+import os
+import zipfile
+from datetime import datetime
+
+from home.src.es.connect import ElasticWrap, IndexPaginate
+from home.src.ta.config import AppConfig
+from home.src.ta.helper import get_mapping, ignore_filelist
+
+
+class ElasticBackup:
+    """dump index to nd-json files for later bulk import"""
+
+    def __init__(self, reason=False):
+        self.config = AppConfig().config
+        self.cache_dir = self.config["application"]["cache_dir"]
+        self.timestamp = datetime.now().strftime("%Y%m%d")
+        self.index_config = get_mapping()
+        self.reason = reason
+
+    def backup_all_indexes(self):
+        """backup all indexes, add reason to init"""
+        print("backup all indexes")
+        if not self.reason:
+            raise ValueError("missing backup reason in ElasticBackup")
+
+        for index in self.index_config:
+            index_name = index["index_name"]
+            print(f"backup: export in progress for {index_name}")
+            if not self.index_exists(index_name):
+                print(f"skip backup for not yet existing index {index_name}")
+                continue
+
+            self.backup_index(index_name)
+
+        self.zip_it()
+        if self.reason == "auto":
+            self.rotate_backup()
+
+    @staticmethod
+    def backup_index(index_name):
+        """export all documents of a single index"""
+        data = {
+            "query": {"match_all": {}},
+            "sort": [{"_doc": {"order": "desc"}}],
+        }
+        paginate = IndexPaginate(
+            f"ta_{index_name}", data, keep_source=True, callback=BackupCallback
+        )
+        _ = paginate.get_results()
+
+    def zip_it(self):
+        """pack it up into single zip file"""
+        file_name = f"ta_backup-{self.timestamp}-{self.reason}.zip"
+        folder = os.path.join(self.cache_dir, "backup")
+
+        to_backup = []
+        for file in os.listdir(folder):
+            if file.endswith(".json"):
+                to_backup.append(os.path.join(folder, file))
+
+        backup_file = os.path.join(folder, file_name)
+
+        comp = zipfile.ZIP_DEFLATED
+        with zipfile.ZipFile(backup_file, "w", compression=comp) as zip_f:
+            for backup_file in to_backup:
+                zip_f.write(backup_file, os.path.basename(backup_file))
+
+        # cleanup
+        for backup_file in to_backup:
+            os.remove(backup_file)
+
+    def post_bulk_restore(self, file_name):
+        """send bulk to es"""
+        file_path = os.path.join(self.cache_dir, file_name)
+        with open(file_path, "r", encoding="utf-8") as f:
+            data = f.read()
+
+        if not data.strip():
+            return
+
+        _, _ = ElasticWrap("_bulk").post(data=data, ndjson=True)
+
+    def get_all_backup_files(self):
+        """build all available backup files for view"""
+        backup_dir = os.path.join(self.cache_dir, "backup")
+        backup_files = os.listdir(backup_dir)
+        all_backup_files = ignore_filelist(backup_files)
+        all_available_backups = [
+            i
+            for i in all_backup_files
+            if i.startswith("ta_") and i.endswith(".zip")
+        ]
+        all_available_backups.sort(reverse=True)
+
+        backup_dicts = []
+        for backup_file in all_available_backups:
+            file_split = backup_file.split("-")
+            if len(file_split) == 2:
+                timestamp = file_split[1].strip(".zip")
+                reason = False
+            elif len(file_split) == 3:
+                timestamp = file_split[1]
+                reason = file_split[2].strip(".zip")
+
+            to_add = {
+                "filename": backup_file,
+                "timestamp": timestamp,
+                "reason": reason,
+            }
+            backup_dicts.append(to_add)
+
+        return backup_dicts
+
+    def restore(self, filename):
+        """
+        restore from backup zip file
+        call reset from ElasitIndexWrap first to start blank
+        """
+        zip_content = self._unpack_zip_backup(filename)
+        self._restore_json_files(zip_content)
+
+    def _unpack_zip_backup(self, filename):
+        """extract backup zip and return filelist"""
+        backup_dir = os.path.join(self.cache_dir, "backup")
+        file_path = os.path.join(backup_dir, filename)
+
+        with zipfile.ZipFile(file_path, "r") as z:
+            zip_content = z.namelist()
+            z.extractall(backup_dir)
+
+        return zip_content
+
+    def _restore_json_files(self, zip_content):
+        """go through the unpacked files and restore"""
+        backup_dir = os.path.join(self.cache_dir, "backup")
+
+        for json_f in zip_content:
+
+            file_name = os.path.join(backup_dir, json_f)
+
+            if not json_f.startswith("es_") or not json_f.endswith(".json"):
+                os.remove(file_name)
+                continue
+
+            print("restoring: " + json_f)
+            self.post_bulk_restore(file_name)
+            os.remove(file_name)
+
+    @staticmethod
+    def index_exists(index_name):
+        """check if index already exists to skip"""
+        _, status_code = ElasticWrap(f"ta_{index_name}").get()
+        exists = status_code == 200
+
+        return exists
+
+    def rotate_backup(self):
+        """delete old backups if needed"""
+        rotate = self.config["scheduler"]["run_backup_rotate"]
+        if not rotate:
+            return
+
+        all_backup_files = self.get_all_backup_files()
+        auto = [i for i in all_backup_files if i["reason"] == "auto"]
+
+        if len(auto) <= rotate:
+            print("no backup files to rotate")
+            return
+
+        backup_dir = os.path.join(self.cache_dir, "backup")
+
+        all_to_delete = auto[rotate:]
+        for to_delete in all_to_delete:
+            file_path = os.path.join(backup_dir, to_delete["filename"])
+            print(f"remove old backup file: {file_path}")
+            os.remove(file_path)
+
+
+class BackupCallback:
+    """handle backup ndjson writer as callback for IndexPaginate"""
+
+    def __init__(self, source, index_name):
+        self.source = source
+        self.index_name = index_name
+        self.timestamp = datetime.now().strftime("%Y%m%d")
+
+    def run(self):
+        """run the junk task"""
+        file_content = self._build_bulk()
+        self._write_es_json(file_content)
+
+    def _build_bulk(self):
+        """build bulk query data from all_results"""
+        bulk_list = []
+
+        for document in self.source:
+            document_id = document["_id"]
+            es_index = document["_index"]
+            action = {"index": {"_index": es_index, "_id": document_id}}
+            source = document["_source"]
+            bulk_list.append(json.dumps(action))
+            bulk_list.append(json.dumps(source))
+
+        # add last newline
+        bulk_list.append("\n")
+        file_content = "\n".join(bulk_list)
+
+        return file_content
+
+    def _write_es_json(self, file_content):
+        """write nd-json file for es _bulk API to disk"""
+        cache_dir = AppConfig().config["application"]["cache_dir"]
+        file_name = f"es_{self.index_name.lstrip('ta_')}-{self.timestamp}.json"
+        file_path = os.path.join(cache_dir, "backup", file_name)
+        with open(file_path, "a+", encoding="utf-8") as f:
+            f.write(file_content)

tubearchivist/home/src/es/connect.py

@@ -34,9 +34,11 @@ class ElasticWrap:
     def get(self, data=False):
         """get data from es"""
         if data:
-            response = requests.get(self.url, json=data, auth=self.auth)
+            response = requests.get(
+                self.url, json=data, auth=self.auth, timeout=10
+            )
         else:
-            response = requests.get(self.url, auth=self.auth)
+            response = requests.get(self.url, auth=self.auth, timeout=10)
         if not response.ok:
             print(response.text)
 
@@ -53,10 +55,16 @@ class ElasticWrap:
 
         if data:
             response = requests.post(
-                self.url, data=payload, headers=headers, auth=self.auth
+                self.url,
+                data=payload,
+                headers=headers,
+                auth=self.auth,
+                timeout=10,
             )
         else:
-            response = requests.post(self.url, headers=headers, auth=self.auth)
+            response = requests.post(
+                self.url, headers=headers, auth=self.auth, timeout=10
+            )
 
         if not response.ok:
             print(response.text)
@@ -67,7 +75,9 @@ class ElasticWrap:
         """put data to es"""
         if refresh:
             self.url = f"{self.url}/?refresh=true"
-        response = requests.put(f"{self.url}", json=data, auth=self.auth)
+        response = requests.put(
+            f"{self.url}", json=data, auth=self.auth, timeout=10
+        )
         if not response.ok:
             print(response.text)
             print(data)
@@ -80,9 +90,11 @@ class ElasticWrap:
         if refresh:
             self.url = f"{self.url}/?refresh=true"
         if data:
-            response = requests.delete(self.url, json=data, auth=self.auth)
+            response = requests.delete(
+                self.url, json=data, auth=self.auth, timeout=10
+            )
         else:
-            response = requests.delete(self.url, auth=self.auth)
+            response = requests.delete(self.url, auth=self.auth, timeout=10)
 
         if not response.ok:
             print(response.text)
@@ -125,8 +137,7 @@ class IndexPaginate:
     def validate_data(self):
         """add pit and size to data"""
         if "sort" not in self.data.keys():
-            print(self.data)
-            raise ValueError("missing sort key in data")
+            self.data.update({"sort": [{"_doc": {"order": "desc"}}]})
 
         self.data["size"] = self.size or self.DEFAULT_SIZE
         self.data["pit"] = {"id": self.pit_id, "keep_alive": "10m"}

tubearchivist/home/src/es/index_mapping.json

@@ -37,8 +37,7 @@
                     "type": "text"
                 },
                 "channel_last_refresh": {
-                    "type": "date",
-                    "format": "epoch_second"
+                    "type": "date"
                 },
                 "channel_overwrites": {
                     "properties": {
@@ -120,8 +119,7 @@
                             "type": "text"
                         },
                         "channel_last_refresh": {
-                            "type": "date",
-                            "format": "epoch_second"
+                            "type": "date"
                         },
                         "channel_overwrites": {
                             "properties": {
@@ -180,6 +178,9 @@
                 "youtube_id": {
                     "type": "keyword"
                 },
+                "vid_type": {
+                    "type": "keyword"
+                },
                 "published": {
                     "type": "date"
                 },
@@ -193,6 +194,9 @@
                         }
                     }
                 },
+                "comment_count": {
+                    "type": "long"
+                },
                 "stats" : {
                     "properties" : {
                         "average_rating" : {
@@ -323,6 +327,9 @@
                 },
                 "youtube_id": {
                     "type": "keyword"
+                },
+                "vid_type": {
+                    "type": "keyword"
                 }
             },
             "expected_set": {
@@ -460,6 +467,75 @@
                 },
                 "number_of_replicas": "0"
             }
+        },
+        {
+            "index_name": "comment",
+            "expected_map": {
+                "youtube_id": {
+                    "type": "keyword"
+                },
+                "comment_last_refresh": {
+                    "type": "date"
+                },
+                "comment_channel_id": {
+                    "type": "keyword"
+                },
+                "comment_comments": {
+                    "properties": {
+                        "comment_id": {
+                            "type": "keyword"
+                        },
+                        "comment_text": {
+                            "type" : "text"
+                        },
+                        "comment_timestamp": {
+                            "type": "date"
+                        },
+                        "comment_time_text": {
+                            "type" : "text"
+                        },
+                        "comment_likecount": {
+                            "type": "long"
+                        },
+                        "comment_is_favorited": {
+                            "type": "boolean"
+                        },
+                        "comment_author": {
+                            "type": "text",
+                            "fields": {
+                                "keyword": {
+                                    "type": "keyword",
+                                    "ignore_above": 256,
+                                    "normalizer": "to_lower"
+                                }
+                            }
+                        },
+                        "comment_author_id": {
+                            "type": "keyword"
+                        },
+                        "comment_author_thumbnail": {
+                            "type": "keyword"
+                        },
+                        "comment_author_is_uploader": {
+                            "type": "boolean"
+                        },
+                        "comment_parent": {
+                            "type": "keyword"
+                        }
+                    }
+                }
+            },
+            "expected_set": {
+                "analysis": {
+                    "normalizer": {
+                        "to_lower": {
+                            "type": "custom",
+                            "filter": ["lowercase"]
+                        }
+                    }
+                },
+                "number_of_replicas": "0"
+            }
         }
     ]
 }

tubearchivist/home/src/es/index_setup.py

@@ -5,22 +5,17 @@ functionality:
 - backup and restore metadata
 """
 
-import json
-import os
-import zipfile
-from datetime import datetime
-
-from home.src.es.connect import ElasticWrap, IndexPaginate
+from home.src.es.backup import ElasticBackup
+from home.src.es.connect import ElasticWrap
+from home.src.es.snapshot import ElasticSnapshot
 from home.src.ta.config import AppConfig
-from home.src.ta.helper import ignore_filelist
+from home.src.ta.helper import get_mapping
 
 
 class ElasticIndex:
-    """
-    handle mapping and settings on elastic search for a given index
-    """
+    """interact with a single index"""
 
-    def __init__(self, index_name, expected_map, expected_set):
+    def __init__(self, index_name, expected_map=False, expected_set=False):
         self.index_name = index_name
         self.expected_map = expected_map
         self.expected_set = expected_set
@@ -61,23 +56,23 @@ class ElasticIndex:
             if list(value.keys()) == ["properties"]:
                 for key_n, value_n in value["properties"].items():
                     if key not in now_map:
-                        print(key_n, value_n)
+                        print(f"detected mapping change: {key_n}, {value_n}")
                         return True
                     if key_n not in now_map[key]["properties"].keys():
-                        print(key_n, value_n)
+                        print(f"detected mapping change: {key_n}, {value_n}")
                         return True
                     if not value_n == now_map[key]["properties"][key_n]:
-                        print(key_n, value_n)
+                        print(f"detected mapping change: {key_n}, {value_n}")
                         return True
 
                 continue
 
             # not nested
             if key not in now_map.keys():
-                print(key, value)
+                print(f"detected mapping change: {key}, {value}")
                 return True
             if not value == now_map[key]:
-                print(key, value)
+                print(f"detected mapping change: {key}, {value}")
                 return True
 
         return False
@@ -100,6 +95,8 @@ class ElasticIndex:
 
     def rebuild_index(self):
         """rebuild with new mapping"""
+        print(f"applying new mappings to index ta_{self.index_name}...")
+        self.create_blank(for_backup=True)
         self.reindex("backup")
         self.delete_index(backup=False)
         self.create_blank()
@@ -126,279 +123,88 @@ class ElasticIndex:
 
         _, _ = ElasticWrap(path).delete()
 
-    def create_blank(self):
+    def create_blank(self, for_backup=False):
         """apply new mapping and settings for blank new index"""
+        print(f"create new blank index with name ta_{self.index_name}...")
+        path = f"ta_{self.index_name}"
+        if for_backup:
+            path = f"{path}_backup"
+
         data = {}
         if self.expected_set:
             data.update({"settings": self.expected_set})
         if self.expected_map:
             data.update({"mappings": {"properties": self.expected_map}})
 
-        _, _ = ElasticWrap(f"ta_{self.index_name}").put(data)
+        _, _ = ElasticWrap(path).put(data)
 
 
-class BackupCallback:
-    """handle backup ndjson writer as callback for IndexPaginate"""
+class ElasitIndexWrap:
+    """interact with all index mapping and setup"""
 
-    def __init__(self, source, index_name):
-        self.source = source
-        self.index_name = index_name
-        self.timestamp = datetime.now().strftime("%Y%m%d")
+    def __init__(self):
+        self.index_config = get_mapping()
+        self.backup_run = False
 
-    def run(self):
-        """run the junk task"""
-        file_content = self._build_bulk()
-        self._write_es_json(file_content)
-
-    def _build_bulk(self):
-        """build bulk query data from all_results"""
-        bulk_list = []
-
-        for document in self.source:
-            document_id = document["_id"]
-            es_index = document["_index"]
-            action = {"index": {"_index": es_index, "_id": document_id}}
-            source = document["_source"]
-            bulk_list.append(json.dumps(action))
-            bulk_list.append(json.dumps(source))
-
-        # add last newline
-        bulk_list.append("\n")
-        file_content = "\n".join(bulk_list)
-
-        return file_content
-
-    def _write_es_json(self, file_content):
-        """write nd-json file for es _bulk API to disk"""
-        cache_dir = AppConfig().config["application"]["cache_dir"]
-        file_name = f"es_{self.index_name.lstrip('ta_')}-{self.timestamp}.json"
-        file_path = os.path.join(cache_dir, "backup", file_name)
-        with open(file_path, "a+", encoding="utf-8") as f:
-            f.write(file_content)
-
-
-class ElasticBackup:
-    """dump index to nd-json files for later bulk import"""
-
-    def __init__(self, index_config, reason):
-        self.config = AppConfig().config
-        self.cache_dir = self.config["application"]["cache_dir"]
-        self.timestamp = datetime.now().strftime("%Y%m%d")
-        self.index_config = index_config
-        self.reason = reason
-
-    @staticmethod
-    def backup_index(index_name):
-        """export all documents of a single index"""
-        data = {
-            "query": {"match_all": {}},
-            "sort": [{"_doc": {"order": "desc"}}],
-        }
-        paginate = IndexPaginate(
-            f"ta_{index_name}", data, keep_source=True, callback=BackupCallback
-        )
-        _ = paginate.get_results()
-
-    def zip_it(self):
-        """pack it up into single zip file"""
-        file_name = f"ta_backup-{self.timestamp}-{self.reason}.zip"
-        folder = os.path.join(self.cache_dir, "backup")
-
-        to_backup = []
-        for file in os.listdir(folder):
-            if file.endswith(".json"):
-                to_backup.append(os.path.join(folder, file))
-
-        backup_file = os.path.join(folder, file_name)
-
-        comp = zipfile.ZIP_DEFLATED
-        with zipfile.ZipFile(backup_file, "w", compression=comp) as zip_f:
-            for backup_file in to_backup:
-                zip_f.write(backup_file, os.path.basename(backup_file))
-
-        # cleanup
-        for backup_file in to_backup:
-            os.remove(backup_file)
-
-    def post_bulk_restore(self, file_name):
-        """send bulk to es"""
-        file_path = os.path.join(self.cache_dir, file_name)
-        with open(file_path, "r", encoding="utf-8") as f:
-            data = f.read()
-
-        if not data.strip():
-            return
-
-        _, _ = ElasticWrap("_bulk").post(data=data, ndjson=True)
-
-    def get_all_backup_files(self):
-        """build all available backup files for view"""
-        backup_dir = os.path.join(self.cache_dir, "backup")
-        backup_files = os.listdir(backup_dir)
-        all_backup_files = ignore_filelist(backup_files)
-        all_available_backups = [
-            i
-            for i in all_backup_files
-            if i.startswith("ta_") and i.endswith(".zip")
-        ]
-        all_available_backups.sort(reverse=True)
-
-        backup_dicts = []
-        for backup_file in all_available_backups:
-            file_split = backup_file.split("-")
-            if len(file_split) == 2:
-                timestamp = file_split[1].strip(".zip")
-                reason = False
-            elif len(file_split) == 3:
-                timestamp = file_split[1]
-                reason = file_split[2].strip(".zip")
-
-            to_add = {
-                "filename": backup_file,
-                "timestamp": timestamp,
-                "reason": reason,
-            }
-            backup_dicts.append(to_add)
-
-        return backup_dicts
-
-    def unpack_zip_backup(self, filename):
-        """extract backup zip and return filelist"""
-        backup_dir = os.path.join(self.cache_dir, "backup")
-        file_path = os.path.join(backup_dir, filename)
-
-        with zipfile.ZipFile(file_path, "r") as z:
-            zip_content = z.namelist()
-            z.extractall(backup_dir)
-
-        return zip_content
-
-    def restore_json_files(self, zip_content):
-        """go through the unpacked files and restore"""
-        backup_dir = os.path.join(self.cache_dir, "backup")
-
-        for json_f in zip_content:
-
-            file_name = os.path.join(backup_dir, json_f)
-
-            if not json_f.startswith("es_") or not json_f.endswith(".json"):
-                os.remove(file_name)
+    def setup(self):
+        """setup elastic index, run at startup"""
+        for index in self.index_config:
+            index_name, expected_map, expected_set = self._config_split(index)
+            handler = ElasticIndex(index_name, expected_map, expected_set)
+            if not handler.exists:
+                handler.create_blank()
                 continue
 
-            print("restoring: " + json_f)
-            self.post_bulk_restore(file_name)
-            os.remove(file_name)
+            rebuild = handler.validate()
+            if rebuild:
+                self._check_backup()
+                handler.rebuild_index()
+                continue
+
+            # else all good
+            print(f"ta_{index_name} index is created and up to date...")
+
+    def reset(self):
+        """reset all indexes to blank"""
+        self.delete_all()
+        self.create_all_blank()
+
+    def delete_all(self):
+        """delete all indexes"""
+        print("reset elastic index")
+        for index in self.index_config:
+            index_name, _, _ = self._config_split(index)
+            handler = ElasticIndex(index_name)
+            handler.delete_index(backup=False)
+
+    def create_all_blank(self):
+        """create all blank indexes"""
+        print("create all new indexes in elastic from template")
+        for index in self.index_config:
+            index_name, expected_map, expected_set = self._config_split(index)
+            handler = ElasticIndex(index_name, expected_map, expected_set)
+            handler.create_blank()
 
     @staticmethod
-    def index_exists(index_name):
-        """check if index already exists to skip"""
-        _, status_code = ElasticWrap(f"ta_{index_name}").get()
-        exists = status_code == 200
-
-        return exists
-
-    def rotate_backup(self):
-        """delete old backups if needed"""
-        rotate = self.config["scheduler"]["run_backup_rotate"]
-        if not rotate:
-            return
-
-        all_backup_files = self.get_all_backup_files()
-        auto = [i for i in all_backup_files if i["reason"] == "auto"]
-
-        if len(auto) <= rotate:
-            print("no backup files to rotate")
-            return
-
-        backup_dir = os.path.join(self.cache_dir, "backup")
-
-        all_to_delete = auto[rotate:]
-        for to_delete in all_to_delete:
-            file_path = os.path.join(backup_dir, to_delete["filename"])
-            print(f"remove old backup file: {file_path}")
-            os.remove(file_path)
-
-
-def get_mapping():
-    """read index_mapping.json and get expected mapping and settings"""
-    with open("home/src/es/index_mapping.json", "r", encoding="utf-8") as f:
-        index_config = json.load(f).get("index_config")
-
-    return index_config
-
-
-def index_check(force_restore=False):
-    """check if all indexes are created and have correct mapping"""
-
-    backed_up = False
-    index_config = get_mapping()
-
-    for index in index_config:
+    def _config_split(index):
+        """split index config keys"""
         index_name = index["index_name"]
         expected_map = index["expected_map"]
         expected_set = index["expected_set"]
-        handler = ElasticIndex(index_name, expected_map, expected_set)
-        # force restore
-        if force_restore:
-            handler.delete_index(backup=False)
-            handler.create_blank()
-            continue
 
-        # create new
-        if not handler.exists:
-            print(f"create new blank index with name ta_{index_name}...")
-            handler.create_blank()
-            continue
+        return index_name, expected_map, expected_set
 
-        # validate index
-        rebuild = handler.validate()
-        if rebuild:
-            # make backup before rebuild
-            if not backed_up:
-                print("running backup first")
-                backup_all_indexes(reason="update")
-                backed_up = True
+    def _check_backup(self):
+        """create backup if needed"""
+        if self.backup_run:
+            return
 
-            print(f"applying new mappings to index ta_{index_name}...")
-            handler.rebuild_index()
-            continue
+        config = AppConfig().config
+        if config["application"]["enable_snapshot"]:
+            # take snapshot if enabled
+            ElasticSnapshot().take_snapshot_now(wait=True)
+        else:
+            # fallback to json backup
+            ElasticBackup(reason="update").backup_all_indexes()
 
-        # else all good
-        print(f"ta_{index_name} index is created and up to date...")
-
-
-def get_available_backups():
-    """return dict of available backups for settings view"""
-    index_config = get_mapping()
-    backup_handler = ElasticBackup(index_config, reason=False)
-    all_backup_files = backup_handler.get_all_backup_files()
-    return all_backup_files
-
-
-def backup_all_indexes(reason):
-    """backup all es indexes to disk"""
-    index_config = get_mapping()
-    backup_handler = ElasticBackup(index_config, reason)
-
-    for index in backup_handler.index_config:
-        index_name = index["index_name"]
-        print(f"backup: export in progress for {index_name}")
-        if not backup_handler.index_exists(index_name):
-            continue
-        backup_handler.backup_index(index_name)
-
-    backup_handler.zip_it()
-
-    if reason == "auto":
-        backup_handler.rotate_backup()
-
-
-def restore_from_backup(filename):
-    """restore indexes from backup file"""
-    # delete
-    index_check(force_restore=True)
-    # recreate
-    index_config = get_mapping()
-    backup_handler = ElasticBackup(index_config, reason=False)
-    zip_content = backup_handler.unpack_zip_backup(filename)
-    backup_handler.restore_json_files(zip_content)
+        self.backup_run = True

tubearchivist/home/src/es/snapshot.py

@@ -5,10 +5,11 @@ functionality:
 
 from datetime import datetime
 from os import environ
+from time import sleep
 from zoneinfo import ZoneInfo
 
 from home.src.es.connect import ElasticWrap
-from home.src.es.index_setup import get_mapping
+from home.src.ta.helper import get_mapping
 
 
 class ElasticSnapshot:
@@ -116,6 +117,7 @@ class ElasticSnapshot:
             "repository": self.REPO,
             "config": {
                 "indices": self.all_indices,
+                "ignore_unavailable": True,
                 "include_global_state": True,
             },
             "retention": {
@@ -133,7 +135,7 @@ class ElasticSnapshot:
             return True
 
         last_stamp = snap_dicts[0]["end_stamp"]
-        now = int(datetime.now().strftime("%s"))
+        now = int(datetime.now().timestamp())
         outdated = (now - last_stamp) / 60 / 60 > 24
         if outdated:
             print("snapshot: is outdated, create new now")
@@ -141,15 +143,42 @@ class ElasticSnapshot:
         print("snapshot: last snapshot is up-to-date")
         return outdated
 
-    def take_snapshot_now(self):
+    def take_snapshot_now(self, wait=False):
         """execute daily snapshot now"""
         path = f"_slm/policy/{self.POLICY}/_execute"
         response, statuscode = ElasticWrap(path).post()
         if statuscode == 200:
             print(f"snapshot: executing now: {response}")
 
+        if wait:
+            self._wait_for_snapshot(response["snapshot_name"])
+
         return response
 
+    def _wait_for_snapshot(self, snapshot_name):
+        """return after snapshot_name completes"""
+        path = f"_snapshot/{self.REPO}/{snapshot_name}"
+
+        while True:
+            # wait for task to be created
+            sleep(1)
+            _, statuscode = ElasticWrap(path).get()
+            if statuscode == 200:
+                break
+
+        while True:
+            # wait for snapshot success
+            response, statuscode = ElasticWrap(path).get()
+            snapshot_state = response["snapshots"][0]["state"]
+            if snapshot_state == "SUCCESS":
+                break
+
+            print(f"snapshot: {snapshot_name} in state {snapshot_state}")
+            print("snapshot: wait to complete")
+            sleep(5)
+
+        print(f"snapshot: completed - {response}")
+
     def get_snapshot_stats(self):
         """get snapshot info for frontend"""
         snapshot_info = self._build_policy_details()

tubearchivist/home/src/frontend/api_calls.py

@@ -4,20 +4,15 @@ Functionality:
 - called via user input
 """
 
-from home.src.download.queue import PendingInteract
 from home.src.download.subscriptions import (
     ChannelSubscription,
     PlaylistSubscription,
 )
-from home.src.frontend.watched import WatchState
-from home.src.index.channel import YoutubeChannel
 from home.src.index.playlist import YoutubePlaylist
-from home.src.index.video import YoutubeVideo
-from home.src.ta.helper import UrlListParser
 from home.src.ta.ta_redis import RedisArchivist, RedisQueue
+from home.src.ta.urlparser import Parser
 from home.tasks import (
     download_pending,
-    download_single,
     index_channel_playlists,
     kill_dl,
     re_sync_thumbs,
@@ -50,12 +45,9 @@ class PostData:
     def exec_map(self):
         """map dict key and return function to execute"""
         exec_map = {
-            "watched": self._watched,
-            "un_watched": self._un_watched,
             "change_view": self._change_view,
             "change_grid": self._change_grid,
             "rescan_pending": self._rescan_pending,
-            "ignore": self._ignore,
             "dl_pending": self._dl_pending,
             "queue": self._queue_handler,
             "unsubscribe": self._unsubscribe,
@@ -63,34 +55,18 @@ class PostData:
             "sort_order": self._sort_order,
             "hide_watched": self._hide_watched,
             "show_subed_only": self._show_subed_only,
-            "dlnow": self._dlnow,
             "show_ignored_only": self._show_ignored_only,
-            "forgetIgnore": self._forget_ignore,
-            "addSingle": self._add_single,
-            "deleteQueue": self._delete_queue,
             "manual-import": self._manual_import,
             "re-embed": self._re_embed,
             "db-backup": self._db_backup,
             "db-restore": self._db_restore,
             "fs-rescan": self._fs_rescan,
-            "delete-video": self._delete_video,
-            "delete-channel": self._delete_channel,
             "delete-playlist": self._delete_playlist,
             "find-playlists": self._find_playlists,
         }
 
         return exec_map[self.to_exec]
 
-    def _watched(self):
-        """mark as watched"""
-        WatchState(self.exec_val).mark_as_watched()
-        return {"success": True}
-
-    def _un_watched(self):
-        """mark as unwatched"""
-        WatchState(self.exec_val).mark_as_unwatched()
-        return {"success": True}
-
     def _change_view(self):
         """process view changes in home, channel, and downloads"""
         origin, new_view = self.exec_val.split(":")
@@ -117,15 +93,6 @@ class PostData:
         update_subscribed.delay()
         return {"success": True}
 
-    def _ignore(self):
-        """ignore from download queue"""
-        video_id = self.exec_val
-        print(f"{video_id}: ignore video from download queue")
-        PendingInteract(video_id=video_id, status="ignore").update_status()
-        # also clear from redis queue
-        RedisQueue().clear_item(video_id)
-        return {"success": True}
-
     @staticmethod
     def _dl_pending():
         """start the download queue"""
@@ -141,7 +108,7 @@ class PostData:
         to_execute = self.exec_val
         if to_execute == "stop":
             print("stopping download queue")
-            RedisQueue().clear()
+            RedisQueue(queue_name="dl_queue").clear()
         elif to_execute == "kill":
             task_id = RedisArchivist().get_message("dl_queue_id")
             if not isinstance(task_id, str):
@@ -156,7 +123,7 @@ class PostData:
         """unsubscribe from channels or playlists"""
         id_unsub = self.exec_val
         print(f"{id_unsub}: unsubscribe")
-        to_unsub_list = UrlListParser(id_unsub).process_list()
+        to_unsub_list = Parser(id_unsub).parse()
         for to_unsub in to_unsub_list:
             unsub_type = to_unsub["type"]
             unsub_id = to_unsub["url"]
@@ -209,16 +176,6 @@ class PostData:
         RedisArchivist().set_message(key, message)
         return {"success": True}
 
-    def _dlnow(self):
-        """start downloading single vid now"""
-        youtube_id = self.exec_val
-        print(f"{youtube_id}: downloading now")
-        running = download_single.delay(youtube_id=youtube_id)
-        task_id = running.id
-        print("set task id: " + task_id)
-        RedisArchivist().set_message("dl_queue_id", task_id)
-        return {"success": True}
-
     def _show_ignored_only(self):
         """switch view on /downloads/ to show ignored only"""
         show_value = self.exec_val
@@ -228,27 +185,6 @@ class PostData:
         RedisArchivist().set_message(key, value)
         return {"success": True}
 
-    def _forget_ignore(self):
-        """delete from ta_download index"""
-        video_id = self.exec_val
-        print(f"{video_id}: forget from download")
-        PendingInteract(video_id=video_id).delete_item()
-        return {"success": True}
-
-    def _add_single(self):
-        """add single youtube_id to download queue"""
-        video_id = self.exec_val
-        print(f"{video_id}: add single vid to download queue")
-        PendingInteract(video_id=video_id, status="pending").update_status()
-        return {"success": True}
-
-    def _delete_queue(self):
-        """delete download queue"""
-        status = self.exec_val
-        print("deleting from download queue: " + status)
-        PendingInteract(status=status).delete_by_status()
-        return {"success": True}
-
     @staticmethod
     def _manual_import():
         """run manual import from settings page"""
@@ -284,18 +220,6 @@ class PostData:
         rescan_filesystem.delay()
         return {"success": True}
 
-    def _delete_video(self):
-        """delete media file, metadata and thumb"""
-        youtube_id = self.exec_val
-        YoutubeVideo(youtube_id).delete_media_file()
-        return {"success": True}
-
-    def _delete_channel(self):
-        """delete channel and all matching videos"""
-        channel_id = self.exec_val
-        YoutubeChannel(channel_id).delete_channel()
-        return {"success": True}
-
     def _delete_playlist(self):
         """delete playlist, only metadata or incl all videos"""
         playlist_dict = self.exec_val

tubearchivist/home/src/frontend/forms.py

@@ -68,12 +68,6 @@ class ApplicationSettingsForm(forms.Form):
         ("1", "enable sponsorblock integration"),
     ]
 
-    CAST_CHOICES = [
-        ("", "-- change Cast integration --"),
-        ("0", "disable Cast"),
-        ("1", "enable Cast"),
-    ]
-
     SNAPSHOT_CHOICES = [
         ("", "-- change snapshot settings --"),
         ("0", "disable system snapshots"),
@@ -92,6 +86,12 @@ class ApplicationSettingsForm(forms.Form):
         ("1", "enable subtitle index"),
     ]
 
+    COMMENT_SORT_CHOICES = [
+        ("", "-- change comments sort settings --"),
+        ("top", "sort comments by top"),
+        ("new", "sort comments by new"),
+    ]
+
     COOKIE_IMPORT_CHOICES = [
         ("", "-- change cookie settings"),
         ("0", "disable cookie"),
@@ -101,6 +101,12 @@ class ApplicationSettingsForm(forms.Form):
     subscriptions_channel_size = forms.IntegerField(
         required=False, min_value=1
     )
+    subscriptions_live_channel_size = forms.IntegerField(
+        required=False, min_value=0
+    )
+    subscriptions_shorts_channel_size = forms.IntegerField(
+        required=False, min_value=0
+    )
     downloads_limit_count = forms.IntegerField(required=False)
     downloads_limit_speed = forms.IntegerField(required=False)
     downloads_throttledratelimit = forms.IntegerField(required=False)
@@ -120,6 +126,10 @@ class ApplicationSettingsForm(forms.Form):
     downloads_subtitle_index = forms.ChoiceField(
         widget=forms.Select, choices=SUBTITLE_INDEX_CHOICES, required=False
     )
+    downloads_comment_max = forms.CharField(required=False)
+    downloads_comment_sort = forms.ChoiceField(
+        widget=forms.Select, choices=COMMENT_SORT_CHOICES, required=False
+    )
     downloads_cookie_import = forms.ChoiceField(
         widget=forms.Select, choices=COOKIE_IMPORT_CHOICES, required=False
     )
@@ -129,9 +139,6 @@ class ApplicationSettingsForm(forms.Form):
     downloads_integrate_sponsorblock = forms.ChoiceField(
         widget=forms.Select, choices=SP_CHOICES, required=False
     )
-    application_enable_cast = forms.ChoiceField(
-        widget=forms.Select, choices=CAST_CHOICES, required=False
-    )
     application_enable_snapshot = forms.ChoiceField(
         widget=forms.Select, choices=SNAPSHOT_CHOICES, required=False
     )

tubearchivist/home/src/frontend/searching.py

@@ -120,7 +120,7 @@ class SearchHandler:
         if "vid_thumb_url" in hit_keys:
             youtube_id = hit["source"]["youtube_id"]
             thumb_path = ThumbManager(youtube_id).vid_thumb_path()
-            hit["source"]["vid_thumb_url"] = thumb_path
+            hit["source"]["vid_thumb_url"] = f"/cache/{thumb_path}"
 
         if "channel_last_refresh" in hit_keys:
             refreshed = hit["source"]["channel_last_refresh"]

tubearchivist/home/src/frontend/watched.py

@@ -6,101 +6,100 @@ functionality:
 from datetime import datetime
 
 from home.src.es.connect import ElasticWrap
-from home.src.ta.helper import UrlListParser
+from home.src.ta.urlparser import Parser
 
 
 class WatchState:
     """handle watched checkbox for videos and channels"""
 
-    def __init__(self, youtube_id):
+    def __init__(self, youtube_id, is_watched):
         self.youtube_id = youtube_id
-        self.stamp = int(datetime.now().strftime("%s"))
+        self.is_watched = is_watched
+        self.stamp = int(datetime.now().timestamp())
+        self.pipeline = f"_ingest/pipeline/watch_{youtube_id}"
 
-    def mark_as_watched(self):
-        """update es with new watched value"""
-        url_type = self.dedect_type()
+    def change(self):
+        """change watched state of item(s)"""
+        print(f"{self.youtube_id}: change watched state to {self.is_watched}")
+        url_type = self._dedect_type()
         if url_type == "video":
-            self.mark_vid_watched()
-        elif url_type == "channel":
-            self.mark_channel_watched()
-        elif url_type == "playlist":
-            self.mark_playlist_watched()
+            self.change_vid_state()
+            return
 
-        print(f"{self.youtube_id}: marked as watched")
+        self._add_pipeline()
+        path = f"ta_video/_update_by_query?pipeline=watch_{self.youtube_id}"
+        data = self._build_update_data(url_type)
+        _, _ = ElasticWrap(path).post(data)
+        self._delete_pipeline()
 
-    def mark_as_unwatched(self):
-        """revert watched state to false"""
-        url_type = self.dedect_type()
-        if url_type == "video":
-            self.mark_vid_watched(revert=True)
-
-        print(f"{self.youtube_id}: revert as unwatched")
-
-    def dedect_type(self):
+    def _dedect_type(self):
         """find youtube id type"""
-        print(self.youtube_id)
-        url_process = UrlListParser(self.youtube_id).process_list()
+        url_process = Parser(self.youtube_id).parse()
         url_type = url_process[0]["type"]
         return url_type
 
-    def mark_vid_watched(self, revert=False):
-        """change watched status of single video"""
+    def change_vid_state(self):
+        """change watched state of video"""
         path = f"ta_video/_update/{self.youtube_id}"
         data = {
-            "doc": {"player": {"watched": True, "watched_date": self.stamp}}
+            "doc": {
+                "player": {
+                    "watched": self.is_watched,
+                    "watched_date": self.stamp,
+                }
+            }
         }
-        if revert:
-            data["doc"]["player"]["watched"] = False
-
         response, status_code = ElasticWrap(path).post(data=data)
         if status_code != 200:
             print(response)
             raise ValueError("failed to mark video as watched")
 
-    def _get_source(self):
-        """build source line for update_by_query script"""
-        source = [
-            "ctx._source.player['watched'] = true",
-            f"ctx._source.player['watched_date'] = {self.stamp}",
-        ]
-        return "; ".join(source)
+    def _build_update_data(self, url_type):
+        """build update by query data based on url_type"""
+        term_key_map = {
+            "channel": "channel.channel_id",
+            "playlist": "playlist.keyword",
+        }
+        term_key = term_key_map.get(url_type)
 
-    def mark_channel_watched(self):
-        """change watched status of every video in channel"""
-        path = "ta_video/_update_by_query"
-        must_list = [
-            {"term": {"channel.channel_id": {"value": self.youtube_id}}},
-            {"term": {"player.watched": {"value": False}}},
-        ]
-        data = {
-            "query": {"bool": {"must": must_list}},
-            "script": {
-                "source": self._get_source(),
-                "lang": "painless",
-            },
+        return {
+            "query": {
+                "bool": {
+                    "must": [
+                        {"term": {term_key: {"value": self.youtube_id}}},
+                        {
+                            "term": {
+                                "player.watched": {
+                                    "value": not self.is_watched
+                                }
+                            }
+                        },
+                    ],
+                }
+            }
         }
 
-        response, status_code = ElasticWrap(path).post(data=data)
-        if status_code != 200:
-            print(response)
-            raise ValueError("failed mark channel as watched")
-
-    def mark_playlist_watched(self):
-        """change watched state of all videos in playlist"""
-        path = "ta_video/_update_by_query"
-        must_list = [
-            {"term": {"playlist.keyword": {"value": self.youtube_id}}},
-            {"term": {"player.watched": {"value": False}}},
-        ]
+    def _add_pipeline(self):
+        """add ingest pipeline"""
         data = {
-            "query": {"bool": {"must": must_list}},
-            "script": {
-                "source": self._get_source(),
-                "lang": "painless",
-            },
+            "description": f"{self.youtube_id}: watched {self.is_watched}",
+            "processors": [
+                {
+                    "set": {
+                        "field": "player.watched",
+                        "value": self.is_watched,
+                    }
+                },
+                {
+                    "set": {
+                        "field": "player.watched_date",
+                        "value": self.stamp,
+                    }
+                },
+            ],
         }
+        _, _ = ElasticWrap(self.pipeline).put(data)
 
-        response, status_code = ElasticWrap(path).post(data=data)
-        if status_code != 200:
-            print(response)
-            raise ValueError("failed mark playlist as watched")
+    def _delete_pipeline(self):
+        """delete pipeline"""
+        ElasticWrap(self.pipeline).delete()

tubearchivist/home/src/index/channel.py

@@ -50,7 +50,7 @@ class ChannelScraper:
         url = f"https://www.youtube.com/channel/{self.channel_id}/about?hl=en"
         cookies = {"CONSENT": "YES+xxxxxxxxxxxxxxxxxxxxxxxxxxx"}
         response = requests.get(
-            url, cookies=cookies, headers=requests_headers()
+            url, cookies=cookies, headers=requests_headers(), timeout=10
         )
         if response.ok:
             channel_page = response.text
@@ -88,7 +88,7 @@ class ChannelScraper:
         # build and return dict
         self.json_data = {
             "channel_active": True,
-            "channel_last_refresh": int(datetime.now().strftime("%s")),
+            "channel_last_refresh": int(datetime.now().timestamp()),
             "channel_subs": self._get_channel_subs(main_tab),
             "channel_name": main_tab["title"],
             "channel_banner_url": self._get_thumbnails(main_tab, "banner"),
@@ -193,6 +193,9 @@ class YoutubeChannel(YouTubeItem):
         if not self.json_data and fallback:
             self._video_fallback(fallback)
 
+        if not self.json_data:
+            return
+
         self.get_channel_art()
 
     def _video_fallback(self, fallback):
@@ -200,7 +203,7 @@ class YoutubeChannel(YouTubeItem):
         print(f"{self.youtube_id}: fallback to video metadata")
         self.json_data = {
             "channel_active": False,
-            "channel_last_refresh": int(datetime.now().strftime("%s")),
+            "channel_last_refresh": int(datetime.now().timestamp()),
             "channel_subs": fallback.get("channel_follower_count", 0),
             "channel_name": fallback["uploader"],
             "channel_banner_url": False,
@@ -275,6 +278,15 @@ class YoutubeChannel(YouTubeItem):
         }
         _, _ = ElasticWrap("ta_video/_delete_by_query").post(data)
 
+    def delete_es_comments(self):
+        """delete all comments from this channel"""
+        data = {
+            "query": {
+                "term": {"comment_channel_id": {"value": self.youtube_id}}
+            }
+        }
+        _, _ = ElasticWrap("ta_comment/_delete_by_query").post(data)
+
     def delete_playlists(self):
         """delete all indexed playlist from es"""
         all_playlists = self.get_indexed_playlists()
@@ -286,6 +298,9 @@ class YoutubeChannel(YouTubeItem):
         """delete channel and all videos"""
         print(f"{self.youtube_id}: delete channel")
         self.get_from_es()
+        if not self.json_data:
+            raise FileNotFoundError
+
         folder_path = self.get_folder_path()
         print(f"{self.youtube_id}: delete all media files")
         try:
@@ -301,6 +316,7 @@ class YoutubeChannel(YouTubeItem):
         self.delete_playlists()
         print(f"{self.youtube_id}: delete indexed videos")
         self.delete_es_videos()
+        self.delete_es_comments()
         self.del_in_es()
 
     def index_channel_playlists(self):
@@ -365,6 +381,17 @@ class YoutubeChannel(YouTubeItem):
 
         return all_youtube_ids
 
+    def get_channel_videos(self):
+        """get all videos from channel"""
+        data = {
+            "query": {
+                "term": {"channel.channel_id": {"value": self.youtube_id}}
+            },
+            "_source": ["youtube_id", "vid_type"],
+        }
+        all_videos = IndexPaginate("ta_video", data).get_results()
+        return all_videos
+
     def get_all_playlists(self):
         """get all playlists owned by this channel"""
         url = (
@@ -376,14 +403,16 @@ class YoutubeChannel(YouTubeItem):
         all_entries = [(i["id"], i["title"]) for i in playlists["entries"]]
         self.all_playlists = all_entries
 
-    def get_indexed_playlists(self):
+    def get_indexed_playlists(self, active_only=False):
         """get all indexed playlists from channel"""
-        data = {
-            "query": {
-                "term": {"playlist_channel_id": {"value": self.youtube_id}}
-            },
-            "sort": [{"playlist_channel.keyword": {"order": "desc"}}],
-        }
+        must_list = [
+            {"term": {"playlist_channel_id": {"value": self.youtube_id}}}
+        ]
+        if active_only:
+            must_list.append({"term": {"playlist_active": {"value": True}}})
+
+        data = {"query": {"bool": {"must": must_list}}}
+
         all_playlists = IndexPaginate("ta_playlist", data).get_results()
         return all_playlists
 
@@ -408,7 +437,8 @@ class YoutubeChannel(YouTubeItem):
                 to_write[key] = False
                 continue
             if value in [0, "0"]:
-                del to_write[key]
+                if key in to_write:
+                    del to_write[key]
                 continue
             if value == "1":
                 to_write[key] = True

tubearchivist/home/src/index/comments.py

@@ -0,0 +1,227 @@
+"""
+Functionality:
+- Download comments
+- Index comments in ES
+- Retrieve comments from ES
+"""
+
+from datetime import datetime
+
+from home.src.download.yt_dlp_base import YtWrap
+from home.src.es.connect import ElasticWrap
+from home.src.ta.config import AppConfig
+from home.src.ta.ta_redis import RedisArchivist
+
+
+class Comments:
+    """interact with comments per video"""
+
+    def __init__(self, youtube_id, config=False):
+        self.youtube_id = youtube_id
+        self.es_path = f"ta_comment/_doc/{youtube_id}"
+        self.json_data = False
+        self.config = config
+        self.is_activated = False
+        self.comments_format = False
+
+    def build_json(self, notify=False):
+        """build json document for es"""
+        print(f"{self.youtube_id}: get comments")
+        self.check_config()
+        if not self.is_activated:
+            return
+
+        self._send_notification(notify)
+        comments_raw, channel_id = self.get_yt_comments()
+        if not comments_raw and not channel_id:
+            return
+
+        self.format_comments(comments_raw)
+
+        self.json_data = {
+            "youtube_id": self.youtube_id,
+            "comment_last_refresh": int(datetime.now().timestamp()),
+            "comment_channel_id": channel_id,
+            "comment_comments": self.comments_format,
+        }
+
+    def check_config(self):
+        """read config if not attached"""
+        if not self.config:
+            self.config = AppConfig().config
+
+        self.is_activated = bool(self.config["downloads"]["comment_max"])
+
+    @staticmethod
+    def _send_notification(notify):
+        """send notification for download post process message"""
+        if not notify:
+            return
+
+        key = "message:download"
+        idx, total_videos = notify
+        message = {
+            "status": key,
+            "level": "info",
+            "title": "Download and index comments",
+            "message": f"Progress: {idx + 1}/{total_videos}",
+        }
+
+        RedisArchivist().set_message(key, message)
+
+    def build_yt_obs(self):
+        """
+        get extractor config
+        max-comments,max-parents,max-replies,max-replies-per-thread
+        """
+        max_comments = self.config["downloads"]["comment_max"]
+        max_comments_list = [i.strip() for i in max_comments.split(",")]
+        comment_sort = self.config["downloads"]["comment_sort"]
+
+        yt_obs = {
+            "skip_download": True,
+            "quiet": False,
+            "getcomments": True,
+            "extractor_args": {
+                "youtube": {
+                    "max_comments": max_comments_list,
+                    "comment_sort": [comment_sort],
+                }
+            },
+        }
+
+        return yt_obs
+
+    def get_yt_comments(self):
+        """get comments from youtube"""
+        yt_obs = self.build_yt_obs()
+        info_json = YtWrap(yt_obs).extract(self.youtube_id)
+        if not info_json:
+            return False, False
+
+        comments_raw = info_json.get("comments")
+        channel_id = info_json.get("channel_id")
+        return comments_raw, channel_id
+
+    def format_comments(self, comments_raw):
+        """process comments to match format"""
+        comments = []
+
+        if comments_raw:
+            for comment in comments_raw:
+                cleaned_comment = self.clean_comment(comment)
+                comments.append(cleaned_comment)
+
+        self.comments_format = comments
+
+    def clean_comment(self, comment):
+        """parse metadata from comment for indexing"""
+        time_text_datetime = datetime.utcfromtimestamp(comment["timestamp"])
+
+        if time_text_datetime.hour == 0 and time_text_datetime.minute == 0:
+            format_string = "%Y-%m-%d"
+        else:
+            format_string = "%Y-%m-%d %H:%M"
+
+        time_text = time_text_datetime.strftime(format_string)
+
+        cleaned_comment = {
+            "comment_id": comment["id"],
+            "comment_text": comment["text"].replace("\xa0", ""),
+            "comment_timestamp": comment["timestamp"],
+            "comment_time_text": time_text,
+            "comment_likecount": comment["like_count"],
+            "comment_is_favorited": comment["is_favorited"],
+            "comment_author": comment["author"],
+            "comment_author_id": comment["author_id"],
+            "comment_author_thumbnail": comment["author_thumbnail"],
+            "comment_author_is_uploader": comment["author_is_uploader"],
+            "comment_parent": comment["parent"],
+        }
+
+        return cleaned_comment
+
+    def upload_comments(self):
+        """upload comments to es"""
+        if not self.is_activated:
+            return
+
+        print(f"{self.youtube_id}: upload comments")
+        _, _ = ElasticWrap(self.es_path).put(self.json_data)
+
+        vid_path = f"ta_video/_update/{self.youtube_id}"
+        data = {"doc": {"comment_count": len(self.comments_format)}}
+        _, _ = ElasticWrap(vid_path).post(data=data)
+
+    def delete_comments(self):
+        """delete comments from es"""
+        print(f"{self.youtube_id}: delete comments")
+        _, _ = ElasticWrap(self.es_path).delete(refresh=True)
+
+    def get_es_comments(self):
+        """get comments from ES"""
+        response, statuscode = ElasticWrap(self.es_path).get()
+        if statuscode == 404:
+            print(f"comments: not found {self.youtube_id}")
+            return False
+
+        return response.get("_source")
+
+    def reindex_comments(self):
+        """update comments from youtube"""
+        self.check_config()
+        if not self.is_activated:
+            return
+
+        self.build_json()
+        if not self.json_data:
+            return
+
+        es_comments = self.get_es_comments()
+
+        if not self.comments_format:
+            return
+
+        if not self.comments_format and es_comments["comment_comments"]:
+            # don't overwrite comments in es
+            return
+
+        self.delete_comments()
+        self.upload_comments()
+
+
+class CommentList:
+    """interact with comments in group"""
+
+    def __init__(self, video_ids):
+        self.video_ids = video_ids
+        self.config = AppConfig().config
+
+    def index(self, notify=False):
+        """index group of videos"""
+        if not self.config["downloads"].get("comment_max"):
+            return
+
+        total_videos = len(self.video_ids)
+        for idx, video_id in enumerate(self.video_ids):
+            comment = Comments(video_id, config=self.config)
+            if notify:
+                notify = (idx, total_videos)
+            comment.build_json(notify=notify)
+            if comment.json_data:
+                comment.upload_comments()
+
+        if notify:
+            self.notify_final(total_videos)
+
+    @staticmethod
+    def notify_final(total_videos):
+        """send final notification"""
+        key = "message:download"
+        message = {
+            "status": key,
+            "level": "info",
+            "title": "Download and index comments finished",
+            "message": f"added comments for {total_videos} videos",
+        }
+        RedisArchivist().set_message(key, message, expire=4)

tubearchivist/home/src/index/filesystem.py

@@ -14,11 +14,10 @@ import subprocess
 from home.src.download.queue import PendingList
 from home.src.download.thumbnails import ThumbManager
 from home.src.es.connect import ElasticWrap
-from home.src.index.reindex import Reindex
+from home.src.index.comments import CommentList
 from home.src.index.video import YoutubeVideo, index_new_video
 from home.src.ta.config import AppConfig
 from home.src.ta.helper import clean_string, ignore_filelist
-from home.src.ta.ta_redis import RedisArchivist
 from PIL import Image, ImageFile
 from yt_dlp.utils import ISO639Utils
 
@@ -603,14 +602,8 @@ def scan_filesystem():
         filesystem_handler.delete_from_index()
     if filesystem_handler.to_index:
         print("index new videos")
-        for missing_vid in filesystem_handler.to_index:
-            youtube_id = missing_vid[2]
+        video_ids = [i[2] for i in filesystem_handler.to_index]
+        for youtube_id in video_ids:
             index_new_video(youtube_id)
 
-
-def reindex_old_documents():
-    """daily refresh of old documents"""
-    handler = Reindex()
-    handler.check_outdated()
-    handler.reindex()
-    RedisArchivist().set_message("last_reindex", handler.now)
+        CommentList(video_ids).index()

tubearchivist/home/src/index/generic.py

@@ -56,11 +56,11 @@ class YouTubeItem:
             "ta_channel": "channel_active",
             "ta_playlist": "playlist_active",
         }
-        update_path = f"{self.index_name}/_update/{self.youtube_id}"
+        path = f"{self.index_name}/_update/{self.youtube_id}?refresh=true"
         data = {
             "script": f"ctx._source.{key_match.get(self.index_name)} = false"
         }
-        _, _ = ElasticWrap(update_path).post(data)
+        _, _ = ElasticWrap(path).post(data)
 
     def del_in_es(self):
         """delete item from elastic search"""

tubearchivist/home/src/index/playlist.py

@@ -41,6 +41,10 @@ class YoutubePlaylist(YouTubeItem):
 
         if scrape or not self.json_data:
             self.get_from_youtube()
+            if not self.youtube_meta:
+                self.json_data = False
+                return
+
             self.process_youtube_meta()
             self.get_entries()
             self.json_data["playlist_entries"] = self.all_members
@@ -62,7 +66,7 @@ class YoutubePlaylist(YouTubeItem):
             "playlist_channel_id": self.youtube_meta["channel_id"],
             "playlist_thumbnail": playlist_thumbnail,
             "playlist_description": self.youtube_meta["description"] or False,
-            "playlist_last_refresh": int(datetime.now().strftime("%s")),
+            "playlist_last_refresh": int(datetime.now().timestamp()),
         }
 
     def get_entries(self, playlistend=False):
@@ -76,12 +80,12 @@ class YoutubePlaylist(YouTubeItem):
                 downloaded = entry["id"] in self.all_youtube_ids
             else:
                 downloaded = False
-            if not entry["uploader"]:
+            if not entry["channel"]:
                 continue
             to_append = {
                 "youtube_id": entry["id"],
                 "title": entry["title"],
-                "uploader": entry["uploader"],
+                "uploader": entry["channel"],
                 "idx": idx,
                 "downloaded": downloaded,
             }

tubearchivist/home/src/index/reindex.py

@@ -4,151 +4,254 @@ functionality:
 - index and update in es
 """
 
+import json
 import os
 import shutil
 from datetime import datetime
-from math import ceil
 from time import sleep
 
 from home.src.download.queue import PendingList
+from home.src.download.subscriptions import ChannelSubscription
 from home.src.download.thumbnails import ThumbManager
 from home.src.download.yt_dlp_base import CookieHandler
 from home.src.download.yt_dlp_handler import VideoDownloader
-from home.src.es.connect import ElasticWrap
+from home.src.es.connect import ElasticWrap, IndexPaginate
 from home.src.index.channel import YoutubeChannel
+from home.src.index.comments import Comments
 from home.src.index.playlist import YoutubePlaylist
 from home.src.index.video import YoutubeVideo
 from home.src.ta.config import AppConfig
+from home.src.ta.ta_redis import RedisArchivist, RedisQueue
 
 
-class Reindex:
-    """check for outdated documents and refresh data from youtube"""
+class ReindexBase:
+    """base config class for reindex task"""
 
-    MATCH_FIELD = {
-        "ta_video": "active",
-        "ta_channel": "channel_active",
-        "ta_playlist": "playlist_active",
+    REINDEX_CONFIG = {
+        "video": {
+            "index_name": "ta_video",
+            "queue_name": "reindex:ta_video",
+            "active_key": "active",
+            "refresh_key": "vid_last_refresh",
+        },
+        "channel": {
+            "index_name": "ta_channel",
+            "queue_name": "reindex:ta_channel",
+            "active_key": "channel_active",
+            "refresh_key": "channel_last_refresh",
+        },
+        "playlist": {
+            "index_name": "ta_playlist",
+            "queue_name": "reindex:ta_playlist",
+            "active_key": "playlist_active",
+            "refresh_key": "playlist_last_refresh",
+        },
     }
+
     MULTIPLY = 1.2
 
     def __init__(self):
-        # config
-        self.now = int(datetime.now().strftime("%s"))
         self.config = AppConfig().config
+        self.now = int(datetime.now().timestamp())
+
+    def populate(self, all_ids, reindex_config):
+        """add all to reindex ids to redis queue"""
+        if not all_ids:
+            return
+
+        RedisQueue(queue_name=reindex_config["queue_name"]).add_list(all_ids)
+
+
+class ReindexOutdated(ReindexBase):
+    """add outdated documents to reindex queue"""
+
+    def __init__(self):
+        super().__init__()
         self.interval = self.config["scheduler"]["check_reindex_days"]
-        # scan
-        self.all_youtube_ids = False
-        self.all_channel_ids = False
-        self.all_playlist_ids = False
 
-    def check_cookie(self):
-        """validate cookie if enabled"""
-        if self.config["downloads"]["cookie_import"]:
-            valid = CookieHandler(self.config).validate()
-            if not valid:
-                return
+    def add_outdated(self):
+        """add outdated documents"""
+        for reindex_config in self.REINDEX_CONFIG.values():
+            total_hits = self._get_total_hits(reindex_config)
+            daily_should = self._get_daily_should(total_hits)
+            all_ids = self._get_outdated_ids(reindex_config, daily_should)
+            self.populate(all_ids, reindex_config)
 
-    def _get_daily(self):
-        """get daily refresh values"""
-        total_videos = self._get_total_hits("ta_video")
-        video_daily = ceil(total_videos / self.interval * self.MULTIPLY)
-        if video_daily >= 10000:
-            video_daily = 9999
-
-        total_channels = self._get_total_hits("ta_channel")
-        channel_daily = ceil(total_channels / self.interval * self.MULTIPLY)
-        total_playlists = self._get_total_hits("ta_playlist")
-        playlist_daily = ceil(total_playlists / self.interval * self.MULTIPLY)
-        return (video_daily, channel_daily, playlist_daily)
-
-    def _get_total_hits(self, index):
+    @staticmethod
+    def _get_total_hits(reindex_config):
         """get total hits from index"""
-        match_field = self.MATCH_FIELD[index]
-        path = f"{index}/_search?filter_path=hits.total"
-        data = {"query": {"match": {match_field: True}}}
+        index_name = reindex_config["index_name"]
+        active_key = reindex_config["active_key"]
+        path = f"{index_name}/_search?filter_path=hits.total"
+        data = {"query": {"match": {active_key: True}}}
         response, _ = ElasticWrap(path).post(data=data)
         total_hits = response["hits"]["total"]["value"]
         return total_hits
 
-    def _get_unrated_vids(self):
-        """get max 200 videos without rating if ryd integration is enabled"""
-        must_not_list = [
-            {"exists": {"field": "stats.average_rating"}},
-            {"term": {"active": {"value": False}}},
-        ]
-        data = {"size": 200, "query": {"bool": {"must_not": must_not_list}}}
-        response, _ = ElasticWrap("ta_video/_search").get(data=data)
+    def _get_daily_should(self, total_hits):
+        """calc how many should reindex daily"""
+        daily_should = int((total_hits // self.interval + 1) * self.MULTIPLY)
+        if daily_should >= 10000:
+            daily_should = 9999
 
-        missing_rating = [i["_id"] for i in response["hits"]["hits"]]
-        self.all_youtube_ids = self.all_youtube_ids + missing_rating
+        return daily_should
 
-    def _get_outdated_vids(self, size):
-        """get daily videos to refresh"""
+    def _get_outdated_ids(self, reindex_config, daily_should):
+        """get outdated from index_name"""
+        index_name = reindex_config["index_name"]
+        refresh_key = reindex_config["refresh_key"]
         now_lte = self.now - self.interval * 24 * 60 * 60
         must_list = [
-            {"match": {"active": True}},
-            {"range": {"vid_last_refresh": {"lte": now_lte}}},
+            {"match": {reindex_config["active_key"]: True}},
+            {"range": {refresh_key: {"lte": now_lte}}},
         ]
         data = {
-            "size": size,
+            "size": daily_should,
             "query": {"bool": {"must": must_list}},
-            "sort": [{"vid_last_refresh": {"order": "asc"}}],
+            "sort": [{refresh_key: {"order": "asc"}}],
             "_source": False,
         }
-        response, _ = ElasticWrap("ta_video/_search").get(data=data)
+        response, _ = ElasticWrap(f"{index_name}/_search").get(data=data)
 
-        all_youtube_ids = [i["_id"] for i in response["hits"]["hits"]]
-        return all_youtube_ids
+        all_ids = [i["_id"] for i in response["hits"]["hits"]]
+        return all_ids
 
-    def _get_outdated_channels(self, size):
-        """get daily channels to refresh"""
-        now_lte = self.now - self.interval * 24 * 60 * 60
-        must_list = [
-            {"match": {"channel_active": True}},
-            {"range": {"channel_last_refresh": {"lte": now_lte}}},
-        ]
+
+class ReindexManual(ReindexBase):
+    """
+    manually add ids to reindex queue from API
+    data_example = {
+        "video": ["video1", "video2", "video3"],
+        "channel": ["channel1", "channel2", "channel3"],
+        "playlist": ["playlist1", "playlist2"],
+    }
+    extract_videos to also reindex all videos of channel/playlist
+    """
+
+    def __init__(self, extract_videos=False):
+        super().__init__()
+        self.extract_videos = extract_videos
+        self.data = False
+
+    def extract_data(self, data):
+        """process data"""
+        self.data = data
+        for key, values in self.data.items():
+            reindex_config = self.REINDEX_CONFIG.get(key)
+            if not reindex_config:
+                print(f"reindex type {key} not valid")
+                raise ValueError
+
+            self.process_index(reindex_config, values)
+
+    def process_index(self, index_config, values):
+        """process values per index"""
+        index_name = index_config["index_name"]
+        if index_name == "ta_video":
+            self._add_videos(values)
+        elif index_name == "ta_channel":
+            self._add_channels(values)
+        elif index_name == "ta_playlist":
+            self._add_playlists(values)
+
+    def _add_videos(self, values):
+        """add list of videos to reindex queue"""
+        if not values:
+            return
+
+        RedisQueue("reindex:ta_video").add_list(values)
+
+    def _add_channels(self, values):
+        """add list of channels to reindex queue"""
+        RedisQueue("reindex:ta_channel").add_list(values)
+
+        if self.extract_videos:
+            for channel_id in values:
+                all_videos = self._get_channel_videos(channel_id)
+                self._add_videos(all_videos)
+
+    def _add_playlists(self, values):
+        """add list of playlists to reindex queue"""
+        RedisQueue("reindex:ta_playlist").add_list(values)
+
+        if self.extract_videos:
+            for playlist_id in values:
+                all_videos = self._get_playlist_videos(playlist_id)
+                self._add_videos(all_videos)
+
+    def _get_channel_videos(self, channel_id):
+        """get all videos from channel"""
         data = {
-            "size": size,
-            "query": {"bool": {"must": must_list}},
-            "sort": [{"channel_last_refresh": {"order": "asc"}}],
-            "_source": False,
+            "query": {"term": {"channel.channel_id": {"value": channel_id}}},
+            "_source": ["youtube_id"],
         }
-        response, _ = ElasticWrap("ta_channel/_search").get(data=data)
+        all_results = IndexPaginate("ta_video", data).get_results()
+        return [i["youtube_id"] for i in all_results]
 
-        all_channel_ids = [i["_id"] for i in response["hits"]["hits"]]
-        return all_channel_ids
-
-    def _get_outdated_playlists(self, size):
-        """get daily outdated playlists to refresh"""
-        now_lte = self.now - self.interval * 24 * 60 * 60
-        must_list = [
-            {"match": {"playlist_active": True}},
-            {"range": {"playlist_last_refresh": {"lte": now_lte}}},
-        ]
+    def _get_playlist_videos(self, playlist_id):
+        """get all videos from playlist"""
         data = {
-            "size": size,
-            "query": {"bool": {"must": must_list}},
-            "sort": [{"playlist_last_refresh": {"order": "asc"}}],
-            "_source": False,
+            "query": {"term": {"playlist.keyword": {"value": playlist_id}}},
+            "_source": ["youtube_id"],
         }
-        response, _ = ElasticWrap("ta_playlist/_search").get(data=data)
+        all_results = IndexPaginate("ta_video", data).get_results()
+        return [i["youtube_id"] for i in all_results]
 
-        all_playlist_ids = [i["_id"] for i in response["hits"]["hits"]]
-        return all_playlist_ids
 
-    def check_outdated(self):
-        """add missing vids and channels"""
-        video_daily, channel_daily, playlist_daily = self._get_daily()
-        self.all_youtube_ids = self._get_outdated_vids(video_daily)
-        self.all_channel_ids = self._get_outdated_channels(channel_daily)
-        self.all_playlist_ids = self._get_outdated_playlists(playlist_daily)
+class Reindex(ReindexBase):
+    """reindex all documents from redis queue"""
 
-        integrate_ryd = self.config["downloads"]["integrate_ryd"]
-        if integrate_ryd:
-            self._get_unrated_vids()
+    def __init__(self):
+        super().__init__()
+        self.all_indexed_ids = False
 
-    @staticmethod
-    def _reindex_single_video(youtube_id):
+    def reindex_all(self):
+        """reindex all in queue"""
+        if not self.cookie_is_valid():
+            print("[reindex] cookie invalid, exiting...")
+            return
+
+        for index_config in self.REINDEX_CONFIG.values():
+            if not RedisQueue(index_config["queue_name"]).has_item():
+                continue
+
+            while True:
+                has_next = self.reindex_index(index_config)
+                if not has_next:
+                    break
+
+        RedisArchivist().set_message("last_reindex", self.now)
+
+    def reindex_index(self, index_config):
+        """reindex all of a single index"""
+        reindex = self.get_reindex_map(index_config["index_name"])
+        youtube_id = RedisQueue(index_config["queue_name"]).get_next()
+        if youtube_id:
+            reindex(youtube_id)
+            sleep_interval = self.config["downloads"].get("sleep_interval", 0)
+            sleep(sleep_interval)
+
+        return bool(youtube_id)
+
+    def get_reindex_map(self, index_name):
+        """return def to run for index"""
+        def_map = {
+            "ta_video": self._reindex_single_video,
+            "ta_channel": self._reindex_single_channel,
+            "ta_playlist": self._reindex_single_playlist,
+        }
+
+        return def_map.get(index_name)
+
+    def _reindex_single_video(self, youtube_id):
+        """wrapper to handle channel name changes"""
+        try:
+            self._reindex_single_video_call(youtube_id)
+        except FileNotFoundError:
+            ChannelUrlFixer(youtube_id, self.config).run()
+            self._reindex_single_video_call(youtube_id)
+
+    def _reindex_single_video_call(self, youtube_id):
         """refresh data for single video"""
         video = YoutubeVideo(youtube_id)
 
@@ -182,6 +285,8 @@ class Reindex:
         thumb_handler.delete_video_thumb()
         thumb_handler.download_video_thumb(video.json_data["vid_thumb_url"])
 
+        Comments(youtube_id, config=self.config).reindex_comments()
+
         return
 
     @staticmethod
@@ -194,6 +299,8 @@ class Reindex:
         channel.get_from_youtube()
         if not channel.json_data:
             channel.deactivate()
+            channel.get_from_es()
+            channel.sync_to_videos()
             return
 
         channel.json_data["channel_subscribed"] = subscribed
@@ -202,13 +309,15 @@ class Reindex:
         channel.upload_to_es()
         channel.sync_to_videos()
 
-    @staticmethod
-    def _reindex_single_playlist(playlist_id, all_indexed_ids):
+        ChannelFullScan(channel_id).scan()
+
+    def _reindex_single_playlist(self, playlist_id):
         """refresh playlist data"""
+        self._get_all_videos()
         playlist = YoutubePlaylist(playlist_id)
         playlist.get_from_es()
         subscribed = playlist.json_data["playlist_subscribed"]
-        playlist.all_youtube_ids = all_indexed_ids
+        playlist.all_youtube_ids = self.all_indexed_ids
         playlist.build_json(scrape=True)
         if not playlist.json_data:
             playlist.deactivate()
@@ -218,37 +327,98 @@ class Reindex:
         playlist.upload_to_es()
         return
 
-    def reindex(self):
-        """reindex what's needed"""
-        sleep_interval = self.config["downloads"]["sleep_interval"]
-        # videos
-        print(f"reindexing {len(self.all_youtube_ids)} videos")
-        for youtube_id in self.all_youtube_ids:
-            try:
-                self._reindex_single_video(youtube_id)
-            except FileNotFoundError:
-                # handle channel name change here
-                ChannelUrlFixer(youtube_id, self.config).run()
-                self._reindex_single_video(youtube_id)
-            if sleep_interval:
-                sleep(sleep_interval)
-        # channels
-        print(f"reindexing {len(self.all_channel_ids)} channels")
-        for channel_id in self.all_channel_ids:
-            self._reindex_single_channel(channel_id)
-            if sleep_interval:
-                sleep(sleep_interval)
-        # playlist
-        print(f"reindexing {len(self.all_playlist_ids)} playlists")
-        if self.all_playlist_ids:
-            handler = PendingList()
-            handler.get_download()
-            handler.get_indexed()
-            all_indexed_ids = [i["youtube_id"] for i in handler.all_videos]
-            for playlist_id in self.all_playlist_ids:
-                self._reindex_single_playlist(playlist_id, all_indexed_ids)
-                if sleep_interval:
-                    sleep(sleep_interval)
+    def _get_all_videos(self):
+        """add all videos for playlist index validation"""
+        if self.all_indexed_ids:
+            return
+
+        handler = PendingList()
+        handler.get_download()
+        handler.get_indexed()
+        self.all_indexed_ids = [i["youtube_id"] for i in handler.all_videos]
+
+    def cookie_is_valid(self):
+        """return true if cookie is enabled and valid"""
+        if not self.config["downloads"]["cookie_import"]:
+            # is not activated, continue reindex
+            return True
+
+        valid = CookieHandler(self.config).validate()
+        return valid
+
+
+class ReindexProgress(ReindexBase):
+    """
+    get progress of reindex task
+    request_type: key of self.REINDEX_CONFIG
+    request_id: id of request_type
+    return = {
+        "state": "running" | "queued" | False
+        "total_queued": int
+        "in_queue_name": "queue_name"
+    }
+    """
+
+    def __init__(self, request_type=False, request_id=False):
+        super().__init__()
+        self.request_type = request_type
+        self.request_id = request_id
+
+    def get_progress(self):
+        """get progress from task"""
+        queue_name, request_type = self._get_queue_name()
+        total = self._get_total_in_queue(queue_name)
+
+        progress = {
+            "total_queued": total,
+            "type": request_type,
+        }
+        state = self._get_state(total, queue_name)
+        progress.update(state)
+
+        return progress
+
+    def _get_queue_name(self):
+        """return queue_name, queue_type, raise exception on error"""
+        if not self.request_type:
+            return "all", "all"
+
+        reindex_config = self.REINDEX_CONFIG.get(self.request_type)
+        if not reindex_config:
+            print(f"reindex_config not found: {self.request_type}")
+            raise ValueError
+
+        return reindex_config["queue_name"], self.request_type
+
+    def _get_total_in_queue(self, queue_name):
+        """get all items in queue"""
+        total = 0
+        if queue_name == "all":
+            queues = [i["queue_name"] for i in self.REINDEX_CONFIG.values()]
+            for queue in queues:
+                total += len(RedisQueue(queue).get_all())
+        else:
+            total += len(RedisQueue(queue_name).get_all())
+
+        return total
+
+    def _get_state(self, total, queue_name):
+        """get state based on request_id"""
+        state_dict = {}
+        if self.request_id:
+            state = RedisQueue(queue_name).in_queue(self.request_id)
+            state_dict.update({"id": self.request_id, "state": state})
+
+            return state_dict
+
+        if total:
+            state = "running"
+        else:
+            state = "empty"
+
+        state_dict.update({"state": state})
+
+        return state_dict
 
 
 class ChannelUrlFixer:
@@ -308,3 +478,74 @@ class ChannelUrlFixer:
         shutil.move(video_path_is, new_path, copy_function=shutil.copyfile)
         VideoDownloader().move_to_archive(self.video.json_data)
         self.video.update_media_url()
+
+
+class ChannelFullScan:
+    """
+    update from v0.3.0 to v0.3.1
+    full scan of channel to fix vid_type mismatch
+    """
+
+    def __init__(self, channel_id):
+        self.channel_id = channel_id
+        self.to_update = False
+
+    def scan(self):
+        """match local with remote"""
+        print(f"{self.channel_id}: start full scan")
+        all_local_videos = self._get_all_local()
+        all_remote_videos = self._get_all_remote()
+        self.to_update = []
+        for video in all_local_videos:
+            video_id = video["youtube_id"]
+            remote_match = [i for i in all_remote_videos if i[0] == video_id]
+            if not remote_match:
+                print(f"{video_id}: no remote match found")
+                continue
+
+            expected_type = remote_match[0][-1].value
+            if video["vid_type"] != expected_type:
+                self.to_update.append(
+                    {
+                        "video_id": video_id,
+                        "vid_type": expected_type,
+                    }
+                )
+
+        self.update()
+
+    def _get_all_remote(self):
+        """get all channel videos"""
+        sub = ChannelSubscription()
+        all_remote_videos = sub.get_last_youtube_videos(
+            self.channel_id, limit=False
+        )
+
+        return all_remote_videos
+
+    def _get_all_local(self):
+        """get all local indexed channel_videos"""
+        channel = YoutubeChannel(self.channel_id)
+        all_local_videos = channel.get_channel_videos()
+
+        return all_local_videos
+
+    def update(self):
+        """build bulk query for updates"""
+        if not self.to_update:
+            print(f"{self.channel_id}: nothing to update")
+            return
+
+        print(f"{self.channel_id}: fixing {len(self.to_update)} videos")
+        bulk_list = []
+        for video in self.to_update:
+            action = {
+                "update": {"_id": video.get("video_id"), "_index": "ta_video"}
+            }
+            source = {"doc": {"vid_type": video.get("vid_type")}}
+            bulk_list.append(json.dumps(action))
+            bulk_list.append(json.dumps(source))
+        # add last newline
+        bulk_list.append("\n")
+        data = "\n".join(bulk_list)
+        _, _ = ElasticWrap("_bulk").post(data=data, ndjson=True)

tubearchivist/home/src/index/subtitle.py

@@ -285,7 +285,7 @@ class SubtitleParser:
             "title": video.json_data.get("title"),
             "subtitle_channel": channel.get("channel_name"),
             "subtitle_channel_id": channel.get("channel_id"),
-            "subtitle_last_refresh": int(datetime.now().strftime("%s")),
+            "subtitle_last_refresh": int(datetime.now().timestamp()),
             "subtitle_lang": self.lang,
             "subtitle_source": source,
         }

tubearchivist/home/src/index/video.py

@@ -11,9 +11,11 @@ import requests
 from django.conf import settings
 from home.src.es.connect import ElasticWrap
 from home.src.index import channel as ta_channel
+from home.src.index import comments as ta_comments
 from home.src.index import playlist as ta_playlist
 from home.src.index.generic import YouTubeItem
 from home.src.index.subtitle import YoutubeSubtitle
+from home.src.index.video_constants import VideoTypeEnum
 from home.src.ta.helper import DurationConverter, clean_string, randomizor
 from home.src.ta.ta_redis import RedisArchivist
 from ryd_client import ryd_client
@@ -27,7 +29,7 @@ class SponsorBlock:
     def __init__(self, user_id=False):
         self.user_id = user_id
         self.user_agent = f"{settings.TA_UPSTREAM} {settings.TA_VERSION}"
-        self.last_refresh = int(datetime.now().strftime("%s"))
+        self.last_refresh = int(datetime.now().timestamp())
 
     def get_sb_id(self):
         """get sponsorblock userid or generate if needed"""
@@ -72,20 +74,14 @@ class SponsorBlock:
 
     def _get_sponsor_dict(self, all_segments):
         """format and process response"""
-        has_unlocked = False
-        cleaned_segments = []
-        for segment in all_segments:
-            if not segment["locked"]:
-                has_unlocked = True
-            del segment["userID"]
-            del segment["description"]
-            cleaned_segments.append(segment)
+        _ = [i.pop("description", None) for i in all_segments]
+        has_unlocked = not any(i.get("locked") for i in all_segments)
 
         sponsor_dict = {
             "last_refresh": self.last_refresh,
             "has_unlocked": has_unlocked,
             "is_enabled": True,
-            "segments": cleaned_segments,
+            "segments": all_segments,
         }
         return sponsor_dict
 
@@ -128,10 +124,16 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
     index_name = "ta_video"
     yt_base = "https://www.youtube.com/watch?v="
 
-    def __init__(self, youtube_id, video_overwrites=False):
+    def __init__(
+        self,
+        youtube_id,
+        video_overwrites=False,
+        video_type=VideoTypeEnum.VIDEOS,
+    ):
         super().__init__(youtube_id)
         self.channel_id = False
         self.video_overwrites = video_overwrites
+        self.video_type = video_type
         self.es_path = f"{self.index_name}/_doc/{youtube_id}"
         self.offline_import = False
 
@@ -179,7 +181,7 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
         upload_date = self.youtube_meta["upload_date"]
         upload_date_time = datetime.strptime(upload_date, "%Y%m%d")
         published = upload_date_time.strftime("%Y-%m-%d")
-        last_refresh = int(datetime.now().strftime("%s"))
+        last_refresh = int(datetime.now().timestamp())
         # base64_blur = ThumbManager().get_base64_blur(self.youtube_id)
         base64_blur = False
         # build json_data basics
@@ -194,6 +196,8 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
             "vid_last_refresh": last_refresh,
             "date_downloaded": last_refresh,
             "youtube_id": self.youtube_id,
+            # Using .value to make json encodable
+            "vid_type": self.video_type.value,
             "active": True,
         }
 
@@ -291,6 +295,9 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
         """delete video file, meta data"""
         print(f"{self.youtube_id}: delete video")
         self.get_from_es()
+        if not self.json_data:
+            raise FileNotFoundError
+
         video_base = self.app_conf["videos"]
         media_url = self.json_data.get("media_url")
         file_path = os.path.join(video_base, media_url)
@@ -302,6 +309,7 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
         self.del_in_playlists()
         self.del_in_es()
         self.delete_subtitles()
+        self.delete_comments()
 
     def del_in_playlists(self):
         """remove downloaded in playlist"""
@@ -326,6 +334,13 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
         print(f"{self.youtube_id}: delete subtitles")
         YoutubeSubtitle(self).delete(subtitles=subtitles)
 
+    def delete_comments(self):
+        """delete comments from es"""
+        comments = ta_comments.Comments(self.youtube_id, config=self.config)
+        comments.check_config()
+        if comments.is_activated:
+            comments.delete_comments()
+
     def _get_ryd_stats(self):
         """get optional stats from returnyoutubedislikeapi.com"""
         # pylint: disable=broad-except
@@ -390,9 +405,13 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
         _, _ = ElasticWrap(path).post(data=data)
 
 
-def index_new_video(youtube_id, video_overwrites=False):
+def index_new_video(
+    youtube_id, video_overwrites=False, video_type=VideoTypeEnum.VIDEOS
+):
     """combined classes to create new video in index"""
-    video = YoutubeVideo(youtube_id, video_overwrites=video_overwrites)
+    video = YoutubeVideo(
+        youtube_id, video_overwrites=video_overwrites, video_type=video_type
+    )
     video.build_json()
     if not video.json_data:
         raise ValueError("failed to get metadata for " + youtube_id)

tubearchivist/home/src/index/video_constants.py

@@ -0,0 +1,12 @@
+"""video constants"""
+
+import enum
+
+
+class VideoTypeEnum(enum.Enum):
+    """all vid_type fields"""
+
+    VIDEOS = "videos"
+    STREAMS = "streams"
+    SHORTS = "shorts"
+    UNKNOWN = "unknown"

tubearchivist/home/src/ta/config.py

@@ -7,8 +7,11 @@ Functionality:
 import json
 import os
 import re
+from random import randint
 
+import requests
 from celery.schedules import crontab
+from django.conf import settings
 from home.src.ta.ta_redis import RedisArchivist
 
 
@@ -47,18 +50,6 @@ class AppConfig:
     @staticmethod
     def get_config_env():
         """read environment application variables"""
-        host_uid_env = os.environ.get("HOST_UID")
-        if host_uid_env:
-            host_uid = int(host_uid_env)
-        else:
-            host_uid = False
-
-        host_gid_env = os.environ.get("HOST_GID")
-        if host_gid_env:
-            host_gid = int(host_gid_env)
-        else:
-            host_gid = False
-
         es_pass = os.environ.get("ELASTIC_PASSWORD")
         es_user = os.environ.get("ELASTIC_USER", default="elastic")
 
@@ -66,8 +57,9 @@ class AppConfig:
             "REDIS_HOST": os.environ.get("REDIS_HOST"),
             "es_url": os.environ.get("ES_URL"),
             "es_auth": (es_user, es_pass),
-            "HOST_UID": host_uid,
-            "HOST_GID": host_gid,
+            "HOST_UID": int(os.environ.get("HOST_UID", False)),
+            "HOST_GID": int(os.environ.get("HOST_GID", False)),
+            "enable_cast": bool(os.environ.get("ENABLE_CAST")),
         }
 
         return application
@@ -126,6 +118,15 @@ class AppConfig:
         self.config["application"]["colors"] = colors
         return colors
 
+    @staticmethod
+    def _build_rand_daily():
+        """build random daily schedule per installation"""
+        return {
+            "minute": randint(0, 59),
+            "hour": randint(0, 23),
+            "day_of_week": "*",
+        }
+
     def load_new_defaults(self):
         """check config.json for missing defaults"""
         default_config = self.get_config_file()
@@ -149,6 +150,9 @@ class AppConfig:
             # missing nested values
             for sub_key, sub_value in value.items():
                 if sub_key not in redis_config[key].keys():
+                    if sub_value == "rand-d":
+                        sub_value = self._build_rand_daily()
+
                     redis_config[key].update({sub_key: sub_value})
                     needs_update = True
 
@@ -165,6 +169,7 @@ class ScheduleBuilder:
         "check_reindex": "0 12 *",
         "thumbnail_check": "0 17 *",
         "run_backup": "0 18 0",
+        "version_check": "0 11 *",
     }
     CONFIG = ["check_reindex_days", "run_backup_rotate"]
     MSG = "message:setting"
@@ -264,18 +269,88 @@ class ScheduleBuilder:
             if not item_conf:
                 continue
 
-            minute = item_conf["minute"]
-            hour = item_conf["hour"]
-            day_of_week = item_conf["day_of_week"]
-            schedule_name = f"schedule_{schedule_item}"
-            to_add = {
-                schedule_name: {
-                    "task": schedule_item,
-                    "schedule": crontab(
-                        minute=minute, hour=hour, day_of_week=day_of_week
-                    ),
+            schedule_dict.update(
+                {
+                    f"schedule_{schedule_item}": {
+                        "task": schedule_item,
+                        "schedule": crontab(
+                            minute=item_conf["minute"],
+                            hour=item_conf["hour"],
+                            day_of_week=item_conf["day_of_week"],
+                        ),
+                    }
                 }
-            }
-            schedule_dict.update(to_add)
+            )
 
         return schedule_dict
+
+
+class ReleaseVersion:
+    """compare local version with remote version"""
+
+    REMOTE_URL = "https://www.tubearchivist.com/api/release/latest/"
+    NEW_KEY = "versioncheck:new"
+
+    def __init__(self):
+        self.local_version = self._parse_version(settings.TA_VERSION)
+        self.is_unstable = settings.TA_VERSION.endswith("-unstable")
+        self.remote_version = False
+        self.is_breaking = False
+        self.response = False
+
+    def check(self):
+        """check version"""
+        print(f"[{self.local_version}]: look for updates")
+        self.get_remote_version()
+        new_version, is_breaking = self._has_update()
+        if new_version:
+            message = {
+                "status": True,
+                "version": new_version,
+                "is_breaking": is_breaking,
+            }
+            RedisArchivist().set_message(self.NEW_KEY, message)
+            print(f"[{self.local_version}]: found new version {new_version}")
+
+    def get_remote_version(self):
+        """read version from remote"""
+        self.response = requests.get(self.REMOTE_URL, timeout=20).json()
+        remote_version_str = self.response["release_version"]
+        self.remote_version = self._parse_version(remote_version_str)
+        self.is_breaking = self.response["breaking_changes"]
+
+    def _has_update(self):
+        """check if there is an update"""
+        for idx, number in enumerate(self.local_version):
+            is_newer = self.remote_version[idx] > number
+            if is_newer:
+                return self.response["release_version"], self.is_breaking
+
+        if self.is_unstable and self.local_version == self.remote_version:
+            return self.response["release_version"], self.is_breaking
+
+        return False, False
+
+    @staticmethod
+    def _parse_version(version):
+        """return version parts"""
+        clean = version.rstrip("-unstable").lstrip("v")
+        return tuple((int(i) for i in clean.split(".")))
+
+    def is_updated(self):
+        """check if update happened in the mean time"""
+        message = self.get_update()
+        if not message:
+            return
+
+        if self._parse_version(message.get("version")) == self.local_version:
+            print(f"[{self.local_version}]: update completed")
+            RedisArchivist().del_message(self.NEW_KEY)
+
+    def get_update(self):
+        """return new version dict if available"""
+        message = RedisArchivist().get_message(self.NEW_KEY)
+        if not message.get("status"):
+            return False
+
+        return message

tubearchivist/home/src/ta/helper.py

@@ -3,15 +3,16 @@ Loose collection of helper functions
 - don't import AppConfig class here to avoid circular imports
 """
 
+import json
+import os
 import random
 import re
 import string
 import subprocess
 import unicodedata
 from datetime import datetime
-from urllib.parse import parse_qs, urlparse
 
-from home.src.download.yt_dlp_base import YtWrap
+import requests
 
 
 def clean_string(file_name):
@@ -117,104 +118,31 @@ def time_parser(timestamp):
     return int(hours) * 60 * 60 + int(minutes) * 60 + float(seconds)
 
 
-class UrlListParser:
-    """take a multi line string and detect valid youtube ids"""
+def clear_dl_cache(config):
+    """clear leftover files from dl cache"""
+    print("clear download cache")
+    cache_dir = os.path.join(config["application"]["cache_dir"], "download")
+    for cached in os.listdir(cache_dir):
+        to_delete = os.path.join(cache_dir, cached)
+        os.remove(to_delete)
 
-    def __init__(self, url_str):
-        self.url_list = [i.strip() for i in url_str.split()]
 
-    def process_list(self):
-        """loop through the list"""
-        youtube_ids = []
-        for url in self.url_list:
-            parsed = urlparse(url)
-            print(f"processing: {url}")
-            print(parsed)
-            if not parsed.netloc:
-                # is not a url
-                id_type = self.find_valid_id(url)
-                youtube_id = url
-            elif "youtube.com" not in url and "youtu.be" not in url:
-                raise ValueError(f"{url} is not a youtube link")
-            elif parsed.path:
-                # is a url
-                youtube_id, id_type = self.detect_from_url(parsed)
-            else:
-                # not detected
-                raise ValueError(f"failed to detect {url}")
+def get_mapping():
+    """read index_mapping.json and get expected mapping and settings"""
+    with open("home/src/es/index_mapping.json", "r", encoding="utf-8") as f:
+        index_config = json.load(f).get("index_config")
 
-            youtube_ids.append({"url": youtube_id, "type": id_type})
+    return index_config
 
-        return youtube_ids
 
-    def detect_from_url(self, parsed):
-        """detect from parsed url"""
-        if parsed.netloc == "youtu.be":
-            # shortened
-            youtube_id = parsed.path.strip("/")
-            _ = self.find_valid_id(youtube_id)
-            return youtube_id, "video"
+def is_shorts(youtube_id):
+    """check if youtube_id is a shorts video, bot not it it's not a shorts"""
+    shorts_url = f"https://www.youtube.com/shorts/{youtube_id}"
+    response = requests.head(
+        shorts_url, headers=requests_headers(), timeout=10
+    )
 
-        if parsed.query:
-            # detect from query string
-            query_parsed = parse_qs(parsed.query)
-            if "v" in query_parsed:
-                youtube_id = query_parsed["v"][0]
-                _ = self.find_valid_id(youtube_id)
-                return youtube_id, "video"
-
-            if "list" in query_parsed:
-                youtube_id = query_parsed["list"][0]
-                return youtube_id, "playlist"
-
-        if parsed.path.startswith("/channel/"):
-            # channel id in url
-            youtube_id = parsed.path.split("/")[2]
-            _ = self.find_valid_id(youtube_id)
-            return youtube_id, "channel"
-
-        # detect channel with yt_dlp
-        youtube_id = self.extract_channel_name(parsed.geturl())
-        return youtube_id, "channel"
-
-    @staticmethod
-    def find_valid_id(id_str):
-        """detect valid id from length of string"""
-        str_len = len(id_str)
-        if str_len == 11:
-            id_type = "video"
-        elif str_len == 24:
-            id_type = "channel"
-        elif str_len in [34, 18] or id_str in ["LL", "WL"]:
-            id_type = "playlist"
-        else:
-            # unable to parse
-            raise ValueError("not a valid id_str: " + id_str)
-
-        return id_type
-
-    @staticmethod
-    def extract_channel_name(url):
-        """find channel id from channel name with yt-dlp help"""
-        obs_request = {
-            "skip_download": True,
-            "extract_flat": True,
-            "playlistend": 0,
-        }
-        url_info = YtWrap(obs_request).extract(url)
-        channel_id = url_info.get("channel_id", False)
-        if channel_id:
-            return channel_id
-
-        url = url_info.get("url", False)
-        if url:
-            # handle old channel name redirect with url path split
-            channel_id = urlparse(url).path.strip("/").split("/")[1]
-
-            return channel_id
-
-        print(f"failed to extract channel id from {url}")
-        raise ValueError
+    return response.status_code == 200
 
 
 class DurationConverter:

tubearchivist/home/src/ta/ta_redis.py

@@ -102,11 +102,11 @@ class RedisArchivist(RedisBase):
 
 
 class RedisQueue(RedisBase):
-    """dynamically interact with the download queue in redis"""
+    """dynamically interact with queues in redis"""
 
-    def __init__(self):
+    def __init__(self, queue_name):
         super().__init__()
-        self.key = self.NAME_SPACE + "dl_queue"
+        self.key = f"{self.NAME_SPACE}{queue_name}"
 
     def get_all(self):
         """return all elements in list"""
@@ -114,6 +114,14 @@ class RedisQueue(RedisBase):
         all_elements = [i.decode() for i in result]
         return all_elements
 
+    def in_queue(self, element):
+        """check if element is in list"""
+        result = self.conn.execute_command("LPOS", self.key, element)
+        if result is not None:
+            return "in_queue"
+
+        return False
+
     def add_list(self, to_add):
         """add list to queue"""
         self.conn.execute_command("RPUSH", self.key, *to_add)

tubearchivist/home/src/ta/urlparser.py

@@ -0,0 +1,133 @@
+"""
+Functionality:
+- detect valid youtube ids and links from multi line string
+- identify vid_type if possible
+"""
+
+from urllib.parse import parse_qs, urlparse
+
+from home.src.download.yt_dlp_base import YtWrap
+from home.src.index.video_constants import VideoTypeEnum
+
+
+class Parser:
+    """take a multi line string and detect valid youtube ids"""
+
+    def __init__(self, url_str):
+        self.url_list = [i.strip() for i in url_str.split()]
+
+    def parse(self):
+        """parse the list"""
+        ids = []
+        for url in self.url_list:
+            parsed = urlparse(url)
+            if parsed.netloc:
+                # is url
+                identified = self.process_url(parsed)
+            else:
+                # is not url
+                identified = self._find_valid_id(url)
+
+            if "vid_type" not in identified:
+                identified.update(self._detect_vid_type(parsed.path))
+
+            ids.append(identified)
+
+        return ids
+
+    def process_url(self, parsed):
+        """process as url"""
+        if parsed.netloc == "youtu.be":
+            # shortened
+            youtube_id = parsed.path.strip("/")
+            return self._validate_expected(youtube_id, "video")
+
+        query_parsed = parse_qs(parsed.query)
+        if "v" in query_parsed:
+            # video from v query str
+            youtube_id = query_parsed["v"][0]
+            return self._validate_expected(youtube_id, "video")
+
+        if "list" in query_parsed:
+            # playlist from list query str
+            youtube_id = query_parsed["list"][0]
+            return self._validate_expected(youtube_id, "playlist")
+
+        all_paths = parsed.path.strip("/").split("/")
+        if all_paths[0] == "shorts":
+            # is shorts video
+            item = self._validate_expected(all_paths[1], "video")
+            item.update({"vid_type": VideoTypeEnum.SHORTS.value})
+            return item
+
+        if all_paths[0] == "channel":
+            return self._validate_expected(all_paths[1], "channel")
+
+        # detect channel
+        channel_id = self._extract_channel_name(parsed.geturl())
+        return {"type": "channel", "url": channel_id}
+
+    def _validate_expected(self, youtube_id, expected_type):
+        """raise value error if not matching"""
+        matched = self._find_valid_id(youtube_id)
+        if matched["type"] != expected_type:
+            raise ValueError(
+                f"{youtube_id} not of expected type {expected_type}"
+            )
+
+        return {"type": expected_type, "url": youtube_id}
+
+    def _find_valid_id(self, id_str):
+        """detect valid id from length of string"""
+        if id_str in ("LL", "WL"):
+            return {"type": "playlist", "url": id_str}
+
+        if id_str.startswith("@"):
+            url = f"https://www.youtube.com/{id_str}"
+            channel_id = self._extract_channel_name(url)
+            return {"type": "channel", "url": channel_id}
+
+        len_id_str = len(id_str)
+        if len_id_str == 11:
+            item_type = "video"
+        elif len_id_str == 24:
+            item_type = "channel"
+        elif len_id_str in (34, 18):
+            item_type = "playlist"
+        else:
+            raise ValueError(f"not a valid id_str: {id_str}")
+
+        return {"type": item_type, "url": id_str}
+
+    @staticmethod
+    def _extract_channel_name(url):
+        """find channel id from channel name with yt-dlp help"""
+        obs_request = {
+            "skip_download": True,
+            "extract_flat": True,
+            "playlistend": 0,
+        }
+        url_info = YtWrap(obs_request).extract(url)
+        channel_id = url_info.get("channel_id", False)
+        if channel_id:
+            return channel_id
+
+        url = url_info.get("url", False)
+        if url:
+            # handle old channel name redirect with url path split
+            channel_id = urlparse(url).path.strip("/").split("/")[1]
+
+            return channel_id
+
+        print(f"failed to extract channel id from {url}")
+        raise ValueError
+
+    def _detect_vid_type(self, path):
+        """try to match enum from path, needs to be serializable"""
+        last = path.strip("/").split("/")[-1]
+        try:
+            vid_type = VideoTypeEnum(last).value
+        except ValueError:
+            vid_type = VideoTypeEnum.UNKNOWN.value
+
+        return {"vid_type": vid_type}

tubearchivist/home/tasks.py

@@ -6,6 +6,7 @@ Functionality:
   because tasks are initiated at application start
 """
 
+import json
 import os
 
 from celery import Celery, shared_task
@@ -17,16 +18,16 @@ from home.src.download.subscriptions import (
 )
 from home.src.download.thumbnails import ThumbFilesystem, ThumbValidator
 from home.src.download.yt_dlp_handler import VideoDownloader
-from home.src.es.index_setup import backup_all_indexes, restore_from_backup
+from home.src.es.backup import ElasticBackup
+from home.src.es.index_setup import ElasitIndexWrap
 from home.src.index.channel import YoutubeChannel
-from home.src.index.filesystem import (
-    ImportFolderScanner,
-    reindex_old_documents,
-    scan_filesystem,
-)
-from home.src.ta.config import AppConfig, ScheduleBuilder
-from home.src.ta.helper import UrlListParser
+from home.src.index.filesystem import ImportFolderScanner, scan_filesystem
+from home.src.index.reindex import Reindex, ReindexManual, ReindexOutdated
+from home.src.index.video_constants import VideoTypeEnum
+from home.src.ta.config import AppConfig, ReleaseVersion, ScheduleBuilder
+from home.src.ta.helper import clear_dl_cache
 from home.src.ta.ta_redis import RedisArchivist, RedisQueue
+from home.src.ta.urlparser import Parser
 
 CONFIG = AppConfig().config
 REDIS_HOST = os.environ.get("REDIS_HOST")
@@ -60,10 +61,22 @@ def update_subscribed():
             missing_from_channels = channel_handler.find_missing()
             playlist_handler = PlaylistSubscription()
             missing_from_playlists = playlist_handler.find_missing()
-            missing = missing_from_channels + missing_from_playlists
-            if missing:
-                youtube_ids = [{"type": "video", "url": i} for i in missing]
-                pending_handler = PendingList(youtube_ids=youtube_ids)
+            if missing_from_channels or missing_from_playlists:
+                channel_videos = [
+                    {"type": "video", "vid_type": vid_type, "url": vid_id}
+                    for vid_id, vid_type in missing_from_channels
+                ]
+                playlist_videos = [
+                    {
+                        "type": "video",
+                        "vid_type": VideoTypeEnum.VIDEOS,
+                        "url": i,
+                    }
+                    for i in missing_from_playlists
+                ]
+                pending_handler = PendingList(
+                    youtube_ids=channel_videos + playlist_videos
+                )
                 pending_handler.parse_url_list()
                 pending_handler.add_to_pending()
 
@@ -96,11 +109,15 @@ def download_pending():
 
 
 @shared_task
-def download_single(youtube_id):
+def download_single(pending_video):
     """start download single video now"""
-    queue = RedisQueue()
-    queue.add_priority(youtube_id)
-    print("Added to queue with priority: " + youtube_id)
+    queue = RedisQueue(queue_name="dl_queue")
+    to_add = {
+        "youtube_id": pending_video["youtube_id"],
+        "vid_type": pending_video["vid_type"],
+    }
+    queue.add_priority(json.dumps(to_add))
+    print(f"Added to queue with priority: {to_add}")
     # start queue if needed
     have_lock = False
     my_lock = RedisArchivist().get_lock("downloading")
@@ -135,9 +152,27 @@ def extrac_dl(youtube_ids):
 
 
 @shared_task(name="check_reindex")
-def check_reindex():
+def check_reindex(data=False, extract_videos=False):
     """run the reindex main command"""
-    reindex_old_documents()
+    if data:
+        ReindexManual(extract_videos=extract_videos).extract_data(data)
+
+    have_lock = False
+    reindex_lock = RedisArchivist().get_lock("reindex")
+
+    try:
+        have_lock = reindex_lock.acquire(blocking=False)
+        if have_lock:
+            if not data:
+                ReindexOutdated().add_outdated()
+
+            Reindex().reindex_all()
+        else:
+            print("Did not acquire reindex lock.")
+
+    finally:
+        if have_lock:
+            reindex_lock.release()
 
 
 @shared_task
@@ -168,7 +203,7 @@ def run_backup(reason="auto"):
     try:
         have_lock = my_lock.acquire(blocking=False)
         if have_lock:
-            backup_all_indexes(reason)
+            ElasticBackup(reason=reason).backup_all_indexes()
         else:
             print("Did not acquire lock for backup task.")
     finally:
@@ -180,7 +215,8 @@ def run_backup(reason="auto"):
 @shared_task
 def run_restore_backup(filename):
     """called from settings page, dump backup to zip file"""
-    restore_from_backup(filename)
+    ElasitIndexWrap().reset()
+    ElasticBackup().restore(filename)
     print("index restore finished")
 
 
@@ -190,13 +226,9 @@ def kill_dl(task_id):
         app.control.revoke(task_id, terminate=True)
 
     _ = RedisArchivist().del_message("dl_queue_id")
-    RedisQueue().clear()
+    RedisQueue(queue_name="dl_queue").clear()
 
-    # clear cache
-    cache_dir = os.path.join(CONFIG["application"]["cache_dir"], "download")
-    for cached in os.listdir(cache_dir):
-        to_delete = os.path.join(cache_dir, cached)
-        os.remove(to_delete)
+    clear_dl_cache(CONFIG)
 
     # notify
     mess_dict = {
@@ -230,9 +262,8 @@ def re_sync_thumbs():
 @shared_task
 def subscribe_to(url_str):
     """take a list of urls to subscribe to"""
-    to_subscribe_list = UrlListParser(url_str).process_list()
-    counter = 1
-    for item in to_subscribe_list:
+    to_subscribe_list = Parser(url_str).parse()
+    for idx, item in enumerate(to_subscribe_list):
         to_sub_id = item["url"]
         if item["type"] == "playlist":
             PlaylistSubscription().process_url_str([item])
@@ -255,10 +286,9 @@ def subscribe_to(url_str):
             "status": key,
             "level": "info",
             "title": "Subscribing to Channels",
-            "message": f"Processing {counter} of {len(to_subscribe_list)}",
+            "message": f"Processing {idx + 1} of {len(to_subscribe_list)}",
         }
         RedisArchivist().set_message(key, message=message, expire=True)
-        counter = counter + 1
 
 
 @shared_task
@@ -277,9 +307,15 @@ def index_channel_playlists(channel_id):
     channel.index_channel_playlists()
 
 
+@shared_task(name="version_check")
+def version_check():
+    """check for new updates"""
+    ReleaseVersion().check()
+
+
 try:
     app.conf.beat_schedule = ScheduleBuilder().build_schedule()
 except KeyError:
-    # update path from v0.0.8 to v0.0.9 to load new defaults
+    # update path to load new defaults
     StartupCheck().sync_redis_state()
     app.conf.beat_schedule = ScheduleBuilder().build_schedule()

tubearchivist/home/templates/home/about.html

@@ -9,7 +9,7 @@
         <h2>Useful Links</h2>
         <p>This project is in active and constant development, take a look at the <a href="https://github.com/tubearchivist/tubearchivist#roadmap" target="_blank">roadmap</a> for a overview.</p>
         <p>For any questions on what a button or a function does, You can find the up-to-date user documentation on <a href="https://github.com/tubearchivist/tubearchivist/wiki" target="_blank">Github</a>.</p>
-        <p>All contributions are welcome: Open an <a href="https://github.com/tubearchivist/tubearchivist/issues" target="_blank">issue</a> for any bugs and errors, join us on <a href="https://discord.gg/AFwz8nE7BK" target="_blank">Discord</a> to discuss details. The <a href="https://github.com/tubearchivist/tubearchivist/blob/master/CONTRIBUTING.md" target="_blank">contributing</a> page is a good place to get started.</p>
+        <p>All contributions are welcome: Open an <a href="https://github.com/tubearchivist/tubearchivist/issues" target="_blank">issue</a> for any bugs and errors, join us on <a href="https://www.tubearchivist.com/discord" target="_blank">Discord</a> to discuss details. The <a href="https://github.com/tubearchivist/tubearchivist/blob/master/CONTRIBUTING.md" target="_blank">contributing</a> page is a good place to get started.</p>
     </div>
     <div class="about-section">
         <h2>Donate</h2>

tubearchivist/home/templates/home/base.html

@@ -132,7 +132,14 @@
     </div>
     <div class="footer">
         <div class="boxed-content">
-            <span>© 2021 - <script type="text/javascript">document.write(new Date().getFullYear());</script> TubeArchivist {{ version }} </span><span><a href="{% url 'about' %}">About</a> | <a href="https://github.com/tubearchivist/tubearchivist" target="_blank">GitHub</a> | <a href="https://hub.docker.com/r/bbilly1/tubearchivist" target="_blank">Docker Hub</a> | <a href="https://discord.gg/AFwz8nE7BK" target="_blank">Discord</a> | <a href="https://www.reddit.com/r/TubeArchivist/">Reddit</a></span>
+            <span>© 2021 - <script type="text/javascript">document.write(new Date().getFullYear());</script></span>
+            <span>TubeArchivist</span>
+            <span>{{ version }}</span>
+            {% if ta_update %}
+                <span class="danger-zone">{{ ta_update.version }} available{% if ta_update.is_breaking %}<span class="danger-zone">Breaking Changes!</span>{% endif %}</span>
+                <span><a href="https://github.com/tubearchivist/tubearchivist/releases/tag/{{ ta_update.version }}" target="_blank">Release Page</a> | </span>
+            {% endif %}
+            <span><a href="{% url 'about' %}">About</a> | <a href="https://github.com/tubearchivist/tubearchivist" target="_blank">GitHub</a> | <a href="https://hub.docker.com/r/bbilly1/tubearchivist" target="_blank">Docker Hub</a> | <a href="https://www.tubearchivist.com/discord" target="_blank">Discord</a> | <a href="https://www.reddit.com/r/TubeArchivist/">Reddit</a></span>
         </div>
     </div>
 </body>

tubearchivist/home/templates/home/channel_id.html

@@ -8,7 +8,15 @@
     </div>
     <div class="info-box-item channel-nav">
         <a href="{% url 'channel_id' channel_info.channel_id %}"><h3>Videos</h3></a>
-        <a href="{% url 'channel_id_playlist' channel_info.channel_id %}"><h3>Playlists</h3></a>
+        {% if has_streams %}
+            <a href="{% url 'channel_id_live' channel_info.channel_id %}"><h3>Streams</h3></a>
+        {% endif %}
+        {% if has_shorts %}
+            <a href="{% url 'channel_id_shorts' channel_info.channel_id %}"><h3>Shorts</h3></a>
+        {% endif %}
+        {% if has_playlists %}
+            <a href="{% url 'channel_id_playlist' channel_info.channel_id %}"><h3>Playlists</h3></a>
+        {% endif %}
         <a href="{% url 'channel_id_about' channel_info.channel_id %}"><h3>About</h3></a>
         {% if has_pending %}
             <a href="{% url 'downloads' %}?channel={{ channel_info.channel_id }}"><h3>Downloads</h3></a>
@@ -100,7 +108,7 @@
                     <a href="#player" data-id="{{ video.source.youtube_id }}" onclick="createPlayer(this)">
                         <div class="video-thumb-wrap {{ view_style }}">
                             <div class="video-thumb">
-                                <img src="/cache/{{ video.source.vid_thumb_url }}" alt="video-thumb">
+                                <img src="{{ video.source.vid_thumb_url }}" alt="video-thumb">
                                 {% if video.source.player.progress %}
                                     <div class="video-progress-bar" id="progress-{{ video.source.youtube_id }}" style="width: {{video.source.player.progress}}%;"></div>
                                 {% else %}

tubearchivist/home/templates/home/channel_id_about.html

@@ -8,7 +8,15 @@
     </div>
     <div class="info-box-item channel-nav">
         <a href="{% url 'channel_id' channel_info.channel_id %}"><h3>Videos</h3></a>
-        <a href="{% url 'channel_id_playlist' channel_info.channel_id %}"><h3>Playlists</h3></a>
+        {% if has_streams %}
+            <a href="{% url 'channel_id_live' channel_info.channel_id %}"><h3>Streams</h3></a>
+        {% endif %}
+        {% if has_shorts %}
+            <a href="{% url 'channel_id_shorts' channel_info.channel_id %}"><h3>Shorts</h3></a>
+        {% endif %}
+        {% if has_playlists %}
+            <a href="{% url 'channel_id_playlist' channel_info.channel_id %}"><h3>Playlists</h3></a>
+        {% endif %}
         <a href="{% url 'channel_id_about' channel_info.channel_id %}"><h3>About</h3></a>
         {% if has_pending %}
             <a href="{% url 'downloads' %}?channel={{ channel_info.channel_id }}"><h3>Downloads</h3></a>
@@ -47,10 +55,20 @@
                 {% elif channel_info.channel_views > 0 %}
                     <p>Channel views: {{ channel_info.channel_views|intcomma }}</p>
                 {% endif %}
-                <button onclick="deleteConfirm()" id="delete-item">Delete Channel</button>
-                <div class="delete-confirm" id="delete-button">
-                    <span>Delete {{ channel_info.channel_name }} including all videos? </span><button class="danger-button" onclick="deleteChannel(this)" data-id="{{ channel_info.channel_id }}">Delete</button> <button onclick="cancelDelete()">Cancel</button>
+                <div class="button-box">
+                    <button onclick="deleteConfirm()" id="delete-item">Delete Channel</button>
+                    <div class="delete-confirm" id="delete-button">
+                        <span>Delete {{ channel_info.channel_name }} including all videos? </span><button class="danger-button" onclick="deleteChannel(this)" data-id="{{ channel_info.channel_id }}">Delete</button> <button onclick="cancelDelete()">Cancel</button>
+                    </div>
                 </div>
+                {% if reindex %}
+                    <p>Reindex scheduled</p>
+                {% else %}
+                    <div id="reindex-button" class="button-box">
+                        <button data-id="{{ channel_info.channel_id }}" data-type="channel" onclick="reindex(this)" title="Reindex Channel {{ channel_info.channel_name }}">Reindex</button>
+                        <button data-id="{{ channel_info.channel_id }}" data-type="channel" data-extract-videos="true" onclick="reindex(this)" title="Reindex Videos of {{ channel_info.channel_name }}">Reindex Videos</button>
+                    </div>
+                {% endif %}
             </div>
         </div>
     </div>

tubearchivist/home/templates/home/channel_id_playlist.html

@@ -8,7 +8,15 @@
     </div>
     <div class="info-box-item channel-nav">
         <a href="{% url 'channel_id' channel_info.channel_id %}"><h3>Videos</h3></a>
-        <a href="{% url 'channel_id_playlist' channel_info.channel_id %}"><h3>Playlists</h3></a>
+        {% if has_streams %}
+            <a href="{% url 'channel_id_live' channel_info.channel_id %}"><h3>Streams</h3></a>
+        {% endif %}
+        {% if has_shorts %}
+            <a href="{% url 'channel_id_shorts' channel_info.channel_id %}"><h3>Shorts</h3></a>
+        {% endif %}
+        {% if has_playlists %}
+            <a href="{% url 'channel_id_playlist' channel_info.channel_id %}"><h3>Playlists</h3></a>
+        {% endif %}
         <a href="{% url 'channel_id_about' channel_info.channel_id %}"><h3>About</h3></a>
         {% if has_pending %}
             <a href="{% url 'downloads' %}?channel={{ channel_info.channel_id }}"><h3>Downloads</h3></a>

tubearchivist/home/templates/home/downloads.html

@@ -72,12 +72,15 @@
                 <div class="video-item {{ view_style }}" id="dl-{{ video.source.youtube_id }}">
                     <div class="video-thumb-wrap {{ view_style }}">
                         <div class="video-thumb">
-                            <img src="/cache/{{ video.source.vid_thumb_url }}" alt="video_thumb">
-                            {% if show_ignored_only %}
-                                <span>ignored</span>
-                            {% else %}
-                                <span>queued</span>
-                            {% endif %}
+                            <img src="{{ video.source.vid_thumb_url }}" alt="video_thumb">
+                            <div class="video-tags">
+                                {% if show_ignored_only %}
+                                    <span>ignored</span>
+                                {% else %}
+                                    <span>queued</span>
+                                {% endif %}
+                                <span>{{ video.source.vid_type }}</span>
+                            </div>
                         </div>
                     </div>
                     <div class="video-desc {{ view_style }}">

tubearchivist/home/templates/home/home.html

@@ -12,7 +12,7 @@
                     <a href="#player" data-id="{{ video.source.youtube_id }}" onclick="createPlayer(this)">
                         <div class="video-thumb-wrap {{ view_style }}">
                             <div class="video-thumb">
-                                <img src="/cache/{{ video.source.vid_thumb_url }}" alt="video-thumb">
+                                <img src="{{ video.source.vid_thumb_url }}" alt="video-thumb">
                                 {% if video.source.player.progress %}
                                     <div class="video-progress-bar" id="progress-{{ video.source.youtube_id }}" style="width: {{video.source.player.progress}}%;"></div>
                                 {% else %}
@@ -98,7 +98,7 @@
                     <a href="#player" data-id="{{ video.source.youtube_id }}" onclick="createPlayer(this)">
                         <div class="video-thumb-wrap {{ view_style }}">
                             <div class="video-thumb">
-                                <img src="/cache/{{ video.source.vid_thumb_url }}" alt="video-thumb">
+                                <img src="{{ video.source.vid_thumb_url }}" alt="video-thumb">
                                 {% if video.source.player.progress %}
                                     <div class="video-progress-bar" id="progress-{{ video.source.youtube_id }}" style="width: {{video.source.player.progress}}%;"></div>
                                 {% else %}

tubearchivist/home/templates/home/playlist_id.html

@@ -52,6 +52,14 @@
                     <p>Total Videos archived: {{ max_hits }}/{{ playlist_info.playlist_entries|length }}</p>
                     <p>Watched: <button title="Mark all videos from {{ playlist_info.playlist_name }} as watched" type="button" id="watched-button" data-id="{{ playlist_info.playlist_id }}" onclick="isWatchedButton(this)">Mark as watched</button></p>
                 {% endif %}
+                {% if reindex %}
+                    <p>Reindex scheduled</p>
+                {% else %}
+                    <div id="reindex-button" class="button-box">
+                        <button data-id="{{ playlist_info.playlist_id }}" data-type="playlist" onclick="reindex(this)" title="Reindex Playlist {{ playlist_info.playlist_name }}">Reindex</button>
+                        <button data-id="{{ playlist_info.playlist_id }}" data-type="playlist" data-extract-videos="true" onclick="reindex(this)" title="Reindex Videos of {{ playlist_info.playlist_name }}">Reindex Videos</button>
+                    </div>
+                {% endif %}
             </div>
         </div>
     </div>
@@ -102,7 +110,7 @@
                     <a href="#player" data-id="{{ video.source.youtube_id }}" onclick="createPlayer(this)">
                         <div class="video-thumb-wrap {{ view_style }}">
                             <div class="video-thumb">
-                                <img src="/cache/{{ video.source.vid_thumb_url }}" alt="video-thumb">
+                                <img src="{{ video.source.vid_thumb_url }}" alt="video-thumb">
                                 {% if video.source.player.progress %}
                                     <div class="video-progress-bar" id="progress-{{ video.source.youtube_id }}" style="width: {{video.source.player.progress}}%;"></div>
                                 {% else %}

tubearchivist/home/templates/home/settings.html

@@ -38,6 +38,16 @@
                 <i>Videos to scan to find new items for the <b>Rescan subscriptions</b> task, max recommended 50.</i><br>
                 {{ app_form.subscriptions_channel_size }}
             </div>
+            <div class="settings-item">
+                <p>YouTube Live page size: <span class="settings-current">{{ config.subscriptions.live_channel_size }}</span></p>
+                <i>Live Videos to scan to find new items for the <b>Rescan subscriptions</b> task, max recommended 50, 0 to disable.</i><br>
+                {{ app_form.subscriptions_live_channel_size }}
+            </div>
+            <div class="settings-item">
+                <p>YouTube Shorts page size: <span class="settings-current">{{ config.subscriptions.shorts_channel_size }}</span></p>
+                <i>Shorts Videos to scan to find new items for the <b>Rescan subscriptions</b> task, max recommended 50, 0 to disable.</i><br>
+                {{ app_form.subscriptions_shorts_channel_size }}
+            </div>
         </div>
         <div class="settings-group">
             <h2 id="downloads">Downloads</h2>
@@ -77,7 +87,7 @@
                 <ul>
                     <li><span class="settings-current">bestvideo[height<=720]+bestaudio/best[height<=720]</span>: best audio and max video height of 720p.</li>
                     <li><span class="settings-current">bestvideo[height<=1080]+bestaudio/best[height<=1080]</span>: best audio and max video height of 1080p.</li>
-                    <li><span class="settings-current">bestvideo[height<=1080][VCODEC*=avc1]+bestaudio[ACODEC*=mp4a]/mp4</span>: Max 1080p video height with iOS compatible video and audio codecs.</li>
+                    <li><span class="settings-current">bestvideo[height<=1080][vcodec*=avc1]+bestaudio[acodec*=mp4a]/mp4</span>: Max 1080p video height with iOS compatible video and audio codecs.</li>
                     <li><span class="settings-current">0</span>: deactivate and download the best quality possible as decided by yt-dlp.</li>
                 </ul>
                 <i>Make sure your custom format gets merged into a single file. Check out the <a href="https://github.com/yt-dlp/yt-dlp#format-selection" target="_blank">documentation</a> for valid configurations.</i><br>
@@ -114,6 +124,24 @@
                 {{ app_form.downloads_subtitle_index }}
             </div>
         </div>
+        <div class="settings-group">
+            <h2 id="comments">Comments</h2>
+            <div class="settings-item">
+                <p>Download and index comments: <span class="settings-current">{{ config.downloads.comment_max }}</span><br>
+                <i>Follow the yt-dlp max_comments documentation, <a href="https://github.com/yt-dlp/yt-dlp#youtube" target="_blank">max-comments,max-parents,max-replies,max-replies-per-thread</a>:</i><br>
+                <p>Example configurations:</p>
+                <ul>
+                    <li><span class="settings-current">all,100,all,30</span>: Get 100 max-parents and 30 max-replies-per-thread.</li>
+                    <li><span class="settings-current">1000,all,all,50</span>: Get a total of 1000 comments over all, 50 replies per thread.</li>
+                </ul>
+                {{ app_form.downloads_comment_max }}</p>
+            </div>
+            <div class="settings-item">
+                <p>Selected comment sort method: <span class="settings-current">{{ config.downloads.comment_sort }}</span><br>
+                <i>Select how many comments and threads to download:</i><br>
+                {{ app_form.downloads_comment_sort }}</p>
+            </div>
+        </div>
         <div class="settings-group">
             <h2 id="format">Cookie</h2>
             <div class="settings-item">
@@ -147,11 +175,6 @@
                 <i>Before activating that, make sure you have a scraping sleep interval of at least 3 secs set to avoid ratelimiting issues.</i><br>
                 {{ app_form.downloads_integrate_sponsorblock }}
             </div>
-            <div class="settings-item">
-                <p>Current Cast integration: <span class="settings-current">{{ config.application.enable_cast }}</span></p>
-                <i>Enabling Cast will load an additional JS library from Google. HTTPS and a supported browser are required for this integration.</i><br>
-                {{ app_form.application_enable_cast }}
-            </div>
         </div>
         <div class="settings-group">
             <h2 id="snapshots">Snapshots</h2>
@@ -186,7 +209,7 @@
             <p>Note:</p>
             <ul>
                 <li>Changes in the scheduler settings require a container restart to take effect.</li>
-                <li>Avoid an unnecessary frequent schedule to not get blocked by YouTube. For that reason <i>*</i> or <i>wildcard</i> for minutes is not supported.</li>
+                <li>Avoid an unnecessary frequent schedule to not get blocked by YouTube. For that reason, the scheduler doesn't support schedules that trigger more than once per hour.</li>
             </ul>
         </div>
     </div>

tubearchivist/home/templates/home/video.html

@@ -2,8 +2,10 @@
 {% block content %}
 {% load static %}
 {% load humanize %}
-<div class="video-main">
-    <div class="video-modal"><span class="video-modal-text"></span></div>
+<div id="player" class="player-wrapper">
+    <div class="video-main">
+        <div class="video-modal"><span class="video-modal-text"></span></div>
+    </div>
 </div>
 <div class="notifications" id="notifications"></div>
 <div class="sponsorblock" id="sponsorblock">
@@ -54,10 +56,19 @@
                 {% else %}
                     <p>Youtube: Deactivated</p>
                 {% endif %}
-                <a download="" href="/media/{{ video.media_url }}"><button id="download-item">Download File</button></a>
-                <button onclick="deleteConfirm()" id="delete-item">Delete Video</button>
-                <div class="delete-confirm" id="delete-button">
-                    <span>Are you sure? </span><button class="danger-button" onclick="deleteVideo(this)" data-id="{{ video.youtube_id }}" data-redirect = "{{ video.channel.channel_id }}">Delete</button> <button onclick="cancelDelete()">Cancel</button>
+                {% if reindex %}
+                    <p>Reindex scheduled</p>
+                {% else %}
+                    <div id="reindex-button" class="button-box">
+                        <button data-id="{{ video.youtube_id }}" data-type="video" onclick="reindex(this)" title="Reindex {{ video.title }}">Reindex</button>
+                    </div>
+                {% endif %}
+                <div class="button-box">
+                    <a download="" href="/media/{{ video.media_url }}"><button id="download-item">Download File</button></a>
+                    <button onclick="deleteConfirm()" id="delete-item">Delete Video</button>
+                    <div class="delete-confirm" id="delete-button">
+                        <span>Are you sure? </span><button class="danger-button" onclick="deleteVideo(this)" data-id="{{ video.youtube_id }}" data-redirect = "{{ video.channel.channel_id }}">Delete</button> <button onclick="cancelDelete()">Cancel</button>
+                    </div>
                 </div>
             </div>
         </div>
@@ -69,15 +80,24 @@
                     <p class="thumb-icon"><img class="dislike" src="{% static 'img/icon-thumb.svg' %}" alt="thumbs-down">: {{ video.stats.dislike_count|intcomma }}</p>
                 {% endif %}
                 {% if video.stats.average_rating %}
-                    <p class="rating-stars">Rating:
+                    <div class="rating-stars">
                         {% for star in video.stats.average_rating %}
                             <img src="/static/img/icon-star-{{ star }}.svg" alt="{{ star }}">
                         {% endfor %}
-                    </p>
+                    </div>
                 {% endif %}
             </div>
         </div>
     </div>
+    {% if video.tags %}
+        <div class="description-box">
+            <div class="video-tag-box">
+                {% for tag in video.tags %}
+                    <span class="video-tag">{{ tag }}</span>
+                {% endfor %}
+            </div>
+        </div>
+    {% endif %}
     {% if video.description %}
         <div class="description-box">
             <p id="text-expand" class="description-text">
@@ -123,6 +143,24 @@
             </div>
         {% endfor %}
     {% endif %}
+    <div class="description-box">
+        <h3>Similar Videos</h3>
+        <div class="video-list grid grid-3" id="similar-videos">
+        </div>
+        <script>getSimilarVideos('{{ video.youtube_id }}')</script>
+    </div>
+    {% if video.comment_count == 0 %}
+    <div class="comments-section">
+        <span>Video has no comments</span>
+    </div>
+    {% elif video.comment_count %}
+    <div class="comments-section">
+        <h3>Comments: {{video.comment_count}}</h3>
+        <div id="comments-list" class="comments-list">
+        </div>
+        <script>getComments('{{ video.youtube_id }}')</script>
+    </div>
+    {% endif %}
 </div>
 <script>
     var videoData = getVideoData('{{ video.youtube_id }}');

tubearchivist/home/urls.py

@@ -7,7 +7,9 @@ from django.urls import path
 from home.views import (
     AboutView,
     ChannelIdAboutView,
+    ChannelIdLiveView,
     ChannelIdPlaylistView,
+    ChannelIdShortsView,
     ChannelIdView,
     ChannelView,
     DownloadView,
@@ -44,6 +46,16 @@ urlpatterns = [
         login_required(ChannelIdView.as_view()),
         name="channel_id",
     ),
+    path(
+        "channel/<slug:channel_id>/streams/",
+        login_required(ChannelIdLiveView.as_view()),
+        name="channel_id_live",
+    ),
+    path(
+        "channel/<slug:channel_id>/shorts/",
+        login_required(ChannelIdShortsView.as_view()),
+        name="channel_id_shorts",
+    ),
     path(
         "channel/<slug:channel_id>/about/",
         login_required(ChannelIdAboutView.as_view()),

tubearchivist/home/views.py

@@ -3,7 +3,7 @@ Functionality:
 - all views for home app
 - holds base classes to inherit from
 """
-
+import enum
 import json
 import urllib.parse
 from time import sleep
@@ -16,8 +16,8 @@ from django.http import JsonResponse
 from django.shortcuts import redirect, render
 from django.views import View
 from home.src.download.yt_dlp_base import CookieHandler
+from home.src.es.backup import ElasticBackup
 from home.src.es.connect import ElasticWrap
-from home.src.es.index_setup import get_available_backups
 from home.src.es.snapshot import ElasticSnapshot
 from home.src.frontend.api_calls import PostData
 from home.src.frontend.forms import (
@@ -35,9 +35,12 @@ from home.src.frontend.searching import SearchHandler
 from home.src.index.channel import YoutubeChannel, channel_overwrites
 from home.src.index.generic import Pagination
 from home.src.index.playlist import YoutubePlaylist
-from home.src.ta.config import AppConfig, ScheduleBuilder
-from home.src.ta.helper import UrlListParser, time_parser
+from home.src.index.reindex import ReindexProgress
+from home.src.index.video_constants import VideoTypeEnum
+from home.src.ta.config import AppConfig, ReleaseVersion, ScheduleBuilder
+from home.src.ta.helper import time_parser
 from home.src.ta.ta_redis import RedisArchivist
+from home.src.ta.urlparser import Parser
 from home.tasks import extrac_dl, index_channel_playlists, subscribe_to
 from rest_framework.authtoken.models import Token
 
@@ -137,6 +140,7 @@ class ArchivistViewConfig(View):
             "show_ignored_only": self._get_show_ignore_only(),
             "show_subed_only": self._get_show_subed_only(),
             "version": settings.TA_VERSION,
+            "ta_update": ReleaseVersion().get_update(),
         }
 
 
@@ -256,6 +260,19 @@ class ArchivistResultsView(ArchivistViewConfig):
         self.context["pagination"] = self.pagination_handler.pagination
 
 
+class MinView(View):
+    """to inherit from for minimal config vars"""
+
+    @staticmethod
+    def get_min_context(request):
+        """build minimal vars for context"""
+        return {
+            "colors": AppConfig(request.user.id).colors,
+            "version": settings.TA_VERSION,
+            "ta_update": ReleaseVersion().get_update(),
+        }
+
+
 class HomeView(ArchivistResultsView):
     """resolves to /
     handle home page and video search post functionality
@@ -297,20 +314,23 @@ class HomeView(ArchivistResultsView):
             self.data["query"] = query
 
 
-class LoginView(View):
+class LoginView(MinView):
     """resolves to /login/
     Greeting and login page
     """
 
     SEC_IN_DAY = 60 * 60 * 24
 
-    @staticmethod
-    def get(request):
+    def get(self, request):
         """handle get requests"""
-        failed = bool(request.GET.get("failed"))
-        colors = AppConfig(request.user.id).colors
-        form = CustomAuthForm()
-        context = {"colors": colors, "form": form, "form_error": failed}
+        context = self.get_min_context(request)
+        context.update(
+            {
+                "form": CustomAuthForm(),
+                "form_error": bool(request.GET.get("failed")),
+            }
+        )
+
         return render(request, "home/login.html", context)
 
     def post(self, request):
@@ -332,19 +352,15 @@ class LoginView(View):
         return redirect("/login?failed=true")
 
 
-class AboutView(View):
+class AboutView(MinView):
     """resolves to /about/
     show helpful how to information
     """
 
-    @staticmethod
-    def get(request):
+    def get(self, request):
         """handle http get"""
-        context = {
-            "title": "About",
-            "colors": AppConfig(request.user.id).colors,
-            "version": settings.TA_VERSION,
-        }
+        context = self.get_min_context(request)
+        context.update({"title": "About"})
         return render(request, "home/about.html", context)
 
 
@@ -441,7 +457,7 @@ class DownloadView(ArchivistResultsView):
             url_str = request.POST.get("vid_url")
             print(url_str)
             try:
-                youtube_ids = UrlListParser(url_str).process_list()
+                youtube_ids = Parser(url_str).parse()
             except ValueError:
                 # failed to process
                 key = "message:add"
@@ -473,6 +489,13 @@ class ChannelIdBaseView(ArchivistResultsView):
 
         return channel_info
 
+    def channel_pages(self, channel_id):
+        """get additional context for channel pages"""
+        self.channel_has_pending(channel_id)
+        self.channel_has_streams(channel_id)
+        self.channel_has_shorts(channel_id)
+        self.channel_has_playlist(channel_id)
+
     def channel_has_pending(self, channel_id):
         """check if channel has pending videos in queue"""
         path = "ta_download/_search"
@@ -486,11 +509,53 @@ class ChannelIdBaseView(ArchivistResultsView):
                     ]
                 }
             },
+            "_source": False,
         }
         response, _ = ElasticWrap(path).get(data=data)
 
         self.context.update({"has_pending": bool(response["hits"]["hits"])})
 
+    def channel_has_streams(self, channel_id):
+        """check if channel has streams videos"""
+        data = self.get_type_data("streams", channel_id)
+        response, _ = ElasticWrap("ta_video/_search").get(data=data)
+
+        self.context.update({"has_streams": bool(response["hits"]["hits"])})
+
+    def channel_has_shorts(self, channel_id):
+        """check if channel has shorts videos"""
+        data = self.get_type_data("shorts", channel_id)
+        response, _ = ElasticWrap("ta_video/_search").get(data=data)
+
+        self.context.update({"has_shorts": bool(response["hits"]["hits"])})
+
+    @staticmethod
+    def get_type_data(vid_type, channel):
+        """build data query for vid_type"""
+        return {
+            "size": 1,
+            "query": {
+                "bool": {
+                    "must": [
+                        {"term": {"vid_type": {"value": vid_type}}},
+                        {"term": {"channel.channel_id": {"value": channel}}},
+                    ]
+                }
+            },
+            "_source": False,
+        }
+
+    def channel_has_playlist(self, channel_id):
+        """check if channel has any playlist indexed"""
+        path = "ta_playlist/_search"
+        data = {
+            "size": 1,
+            "query": {"term": {"playlist_channel_id": {"value": channel_id}}},
+            "_source": False,
+        }
+        response, _ = ElasticWrap(path).get(data=data)
+        self.context.update({"has_playlists": bool(response["hits"]["hits"])})
+
 
 class ChannelIdView(ChannelIdBaseView):
     """resolves to /channel/<channel-id>/
@@ -499,6 +564,7 @@ class ChannelIdView(ChannelIdBaseView):
 
     view_origin = "home"
     es_search = "ta_video/_search"
+    video_types = [VideoTypeEnum.VIDEOS]
 
     def get(self, request, channel_id):
         """get request"""
@@ -506,7 +572,7 @@ class ChannelIdView(ChannelIdBaseView):
         self._update_view_data(channel_id)
         self.find_results()
         self.match_progress()
-        self.channel_has_pending(channel_id)
+        self.channel_pages(channel_id)
 
         if self.context["results"]:
             channel_info = self.context["results"][0]["source"]["channel"]
@@ -528,10 +594,20 @@ class ChannelIdView(ChannelIdBaseView):
 
     def _update_view_data(self, channel_id):
         """update view specific data dict"""
+        vid_type_terms = []
+        for t in self.video_types:
+            if t and isinstance(t, enum.Enum):
+                vid_type_terms.append(t.value)
+            else:
+                print(
+                    "Invalid value passed into video_types on "
+                    + f"ChannelIdView: {t}"
+                )
         self.data["query"] = {
             "bool": {
                 "must": [
-                    {"term": {"channel.channel_id": {"value": channel_id}}}
+                    {"term": {"channel.channel_id": {"value": channel_id}}},
+                    {"terms": {"vid_type": vid_type_terms}},
                 ]
             }
         }
@@ -557,6 +633,22 @@ class ChannelIdView(ChannelIdBaseView):
         return redirect("channel_id", channel_id, permanent=True)
 
 
+class ChannelIdLiveView(ChannelIdView):
+    """resolves to /channel/<channel-id>/streams/
+    display single channel page from channel_id
+    """
+
+    video_types = [VideoTypeEnum.STREAMS]
+
+
+class ChannelIdShortsView(ChannelIdView):
+    """resolves to /channel/<channel-id>/shorts/
+    display single channel page from channel_id
+    """
+
+    video_types = [VideoTypeEnum.SHORTS]
+
+
 class ChannelIdAboutView(ChannelIdBaseView):
     """resolves to /channel/<channel-id>/about/
     show metadata, handle per channel conf
@@ -567,19 +659,20 @@ class ChannelIdAboutView(ChannelIdBaseView):
     def get(self, request, channel_id):
         """handle get request"""
         self.initiate_vars(request)
-        self.channel_has_pending(channel_id)
-
-        path = f"ta_channel/_doc/{channel_id}"
-        response, _ = ElasticWrap(path).get()
+        self.channel_pages(channel_id)
 
+        response, _ = ElasticWrap(f"ta_channel/_doc/{channel_id}").get()
         channel_info = SearchProcess(response).process()
-        channel_name = channel_info["channel_name"]
+        reindex = ReindexProgress(
+            request_type="channel", request_id=channel_id
+        ).get_progress()
 
         self.context.update(
             {
-                "title": "Channel: About " + channel_name,
+                "title": "Channel: About " + channel_info["channel_name"],
                 "channel_info": channel_info,
                 "channel_overwrite_form": ChannelOverwriteForm,
+                "reindex": reindex.get("state"),
             }
         )
 
@@ -614,7 +707,7 @@ class ChannelIdPlaylistView(ChannelIdBaseView):
         self.initiate_vars(request)
         self._update_view_data(channel_id)
         self.find_results()
-        self.channel_has_pending(channel_id)
+        self.channel_pages(channel_id)
 
         channel_info = self.get_channel_meta(channel_id)
         channel_name = channel_info["channel_name"]
@@ -706,12 +799,17 @@ class PlaylistIdView(ArchivistResultsView):
         self._update_view_data(playlist_id, playlist_info)
         self.find_results()
         self.match_progress()
+        reindex = ReindexProgress(
+            request_type="playlist", request_id=playlist_id
+        ).get_progress()
+
         self.context.update(
             {
                 "title": "Playlist: " + playlist_name,
                 "playlist_info": playlist_info,
                 "playlist_name": playlist_name,
                 "channel_info": channel_info,
+                "reindex": reindex.get("state"),
             }
         )
         return render(request, "home/playlist_id.html", self.context)
@@ -836,7 +934,7 @@ class PlaylistView(ArchivistResultsView):
         return redirect("playlist")
 
 
-class VideoView(View):
+class VideoView(MinView):
     """resolves to /video/<video-id>/
     display details about a single video
     """
@@ -844,11 +942,8 @@ class VideoView(View):
     def get(self, request, video_id):
         """get single video"""
         config_handler = AppConfig(request.user.id)
-        position = time_parser(request.GET.get("t"))
-        path = f"ta_video/_doc/{video_id}"
-        look_up = SearchHandler(path, config=False)
-        video_hit = look_up.get_data()
-        video_data = video_hit[0]["source"]
+        look_up = SearchHandler(f"ta_video/_doc/{video_id}", config=False)
+        video_data = look_up.get_data()[0]["source"]
         try:
             rating = video_data["stats"]["average_rating"]
             video_data["stats"]["average_rating"] = self.star_creator(rating)
@@ -861,17 +956,22 @@ class VideoView(View):
         else:
             playlist_nav = False
 
-        video_title = video_data["title"]
-        context = {
-            "video": video_data,
-            "playlist_nav": playlist_nav,
-            "title": video_title,
-            "colors": config_handler.colors,
-            "cast": config_handler.config["application"]["enable_cast"],
-            "version": settings.TA_VERSION,
-            "config": config_handler.config,
-            "position": position,
-        }
+        reindex = ReindexProgress(
+            request_type="video", request_id=video_id
+        ).get_progress()
+
+        context = self.get_min_context(request)
+        context.update(
+            {
+                "video": video_data,
+                "playlist_nav": playlist_nav,
+                "title": video_data.get("title"),
+                "cast": config_handler.config["application"]["enable_cast"],
+                "config": config_handler.config,
+                "position": time_parser(request.GET.get("t")),
+                "reindex": reindex.get("state"),
+            }
+        )
         return render(request, "home/video.html", context)
 
     @staticmethod
@@ -928,7 +1028,7 @@ class SearchView(ArchivistResultsView):
         return render(request, "home/search.html", self.context)
 
 
-class SettingsView(View):
+class SettingsView(MinView):
     """resolves to /settings/
     handle the settings page, display current settings,
     take post request from the form to update settings
@@ -936,28 +1036,19 @@ class SettingsView(View):
 
     def get(self, request):
         """read and display current settings"""
-        config_handler = AppConfig(request.user.id)
-        colors = config_handler.colors
-
-        available_backups = get_available_backups()
-        user_form = UserSettingsForm()
-        app_form = ApplicationSettingsForm()
-        scheduler_form = SchedulerSettingsForm()
-        snapshots = ElasticSnapshot().get_snapshot_stats()
-        token = self.get_token(request)
-
-        context = {
-            "title": "Settings",
-            "config": config_handler.config,
-            "api_token": token,
-            "colors": colors,
-            "available_backups": available_backups,
-            "user_form": user_form,
-            "app_form": app_form,
-            "scheduler_form": scheduler_form,
-            "snapshots": snapshots,
-            "version": settings.TA_VERSION,
-        }
+        context = self.get_min_context(request)
+        context.update(
+            {
+                "title": "Settings",
+                "config": AppConfig(request.user.id).config,
+                "api_token": self.get_token(request),
+                "available_backups": ElasticBackup().get_all_backup_files(),
+                "user_form": UserSettingsForm(),
+                "app_form": ApplicationSettingsForm(),
+                "scheduler_form": SchedulerSettingsForm(),
+                "snapshots": ElasticSnapshot().get_snapshot_stats(),
+            }
+        )
 
         return render(request, "home/settings.html", context)
 

tubearchivist/requirements.txt

@@ -1,13 +1,13 @@
 beautifulsoup4==4.11.1
 celery==5.2.7
-Django==4.1.3
+Django==4.1.5
 django-auth-ldap==4.1.0
 django-cors-headers==3.13.0
 djangorestframework==3.14.0
-Pillow==9.3.0
-redis==4.3.4
+Pillow==9.4.0
+redis==4.4.2
 requests==2.28.1
 ryd-client==0.0.6
 uWSGI==2.0.21
-whitenoise==6.2.0
-yt_dlp==2022.10.4
+whitenoise==6.3.0
+yt_dlp==2023.1.6

tubearchivist/static/css/style.css

@@ -119,6 +119,10 @@ button:hover {
     color: var(--main-bg);
 }
 
+.button-box {
+    padding: 5px 0;
+}
+
 .unsubscribe {
     background-color: var(--accent-font-light);
 }
@@ -458,7 +462,7 @@ video:-webkit-full-screen {
     overflow: hidden;
 }
 
-.video-item:hover .video-thumb span {
+.video-item:hover .video-tags {
     opacity: 1;
 }
 
@@ -482,16 +486,20 @@ video:-webkit-full-screen {
     position: relative;
 }
 
-.video-thumb span {
+.video-tags {
     position: absolute;
     top: 5px;
-    left: 5px;
-    background-color: var(--accent-font-light);
+    left: 0;
     padding: 5px;
     opacity: 0;
     transition: 300ms ease-in-out;
 }
 
+.video-tags span {
+    background-color: var(--accent-font-light);
+    padding: 5px;
+}
+
 .video-play img {
     width: 40px;
     filter: var(--img-filter);
@@ -619,7 +627,8 @@ video:-webkit-full-screen {
     margin-top: 1rem;
 }
 
-.description-box {
+.description-box,
+.comments-section {
     margin-top: 1rem;
     padding: 15px;
     background-color: var(--highlight-bg);
@@ -640,6 +649,7 @@ video:-webkit-full-screen {
 
 .info-box-item {
     display: flex;
+    flex-wrap: wrap;
     align-items: center;
     padding: 15px;
     background-color: var(--highlight-bg);
@@ -778,10 +788,26 @@ video:-webkit-full-screen {
     margin-left: 5px;
 }
 
+.thumb-icon {
+    display: flex;
+}
+
+.video-tag-box {
+    display: flex;
+    flex-wrap: wrap;
+    justify-content: center;
+}
+
+.video-tag {
+    padding: 5px 10px;
+    margin: 5px;
+    border: 1px solid var(--accent-font-light);
+}
+
 .thumb-icon img,
 .rating-stars img {
     width: 20px;
-    margin: 0;
+    margin: 0 5px;
     filter: var(--img-filter);
 }
 
@@ -819,6 +845,44 @@ video:-webkit-full-screen {
     width: 100%;
 }
 
+.comment-box {
+    padding-bottom: 1rem;
+    overflow: hidden;
+}
+
+.comment-box h3 {
+    line-break: anywhere;
+}
+
+.comments-replies {
+    display: none;
+    padding-left: 1rem;
+    border-left: 1px solid var(--accent-font-light);
+    margin-top: 1rem;
+}
+
+.comment-highlight {
+    background-color: var(--main-font);
+    padding: 3px;
+    color: var(--accent-font-dark);
+    font-family: Sen-bold, sans-serif;
+    width: fit-content;
+}
+
+.comment-meta {
+    display: flex;
+}
+
+.space-carrot {
+    margin: 0 5px;
+}
+
+.comment-like img {
+    width: 20px;
+    margin-left: 5px;
+    filter: var(--img-filter-error);
+}
+
 /* multi search page */
 .multi-search-box {
     padding-right: 20px;

tubearchivist/static/img/icon-eye.svg

@@ -1,63 +1,17 @@
-<?xml version="1.0" encoding="UTF-8" standalone="no"?>
-<!-- Created with Inkscape (http://www.inkscape.org/) -->
-
-<svg
-   xmlns:dc="http://purl.org/dc/elements/1.1/"
-   xmlns:cc="http://creativecommons.org/ns#"
-   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
-   xmlns:svg="http://www.w3.org/2000/svg"
-   xmlns="http://www.w3.org/2000/svg"
-   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
-   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
-   width="500"
-   height="500"
-   viewBox="0 0 132.29197 132.29167"
-   version="1.1"
-   id="svg1303"
-   inkscape:version="0.92.4 (5da689c313, 2019-01-14)"
-   sodipodi:docname="Icons_seen.svg">
-  <defs
-     id="defs1297" />
-  <sodipodi:namedview
-     id="base"
-     pagecolor="#ffffff"
-     bordercolor="#666666"
-     borderopacity="1.0"
-     inkscape:pageopacity="0.0"
-     inkscape:pageshadow="2"
-     inkscape:zoom="1.0105705"
-     inkscape:cx="84.208758"
-     inkscape:cy="136.94831"
-     inkscape:document-units="mm"
-     inkscape:current-layer="layer1"
-     showgrid="false"
-     units="px"
-     inkscape:window-width="1920"
-     inkscape:window-height="1017"
-     inkscape:window-x="-8"
-     inkscape:window-y="-8"
-     inkscape:window-maximized="1" />
-  <metadata
-     id="metadata1300">
-    <rdf:RDF>
-      <cc:Work
-         rdf:about="">
-        <dc:format>image/svg+xml</dc:format>
-        <dc:type
-           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
-        <dc:title />
-      </cc:Work>
-    </rdf:RDF>
-  </metadata>
-  <g
-     inkscape:label="Ebene 1"
-     inkscape:groupmode="layer"
-     id="layer1"
-     transform="translate(0,-164.70764)">
-    <path
-       style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;paint-order:markers fill stroke"
-       d="M 66.145984,191.3255 A 71.797122,73.404487 0 0 0 2.5025987,230.88314 71.797122,73.404487 0 0 0 66.145984,270.38145 71.797122,73.404487 0 0 0 129.78937,230.82387 71.797122,73.404487 0 0 0 66.145984,191.3255 Z m -4.921549,7.22394 a 32.724755,32.724755 0 0 0 -13.334395,5.17759 12.828107,12.828107 0 0 1 2.180958,-0.18652 12.828107,12.828107 0 0 1 12.828141,12.82816 12.828107,12.828107 0 0 1 -12.828141,12.82822 12.828107,12.828107 0 0 1 -12.828192,-12.82822 12.828107,12.828107 0 0 1 0.04509,-0.91548 32.724755,32.724755 0 0 0 -3.86581,15.39964 32.724755,32.724755 0 0 0 27.161922,32.24981 A 59.09757,60.420619 0 0 1 13.7604,230.8774 59.09757,60.420619 0 0 1 61.224664,198.54948 Z m 10.482345,0.0563 a 59.09757,60.420619 0 0 1 46.82502,32.22523 59.09757,60.420619 0 0 1 -47.393377,32.32497 32.724755,32.724755 0 0 0 27.73169,-32.30187 32.724755,32.724755 0 0 0 -27.163333,-32.24833 z"
-       id="path1091"
-       inkscape:connector-curvature="0" />
-  </g>
-</svg>
+<?xml version="1.0" encoding="utf-8"?>
+<!-- Generator: Adobe Illustrator 26.5.0, SVG Export Plug-In . SVG Version: 6.00 Build 0)  -->
+<svg version="1.1"
+	 id="svg1303" xmlns:cc="http://creativecommons.org/ns#" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" xmlns:svg="http://www.w3.org/2000/svg" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" inkscape:version="0.92.4 (5da689c313, 2019-01-14)" sodipodi:docname="Icons_seen.svg"
+	 xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px" viewBox="0 0 500 500"
+	 style="enable-background:new 0 0 500 500;" xml:space="preserve">
+<sodipodi:namedview  bordercolor="#666666" borderopacity="1.0" id="base" inkscape:current-layer="layer1" inkscape:cx="84.208758" inkscape:cy="136.94831" inkscape:document-units="mm" inkscape:pageopacity="0.0" inkscape:pageshadow="2" inkscape:window-height="1017" inkscape:window-maximized="1" inkscape:window-width="1920" inkscape:window-x="-8" inkscape:window-y="-8" inkscape:zoom="1.0105705" pagecolor="#ffffff" showgrid="false" units="px">
+	</sodipodi:namedview>
+<g id="layer1" transform="translate(0,-164.70764)" inkscape:groupmode="layer" inkscape:label="Ebene 1">
+	<path id="path1091" inkscape:connector-curvature="0" d="M250,259.4c-105.2,0.1-201.5,60-250,155.4C48.5,510.1,144.9,569.9,250,570
+		c105.2-0.1,201.5-60,250-155.4C451.5,319.3,355.1,259.5,250,259.4z M230.7,287.8c-18.7,2.8-36.6,9.8-52.4,20.3
+		c2.8-0.5,5.7-0.7,8.6-0.7c27.8,0,50.4,22.6,50.4,50.4c0,0,0,0,0,0c0,27.8-22.6,50.4-50.4,50.4c0,0,0,0,0,0
+		c-27.8,0-50.4-22.6-50.4-50.4c0,0,0,0,0,0c0-1.2,0.1-2.4,0.2-3.6c-10,18.6-15.2,39.4-15.2,60.5c0,62.6,45,116,106.7,126.7
+		c-78.3-7.6-147.6-55.2-183.9-126.6C81,342.6,151.3,294.7,230.7,287.8L230.7,287.8z M271.8,288c78.3,7.6,147.6,55.2,183.9,126.6
+		c-36.7,72.1-106.9,120-186.2,127c62.6-9.7,108.9-63.5,108.9-126.9C378.5,352.1,333.5,298.7,271.8,288L271.8,288z"/>
+</g>
+</svg>

tubearchivist/static/img/icon-heart.svg

@@ -0,0 +1,8 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!-- Generator: Adobe Illustrator 26.5.0, SVG Export Plug-In . SVG Version: 6.00 Build 0)  -->
+<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
+	 viewBox="0 0 500 500" style="enable-background:new 0 0 500 500;" xml:space="preserve">
+<path d="M499.9,159L499.9,159c0.1-1.7,0.1-3.4,0.1-5.2c0-69.5-58.6-129.7-130.9-129.7c-52.9,0-98.4,34-119,77.4h0
+	c-20.7-43.4-66.2-77.4-119-77.4C58.6,24.1,0,84.4,0,153.9c0,1.7,0.1,3.4,0.1,5.2h0c0,0-7.4,82.6,84.5,172.7
+	c41.8,41.9,88.5,81.6,165.4,144.1c76.9-62.5,123.6-102.3,165.4-144.1C507.2,241.6,499.9,159,499.9,159z"/>
+</svg>

tubearchivist/static/img/icon-thumb.svg

@@ -1,76 +1,18 @@
-<?xml version="1.0" encoding="UTF-8" standalone="no"?>
-<!-- Created with Inkscape (http://www.inkscape.org/) -->
-
-<svg
-   xmlns:dc="http://purl.org/dc/elements/1.1/"
-   xmlns:cc="http://creativecommons.org/ns#"
-   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
-   xmlns:svg="http://www.w3.org/2000/svg"
-   xmlns="http://www.w3.org/2000/svg"
-   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
-   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
-   width="500"
-   height="500"
-   viewBox="0 0 132.29197 132.29167"
-   version="1.1"
-   id="svg1303"
-   inkscape:version="0.92.4 (5da689c313, 2019-01-14)"
-   sodipodi:docname="icon_thumb.svg">
-  <defs
-     id="defs1297" />
-  <sodipodi:namedview
-     id="base"
-     pagecolor="#ffffff"
-     bordercolor="#666666"
-     borderopacity="1.0"
-     inkscape:pageopacity="0.0"
-     inkscape:pageshadow="2"
-     inkscape:zoom="1.5046797"
-     inkscape:cx="35.718548"
-     inkscape:cy="203.39339"
-     inkscape:document-units="mm"
-     inkscape:current-layer="layer1"
-     showgrid="false"
-     units="px"
-     inkscape:window-width="1920"
-     inkscape:window-height="1009"
-     inkscape:window-x="-8"
-     inkscape:window-y="-8"
-     inkscape:window-maximized="1"
-     showguides="true"
-     inkscape:guide-bbox="true">
-    <sodipodi:guide
-       position="80.99058,65.965029"
-       orientation="0,1"
-       id="guide816"
-       inkscape:locked="false" />
-    <sodipodi:guide
-       position="65.965178,49.107299"
-       orientation="1,0"
-       id="guide818"
-       inkscape:locked="false" />
-  </sodipodi:namedview>
-  <metadata
-     id="metadata1300">
-    <rdf:RDF>
-      <cc:Work
-         rdf:about="">
-        <dc:format>image/svg+xml</dc:format>
-        <dc:type
-           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
-        <dc:title />
-      </cc:Work>
-    </rdf:RDF>
-  </metadata>
-  <g
-     inkscape:label="Ebene 1"
-     inkscape:groupmode="layer"
-     id="layer1"
-     transform="translate(0,-164.70764)">
-    <path
-       style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;paint-order:markers fill stroke"
-       d="m 72.35681,178.12175 c -0.545876,-0.0192 -1.121779,0.0192 -1.729242,0.12259 -12.959134,2.14891 4.226681,21.04642 -10.867262,31.83193 -12.617074,10.52505 -17.206737,7.22264 -17.332488,10.67172 v 33.88171 c 1.891308,5.00813 14.483922,10.12847 27.868234,11.21944 13.384309,1.09096 32.098988,-1.73801 32.305468,-7.56421 0.10989,-5.27572 -2.896223,-5.39662 -2.896223,-5.39662 0,0 6.589043,-0.40752 6.808883,-6.12278 0.10966,-5.60545 -6.38265,-6.40691 -6.38265,-6.40691 0,0 8.31589,-0.23607 8.31589,-6.01832 0,-5.91485 -6.39662,-6.38072 -8.37539,-6.48104 1.22776,-0.0747 6.07353,-1.75141 6.04505,-7.59582 -0.0287,-6.03384 -9.280896,-5.53597 -25.919173,-6.63279 1.919791,-11.19377 3.258461,-35.14851 -7.841097,-35.50894 z m -45.69253,38.36631 c -1.562226,0 -2.819733,1.04793 -2.819733,2.34979 v 38.25943 c 0,1.30191 1.257507,2.34983 2.819733,2.34983 h 10.526021 c 1.562231,0 2.819739,-1.04792 2.819739,-2.34983 v -38.25943 c 0,-1.30186 -1.257508,-2.34979 -2.819739,-2.34979 z"
-       id="rect1278"
-       inkscape:connector-curvature="0" />
-  </g>
-</svg>
+<?xml version="1.0" encoding="utf-8"?>
+<!-- Generator: Adobe Illustrator 26.5.0, SVG Export Plug-In . SVG Version: 6.00 Build 0)  -->
+<svg version="1.1"
+	 id="svg1303" xmlns:cc="http://creativecommons.org/ns#" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" xmlns:svg="http://www.w3.org/2000/svg" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" inkscape:version="0.92.4 (5da689c313, 2019-01-14)" sodipodi:docname="icon_thumb.svg"
+	 xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px" viewBox="0 0 500 500"
+	 style="enable-background:new 0 0 500 500;" xml:space="preserve">
+<sodipodi:namedview  bordercolor="#666666" borderopacity="1.0" id="base" inkscape:current-layer="layer1" inkscape:cx="35.718548" inkscape:cy="203.39339" inkscape:document-units="mm" inkscape:guide-bbox="true" inkscape:pageopacity="0.0" inkscape:pageshadow="2" inkscape:window-height="1009" inkscape:window-maximized="1" inkscape:window-width="1920" inkscape:window-x="-8" inkscape:window-y="-8" inkscape:zoom="1.5046797" pagecolor="#ffffff" showgrid="false" showguides="true" units="px">
+	<sodipodi:guide  id="guide816" inkscape:locked="false" orientation="0,1" position="80.99058,65.965029"></sodipodi:guide>
+	<sodipodi:guide  id="guide818" inkscape:locked="false" orientation="1,0" position="65.965178,49.107299"></sodipodi:guide>
+</sodipodi:namedview>
+<g id="layer1" transform="translate(0,-164.70764)" inkscape:groupmode="layer" inkscape:label="Ebene 1">
+	<path id="rect1278" inkscape:connector-curvature="0" d="M285.3,164.7c-3.1-0.1-6.4,0.1-9.8,0.7c-73.7,12.2,24,119.6-61.8,180.9
+		c-71.7,59.8-97.8,41.1-98.5,60.7v192.6c10.8,28.5,82.3,57.6,158.4,63.8c76.1,6.2,182.5-9.9,183.6-43c0.6-30-16.5-30.7-16.5-30.7
+		s37.5-2.3,38.7-34.8c0.6-31.9-36.3-36.4-36.3-36.4s47.3-1.3,47.3-34.2c0-33.6-36.4-36.3-47.6-36.8c7-0.4,34.5-10,34.4-43.2
+		c-0.2-34.3-52.8-31.5-147.3-37.7C340.8,302.9,348.4,166.8,285.3,164.7L285.3,164.7z M25.6,382.8c-8.9,0-16,6-16,13.4v217.5
+		c0,7.4,7.1,13.4,16,13.4h59.8c8.9,0,16-6,16-13.4V396.2c0-7.4-7.1-13.4-16-13.4H25.6z"/>
+</g>
+</svg>

tubearchivist/static/script.js

@@ -23,15 +23,14 @@ function updateVideoWatchStatus(input1, videoCurrentWatchStatus) {
   postVideoProgress(videoId, 0); // Reset video progress on watched/unwatched;
   removeProgressBar(videoId);
 
-  let watchStatusIndicator, payload;
+  let watchStatusIndicator;
+  let apiEndpoint = '/api/watched/';
   if (videoCurrentWatchStatus === 'watched') {
     watchStatusIndicator = createWatchStatusIndicator(videoId, 'unwatched');
-    payload = JSON.stringify({ un_watched: videoId });
-    sendPost(payload);
+    apiRequest(apiEndpoint, 'POST', { id: videoId, is_watched: false });
   } else if (videoCurrentWatchStatus === 'unwatched') {
     watchStatusIndicator = createWatchStatusIndicator(videoId, 'watched');
-    payload = JSON.stringify({ watched: videoId });
-    sendPost(payload);
+    apiRequest(apiEndpoint, 'POST', { id: videoId, is_watched: true });
   }
 
   let watchButtons = document.getElementsByClassName('watch-button');
@@ -76,9 +75,10 @@ function removeProgressBar(videoId) {
 
 function isWatchedButton(button) {
   let youtube_id = button.getAttribute('data-id');
-  let payload = JSON.stringify({ watched: youtube_id });
+  let apiEndpoint = '/api/watched/';
+  let data = { id: youtube_id, is_watched: true };
   button.remove();
-  sendPost(payload);
+  apiRequest(apiEndpoint, 'POST', data);
   setTimeout(function () {
     location.reload();
   }, 1000);
@@ -147,6 +147,24 @@ function toggleCheckbox(checkbox) {
   }, 500);
 }
 
+// start reindex task
+function reindex(button) {
+  let apiEndpoint = '/api/refresh/';
+  if (button.getAttribute('data-extract-videos')) {
+    apiEndpoint += '?extract_videos=true';
+  }
+  let type = button.getAttribute('data-type');
+  let id = button.getAttribute('data-id');
+
+  let data = {};
+  data[type] = [id];
+
+  apiRequest(apiEndpoint, 'POST', data);
+  let message = document.createElement('p');
+  message.innerText = 'Reindex scheduled';
+  document.getElementById('reindex-button').replaceWith(message);
+}
+
 // download page buttons
 function rescanPending() {
   let payload = JSON.stringify({ rescan_pending: true });
@@ -168,15 +186,15 @@ function dlPending() {
 
 function toIgnore(button) {
   let youtube_id = button.getAttribute('data-id');
-  let payload = JSON.stringify({ ignore: youtube_id });
-  sendPost(payload);
+  let apiEndpoint = '/api/download/' + youtube_id + '/';
+  apiRequest(apiEndpoint, 'POST', { status: 'ignore' });
   document.getElementById('dl-' + youtube_id).remove();
 }
 
 function downloadNow(button) {
   let youtube_id = button.getAttribute('data-id');
-  let payload = JSON.stringify({ dlnow: youtube_id });
-  sendPost(payload);
+  let apiEndpoint = '/api/download/' + youtube_id + '/';
+  apiRequest(apiEndpoint, 'POST', { status: 'priority' });
   document.getElementById(youtube_id).remove();
   setTimeout(function () {
     checkMessages();
@@ -185,15 +203,15 @@ function downloadNow(button) {
 
 function forgetIgnore(button) {
   let youtube_id = button.getAttribute('data-id');
-  let payload = JSON.stringify({ forgetIgnore: youtube_id });
-  sendPost(payload);
+  let apiEndpoint = '/api/download/' + youtube_id + '/';
+  apiRequest(apiEndpoint, 'DELETE');
   document.getElementById('dl-' + youtube_id).remove();
 }
 
 function addSingle(button) {
   let youtube_id = button.getAttribute('data-id');
-  let payload = JSON.stringify({ addSingle: youtube_id });
-  sendPost(payload);
+  let apiEndpoint = '/api/download/' + youtube_id + '/';
+  apiRequest(apiEndpoint, 'POST', { status: 'pending' });
   document.getElementById('dl-' + youtube_id).remove();
   setTimeout(function () {
     checkMessages();
@@ -202,8 +220,8 @@ function addSingle(button) {
 
 function deleteQueue(button) {
   let to_delete = button.getAttribute('data-id');
-  let payload = JSON.stringify({ deleteQueue: to_delete });
-  sendPost(payload);
+  let apiEndpoint = '/api/download/?filter=' + to_delete;
+  apiRequest(apiEndpoint, 'DELETE');
   // clear button
   let message = document.createElement('p');
   message.innerText = 'deleting download queue: ' + to_delete;
@@ -287,7 +305,7 @@ function resetToken() {
   document.getElementById('text-reveal').replaceWith(message);
 }
 
-// restore from snapshot 
+// restore from snapshot
 function restoreSnapshot(snapshotId) {
   console.log('restore ' + snapshotId);
   let apiEndpoint = '/api/snapshot/' + snapshotId + '/';
@@ -316,8 +334,8 @@ function deleteConfirm() {
 function deleteVideo(button) {
   let to_delete = button.getAttribute('data-id');
   let to_redirect = button.getAttribute('data-redirect');
-  let payload = JSON.stringify({ 'delete-video': to_delete });
-  sendPost(payload);
+  let apiEndpoint = '/api/video/' + to_delete + '/';
+  apiRequest(apiEndpoint, 'DELETE');
   setTimeout(function () {
     let redirect = '/channel/' + to_redirect;
     window.location.replace(redirect);
@@ -326,8 +344,8 @@ function deleteVideo(button) {
 
 function deleteChannel(button) {
   let to_delete = button.getAttribute('data-id');
-  let payload = JSON.stringify({ 'delete-channel': to_delete });
-  sendPost(payload);
+  let apiEndpoint = '/api/channel/' + to_delete + '/';
+  apiRequest(apiEndpoint, 'DELETE');
   setTimeout(function () {
     window.location.replace('/channel/');
   }, 1000);
@@ -793,7 +811,11 @@ function apiRequest(apiEndpoint, method, data) {
   xhttp.setRequestHeader('Authorization', 'Token ' + sessionToken);
   xhttp.setRequestHeader('Content-Type', 'application/json');
   xhttp.send(JSON.stringify(data));
-  return JSON.parse(xhttp.responseText);
+  if (xhttp.status === 404) {
+    return false;
+  } else {
+    return JSON.parse(xhttp.responseText);
+  }
 }
 
 // Gets origin URL
@@ -951,7 +973,7 @@ function createVideo(video, viewStyle) {
   // create video item div from template
   const videoId = video.youtube_id;
   // const mediaUrl = video.media_url;
-  const thumbUrl = '/cache/' + video.vid_thumb_url;
+  // const thumbUrl = '/cache/' + video.vid_thumb_url;
   const videoTitle = video.title;
   const videoPublished = video.published;
   const videoDuration = video.player.duration_str;
@@ -968,7 +990,7 @@ function createVideo(video, viewStyle) {
     <a href="#player" data-id="${videoId}" onclick="createPlayer(this)">
         <div class="video-thumb-wrap ${viewStyle}">
             <div class="video-thumb">
-                <img src="${thumbUrl}" alt="video-thumb">
+                <img src="${video.vid_thumb_url}" alt="video-thumb">
             </div>
             <div class="video-play">
                 <img src="/static/img/icon-play.svg" alt="play-icon">
@@ -1104,6 +1126,138 @@ function createFulltext(fullText) {
   return fullTextDiv;
 }
 
+function getComments(videoId) {
+  let apiEndpoint = '/api/video/' + videoId + '/comment/';
+  let response = apiRequest(apiEndpoint, 'GET');
+  let allComments = response.data;
+
+  writeComments(allComments);
+}
+
+function writeComments(allComments) {
+  let commentsListBox = document.getElementById('comments-list');
+  for (let i = 0; i < allComments.length; i++) {
+    const rootComment = allComments[i];
+
+    let commentBox = createCommentBox(rootComment, true);
+
+    // add replies to commentBox
+    if (rootComment.comment_replies) {
+      let commentReplyBox = document.createElement('div');
+      commentReplyBox.setAttribute('class', 'comments-replies');
+      commentReplyBox.setAttribute('id', rootComment.comment_id + '-replies');
+      let totalReplies = rootComment.comment_replies.length;
+      if (totalReplies > 0) {
+        let replyButton = createReplyButton(rootComment.comment_id + '-replies', totalReplies);
+        commentBox.appendChild(replyButton);
+      }
+      for (let j = 0; j < totalReplies; j++) {
+        const commentReply = rootComment.comment_replies[j];
+        let commentReplyDiv = createCommentBox(commentReply, false);
+        commentReplyBox.appendChild(commentReplyDiv);
+      }
+      if (totalReplies > 0) {
+        commentBox.appendChild(commentReplyBox);
+      }
+    }
+    commentsListBox.appendChild(commentBox);
+  }
+}
+
+function createReplyButton(replyId, totalReplies) {
+  let replyButton = document.createElement('button');
+  replyButton.innerHTML = `<span id="toggle-icon">+</span> ${totalReplies} replies`;
+  replyButton.setAttribute('data-id', replyId);
+  replyButton.setAttribute('onclick', 'toggleCommentReplies(this)');
+  return replyButton;
+}
+
+function toggleCommentReplies(button) {
+  let commentReplyId = button.getAttribute('data-id');
+  let state = document.getElementById(commentReplyId).style.display;
+
+  if (state === 'none' || state === '') {
+    document.getElementById(commentReplyId).style.display = 'block';
+    button.querySelector('#toggle-icon').innerHTML = '-';
+  } else {
+    document.getElementById(commentReplyId).style.display = 'none';
+    button.querySelector('#toggle-icon').innerHTML = '+';
+  }
+}
+
+function createCommentBox(comment, isRoot) {
+  let commentBox = document.createElement('div');
+  commentBox.setAttribute('class', 'comment-box');
+
+  let commentClass;
+  if (isRoot) {
+    commentClass = 'root-comment';
+  } else {
+    commentClass = 'reply-comment';
+  }
+
+  commentBox.classList.add = commentClass;
+
+  let commentAuthor = document.createElement('h3');
+  commentAuthor.innerText = comment.comment_author;
+  if (comment.comment_author_is_uploader) {
+    commentAuthor.setAttribute('class', 'comment-highlight');
+  }
+  commentBox.appendChild(commentAuthor);
+
+  let commentText = document.createElement('p');
+  commentText.innerText = comment.comment_text;
+  commentBox.appendChild(commentText);
+
+  const spacer = '<span class="space-carrot">|</span>';
+  let commentMeta = document.createElement('div');
+  commentMeta.setAttribute('class', 'comment-meta');
+
+  commentMeta.innerHTML = `<span>${comment.comment_time_text}</span>`;
+
+  if (comment.comment_likecount > 0) {
+    let numberFormatted = formatNumbers(comment.comment_likecount);
+    commentMeta.innerHTML += `${spacer}<span class="thumb-icon"><img src="/static/img/icon-thumb.svg"> ${numberFormatted}</span>`;
+  }
+
+  if (comment.comment_is_favorited) {
+    commentMeta.innerHTML += `${spacer}<span class="comment-like"><img src="/static/img/icon-heart.svg"></span>`;
+  }
+
+  commentBox.appendChild(commentMeta);
+
+  return commentBox;
+}
+
+function getSimilarVideos(videoId) {
+  let apiEndpoint = '/api/video/' + videoId + '/similar/';
+  let response = apiRequest(apiEndpoint, 'GET');
+  if (!response) {
+    populateEmpty();
+    return;
+  }
+  let allSimilar = response.data;
+  if (allSimilar.length > 0) {
+    populateSimilar(allSimilar);
+  }
+}
+
+function populateSimilar(allSimilar) {
+  let similarBox = document.getElementById('similar-videos');
+  for (let i = 0; i < allSimilar.length; i++) {
+    const similarRaw = allSimilar[i];
+    let similarDiv = createVideo(similarRaw, 'grid');
+    similarBox.appendChild(similarDiv);
+  }
+}
+
+function populateEmpty() {
+  let similarBox = document.getElementById('similar-videos');
+  let emptyMessage = document.createElement('p');
+  emptyMessage.innerText = 'No similar videos found.';
+  similarBox.appendChild(emptyMessage);
+}
+
 // generic
 
 function sendPost(payload) {