remove redundant latest tags

Changed
- lock creating superuser at startup by @dshoreman
- auto validate cookie
- API: return cookie validation state for browser extension
- Finetuning CSS mobile layout

CONTRIBUTING.md

@@ -5,13 +5,13 @@ If you haven't already, the best place to start is the README. This will give yo
 
 ## Report a bug
 
-If you notice something is not working as expected, check to see if it has been previously reported in the [open issues](https://github.com/bbilly1/tubearchivist/issues).
+If you notice something is not working as expected, check to see if it has been previously reported in the [open issues](https://github.com/tubearchivist/tubearchivist/issues).
 If it has not yet been disclosed, go ahead and create an issue.  
 If the issue doesn't move forward due to a lack of response, I assume it's solved and will close it after some time to keep the list fresh. 
 
 ## Wiki
 
-The wiki is where all user functions are explained in detail. These pages are mirrored into the **docs** folder of the repo. This allows for pull requests and all other features like regular code. Make any changes there, and I'll sync them with the wiki tab.
+The wiki is where all user functions are documented in detail. These pages are mirrored into the **docs** folder of the repo. This allows for pull requests and all other features like regular code. Make any changes there, and I'll sync them with the wiki tab.
 
 ## Development Environment
 
@@ -19,7 +19,7 @@ I have learned the hard way, that working on a dockerized application outside of
 
 This is my setup I have landed on, YMMV:
 - Clone the repo, work on it with your favorite code editor in your local filesystem. *testing* branch is the where all the changes are happening, might be unstable and is WIP.
-- Then I have a VM on KVM hypervisor running standard Ubuntu Server LTS with docker installed. The VM keeps my projects separate and offers convenient snapshot functionality. The VM also offers ways to simulate lowend environments by limiting CPU cores and memory. But you could also just run docker on your host system.
+- Then I have a VM running standard Ubuntu Server LTS with docker installed. The VM keeps my projects separate and offers convenient snapshot functionality. The VM also offers ways to simulate lowend environments by limiting CPU cores and memory. You can use this [Ansible Docker Ubuntu](https://github.com/bbilly1/ansible-playbooks) playbook to get started quickly. But you could also just run docker on your host system.
 - The `Dockerfile` is structured in a way that the actual application code is in the last layer so rebuilding the image with only code changes utilizes the build cache for everything else and will just take a few seconds.
 - Take a look at the `deploy.sh` file. I have my local DNS resolve `tubearchivist.local` to the IP of the VM for convenience. To deploy the latest changes and rebuild the application to the testing VM run:
 ```bash
@@ -27,16 +27,16 @@ This is my setup I have landed on, YMMV:
 ```
 - The command above will call the docker build command with `--build-arg INSTALL_DEBUG=1` to install additional useful debug tools.
 - The `test` argument takes another optional argument to build for a specific architecture valid options are: `amd64`, `arm64` and `multi`, default is `amd64`.
-- This `deploy.sh` file is not meant to be universally usable for every possible environment but could serve as an idea on how to automatically rebuild containers to test changes - customize to your liking. 
+- This `deploy.sh` script is not meant to be universally usable for every possible environment but could serve as an idea on how to automatically rebuild containers to test changes - customize to your liking. 
 
 ## Working with Elasticsearch
 Additionally to the required services as listed in the example docker-compose file, the **Dev Tools** of [Kibana](https://www.elastic.co/guide/en/kibana/current/docker.html) are invaluable for running and testing Elasticsearch queries.  
 
-If you want to run queries in on the Elasticsearch container directly from your host with for example `curl` or something like *postman*, you might want to **publish** the port 9200 instead of just **exposing** it.
+If you want to run queries on the Elasticsearch container directly from your host with for example `curl` or something like *postman*, you might want to **publish** the port 9200 instead of just **exposing** it.
 
 ## Implementing a new feature
 
-Do you see anything on the roadmap that you would like to take a closer look at but you are not sure, what's the best way to tackle that? Or anything not on there yet you'd like to implement but are not sure how? Open up an issue and we try to find a solution together.
+Do you see anything on the roadmap that you would like to take a closer look at but you are not sure, what's the best way to tackle that? Or anything not on there yet you'd like to implement but are not sure how? Reach out on Discord and we'll look into it together.
 
 ## Making changes
 
@@ -53,10 +53,10 @@ If you want to see what's in your container, checkout the matching release tag.
 
 ## Code formatting and linting
 
-To keep things clean and consistent for everybody, there is a github action setup to lint and check the changes. You can test your code locally first if you want. For example if you made changes in the **download** module, run
+To keep things clean and consistent for everybody, there is a github action setup to lint and check the changes. You can test your code locally first if you want. For example if you made changes in the **video** module, run
 
 ```shell
-./deploy.sh validate tubearchivist/home/src/download.py
+./deploy.sh validate tubearchivist/home/src/index/video.py
 ```
 
-to validate your changes. If you omit the path, all the project files will get checked. This is subject to change as the codebase improves. 
+to validate your changes. If you omit the path, all the project files will get checked. This is subject to change as the codebase improves. 

Dockerfile

@@ -1,17 +1,36 @@
-# build the tube archivist image from default python slim image
+# multi stage to build tube archivist
+# first stage to build python wheel, copy into final image
+
+
+# First stage to build python wheel
+FROM python:3.10.5-slim-bullseye AS builder
+ARG TARGETPLATFORM
+
+RUN apt-get update
+RUN apt-get install -y --no-install-recommends build-essential gcc
+
+# install requirements
+COPY ./tubearchivist/requirements.txt /requirements.txt
+RUN pip install --user -r requirements.txt
+
+# build final image
+FROM python:3.10.5-slim-bullseye as tubearchivist
 
-FROM python:3.10.3-slim-bullseye
 ARG TARGETPLATFORM
 ARG INSTALL_DEBUG
 
 ENV PYTHONUNBUFFERED 1
 
+# copy build requirements
+COPY --from=builder /root/.local /root/.local
+ENV PATH=/root/.local/bin:$PATH
+
 # install distro packages needed
 RUN apt-get clean && apt-get -y update && apt-get -y install --no-install-recommends \
-    build-essential \
     nginx \
     atomicparsley \
-    curl && rm -rf /var/lib/apt/lists/*
+    curl \
+    xz-utils && rm -rf /var/lib/apt/lists/*
 
 # get newest patched ffmpeg and ffprobe builds for amd64 fall back to repo ffmpeg for arm64
 RUN if [ "$TARGETPLATFORM" = "linux/amd64" ] ; then \
@@ -31,7 +50,7 @@ RUN if [ "$TARGETPLATFORM" = "linux/amd64" ] ; then \
 RUN if [ "$INSTALL_DEBUG" ] ; then \
         apt-get -y update && apt-get -y install --no-install-recommends \
         vim htop bmon net-tools iputils-ping procps \
-        && pip install --no-cache-dir ipython --src /usr/local/src \
+        && pip install --user ipython \
     ; fi
 
 # make folders
@@ -39,12 +58,8 @@ RUN mkdir /cache
 RUN mkdir /youtube
 RUN mkdir /app
 
-# install python dependencies
-COPY ./tubearchivist/requirements.txt /requirements.txt
-RUN pip install --no-cache-dir -r requirements.txt --src /usr/local/src
-
 # copy config files
-COPY docker_assets/nginx.conf /etc/nginx/conf.d/
+COPY docker_assets/nginx.conf /etc/nginx/sites-available/default
 
 # copy application into container
 COPY ./tubearchivist /app

README.md

@@ -1,13 +1,19 @@
 ![Tube Archivist](assets/tube-archivist-banner.jpg?raw=true "Tube Archivist Banner")  
 
-<center><h1>Your self hosted YouTube media server</h1></center>
+<h1 align="center">Your self hosted YouTube media server</h1>
+<div align="center">
+<a href="https://github.com/bbilly1/tilefy" target="_blank"><img src="https://tiles.tilefy.me/t/tubearchivist-docker.png" alt="tubearchivist-docker" title="Tube Archivist Docker Pulls" height="50" width="200"/></a>
+<a href="https://github.com/bbilly1/tilefy" target="_blank"><img src="https://tiles.tilefy.me/t/tubearchivist-github-star.png" alt="tubearchivist-github-star" title="Tube Archivist GitHub Stars" height="50" width="200"/></a>
+<a href="https://github.com/bbilly1/tilefy" target="_blank"><img src="https://tiles.tilefy.me/t/tubearchivist-github-forks.png" alt="tubearchivist-github-forks" title="Tube Archivist GitHub Forks" height="50" width="200"/></a>
+</div>
 
 ## Table of contents:
-* [Wiki](https://github.com/bbilly1/tubearchivist/wiki) for a detailed documentation, with [FAQ](https://github.com/bbilly1/tubearchivist/wiki/FAQ)
+* [Wiki](https://github.com/tubearchivist/tubearchivist/wiki) with [FAQ](https://github.com/tubearchivist/tubearchivist/wiki/FAQ)
 * [Core functionality](#core-functionality)
 * [Screenshots](#screenshots)
 * [Problem Tube Archivist tries to solve](#problem-tube-archivist-tries-to-solve)
 * [Connect](#connect)
+* [Extended Universe](#extended-universe)
 * [Installing and updating](#installing-and-updating)
 * [Getting Started](#getting-started)
 * [Potential pitfalls](#potential-pitfalls)
@@ -24,6 +30,9 @@
 * Play videos
 * Keep track of viewed and unviewed videos
 
+## Tube Archivist on YouTube
+[![ibracorp-youtube-video-thumb](assets/tube-archivist-ibracorp-O8H8Z01c0Ys.jpg)](https://www.youtube.com/watch?v=O8H8Z01c0Ys)
+
 ## Screenshots
 ![home screenshot](assets/tube-archivist-screenshot-home.png?raw=true "Tube Archivist Home")  
 *Home Page*
@@ -44,8 +53,12 @@
 Once your YouTube video collection grows, it becomes hard to search and find a specific video. That's where Tube Archivist comes in: By indexing your video collection with metadata from YouTube, you can organize, search and enjoy your archived YouTube videos without hassle offline through a convenient web interface.
 
 ## Connect
-- [Discord](https://discord.gg/AFwz8nE7BK): Connect with us on our brand new Discord server.
-- [r/TubeArchivist](https://www.reddit.com/r/TubeArchivist/): Join our brand new Subreddit.
+- [Discord](https://discord.gg/AFwz8nE7BK): Connect with us on our Discord server.
+- [r/TubeArchivist](https://www.reddit.com/r/TubeArchivist/): Join our Subreddit.
+
+## Extended Universe
+- [Browser Extension](https://github.com/tubearchivist/browser-extension) Tube Archivist Companion, for [Firefox](https://addons.mozilla.org/addon/tubearchivist-companion/) and [Chrome](https://chrome.google.com/webstore/detail/tubearchivist-companion/jjnkmicfnfojkkgobdfeieblocadmcie)
+- [Tube Archivist Metrics](https://github.com/tubearchivist/tubearchivist-metrics) to create statistics in Prometheus/OpenMetrics format.
 
 ## Installing and updating
 Take a look at the example `docker-compose.yml` file provided. Use the *latest* or the named semantic version tag. The *unstable* tag is for intermediate testing and as the name implies, is **unstable** and not be used on your main installation but in a [testing environment](CONTRIBUTING.md).  
@@ -73,6 +86,10 @@ Should that not be an option, the Tube Archivist container takes these two addit
 Changing any of these two environment variables will change the files *nginx.conf* and *uwsgi.ini* at startup using `sed` in your container.
 
 ### Elasticsearch
+**Note**: Newest Tube Archivist depends on Elasticsearch version 7.17 to provide an automatic updatepath in the future. 
+
+Use `bbilly1/tubearchivist-es` to automatically get the recommended version, or use the official image with the version tag in the docker-compose file.
+
 Stores video meta data and makes everything searchable. Also keeps track of the download queue.
   - Needs to be accessible over the default port `9200`
   - Needs a volume at **/usr/share/elasticsearch/data** to store data
@@ -93,11 +110,11 @@ For some architectures it might be required to run Redis JSON on a nonstandard p
 ### Updating Tube Archivist
 You will see the current version number of **Tube Archivist** in the footer of the interface so you can compare it with the latest release to make sure you are running the *latest and greatest*.  
 * There can be breaking changes between updates, particularly as the application grows, new environment variables or settings might be required for you to set in the your docker-compose file. *Always* check the **release notes**: Any breaking changes will be marked there.  
-* All testing and development is done with the Elasticsearch version number as mentioned in the provided *docker-compose.yml* file. This will be updated when a new release of Elasticsearch is available. Running an older version of Elasticsearch is most likely not going to result in any issues, but it's still recommended to run the same version as mentioned.
+* All testing and development is done with the Elasticsearch version number as mentioned in the provided *docker-compose.yml* file. This will be updated when a new release of Elasticsearch is available. Running an older version of Elasticsearch is most likely not going to result in any issues, but it's still recommended to run the same version as mentioned. Use `bbilly1/tubearchivist-es` to automatically get the recommended version.
 
 ### Alternative installation instructions:
 - **arm64**: The Tube Archivist container is multi arch, so is Elasticsearch. RedisJSON doesn't offer arm builds, you can use `bbilly1/rejson`, an unofficial rebuild for arm64.
-- **Synology**: There is a [discussion thread](https://github.com/bbilly1/tubearchivist/discussions/48) with Synology installation instructions.
+- **Synology**: There is a [discussion thread](https://github.com/tubearchivist/tubearchivist/discussions/48) with Synology installation instructions.
 - **Unraid**: The three containers needed are all in the Community Applications. First install `TubeArchivist RedisJSON` followed by `TubeArchivist ES`, and finally you can install `TubeArchivist`. If you have unraid specific issues, report those to the [support thread](https://forums.unraid.net/topic/114073-support-crocs-tube-archivist/ "support thread").
 - **Helm Chart**: There is a Helm Chart available at https://github.com/insuusvenerati/helm-charts. Mostly self-explanatory but feel free to ask questions in the discord / subreddit.
 
@@ -125,7 +142,9 @@ chown 1000:0 /path/to/mount/point
 This will match the permissions with the **UID** and **GID** of elasticsearch within the container and should fix the issue.
 
 ### Disk usage
-The Elasticsearch index will turn to *read only* if the disk usage of the container goes above 95% until the usage drops below 90% again. Similar to that, TubeArchivist will become all sorts of messed up when running out of disk space. There are some error messages in the logs when that happens, but it's best to make sure to have enough disk space before starting to download.
+The Elasticsearch index will turn to *read only* if the disk usage of the container goes above 95% until the usage drops below 90% again, you will see error messages like `disk usage exceeded flood-stage watermark`, [link](https://github.com/tubearchivist/tubearchivist#disk-usage).  
+
+Similar to that, TubeArchivist will become all sorts of messed up when running out of disk space. There are some error messages in the logs when that happens, but it's best to make sure to have enough disk space before starting to download.
 
 ## Getting Started
 1. Go through the **settings** page and look at the available options. Particularly set *Download Format* to your desired video quality before downloading. **Tube Archivist** downloads the best available quality by default. To support iOS or MacOS and some other browsers a compatible format must be specified. For example:
@@ -138,24 +157,30 @@ bestvideo[VCODEC=avc1]+bestaudio[ACODEC=mp4a]/mp4
 5. Enjoy your archived collection!
   
 ## Roadmap
-We have come far, nonetheless we are not short of ideas on how to improve and extend this project:
+We have come far, nonetheless we are not short of ideas on how to improve and extend this project. Issues waiting for you to be tackled in no particular order:
 
 - [ ] User roles
 - [ ] Podcast mode to serve channel as mp3
 - [ ] Implement [PyFilesystem](https://github.com/PyFilesystem/pyfilesystem2) for flexible video storage
-- [ ] Implement [Apprise](https://github.com/caronc/apprise) for notifications
-- [ ] Add [SponsorBlock](https://sponsor.ajay.app/) integration
-- [ ] Implement per channel settings
-- [ ] User created playlists
-- [ ] Auto play or play next link
-- [ ] SSO / LDAP support
+- [ ] Implement [Apprise](https://github.com/caronc/apprise) for notifications ([#97](https://github.com/tubearchivist/tubearchivist/issues/97))
+- [ ] User created playlists, random and repeat controls ([#108](https://github.com/tubearchivist/tubearchivist/issues/108), [#220](https://github.com/tubearchivist/tubearchivist/issues/220))
+- [ ] Auto play or play next link ([#226](https://github.com/tubearchivist/tubearchivist/issues/226))
 - [ ] Show similar videos on video page
 - [ ] Multi language support
 - [ ] Show total video downloaded vs total videos available in channel
-- [ ] Make items in grid row configurable
 - [ ] Add statistics of index
+- [ ] Download speed schedule ([#198](https://github.com/tubearchivist/tubearchivist/issues/198))
+- [ ] Implement complete offline media file import from json file ([#138](https://github.com/tubearchivist/tubearchivist/issues/138))
+- [ ] Filter and query in search form, search by url query ([#134](https://github.com/tubearchivist/tubearchivist/issues/134), [#139](https://github.com/tubearchivist/tubearchivist/issues/139))
+- [ ] Auto ignore videos by keyword ([#163](https://github.com/tubearchivist/tubearchivist/issues/163))
+- [ ] Custom searchable notes to videos, channels, playlists ([#144](https://github.com/tubearchivist/tubearchivist/issues/144))
+- [ ] Download video comments
 
 Implemented:
+- [X] Make items in grid row configurable to use more of the screen [2022-06-04]
+- [X] Add passing browser cookies to yt-dlp [2022-05-08]
+- [X] Add [SponsorBlock](https://sponsor.ajay.app/) integration [2022-04-16]
+- [X] Implement per channel settings [2022-03-26]
 - [X] Subtitle download & indexing [2022-02-13]
 - [X] Fancy advanced unified search interface [2022-01-08]
 - [X] Auto rescan and auto download on a schedule [2021-12-17]

browser_extension/README.md

@@ -1,26 +0,0 @@
-# Tube Archivist Companion
-A browser extension to directly add videos from YouTube to Tube Archivist.
-
-## MVP or better *bearly viable product*
-This is a proof of concept with the following functionality:
-- Add your Tube Archivist connection details in the addon popup
-- Inject a download button into youtube search results page
-- Clicking the button will automatically add the video to the your download queue
-
-## Test this extension
-- Firefox
-  - Open `about:debugging#/runtime/this-firefox`
-  - Click on *Load Temporary Add-on*
-  - Select the *manifest.json* file to load the addon. 
-- Chrome / Chromium
-  - Open `chrome://extensions/`
-  - Toggle *Developer mode* on top right
-  - Click on *Load unpacked*
-  - Open the folder containing the *manifest.json* file.
-
-## Help needed
-This is only minimally useful in this state. Join us on our Discord and please help us improve that.
-
-## Note:
-- For mysterious reasons sometimes the download buttons will only load when refreshing the YouTube search page and not on first load... Hence: Help needed!
-- For your testing environment only for now: Point the extension to the newest *unstable* build.

browser_extension/extension/background.js

@@ -1,94 +0,0 @@
-/*
-extension background script listening for events
-*/
-
-console.log("running background.js");
-
-let browserType = getBrowser();
-
-
-// boilerplate to dedect browser type api
-function getBrowser() {
-    if (typeof chrome !== "undefined") {
-        if (typeof browser !== "undefined") {
-            console.log("detected firefox");
-            return browser;
-        } else {
-            console.log("detected chrome");
-            return chrome;
-        }
-    } else {
-        console.log("failed to dedect browser");
-        throw "browser detection error"
-    };
-}
-
-
-// send post request to API backend
-async function sendPayload(url, token, payload) {
-
-    const rawResponse = await fetch(url, {
-        method: "POST",
-        headers: {
-            "Accept": "application/json",
-            "Content-Type": "application/json",
-            "Authorization": token,
-            "mode": "no-cors"
-        },
-        body: JSON.stringify(payload)
-    });
-
-    const content = await rawResponse.json();
-    return content;
-}
-
-
-// read access storage and send
-function forwardRequest(payload) {
-
-    console.log("running forwardRequest");
-
-    function onGot(item) {
-        console.log(item.access);
-
-        const url = `${item.access.url}:${item.access.port}/api/download/`;
-        console.log(`sending to ${url}`);
-        const token = `Token ${item.access.apiKey}`;
-
-        sendPayload(url, token, payload).then(content => {
-            console.log(content);
-        })
-
-    };
-
-    function onError(error) {
-        console.local("failed to get access details");
-        console.log(`Error: ${error}`);
-    };
-
-    browserType.storage.local.get("access", function(result) {
-        onGot(result)
-    });
-
-}
-
-
-// listen for messages
-browserType.runtime.onMessage.addListener(
-    function(request, sender, sendResponse) {
-        console.log("responding from background.js listener");
-        console.log(JSON.stringify(request));
-        if (request.download) {
-            console.log("found new download task");
-            let payload = {
-                "data": [
-                    {
-                        "youtube_id": request.download["videoId"],
-                        "status": "pending",
-                    }
-                ]
-            }
-            forwardRequest(payload);
-        };
-    }
-);

browser_extension/extension/images/logo.svg

@@ -1,102 +0,0 @@
-<?xml version="1.0" encoding="utf-8"?>
-<!-- Generator: Adobe Illustrator 26.0.2, SVG Export Plug-In . SVG Version: 6.00 Build 0)  -->
-<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
-	 viewBox="0 0 350 75" style="enable-background:new 0 0 350 75;" xml:space="preserve">
-<style type="text/css">
-	.st0{fill:#FFFFFF;}
-	.st1{fill:#2C9D8A;}
-</style>
-<g id="Layer_1_00000061458428874772671790000018384579824765206440_">
-	<g id="archivist">
-		<path class="st0" d="M218.5,15.2v4.6c-0.7-0.5-1.5-0.8-2.5-0.8c-1.8,0-3,1.4-3.9,3.1v8.9h-4v-16h4v3c1.1-1.9,2.6-3.3,4.5-3.3
-			C217.3,14.7,217.9,14.9,218.5,15.2z"/>
-		<path class="st0" d="M233.6,16.3V20c-1.1-0.8-2.4-1.4-4.4-1.4c-2.4,0-4.5,1.9-4.5,4.5c0,2.6,1.9,4.5,4.5,4.5c2,0,3.3-0.6,4.4-1.4
-			v3.7c-0.9,0.8-2.3,1.6-4.7,1.6c-4.8,0-8.4-3.9-8.4-8.4c0-4.5,3.6-8.5,8.4-8.5C231.3,14.6,232.7,15.6,233.6,16.3z"/>
-		<path class="st0" d="M251.8,21.2v9.8h-4l0-9.3c0-2.2-1.1-3.2-2.9-3.2c-1.4,0-2.7,1.1-3.6,2.3v10.1h-4V8.2h4v9.1
-			c1-1.3,2.7-2.7,4.8-2.7C249.3,14.6,251.8,16.9,251.8,21.2z"/>
-		<path class="st0" d="M262.1,31.1h-4V18.5h-2.5v-3.4h6.5V31.1z M256.9,10.1c0-1.5,1.2-2.6,2.6-2.6s2.7,1.2,2.7,2.6
-			c0,1.5-1.2,2.7-2.7,2.7S256.9,11.6,256.9,10.1z"/>
-		<path class="st0" d="M272.2,31.1l-7.5-16h4.7l4,9.9l4.2-9.9h4.7l-7.6,16H272.2z"/>
-		<path class="st0" d="M290.8,31.1h-4V18.5h-2.5v-3.4h6.5V31.1z M285.6,10.1c0-1.5,1.2-2.6,2.6-2.6c1.5,0,2.7,1.2,2.7,2.6
-			c0,1.5-1.2,2.7-2.7,2.7C286.8,12.8,285.6,11.6,285.6,10.1z"/>
-		<path class="st0" d="M305.5,16.1v3.5c-1-0.7-2.5-1.5-4.1-1.5c-1.5,0-2.5,0.8-2.5,1.7c0,0.7,0.4,1.2,1.8,1.7l1.8,0.6
-			c2.5,0.7,3.6,2.2,3.6,4.2c0,2.9-2.8,5.1-6.2,5.1c-2.2,0-4.4-0.8-5.5-1.6v-3.5c1.2,0.8,3.4,1.6,4.9,1.6c1.6,0,2.5-0.6,2.5-1.6
-			c0-0.8-0.7-1.2-2.1-1.6l-2-0.7c-2.4-0.8-3.2-2.3-3.2-4.3c0-3,2.5-5.1,6.1-5.1C302.7,14.6,304.5,15.3,305.5,16.1z"/>
-		<path class="st0" d="M319.8,15.1v3.4h-4.4v6.9c0,1.4,0.7,2.3,2,2.3c1.1,0,1.9-0.3,2.6-0.7v3.2c-0.8,0.8-1.9,1.4-3.8,1.4
-			c-2.9,0-4.8-2-4.8-5.2v-8h-2.9v-2.1c2.1-0.7,4.2-2.7,5.2-6h1.7v4.7H319.8z"/>
-		<polygon id="A" class="st0" points="183.9,31.1 192.7,8 196.6,8 205.4,31.1 201,31.1 194.7,13.5 188.3,31.1 		"/>
-		<path class="st1" d="M197.7,26.2l-1.8,1.6v-5.2v-0.7v-1.8c0-0.8-0.6-1.4-1.3-1.4s-1.3,0.6-1.3,1.4v1.8v0.7v5.2l-1.8-1.6l-1.2,1.2
-			l4.3,3.9l4.3-3.9L197.7,26.2z"/>
-	</g>
-	<g id="tube">
-		<g id="T_00000039852614045324270430000005353782644941079486_">
-			<polygon class="st0" points="94.7,8.3 94.7,12.3 101.3,12.3 101.3,18.4 105.5,18.4 105.5,12.3 111.9,12.3 111.9,8.3 			"/>
-			<rect x="101.3" y="22.6" class="st0" width="4.1" height="9"/>
-			<path class="st1" d="M105.8,21.7H101c-0.7,0-1.3-0.6-1.3-1.3l0,0c0-0.7,0.6-1.3,1.3-1.3h4.8c0.7,0,1.3,0.6,1.3,1.3l0,0
-				C107.1,21.1,106.5,21.7,105.8,21.7z"/>
-		</g>
-		<path class="st0" d="M127.4,24.5c0,4.2-3,6.8-7.1,6.8c-4.1,0-7.1-2.6-7.1-6.8v-9.4h4l0,8.7c0,2.4,1.3,3.7,3.1,3.7
-			c1.8,0,3.1-1.2,3.1-3.7v-8.7h4V24.5z"/>
-		<path class="st0" d="M136,17.3c1-1.8,2.8-3.1,5.2-3.1c3.9,0,7.3,3,7.3,8.4c0,5.4-3.4,8.7-7.5,8.7c-1.8,0-3.7-1-4.9-2.3v1.8h-4.1
-			V7.7h4.1V17.3z M136,22.9v2.3c1.2,1.2,2.5,2.1,4.1,2.1c2.5,0,4.1-2.3,4.1-4.7c0-2.4-1.6-4.3-4-4.3C138,18.3,135.9,20,136,22.9z"/>
-		<path class="st0" d="M167,24.1h-11.7c0.4,2.2,2.3,3.7,4.8,3.7c2.4,0,3.9-1,4.9-1.6v3.7c-0.9,0.7-2.6,1.6-5.1,1.6
-			c-4.8,0-8.7-3.5-8.7-8.4c0-4.7,3.6-8.7,8.2-8.7c4.8,0,7.6,3.3,7.6,8.1V24.1z M163.1,21.5c-0.2-2.3-2-3.5-3.8-3.5
-			c-2.1,0-3.5,1.4-3.9,3.5H163.1z"/>
-	</g>
-	<g id="icon">
-		<path class="st1" d="M27.6,33.7l1,0.4l0.5-2l-6.7-2l-0.7,2l0.9,0.4C20.9,44.4,25,54.5,34,61.9l2.9-4
-			C29.6,51.4,25.8,43.2,27.6,33.7z"/>
-		<path class="st1" d="M41.2,60.5l0.6-0.8L40,58.4L35.7,64l1.8,1.3l0.6-0.7c10.7,5.7,21.9,4.7,31.8-1.5L66.8,59
-			C58.3,63.9,49.9,65.3,41.2,60.5z"/>
-		<path class="st1" d="M70.8,55.6l-0.7-0.7l-1.8,1.3l4.3,5.9l1.8-1.3L73.9,60c8.7-8.5,11.2-18.7,8.4-30l-4.9,1.4
-			C79.3,40.9,78,48.7,70.8,55.6z"/>
-		<path class="st1" d="M75.5,26.5l-0.9,0.4l0.6,2.4l7.2-2.1l-0.7-2.4l-0.9,0.3c-5.7-10.7-15-17.4-26.6-17.9v5.4
-			C63.8,13.4,70.9,17.6,75.5,26.5z"/>
-		<path class="st1" d="M49,12.8l0.1,0.9l2.3-0.1v-7h-2.3v0.9C40,8.8,28.6,14.8,24,27.6l4.9,1.4C32.4,19,40.2,14.7,49,12.8z"/>
-		<polygon id="A_00000111884739883312047940000001222474654974762168_" class="st0" points="48,54 60.6,20.9 66.1,20.9 78.8,54 
-			72.5,54 63.4,28.8 54.3,54 		"/>
-		<g id="T">
-			<polygon class="st0" points="27.6,20.4 27.6,26.1 37.1,26.1 37.1,35 43.1,35 43.1,26.1 52.4,26.1 52.4,20.4 			"/>
-			<rect x="37.1" y="41" class="st0" width="6" height="13"/>
-		</g>
-		<path class="st1" d="M43.5,39.7h-6.9c-1,0-1.9-0.8-1.9-1.9l0,0c0-1,0.8-1.9,1.9-1.9h6.9c1,0,1.9,0.8,1.9,1.9l0,0
-			C45.4,38.9,44.5,39.7,43.5,39.7z"/>
-		<path class="st1" d="M67.8,47l-2.5,2.3v-7.5v-1v-2.5c0-1.1-0.8-2-1.9-2s-1.9,0.9-1.9,2v2.5v1v7.5L58.9,47l-1.7,1.7l6.2,5.6
-			l6.1-5.6L67.8,47z"/>
-	</g>
-</g>
-<g id="Layer_2">
-	<g>
-		<path class="st0" d="M142.1,58c0,6-4.3,9.6-10.1,9.6c-5.8,0-10.1-3.6-10.1-9.6V44.7h5.7l0,12.3c0,3.4,1.8,5.2,4.4,5.2
-			c2.6,0,4.4-1.7,4.4-5.2V44.7h5.7V58z"/>
-		<path class="st0" d="M154.1,48.5c1.3-2.4,3.8-4.3,7.2-4.3c5.4,0,10,4.1,10,11.6c0,7.4-4.6,12-10.4,12c-2.4,0-5.1-1.3-6.8-3.2V67
-			h-5.6V35.2h5.6V48.5z M154.1,56.1v3.2c1.6,1.7,3.4,2.9,5.6,2.9c3.5,0,5.7-3.2,5.7-6.5c0-3.4-2.2-6-5.6-6
-			C156.9,49.7,154.1,52.1,154.1,56.1z"/>
-		<path class="st0" d="M190.4,46.2v5c-1.4-0.9-3.5-2-5.7-2c-2.1,0-3.4,1.1-3.4,2.4c0,1,0.6,1.7,2.6,2.3l2.5,0.8c3.4,1,5.1,3,5.1,5.9
-			c0,4.1-3.9,7.2-8.7,7.2c-3,0-6.1-1.1-7.6-2.3v-4.9c1.7,1.1,4.7,2.3,6.9,2.3c2.3,0,3.5-0.8,3.5-2.3c0-1.2-1-1.6-3-2.3l-2.7-1
-			c-3.4-1.1-4.5-3.2-4.5-6.1c0-4.2,3.5-7.1,8.6-7.1C186.6,44.1,189.2,45.1,190.4,46.2z"/>
-		<path class="st0" d="M212.9,46.5v5.2c-1.5-1.1-3.4-2-6.1-2c-3.4,0-6.2,2.7-6.2,6.3s2.6,6.3,6.2,6.3c2.7,0,4.6-0.8,6.1-1.9v5.1
-			c-1.2,1.1-3.3,2.3-6.6,2.3c-6.7,0-11.7-5.4-11.7-11.7s5-11.9,11.7-11.9C209.7,44.1,211.7,45.5,212.9,46.5z"/>
-		<path class="st0" d="M232.5,44.9v6.4c-1-0.7-2.1-1.1-3.4-1.1c-2.6,0-4.2,1.9-5.5,4.4V67h-5.6V44.7h5.6v4.2
-			c1.5-2.7,3.6-4.6,6.3-4.6C230.9,44.3,231.7,44.5,232.5,44.9z"/>
-		<path class="st0" d="M244.7,67h-5.6V49.5h-3.5v-4.8h9.1V67z M237.5,37.8c0-2,1.6-3.7,3.7-3.7s3.7,1.6,3.7,3.7s-1.7,3.7-3.7,3.7
-			S237.5,39.8,237.5,37.8z"/>
-		<path class="st0" d="M257,48.5c1.3-2.4,3.8-4.3,7.2-4.3c5.4,0,10,4.1,10,11.6c0,7.4-4.6,12-10.4,12c-2.4,0-5.1-1.3-6.8-3.2V67
-			h-5.6V35.2h5.6V48.5z M257,56.1v3.2c1.6,1.7,3.4,2.9,5.6,2.9c3.5,0,5.7-3.2,5.7-6.5c0-3.4-2.2-6-5.6-6
-			C259.8,49.7,257,52.1,257,56.1z"/>
-		<path class="st0" d="M299.8,57.6h-16.2c0.6,3.1,3.1,5,6.5,5c3.3,0,5.4-1.4,6.8-2.3v5.1c-1.2,0.9-3.6,2.3-7,2.3
-			c-6.6,0-12-4.9-12-11.6c0-6.5,5-12,11.3-12c6.6,0,10.5,4.6,10.5,11.1V57.6z M294.4,54c-0.2-3.2-2.7-4.8-5.3-4.8
-			c-2.9,0-4.8,1.9-5.4,4.8H294.4z"/>
-		<path class="st0" d="M319.5,44.9v6.4c-1-0.7-2.1-1.1-3.4-1.1c-2.6,0-4.2,1.9-5.5,4.4V67H305V44.7h5.6v4.2c1.5-2.7,3.6-4.6,6.3-4.6
-			C317.9,44.3,318.7,44.5,319.5,44.9z"/>
-	</g>
-	<g>
-		<path class="st0" d="M102.6,54l2.4,0.9v-6.1c0-0.2,0-0.3,0.1-0.5c-1.2-0.6-1.9-1.4-1.9-2.6c0-2.3,2.6-3.2,4.9-3.2
-			c3.1,0,5.4,1.3,7.5,2.7v-6.1c-1.9-1.2-4.8-2.3-8.3-2.3c-5.5,0-10.5,3.5-10.5,9C96.7,49.3,98,52.3,102.6,54z"/>
-		<path class="st0" d="M111.1,51l-1.9-0.8v6.6c0.8,0.5,1.3,1.2,1.3,2.1c0,1.9-2,2.9-4.5,2.9c-3.5,0-6.7-1.5-9.3-3.4v6.1
-			c2.4,1.7,5.7,3.1,9.2,3.1c6.1,0,11.2-3.8,11.2-9C117.2,55.3,115.3,52.8,111.1,51z"/>
-	</g>
-</g>
-<path class="st1" d="M105.5,56.1v-6.9c0-1,0.8-1.9,1.6-1.9l0,0c0.9,0,1.6,0.9,1.6,1.9v6.9c0,1-0.8,1.9-1.6,1.9l0,0
-	C106.2,58,105.5,57.1,105.5,56.1z"/>
-</svg>

browser_extension/extension/images/question.svg

@@ -1,16 +0,0 @@
-<?xml version="1.0" encoding="utf-8"?>
-<!-- Generator: Adobe Illustrator 26.0.2, SVG Export Plug-In . SVG Version: 6.00 Build 0)  -->
-<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
-	 viewBox="0 0 374.4 375.1" style="enable-background:new 0 0 374.4 375.1;" xml:space="preserve">
-<style type="text/css">
-	.st0{fill:#FFFFFF;}
-</style>
-<path class="st0" d="M187.5,0C84.2-0.1,0.1,83.8,0,187.2C-0.2,290.8,83.7,375,187.1,375.1c103.3,0,187.2-83.9,187.3-187.4
-	C374.5,84.3,290.7,0.2,187.5,0z M187.9,304.5c-16,0-29.2-13.3-29.2-29.3c0.1-16,13.2-29,29.1-29.1c16,0,29.2,13.3,29.2,29.3
-	C216.9,291.4,203.8,304.5,187.9,304.5z M238.2,195.7c-4.8,2.9-9.7,5.8-14.5,8.7c-9.1,5.5-13.6,13.6-13.9,24.2c0,0.8,0,1.7,0,2.7
-	c-14.6,0-29,0-43.8,0c0.1-2.8,0.2-5.6,0.4-8.5c0.8-10.6,3.2-20.9,8.4-30.3c5.9-11,14.9-18.9,25.5-25.1c4.1-2.4,8.4-4.7,12.5-7.1
-	c12.7-7.7,13.1-19.1,6.2-29.9c-6.5-10.3-16.3-14.9-28.1-15.7c-8-0.5-15.7,0.7-22.6,4.9c-10.1,6.2-15,15.6-16.9,26.9
-	c-0.1,0.4-0.1,0.9-0.2,1.6c-14.5-1.9-28.9-3.8-43.9-5.8c1.4-5.4,2.5-10.8,4.2-15.9c5.2-14.9,13.6-27.8,25.8-38
-	c11.1-9.3,23.9-14.7,38.2-16.6c2.3-0.3,4.6-0.6,6.9-1c3.2,0,6.4,0,9.6,0c4.4,0.6,8.9,1.1,13.3,1.9c29.8,5.9,55.4,28.1,61.4,59.3
-	C271.5,157.5,260.7,182.2,238.2,195.7z"/>
-</svg>

browser_extension/extension/images/social/discord.svg

@@ -1,24 +0,0 @@
-<?xml version="1.0" encoding="utf-8"?>
-<!-- Generator: Adobe Illustrator 26.0.2, SVG Export Plug-In . SVG Version: 6.00 Build 0)  -->
-<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
-	 viewBox="0 0 384 384" style="enable-background:new 0 0 384 384;" xml:space="preserve">
-<style type="text/css">
-	.st0{fill:#FFFFFF;}
-</style>
-<g>
-	<path class="st0" d="M191.57,379.5C88.15,379.44,4.32,295.25,4.46,191.57C4.6,88.18,88.66,4.28,191.95,4.42
-		c103.29,0.14,187.02,84.23,186.93,187.72C378.8,295.6,294.86,379.55,191.57,379.5z M244.55,271.24
-		c4.89,5.91,9.44,11.19,13.73,16.69c1.8,2.32,3.72,3.11,6.64,2.93c28.39-1.75,53.17-11.56,72.96-32.55c1.12-1.19,2.08-3.2,1.97-4.76
-		c-0.79-11.71-1.27-23.48-2.97-35.07c-5.07-34.65-16.23-67.36-32.74-98.23c-0.73-1.36-1.88-2.67-3.15-3.55
-		c-19.34-13.43-40.61-21.62-64.16-23.63c-3.19-0.27-5.5,0.48-6.89,3.09c10.27,3.97,20.59,7.65,30.65,11.94
-		c10.1,4.32,19.64,9.74,28.21,16.92c-64.82-29.49-129.53-29.49-194.34,0.02c17.77-13.75,38.02-22.07,59.27-28.39
-		c-1.74-3.15-4.12-3.86-7.32-3.58c-23.15,2.02-44.13,9.94-63.2,23.08c-1.76,1.21-3.51,2.96-4.4,4.86
-		c-5.36,11.39-11.02,22.69-15.57,34.41c-12.22,31.43-19.35,64-19.75,97.86c-0.02,1.71,0.88,3.85,2.07,5.1
-		c18.74,19.74,41.98,29.72,68.88,32.19c5.73,0.53,9.64-0.46,12.96-5.26c3.36-4.86,7.35-9.28,11.2-14.05
-		c-22.46-8.05-39.37-18.53-42.93-26.17c63.97,36.11,128.17,36.13,192.51-0.02C276.69,258.45,261.71,265.95,244.55,271.24z"/>
-	<path class="st0" d="M237.28,232.74c-14.34-0.02-26.07-12-25.9-26.45c0.17-14.67,11.86-26.25,26.56-26.33
-		c13.63-0.07,25.18,12,25.21,26.35C263.18,220.98,251.66,232.76,237.28,232.74z"/>
-	<path class="st0" d="M146.11,232.74c-14.35,0.05-25.9-11.7-25.92-26.38c-0.02-14.3,11.56-26.46,25.16-26.41
-		c14.85,0.06,26.74,11.96,26.62,26.66C171.84,221.04,160.32,232.69,146.11,232.74z"/>
-</g>
-</svg>

browser_extension/extension/images/social/github.svg

@@ -1,23 +0,0 @@
-<?xml version="1.0" encoding="utf-8"?>
-<!-- Generator: Adobe Illustrator 26.0.2, SVG Export Plug-In . SVG Version: 6.00 Build 0)  -->
-<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
-	 viewBox="0 0 384 384" style="enable-background:new 0 0 384 384;" xml:space="preserve">
-<style type="text/css">
-	.st0{fill:#FFFFFF;}
-</style>
-<g>
-	<path class="st0" d="M244.51,375.15c0-10.59-0.31-21.12,0.08-31.61c0.5-13.65,0.02-26.89-10.41-37.88
-		c8.18-2.06,15.59-3.84,22.95-5.83c3.51-0.95,6.95-2.22,10.37-3.47c27.26-9.99,40.88-31.15,45.79-58.73
-		c2.71-15.21,4.11-30.82-0.74-45.88c-2.65-8.23-6.01-16.69-11.08-23.5c-3.83-5.14-3.43-9.05-2.23-14.27
-		c2.86-12.48,0.71-24.71-2.64-36.79c-0.99-3.58-3.09-4.99-6.88-4.6c-11.45,1.18-21.48,6.12-31.34,11.6
-		c-1.42,0.79-2.97,1.46-4.16,2.52c-5.55,4.99-11.4,5.13-18.58,3.42c-25.28-6.04-50.67-5.14-75.85,1.12
-		c-3.59,0.89-6.27,0.27-9.31-1.86c-11.08-7.73-22.87-14.02-36.37-16.31c-0.13-0.02-0.26-0.07-0.4-0.09
-		c-7.12-1.23-8.12-0.76-10.11,6.16c-3.51,12.23-5.18,24.6-1.55,37.14c1.14,3.94,0.44,7.16-2.01,10.61
-		c-23.14,32.46-16.36,80.65,0.11,103.41c12.16,16.81,29.02,26.2,48.63,31.06c5.49,1.36,11.11,2.15,17.1,3.28
-		c-0.24,0.79-0.43,1.81-0.85,2.71c-3.15,6.91-6.27,13.83-9.6,20.65c-0.67,1.38-2.06,3.14-3.34,3.35c-7.3,1.18-14.65,2.46-22.02,2.7
-		c-6.5,0.21-12.09-3.32-16.37-8c-5.11-5.59-9.65-11.73-14.19-17.83c-6.07-8.15-14.2-12.42-24.05-13.3c-2.22-0.2-4.58,1.12-7.99,2.05
-		c2.69,3.23,4.46,6.17,6.96,8.2c10.32,8.35,17.42,19.1,22.26,31.24c4.72,11.85,14.1,16.55,25.31,19.27
-		c9.24,2.24,18.51,1.5,27.83,0.78c6.07-0.47,6.09-0.22,6.09,5.98c0.01,5.58,0,11.17,0,16.81C86.7,368.22,7.24,301.96,4.67,198.28
-		C2.19,97.88,77.65,13.69,175.94,5.34c103.69-8.81,192.95,68.52,202.14,171.23C387.22,278.72,317.17,355.78,244.51,375.15z"/>
-</g>
-</svg>

browser_extension/extension/images/social/reddit.svg

@@ -1,35 +0,0 @@
-<?xml version="1.0" encoding="utf-8"?>
-<!-- Generator: Adobe Illustrator 26.0.2, SVG Export Plug-In . SVG Version: 6.00 Build 0)  -->
-<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
-	 viewBox="0 0 384 384" style="enable-background:new 0 0 384 384;" xml:space="preserve">
-<style type="text/css">
-	.st0{fill:#FFFFFF;}
-</style>
-<g>
-	<path class="st0" d="M240.82,274c-3.86,3.78-8.45,6.33-13.43,8.26c-9.77,3.79-19.98,5.34-30.38,5.85
-		c-5.42,0.26-10.85,0.12-16.26-0.44c-5.89-0.61-11.72-1.56-17.42-3.18c-7.65-2.17-14.86-5.18-20.68-10.86
-		c-0.98-0.96-2.26-1.44-3.64-1.48c-2.72-0.08-4.71,1.17-5.94,3.55c-1.14,2.2-0.8,4.87,0.94,6.7c1.28,1.35,2.7,2.59,4.17,3.73
-		c5.88,4.57,12.58,7.49,19.65,9.63c11.11,3.37,22.52,4.66,36.04,4.85c2.63-0.18,7.2-0.42,11.76-0.82c6.75-0.6,13.38-1.86,19.88-3.8
-		c6.56-1.95,12.83-4.56,18.42-8.57c1.94-1.4,3.82-2.93,5.52-4.61c2.44-2.4,2.35-6.62,0.01-8.95
-		C247.1,271.52,243.37,271.51,240.82,274z"/>
-	<path class="st0" d="M192.39,4.72C89.03,4.5,5.07,88.18,4.86,191.61c-0.21,103.43,83.4,187.46,186.75,187.67
-		c103.35,0.21,187.31-83.46,187.53-186.89C379.36,88.96,295.74,4.93,192.39,4.72z M327.66,225.47c0.14,1.32,0.27,2.64,0.41,3.96
-		c0.53,4.98,0.43,9.96-0.23,14.91c-2.24,16.85-10.03,30.98-21.45,43.28c-8.94,9.62-19.42,17.22-30.95,23.42
-		c-13.96,7.52-28.79,12.64-44.3,15.87c-8.46,1.76-17,2.91-25.62,3.51c-6.58,0.46-13.16,0.6-19.75,0.39
-		c-6.8-0.22-13.58-0.8-20.34-1.72c-11.93-1.64-23.6-4.38-34.96-8.39c-14.15-4.99-27.38-11.73-39.34-20.86
-		c-9.74-7.44-18.15-16.12-24.63-26.59c-5.11-8.26-8.64-17.14-10.2-26.75c-1.1-6.76-1.23-13.55-0.14-20.33c0.03-0.21,0-0.44,0-0.7
-		c-10.51-5.25-17.19-13.54-19.32-25.14c-1.71-9.32,0.32-17.97,5.8-25.72c11.73-16.6,35.97-19.53,51.41-4.96
-		c27.77-19.23,59-27.5,92.46-28.92c0.46-2.12,0.92-4.2,1.36-6.28c3.46-16.26,6.92-32.53,10.37-48.79
-		c1.82-8.57,3.63-17.14,5.43-25.71c0.26-1.23,0.59-2.42,1.44-3.4c1.68-1.92,3.83-2.34,6.2-1.85c6.31,1.3,12.62,2.66,18.92,3.99
-		c10.35,2.19,20.7,4.39,31.05,6.59c2,0.42,4,0.84,6.09,1.28c3.7-6.86,9.29-11.3,16.92-12.93c5.54-1.19,10.89-0.41,15.96,2.14
-		c10.03,5.06,15.32,16.56,12.65,27.42c-2.74,11.12-12.68,18.73-24.01,18.65c-11.69-0.08-23.34-9.38-24.16-23.21
-		c-1.43-0.6-49.41-10.81-50.5-10.7c-5.15,24.12-10.32,48.27-15.51,72.57c0.64,0.04,1.07,0.07,1.51,0.09
-		c19.74,0.93,38.91,4.68,57.37,11.79c11.05,4.25,21.5,9.64,31.21,16.44c0.29,0.2,0.6,0.38,0.94,0.59
-		c8.62-7.75,18.62-11.01,30.07-8.89c9.47,1.76,16.97,6.78,22.22,14.87c5.67,8.73,7.06,18.26,4.21,28.25
-		C343.42,213.65,337.03,220.82,327.66,225.47z"/>
-	<path class="st0" d="M162.26,218.68c0.08-13.49-10.62-24.33-24.09-24.41c-13.86,0-24.09,11.31-24.22,23.89
-		c-0.13,12.52,9.75,24.24,23.96,24.42C151.22,242.73,162.19,231.9,162.26,218.68z"/>
-	<path class="st0" d="M245.16,194.56c-13.42,0.03-24.09,10.83-24.11,24.24c-0.02,12.58,10.14,24.01,24.17,24.05
-		c13.33,0.04,24.16-10.89,24.12-24.26C269.32,205.3,258.48,194.52,245.16,194.56z"/>
-</g>
-</svg>

browser_extension/extension/index.html

@@ -1,93 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-<head>
-    <meta charset="UTF-8">
-    <meta http-equiv="X-UA-Compatible" content="IE=edge">
-    <meta name="viewport" content="width=device-width, initial-scale=1.0">
-    <title>TubeArchivist Companion</title>
-    <link rel="stylesheet" href="https://fonts.googleapi.com/css?family=Sen" type="text/css">
-    <style>
-        body {
-            font-family: Sen-Regular, sans-serif;
-            background-color: #00202f;
-            color: #97d4c8;
-        }
-        .container {
-            padding: 10px;
-            min-width: 300px;
-        }
-        hr {
-            height: 1px;
-            color: white;
-            background-color: white;
-        }
-        .login-form {
-            display: grid;
-            grid-template-columns: 1fr 1fr;
-        }
-        .login-form label,
-        .login-form input {
-            margin: 3px 0;
-        }
-        .submit button {
-            margin: 10px auto 15px auto;
-            display: block;
-            border-radius: 0;
-            padding: 5px 13px;
-            border: none;
-            cursor: pointer;
-            background-color: #259485;
-            color: #ffffff;
-        }
-        .submit button:hover {
-            background-color: #97d4c8;
-            transform: scale(1.05);
-            color: #00202f;
-        }
-        .icons {
-            display: flex;
-            grid-template-columns: 1fr 1fr;
-            justify-content: space-between;
-        }
-        .icons img {
-            width: 25px;
-        }
-    </style>
-</head>
-<body>
-    <div class="container">
-        <img src="/images/logo.svg" alt="ta-logo">
-        <hr>
-        <form class="login-form">
-            <label for="url">Tube Archivist Url:</label>
-            <input type="text" id="url" name="url">
-            <label for="port">Tube Archivist Port:</label>
-            <input type="text" id="port" name="port">
-            <label for="api-key">API key:</label>
-            <input type="text" id="api-key" name="api-key">
-        </form>
-        <div class="submit">
-            <button onclick="storeLogin()" id="save-login">Save</button>
-        </div>
-        <div class="icons">
-            <div>
-                <a href="https://www.reddit.com/r/TubeArchivist/" target="_blank">
-                    <img src="/images/social/reddit.svg" alt="reddit-icon">
-                </a>
-                <a href="https://discord.gg/AFwz8nE7BK" target="_blank">
-                    <img src="/images/social/discord.svg" alt="discord-icon">
-                </a>
-                <a href="https://github.com/bbilly1/tubearchivist" target="_blank">
-                    <img src="/images/social/github.svg" alt="github-icon">
-                </a>
-            </div>
-            <div>
-                <a href="#">
-                    <img src="/images/question.svg" alt="question-icon">
-                </a>
-            </div>
-        </div>
-    </div>
-    <script type="text/javascript" src="popup.js"></script>
-</body>
-</html>

browser_extension/extension/manifest.json

@@ -1,25 +0,0 @@
-{
-    "manifest_version": 2,
-    "name": "TubeArchivist Companion",
-    "description": "Interact with your selhosted TA server.",
-    "version": "0.0.1",
-    "icons": {
-        "128": "/images/icon128.png"
-    },
-    "browser_action": {
-        "default_icon": "/images/icon.png",
-        "default_popup": "index.html"
-    },
-    "permissions": [
-        "storage"
-    ],
-    "content_scripts": [
-        {
-            "matches": ["https://www.youtube.com/results*"],
-            "js": ["script.js"]
-        }
-    ],
-    "background": {
-        "scripts": ["background.js"]
-    }
-}

browser_extension/extension/popup.js

@@ -1,63 +0,0 @@
-/*
-Loaded into popup index.html
-*/
-
-let browserType = getBrowser();
-
-// boilerplate to dedect browser type api
-function getBrowser() {
-    if (typeof chrome !== "undefined") {
-        if (typeof browser !== "undefined") {
-            console.log("detected firefox");
-            return browser;
-        } else {
-            console.log("detected chrome");
-            return chrome;
-        }
-    } else {
-        console.log("failed to dedect browser");
-        throw "browser detection error"
-    };
-}
-
-// store access details
-document.getElementById("save-login").addEventListener("click", function () {
-    console.log("save form");
-    let toStore = {
-        "access": {
-            "url": document.getElementById("url").value,
-            "port": document.getElementById("port").value,
-            "apiKey": document.getElementById("api-key").value
-        }
-    };
-    console.log(toStore);
-    browserType.storage.local.set(toStore, function() {
-        console.log("Stored connection details: " + JSON.stringify(toStore));
-    });
-})
-
-// fill in form
-document.addEventListener("DOMContentLoaded", async () => {
-
-    console.log("executing dom loader");
-
-    function onGot(item) {
-        if (!item.access) {
-            console.log("no access details found");
-            return
-        }
-        console.log(item.access);
-        document.getElementById("url").value = item.access.url;
-        document.getElementById("port").value = item.access.port;
-        document.getElementById("api-key").value = item.access.apiKey;
-    };
-    
-    function onError(error) {
-        console.log(`Error: ${error}`);
-    };
-
-    browserType.storage.local.get("access", function(result) {
-        onGot(result)
-    });
-
-})

browser_extension/extension/script.js

@@ -1,72 +0,0 @@
-/*
-content script running on youtube.com
-*/
-
-console.log("running script.js");
-
-let browserType = getBrowser();
-
-setTimeout(function(){
-    console.log("running setimeout")
-    linkFinder();
-    return false;
-}, 2000);
-
-
-// boilerplate to dedect browser type api
-function getBrowser() {
-    if (typeof chrome !== "undefined") {
-        if (typeof browser !== "undefined") {
-            console.log("detected firefox");
-            return browser;
-        } else {
-            console.log("detected chrome");
-            return chrome;
-        }
-    } else {
-        console.log("failed to dedect browser");
-        throw "browser detection error"
-    };
-}
-
-
-// event handler for download task
-function addToDownload(videoId) {
-
-    console.log(`downloading ${videoId}`);
-    let payload = {
-        "download": {
-            "videoId": videoId
-        }
-    };
-
-    browserType.runtime.sendMessage(payload);
-
-}
-
-
-// find relevant links to add a button to
-function linkFinder() {
-
-    console.log("running link finder");
-
-    var allLinks = document.links;
-    for (let i = 0; i < allLinks.length; i++) {
-        
-        const linkItem = allLinks[i];
-        const linkDest = linkItem.getAttribute("href");
-
-        if (linkDest.startsWith("/watch?v=") && linkItem.id == "video-title") {
-            var dlButton = document.createElement("button");
-            dlButton.innerText = "download";
-            var videoId = linkDest.split("=")[1];
-            dlButton.setAttribute("data-id", videoId);
-            dlButton.setAttribute("id", "ta-dl-" + videoId);
-            dlButton.onclick = function(event) {
-                var videoId = this.getAttribute("data-id");
-                addToDownload(videoId);
-            };
-            linkItem.parentElement.appendChild(dlButton);
-        }
-    }
-}

deploy.sh

@@ -12,7 +12,7 @@
 # docker buildx inspect --bootstrap
 
 # more details:
-# https://github.com/bbilly1/tubearchivist/issues/6
+# https://github.com/tubearchivist/tubearchivist/issues/6
 
 set -e
 
@@ -33,7 +33,7 @@ function sync_blackhole {
         . -e ssh "$host":tubearchivist
 
     echo "$PASS" | ssh "$host" 'sudo -S docker buildx build --platform linux/amd64 -t bbilly1/tubearchivist:latest tubearchivist --load 2>/dev/null'
-    echo "$PASS" | ssh "$host" 'sudo -S docker-compose up -d 2>/dev/null'
+    echo "$PASS" | ssh "$host" 'sudo -S docker compose up -d 2>/dev/null'
 
 }
 
@@ -69,7 +69,7 @@ function sync_test {
     fi
 
     ssh "$host" "docker buildx build --build-arg INSTALL_DEBUG=1 --platform $platform -t bbilly1/tubearchivist:latest tubearchivist --load"
-    ssh "$host" 'docker-compose -f docker/docker-compose.yml up -d'
+    ssh "$host" 'docker compose -f docker/docker-compose.yml up -d'
 
 }
 
@@ -100,6 +100,34 @@ function validate {
 }
 
 
+# update latest tag compatible es for set and forget
+function sync_latest_es {
+
+    VERSION=$(grep "bbilly1/tubearchivist-es" docker-compose.yml | awk '{print $NF}')
+    printf "\nsync new ES version %s\nContinue?\n" "$VERSION"
+    read -rn 1
+
+    if [[ $(systemctl is-active docker) != 'active' ]]; then
+        echo "starting docker"
+        sudo systemctl start docker
+    fi
+
+    sudo docker image pull docker.elastic.co/elasticsearch/elasticsearch:"$VERSION"
+
+    sudo docker tag \
+        docker.elastic.co/elasticsearch/elasticsearch:"$VERSION" \
+        bbilly1/tubearchivist-es
+
+    sudo docker tag \
+        docker.elastic.co/elasticsearch/elasticsearch:"$VERSION" \
+        bbilly1/tubearchivist-es:"$VERSION"
+
+    sudo docker push bbilly1/tubearchivist-es
+    sudo docker push bbilly1/tubearchivist-es:"$VERSION"
+
+}
+
+
 # publish unstable tag to docker
 function sync_unstable {
 
@@ -116,6 +144,7 @@ function sync_unstable {
 }
 
 
+# new function, sync only tag, build with build server
 function sync_docker {
 
     # check things
@@ -124,16 +153,40 @@ function sync_docker {
         return
     fi
 
+    echo "latest tags:"
+    git tag | tail -n 5 | sort -r
+
+    printf "\ncreate new version:\n"
+    read -r VERSION
+
+    echo "push new tag: $VERSION?"
+    read -rn 1
+
+    # create release tag
+    echo "commits since last version:"
+    git log "$(git describe --tags --abbrev=0)"..HEAD --oneline
+    git tag -a "$VERSION" -m "new release version $VERSION"
+    git push origin "$VERSION"
+
+}
+
+
+# old builder, sync tag, build and push locally
+function sync_docker_old {
+
+    # check things
+    if [[ $(git branch --show-current) != 'master' ]]; then
+        echo 'you are not on master, dummy!'
+        return
+    fi
+
     if [[ $(systemctl is-active docker) != 'active' ]]; then
         echo "starting docker"
         sudo systemctl start docker
     fi
 
     echo "latest tags:"
-    git tag
-
-    echo "latest docker images:"
-    sudo docker image ls bbilly1/tubearchivist
+    git tag | tail -n 5 | sort -r
 
     printf "\ncreate new version:\n"
     read -r VERSION
@@ -151,28 +204,27 @@ function sync_docker {
     echo "commits since last version:"
     git log "$(git describe --tags --abbrev=0)"..HEAD --oneline
     git tag -a "$VERSION" -m "new release version $VERSION"
-    git push all "$VERSION"
+    git push origin "$VERSION"
 
 }
 
 
-# check package versions in requirements.txt for updates
-python version_check.py
-
-
 if [[ $1 == "blackhole" ]]; then
     sync_blackhole
 elif [[ $1 == "test" ]]; then
     sync_test "$2"
 elif [[ $1 == "validate" ]]; then
+    # check package versions in requirements.txt for updates
+    python version_check.py
     validate "$2"
 elif [[ $1 == "docker" ]]; then
     sync_docker
-    sync_unstable
 elif [[ $1 == "unstable" ]]; then
     sync_unstable
+elif [[ $1 == "es" ]]; then
+    sync_latest_es
 else
-    echo "valid options are: blackhole | test | validate | docker | unstable"
+    echo "valid options are: blackhole | test | validate | docker | unstable | es"
 fi
 
 

docker-compose.yml

@@ -3,29 +3,29 @@ version: '3.3'
 services:
   tubearchivist:
     container_name: tubearchivist
-    restart: always
-    image: bbilly1/tubearchivist:latest
+    restart: unless-stopped
+    image: bbilly1/tubearchivist
     ports:
       - 8000:8000
     volumes:
       - media:/youtube
       - cache:/cache
     environment:
-      - ES_URL=http://archivist-es:9200
-      - REDIS_HOST=archivist-redis
+      - ES_URL=http://archivist-es:9200     # needs protocol e.g. http and port
+      - REDIS_HOST=archivist-redis          # don't add protocol
       - HOST_UID=1000
       - HOST_GID=1000
-      - TA_USERNAME=tubearchivist
-      - TA_PASSWORD=verysecret
-      - ELASTIC_PASSWORD=verysecret
-      - TZ=America/New_York
+      - TA_USERNAME=tubearchivist           # your initial TA credentials
+      - TA_PASSWORD=verysecret              # your initial TA credentials
+      - ELASTIC_PASSWORD=verysecret         # set password for Elasticsearch
+      - TZ=America/New_York                 # set your time zone
     depends_on:
       - archivist-es
       - archivist-redis
   archivist-redis:
-    image: redislabs/rejson:latest      # For arm64 just update this line with bbilly1/rejson:latest
+    image: redislabs/rejson                 # for arm64 use bbilly1/rejson
     container_name: archivist-redis
-    restart: always
+    restart: unless-stopped
     expose:
       - "6379"
     volumes:
@@ -33,12 +33,12 @@ services:
     depends_on:
       - archivist-es
   archivist-es:
-    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.1
+    image: bbilly1/tubearchivist-es         # only for amd64, or use official es 7.17.4
     container_name: archivist-es
-    restart: always
+    restart: unless-stopped
     environment:
       - "xpack.security.enabled=true"
-      - "ELASTIC_PASSWORD=verysecret"
+      - "ELASTIC_PASSWORD=verysecret"       # matching Elasticsearch password
       - "discovery.type=single-node"
       - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
     ulimits:
@@ -46,7 +46,7 @@ services:
         soft: -1
         hard: -1
     volumes:
-      - es:/usr/share/elasticsearch/data
+      - es:/usr/share/elasticsearch/data    # check for permission error when using bind mount, see readme
     expose:
       - "9200"
 

docker_assets/nginx.conf

@@ -1,30 +1,29 @@
 server {
 
     listen 8000;
-
+    
     location /cache/videos/ {
         alias /cache/videos/;
     }
-
+    
     location /cache/channels/ {
         alias /cache/channels/;
     }
-
+    
     location /cache/playlists/ {
         alias /cache/playlists/;
     }
-
+    
     location /media/ {
         alias /youtube/;
-
         types {
             text/vtt vtt;
         }
     }
-
+    
     location / {
         include uwsgi_params;
         uwsgi_pass localhost:8080;
     }
-
+    
 }

docker_assets/run.sh

@@ -5,21 +5,22 @@ if [[ -z "$ELASTIC_USER" ]]; then
     export ELASTIC_USER=elastic
 fi
 
-ENV_VARS=("TA_USERNAME" "TA_PASSWORD" "ELASTIC_PASSWORD" "ELASTIC_USER")
-for each in "${ENV_VARS[@]}"; do
-    if ! [[ -v $each ]]; then
-        echo "missing environment variable $each"
-        exit 1
-    fi
-done
+cachedir=/cache
+[[ -d $cachedir ]] || cachedir=.
+lockfile=${cachedir}/initsu.lock
+
+required="Missing required environment variable"
+[[ -f $lockfile ]] || : "${TA_USERNAME:?$required}"
+: "${TA_PASSWORD:?$required}"
+: "${ELASTIC_PASSWORD:?$required}"
 
 # ugly nginx and uwsgi port overwrite with env vars
 if [[ -n "$TA_PORT" ]]; then
-    sed -i "s/8000/$TA_PORT/g" /etc/nginx/conf.d/nginx.conf
+    sed -i "s/8000/$TA_PORT/g" /etc/nginx/sites-available/default
 fi
 
 if [[ -n "$TA_UWSGI_PORT" ]]; then
-    sed -i "s/8080/$TA_UWSGI_PORT/g" /etc/nginx/conf.d/nginx.conf
+    sed -i "s/8080/$TA_UWSGI_PORT/g" /etc/nginx/sites-available/default
     sed -i "s/8080/$TA_UWSGI_PORT/g" /app/uwsgi.ini
 fi
 
@@ -39,12 +40,27 @@ done
 # start python application
 python manage.py makemigrations
 python manage.py migrate
-export DJANGO_SUPERUSER_PASSWORD=$TA_PASSWORD && \
-    python manage.py createsuperuser --noinput --name "$TA_USERNAME"
+
+if [[ -f $lockfile ]]; then
+    echo -e "\e[33;1m[WARNING]\e[0m This is not the first run! Skipping" \
+        "superuser creation.\nTo force it, remove $lockfile"
+else
+    export DJANGO_SUPERUSER_PASSWORD=$TA_PASSWORD
+    output="$(python manage.py createsuperuser --noinput --name "$TA_USERNAME" 2>&1)"
+
+    case "$output" in
+        *"Superuser created successfully"*)
+            echo "$output" && touch $lockfile ;;
+        *"That name is already taken."*)
+            echo "Superuser already exists. Creation will be skipped on next start."
+            touch $lockfile ;;
+        *) echo "$output" && exit 1
+    esac
+fi
 
 python manage.py collectstatic --noinput -c
 nginx &
 celery -A home.tasks worker --loglevel=INFO &
 celery -A home beat --loglevel=INFO \
-    -s "${BEAT_SCHEDULE_PATH:-/cache/celerybeat-schedule}" &
+    -s "${BEAT_SCHEDULE_PATH:-${cachedir}/celerybeat-schedule}" &
 uwsgi --ini uwsgi.ini

docs/Channels.md

@@ -28,4 +28,5 @@ Each channel will get a dedicated channel detail page accessible at `/channel/<c
 Clicking on the *Configure* button will open a form with options to configure settings on a per channel basis. Any configurations here will overwrite your settings from the [settings](Settings) page.
 - **Download Format**: Overwrite the download qualities for videos from this channel.
 - **Auto Delete**: Automatically delete watched videos from this channel after selected days.
-- **Index Playlists**: Automatically add all Playlists with at least a video downloaded to your index. Only do this for channels where you care about playlists as this will slow down indexing new videos for having to check which playlist this belongs to.
+- **Index Playlists**: Automatically add all Playlists with at least a video downloaded to your index. Only do this for channels where you care about playlists as this will slow down indexing new videos for having to check which playlist this belongs to.
+- **SponsorBlock**: Using [SponsorBlock](https://sponsor.ajay.app/) to get and skip sponsored content. Customize per channel: You can *disable* or *enable* SponsorBlock for certain channels only to overwrite the behavior set on the [Settings](settings) page. Selecting *unset* will remove the overwrite and your setting will fall back to the default on the settings page.

docs/Downloads.md

@@ -33,11 +33,9 @@ Every video in the download queue has two buttons:
 - **Ignore**: This will remove that video from the download queue and this video will not get added again, even when you **Rescan Subscriptions**.
 - **Download now**: This will give priority to this video. If the download process is already running, the prioritized video will get downloaded as soon as the current video is finished. If there is no download process running, this will start downloading this single video and stop after that.  
 
-The button **Delete all queued** will delete all pending videos from the download queue.
-
 You can flip the view by activating **Show Only Ignored Videos**. This will show all videos you have previously *ignored*.  
 Every video in the ignored list has two buttons:
 - **Forget**: This will delete the item form the ignored list.
 - **Add to Queue**: This will add the ignored video back to the download queue.  
 
-The button **Delete all ignored** will delete all videos you have previously ignored.
+You can delete your download queue from the [Settings](Settings#actions) page.

docs/FAQ.md

@@ -29,3 +29,6 @@ So Docker is the only supported installation method. If you don't have any exper
 
 ## 4. Finetuning Elasticsearch
 A minimal configuration of Elasticsearch (ES) is provided in the example docker-compose.yml file. ES is highly configurable and very interesting to learn more about. Refer to the [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html) if you want to get into it.
+
+## 5. Advanced Authentication
+If you like to use things like SSO, LDAP or 2FA to login, consider using something like Authelia as a reverse proxy so this project can focus on the core task. Tube Archivist has a *remember me* checkbox at login to extend your sessions lifetime in your browser. 

docs/Home.md

@@ -9,6 +9,7 @@ Table of contents:
 * [Settings](Settings): All the configuration options
 * [Video](Video): All details of a single video and playlist navigation.
 * [Users](Users): User management admin interface
+* [Installation](Installation): WIP - detailed installation instructions for various platforms.
 
 ## Getting Started
 1. [Subscribe](Channels#channels-overview) to some of your favourite YouTube channels.
@@ -27,4 +28,4 @@ Table of contents:
 
 An empty checkbox icon <img src="assets/icon-unseen.png?raw=true" alt="unseen icon" width="20px" style="margin:0 5px;"> will show for videos you haven't marked as watched. Click on it and the icon will change to a filled checkbox <img src="assets/icon-seen.png?raw=true" alt="seen icon" width="20px" style="margin:0 5px;"> indicating it as watched - click again to revert.
 
-When available the <img src="assets/icon-gridview.png?raw=true" alt="gridview icon" width="20px" style="margin:0 5px;"> gridview icon will display the list in a grid, the <img src="assets/icon-listview.png?raw=true" alt="listview icon" width="20px" style="margin:0 5px;"> listview icon will arrange the items in a list. The sort icon <img src="assets/icon-sort.png?raw=true" alt="listview icon" width="20px" style="margin:0 5px;"> will open additional sort options.
+When available the <img src="assets/icon-gridview.png?raw=true" alt="gridview icon" width="20px" style="margin:0 5px;"> gridview icon will display the list in a grid. A grid row holds 3 items by default, use the <img src="assets/icon-add.png?raw=true" alt="listview icon" width="20px" style="margin:0 5px;"> icon to add more or the <img src="assets/icon-substract.png?raw=true" alt="listview icon" width="20px" style="margin:0 5px;"> icon to remove items per row, depending on your screen size. The <img src="assets/icon-listview.png?raw=true" alt="listview icon" width="20px" style="margin:0 5px;"> listview icon will arrange the items in a list. The sort icon <img src="assets/icon-sort.png?raw=true" alt="listview icon" width="20px" style="margin:0 5px;"> will open additional sort options.

docs/Installation.md

@@ -0,0 +1,69 @@
+# Detailed Installation Instructions for Various Platforms
+
+## Unraid
+
+Tube Archivist, and all if it's dependencies are located in the [community applications](https://forums.unraid.net/topic/38582-plug-in-community-applications/) store. The three containers you will need are as follows:
+
+-   **TubeArchivist-RedisJSON**: This container acts as a cache and temporary link between the application and the file system. Used to store and display messages and configuration variables.
+-   **TubeArchivist-ES**: ElasticSearch stores video meta data and makes everything searchable. Also keeps track of the download queue.
+-   **TubeArchivist**: Once your YouTube video collection grows, it becomes hard to search and find a specific video. That's where Tube Archivist comes in: By indexing your video collection with metadata from YouTube, you can organize, search and enjoy your archived YouTube videos without hassle offline through a convenient web interface.
+
+### Step 1: Install  `TubeArchivist-RedisJSON`
+
+![enter image description here](https://i.imgur.com/ycAqFRU.png)
+This is the easiest container to setup of the thee, just make sure that you do not have any port conflicts, and that your `/data` is mounted to the correct path. The other containers will map to the same directory.
+
+If you need to install `TubeArchivist-RedisJSON`on a different port, you'll have to follow [these steps](https://github.com/tubearchivist/tubearchivist#redis-on-a-custom-port) later on when installing the `TubeArchivist` container
+
+
+### Step 2: Install  `TubeArchivist-ES`
+![enter image description here](https://i.imgur.com/o6tsTdt.png)
+ElasticSeach is also pretty easy to setup. Again, make sure you have no port conflicts, make sure that you mapped `/usr/share/elasticsearch/data` to the same directory as `RedisJSON`, and make sure to change the default password to something more secure. 
+
+There is three additional settings in the "show more settings" area, but leave those as they are.
+
+
+### Step 3: Install  `TubeArchivist`
+
+![enter image description here](https://i.imgur.com/dwSCfgO.png)
+It's finally time to set up TubeArchivist!
+
+ - `Port:`Again, make sure that you have no port conflicts on 8000.
+   
+ - `Youtube Media Path:` is where you'll download all of your videos to.
+   Make sure that this is an empty directory to not cause confusion when
+   starting the application. If you have existing videos that you'd like
+   to import into Tube Archivist, please checkout the [settings
+   wiki.](https://github.com/tubearchivist/tubearchivist/wiki/Settings#manual-media-files-import)
+   
+   
+- `Appdata:` This should be the same base path as the other two containers.
+   
+ - `TA Username:`This will be your username for TubeArchivist.
+   
+ - `TA Password:`This will be your password for TubeArchivist.
+   
+ - `Redis` This will be JUST the ip address of your redis container
+
+ - `ElasticSearch Password:`This is the password you defined in the `TubeArchivist-ES` container.
+ - `ElasticSearch:` This seems to cause some confusion, but it's a pretty simple step, just replace the IP and Port to match you `TubeArchivist-ES` container.
+
+ (example: if your IP is 192.168.1.15, the value should be http://192.168.1.15:9200)
+
+ - `Time Zone:` This is an important step for your scheduler, to find your timezone, use a site like [TimeZoneConverter](http://www.timezoneconverter.com/cgi-bin/findzone.tzc) 
+
+### From there, you should be able to start up your containers and you're good to go!
+If you're still having trouble, join us on [discord](https://discord.gg/AFwz8nE7BK) and come to the #unraid channel.
+
+<br />
+<br />
+
+## Truenas Scale 
+
+Truenas Scale can be a bit confusing, with its k3s kubernetes implementation. 
+
+However, there is a step by step guide available for it's users here:
+
+https://heavysetup.info/applications/tube-archivist/dataset/
+
+- Ensure you are navigating the columns under `Tube Archivist` on the left hand side of the screen

docs/Settings.md

@@ -32,10 +32,37 @@ Additional settings passed to yt-dlp.
 - **Source Settings**: User created subtitles are provided from the uploader and are usually the video script. Auto generated is from YouTube, quality varies, particularly for auto translated tracks.
 - **Index Settings**: Enabling subtitle indexing will add the lines to Elasticsearch and will make subtitles searchable. This will increase the index size and is not recommended on low-end hardware.
 
+## Cookie
+Importing your YouTube Cookie into Tube Archivist allows yt-dlp to bypass age restrictions, gives access to private videos and your *watch later* or *liked videos*.
+
+### Security concerns
+Cookies are used to store your session and contain your access token to your google account, this information can be used to take over your account. Treat that data with utmost care as you would any other password or credential. *Tube Archivist* stores your cookie in Redis and will automatically append it to yt-dlp for every request.
+
+### Auto import
+Easiest way to import your cookie is to use the **Tube Archivist Companion** [browser extension](https://github.com/tubearchivist/browser-extension) for Firefox and Chrome.
+
+### Alternative Manual Export your cookie
+- Install **Cookies.txt** addon for [chrome](https://chrome.google.com/webstore/detail/get-cookiestxt/bgaddhkoddajcdgocldbbfleckgcbcid) or [firefox](https://addons.mozilla.org/firefox/addon/cookies-txt).
+- Visit YouTube and login with whichever YouTube account you wish to use to generate the cookies.
+- Click on the extension icon in the toolbar - it will drop down showing the active cookies for YT.
+- Click Export to export the cookies, filename is by default *cookies.google.txt*.
+
+### Alternative Manual Import your cookie
+Place the file *cookies.google.txt* into the *cache/import* folder of Tube Archivist and enable the cookie import. Once you click on *Update Application Configurations* to save your changes, your cookie will get imported and stored internally.
+
+Once imported, a **Validate Cookie File** button will show, where you can confirm if your cookie is working or not.
+
+### Use your cookie
+Once imported, additionally to the advantages above, your [Watch Later](https://www.youtube.com/playlist?list=WL) and [Liked Videos](https://www.youtube.com/playlist?list=LL) become a regular playlist you can download and subscribe to as any other [playlist](Playlists).
+
+### Limitation
+There is only one cookie per Tube Archivist instance, this will be shared between all users.
+
 ## Integrations
 All third party integrations of TubeArchivist will **always** be *opt in*.
-- **API**: Your access token for the Tube Archivist API. 
+- **API**: Your access token for the Tube Archivist API.
 - **returnyoutubedislike.com**: This will get return dislikes and average ratings for each video by integrating with the API from [returnyoutubedislike.com](https://www.returnyoutubedislike.com/).
+- **SponsorBlock**: Using [SponsorBlock](https://sponsor.ajay.app/) to get and skip sponsored content. If a video doesn't have timestamps, or has unlocked timestamps, use the browser addon to contribute to this excellent project. Can also be activated and deactivated as a per [channel overwrite](Settings#channel-customize).
 - **Cast**: Enabling the cast integration in the settings page will load an additional JS library from **Google**.
     * Requirements
         - HTTPS
@@ -55,7 +82,7 @@ Examples:
 - **0 15 \***: Run task every day at 15:00 in the afternoon.
 - **30 8 \*/2**: Run task every second day of the week (Sun, Tue, Thu, Sat) at 08:30 in the morning.
 - **0 \*/3,8-17 \***: Execute every hour divisible by 3, and every hour during office hours (8 in the morning - 5 in the afternoon).
-- **0 8,16 \***: Execute every day at 8 in the morning and at 4 in the afternoon. 
+- **0 8,16 \***: Execute every day at 8 in the morning and at 4 in the afternoon.
 - **auto**: Sensible default.
 - **0**: (zero), deactivate that task.
 
@@ -71,9 +98,9 @@ That's the equivalent task as run from the downloads page looking through your c
 Start downloading all videos currently in the download queue.
 
 ## Refresh Metadata
-Rescan videos, channels and playlists on youtube and update metadata periodically. This will also deactivate an item and exclude it from future refreshes if the link on YouTube is no longer available. This task is meant to be run once per day, set your schedule accordingly. 
+Rescan videos, channels and playlists on youtube and update metadata periodically. This will also refresh your subtitles based on your current settings. If an item is no longer available on YouTube, this will deactivate it and exclude it from future refreshes. This task is meant to be run once per day, set your schedule accordingly.
 
-The field **Refresh older than x days** takes a number where TubeArchivist will consider an item as *outdated*. This value is used to calculate how many items need to be refreshed today based on the total indexed. This will spread out the requests to YouTube. Sensible value here is **90** days. 
+The field **Refresh older than x days** takes a number where TubeArchivist will consider an item as *outdated*. This value is used to calculate how many items need to be refreshed today based on the total indexed. This will spread out the requests to YouTube. Sensible value here is **90** days.
 
 ## Thumbnail check
 This will check if all expected thumbnails are there and will delete any artwork without matching video.
@@ -85,6 +112,9 @@ Create a zip file of the metadata and select **Max auto backups to keep** to aut
 # Actions
 Additional database functionality.
 
+## Delete download queue
+The button **Delete all queued** will delete all pending videos from the download queue. The button **Delete all ignored** will delete all videos you have previously ignored.
+
 ## Manual Media Files Import
 So far this depends on the video you are trying to import to be still available on YouTube to get the metadata. Add the files you'd like to import to the */cache/import* folder. Then start the process from the settings page *Manual Media Files Import*. Make sure to follow one of the two methods below.
 
@@ -103,6 +133,7 @@ Detect the YouTube ID from filename, this accepts the default yt-dlp naming conv
 
 ### Some notes:
 - This will **consume** the files you put into the import folder: Files will get converted to mp4 if needed (this might take a long time...) and moved to the archive, *.json* files will get deleted upon completion to avoid having duplicates on the next run.
+- For best file transcoding quality, convert your media files with desired settings first before importing (#138).
 - There should be no subdirectories added to */cache/import*, only video files. If your existing video library has video files inside subdirectories, you can get all the files into one directory by running `find ./ -mindepth 2 -type f -exec mv '{}' . \;` from the top-level directory of your existing video library. You can also delete any remaining empty subdirectories with `find ./ -mindepth 1 -type d -delete`.
 - Maybe start with a subset of your files to import to make sure everything goes well...
 - Follow the logs to monitor progress and errors: `docker-compose logs -f tubearchivist`.
@@ -129,5 +160,6 @@ This function will go through all your media files and looks at the whole index
 - Should the filename not match with the indexed media url, this will rename the video files correctly and update the index with the new link.
 - When you delete media files from the filesystem outside of the Tube Archivist interface, this will delete leftover metadata from the index.
 - When you have media files that are not indexed yet, this will grab the metadata from YouTube like it was a newly downloaded video. This can be useful when restoring from an older backup file with missing metadata but already downloaded mediafiles. NOTE: This only works if the media files are named in the same convention as Tube Archivist does, particularly the YouTube ID needs to be at the same index in the filename, alternatively see above for *Manual Media Files Import*.
+-This will also check all of your thumbnails and download any that are missing.
 
 BE AWARE: There is no undo.

tubearchivist/api/README.md

@@ -1,6 +1,9 @@
 # TubeArchivist API
 Documentation of available API endpoints.  
-**Note: This is very early alpha and will change!**
+
+Note:  
+- This is very early stages and will change!
+- Check the commit history to see if a documented feature is already in your release
 
 ## Authentication
 API token will get automatically created, accessible on the settings page. Token needs to be passed as an authorization header with every request. Additionally session based authentication is enabled too: When you are logged into your TubeArchivist instance, you'll have access to the api in the browser for testing.
@@ -20,6 +23,40 @@ headers = {"Authorization": "Token xxxxxxxxxx"}
 response = requests.get(url, headers=headers)
 ```
 
+## Pagination
+The list views return a paginate object with the following keys:
+- page_size: int current page size set in config
+- page_from: int first result idx
+- prev_pages: array of ints of previous pages, if available
+- current_page: int current page from query
+- max_hits: reached: bool if max of 10k results is reached
+- last_page: int of last page link
+- next_pages: array of ints of next pages
+- total_hits: int total results
+
+Pass page number as a query parameter: `page=2`. Defaults to *0*, `page=1` is redundant and falls back to *0*. If a page query doesn't return any results, you'll get `HTTP 404 Not Found`.
+
+## Login View
+Return token and user ID for username and password:  
+POST /api/login
+```json
+{
+    "username": "tubearchivist",
+    "password": "verysecret"
+}
+```
+
+after successful login returns 
+```json
+{
+    "token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
+    "user_id": 1
+}
+```
+
+## Video List View
+/api/video/
+
 ## Video Item View
 /api/video/\<video_id>/
 
@@ -49,6 +86,43 @@ POST /api/video/\<video_id>/progress
 ### Delete player position of video
 DELETE /api/video/\<video_id>/progress  
 
+
+## Sponsor Block View
+/api/video/\<video_id>/sponsor/
+
+Integrate with sponsorblock
+
+### Get list of segments
+GET /api/video/\<video_id>/sponsor/
+
+
+### Vote on existing segment
+**This only simulates the request**  
+POST /api/video/\<video_id>/sponsor/
+```json
+{
+    "vote": {
+        "uuid": "<uuid>",
+        "yourVote": 1
+    }
+}
+```
+yourVote needs to be *int*: 0 for downvote, 1 for upvote, 20 to undo vote
+
+### Create new segment
+**This only simulates the request**  
+POST /api/video/\<video_id>/sponsor/
+```json
+{
+    "segment": {
+        "startTime": 5,
+        "endTime": 10
+    }
+}
+```
+Timestamps either *int* or *float*, end time can't be before start time.
+
+
 ## Channel List View
 /api/channel/
 
@@ -65,11 +139,23 @@ POST /api/channel/
 ## Channel Item View
 /api/channel/\<channel_id>/
 
+## Channel Videos View
+/api/channel/\<channel_id>/video/
+
+## Playlist List View
+/api/playlist/
+
 ## Playlists Item View
 /api/playlist/\<playlist_id>/
 
+## Playlist Videos View
+/api/playlist/\<playlist_id>/video/
+
 ## Download Queue List View
-/api/download/
+GET /api/download/
+
+Parameter:
+- filter: pending, ignore
 
 ### Add list of videos to download queue
 POST /api/download/
@@ -81,5 +167,109 @@ POST /api/download/
 }
 ```
 
+### Delete download queue items by filter
+DELETE /api/download/?filter=ignore  
+DELETE /api/download/?filter=pending
+
 ## Download Queue Item View
-/api/download/\<video_id>/
+GET /api/download/\<video_id>/  
+POST /api/download/\<video_id>/
+
+Ignore video in download queue:
+```json
+{
+    "status": "ignore"
+}
+```
+
+Add to queue previously ignored video:
+```json
+{
+    "status": "pending"
+}
+```
+
+DELETE /api/download/\<video_id>/  
+Forget or delete from download queue
+
+## Ping View
+Validate your connection with the API  
+GET /api/ping
+
+When valid returns message with user id: 
+```json
+{
+    "response": "pong",
+    "user": 1
+}
+```
+
+## Task View
+GET /api/task/
+POST /api/task/
+
+Check if there is an ongoing task:
+GET /api/task/
+
+Returns:
+```json
+{
+    "rescan": false,
+    "downloading": false
+}
+```
+
+Start a background task  
+POST /api/task/
+```json
+{
+    "run": "task_name"
+}
+```
+
+List of valid task names:
+- **download_pending**: Start the download queue
+- **rescan_pending**: Rescan your subscriptions
+
+
+## Cookie View
+Check your youtube cookie settings, *status* turns to `true` if cookie has been validated.  
+GET /api/cookie/
+```json
+{
+    "cookie_enabled": true,
+    "status": true,
+    "validated": <timestamp>,
+    "validated_str": "timestamp"
+}
+```
+
+POST /api/cookie/  
+Send empty post request to validate cookie.
+```json
+{
+    "cookie_validated": true
+}
+```
+
+PUT /api/cookie/  
+Send put request containing the cookie as a string:
+```json
+{
+    "cookie": "your-cookie-as-string"
+}
+```
+Imports and validates cookie, returns on success:
+```json
+{
+    "cookie_import": "done",
+    "cookie_validated": true
+}
+```
+Or returns status code 400 on failure:
+```json
+{
+    "cookie_import": "fail",
+    "cookie_validated": false
+}
+```

tubearchivist/api/src/search_processor.py

@@ -0,0 +1,125 @@
+"""
+Functionality:
+- processing search results for frontend
+- this is duplicated code from home.src.frontend.searching.SearchHandler
+"""
+
+import urllib.parse
+
+from home.src.download.thumbnails import ThumbManager
+from home.src.ta.config import AppConfig
+from home.src.ta.helper import date_praser
+
+
+class SearchProcess:
+    """process search results"""
+
+    CONFIG = AppConfig().config
+    CACHE_DIR = CONFIG["application"]["cache_dir"]
+
+    def __init__(self, response):
+        self.response = response
+        self.processed = False
+
+    def process(self):
+        """dedect type and process"""
+        if "_source" in self.response.keys():
+            # single
+            self.processed = self._process_result(self.response)
+
+        elif "hits" in self.response.keys():
+            # multiple
+            self.processed = []
+            all_sources = self.response["hits"]["hits"]
+            for result in all_sources:
+                self.processed.append(self._process_result(result))
+
+        return self.processed
+
+    def _process_result(self, result):
+        """dedect which type of data to process"""
+        index = result["_index"]
+        processed = False
+        if index == "ta_video":
+            processed = self._process_video(result["_source"])
+        if index == "ta_channel":
+            processed = self._process_channel(result["_source"])
+        if index == "ta_playlist":
+            processed = self._process_playlist(result["_source"])
+        if index == "ta_download":
+            processed = self._process_download(result["_source"])
+
+        return processed
+
+    @staticmethod
+    def _process_channel(channel_dict):
+        """run on single channel"""
+        channel_id = channel_dict["channel_id"]
+        art_base = f"/cache/channels/{channel_id}"
+        date_str = date_praser(channel_dict["channel_last_refresh"])
+        channel_dict.update(
+            {
+                "channel_last_refresh": date_str,
+                "channel_banner_url": f"{art_base}_banner.jpg",
+                "channel_thumb_url": f"{art_base}_thumb.jpg",
+                "channel_tvart_url": False,
+            }
+        )
+
+        return dict(sorted(channel_dict.items()))
+
+    def _process_video(self, video_dict):
+        """run on single video dict"""
+        video_id = video_dict["youtube_id"]
+        media_url = urllib.parse.quote(video_dict["media_url"])
+        vid_last_refresh = date_praser(video_dict["vid_last_refresh"])
+        published = date_praser(video_dict["published"])
+        vid_thumb_url = ThumbManager().vid_thumb_path(video_id)
+        channel = self._process_channel(video_dict["channel"])
+
+        if "subtitles" in video_dict:
+            for idx, _ in enumerate(video_dict["subtitles"]):
+                url = video_dict["subtitles"][idx]["media_url"]
+                video_dict["subtitles"][idx]["media_url"] = f"/media/{url}"
+
+        video_dict.update(
+            {
+                "channel": channel,
+                "media_url": f"/media/{media_url}",
+                "vid_last_refresh": vid_last_refresh,
+                "published": published,
+                "vid_thumb_url": f"{self.CACHE_DIR}/{vid_thumb_url}",
+            }
+        )
+
+        return dict(sorted(video_dict.items()))
+
+    @staticmethod
+    def _process_playlist(playlist_dict):
+        """run on single playlist dict"""
+        playlist_id = playlist_dict["playlist_id"]
+        playlist_last_refresh = date_praser(
+            playlist_dict["playlist_last_refresh"]
+        )
+        playlist_dict.update(
+            {
+                "playlist_thumbnail": f"/cache/playlists/{playlist_id}.jpg",
+                "playlist_last_refresh": playlist_last_refresh,
+            }
+        )
+
+        return dict(sorted(playlist_dict.items()))
+
+    def _process_download(self, download_dict):
+        """run on single download item"""
+        video_id = download_dict["youtube_id"]
+        vid_thumb_url = ThumbManager().vid_thumb_path(video_id)
+        published = date_praser(download_dict["published"])
+
+        download_dict.update(
+            {
+                "vid_thumb_url": f"{self.CACHE_DIR}/{vid_thumb_url}",
+                "published": published,
+            }
+        )
+        return dict(sorted(download_dict.items()))

tubearchivist/api/src/task_processor.py

@@ -0,0 +1,54 @@
+"""
+Functionality:
+- process tasks from API
+- validate
+- handover to celery
+"""
+
+from home.src.ta.ta_redis import RedisArchivist
+from home.tasks import download_pending, update_subscribed
+
+
+class TaskHandler:
+    """handle tasks from api"""
+
+    def __init__(self, data):
+        self.data = data
+
+    def run_task(self):
+        """map data and run"""
+        task_name = self.data["run"]
+        try:
+            to_run = self.exec_map(task_name)
+        except KeyError as err:
+            print(f"invalid task name {task_name}")
+            raise ValueError from err
+
+        response = to_run()
+        response.update({"task": task_name})
+        return response
+
+    def exec_map(self, task_name):
+        """map dict key and return function to execute"""
+        exec_map = {
+            "download_pending": self._download_pending,
+            "rescan_pending": self._rescan_pending,
+        }
+
+        return exec_map[task_name]
+
+    @staticmethod
+    def _rescan_pending():
+        """look for new items in subscribed channels"""
+        print("rescan subscribed channels")
+        update_subscribed.delay()
+        return {"success": True}
+
+    @staticmethod
+    def _download_pending():
+        """start the download queue"""
+        print("download pending")
+        running = download_pending.delay()
+        print("set task id: " + running.id)
+        RedisArchivist().set_message("dl_queue_id", running.id)
+        return {"success": True}

tubearchivist/api/urls.py

@@ -2,16 +2,32 @@
 
 from api.views import (
     ChannelApiListView,
+    ChannelApiVideoView,
     ChannelApiView,
+    CookieView,
     DownloadApiListView,
     DownloadApiView,
+    LoginApiView,
+    PingView,
+    PlaylistApiListView,
+    PlaylistApiVideoView,
     PlaylistApiView,
+    TaskApiView,
+    VideoApiListView,
     VideoApiView,
     VideoProgressView,
+    VideoSponsorView,
 )
 from django.urls import path
 
 urlpatterns = [
+    path("ping/", PingView.as_view(), name="ping"),
+    path("login/", LoginApiView.as_view(), name="api-login"),
+    path(
+        "video/",
+        VideoApiListView.as_view(),
+        name="api-video-list",
+    ),
     path(
         "video/<slug:video_id>/",
         VideoApiView.as_view(),
@@ -22,6 +38,11 @@ urlpatterns = [
         VideoProgressView.as_view(),
         name="api-video-progress",
     ),
+    path(
+        "video/<slug:video_id>/sponsor/",
+        VideoSponsorView.as_view(),
+        name="api-video-sponsor",
+    ),
     path(
         "channel/",
         ChannelApiListView.as_view(),
@@ -32,11 +53,26 @@ urlpatterns = [
         ChannelApiView.as_view(),
         name="api-channel",
     ),
+    path(
+        "channel/<slug:channel_id>/video/",
+        ChannelApiVideoView.as_view(),
+        name="api-channel-video",
+    ),
+    path(
+        "playlist/",
+        PlaylistApiListView.as_view(),
+        name="api-playlist-list",
+    ),
     path(
         "playlist/<slug:playlist_id>/",
         PlaylistApiView.as_view(),
         name="api-playlist",
     ),
+    path(
+        "playlist/<slug:playlist_id>/video/",
+        PlaylistApiVideoView.as_view(),
+        name="api-playlist-video",
+    ),
     path(
         "download/",
         DownloadApiListView.as_view(),
@@ -47,4 +83,14 @@ urlpatterns = [
         DownloadApiView.as_view(),
         name="api-download",
     ),
+    path(
+        "task/",
+        TaskApiView.as_view(),
+        name="api-task",
+    ),
+    path(
+        "cookie/",
+        CookieView.as_view(),
+        name="api-cookie",
+    ),
 ]

tubearchivist/api/views.py

@@ -1,15 +1,22 @@
 """all API views"""
 
-from home.src.download.thumbnails import ThumbManager
+from api.src.search_processor import SearchProcess
+from api.src.task_processor import TaskHandler
+from home.src.download.queue import PendingInteract
+from home.src.download.yt_dlp_base import CookieHandler
 from home.src.es.connect import ElasticWrap
+from home.src.index.generic import Pagination
+from home.src.index.video import SponsorBlock
 from home.src.ta.config import AppConfig
 from home.src.ta.helper import UrlListParser
-from home.src.ta.ta_redis import RedisArchivist
+from home.src.ta.ta_redis import RedisArchivist, RedisQueue
 from home.tasks import extrac_dl, subscribe_to
 from rest_framework.authentication import (
     SessionAuthentication,
     TokenAuthentication,
 )
+from rest_framework.authtoken.models import Token
+from rest_framework.authtoken.views import ObtainAuthToken
 from rest_framework.permissions import IsAuthenticated
 from rest_framework.response import Response
 from rest_framework.views import APIView
@@ -21,12 +28,15 @@ class ApiBaseView(APIView):
     authentication_classes = [SessionAuthentication, TokenAuthentication]
     permission_classes = [IsAuthenticated]
     search_base = False
+    data = False
 
     def __init__(self):
         super().__init__()
         self.response = {"data": False, "config": AppConfig().config}
+        self.data = {"query": {"match_all": {}}}
         self.status_code = False
         self.context = False
+        self.pagination_handler = False
 
     def get_document(self, document_id):
         """get single document from es"""
@@ -34,42 +44,38 @@ class ApiBaseView(APIView):
         print(path)
         response, status_code = ElasticWrap(path).get()
         try:
-            self.response["data"] = response["_source"]
+            self.response["data"] = SearchProcess(response).process()
         except KeyError:
             print(f"item not found: {document_id}")
             self.response["data"] = False
         self.status_code = status_code
 
-    def process_keys(self):
-        """process keys for frontend"""
-        all_keys = self.response["data"].keys()
-        if "media_url" in all_keys:
-            media_url = self.response["data"]["media_url"]
-            self.response["data"]["media_url"] = f"/media/{media_url}"
-        if "vid_thumb_url" in all_keys:
-            youtube_id = self.response["data"]["youtube_id"]
-            vid_thumb_url = ThumbManager().vid_thumb_path(youtube_id)
-            cache_dir = self.response["config"]["application"]["cache_dir"]
-            new_thumb = f"{cache_dir}/{vid_thumb_url}"
-            self.response["data"]["vid_thumb_url"] = new_thumb
-        if "subtitles" in all_keys:
-            all_subtitles = self.response["data"]["subtitles"]
-            for idx, _ in enumerate(all_subtitles):
-                url = self.response["data"]["subtitles"][idx]["media_url"]
-                new_url = f"/media/{url}"
-                self.response["data"]["subtitles"][idx]["media_url"] = new_url
+    def initiate_pagination(self, request):
+        """set initial pagination values"""
+        user_id = request.user.id
+        page_get = int(request.GET.get("page", 0))
+        self.pagination_handler = Pagination(page_get, user_id)
+        self.data.update(
+            {
+                "size": self.pagination_handler.pagination["page_size"],
+                "from": self.pagination_handler.pagination["page_from"],
+            }
+        )
 
-    def get_paginate(self):
-        """add pagination detail to response"""
-        self.response["paginate"] = False
-
-    def get_document_list(self, data):
+    def get_document_list(self, request):
         """get a list of results"""
         print(self.search_base)
-        response, status_code = ElasticWrap(self.search_base).get(data=data)
-        all_hits = response["hits"]["hits"]
-        self.response["data"] = [i["_source"] for i in all_hits]
-        self.status_code = status_code
+        self.initiate_pagination(request)
+        es_handler = ElasticWrap(self.search_base)
+        response, status_code = es_handler.get(data=self.data)
+        self.response["data"] = SearchProcess(response).process()
+        if self.response["data"]:
+            self.status_code = status_code
+        else:
+            self.status_code = 404
+
+        self.pagination_handler.validate(response["hits"]["total"]["value"])
+        self.response["paginate"] = self.pagination_handler.pagination
 
 
 class VideoApiView(ApiBaseView):
@@ -83,10 +89,24 @@ class VideoApiView(ApiBaseView):
         # pylint: disable=unused-argument
         """get request"""
         self.get_document(video_id)
-        self.process_keys()
         return Response(self.response, status=self.status_code)
 
 
+class VideoApiListView(ApiBaseView):
+    """resolves to /api/video/
+    GET: returns list of videos
+    """
+
+    search_base = "ta_video/_search/"
+
+    def get(self, request):
+        """get request"""
+        self.data.update({"sort": [{"published": {"order": "desc"}}]})
+        self.get_document_list(request)
+
+        return Response(self.response)
+
+
 class VideoProgressView(ApiBaseView):
     """resolves to /api/video/<video_id>/
     handle progress status for video
@@ -111,7 +131,7 @@ class VideoProgressView(ApiBaseView):
         position = request.data.get("position", 0)
         key = f"{request.user.id}:progress:{video_id}"
         message = {"position": position, "youtube_id": video_id}
-        RedisArchivist().set_message(key, message, expire=False)
+        RedisArchivist().set_message(key, message)
         self.response = request.data
 
         return Response(self.response)
@@ -125,6 +145,55 @@ class VideoProgressView(ApiBaseView):
         return Response(self.response)
 
 
+class VideoSponsorView(ApiBaseView):
+    """resolves to /api/video/<video_id>/sponsor/
+    handle sponsor block integration
+    """
+
+    search_base = "ta_video/_doc/"
+
+    def get(self, request, video_id):
+        """get sponsor info"""
+        # pylint: disable=unused-argument
+
+        self.get_document(video_id)
+        sponsorblock = self.response["data"].get("sponsorblock")
+
+        return Response(sponsorblock)
+
+    def post(self, request, video_id):
+        """post verification and timestamps"""
+        if "segment" in request.data:
+            response, status_code = self._create_segment(request, video_id)
+        elif "vote" in request.data:
+            response, status_code = self._vote_on_segment(request)
+
+        return Response(response, status=status_code)
+
+    @staticmethod
+    def _create_segment(request, video_id):
+        """create segment in API"""
+        start_time = request.data["segment"]["startTime"]
+        end_time = request.data["segment"]["endTime"]
+        response, status_code = SponsorBlock(request.user.id).post_timestamps(
+            video_id, start_time, end_time
+        )
+
+        return response, status_code
+
+    @staticmethod
+    def _vote_on_segment(request):
+        """validate on existing segment"""
+        user_id = request.user.id
+        uuid = request.data["vote"]["uuid"]
+        vote = request.data["vote"]["yourVote"]
+        response, status_code = SponsorBlock(user_id).vote_on_segment(
+            uuid, vote
+        )
+
+        return response, status_code
+
+
 class ChannelApiView(ApiBaseView):
     """resolves to /api/channel/<channel_id>/
     GET: returns metadata dict of channel
@@ -148,11 +217,11 @@ class ChannelApiListView(ApiBaseView):
     search_base = "ta_channel/_search/"
 
     def get(self, request):
-        # pylint: disable=unused-argument
         """get request"""
-        data = {"query": {"match_all": {}}}
-        self.get_document_list(data)
-        self.get_paginate()
+        self.get_document_list(request)
+        self.data.update(
+            {"sort": [{"channel_name.keyword": {"order": "asc"}}]}
+        )
 
         return Response(self.response)
 
@@ -174,6 +243,44 @@ class ChannelApiListView(ApiBaseView):
         return Response(data)
 
 
+class ChannelApiVideoView(ApiBaseView):
+    """resolves to /api/channel/<channel-id>/video
+    GET: returns a list of videos of channel
+    """
+
+    search_base = "ta_video/_search/"
+
+    def get(self, request, channel_id):
+        """handle get request"""
+        self.data.update(
+            {
+                "query": {
+                    "term": {"channel.channel_id": {"value": channel_id}}
+                },
+                "sort": [{"published": {"order": "desc"}}],
+            }
+        )
+        self.get_document_list(request)
+
+        return Response(self.response, status=self.status_code)
+
+
+class PlaylistApiListView(ApiBaseView):
+    """resolves to /api/playlist/
+    GET: returns list of indexed playlists
+    """
+
+    search_base = "ta_playlist/_search/"
+
+    def get(self, request):
+        """handle get request"""
+        self.data.update(
+            {"sort": [{"playlist_name.keyword": {"order": "asc"}}]}
+        )
+        self.get_document_list(request)
+        return Response(self.response)
+
+
 class PlaylistApiView(ApiBaseView):
     """resolves to /api/playlist/<playlist_id>/
     GET: returns metadata dict of playlist
@@ -188,12 +295,33 @@ class PlaylistApiView(ApiBaseView):
         return Response(self.response, status=self.status_code)
 
 
+class PlaylistApiVideoView(ApiBaseView):
+    """resolves to /api/playlist/<playlist_id>/video
+    GET: returns list of videos in playlist
+    """
+
+    search_base = "ta_video/_search/"
+
+    def get(self, request, playlist_id):
+        """handle get request"""
+        self.data["query"] = {
+            "term": {"playlist.keyword": {"value": playlist_id}}
+        }
+        self.data.update({"sort": [{"published": {"order": "desc"}}]})
+
+        self.get_document_list(request)
+        return Response(self.response, status=self.status_code)
+
+
 class DownloadApiView(ApiBaseView):
     """resolves to /api/download/<video_id>/
     GET: returns metadata dict of an item in the download queue
+    POST: update status of item to pending or ignore
+    DELETE: forget from download queue
     """
 
     search_base = "ta_download/_doc/"
+    valid_status = ["pending", "ignore"]
 
     def get(self, request, video_id):
         # pylint: disable=unused-argument
@@ -201,21 +329,53 @@ class DownloadApiView(ApiBaseView):
         self.get_document(video_id)
         return Response(self.response, status=self.status_code)
 
+    def post(self, request, video_id):
+        """post to video to change status"""
+        item_status = request.data["status"]
+        if item_status not in self.valid_status:
+            message = f"{video_id}: invalid status {item_status}"
+            print(message)
+            return Response({"message": message}, status=400)
+
+        print(f"{video_id}: change status to {item_status}")
+        PendingInteract(video_id=video_id, status=item_status).update_status()
+        RedisQueue().clear_item(video_id)
+
+        return Response(request.data)
+
+    @staticmethod
+    def delete(request, video_id):
+        # pylint: disable=unused-argument
+        """delete single video from queue"""
+        print(f"{video_id}: delete from queue")
+        PendingInteract(video_id=video_id).delete_item()
+
+        return Response({"success": True})
+
 
 class DownloadApiListView(ApiBaseView):
     """resolves to /api/download/
     GET: returns latest videos in the download queue
     POST: add a list of videos to download queue
+    DELETE: remove items based on query filter
     """
 
     search_base = "ta_download/_search/"
+    valid_filter = ["pending", "ignore"]
 
     def get(self, request):
-        # pylint: disable=unused-argument
         """get request"""
-        data = {"query": {"match_all": {}}}
-        self.get_document_list(data)
-        self.get_paginate()
+        query_filter = request.GET.get("filter", False)
+        self.data.update({"sort": [{"timestamp": {"order": "asc"}}]})
+        if query_filter:
+            if query_filter not in self.valid_filter:
+                message = f"invalid url query filder: {query_filter}"
+                print(message)
+                return Response({"message": message}, status=400)
+
+            self.data["query"] = {"term": {"status": {"value": query_filter}}}
+
+        self.get_document_list(request)
         return Response(self.response)
 
     @staticmethod
@@ -242,3 +402,127 @@ class DownloadApiListView(ApiBaseView):
         extrac_dl.delay(youtube_ids)
 
         return Response(data)
+
+    def delete(self, request):
+        """delete download queue"""
+        query_filter = request.GET.get("filter", False)
+        if query_filter not in self.valid_filter:
+            message = f"invalid url query filter: {query_filter}"
+            print(message)
+            return Response({"message": message}, status=400)
+
+        message = f"delete queue by status: {query_filter}"
+        print(message)
+        PendingInteract(status=query_filter).delete_by_status()
+
+        return Response({"message": message})
+
+
+class PingView(ApiBaseView):
+    """resolves to /api/ping/
+    GET: test your connection
+    """
+
+    @staticmethod
+    def get(request):
+        """get pong"""
+        data = {"response": "pong", "user": request.user.id}
+        return Response(data)
+
+
+class LoginApiView(ObtainAuthToken):
+    """resolves to /api/login/
+    POST: return token and username after successful login
+    """
+
+    def post(self, request, *args, **kwargs):
+        """post data"""
+        # pylint: disable=no-member
+        serializer = self.serializer_class(
+            data=request.data, context={"request": request}
+        )
+        serializer.is_valid(raise_exception=True)
+        user = serializer.validated_data["user"]
+        token, _ = Token.objects.get_or_create(user=user)
+
+        print(f"returning token for user with id {user.pk}")
+
+        return Response({"token": token.key, "user_id": user.pk})
+
+
+class TaskApiView(ApiBaseView):
+    """resolves to /api/task/
+    GET: check if ongoing background task
+    POST: start a new background task
+    """
+
+    @staticmethod
+    def get(request):
+        """handle get request"""
+        # pylint: disable=unused-argument
+        response = {"rescan": False, "downloading": False}
+        for key in response.keys():
+            response[key] = RedisArchivist().is_locked(key)
+
+        return Response(response)
+
+    def post(self, request):
+        """handle post request"""
+
+        data = request.data
+        print(data)
+        response = TaskHandler(data).run_task()
+
+        return Response(response)
+
+
+class CookieView(ApiBaseView):
+    """resolves to /api/cookie/
+    GET: check if cookie is enabled
+    POST: verify validity of cookie
+    PUT: import cookie
+    """
+
+    @staticmethod
+    def get(request):
+        """handle get request"""
+        # pylint: disable=unused-argument
+        config = AppConfig().config
+        valid = RedisArchivist().get_message("cookie:valid")
+        response = {"cookie_enabled": config["downloads"]["cookie_import"]}
+        response.update(valid)
+
+        return Response(response)
+
+    @staticmethod
+    def post(request):
+        """handle post request"""
+        # pylint: disable=unused-argument
+        config = AppConfig().config
+        validated = CookieHandler(config).validate()
+
+        return Response({"cookie_validated": validated})
+
+    @staticmethod
+    def put(request):
+        """handle put request"""
+        # pylint: disable=unused-argument
+        config = AppConfig().config
+        cookie = request.data.get("cookie")
+        if not cookie:
+            message = "missing cookie key in request data"
+            print(message)
+            return Response({"message": message}, status=400)
+
+        print(f"cookie preview:\n\n{cookie[:300]}")
+        handler = CookieHandler(config)
+        handler.set_cookie(cookie)
+        validated = handler.validate()
+        if not validated:
+            handler.revoke()
+            message = {"cookie_import": "fail", "cookie_validated": validated}
+            print(f"cookie: {message}")
+            return Response({"message": message}, status=400)
+
+        message = {"cookie_import": "done", "cookie_validated": validated}
+        return Response(message)

tubearchivist/config/settings.py

@@ -146,8 +146,21 @@ LOGOUT_REDIRECT_URL = "/login/"
 
 # Cors needed for browser extension
 # background.js makes the request so HTTP_ORIGIN will be from extension
-CORS_ALLOWED_ORIGIN_REGEXES = [r"moz-extension://*", r"chrome-extension://*"]
+if environ.get("DISABLE_CORS"):
+    # disable cors
+    CORS_ORIGIN_ALLOW_ALL = True
+else:
+    CORS_ALLOWED_ORIGIN_REGEXES = [
+        r"moz-extension://*",
+        r"chrome-extension://*",
+    ]
+    CORS_ALLOWED_ORIGINS = ["http://localhost:3000"]
+
 
 CORS_ALLOW_HEADERS = list(default_headers) + [
     "mode",
 ]
+
+# TA application settings
+TA_UPSTREAM = "https://github.com/tubearchivist/tubearchivist"
+TA_VERSION = "v0.1.7"

tubearchivist/home/apps.py

@@ -28,6 +28,7 @@ class StartupCheck:
         self.release_lock()
         index_check()
         self.sync_redis_state()
+        self.set_redis_conf()
         self.make_folders()
         self.set_has_run()
 
@@ -45,6 +46,10 @@ class StartupCheck:
         print("sync redis")
         self.config_handler.load_new_defaults()
 
+    def set_redis_conf(self):
+        """set conf values for redis"""
+        self.redis_con.conn.config_set("timeout", 3600)
+
     def make_folders(self):
         """make needed cache folders here so docker doesn't mess it up"""
         folders = [
@@ -97,9 +102,8 @@ class StartupCheck:
 
         if invalid:
             print(
-                "minial required elasticsearch version: "
-                + f"{self.MIN_MAJOR}.{self.MIN_MINOR}, "
-                + "please update to recommended version."
+                "required elasticsearch version: "
+                + f"{self.MIN_MAJOR}.{self.MIN_MINOR}"
             )
             sys.exit(1)
 

tubearchivist/home/config.json

@@ -8,7 +8,8 @@
         "home": "grid",
         "channel": "list",
         "downloads": "list",
-        "playlist": "grid"
+        "playlist": "grid",
+        "grid_items": 3
     },
     "subscriptions": {
         "auto_search": false,
@@ -26,8 +27,10 @@
         "subtitle": false,
         "subtitle_source": false,
         "subtitle_index": false,
+        "cookie_import": false,
         "throttledratelimit": false,
-        "integrate_ryd": false
+        "integrate_ryd": false,
+        "integrate_sponsorblock": false
     },
     "application": {
         "app_root": "/app",

tubearchivist/home/src/download/queue.py

@@ -7,14 +7,15 @@ Functionality:
 import json
 from datetime import datetime
 
-import yt_dlp
 from home.src.download.subscriptions import (
     ChannelSubscription,
     PlaylistSubscription,
 )
 from home.src.download.thumbnails import ThumbManager
+from home.src.download.yt_dlp_base import YtWrap
 from home.src.es.connect import ElasticWrap, IndexPaginate
 from home.src.index.playlist import YoutubePlaylist
+from home.src.ta.config import AppConfig
 from home.src.ta.helper import DurationConverter
 from home.src.ta.ta_redis import RedisArchivist
 
@@ -101,7 +102,7 @@ class PendingInteract:
     def delete_item(self):
         """delete single item from pending"""
         path = f"ta_download/_doc/{self.video_id}"
-        _, _ = ElasticWrap(path).delete()
+        _, _ = ElasticWrap(path).delete(refresh=True)
 
     def delete_by_status(self):
         """delete all matching item by status"""
@@ -119,8 +120,19 @@ class PendingInteract:
 class PendingList(PendingIndex):
     """manage the pending videos list"""
 
+    yt_obs = {
+        "default_search": "ytsearch",
+        "quiet": True,
+        "check_formats": "selected",
+        "noplaylist": True,
+        "writethumbnail": True,
+        "simulate": True,
+        "socket_timeout": 3,
+    }
+
     def __init__(self, youtube_ids=False):
         super().__init__()
+        self.config = AppConfig().config
         self.youtube_ids = youtube_ids
         self.to_skip = False
         self.missing_videos = False
@@ -138,7 +150,7 @@ class PendingList(PendingIndex):
                 "title": "Adding to download queue.",
                 "message": "Extracting lists",
             }
-            RedisArchivist().set_message("message:add", mess_dict)
+            RedisArchivist().set_message("message:add", mess_dict, expire=True)
             self._process_entry(entry)
 
     def _process_entry(self, entry):
@@ -160,6 +172,8 @@ class PendingList(PendingIndex):
         """add video to list"""
         if url not in self.missing_videos and url not in self.to_skip:
             self.missing_videos.append(url)
+        else:
+            print(f"{url}: skipped adding already indexed video to download.")
 
     def _parse_channel(self, url):
         """add all videos of channel to list"""
@@ -199,10 +213,11 @@ class PendingList(PendingIndex):
             thumb_handler.download_vid(thumb_needed)
             self._notify_add(idx)
 
-        # add last newline
-        bulk_list.append("\n")
-        query_str = "\n".join(bulk_list)
-        _, _ = ElasticWrap("_bulk").post(query_str, ndjson=True)
+        if bulk_list:
+            # add last newline
+            bulk_list.append("\n")
+            query_str = "\n".join(bulk_list)
+            _, _ = ElasticWrap("_bulk").post(query_str, ndjson=True)
 
     def _notify_add(self, idx):
         """send notification for adding videos to download queue"""
@@ -214,27 +229,23 @@ class PendingList(PendingIndex):
             "message": "Progress: " + progress,
         }
         if idx + 1 == len(self.missing_videos):
-            RedisArchivist().set_message("message:add", mess_dict, expire=4)
+            expire = 4
         else:
-            RedisArchivist().set_message("message:add", mess_dict)
+            expire = True
 
+        RedisArchivist().set_message("message:add", mess_dict, expire=expire)
         if idx + 1 % 25 == 0:
             print("adding to queue progress: " + progress)
 
     def get_youtube_details(self, youtube_id):
         """get details from youtubedl for single pending video"""
-        obs = {
-            "default_search": "ytsearch",
-            "quiet": True,
-            "check_formats": "selected",
-            "noplaylist": True,
-            "writethumbnail": True,
-            "simulate": True,
-        }
-        try:
-            vid = yt_dlp.YoutubeDL(obs).extract_info(youtube_id)
-        except yt_dlp.utils.DownloadError:
-            print("failed to extract info for: " + youtube_id)
+        vid = YtWrap(self.yt_obs, self.config).extract(youtube_id)
+        if not vid:
+            return False
+
+        if vid.get("id") != youtube_id:
+            # skip premium videos with different id
+            print(f"{youtube_id}: skipping premium video, id not matching")
             return False
         # stop if video is streaming live now
         if vid["is_live"]:
@@ -259,9 +270,12 @@ class PendingList(PendingIndex):
             "vid_thumb_url": vid["thumbnail"],
             "title": vid["title"],
             "channel_id": vid["channel_id"],
-            "channel_indexed": vid["channel_id"] in self.all_channels,
             "duration": duration_str,
             "published": published,
             "timestamp": int(datetime.now().strftime("%s")),
         }
+        if self.all_channels:
+            youtube_details.update(
+                {"channel_indexed": vid["channel_id"] in self.all_channels}
+            )
         return youtube_details

tubearchivist/home/src/download/subscriptions.py

@@ -4,8 +4,8 @@ Functionality:
 - handle playlist subscriptions
 """
 
-import yt_dlp
 from home.src.download import queue  # partial import
+from home.src.download.yt_dlp_base import YtWrap
 from home.src.es.connect import IndexPaginate
 from home.src.index.channel import YoutubeChannel
 from home.src.index.playlist import YoutubePlaylist
@@ -17,10 +17,7 @@ class ChannelSubscription:
     """manage the list of channels subscribed"""
 
     def __init__(self):
-        config = AppConfig().config
-        self.es_url = config["application"]["es_url"]
-        self.es_auth = config["application"]["es_auth"]
-        self.channel_size = config["subscriptions"]["channel_size"]
+        self.config = AppConfig().config
 
     @staticmethod
     def get_channels(subscribed_only=True):
@@ -39,23 +36,19 @@ class ChannelSubscription:
 
     def get_last_youtube_videos(self, channel_id, limit=True):
         """get a list of last videos from channel"""
-        url = f"https://www.youtube.com/channel/{channel_id}/videos"
         obs = {
-            "default_search": "ytsearch",
-            "quiet": True,
             "skip_download": True,
             "extract_flat": True,
         }
         if limit:
-            obs["playlistend"] = self.channel_size
+            obs["playlistend"] = self.config["subscriptions"]["channel_size"]
 
-        try:
-            chan = yt_dlp.YoutubeDL(obs).extract_info(url, download=False)
-        except yt_dlp.utils.DownloadError:
-            print(f"{channel_id}: failed to extract videos, skipping.")
+        url = f"https://www.youtube.com/channel/{channel_id}/videos"
+        channel = YtWrap(obs, self.config).extract(url)
+        if not channel:
             return False
 
-        last_videos = [(i["id"], i["title"]) for i in chan["entries"]]
+        last_videos = [(i["id"], i["title"]) for i in channel["entries"]]
         return last_videos
 
     def find_missing(self):
@@ -83,11 +76,13 @@ class ChannelSubscription:
                 "message": f"Progress: {idx + 1}/{len(all_channels)}",
             }
             if idx + 1 == len(all_channels):
-                RedisArchivist().set_message(
-                    "message:rescan", message=message, expire=4
-                )
+                expire = 4
             else:
-                RedisArchivist().set_message("message:rescan", message=message)
+                expire = True
+
+            RedisArchivist().set_message(
+                "message:rescan", message=message, expire=expire
+            )
 
         return missing_videos
 
@@ -159,7 +154,7 @@ class PlaylistSubscription:
                 "message": f"Processing {idx + 1} of {len(new_playlists)}",
             }
             RedisArchivist().set_message(
-                "message:subplaylist", message=message
+                "message:subplaylist", message=message, expire=True
             )
 
         return new_thumbs
@@ -168,7 +163,7 @@ class PlaylistSubscription:
     def channel_validate(channel_id):
         """make sure channel of playlist is there"""
         channel = YoutubeChannel(channel_id)
-        channel.build_json()
+        channel.build_json(upload=True)
 
     @staticmethod
     def change_subscribe(playlist_id, subscribe_status):
@@ -213,7 +208,9 @@ class PlaylistSubscription:
                 "title": "Scanning playlists: Looking for new videos.",
                 "message": f"Progress: {idx + 1}/{len(all_playlists)}",
             }
-            RedisArchivist().set_message("message:rescan", message=message)
+            RedisArchivist().set_message(
+                "message:rescan", message=message, expire=True
+            )
 
             for video in all_missing:
                 youtube_id = video["youtube_id"]

tubearchivist/home/src/download/thumbnails.py

@@ -4,8 +4,10 @@ functionality:
 - check for missing thumbnails
 """
 
+import base64
 import os
 from collections import Counter
+from io import BytesIO
 from time import sleep
 
 import requests
@@ -15,7 +17,9 @@ from home.src.ta.config import AppConfig
 from home.src.ta.helper import ignore_filelist
 from home.src.ta.ta_redis import RedisArchivist
 from mutagen.mp4 import MP4, MP4Cover
-from PIL import Image
+from PIL import Image, ImageFile, ImageFilter
+
+ImageFile.LOAD_TRUNCATED_IMAGES = True
 
 
 class ThumbManager:
@@ -189,11 +193,13 @@ class ThumbManager:
                     "message": "Downloading Thumbnails, Progress: " + progress,
                 }
                 if idx + 1 == len(missing_thumbs):
-                    RedisArchivist().set_message(
-                        "message:add", mess_dict, expire=4
-                    )
+                    expire = 4
                 else:
-                    RedisArchivist().set_message("message:add", mess_dict)
+                    expire = True
+
+                RedisArchivist().set_message(
+                    "message:add", mess_dict, expire=expire
+                )
 
             if idx + 1 % 25 == 0:
                 print("thumbnail progress: " + progress)
@@ -222,7 +228,8 @@ class ThumbManager:
                 "title": "Processing Channels",
                 "message": "Downloading Channel Art.",
             }
-            RedisArchivist().set_message("message:download", mess_dict)
+            key = "message:download"
+            RedisArchivist().set_message(key, mess_dict, expire=True)
 
     def download_playlist(self, missing_playlists):
         """download needed artwork for playlists"""
@@ -239,7 +246,23 @@ class ThumbManager:
                 "title": "Processing Playlists",
                 "message": "Downloading Playlist Art.",
             }
-            RedisArchivist().set_message("message:download", mess_dict)
+            key = "message:download"
+            RedisArchivist().set_message(key, mess_dict, expire=True)
+
+    def get_base64_blur(self, youtube_id):
+        """return base64 encoded placeholder"""
+        img_path = self.vid_thumb_path(youtube_id)
+        file_path = os.path.join(self.CACHE_DIR, img_path)
+        img_raw = Image.open(file_path)
+        img_raw.thumbnail((img_raw.width // 20, img_raw.height // 20))
+        img_blur = img_raw.filter(ImageFilter.BLUR)
+        buffer = BytesIO()
+        img_blur.save(buffer, format="JPEG")
+        img_data = buffer.getvalue()
+        img_base64 = base64.b64encode(img_data).decode()
+        data_url = f"data:image/jpg;base64,{img_base64}"
+
+        return data_url
 
     @staticmethod
     def vid_thumb_path(youtube_id):
@@ -279,6 +302,7 @@ class ThumbManager:
     def get_thumb_list(self):
         """get list of mediafiles and matching thumbnails"""
         pending = queue.PendingList()
+        pending.get_download()
         pending.get_indexed()
 
         video_list = []

tubearchivist/home/src/download/yt_dlp_base.py

@@ -0,0 +1,151 @@
+"""
+functionality:
+- base class to make all calls to yt-dlp
+- handle yt-dlp errors
+"""
+
+import os
+from datetime import datetime
+from http import cookiejar
+from io import StringIO
+
+import yt_dlp
+from home.src.ta.ta_redis import RedisArchivist
+
+
+class YtWrap:
+    """wrap calls to yt"""
+
+    OBS_BASE = {
+        "default_search": "ytsearch",
+        "quiet": True,
+        "check_formats": "selected",
+        "socket_timeout": 2,
+    }
+
+    def __init__(self, obs_request, config=False):
+        self.obs_request = obs_request
+        self.config = config
+        self.build_obs()
+
+    def build_obs(self):
+        """build yt-dlp obs"""
+        self.obs = self.OBS_BASE.copy()
+        self.obs.update(self.obs_request)
+        if self.config:
+            self.add_cookie()
+
+    def add_cookie(self):
+        """add cookie if enabled"""
+        if self.config["downloads"]["cookie_import"]:
+            cookie_io = CookieHandler(self.config).get()
+            self.obs["cookiefile"] = cookie_io
+
+    def download(self, url):
+        """make download request"""
+        with yt_dlp.YoutubeDL(self.obs) as ydl:
+            try:
+                ydl.download([url])
+            except yt_dlp.utils.DownloadError:
+                print(f"{url}: failed to download.")
+                return False
+
+        return True
+
+    def extract(self, url):
+        """make extract request"""
+        try:
+            response = yt_dlp.YoutubeDL(self.obs).extract_info(url)
+        except cookiejar.LoadError:
+            print("cookie file is invalid")
+            return False
+        except (yt_dlp.utils.ExtractorError, yt_dlp.utils.DownloadError):
+            print(f"{url}: failed to get info from youtube")
+            return False
+
+        return response
+
+
+class CookieHandler:
+    """handle youtube cookie for yt-dlp"""
+
+    def __init__(self, config):
+        self.cookie_io = False
+        self.config = config
+
+    def get(self):
+        """get cookie io stream"""
+        cookie = RedisArchivist().get_message("cookie")
+        self.cookie_io = StringIO(cookie)
+        return self.cookie_io
+
+    def import_cookie(self):
+        """import cookie from file"""
+        cache_path = self.config["application"]["cache_dir"]
+        import_path = os.path.join(cache_path, "import", "cookies.google.txt")
+        with open(import_path, encoding="utf-8") as cookie_file:
+            cookie = cookie_file.read()
+
+        self.set_cookie(cookie)
+
+        os.remove(import_path)
+        print("cookie: import successful")
+
+    def set_cookie(self, cookie):
+        """set cookie str and activate in cofig"""
+        RedisArchivist().set_message("cookie", cookie)
+        path = ".downloads.cookie_import"
+        RedisArchivist().set_message("config", True, path=path)
+        self.config["downloads"]["cookie_import"] = True
+        print("cookie: activated and stored in Redis")
+
+    @staticmethod
+    def revoke():
+        """revoke cookie"""
+        RedisArchivist().del_message("cookie")
+        RedisArchivist().del_message("cookie:valid")
+        RedisArchivist().set_message(
+            "config", False, path=".downloads.cookie_import"
+        )
+        print("cookie: revoked")
+
+    def validate(self):
+        """validate cookie using the liked videos playlist"""
+        print("validating cookie")
+        obs_request = {
+            "skip_download": True,
+            "extract_flat": True,
+        }
+        validator = YtWrap(obs_request, self.config)
+        response = bool(validator.extract("LL"))
+        self.store_validation(response)
+
+        # update in redis to avoid expiring
+        modified = validator.obs["cookiefile"].getvalue()
+        if modified:
+            RedisArchivist().set_message("cookie", modified)
+
+        if not response:
+            mess_dict = {
+                "status": "message:download",
+                "level": "error",
+                "title": "Cookie validation failed, exiting...",
+                "message": "",
+            }
+            RedisArchivist().set_message(
+                "message:download", mess_dict, expire=4
+            )
+            print("cookie validation failed, exiting...")
+
+        return response
+
+    @staticmethod
+    def store_validation(response):
+        """remember last validation"""
+        now = datetime.now()
+        message = {
+            "status": response,
+            "validated": int(now.strftime("%s")),
+            "validated_str": now.strftime("%Y-%m-%d %H:%M"),
+        }
+        RedisArchivist().set_message("cookie:valid", message)

tubearchivist/home/src/download/yt_dlp_handler.py

@@ -9,11 +9,10 @@ functionality:
 import os
 import shutil
 from datetime import datetime
-from time import sleep
 
-import yt_dlp
 from home.src.download.queue import PendingList
 from home.src.download.subscriptions import PlaylistSubscription
+from home.src.download.yt_dlp_base import CookieHandler, YtWrap
 from home.src.es.connect import ElasticWrap, IndexPaginate
 from home.src.index.channel import YoutubeChannel
 from home.src.index.playlist import YoutubePlaylist
@@ -126,18 +125,19 @@ class DownloadPostProcess:
             + f"{id_c + 1}/{len(self.download.channels)}"
         )
         message = f"Progress: {id_p + 1}/{len(all_channel_playlist)}"
+        key = "message:download"
         mess_dict = {
-            "status": "message:download",
+            "status": key,
             "level": "info",
             "title": title,
             "message": message,
         }
         if id_p + 1 == len(all_channel_playlist):
-            RedisArchivist().set_message(
-                "message:download", mess_dict, expire=4
-            )
+            expire = 4
         else:
-            RedisArchivist().set_message("message:download", mess_dict)
+            expire = True
+
+        RedisArchivist().set_message(key, mess_dict, expire=expire)
 
 
 class VideoDownloader:
@@ -146,6 +146,8 @@ class VideoDownloader:
     if not initiated with list, take from queue
     """
 
+    MSG = "message:download"
+
     def __init__(self, youtube_id_list=False):
         self.obs = False
         self.video_overwrites = False
@@ -156,10 +158,7 @@ class VideoDownloader:
 
     def run_queue(self):
         """setup download queue in redis loop until no more items"""
-        pending = PendingList()
-        pending.get_download()
-        pending.get_channels()
-        self.video_overwrites = pending.video_overwrites
+        self._setup_queue()
 
         queue = RedisQueue()
 
@@ -172,30 +171,57 @@ class VideoDownloader:
             if not youtube_id:
                 break
 
-            try:
-                self._dl_single_vid(youtube_id)
-            except yt_dlp.utils.DownloadError:
-                print("failed to download " + youtube_id)
+            success = self._dl_single_vid(youtube_id)
+            if not success:
                 continue
-            vid_dict = index_new_video(youtube_id)
+
+            vid_dict = index_new_video(
+                youtube_id, video_overwrites=self.video_overwrites
+            )
             self.channels.add(vid_dict["channel"]["channel_id"])
+            mess_dict = {
+                "status": self.MSG,
+                "level": "info",
+                "title": "Moving....",
+                "message": "Moving downloaded file to storage folder",
+            }
+            RedisArchivist().set_message(self.MSG, mess_dict)
+
             self.move_to_archive(vid_dict)
+            mess_dict = {
+                "status": self.MSG,
+                "level": "info",
+                "title": "Completed",
+                "message": "",
+            }
+            RedisArchivist().set_message(self.MSG, mess_dict, expire=10)
             self._delete_from_pending(youtube_id)
 
         # post processing
         self._add_subscribed_channels()
         DownloadPostProcess(self).run()
 
-    @staticmethod
-    def add_pending():
+    def _setup_queue(self):
+        """setup required and validate"""
+        if self.config["downloads"]["cookie_import"]:
+            valid = CookieHandler(self.config).validate()
+            if not valid:
+                return
+
+        pending = PendingList()
+        pending.get_download()
+        pending.get_channels()
+        self.video_overwrites = pending.video_overwrites
+
+    def add_pending(self):
         """add pending videos to download queue"""
         mess_dict = {
-            "status": "message:download",
+            "status": self.MSG,
             "level": "info",
             "title": "Looking for videos to download",
             "message": "Scanning your download queue.",
         }
-        RedisArchivist().set_message("message:download", mess_dict)
+        RedisArchivist().set_message(self.MSG, mess_dict, expire=True)
         pending = PendingList()
         pending.get_download()
         to_add = [i["youtube_id"] for i in pending.all_pending]
@@ -203,18 +229,17 @@ class VideoDownloader:
             # there is nothing pending
             print("download queue is empty")
             mess_dict = {
-                "status": "message:download",
+                "status": self.MSG,
                 "level": "error",
                 "title": "Download queue is empty",
                 "message": "Add some videos to the queue first.",
             }
-            RedisArchivist().set_message("message:download", mess_dict)
+            RedisArchivist().set_message(self.MSG, mess_dict, expire=True)
             return
 
         RedisQueue().add_list(to_add)
 
-    @staticmethod
-    def _progress_hook(response):
+    def _progress_hook(self, response):
         """process the progress_hooks from yt_dlp"""
         # title
         path = os.path.split(response["filename"])[-1][12:]
@@ -231,12 +256,12 @@ class VideoDownloader:
         except KeyError:
             message = "processing"
         mess_dict = {
-            "status": "message:download",
+            "status": self.MSG,
             "level": "info",
             "title": title,
             "message": message,
         }
-        RedisArchivist().set_message("message:download", mess_dict)
+        RedisArchivist().set_message(self.MSG, mess_dict, expire=True)
 
     def _build_obs(self):
         """collection to build all obs passed to yt-dlp"""
@@ -263,6 +288,7 @@ class VideoDownloader:
             "writethumbnail": False,
             "noplaylist": True,
             "check_formats": "selected",
+            "socket_timeout": 3,
         }
 
     def _build_obs_user(self):
@@ -290,6 +316,17 @@ class VideoDownloader:
                     "add_metadata": True,
                 }
             )
+            postprocessors.append(
+                {
+                    "key": "MetadataFromField",
+                    "formats": [
+                        "%(title)s:%(meta_title)s",
+                        "%(uploader)s:%(meta_artist)s",
+                        ":(?P<album>)",
+                    ],
+                    "when": "pre_process",
+                }
+            )
 
         if self.config["downloads"]["add_thumbnail"]:
             postprocessors.append(
@@ -325,13 +362,7 @@ class VideoDownloader:
             if youtube_id in file_name:
                 obs["outtmpl"] = os.path.join(dl_cache, file_name)
 
-        with yt_dlp.YoutubeDL(obs) as ydl:
-            try:
-                ydl.download([youtube_id])
-            except yt_dlp.utils.DownloadError:
-                print("retry failed download: " + youtube_id)
-                sleep(10)
-                ydl.download([youtube_id])
+        success = YtWrap(obs, self.config).download(youtube_id)
 
         if self.obs["writethumbnail"]:
             # webp files don't get cleaned up automatically
@@ -341,6 +372,8 @@ class VideoDownloader:
                 file_path = os.path.join(dl_cache, file_name)
                 os.remove(file_path)
 
+        return success
+
     def move_to_archive(self, vid_dict):
         """move downloaded video from cache to archive"""
         videos = self.config["application"]["videos"]

tubearchivist/home/src/es/connect.py

@@ -75,8 +75,10 @@ class ElasticWrap:
 
         return response.json(), response.status_code
 
-    def delete(self, data=False):
+    def delete(self, data=False, refresh=False):
         """delete document from es"""
+        if refresh:
+            self.url = f"{self.url}/?refresh=true"
         if data:
             response = requests.delete(self.url, json=data, auth=self.auth)
         else:

tubearchivist/home/src/es/index_mapping.json

@@ -50,6 +50,9 @@
                         },
                         "index_playlists": {
                             "type": "boolean"
+                        },
+                        "integrate_sponsorblock": {
+                            "type" : "boolean"
                         }
                     }
                 }
@@ -73,6 +76,10 @@
                     "type": "text",
                     "index": false
                 },
+                "vid_thumb_base64": {
+                    "type": "text",
+                    "index": false
+                },
                 "date_downloaded": {
                     "type": "date"
                 },
@@ -126,6 +133,9 @@
                                 },
                                 "index_playlists": {
                                     "type": "boolean"
+                                },
+                                "integrate_sponsorblock": {
+                                    "type" : "boolean"
                                 }
                             }
                         }
@@ -183,6 +193,22 @@
                         }
                     }
                 },
+                "stats" : {
+                    "properties" : {
+                        "average_rating" : {
+                            "type" : "float"
+                        },
+                        "dislike_count" : {
+                            "type" : "long"
+                        },
+                        "like_count" : {
+                            "type" : "long"
+                        },
+                        "view_count" : {
+                            "type" : "long"
+                        }
+                    }
+                },
                 "subtitles": {
                     "properties": {
                         "ext": {
@@ -208,6 +234,44 @@
                             "index": false
                         }
                     }
+                },
+                "sponsorblock": {
+                    "properties": {
+                        "last_refresh": {
+                            "type": "date"
+                        },
+                        "has_unlocked": {
+                            "type": "boolean"
+                        },
+                        "is_enabled": {
+                            "type": "boolean"
+                        },
+                        "segments" : {
+                            "properties" : {
+                                "UUID" : {
+                                    "type": "keyword"
+                                },
+                                "actionType" : {
+                                    "type": "keyword"
+                                },
+                                "category" : {
+                                    "type": "keyword"
+                                },
+                                "locked" : {
+                                    "type" : "short"
+                                },
+                                "segment" : {
+                                    "type" : "float"
+                                },
+                                "videoDuration" : {
+                                    "type" : "float"
+                                },
+                                "votes" : {
+                                    "type" : "long"
+                                }
+                            }
+                        }
+                    }
                 }
             },
             "expected_set": {

tubearchivist/home/src/es/index_setup.py

@@ -377,6 +377,7 @@ def backup_all_indexes(reason):
 
     for index in backup_handler.index_config:
         index_name = index["index_name"]
+        print(f"backup: export in progress for {index_name}")
         if not backup_handler.index_exists(index_name):
             continue
         all_results = backup_handler.get_all_documents(index_name)

tubearchivist/home/src/frontend/api_calls.py

@@ -19,7 +19,6 @@ from home.src.ta.ta_redis import RedisArchivist, RedisQueue
 from home.tasks import (
     download_pending,
     download_single,
-    extrac_dl,
     index_channel_playlists,
     kill_dl,
     re_sync_thumbs,
@@ -55,6 +54,7 @@ class PostData:
             "watched": self._watched,
             "un_watched": self._un_watched,
             "change_view": self._change_view,
+            "change_grid": self._change_grid,
             "rescan_pending": self._rescan_pending,
             "ignore": self._ignore,
             "dl_pending": self._dl_pending,
@@ -98,7 +98,18 @@ class PostData:
         origin, new_view = self.exec_val.split(":")
         key = f"{self.current_user}:view:{origin}"
         print(f"change view: {key} to {new_view}")
-        RedisArchivist().set_message(key, {"status": new_view}, expire=False)
+        RedisArchivist().set_message(key, {"status": new_view})
+        return {"success": True}
+
+    def _change_grid(self):
+        """process change items in grid"""
+        grid_items = int(self.exec_val)
+        grid_items = max(grid_items, 3)
+        grid_items = min(grid_items, 7)
+
+        key = f"{self.current_user}:grid_items"
+        print(f"change grid items: {grid_items}")
+        RedisArchivist().set_message(key, {"status": grid_items})
         return {"success": True}
 
     @staticmethod
@@ -111,7 +122,7 @@ class PostData:
     def _ignore(self):
         """ignore from download queue"""
         video_id = self.exec_val
-        print(f"ignore video {video_id}")
+        print(f"{video_id}: ignore video from download queue")
         PendingInteract(video_id=video_id, status="ignore").update_status()
         # also clear from redis queue
         RedisQueue().clear_item(video_id)
@@ -123,8 +134,8 @@ class PostData:
         print("download pending")
         running = download_pending.delay()
         task_id = running.id
-        print("set task id: " + task_id)
-        RedisArchivist().set_message("dl_queue_id", task_id, expire=False)
+        print(f"{task_id}: set task id")
+        RedisArchivist().set_message("dl_queue_id", task_id)
         return {"success": True}
 
     def _queue_handler(self):
@@ -146,7 +157,7 @@ class PostData:
     def _unsubscribe(self):
         """unsubscribe from channels or playlists"""
         id_unsub = self.exec_val
-        print("unsubscribe from " + id_unsub)
+        print(f"{id_unsub}: unsubscribe")
         to_unsub_list = UrlListParser(id_unsub).process_list()
         for to_unsub in to_unsub_list:
             unsub_type = to_unsub["type"]
@@ -167,7 +178,7 @@ class PostData:
     def _subscribe(self):
         """subscribe to channel or playlist, called from js buttons"""
         id_sub = self.exec_val
-        print("subscribe to " + id_sub)
+        print(f"{id_sub}: subscribe")
         subscribe_to.delay(id_sub)
         return {"success": True}
 
@@ -176,11 +187,11 @@ class PostData:
         sort_order = {"status": self.exec_val}
         if self.exec_val in ["asc", "desc"]:
             RedisArchivist().set_message(
-                f"{self.current_user}:sort_order", sort_order, expire=False
+                f"{self.current_user}:sort_order", sort_order
             )
         else:
             RedisArchivist().set_message(
-                f"{self.current_user}:sort_by", sort_order, expire=False
+                f"{self.current_user}:sort_by", sort_order
             )
         return {"success": True}
 
@@ -189,7 +200,7 @@ class PostData:
         key = f"{self.current_user}:hide_watched"
         message = {"status": bool(int(self.exec_val))}
         print(f"toggle {key}: {message}")
-        RedisArchivist().set_message(key, message, expire=False)
+        RedisArchivist().set_message(key, message)
         return {"success": True}
 
     def _show_subed_only(self):
@@ -197,17 +208,17 @@ class PostData:
         key = f"{self.current_user}:show_subed_only"
         message = {"status": bool(int(self.exec_val))}
         print(f"toggle {key}: {message}")
-        RedisArchivist().set_message(key, message, expire=False)
+        RedisArchivist().set_message(key, message)
         return {"success": True}
 
     def _dlnow(self):
         """start downloading single vid now"""
         youtube_id = self.exec_val
-        print("downloading: " + youtube_id)
+        print(f"{youtube_id}: downloading now")
         running = download_single.delay(youtube_id=youtube_id)
         task_id = running.id
         print("set task id: " + task_id)
-        RedisArchivist().set_message("dl_queue_id", task_id, expire=False)
+        RedisArchivist().set_message("dl_queue_id", task_id)
         return {"success": True}
 
     def _show_ignored_only(self):
@@ -216,23 +227,21 @@ class PostData:
         key = f"{self.current_user}:show_ignored_only"
         value = {"status": show_value}
         print(f"Filter download view ignored only: {show_value}")
-        RedisArchivist().set_message(key, value, expire=False)
+        RedisArchivist().set_message(key, value)
         return {"success": True}
 
     def _forget_ignore(self):
         """delete from ta_download index"""
         video_id = self.exec_val
-        print(f"forgetting from download index: {video_id}")
+        print(f"{video_id}: forget from download")
         PendingInteract(video_id=video_id).delete_item()
         return {"success": True}
 
     def _add_single(self):
         """add single youtube_id to download queue"""
         video_id = self.exec_val
-        print(f"add vid to dl queue: {video_id}")
-        PendingInteract(video_id=video_id).delete_item()
-        video_ids = UrlListParser(video_id).process_list()
-        extrac_dl.delay(video_ids)
+        print(f"{video_id}: add single vid to download queue")
+        PendingInteract(video_id=video_id, status="pending").update_status()
         return {"success": True}
 
     def _delete_queue(self):

tubearchivist/home/src/frontend/forms.py

@@ -62,6 +62,12 @@ class ApplicationSettingsForm(forms.Form):
         ("1", "enable ryd integration"),
     ]
 
+    SP_CHOICES = [
+        ("", "-- change sponsorblock integrations"),
+        ("0", "disable sponsorblock integration"),
+        ("1", "enable sponsorblock integration"),
+    ]
+
     CAST_CHOICES = [
         ("", "-- change Cast integration --"),
         ("0", "disable Cast"),
@@ -80,6 +86,12 @@ class ApplicationSettingsForm(forms.Form):
         ("1", "enable subtitle index"),
     ]
 
+    COOKIE_IMPORT_CHOICES = [
+        ("", "-- change cookie settings"),
+        ("0", "disable cookie"),
+        ("1", "enable cookie"),
+    ]
+
     subscriptions_channel_size = forms.IntegerField(required=False)
     downloads_limit_count = forms.IntegerField(required=False)
     downloads_limit_speed = forms.IntegerField(required=False)
@@ -100,9 +112,15 @@ class ApplicationSettingsForm(forms.Form):
     downloads_subtitle_index = forms.ChoiceField(
         widget=forms.Select, choices=SUBTITLE_INDEX_CHOICES, required=False
     )
+    downloads_cookie_import = forms.ChoiceField(
+        widget=forms.Select, choices=COOKIE_IMPORT_CHOICES, required=False
+    )
     downloads_integrate_ryd = forms.ChoiceField(
         widget=forms.Select, choices=RYD_CHOICES, required=False
     )
+    downloads_integrate_sponsorblock = forms.ChoiceField(
+        widget=forms.Select, choices=SP_CHOICES, required=False
+    )
     application_enable_cast = forms.ChoiceField(
         widget=forms.Select, choices=CAST_CHOICES, required=False
     )
@@ -189,8 +207,18 @@ class ChannelOverwriteForm(forms.Form):
         ("1", "Enable playlist index"),
     ]
 
+    SP_CHOICES = [
+        ("", "-- change sponsorblock integrations"),
+        ("disable", "disable sponsorblock integration"),
+        ("1", "enable sponsorblock integration"),
+        ("0", "unset sponsorblock integration"),
+    ]
+
     download_format = forms.CharField(label=False, required=False)
     autodelete_days = forms.IntegerField(label=False, required=False)
     index_playlists = forms.ChoiceField(
         widget=forms.Select, choices=PLAYLIST_INDEX, required=False
     )
+    integrate_sponsorblock = forms.ChoiceField(
+        widget=forms.Select, choices=SP_CHOICES, required=False
+    )

tubearchivist/home/src/index/channel.py

@@ -10,10 +10,10 @@ import re
 from datetime import datetime
 
 import requests
-import yt_dlp
 from bs4 import BeautifulSoup
 from home.src.download import queue  # partial import
 from home.src.download.thumbnails import ThumbManager
+from home.src.download.yt_dlp_base import YtWrap
 from home.src.es.connect import ElasticWrap, IndexPaginate
 from home.src.index.generic import YouTubeItem
 from home.src.index.playlist import YoutubePlaylist
@@ -153,6 +153,7 @@ class YoutubeChannel(YouTubeItem):
     es_path = False
     index_name = "ta_channel"
     yt_base = "https://www.youtube.com/channel/"
+    msg = "message:playlistscan"
 
     def __init__(self, youtube_id):
         super().__init__(youtube_id)
@@ -252,12 +253,12 @@ class YoutubeChannel(YouTubeItem):
         self.get_from_es()
         channel_name = self.json_data["channel_name"]
         mess_dict = {
-            "status": "message:playlistscan",
+            "status": self.msg,
             "level": "info",
             "title": "Looking for playlists",
             "message": f"{channel_name}: Scanning channel in progress",
         }
-        RedisArchivist().set_message("message:playlistscan", mess_dict)
+        RedisArchivist().set_message(self.msg, mess_dict, expire=True)
         self.get_all_playlists()
         if not self.all_playlists:
             print(f"{self.youtube_id}: no playlists found.")
@@ -272,12 +273,12 @@ class YoutubeChannel(YouTubeItem):
         """send notification"""
         channel_name = self.json_data["channel_name"]
         mess_dict = {
-            "status": "message:playlistscan",
+            "status": self.msg,
             "level": "info",
             "title": f"{channel_name}: Scanning channel for playlists",
             "message": f"Progress: {idx + 1}/{len(self.all_playlists)}",
         }
-        RedisArchivist().set_message("message:playlistscan", mess_dict)
+        RedisArchivist().set_message(self.msg, mess_dict, expire=True)
         print("add playlist: " + playlist[1])
 
     @staticmethod
@@ -314,12 +315,8 @@ class YoutubeChannel(YouTubeItem):
             f"https://www.youtube.com/channel/{self.youtube_id}"
             + "/playlists?view=1&sort=dd&shelf_id=0"
         )
-        obs = {
-            "quiet": True,
-            "skip_download": True,
-            "extract_flat": True,
-        }
-        playlists = yt_dlp.YoutubeDL(obs).extract_info(url)
+        obs = {"skip_download": True, "extract_flat": True}
+        playlists = YtWrap(obs, self.config).extract(url)
         all_entries = [(i["id"], i["title"]) for i in playlists["entries"]]
         self.all_playlists = all_entries
 
@@ -340,12 +337,20 @@ class YoutubeChannel(YouTubeItem):
 
     def set_overwrites(self, overwrites):
         """set per channel overwrites"""
-        valid_keys = ["download_format", "autodelete_days", "index_playlists"]
+        valid_keys = [
+            "download_format",
+            "autodelete_days",
+            "index_playlists",
+            "integrate_sponsorblock",
+        ]
 
         to_write = self.json_data.get("channel_overwrites", {})
         for key, value in overwrites.items():
             if key not in valid_keys:
                 raise ValueError(f"invalid overwrite key: {key}")
+            if value == "disable":
+                to_write[key] = False
+                continue
             if value in [0, "0"]:
                 del to_write[key]
                 continue

tubearchivist/home/src/index/filesystem.py

@@ -310,4 +310,4 @@ def reindex_old_documents():
     handler = Reindex()
     handler.check_outdated()
     handler.reindex()
-    RedisArchivist().set_message("last_reindex", handler.now, expire=False)
+    RedisArchivist().set_message("last_reindex", handler.now)

tubearchivist/home/src/index/generic.py

@@ -5,7 +5,7 @@ functionality:
 
 import math
 
-import yt_dlp
+from home.src.download.yt_dlp_base import YtWrap
 from home.src.es.connect import ElasticWrap
 from home.src.ta.config import AppConfig
 from home.src.ta.ta_redis import RedisArchivist
@@ -18,38 +18,22 @@ class YouTubeItem:
     index_name = False
     yt_base = False
     yt_obs = {
-        "quiet": True,
-        "default_search": "ytsearch",
         "skip_download": True,
-        "check_formats": "selected",
         "noplaylist": True,
     }
 
     def __init__(self, youtube_id):
         self.youtube_id = youtube_id
-        self.config = False
-        self.app_conf = False
-        self.youtube_meta = False
-        self.json_data = False
-        self._get_conf()
-
-    def _get_conf(self):
-        """read user conf"""
         self.config = AppConfig().config
         self.app_conf = self.config["application"]
+        self.youtube_meta = False
+        self.json_data = False
 
     def get_from_youtube(self):
         """use yt-dlp to get meta data from youtube"""
         print(f"{self.youtube_id}: get metadata from youtube")
-        try:
-            yt_item = yt_dlp.YoutubeDL(self.yt_obs)
-            response = yt_item.extract_info(self.yt_base + self.youtube_id)
-        except (
-            yt_dlp.utils.ExtractorError,
-            yt_dlp.utils.DownloadError,
-        ):
-            print(f"{self.youtube_id}: failed to get info from youtube")
-            response = False
+        url = self.yt_base + self.youtube_id
+        response = YtWrap(self.yt_obs, self.config).extract(url)
 
         self.youtube_meta = response
 
@@ -81,7 +65,7 @@ class YouTubeItem:
     def del_in_es(self):
         """delete item from elastic search"""
         print(f"{self.youtube_id}: delete from es")
-        _, _ = ElasticWrap(self.es_path).delete()
+        _, _ = ElasticWrap(self.es_path).delete(refresh=True)
 
 
 class Pagination:
@@ -133,7 +117,7 @@ class Pagination:
         """validate pagination with total_hits after making api call"""
         page_get = self.page_get
         max_pages = math.ceil(total_hits / self.page_size)
-        if total_hits > 10000:
+        if total_hits >= 10000:
             # es returns maximal 10000 results
             self.pagination["max_hits"] = True
             max_pages = max_pages - 1
@@ -147,3 +131,4 @@ class Pagination:
         ]
 
         self.pagination["next_pages"] = next_pages
+        self.pagination["total_hits"] = total_hits

tubearchivist/home/src/index/playlist.py

@@ -18,12 +18,7 @@ class YoutubePlaylist(YouTubeItem):
 
     es_path = False
     index_name = "ta_playlist"
-    yt_obs = {
-        "default_search": "ytsearch",
-        "quiet": True,
-        "skip_download": True,
-        "extract_flat": True,
-    }
+    yt_obs = {"extract_flat": True}
     yt_base = "https://www.youtube.com/playlist?list="
 
     def __init__(self, youtube_id):

tubearchivist/home/src/index/reindex.py

@@ -4,12 +4,16 @@ functionality:
 - index and update in es
 """
 
+import os
+import shutil
 from datetime import datetime
 from math import ceil
 from time import sleep
 
 from home.src.download.queue import PendingList
 from home.src.download.thumbnails import ThumbManager
+from home.src.download.yt_dlp_base import CookieHandler
+from home.src.download.yt_dlp_handler import VideoDownloader
 from home.src.es.connect import ElasticWrap
 from home.src.index.channel import YoutubeChannel
 from home.src.index.playlist import YoutubePlaylist
@@ -37,10 +41,20 @@ class Reindex:
         self.all_channel_ids = False
         self.all_playlist_ids = False
 
+    def check_cookie(self):
+        """validate cookie if enabled"""
+        if self.config["downloads"]["cookie_import"]:
+            valid = CookieHandler(self.config).validate()
+            if not valid:
+                return
+
     def _get_daily(self):
         """get daily refresh values"""
         total_videos = self._get_total_hits("ta_video")
         video_daily = ceil(total_videos / self.interval * self.MULTIPLY)
+        if video_daily >= 10000:
+            video_daily = 9999
+
         total_channels = self._get_total_hits("ta_channel")
         channel_daily = ceil(total_channels / self.interval * self.MULTIPLY)
         total_playlists = self._get_total_hits("ta_playlist")
@@ -147,6 +161,7 @@ class Reindex:
         date_downloaded = video.json_data["date_downloaded"]
         channel_dict = video.json_data["channel"]
         playlist = video.json_data.get("playlist")
+        subtitles = video.json_data.get("subtitles")
 
         # get new
         video.build_json()
@@ -154,7 +169,7 @@ class Reindex:
             video.deactivate()
             return
 
-        video.delete_subtitles()
+        video.delete_subtitles(subtitles=subtitles)
         video.check_subtitles()
 
         # add back
@@ -208,7 +223,12 @@ class Reindex:
         # videos
         print(f"reindexing {len(self.all_youtube_ids)} videos")
         for youtube_id in self.all_youtube_ids:
-            self._reindex_single_video(youtube_id)
+            try:
+                self._reindex_single_video(youtube_id)
+            except FileNotFoundError:
+                # handle channel name change here
+                ChannelUrlFixer(youtube_id, self.config).run()
+                self._reindex_single_video(youtube_id)
             if sleep_interval:
                 sleep(sleep_interval)
         # channels
@@ -228,3 +248,62 @@ class Reindex:
                 self._reindex_single_playlist(playlist_id, all_indexed_ids)
                 if sleep_interval:
                     sleep(sleep_interval)
+
+
+class ChannelUrlFixer:
+    """fix not matching channel names in reindex"""
+
+    def __init__(self, youtube_id, config):
+        self.youtube_id = youtube_id
+        self.config = config
+        self.video = False
+
+    def run(self):
+        """check and run if needed"""
+        print(f"{self.youtube_id}: failed to build channel path, try to fix.")
+        video_path_is, video_folder_is = self.get_as_is()
+        if not os.path.exists(video_path_is):
+            print(f"giving up reindex, video in video: {self.video.json_data}")
+            raise ValueError
+
+        _, video_folder_should = self.get_as_should()
+
+        if video_folder_is != video_folder_should:
+            self.process(video_path_is)
+        else:
+            print(f"{self.youtube_id}: skip channel url fixer")
+
+    def get_as_is(self):
+        """get video object as is"""
+        self.video = YoutubeVideo(self.youtube_id)
+        self.video.get_from_es()
+        video_path_is = os.path.join(
+            self.config["application"]["videos"],
+            self.video.json_data["media_url"],
+        )
+        video_folder_is = os.path.split(video_path_is)[0]
+
+        return video_path_is, video_folder_is
+
+    def get_as_should(self):
+        """add fresh metadata from remote"""
+        self.video.get_from_youtube()
+        self.video.add_file_path()
+
+        video_path_should = os.path.join(
+            self.config["application"]["videos"],
+            self.video.json_data["media_url"],
+        )
+        video_folder_should = os.path.split(video_path_should)[0]
+        return video_path_should, video_folder_should
+
+    def process(self, video_path_is):
+        """fix filepath"""
+        print(f"{self.youtube_id}: fixing channel rename.")
+        cache_dir = self.config["application"]["cache_dir"]
+        new_file_path = os.path.join(
+            cache_dir, "download", self.youtube_id + ".mp4"
+        )
+        shutil.move(video_path_is, new_file_path)
+        VideoDownloader().move_to_archive(self.video.json_data)
+        self.video.update_media_url()

tubearchivist/home/src/index/video.py

@@ -9,14 +9,18 @@ import os
 from datetime import datetime
 
 import requests
+from django.conf import settings
 from home.src.es.connect import ElasticWrap
 from home.src.index import channel as ta_channel
+from home.src.index import playlist as ta_playlist
 from home.src.index.generic import YouTubeItem
 from home.src.ta.helper import (
     DurationConverter,
     clean_string,
+    randomizor,
     requests_headers,
 )
+from home.src.ta.ta_redis import RedisArchivist
 from ryd_client import ryd_client
 
 
@@ -63,7 +67,7 @@ class YoutubeSubtitle:
             return False
 
         video_media_url = self.video.json_data["media_url"]
-        media_url = video_media_url.replace(".mp4", f"-{lang}.vtt")
+        media_url = video_media_url.replace(".mp4", f".{lang}.vtt")
         all_formats = all_subtitles.get(lang)
         if not all_formats:
             return False
@@ -99,7 +103,7 @@ class YoutubeSubtitle:
             return False
 
         video_media_url = self.video.json_data["media_url"]
-        media_url = video_media_url.replace(".mp4", f"-{lang}.vtt")
+        media_url = video_media_url.replace(".mp4", f".{lang}.vtt")
         all_formats = all_subtitles.get(lang)
         if not all_formats:
             # no user subtitles found
@@ -148,6 +152,30 @@ class YoutubeSubtitle:
         """send subtitle to es for indexing"""
         _, _ = ElasticWrap("_bulk").post(data=query_str, ndjson=True)
 
+    def delete(self, subtitles=False):
+        """delete subtitles from index and filesystem"""
+        youtube_id = self.video.youtube_id
+        videos_base = self.video.config["application"]["videos"]
+        # delete files
+        if subtitles:
+            files = [i["media_url"] for i in subtitles]
+        else:
+            if not self.video.json_data.get("subtitles"):
+                return
+
+            files = [i["media_url"] for i in self.video.json_data["subtitles"]]
+
+        for file_name in files:
+            file_path = os.path.join(videos_base, file_name)
+            try:
+                os.remove(file_path)
+            except FileNotFoundError:
+                print(f"{youtube_id}: {file_path} failed to delete")
+        # delete from index
+        path = "ta_subtitle/_delete_by_query?refresh=true"
+        data = {"query": {"term": {"youtube_id": {"value": youtube_id}}}}
+        _, _ = ElasticWrap(path).post(data=data)
+
 
 class SubtitleParser:
     """parse subtitle str from youtube"""
@@ -166,6 +194,11 @@ class SubtitleParser:
 
         self.all_cues = []
         for idx, event in enumerate(all_events):
+            if "dDurationMs" not in event or "segs" not in event:
+                # some events won't have a duration or segs
+                print(f"skipping subtitle event without content: {event}")
+                continue
+
             cue = {
                 "start": self._ms_conv(event["tStartMs"]),
                 "end": self._ms_conv(event["tStartMs"] + event["dDurationMs"]),
@@ -187,10 +220,16 @@ class SubtitleParser:
 
             if flatten:
                 # fix overlapping retiming issue
-                last_end = flatten[-1]["tStartMs"] + flatten[-1]["dDurationMs"]
+                last = flatten[-1]
+                if "dDurationMs" not in last or "segs" not in last:
+                    # some events won't have a duration or segs
+                    print(f"skipping subtitle event without content: {event}")
+                    continue
+
+                last_end = last["tStartMs"] + last["dDurationMs"]
                 if event["tStartMs"] < last_end:
-                    joined = flatten[-1]["segs"][0]["utf8"] + "\n" + text
-                    flatten[-1]["segs"][0]["utf8"] = joined
+                    joined = last["segs"][0]["utf8"] + "\n" + text
+                    last["segs"][0]["utf8"] = joined
                     continue
 
             event.update({"segs": [{"utf8": text}]})
@@ -280,6 +319,100 @@ class SubtitleParser:
         return chunk_list
 
 
+class SponsorBlock:
+    """handle sponsor block integration"""
+
+    API = "https://sponsor.ajay.app/api"
+
+    def __init__(self, user_id=False):
+        self.user_id = user_id
+        self.user_agent = f"{settings.TA_UPSTREAM} {settings.TA_VERSION}"
+        self.last_refresh = int(datetime.now().strftime("%s"))
+
+    def get_sb_id(self):
+        """get sponsorblock userid or generate if needed"""
+        if not self.user_id:
+            print("missing request user id")
+            raise ValueError
+
+        key = f"{self.user_id}:id_sponsorblock"
+        sb_id = RedisArchivist().get_message(key)
+        if not sb_id["status"]:
+            sb_id = {"status": randomizor(32)}
+            RedisArchivist().set_message(key, sb_id)
+
+        return sb_id
+
+    def get_timestamps(self, youtube_id):
+        """get timestamps from the API"""
+        url = f"{self.API}/skipSegments?videoID={youtube_id}"
+        headers = {"User-Agent": self.user_agent}
+        print(f"{youtube_id}: get sponsorblock timestamps")
+        response = requests.get(url, headers=headers)
+        if not response.ok:
+            print(f"{youtube_id}: sponsorblock failed: {response.text}")
+            sponsor_dict = {
+                "last_refresh": self.last_refresh,
+                "is_enabled": True,
+                "segments": [],
+            }
+        else:
+            all_segments = response.json()
+            sponsor_dict = self._get_sponsor_dict(all_segments)
+
+        return sponsor_dict
+
+    def _get_sponsor_dict(self, all_segments):
+        """format and process response"""
+        has_unlocked = False
+        cleaned_segments = []
+        for segment in all_segments:
+            if not segment["locked"]:
+                has_unlocked = True
+            del segment["userID"]
+            del segment["description"]
+            cleaned_segments.append(segment)
+
+        sponsor_dict = {
+            "last_refresh": self.last_refresh,
+            "has_unlocked": has_unlocked,
+            "is_enabled": True,
+            "segments": cleaned_segments,
+        }
+        return sponsor_dict
+
+    def post_timestamps(self, youtube_id, start_time, end_time):
+        """post timestamps to api"""
+        user_id = self.get_sb_id().get("status")
+        data = {
+            "videoID": youtube_id,
+            "startTime": start_time,
+            "endTime": end_time,
+            "category": "sponsor",
+            "userID": user_id,
+            "userAgent": self.user_agent,
+        }
+        url = f"{self.API}/skipSegments?videoID={youtube_id}"
+        print(f"post: {data}")
+        print(f"to: {url}")
+
+        return {"success": True}, 200
+
+    def vote_on_segment(self, uuid, vote):
+        """send vote on existing segment"""
+        user_id = self.get_sb_id().get("status")
+        data = {
+            "UUID": uuid,
+            "userID": user_id,
+            "type": vote,
+        }
+        url = f"{self.API}/api/voteOnSponsorTime"
+        print(f"post: {data}")
+        print(f"to: {url}")
+
+        return {"success": True}, 200
+
+
 class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
     """represents a single youtube video"""
 
@@ -287,9 +420,10 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
     index_name = "ta_video"
     yt_base = "https://www.youtube.com/watch?v="
 
-    def __init__(self, youtube_id):
+    def __init__(self, youtube_id, video_overwrites=False):
         super().__init__(youtube_id)
         self.channel_id = False
+        self.video_overwrites = video_overwrites
         self.es_path = f"{self.index_name}/_doc/{youtube_id}"
 
     def build_json(self):
@@ -306,8 +440,25 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
         if self.config["downloads"]["integrate_ryd"]:
             self._get_ryd_stats()
 
+        if self._check_get_sb():
+            self._get_sponsorblock()
+
         return
 
+    def _check_get_sb(self):
+        """check if need to run sponsor block"""
+        integrate = self.config["downloads"]["integrate_sponsorblock"]
+
+        if self.video_overwrites:
+            single_overwrite = self.video_overwrites.get(self.youtube_id)
+            if not single_overwrite:
+                return integrate
+
+            if "integrate_sponsorblock" in single_overwrite:
+                return single_overwrite.get("integrate_sponsorblock")
+
+        return integrate
+
     def _process_youtube_meta(self):
         """extract relevant fields from youtube"""
         # extract
@@ -316,12 +467,15 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
         upload_date_time = datetime.strptime(upload_date, "%Y%m%d")
         published = upload_date_time.strftime("%Y-%m-%d")
         last_refresh = int(datetime.now().strftime("%s"))
+        # base64_blur = ThumbManager().get_base64_blur(self.youtube_id)
+        base64_blur = False
         # build json_data basics
         self.json_data = {
             "title": self.youtube_meta["title"],
             "description": self.youtube_meta["description"],
             "category": self.youtube_meta["categories"],
             "vid_thumb_url": self.youtube_meta["thumbnail"],
+            "vid_thumb_base64": base64_blur,
             "tags": self.youtube_meta["tags"],
             "published": published,
             "vid_last_refresh": last_refresh,
@@ -412,21 +566,43 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
 
     def delete_media_file(self):
         """delete video file, meta data"""
+        print(f"{self.youtube_id}: delete video")
         self.get_from_es()
         video_base = self.app_conf["videos"]
-        to_del = [self.json_data.get("media_url")]
-
-        all_subtitles = self.json_data.get("subtitles")
-        if all_subtitles:
-            to_del = to_del + [i.get("media_url") for i in all_subtitles]
-
-        for media_url in to_del:
-            file_path = os.path.join(video_base, media_url)
+        media_url = self.json_data.get("media_url")
+        file_path = os.path.join(video_base, media_url)
+        try:
             os.remove(file_path)
+        except FileNotFoundError:
+            print(f"{self.youtube_id}: failed {media_url}, continue.")
 
+        self.del_in_playlists()
         self.del_in_es()
         self.delete_subtitles()
 
+    def del_in_playlists(self):
+        """remove downloaded in playlist"""
+        all_playlists = self.json_data.get("playlist")
+        if not all_playlists:
+            return
+
+        for playlist_id in all_playlists:
+            print(f"{playlist_id}: delete video {self.youtube_id}")
+            playlist = ta_playlist.YoutubePlaylist(playlist_id)
+            playlist.get_from_es()
+            entries = playlist.json_data["playlist_entries"]
+            for idx, entry in enumerate(entries):
+                if entry["youtube_id"] == self.youtube_id:
+                    playlist.json_data["playlist_entries"][idx].update(
+                        {"downloaded": False}
+                    )
+            playlist.upload_to_es()
+
+    def delete_subtitles(self, subtitles=False):
+        """delete indexed subtitles"""
+        print(f"{self.youtube_id}: delete subtitles")
+        YoutubeSubtitle(self).delete(subtitles=subtitles)
+
     def _get_ryd_stats(self):
         """get optional stats from returnyoutubedislikeapi.com"""
         try:
@@ -447,6 +623,12 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
 
         return True
 
+    def _get_sponsorblock(self):
+        """get optional sponsorblock timestamps from sponsor.ajay.app"""
+        sponsorblock = SponsorBlock().get_timestamps(self.youtube_id)
+        if sponsorblock:
+            self.json_data["sponsorblock"] = sponsorblock
+
     def check_subtitles(self):
         """optionally add subtitles"""
         handler = YoutubeSubtitle(self)
@@ -455,16 +637,16 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
             self.json_data["subtitles"] = subtitles
             handler.download_subtitles(relevant_subtitles=subtitles)
 
-    def delete_subtitles(self):
-        """delete indexed subtitles"""
-        path = "ta_subtitle/_delete_by_query?refresh=true"
-        data = {"query": {"term": {"youtube_id": {"value": self.youtube_id}}}}
+    def update_media_url(self):
+        """update only media_url in es for reindex channel rename"""
+        data = {"doc": {"media_url": self.json_data["media_url"]}}
+        path = f"{self.index_name}/_update/{self.youtube_id}"
         _, _ = ElasticWrap(path).post(data=data)
 
 
-def index_new_video(youtube_id):
+def index_new_video(youtube_id, video_overwrites=False):
     """combined classes to create new video in index"""
-    video = YoutubeVideo(youtube_id)
+    video = YoutubeVideo(youtube_id, video_overwrites=video_overwrites)
     video.build_json()
     if not video.json_data:
         raise ValueError("failed to get metadata for " + youtube_id)

tubearchivist/home/src/ta/config.py

@@ -83,33 +83,35 @@ class AppConfig:
 
     def update_config(self, form_post):
         """update config values from settings form"""
-        config = self.config
+        updated = []
         for key, value in form_post.items():
-            to_write = value[0]
-            if len(to_write):
-                if to_write == "0":
-                    to_write = False
-                elif to_write == "1":
-                    to_write = True
-                elif to_write.isdigit():
-                    to_write = int(to_write)
+            if not value and not isinstance(value, int):
+                continue
 
-                config_dict, config_value = key.split("_", maxsplit=1)
-                config[config_dict][config_value] = to_write
+            if value in ["0", 0]:
+                to_write = False
+            elif value == "1":
+                to_write = True
+            else:
+                to_write = value
 
-        RedisArchivist().set_message("config", config, expire=False)
+            config_dict, config_value = key.split("_", maxsplit=1)
+            self.config[config_dict][config_value] = to_write
+            updated.append((config_value, to_write))
+
+        RedisArchivist().set_message("config", self.config)
+        return updated
 
     @staticmethod
     def set_user_config(form_post, user_id):
         """set values in redis for user settings"""
         for key, value in form_post.items():
-            to_write = value[0]
-            if len(to_write):
-                if to_write.isdigit():
-                    to_write = int(to_write)
-                message = {"status": to_write}
-                redis_key = f"{user_id}:{key}"
-                RedisArchivist().set_message(redis_key, message, expire=False)
+            if not value:
+                continue
+
+            message = {"status": value}
+            redis_key = f"{user_id}:{key}"
+            RedisArchivist().set_message(redis_key, message)
 
     def get_colors(self):
         """overwrite config if user has set custom values"""
@@ -149,7 +151,7 @@ class AppConfig:
                     needs_update = True
 
         if needs_update:
-            RedisArchivist().set_message("config", redis_config, expire=False)
+            RedisArchivist().set_message("config", redis_config)
 
 
 class ScheduleBuilder:
@@ -163,6 +165,7 @@ class ScheduleBuilder:
         "run_backup": "0 18 0",
     }
     CONFIG = ["check_reindex_days", "run_backup_rotate"]
+    MSG = "message:setting"
 
     def __init__(self):
         self.config = AppConfig().config
@@ -172,64 +175,84 @@ class ScheduleBuilder:
         print("processing form, restart container for changes to take effect")
         redis_config = self.config
         for key, value in form_post.items():
-            to_check = value[0]
-            if key in self.SCHEDULES and to_check:
+            if key in self.SCHEDULES and value:
                 try:
-                    to_write = self.value_builder(key, to_check)
+                    to_write = self.value_builder(key, value)
                 except ValueError:
-                    print(f"failed: {key} {to_check}")
+                    print(f"failed: {key} {value}")
                     mess_dict = {
-                        "status": "message:setting",
+                        "status": self.MSG,
                         "level": "error",
                         "title": "Scheduler update failed.",
                         "message": "Invalid schedule input",
                     }
-                    RedisArchivist().set_message("message:setting", mess_dict)
+                    RedisArchivist().set_message(
+                        self.MSG, mess_dict, expire=True
+                    )
                     return
 
                 redis_config["scheduler"][key] = to_write
-            if key in self.CONFIG and to_check:
-                redis_config["scheduler"][key] = int(to_check)
-        RedisArchivist().set_message("config", redis_config, expire=False)
+            if key in self.CONFIG and value:
+                redis_config["scheduler"][key] = int(value)
+        RedisArchivist().set_message("config", redis_config)
         mess_dict = {
-            "status": "message:setting",
+            "status": self.MSG,
             "level": "info",
             "title": "Scheduler changed.",
             "message": "Please restart container for changes to take effect",
         }
-        RedisArchivist().set_message("message:setting", mess_dict)
+        RedisArchivist().set_message(self.MSG, mess_dict, expire=True)
 
-    def value_builder(self, key, to_check):
+    def value_builder(self, key, value):
         """validate single cron form entry and return cron dict"""
-        print(f"change schedule for {key} to {to_check}")
-        if to_check == "0":
+        print(f"change schedule for {key} to {value}")
+        if value == "0":
             # deactivate this schedule
             return False
-        if re.search(r"[\d]{1,2}\/[\d]{1,2}", to_check):
+        if re.search(r"[\d]{1,2}\/[\d]{1,2}", value):
             # number/number cron format will fail in celery
             print("number/number schedule formatting not supported")
             raise ValueError
 
         keys = ["minute", "hour", "day_of_week"]
-        if to_check == "auto":
+        if value == "auto":
             # set to sensible default
             values = self.SCHEDULES[key].split()
         else:
-            values = to_check.split()
+            values = value.split()
 
         if len(keys) != len(values):
-            print(f"failed to parse {to_check} for {key}")
+            print(f"failed to parse {value} for {key}")
             raise ValueError("invalid input")
 
         to_write = dict(zip(keys, values))
-        try:
-            int(to_write["minute"])
-        except ValueError as error:
-            print("too frequent: only number in minutes are supported")
-            raise ValueError("invalid input") from error
+        self._validate_cron(to_write)
 
         return to_write
 
+    @staticmethod
+    def _validate_cron(to_write):
+        """validate all fields, raise value error for impossible schedule"""
+        all_hours = list(re.split(r"\D+", to_write["hour"]))
+        for hour in all_hours:
+            if hour.isdigit() and int(hour) > 23:
+                print("hour can not be greater than 23")
+                raise ValueError("invalid input")
+
+        all_days = list(re.split(r"\D+", to_write["day_of_week"]))
+        for day in all_days:
+            if day.isdigit() and int(day) > 6:
+                print("day can not be greater than 6")
+                raise ValueError("invalid input")
+
+        if not to_write["minute"].isdigit():
+            print("too frequent: only number in minutes are supported")
+            raise ValueError("invalid input")
+
+        if int(to_write["minute"]) > 59:
+            print("minutes can not be greater than 59")
+            raise ValueError("invalid input")
+
     def build_schedule(self):
         """build schedule dict as expected by app.conf.beat_schedule"""
         schedule_dict = {}

tubearchivist/home/src/ta/helper.py

@@ -8,9 +8,10 @@ import re
 import string
 import subprocess
 import unicodedata
+from datetime import datetime
 from urllib.parse import parse_qs, urlparse
 
-import yt_dlp
+from home.src.download.yt_dlp_base import YtWrap
 
 
 def clean_string(file_name):
@@ -36,6 +37,12 @@ def ignore_filelist(filelist):
     return cleaned
 
 
+def randomizor(length):
+    """generate random alpha numeric string"""
+    pool = string.digits + string.ascii_letters
+    return "".join(random.choice(pool) for i in range(length))
+
+
 def requests_headers():
     """build header with random user agent for requests outside of yt-dlp"""
 
@@ -88,6 +95,16 @@ def requests_headers():
     return {"User-Agent": template}
 
 
+def date_praser(timestamp):
+    """return formatted date string"""
+    if isinstance(timestamp, int):
+        date_obj = datetime.fromtimestamp(timestamp)
+    elif isinstance(timestamp, str):
+        date_obj = datetime.strptime(timestamp, "%Y-%m-%d")
+
+    return datetime.strftime(date_obj, "%d %b, %Y")
+
+
 class UrlListParser:
     """take a multi line string and detect valid youtube ids"""
 
@@ -156,7 +173,7 @@ class UrlListParser:
             id_type = "video"
         elif str_len == 24:
             id_type = "channel"
-        elif str_len in [34, 18]:
+        elif str_len in [34, 18] or id_str in ["LL", "WL"]:
             id_type = "playlist"
         else:
             # unable to parse
@@ -167,14 +184,12 @@ class UrlListParser:
     @staticmethod
     def extract_channel_name(url):
         """find channel id from channel name with yt-dlp help"""
-        obs = {
-            "default_search": "ytsearch",
-            "quiet": True,
+        obs_request = {
             "skip_download": True,
             "extract_flat": True,
             "playlistend": 0,
         }
-        url_info = yt_dlp.YoutubeDL(obs).extract_info(url, download=False)
+        url_info = YtWrap(obs_request).extract(url)
         try:
             channel_id = url_info["channel_id"]
         except KeyError as error:

tubearchivist/home/src/ta/ta_redis.py

@@ -8,7 +8,6 @@ import json
 import os
 
 import redis
-from home.src.ta.helper import ignore_filelist
 
 
 class RedisBase:
@@ -35,10 +34,10 @@ class RedisArchivist(RedisBase):
         "setting",
     ]
 
-    def set_message(self, key, message, expire=True):
+    def set_message(self, key, message, path=".", expire=False):
         """write new message to redis"""
         self.conn.execute_command(
-            "JSON.SET", self.NAME_SPACE + key, ".", json.dumps(message)
+            "JSON.SET", self.NAME_SPACE + key, path, json.dumps(message)
         )
 
         if expire:
@@ -81,6 +80,12 @@ class RedisArchivist(RedisBase):
         redis_lock = self.conn.lock(self.NAME_SPACE + lock_key)
         return redis_lock
 
+    def is_locked(self, lock_key):
+        """check if lock is set"""
+        lock_name = self.NAME_SPACE + lock_key
+        lock_status = bool(self.conn.execute_command("GET", lock_name))
+        return lock_status
+
     def get_progress(self):
         """get a list of all progress messages"""
         all_messages = []
@@ -95,27 +100,6 @@ class RedisArchivist(RedisBase):
 
         return all_messages
 
-    @staticmethod
-    def monitor_cache_dir(cache_dir):
-        """
-        look at download cache dir directly as alternative progress info
-        """
-        dl_cache = os.path.join(cache_dir, "download")
-        all_cache_file = os.listdir(dl_cache)
-        cache_file = ignore_filelist(all_cache_file)
-        if cache_file:
-            filename = cache_file[0][12:].replace("_", " ").split(".")[0]
-            mess_dict = {
-                "status": "message:download",
-                "level": "info",
-                "title": "Downloading: " + filename,
-                "message": "",
-            }
-        else:
-            return False
-
-        return mess_dict
-
 
 class RedisQueue(RedisBase):
     """dynamically interact with the download queue in redis"""

tubearchivist/home/tasks.py

@@ -48,7 +48,7 @@ def update_subscribed():
         "title": "Rescanning channels and playlists.",
         "message": "Looking for new videos.",
     }
-    RedisArchivist().set_message("message:rescan", message)
+    RedisArchivist().set_message("message:rescan", message, expire=True)
 
     have_lock = False
     my_lock = RedisArchivist().get_lock("rescan")
@@ -108,13 +108,14 @@ def download_single(youtube_id):
     try:
         have_lock = my_lock.acquire(blocking=False)
         if have_lock:
+            key = "message:download"
             mess_dict = {
-                "status": "message:download",
+                "status": key,
                 "level": "info",
                 "title": "Download single video",
                 "message": "processing",
             }
-            RedisArchivist().set_message("message:download", mess_dict)
+            RedisArchivist().set_message(key, mess_dict, expire=True)
             VideoDownloader().run_queue()
         else:
             print("Download queue already running.")
@@ -196,7 +197,7 @@ def kill_dl(task_id):
         "title": "Canceling download process",
         "message": "Canceling download queue now.",
     }
-    RedisArchivist().set_message("message:download", mess_dict)
+    RedisArchivist().set_message("message:download", mess_dict, expire=True)
 
 
 @shared_task
@@ -245,13 +246,14 @@ def subscribe_to(url_str):
             channel_id_sub, channel_subscribed=True
         )
         # notify for channels
+        key = "message:subchannel"
         message = {
-            "status": "message:subchannel",
+            "status": key,
             "level": "info",
             "title": "Subscribing to Channels",
             "message": f"Processing {counter} of {len(to_subscribe_list)}",
         }
-        RedisArchivist().set_message("message:subchannel", message=message)
+        RedisArchivist().set_message(key, message=message, expire=True)
         counter = counter + 1
 
 
@@ -260,13 +262,14 @@ def index_channel_playlists(channel_id):
     """add all playlists of channel to index"""
     channel = YoutubeChannel(channel_id)
     # notify
+    key = "message:playlistscan"
     mess_dict = {
-        "status": "message:playlistscan",
+        "status": key,
         "level": "info",
         "title": "Looking for playlists",
         "message": f'Scanning channel "{channel.youtube_id}" in progress',
     }
-    RedisArchivist().set_message("message:playlistscan", mess_dict)
+    RedisArchivist().set_message(key, mess_dict, expire=True)
     channel.index_channel_playlists()
 
 

tubearchivist/home/templates/home/about.html

@@ -7,13 +7,13 @@
     </div>
     <div class="about-section">
         <h2>Useful Links</h2>
-        <p>This project is in active and constant development, take a look at the <a href="https://github.com/bbilly1/tubearchivist#roadmap" target="_blank">roadmap</a> for a overview.</p>
-        <p>For any questions on what a button or a function does, You can find the up-to-date user documentation on <a href="https://github.com/bbilly1/tubearchivist/wiki" target="_blank">Github</a>.</p>
-        <p>All contributions are welcome: Open an <a href="https://github.com/bbilly1/tubearchivist/issues" target="_blank">issue</a> for any bugs and errors, start a <a href="https://github.com/bbilly1/tubearchivist/discussions" target="_blank">discussion</a> for anything that will require a more indepth look. The <a href="https://github.com/bbilly1/tubearchivist/blob/master/CONTRIBUTING.md" target="_blank">contributing</a> page is a good place to get started.</p>
+        <p>This project is in active and constant development, take a look at the <a href="https://github.com/tubearchivist/tubearchivist#roadmap" target="_blank">roadmap</a> for a overview.</p>
+        <p>For any questions on what a button or a function does, You can find the up-to-date user documentation on <a href="https://github.com/tubearchivist/tubearchivist/wiki" target="_blank">Github</a>.</p>
+        <p>All contributions are welcome: Open an <a href="https://github.com/tubearchivist/tubearchivist/issues" target="_blank">issue</a> for any bugs and errors, join us on <a href="https://discord.gg/AFwz8nE7BK" target="_blank">Discord</a> to discuss details. The <a href="https://github.com/tubearchivist/tubearchivist/blob/master/CONTRIBUTING.md" target="_blank">contributing</a> page is a good place to get started.</p>
     </div>
     <div class="about-section">
         <h2>Donate</h2>
-        <p>Here are <a href="https://github.com/bbilly1/tubearchivist#donate" target="_blank">some links</a>, if you want to buy the developer a coffee. Thank you for your support!</p>
+        <p>Here are <a href="https://github.com/tubearchivist/tubearchivist#donate" target="_blank">some links</a>, if you want to buy the developer a coffee. Thank you for your support!</p>
     </div>
 </div>
 {% endblock content %}

tubearchivist/home/templates/home/base.html

@@ -132,7 +132,7 @@
     </div>
     <div class="footer">
         <div class="boxed-content">
-            <span>© 2021 - <script type="text/javascript">document.write(new Date().getFullYear());</script> TubeArchivist v0.1.3 </span><span><a href="{% url 'about' %}">About</a> | <a href="https://github.com/bbilly1/tubearchivist" target="_blank">GitHub</a> | <a href="https://hub.docker.com/r/bbilly1/tubearchivist" target="_blank">Docker Hub</a> | <a href="https://discord.gg/AFwz8nE7BK" target="_blank">Discord</a> | <a href="https://www.reddit.com/r/TubeArchivist/">Reddit</a></span>
+            <span>© 2021 - <script type="text/javascript">document.write(new Date().getFullYear());</script> TubeArchivist {{ version }} </span><span><a href="{% url 'about' %}">About</a> | <a href="https://github.com/tubearchivist/tubearchivist" target="_blank">GitHub</a> | <a href="https://hub.docker.com/r/bbilly1/tubearchivist" target="_blank">Docker Hub</a> | <a href="https://discord.gg/AFwz8nE7BK" target="_blank">Discord</a> | <a href="https://www.reddit.com/r/TubeArchivist/">Reddit</a></span>
         </div>
     </div>
 </body>

tubearchivist/home/templates/home/channel_id.html

@@ -89,7 +89,17 @@
                         {% endif %}</span></p>
                     {{ channel_overwrite_form.index_playlists }}<br>
                 </div>
-                <div class="overwrite-form-item"></div>
+                <div class="overwrite-form-item">
+                    <p>Enable <a href="https://sponsor.ajay.app/" target="_blank">SponsorBlock</a>: <span class="settings-current">
+                        {% if channel_info.channel_overwrites.integrate_sponsorblock %}
+                            {{ channel_info.channel_overwrites.integrate_sponsorblock }}
+                        {% elif channel_info.channel_overwrites.integrate_sponsorblock == False %}
+                            Disabled
+                        {% else %}
+                            False
+                        {% endif %}</span></p>
+                    {{ channel_overwrite_form.integrate_sponsorblock }}<br>
+                </div>
                 <button type="submit">Save Channel Overwrites</button>
             </form>
         </div>
@@ -102,6 +112,8 @@
             </div>
         </div>
     {% endif %}
+</div>
+<div class="boxed-content {% if view_style == "grid" %}boxed-{{ grid_items }}{% endif %}">
     <div class="view-controls">
         <div class="toggle">
             <span>Hide watched videos:</span>
@@ -131,14 +143,24 @@
         </div>
         <div class="view-icons">
             <img src="{% static 'img/icon-sort.svg' %}" alt="sort-icon" onclick="showForm()" id="animate-icon">
+            {% if view_style == "grid" %}
+            <div class="grid-count">
+                {% if grid_items < 7 %}
+                    <img src="{% static 'img/icon-add.svg' %}" onclick="changeGridItems(this)" data-value="{{ grid_items|add:"1"}}" alt="grid plus row">
+                {% endif %}
+                {% if grid_items > 3 %}
+                    <img src="{% static 'img/icon-substract.svg' %}" onclick="changeGridItems(this)" data-value="{{ grid_items|add:"-1"}}" alt="grid minus row">
+                {% endif %}
+            </div>
+            {% endif %}
             <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="grid" alt="grid view">
             <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="list" alt="list view">
         </div>
     </div>
 </div>
 <div id="player" class="player-wrapper"></div>
-<div class="boxed-content">
-    <div class="video-list {{ view_style }}">
+<div class="boxed-content {% if view_style == "grid" %}boxed-{{ grid_items }}{% endif %}">
+    <div class="video-list {{ view_style }} {% if view_style == "grid" %}grid-{{ grid_items }}{% endif %}">
         {% if results %}
             {% for video in results %}
                 <div class="video-item {{ view_style }}">

tubearchivist/home/templates/home/downloads.html

@@ -45,15 +45,6 @@
             <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="downloads" data-value="list" alt="list view">
         </div>
     </div>
-    <div class="title-split">
-        {% if show_ignored_only %}
-            <h2>Ignored from download</h2>
-            <button onclick="deleteQueue(this)" data-id="ignore" title="Delete all previously ignored videos from the queue">Delete all ignored</button>
-        {% else %}
-            <h2>Download queue</h2>
-            <button onclick="deleteQueue(this)" data-id="pending" title="Delete all pending videos from the queue">Delete all queued</button>
-        {% endif %}
-    </div>
     <h3>Total videos: {{ max_hits }}{% if max_hits == 10000 %}+{% endif %}</h3>
     <div class="dl-list {{ view_style }}">
         {% if results %}

tubearchivist/home/templates/home/home.html

@@ -1,12 +1,12 @@
 {% extends "home/base.html" %}
 {% block content %}
 {% load static %}
-<div class="boxed-content">
+<div class="boxed-content {% if view_style == "grid" %}boxed-{{ grid_items }}{% endif %}">
     {% if continue_vids %}
         <div class="title-bar">
             <h1>Continue Watching</h1>
         </div>
-        <div class="video-list {{ view_style }}">
+        <div class="video-list {{ view_style }} {% if view_style == "grid" %}grid-{{ grid_items }}{% endif %}">
             {% for video in continue_vids %}
                 <div class="video-item {{ view_style }}">
                     <a href="#player" data-id="{{ video.source.youtube_id }}" onclick="createPlayer(this)">
@@ -74,14 +74,24 @@
         </div>
         <div class="view-icons">
             <img src="{% static 'img/icon-sort.svg' %}" alt="sort-icon" onclick="showForm()" id="animate-icon">
+            {% if view_style == "grid" %}
+                <div class="grid-count">
+                    {% if grid_items < 7 %}
+                        <img src="{% static 'img/icon-add.svg' %}" onclick="changeGridItems(this)" data-value="{{ grid_items|add:"1"}}" alt="grid plus row">
+                    {% endif %}
+                    {% if grid_items > 3 %}
+                        <img src="{% static 'img/icon-substract.svg' %}" onclick="changeGridItems(this)" data-value="{{ grid_items|add:"-1"}}" alt="grid minus row">
+                    {% endif %}
+                </div>
+            {% endif %}
             <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="grid" alt="grid view">
             <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="list" alt="list view">
         </div>
     </div>
 </div>
 <div id="player" class="player-wrapper"></div>
-<div class="boxed-content">
-    <div class="video-list {{ view_style }}">
+<div class="boxed-content {% if view_style == "grid" %}boxed-{{ grid_items }}{% endif %}">
+    <div class="video-list {{ view_style }} {% if view_style == "grid" %}grid-{{ grid_items }}{% endif %}">
         {% if results %}
             {% for video in results %}
                 <div class="video-item {{ view_style }}">

tubearchivist/home/templates/home/login.html

@@ -45,7 +45,7 @@
             <input type="hidden" name="next" value="{{ request.GET.next }}" />
             <button type="submit">Login</button>
         </form>
-        <p class="login-links"><span><a href="https://github.com/bbilly1/tubearchivist" target="_blank">Github</a></span> <span><a href="https://github.com/bbilly1/tubearchivist#donate" target="_blank">Donate</a></span></p>
+        <p class="login-links"><span><a href="https://github.com/tubearchivist/tubearchivist" target="_blank">Github</a></span> <span><a href="https://github.com/tubearchivist/tubearchivist#donate" target="_blank">Donate</a></span></p>
     </div>
     <div class="footer-colors">
         <div class="col-1"></div>

tubearchivist/home/templates/home/playlist_id.html

@@ -63,6 +63,8 @@
             </div>
         </div>
     {% endif %}
+</div>
+<div class="boxed-content {% if view_style == "grid" %}boxed-{{ grid_items }}{% endif %}">
     <div class="view-controls">
         <div class="toggle">
             <span>Hide watched videos:</span>
@@ -76,14 +78,24 @@
             </div>
         </div>
         <div class="view-icons">
+            {% if view_style == "grid" %}
+            <div class="grid-count">
+                {% if grid_items < 7 %}
+                    <img src="{% static 'img/icon-add.svg' %}" onclick="changeGridItems(this)" data-value="{{ grid_items|add:"1"}}" alt="grid plus row">
+                {% endif %}
+                {% if grid_items > 3 %}
+                    <img src="{% static 'img/icon-substract.svg' %}" onclick="changeGridItems(this)" data-value="{{ grid_items|add:"-1"}}" alt="grid minus row">
+                {% endif %}
+            </div>
+            {% endif %}
             <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="grid" alt="grid view">
             <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="list" alt="list view">
         </div>
     </div>
 </div>
 <div id="player" class="player-wrapper"></div>
-<div class="boxed-content">
-    <div class="video-list {{ view_style }}">
+<div class="boxed-content {% if view_style == "grid" %}boxed-{{ grid_items }}{% endif %}">
+    <div class="video-list {{ view_style }} {% if view_style == "grid" %}grid-{{ grid_items }}{% endif %}">
         {% if results %}
             {% for video in results %}
                 <div class="video-item {{ view_style }}">

tubearchivist/home/templates/home/search.html

@@ -1,16 +1,16 @@
 {% extends "home/base.html" %}
 {% block content %}
 <div id="player" class="player-wrapper"></div>
-<div class="boxed-content">
+<div class="boxed-content {% if all_styles.home == "grid" %}boxed-{{ grid_items }}{% endif %}">
     <div class="title-bar">
-        <h1>Search</h1>
+        <h1>Search your Archive</h1>
     </div>
     <div class="multi-search-box">
         {{ search_form }}
     </div>
     <div class="multi-search-result">
         <h2>Video Results</h2>
-        <div id="video-results" class="video-list {{ all_styles.home }}">
+        <div id="video-results" class="video-list {{ all_styles.home }} {% if all_styles.home == "grid" %}grid-{{ grid_items }}{% endif %}">
             <p>No videos found.</p>
         </div>
     </div>

tubearchivist/home/templates/home/settings.html

@@ -48,7 +48,7 @@
             </div>
             <div class="settings-item">
                 <p>Current download speed limit in KB/s: <span class="settings-current">{{ config.downloads.limit_speed }}</span></p>
-                <i>Limit download speed. 0 (zero) to deactivate, e.g. 1000 (1MB/s). Speeds are in KB/s.</i><br>
+                <i>Limit download speed. 0 (zero) to deactivate, e.g. 1000 (1MB/s). Speeds are in KB/s. Setting takes effect on new download jobs or application restart.</i><br>
                 {{ app_form.downloads_limit_speed }}
             </div>
             <div class="settings-item">
@@ -114,6 +114,20 @@
                 {{ app_form.downloads_subtitle_index }}
             </div>
         </div>
+        <div class="settings-group">
+            <h2 id="format">Cookie</h2>
+            <div class="settings-item">
+                <p>Import YouTube cookie: <span class="settings-current">{{ config.downloads.cookie_import }}</span><br></p>
+                <p>For automatic cookie import use <b>Tube Archivist Companion</b> <a href="https://github.com/tubearchivist/browser-extension" target="_blank">browser extension</a>.</p> 
+                <i>For manual cookie import, place your cookie file named <span class="settings-current">cookies.google.txt</span> in <span class="settings-current">cache/import</span> before enabling. Instructions in the <a href="https://github.com/tubearchivist/tubearchivist/wiki/Settings" target="_blank">Wiki.</a></i><br>
+                {{ app_form.downloads_cookie_import }}<br>
+                {% if config.downloads.cookie_import %}
+                    <div id="cookieMessage">
+                        <button onclick="handleCookieValidate()" type="button" id="cookieButton">Validate Cookie File</button>
+                    </div>
+                {% endif %}
+            </div>
+        </div>
         <div class="settings-group">
             <h2 id="integrations">Integrations</h2>
             <div class="settings-item">
@@ -124,10 +138,15 @@
                 </div>
             </div>
             <div class="settings-item">
-                <p>Integrate with <a href="https://returnyoutubedislike.com/">returnyoutubedislike.com</a> to get dislikes and average ratings back: <span class="settings-current">{{ config.downloads.integrate_ryd }}</span></p>
+                <p>Integrate with <a href="https://returnyoutubedislike.com/" target="_blank">returnyoutubedislike.com</a> to get dislikes and average ratings back: <span class="settings-current">{{ config.downloads.integrate_ryd }}</span></p>
                 <i>Before activating that, make sure you have a scraping sleep interval of at least 3 secs set to avoid ratelimiting issues.</i><br>
                 {{ app_form.downloads_integrate_ryd }}
             </div>
+            <div class="settings-item">
+                <p>Integrate with <a href="https://sponsor.ajay.app/" target="_blank">SponsorBlock</a> to get sponsored timestamps: <span class="settings-current">{{ config.downloads.integrate_sponsorblock }}</span></p>
+                <i>Before activating that, make sure you have a scraping sleep interval of at least 3 secs set to avoid ratelimiting issues.</i><br>
+                {{ app_form.downloads_integrate_sponsorblock }}
+            </div>
             <div class="settings-item">
                 <p>Current Cast integration: <span class="settings-current">{{ config.application.enable_cast }}</span></p>
                 <i>Enabling Cast will load an additional JS library from Google. HTTPS and a supported browser are required for this integration.</i><br>
@@ -251,9 +270,15 @@
     <div class="title-bar">
         <h1>Actions</h1>
     </div>
+    <div class="settings-group">
+        <h2>Delete download queue</h2>
+        <p>Delete your pending or previously ignored videos from your download queue.<p>
+        <button onclick="deleteQueue(this)" id="ignore-button" data-id="ignore" title="Delete all previously ignored videos from the queue">Delete all ignored</button>
+        <button onclick="deleteQueue(this)" id="pending-button" data-id="pending" title="Delete all pending videos from the queue">Delete all queued</button>
+    </div>
     <div class="settings-group">
         <h2>Manual media files import.</h2>
-        <p>Add files to the <span class="settings-current">cache/import</span> folder. Make sure to follow the instructions in the Github <a href="https://github.com/bbilly1/tubearchivist/wiki/Settings" target="_blank">Wiki</a>.</p>
+        <p>Add files to the <span class="settings-current">cache/import</span> folder. Make sure to follow the instructions in the Github <a href="https://github.com/tubearchivist/tubearchivist/wiki/Settings" target="_blank">Wiki</a>.</p>
         <div id="manual-import">
             <button onclick="manualImport()">Start import</button>
         </div>
@@ -298,7 +323,7 @@
     <div class="settings-group">
         <h2>Rescan filesystem</h2>
         <p><span class="danger-zone">Danger Zone</span>: This will delete the metadata of deleted videos from the filesystem.</p>
-        <p>Rescan your media folder looking for missing videos and clean up index. More infos on the Github <a href="https://github.com/bbilly1/tubearchivist/wiki/Settings" target="_blank">Wiki</a>.</p>
+        <p>Rescan your media folder looking for missing videos and clean up index. More infos on the Github <a href="https://github.com/tubearchivist/tubearchivist/wiki/Settings" target="_blank">Wiki</a>.</p>
         <div id="fs-rescan">
             <button onclick="fsRescan()">Rescan filesystem</button>
         </div>

tubearchivist/home/templates/home/video.html

@@ -3,6 +3,16 @@
 {% load static %}
 {% load humanize %}
 <div class="video-main"></div>
+<div class="notifications" id="notifications"></div>
+<div class="sponsorblock" id="sponsorblock">
+    {% if video.sponsorblock.is_enabled %}
+        {% if video.sponsorblock.segments|length == 0 %}
+            <h4>This video doesn't have any sponsor segments added. To add a segment go to <u><a href="https://www.youtube.com/watch?v={{ video.youtube_id }}">this video on YouTube</a></u> and add a segment using the <u><a href="https://sponsor.ajay.app/">SponsorBlock</a></u> extension.</h4>
+        {% elif video.sponsorblock.has_unlocked %}
+            <h4>This video has unlocked sponsor segments. Go to <u><a href="https://www.youtube.com/watch?v={{ video.youtube_id }}">this video on YouTube</a></u> and vote on the segments using the <u><a href="https://sponsor.ajay.app/">SponsorBlock</a></u> extension.</h4>
+        {% endif %}
+    {% endif %}
+</div>
 <div class="boxed-content">
     <div class="title-bar">
         {% if cast %}
@@ -114,6 +124,7 @@
 </div>
 <script>
     var videoData = getVideoData('{{ video.youtube_id }}');
+    sponsorBlock = videoData.data.sponsorblock;
     var videoProgress = getVideoProgress('{{ video.youtube_id }}').position;
     window.onload = insertVideoTag(videoData, videoProgress);
 </script>

tubearchivist/home/views.py

@@ -8,12 +8,13 @@ import json
 import urllib.parse
 from time import sleep
 
-from django import forms
+from django.conf import settings
 from django.contrib.auth import login
 from django.contrib.auth.forms import AuthenticationForm
 from django.http import JsonResponse
 from django.shortcuts import redirect, render
 from django.views import View
+from home.src.download.yt_dlp_base import CookieHandler
 from home.src.es.index_setup import get_available_backups
 from home.src.frontend.api_calls import PostData
 from home.src.frontend.forms import (
@@ -76,6 +77,15 @@ class ArchivistViewConfig(View):
 
         return view_style
 
+    def _get_grid_items(self):
+        """return items per row to show in grid view"""
+        grid_key = f"{self.user_id}:grid_items"
+        grid_items = self.user_conf.get_message(grid_key)["status"]
+        if not grid_items:
+            grid_items = self.default_conf["default_view"]["grid_items"]
+
+        return grid_items
+
     def get_all_view_styles(self):
         """get dict of all view stiles for search form"""
         all_keys = ["channel", "playlist", "home"]
@@ -111,7 +121,7 @@ class ArchivistViewConfig(View):
         """build default context for every view"""
         self.user_id = user_id
         self.user_conf = RedisArchivist()
-        self.default_conf = AppConfig().config
+        self.default_conf = AppConfig(self.user_id).config
 
         self.context = {
             "colors": self.default_conf["application"]["colors"],
@@ -119,9 +129,11 @@ class ArchivistViewConfig(View):
             "sort_by": self._get_sort_by(),
             "sort_order": self._get_sort_order(),
             "view_style": self._get_view_style(),
+            "grid_items": self._get_grid_items(),
             "hide_watched": self._get_hide_watched(),
             "show_ignored_only": self._get_show_ignore_only(),
             "show_subed_only": self._get_show_subed_only(),
+            "version": settings.TA_VERSION,
         }
 
 
@@ -329,8 +341,11 @@ class AboutView(View):
     @staticmethod
     def get(request):
         """handle http get"""
-        colors = AppConfig(request.user.id).colors
-        context = {"title": "About", "colors": colors}
+        context = {
+            "title": "About",
+            "colors": AppConfig(request.user.id).colors,
+            "version": settings.TA_VERSION,
+        }
         return render(request, "home/about.html", context)
 
 
@@ -379,14 +394,15 @@ class DownloadView(ArchivistResultsView):
                 youtube_ids = UrlListParser(url_str).process_list()
             except ValueError:
                 # failed to process
+                key = "message:add"
                 print(f"failed to parse: {url_str}")
                 mess_dict = {
-                    "status": "message:add",
+                    "status": key,
                     "level": "error",
                     "title": "Failed to extract links.",
                     "message": "Not a video, channel or playlist ID or URL",
                 }
-                RedisArchivist().set_message("message:add", mess_dict)
+                RedisArchivist().set_message(key, mess_dict, expire=True)
                 return redirect("downloads")
 
             print(youtube_ids)
@@ -497,13 +513,14 @@ class ChannelView(ArchivistResultsView):
         """handle http post requests"""
         subscribe_form = SubscribeToChannelForm(data=request.POST)
         if subscribe_form.is_valid():
+            key = "message:subchannel"
             message = {
-                "status": "message:subchannel",
+                "status": key,
                 "level": "info",
                 "title": "Subscribing to Channels",
                 "message": "Parsing form data",
             }
-            RedisArchivist().set_message("message:subchannel", message=message)
+            RedisArchivist().set_message(key, message=message, expire=True)
             url_str = request.POST.get("subscribe")
             print(url_str)
             subscribe_to.delay(url_str)
@@ -644,15 +661,14 @@ class PlaylistView(ArchivistResultsView):
         if subscribe_form.is_valid():
             url_str = request.POST.get("subscribe")
             print(url_str)
+            key = "message:subplaylist"
             message = {
-                "status": "message:subplaylist",
+                "status": key,
                 "level": "info",
                 "title": "Subscribing to Playlists",
                 "message": "Parsing form data",
             }
-            RedisArchivist().set_message(
-                "message:subplaylist", message=message
-            )
+            RedisArchivist().set_message(key, message=message, expire=True)
             subscribe_to.delay(url_str)
 
         sleep(1)
@@ -666,7 +682,7 @@ class VideoView(View):
 
     def get(self, request, video_id):
         """get single video"""
-        colors, cast = self.read_config(user_id=request.user.id)
+        config_handler = AppConfig(request.user.id)
         path = f"ta_video/_doc/{video_id}"
         look_up = SearchHandler(path, config=False)
         video_hit = look_up.get_data()
@@ -688,8 +704,10 @@ class VideoView(View):
             "video": video_data,
             "playlist_nav": playlist_nav,
             "title": video_title,
-            "colors": colors,
-            "cast": cast,
+            "colors": config_handler.colors,
+            "cast": config_handler.config["application"]["enable_cast"],
+            "version": settings.TA_VERSION,
+            "config": config_handler.config,
         }
         return render(request, "home/video.html", context)
 
@@ -706,14 +724,6 @@ class VideoView(View):
 
         return all_navs
 
-    @staticmethod
-    def read_config(user_id):
-        """read config file"""
-        config_handler = AppConfig(user_id)
-        cast = config_handler.config["application"]["enable_cast"]
-        colors = config_handler.colors
-        return colors, cast
-
     @staticmethod
     def star_creator(rating):
         """convert rating float to stars"""
@@ -746,7 +756,10 @@ class SearchView(ArchivistResultsView):
         all_styles = self.get_all_view_styles()
         self.context.update({"all_styles": all_styles})
         self.context.update(
-            {"search_form": MultiSearchForm(initial=all_styles)}
+            {
+                "search_form": MultiSearchForm(initial=all_styles),
+                "version": settings.TA_VERSION,
+            }
         )
 
         return render(request, "home/search.html", self.context)
@@ -778,6 +791,7 @@ class SettingsView(View):
             "user_form": user_form,
             "app_form": app_form,
             "scheduler_form": scheduler_form,
+            "version": settings.TA_VERSION,
         }
 
         return render(request, "home/settings.html", context)
@@ -789,30 +803,62 @@ class SettingsView(View):
         token = Token.objects.get_or_create(user=request.user)[0]
         return token
 
-    @staticmethod
-    def post(request):
+    def post(self, request):
         """handle form post to update settings"""
+        user_form = UserSettingsForm(request.POST)
+        config_handler = AppConfig()
+        if user_form.is_valid():
+            user_form_post = user_form.cleaned_data
+            if any(user_form_post.values()):
+                config_handler.set_user_config(user_form_post, request.user.id)
 
-        form_response = forms.Form(request.POST)
-        if form_response.is_valid():
-            form_post = dict(request.POST)
-            print(form_post)
-            del form_post["csrfmiddlewaretoken"]
-            config_handler = AppConfig()
-            if "application-settings" in form_post:
-                del form_post["application-settings"]
-                config_handler.update_config(form_post)
-            elif "user-settings" in form_post:
-                del form_post["user-settings"]
-                config_handler.set_user_config(form_post, request.user.id)
-            elif "scheduler-settings" in form_post:
-                del form_post["scheduler-settings"]
-                print(form_post)
-                ScheduleBuilder().update_schedule_conf(form_post)
+        app_form = ApplicationSettingsForm(request.POST)
+        if app_form.is_valid():
+            app_form_post = app_form.cleaned_data
+            if app_form_post:
+                print(app_form_post)
+                updated = config_handler.update_config(app_form_post)
+                self.post_process_updated(updated, config_handler.config)
+
+        scheduler_form = SchedulerSettingsForm(request.POST)
+        if scheduler_form.is_valid():
+            scheduler_form_post = scheduler_form.cleaned_data
+            if any(scheduler_form_post.values()):
+                print(scheduler_form_post)
+                ScheduleBuilder().update_schedule_conf(scheduler_form_post)
 
         sleep(1)
         return redirect("settings", permanent=True)
 
+    def post_process_updated(self, updated, config):
+        """apply changes for config"""
+        if not updated:
+            return
+
+        for config_value, updated_value in updated:
+            if config_value == "cookie_import":
+                self.process_cookie(config, updated_value)
+
+    @staticmethod
+    def process_cookie(config, updated_value):
+        """import and validate cookie"""
+        handler = CookieHandler(config)
+        if updated_value:
+            handler.import_cookie()
+            valid = handler.validate()
+            if not valid:
+                handler.revoke()
+                key = "message:setting"
+                message = {
+                    "status": key,
+                    "level": "error",
+                    "title": "Cookie import failed",
+                    "message": "",
+                }
+                RedisArchivist().set_message(key, message=message, expire=True)
+        else:
+            handler.revoke()
+
 
 def progress(request):
     # pylint: disable=unused-argument

tubearchivist/requirements.txt

@@ -1,12 +1,12 @@
-beautifulsoup4==4.10.0
-celery==5.2.3
-Django==4.0.3
-django-cors-headers==3.11.0
+beautifulsoup4==4.11.1
+celery==5.2.7
+Django==4.0.5
+django-cors-headers==3.13.0
 djangorestframework==3.13.1
-Pillow==9.0.1
-redis==4.2.0
-requests==2.27.1
+Pillow==9.2.0
+redis==4.3.4
+requests==2.28.1
 ryd-client==0.0.3
 uWSGI==2.0.20
-whitenoise==6.0.0
-yt_dlp==2022.3.8.2
+whitenoise==6.2.0
+yt_dlp==2022.6.29

tubearchivist/static/css/style.css

@@ -62,6 +62,13 @@ h3 {
     color: var(--accent-font-light);
 }
 
+h4 {
+    font-size: 0.7em;
+    margin-bottom: 7px;
+    font-family: Sen-Regular, sans-serif;
+    color: var(--accent-font-light);
+}
+
 p, i, li {
     font-family: Sen-Regular, sans-serif;
     margin-bottom: 10px;
@@ -126,6 +133,18 @@ button:hover {
     margin: 0 auto;
 }
 
+.boxed-content.boxed-4 {
+    max-width: 1200px;
+    width: 80%;
+}
+
+.boxed-content.boxed-5,
+.boxed-content.boxed-6,
+.boxed-content.boxed-7 {
+    max-width: unset;
+    width: 85%;
+}
+
 .round-img img {
     border-radius: 50%;
 }
@@ -321,7 +340,8 @@ button:hover {
     margin: 15px 0;
 }
 
-.view-icons {
+.view-icons,
+.grid-count {
     display: flex;
     justify-content: end;
 }
@@ -333,6 +353,13 @@ button:hover {
     filter: var(--img-filter);
 }
 
+.grid-count img {
+    width: 15px;
+    margin: 5px;
+    cursor: pointer;
+    filter: var(--img-filter);
+}
+
 #hidden-form {
     display: none;
 }
@@ -355,6 +382,18 @@ button:hover {
     height: 100vh;
 }
 
+.notifications {
+    text-align: center;
+    width: 80%;
+    margin: auto;
+}
+
+.sponsorblock {
+    text-align: center;
+    width: 80%;
+    margin: auto;
+}
+
 .video-player video,
 .video-main video {
     max-height: 80vh;
@@ -376,10 +415,26 @@ button:hover {
     margin-top: 1rem;
 }
 
-.video-list.grid {
+.video-list.grid.grid-3 {
     grid-template-columns: 1fr 1fr 1fr;
 }
 
+.video-list.grid.grid-4 {
+    grid-template-columns: 1fr 1fr 1fr 1fr;
+}
+
+.video-list.grid.grid-5 {
+    grid-template-columns: 1fr 1fr 1fr 1fr 1fr;
+}
+
+.video-list.grid.grid-6 {
+    grid-template-columns: 1fr 1fr 1fr 1fr 1fr 1fr;
+}
+
+.video-list.grid.grid-7 {
+    grid-template-columns: 1fr 1fr 1fr 1fr 1fr 1fr 1fr;
+}
+
 .video-list.list {
     grid-template-columns: unset;
 }
@@ -1031,11 +1086,19 @@ button:hover {
 }
 
 /* tablet */
-@media screen and (max-width: 1000px) {
-    .boxed-content {
+@media screen and (max-width: 1000px), screen and (max-height: 850px) {
+    .boxed-content,
+    .boxed-content.boxed-4,
+    .boxed-content.boxed-5,
+    .boxed-content.boxed-6,
+    .boxed-content.boxed-7 {
         width: 90%;
     }
-    .video-list.grid,
+    .video-list.grid.grid-3,
+    .video-list.grid.grid-4,
+    .video-list.grid.grid-5,
+    .video-list.grid.grid-6,
+    .video-list.grid.grid-7,
     .dl-list.grid,
     .channel-list.grid,
     .playlist-list.grid {
@@ -1065,6 +1128,9 @@ button:hover {
         position: unset;
         transform: unset;
     }
+    .grid-count {
+        display: none;
+    }
     .video-player {
         height: unset;
         padding: 20px 0
@@ -1079,7 +1145,11 @@ button:hover {
     * {
         word-wrap: anywhere;
     }
-    .video-list.grid,
+    .video-list.grid.grid-3,
+    .video-list.grid.grid-4,
+    .video-list.grid.grid-5,
+    .video-list.grid.grid-6,
+    .video-list.grid.grid-7,
     .dl-list.grid,
     .channel-list.grid,
     .video-item.list,

tubearchivist/static/img/icon-substract.svg

@@ -0,0 +1,75 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   width="474.99609"
+   height="78.428696"
+   viewBox="0 0 125.67634 20.750926"
+   version="1.1"
+   id="svg1303"
+   inkscape:version="0.92.4 (5da689c313, 2019-01-14)"
+   sodipodi:docname="icon-add.svg"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:svg="http://www.w3.org/2000/svg"
+   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+   xmlns:cc="http://creativecommons.org/ns#"
+   xmlns:dc="http://purl.org/dc/elements/1.1/">
+  <defs
+     id="defs1297" />
+  <sodipodi:namedview
+     id="base"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageopacity="0.0"
+     inkscape:pageshadow="2"
+     inkscape:zoom="0.85859018"
+     inkscape:cx="-97.380081"
+     inkscape:cy="261.09215"
+     inkscape:document-units="mm"
+     inkscape:current-layer="layer1"
+     showgrid="false"
+     units="px"
+     inkscape:window-width="1920"
+     inkscape:window-height="1017"
+     inkscape:window-x="-8"
+     inkscape:window-y="-8"
+     inkscape:window-maximized="1"
+     showguides="true"
+     inkscape:guide-bbox="true"
+     inkscape:showpageshadow="2"
+     inkscape:pagecheckerboard="0"
+     inkscape:deskcolor="#d1d1d1">
+    <sodipodi:guide
+       position="-225.18364,87.524084"
+       orientation="1,0"
+       id="guide1072"
+       inkscape:locked="false" />
+  </sodipodi:namedview>
+  <metadata
+     id="metadata1300">
+    <rdf:RDF>
+      <cc:Work
+         rdf:about="">
+        <dc:format>image/svg+xml</dc:format>
+        <dc:type
+           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+        <dc:title />
+      </cc:Work>
+    </rdf:RDF>
+  </metadata>
+  <g
+     inkscape:label="Ebene 1"
+     inkscape:groupmode="layer"
+     id="layer1"
+     transform="translate(-3.3077777,-220.4781)">
+    <path
+       style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;paint-order:markers fill stroke"
+       d="m 12.291019,220.47796 c -4.9648228,-0.009 -8.9691711,3.98046 -8.9784054,8.94536 l -0.00482,2.62846 c -0.00925,4.9649 3.9804459,8.96933 8.9452674,8.97832 107.703889,0.29274 22.266017,0.0414 107.747629,0.19881 4.96484,0.009 8.96917,-3.98046 8.9784,-8.94534 l 0.005,-2.62847 c 0.009,-4.96489 -3.98037,-8.96923 -8.94525,-8.97831 -107.850822,-0.0255 -20.413204,-0.038 -107.747821,-0.19883 z"
+       id="rect1073"
+       inkscape:connector-curvature="0"
+       sodipodi:nodetypes="ccccccccc" />
+  </g>
+</svg>

tubearchivist/static/script.js

@@ -121,6 +121,16 @@ function changeView(image) {
     }, 500);
 }
 
+function changeGridItems(image) {
+    var operator = image.getAttribute("data-value");
+    var payload = JSON.stringify({'change_grid': operator});
+    sendPost(payload);
+    setTimeout(function(){
+        location.reload();
+        return false;
+    }, 500);
+}
+
 function toggleCheckbox(checkbox) {
     // pass checkbox id as key and checkbox.checked as value
     var toggleId = checkbox.id;
@@ -193,10 +203,10 @@ function deleteQueue(button) {
     var to_delete = button.getAttribute('data-id');
     var payload = JSON.stringify({'deleteQueue': to_delete});
     sendPost(payload);
-    setTimeout(function(){
-        location.reload();
-        return false;
-    }, 1000);
+    // clear button
+    var message = document.createElement('p');
+    message.innerText = 'deleting download queue: ' + to_delete;
+    document.getElementById(button.id).replaceWith(message);
 }
 
 function stopQueue() {
@@ -327,9 +337,33 @@ function cancelDelete() {
 }
 
 // player
+var sponsorBlock = [];
 function createPlayer(button) {
     var videoId = button.getAttribute('data-id');
     var videoData = getVideoData(videoId);
+
+    var sponsorBlockElements = '';
+    if (videoData.data.sponsorblock && videoData.data.sponsorblock.is_enabled) {
+        sponsorBlock = videoData.data.sponsorblock;
+        if (sponsorBlock.segments.length == 0) {
+            sponsorBlockElements = `
+            <div class="sponsorblock" id="sponsorblock">
+                <h4>This video doesn't have any sponsor segments added. To add a segment go to <u><a href="https://www.youtube.com/watch?v=${videoId}">this video on Youtube</a></u> and add a segment using the <u><a href="https://sponsor.ajay.app/">SponsorBlock</a></u> extension.</h4>
+            </div>
+            `;
+        } else {
+            if (sponsorBlock.has_unlocked) {
+                sponsorBlockElements = `
+                <div class="sponsorblock" id="sponsorblock">
+                    <h4>This video has unlocked sponsor segments. Go to <u><a href="https://www.youtube.com/watch?v=${videoId}">this video on YouTube</a></u> and vote on the segments using the <u><a href="https://sponsor.ajay.app/">SponsorBlock</a></u> extension.</h4>
+                </div>
+                `;
+            }
+        }
+    } else {
+        sponsorBlock = null;
+    }
+
     var videoProgress = getVideoProgress(videoId).position;
     var videoName = videoData.data.title;
 
@@ -353,7 +387,6 @@ function createPlayer(button) {
     var channelName = videoData.data.channel.channel_name;
 
     removePlayer();
-    // document.getElementById(videoId).outerHTML = ''; // Remove watch indicator from video info
 
     // If cast integration is enabled create cast button
     var castButton = '';
@@ -383,6 +416,8 @@ function createPlayer(button) {
     const markup = `
     <div class="video-player" data-id="${videoId}">
         ${videoTag}
+        <div class="notifications" id="notifications"></div>
+        ${sponsorBlockElements}
         <div class="player-title boxed-content">
             <img class="close-button" src="/static/img/icon-close.svg" alt="close-icon" data="${videoId}" onclick="removePlayer()" title="Close player">
             ${watchStatusIndicator}
@@ -483,11 +518,30 @@ function getVideoPlayerWatchStatus() {
     return watched;
 }
 
-// Runs on video playback, marks video as watched if video gets to 90% or higher, sends position to api
+// Runs on video playback, marks video as watched if video gets to 90% or higher, sends position to api, SB skipping
 function onVideoProgress() {
     var videoId = getVideoPlayerVideoId();
     var currentTime = getVideoPlayerCurrentTime();
     var duration = getVideoPlayerDuration();
+    var videoElement = getVideoPlayer();
+    var notificationsElement = document.getElementById("notifications");
+    if (sponsorBlock && sponsorBlock.segments) {
+        for(let i in sponsorBlock.segments) {
+            if(currentTime >= sponsorBlock.segments[i].segment[0] && currentTime <= sponsorBlock.segments[i].segment[0] + 0.3) {
+                videoElement.currentTime = sponsorBlock.segments[i].segment[1];
+                var notificationElement = document.getElementById("notification-" + sponsorBlock.segments[i].UUID);
+                if (!notificationElement) {
+                    notificationsElement.innerHTML += `<h3 id="notification-${sponsorBlock.segments[i].UUID}">Skipped sponsor segment from ${formatTime(sponsorBlock.segments[i].segment[0])} to ${formatTime(sponsorBlock.segments[i].segment[1])}.</h3>`;
+                }
+            }
+            if(currentTime > sponsorBlock.segments[i].segment[1] + 10) {
+                var notificationsElementUUID = document.getElementById("notification-" + sponsorBlock.segments[i].UUID);
+                if(notificationsElementUUID) {
+                    notificationsElementUUID.outerHTML = '';
+                }
+            }
+        }
+    }
     if ((currentTime % 10).toFixed(1) <= 0.2) { // Check progress every 10 seconds or else progress is checked a few times a second
         postVideoProgress(videoId, currentTime);
         if (!getVideoPlayerWatchStatus()) { // Check if video is already marked as watched
@@ -504,6 +558,12 @@ function onVideoEnded() {
     if (!getVideoPlayerWatchStatus()) { // Check if video is already marked as watched
         updateVideoWatchStatus(videoId, "unwatched");
     }
+    for(let i in sponsorBlock.segments) {
+        var notificationsElementUUID = document.getElementById("notification-" + sponsorBlock.segments[i].UUID);
+        if(notificationsElementUUID) {
+            notificationsElementUUID.outerHTML = '';
+        }
+    }
 }
 
 function watchedThreshold(currentTime, duration) {
@@ -542,6 +602,32 @@ function formatNumbers(number) {
     return numberFormatted;
 }
 
+// Formats times in seconds for frontend
+function formatTime(time) {
+    var hoursUnformatted = time / 3600;
+    var minutesUnformatted  = (time % 3600) / 60;
+    var secondsUnformatted  = time % 60;
+
+    var hoursFormatted = Math.trunc(hoursUnformatted);
+    if(minutesUnformatted < 10 && hoursFormatted > 0) {
+        var minutesFormatted  = "0" + Math.trunc(minutesUnformatted);
+    } else {
+        var minutesFormatted = Math.trunc(minutesUnformatted);
+    }
+    if(secondsUnformatted < 10) {
+        var secondsFormatted = "0" + Math.trunc(secondsUnformatted);
+    } else {
+        var secondsFormatted = Math.trunc(secondsUnformatted);
+    }
+
+    var timeUnformatted = '';
+    if(hoursFormatted > 0) {
+        timeUnformatted = hoursFormatted + ":"
+    }
+    var timeFormatted = timeUnformatted.concat(minutesFormatted, ":", secondsFormatted);
+    return timeFormatted;
+}
+
 // Gets video data when passed video ID
 function getVideoData(videoId) {
     var apiEndpoint = "/api/video/" + videoId + "/";
@@ -599,6 +685,48 @@ function postVideoProgress(videoId, videoProgress) {
     }
 }
 
+// Send sponsor segment when given video id and and timestamps
+function postSponsorSegment(videoId, startTime, endTime) {
+    var apiEndpoint = "/api/video/" + videoId + "/sponsor/";
+    var data = {
+        "segment": {
+            "startTime": startTime,
+            "endTime": endTime
+        }
+    };
+    apiRequest(apiEndpoint, "POST", data);
+}
+
+// Send sponsor segment when given video id and and timestamps
+function postSponsorSegmentVote(videoId, uuid, vote) {
+    var apiEndpoint = "/api/video/" + videoId + "/sponsor/";
+    var data = {
+        "vote": {
+            "uuid": uuid,
+            "yourVote": vote
+        }
+    };
+    apiRequest(apiEndpoint, "POST", data);
+}
+
+function handleCookieValidate() {
+    document.getElementById("cookieButton").remove();
+    var cookieMessageElement = document.getElementById("cookieMessage");
+    cookieMessageElement.innerHTML = `<span>Processing.</span>`;
+    response = postCookieValidate();
+    if (response.cookie_validated == true) {
+        cookieMessageElement.innerHTML = `<span>The cookie file is valid.</span>`;
+    } else {
+        cookieMessageElement.innerHTML = `<span class="danger-zone">Warning, the cookie file is invalid.</span>`;
+    }
+}
+
+// Check youtube cookie settings
+function postCookieValidate() {
+    var apiEndpoint = "/api/cookie/";
+    return apiRequest(apiEndpoint, "POST");
+}
+
 // Makes api requests when passed an endpoint and method ("GET", "POST", "DELETE")
 function apiRequest(apiEndpoint, method, data) {
     const xhttp = new XMLHttpRequest();
@@ -657,21 +785,25 @@ function setProgressBar(videoId, currentTime, duration) {
 }
 
 // multi search form
+let searchTimeout = null;
 function searchMulti(query) {
-    if (query.length > 1) {
-        var payload = JSON.stringify({'multi_search': query});
-        var http = new XMLHttpRequest();
-        http.onreadystatechange = function() {
-            if (http.readyState === 4) {
-                allResults = JSON.parse(http.response).results;
-                populateMultiSearchResults(allResults);
-            }
-        };
-        http.open("POST", "/process/", true);
-        http.setRequestHeader("X-CSRFToken", getCookie("csrftoken"));
-        http.setRequestHeader("Content-type", "application/json");
-        http.send(payload);
-    }
+    clearTimeout(searchTimeout);
+    searchTimeout = setTimeout(function () {
+        if (query.length > 1) {
+            var payload = JSON.stringify({'multi_search': query});
+            var http = new XMLHttpRequest();
+            http.onreadystatechange = function() {
+                if (http.readyState === 4) {
+                    allResults = JSON.parse(http.response).results;
+                    populateMultiSearchResults(allResults);
+                }
+            };
+            http.open("POST", "/process/", true);
+            http.setRequestHeader("X-CSRFToken", getCookie("csrftoken"));
+            http.setRequestHeader("Content-type", "application/json");
+            http.send(payload);
+        }
+    }, 500);
 }
 
 function getViewDefaults(view) {