Merge video progress bar to master

* Reduce API calls

* Fix video id

* Updated `createVideoTag()` description.

* Fixed URL used for cast integration

* Check video duration

* Updates progress bar on watched and close.

* Set progress bar width to 0% by default

* Cleanup, function descriptions

* Cleanup console logging

* Update progress bar on cast progress every 10s

* Catch short <30s videos and mark as watched

.github/workflows/lint_python.yml

@@ -9,7 +9,6 @@ jobs:
       - run: pip install --upgrade pip wheel
       - run: pip install bandit black codespell flake8 flake8-bugbear
                          flake8-comprehensions isort
-      - run: bandit --recursive  --skip B105,B108,B404,B603,B607 .
       - run: black --check --diff --line-length 79 .
       - run: codespell
       - run: flake8 . --count --max-complexity=12 --max-line-length=79

.gitignore

@@ -3,3 +3,6 @@ __pycache__
 
 # django testing db
 db.sqlite3
+
+# vscode custom conf
+.vscode

.vscode/settings.json

@@ -1,5 +0,0 @@
-{
-    "python.linting.pylintEnabled": true,
-    "python.linting.pycodestyleEnabled": false,
-    "python.linting.enabled": true
-}

Dockerfile

@@ -1,6 +1,6 @@
 # build the tube archivist image from default python slim image
 
-FROM python:3.10.1-slim-bullseye
+FROM python:3.10.2-slim-bullseye
 ARG TARGETPLATFORM
 
 ENV PYTHONUNBUFFERED 1
@@ -9,13 +9,14 @@ ENV PYTHONUNBUFFERED 1
 RUN apt-get clean && apt-get -y update && apt-get -y install --no-install-recommends \
     build-essential \
     nginx \
+    atomicparsley \
     curl && rm -rf /var/lib/apt/lists/*
 
 # get newest patched ffmpeg and ffprobe builds for amd64 fall back to repo ffmpeg for arm64
 RUN if [ "$TARGETPLATFORM" = "linux/amd64" ] ; then \
     curl -s https://api.github.com/repos/yt-dlp/FFmpeg-Builds/releases/latest \
         | grep browser_download_url \
-        | grep linux64-gpl-4.4.tar.xz \
+        | grep ".*master.*linux64.*tar.xz" \
         | cut -d '"' -f 4 \
         | xargs curl -L --output ffmpeg.tar.xz && \
         tar -xf ffmpeg.tar.xz --strip-components=2 --no-anchored -C /usr/bin/ "ffmpeg" && \
@@ -35,12 +36,12 @@ COPY ./tubearchivist/requirements.txt /requirements.txt
 RUN pip install --no-cache-dir -r requirements.txt --src /usr/local/src
 
 # copy config files
-COPY nginx.conf /etc/nginx/conf.d/
+COPY docker_assets/nginx.conf /etc/nginx/conf.d/
 
 # copy application into container
 COPY ./tubearchivist /app
-COPY ./run.sh /app
-COPY ./uwsgi.ini /app
+COPY ./docker_assets/run.sh /app
+COPY ./docker_assets/uwsgi.ini /app
 
 # volumes
 VOLUME /cache

README.md

@@ -3,7 +3,7 @@
 <center><h1>Your self hosted YouTube media server</h1></center>
 
 ## Table of contents:
-* [Wiki](https://github.com/bbilly1/tubearchivist/wiki) for a detailed documentation
+* [Wiki](https://github.com/bbilly1/tubearchivist/wiki) for a detailed documentation, with [FAQ](https://github.com/bbilly1/tubearchivist/wiki/FAQ)
 * [Core functionality](#core-functionality)
 * [Screenshots](#screenshots)
 * [Problem Tube Archivist tries to solve](#problem-tube-archivist-tries-to-solve)
@@ -99,6 +99,7 @@ You will see the current version number of **Tube Archivist** in the footer of t
 - **arm64**: The Tube Archivist container is multi arch, so is Elasticsearch. RedisJSON doesn't offer arm builds, you can use `bbilly1/rejson`, an unofficial rebuild for arm64.
 - **Synology**: There is a [discussion thread](https://github.com/bbilly1/tubearchivist/discussions/48) with Synology installation instructions.
 - **Unraid**: The three containers needed are all in the Community Applications. First install `TubeArchivist RedisJSON` followed by `TubeArchivist ES`, and finally you can install `TubeArchivist`. If you have unraid specific issues, report those to the [support thread](https://forums.unraid.net/topic/114073-support-crocs-tube-archivist/ "support thread").
+- **Helm Chart**: There is a Helm Chart available at https://github.com/insuusvenerati/helm-charts. Mostly self-explanatory but feel free to ask questions in the discord / subreddit.
 
 
 ## Potential pitfalls
@@ -137,37 +138,39 @@ bestvideo[VCODEC=avc1]+bestaudio[ACODEC=mp4a]/mp4
 5. Enjoy your archived collection!
   
 ## Roadmap
-This should be considered as a **minimal viable product**, there is an extensive list of future functions and improvements planned.
+We have come far, nonetheless we are not short of ideas on how to improve and extend this project:
 
-### Functionality
 - [ ] User roles
 - [ ] Podcast mode to serve channel as mp3
 - [ ] Implement [PyFilesystem](https://github.com/PyFilesystem/pyfilesystem2) for flexible video storage
 - [ ] Implement [Apprise](https://github.com/caronc/apprise) for notifications
-- [ ] Subtitle download & indexing
+- [ ] Add [SponsorBlock](https://sponsor.ajay.app/) integration
+- [ ] Implement per channel settings
 - [ ] User created playlists
 - [ ] Auto play or play next link
 - [ ] SSO / LDAP support
+- [ ] Show similar videos on video page
+- [ ] Multi language support
+- [ ] Show total video downloaded vs total videos available in channel
+- [ ] Make items in grid row configurable
+- [ ] Add statistics of index
+
+Implemented:
+- [X] Subtitle download & indexing [2022-02-13]
+- [X] Fancy advanced unified search interface [2022-01-08]
 - [X] Auto rescan and auto download on a schedule [2021-12-17]
 - [X] Optional automatic deletion of watched items after a specified time [2021-12-17]
 - [X] Create playlists [2021-11-27]
 - [X] Access control [2021-11-01]
 - [X] Delete videos and channel [2021-10-16]
 - [X] Add thumbnail embed option [2021-10-16]
+- [X] Create a github wiki for user documentation [2021-10-03]
+- [X] Grid and list view for both channel and video list pages [2021-10-03]
 - [X] Un-ignore videos [2021-10-03]
 - [X] Dynamic download queue [2021-09-26]
 - [X] Backup and restore [2021-09-22]
 - [X] Scan your file system to index already downloaded videos [2021-09-14]
 
-### UI
-- [ ] Fancy advanced unified search interface
-- [ ] Show similar videos on video page
-- [ ] Multi language support
-- [ ] Show total video downloaded vs total videos available in channel
-- [X] Grid and list view for both channel and video list pages [2021-10-03]
-- [X] Create a github wiki for user documentation [2021-10-03]
-
-
 ## Known limitations
 - Video files created by Tube Archivist need to be playable in your browser of choice. Not every codec is compatible with every browser and might require some testing with format selection. 
 - Every limitation of **yt-dlp** will also be present in Tube Archivist. If **yt-dlp** can't download or extract a video for any reason, Tube Archivist won't be able to either.
@@ -180,4 +183,4 @@ The best donation to **Tube Archivist** is your time, take a look at the [contri
 Second best way to support the development is to provide for caffeinated beverages:
 * [Paypal.me](https://paypal.me/bbilly1) for a one time coffee
 * [Paypal Subscription](https://www.paypal.com/webapps/billing/plans/subscribe?plan_id=P-03770005GR991451KMFGVPMQ) for a monthly coffee
-* [co-fi.com](https://ko-fi.com/bbilly1) for an alternative platform
+* [ko-fi.com](https://ko-fi.com/bbilly1) for an alternative platform

browser_extension/README.md

@@ -0,0 +1,26 @@
+# Tube Archivist Companion
+A browser extension to directly add videos from YouTube to Tube Archivist.
+
+## MVP or better *bearly viable product*
+This is a proof of concept with the following functionality:
+- Add your Tube Archivist connection details in the addon popup
+- Inject a download button into youtube search results page
+- Clicking the button will automatically add the video to the your download queue
+
+## Test this extension
+- Firefox
+  - Open `about:debugging#/runtime/this-firefox`
+  - Click on *Load Temporary Add-on*
+  - Select the *manifest.json* file to load the addon. 
+- Chrome / Chromium
+  - Open `chrome://extensions/`
+  - Toggle *Developer mode* on top right
+  - Click on *Load unpacked*
+  - Open the folder containing the *manifest.json* file.
+
+## Help needed
+This is only minimally useful in this state. Join us on our Discord and please help us improve that.
+
+## Note:
+- For mysterious reasons sometimes the download buttons will only load when refreshing the YouTube search page and not on first load... Hence: Help needed!
+- For your testing environment only for now: Point the extension to the newest *unstable* build.

browser_extension/extension/background.js

@@ -0,0 +1,94 @@
+/*
+extension background script listening for events
+*/
+
+console.log("running background.js");
+
+let browserType = getBrowser();
+
+
+// boilerplate to dedect browser type api
+function getBrowser() {
+    if (typeof chrome !== "undefined") {
+        if (typeof browser !== "undefined") {
+            console.log("detected firefox");
+            return browser;
+        } else {
+            console.log("detected chrome");
+            return chrome;
+        }
+    } else {
+        console.log("failed to dedect browser");
+        throw "browser detection error"
+    };
+}
+
+
+// send post request to API backend
+async function sendPayload(url, token, payload) {
+
+    const rawResponse = await fetch(url, {
+        method: "POST",
+        headers: {
+            "Accept": "application/json",
+            "Content-Type": "application/json",
+            "Authorization": token,
+            "mode": "no-cors"
+        },
+        body: JSON.stringify(payload)
+    });
+
+    const content = await rawResponse.json();
+    return content;
+}
+
+
+// read access storage and send
+function forwardRequest(payload) {
+
+    console.log("running forwardRequest");
+
+    function onGot(item) {
+        console.log(item.access);
+
+        const url = `${item.access.url}:${item.access.port}/api/download/`;
+        console.log(`sending to ${url}`);
+        const token = `Token ${item.access.apiKey}`;
+
+        sendPayload(url, token, payload).then(content => {
+            console.log(content);
+        })
+
+    };
+
+    function onError(error) {
+        console.local("failed to get access details");
+        console.log(`Error: ${error}`);
+    };
+
+    browserType.storage.local.get("access", function(result) {
+        onGot(result)
+    });
+
+}
+
+
+// listen for messages
+browserType.runtime.onMessage.addListener(
+    function(request, sender, sendResponse) {
+        console.log("responding from background.js listener");
+        console.log(JSON.stringify(request));
+        if (request.download) {
+            console.log("found new download task");
+            let payload = {
+                "data": [
+                    {
+                        "youtube_id": request.download["videoId"],
+                        "status": "pending",
+                    }
+                ]
+            }
+            forwardRequest(payload);
+        };
+    }
+);

browser_extension/extension/images/logo.svg

@@ -0,0 +1,102 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!-- Generator: Adobe Illustrator 26.0.2, SVG Export Plug-In . SVG Version: 6.00 Build 0)  -->
+<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
+	 viewBox="0 0 350 75" style="enable-background:new 0 0 350 75;" xml:space="preserve">
+<style type="text/css">
+	.st0{fill:#FFFFFF;}
+	.st1{fill:#2C9D8A;}
+</style>
+<g id="Layer_1_00000061458428874772671790000018384579824765206440_">
+	<g id="archivist">
+		<path class="st0" d="M218.5,15.2v4.6c-0.7-0.5-1.5-0.8-2.5-0.8c-1.8,0-3,1.4-3.9,3.1v8.9h-4v-16h4v3c1.1-1.9,2.6-3.3,4.5-3.3
+			C217.3,14.7,217.9,14.9,218.5,15.2z"/>
+		<path class="st0" d="M233.6,16.3V20c-1.1-0.8-2.4-1.4-4.4-1.4c-2.4,0-4.5,1.9-4.5,4.5c0,2.6,1.9,4.5,4.5,4.5c2,0,3.3-0.6,4.4-1.4
+			v3.7c-0.9,0.8-2.3,1.6-4.7,1.6c-4.8,0-8.4-3.9-8.4-8.4c0-4.5,3.6-8.5,8.4-8.5C231.3,14.6,232.7,15.6,233.6,16.3z"/>
+		<path class="st0" d="M251.8,21.2v9.8h-4l0-9.3c0-2.2-1.1-3.2-2.9-3.2c-1.4,0-2.7,1.1-3.6,2.3v10.1h-4V8.2h4v9.1
+			c1-1.3,2.7-2.7,4.8-2.7C249.3,14.6,251.8,16.9,251.8,21.2z"/>
+		<path class="st0" d="M262.1,31.1h-4V18.5h-2.5v-3.4h6.5V31.1z M256.9,10.1c0-1.5,1.2-2.6,2.6-2.6s2.7,1.2,2.7,2.6
+			c0,1.5-1.2,2.7-2.7,2.7S256.9,11.6,256.9,10.1z"/>
+		<path class="st0" d="M272.2,31.1l-7.5-16h4.7l4,9.9l4.2-9.9h4.7l-7.6,16H272.2z"/>
+		<path class="st0" d="M290.8,31.1h-4V18.5h-2.5v-3.4h6.5V31.1z M285.6,10.1c0-1.5,1.2-2.6,2.6-2.6c1.5,0,2.7,1.2,2.7,2.6
+			c0,1.5-1.2,2.7-2.7,2.7C286.8,12.8,285.6,11.6,285.6,10.1z"/>
+		<path class="st0" d="M305.5,16.1v3.5c-1-0.7-2.5-1.5-4.1-1.5c-1.5,0-2.5,0.8-2.5,1.7c0,0.7,0.4,1.2,1.8,1.7l1.8,0.6
+			c2.5,0.7,3.6,2.2,3.6,4.2c0,2.9-2.8,5.1-6.2,5.1c-2.2,0-4.4-0.8-5.5-1.6v-3.5c1.2,0.8,3.4,1.6,4.9,1.6c1.6,0,2.5-0.6,2.5-1.6
+			c0-0.8-0.7-1.2-2.1-1.6l-2-0.7c-2.4-0.8-3.2-2.3-3.2-4.3c0-3,2.5-5.1,6.1-5.1C302.7,14.6,304.5,15.3,305.5,16.1z"/>
+		<path class="st0" d="M319.8,15.1v3.4h-4.4v6.9c0,1.4,0.7,2.3,2,2.3c1.1,0,1.9-0.3,2.6-0.7v3.2c-0.8,0.8-1.9,1.4-3.8,1.4
+			c-2.9,0-4.8-2-4.8-5.2v-8h-2.9v-2.1c2.1-0.7,4.2-2.7,5.2-6h1.7v4.7H319.8z"/>
+		<polygon id="A" class="st0" points="183.9,31.1 192.7,8 196.6,8 205.4,31.1 201,31.1 194.7,13.5 188.3,31.1 		"/>
+		<path class="st1" d="M197.7,26.2l-1.8,1.6v-5.2v-0.7v-1.8c0-0.8-0.6-1.4-1.3-1.4s-1.3,0.6-1.3,1.4v1.8v0.7v5.2l-1.8-1.6l-1.2,1.2
+			l4.3,3.9l4.3-3.9L197.7,26.2z"/>
+	</g>
+	<g id="tube">
+		<g id="T_00000039852614045324270430000005353782644941079486_">
+			<polygon class="st0" points="94.7,8.3 94.7,12.3 101.3,12.3 101.3,18.4 105.5,18.4 105.5,12.3 111.9,12.3 111.9,8.3 			"/>
+			<rect x="101.3" y="22.6" class="st0" width="4.1" height="9"/>
+			<path class="st1" d="M105.8,21.7H101c-0.7,0-1.3-0.6-1.3-1.3l0,0c0-0.7,0.6-1.3,1.3-1.3h4.8c0.7,0,1.3,0.6,1.3,1.3l0,0
+				C107.1,21.1,106.5,21.7,105.8,21.7z"/>
+		</g>
+		<path class="st0" d="M127.4,24.5c0,4.2-3,6.8-7.1,6.8c-4.1,0-7.1-2.6-7.1-6.8v-9.4h4l0,8.7c0,2.4,1.3,3.7,3.1,3.7
+			c1.8,0,3.1-1.2,3.1-3.7v-8.7h4V24.5z"/>
+		<path class="st0" d="M136,17.3c1-1.8,2.8-3.1,5.2-3.1c3.9,0,7.3,3,7.3,8.4c0,5.4-3.4,8.7-7.5,8.7c-1.8,0-3.7-1-4.9-2.3v1.8h-4.1
+			V7.7h4.1V17.3z M136,22.9v2.3c1.2,1.2,2.5,2.1,4.1,2.1c2.5,0,4.1-2.3,4.1-4.7c0-2.4-1.6-4.3-4-4.3C138,18.3,135.9,20,136,22.9z"/>
+		<path class="st0" d="M167,24.1h-11.7c0.4,2.2,2.3,3.7,4.8,3.7c2.4,0,3.9-1,4.9-1.6v3.7c-0.9,0.7-2.6,1.6-5.1,1.6
+			c-4.8,0-8.7-3.5-8.7-8.4c0-4.7,3.6-8.7,8.2-8.7c4.8,0,7.6,3.3,7.6,8.1V24.1z M163.1,21.5c-0.2-2.3-2-3.5-3.8-3.5
+			c-2.1,0-3.5,1.4-3.9,3.5H163.1z"/>
+	</g>
+	<g id="icon">
+		<path class="st1" d="M27.6,33.7l1,0.4l0.5-2l-6.7-2l-0.7,2l0.9,0.4C20.9,44.4,25,54.5,34,61.9l2.9-4
+			C29.6,51.4,25.8,43.2,27.6,33.7z"/>
+		<path class="st1" d="M41.2,60.5l0.6-0.8L40,58.4L35.7,64l1.8,1.3l0.6-0.7c10.7,5.7,21.9,4.7,31.8-1.5L66.8,59
+			C58.3,63.9,49.9,65.3,41.2,60.5z"/>
+		<path class="st1" d="M70.8,55.6l-0.7-0.7l-1.8,1.3l4.3,5.9l1.8-1.3L73.9,60c8.7-8.5,11.2-18.7,8.4-30l-4.9,1.4
+			C79.3,40.9,78,48.7,70.8,55.6z"/>
+		<path class="st1" d="M75.5,26.5l-0.9,0.4l0.6,2.4l7.2-2.1l-0.7-2.4l-0.9,0.3c-5.7-10.7-15-17.4-26.6-17.9v5.4
+			C63.8,13.4,70.9,17.6,75.5,26.5z"/>
+		<path class="st1" d="M49,12.8l0.1,0.9l2.3-0.1v-7h-2.3v0.9C40,8.8,28.6,14.8,24,27.6l4.9,1.4C32.4,19,40.2,14.7,49,12.8z"/>
+		<polygon id="A_00000111884739883312047940000001222474654974762168_" class="st0" points="48,54 60.6,20.9 66.1,20.9 78.8,54 
+			72.5,54 63.4,28.8 54.3,54 		"/>
+		<g id="T">
+			<polygon class="st0" points="27.6,20.4 27.6,26.1 37.1,26.1 37.1,35 43.1,35 43.1,26.1 52.4,26.1 52.4,20.4 			"/>
+			<rect x="37.1" y="41" class="st0" width="6" height="13"/>
+		</g>
+		<path class="st1" d="M43.5,39.7h-6.9c-1,0-1.9-0.8-1.9-1.9l0,0c0-1,0.8-1.9,1.9-1.9h6.9c1,0,1.9,0.8,1.9,1.9l0,0
+			C45.4,38.9,44.5,39.7,43.5,39.7z"/>
+		<path class="st1" d="M67.8,47l-2.5,2.3v-7.5v-1v-2.5c0-1.1-0.8-2-1.9-2s-1.9,0.9-1.9,2v2.5v1v7.5L58.9,47l-1.7,1.7l6.2,5.6
+			l6.1-5.6L67.8,47z"/>
+	</g>
+</g>
+<g id="Layer_2">
+	<g>
+		<path class="st0" d="M142.1,58c0,6-4.3,9.6-10.1,9.6c-5.8,0-10.1-3.6-10.1-9.6V44.7h5.7l0,12.3c0,3.4,1.8,5.2,4.4,5.2
+			c2.6,0,4.4-1.7,4.4-5.2V44.7h5.7V58z"/>
+		<path class="st0" d="M154.1,48.5c1.3-2.4,3.8-4.3,7.2-4.3c5.4,0,10,4.1,10,11.6c0,7.4-4.6,12-10.4,12c-2.4,0-5.1-1.3-6.8-3.2V67
+			h-5.6V35.2h5.6V48.5z M154.1,56.1v3.2c1.6,1.7,3.4,2.9,5.6,2.9c3.5,0,5.7-3.2,5.7-6.5c0-3.4-2.2-6-5.6-6
+			C156.9,49.7,154.1,52.1,154.1,56.1z"/>
+		<path class="st0" d="M190.4,46.2v5c-1.4-0.9-3.5-2-5.7-2c-2.1,0-3.4,1.1-3.4,2.4c0,1,0.6,1.7,2.6,2.3l2.5,0.8c3.4,1,5.1,3,5.1,5.9
+			c0,4.1-3.9,7.2-8.7,7.2c-3,0-6.1-1.1-7.6-2.3v-4.9c1.7,1.1,4.7,2.3,6.9,2.3c2.3,0,3.5-0.8,3.5-2.3c0-1.2-1-1.6-3-2.3l-2.7-1
+			c-3.4-1.1-4.5-3.2-4.5-6.1c0-4.2,3.5-7.1,8.6-7.1C186.6,44.1,189.2,45.1,190.4,46.2z"/>
+		<path class="st0" d="M212.9,46.5v5.2c-1.5-1.1-3.4-2-6.1-2c-3.4,0-6.2,2.7-6.2,6.3s2.6,6.3,6.2,6.3c2.7,0,4.6-0.8,6.1-1.9v5.1
+			c-1.2,1.1-3.3,2.3-6.6,2.3c-6.7,0-11.7-5.4-11.7-11.7s5-11.9,11.7-11.9C209.7,44.1,211.7,45.5,212.9,46.5z"/>
+		<path class="st0" d="M232.5,44.9v6.4c-1-0.7-2.1-1.1-3.4-1.1c-2.6,0-4.2,1.9-5.5,4.4V67h-5.6V44.7h5.6v4.2
+			c1.5-2.7,3.6-4.6,6.3-4.6C230.9,44.3,231.7,44.5,232.5,44.9z"/>
+		<path class="st0" d="M244.7,67h-5.6V49.5h-3.5v-4.8h9.1V67z M237.5,37.8c0-2,1.6-3.7,3.7-3.7s3.7,1.6,3.7,3.7s-1.7,3.7-3.7,3.7
+			S237.5,39.8,237.5,37.8z"/>
+		<path class="st0" d="M257,48.5c1.3-2.4,3.8-4.3,7.2-4.3c5.4,0,10,4.1,10,11.6c0,7.4-4.6,12-10.4,12c-2.4,0-5.1-1.3-6.8-3.2V67
+			h-5.6V35.2h5.6V48.5z M257,56.1v3.2c1.6,1.7,3.4,2.9,5.6,2.9c3.5,0,5.7-3.2,5.7-6.5c0-3.4-2.2-6-5.6-6
+			C259.8,49.7,257,52.1,257,56.1z"/>
+		<path class="st0" d="M299.8,57.6h-16.2c0.6,3.1,3.1,5,6.5,5c3.3,0,5.4-1.4,6.8-2.3v5.1c-1.2,0.9-3.6,2.3-7,2.3
+			c-6.6,0-12-4.9-12-11.6c0-6.5,5-12,11.3-12c6.6,0,10.5,4.6,10.5,11.1V57.6z M294.4,54c-0.2-3.2-2.7-4.8-5.3-4.8
+			c-2.9,0-4.8,1.9-5.4,4.8H294.4z"/>
+		<path class="st0" d="M319.5,44.9v6.4c-1-0.7-2.1-1.1-3.4-1.1c-2.6,0-4.2,1.9-5.5,4.4V67H305V44.7h5.6v4.2c1.5-2.7,3.6-4.6,6.3-4.6
+			C317.9,44.3,318.7,44.5,319.5,44.9z"/>
+	</g>
+	<g>
+		<path class="st0" d="M102.6,54l2.4,0.9v-6.1c0-0.2,0-0.3,0.1-0.5c-1.2-0.6-1.9-1.4-1.9-2.6c0-2.3,2.6-3.2,4.9-3.2
+			c3.1,0,5.4,1.3,7.5,2.7v-6.1c-1.9-1.2-4.8-2.3-8.3-2.3c-5.5,0-10.5,3.5-10.5,9C96.7,49.3,98,52.3,102.6,54z"/>
+		<path class="st0" d="M111.1,51l-1.9-0.8v6.6c0.8,0.5,1.3,1.2,1.3,2.1c0,1.9-2,2.9-4.5,2.9c-3.5,0-6.7-1.5-9.3-3.4v6.1
+			c2.4,1.7,5.7,3.1,9.2,3.1c6.1,0,11.2-3.8,11.2-9C117.2,55.3,115.3,52.8,111.1,51z"/>
+	</g>
+</g>
+<path class="st1" d="M105.5,56.1v-6.9c0-1,0.8-1.9,1.6-1.9l0,0c0.9,0,1.6,0.9,1.6,1.9v6.9c0,1-0.8,1.9-1.6,1.9l0,0
+	C106.2,58,105.5,57.1,105.5,56.1z"/>
+</svg>

browser_extension/extension/images/question.svg

@@ -0,0 +1,16 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!-- Generator: Adobe Illustrator 26.0.2, SVG Export Plug-In . SVG Version: 6.00 Build 0)  -->
+<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
+	 viewBox="0 0 374.4 375.1" style="enable-background:new 0 0 374.4 375.1;" xml:space="preserve">
+<style type="text/css">
+	.st0{fill:#FFFFFF;}
+</style>
+<path class="st0" d="M187.5,0C84.2-0.1,0.1,83.8,0,187.2C-0.2,290.8,83.7,375,187.1,375.1c103.3,0,187.2-83.9,187.3-187.4
+	C374.5,84.3,290.7,0.2,187.5,0z M187.9,304.5c-16,0-29.2-13.3-29.2-29.3c0.1-16,13.2-29,29.1-29.1c16,0,29.2,13.3,29.2,29.3
+	C216.9,291.4,203.8,304.5,187.9,304.5z M238.2,195.7c-4.8,2.9-9.7,5.8-14.5,8.7c-9.1,5.5-13.6,13.6-13.9,24.2c0,0.8,0,1.7,0,2.7
+	c-14.6,0-29,0-43.8,0c0.1-2.8,0.2-5.6,0.4-8.5c0.8-10.6,3.2-20.9,8.4-30.3c5.9-11,14.9-18.9,25.5-25.1c4.1-2.4,8.4-4.7,12.5-7.1
+	c12.7-7.7,13.1-19.1,6.2-29.9c-6.5-10.3-16.3-14.9-28.1-15.7c-8-0.5-15.7,0.7-22.6,4.9c-10.1,6.2-15,15.6-16.9,26.9
+	c-0.1,0.4-0.1,0.9-0.2,1.6c-14.5-1.9-28.9-3.8-43.9-5.8c1.4-5.4,2.5-10.8,4.2-15.9c5.2-14.9,13.6-27.8,25.8-38
+	c11.1-9.3,23.9-14.7,38.2-16.6c2.3-0.3,4.6-0.6,6.9-1c3.2,0,6.4,0,9.6,0c4.4,0.6,8.9,1.1,13.3,1.9c29.8,5.9,55.4,28.1,61.4,59.3
+	C271.5,157.5,260.7,182.2,238.2,195.7z"/>
+</svg>

browser_extension/extension/images/social/discord.svg

@@ -0,0 +1,24 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!-- Generator: Adobe Illustrator 26.0.2, SVG Export Plug-In . SVG Version: 6.00 Build 0)  -->
+<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
+	 viewBox="0 0 384 384" style="enable-background:new 0 0 384 384;" xml:space="preserve">
+<style type="text/css">
+	.st0{fill:#FFFFFF;}
+</style>
+<g>
+	<path class="st0" d="M191.57,379.5C88.15,379.44,4.32,295.25,4.46,191.57C4.6,88.18,88.66,4.28,191.95,4.42
+		c103.29,0.14,187.02,84.23,186.93,187.72C378.8,295.6,294.86,379.55,191.57,379.5z M244.55,271.24
+		c4.89,5.91,9.44,11.19,13.73,16.69c1.8,2.32,3.72,3.11,6.64,2.93c28.39-1.75,53.17-11.56,72.96-32.55c1.12-1.19,2.08-3.2,1.97-4.76
+		c-0.79-11.71-1.27-23.48-2.97-35.07c-5.07-34.65-16.23-67.36-32.74-98.23c-0.73-1.36-1.88-2.67-3.15-3.55
+		c-19.34-13.43-40.61-21.62-64.16-23.63c-3.19-0.27-5.5,0.48-6.89,3.09c10.27,3.97,20.59,7.65,30.65,11.94
+		c10.1,4.32,19.64,9.74,28.21,16.92c-64.82-29.49-129.53-29.49-194.34,0.02c17.77-13.75,38.02-22.07,59.27-28.39
+		c-1.74-3.15-4.12-3.86-7.32-3.58c-23.15,2.02-44.13,9.94-63.2,23.08c-1.76,1.21-3.51,2.96-4.4,4.86
+		c-5.36,11.39-11.02,22.69-15.57,34.41c-12.22,31.43-19.35,64-19.75,97.86c-0.02,1.71,0.88,3.85,2.07,5.1
+		c18.74,19.74,41.98,29.72,68.88,32.19c5.73,0.53,9.64-0.46,12.96-5.26c3.36-4.86,7.35-9.28,11.2-14.05
+		c-22.46-8.05-39.37-18.53-42.93-26.17c63.97,36.11,128.17,36.13,192.51-0.02C276.69,258.45,261.71,265.95,244.55,271.24z"/>
+	<path class="st0" d="M237.28,232.74c-14.34-0.02-26.07-12-25.9-26.45c0.17-14.67,11.86-26.25,26.56-26.33
+		c13.63-0.07,25.18,12,25.21,26.35C263.18,220.98,251.66,232.76,237.28,232.74z"/>
+	<path class="st0" d="M146.11,232.74c-14.35,0.05-25.9-11.7-25.92-26.38c-0.02-14.3,11.56-26.46,25.16-26.41
+		c14.85,0.06,26.74,11.96,26.62,26.66C171.84,221.04,160.32,232.69,146.11,232.74z"/>
+</g>
+</svg>

browser_extension/extension/images/social/github.svg

@@ -0,0 +1,23 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!-- Generator: Adobe Illustrator 26.0.2, SVG Export Plug-In . SVG Version: 6.00 Build 0)  -->
+<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
+	 viewBox="0 0 384 384" style="enable-background:new 0 0 384 384;" xml:space="preserve">
+<style type="text/css">
+	.st0{fill:#FFFFFF;}
+</style>
+<g>
+	<path class="st0" d="M244.51,375.15c0-10.59-0.31-21.12,0.08-31.61c0.5-13.65,0.02-26.89-10.41-37.88
+		c8.18-2.06,15.59-3.84,22.95-5.83c3.51-0.95,6.95-2.22,10.37-3.47c27.26-9.99,40.88-31.15,45.79-58.73
+		c2.71-15.21,4.11-30.82-0.74-45.88c-2.65-8.23-6.01-16.69-11.08-23.5c-3.83-5.14-3.43-9.05-2.23-14.27
+		c2.86-12.48,0.71-24.71-2.64-36.79c-0.99-3.58-3.09-4.99-6.88-4.6c-11.45,1.18-21.48,6.12-31.34,11.6
+		c-1.42,0.79-2.97,1.46-4.16,2.52c-5.55,4.99-11.4,5.13-18.58,3.42c-25.28-6.04-50.67-5.14-75.85,1.12
+		c-3.59,0.89-6.27,0.27-9.31-1.86c-11.08-7.73-22.87-14.02-36.37-16.31c-0.13-0.02-0.26-0.07-0.4-0.09
+		c-7.12-1.23-8.12-0.76-10.11,6.16c-3.51,12.23-5.18,24.6-1.55,37.14c1.14,3.94,0.44,7.16-2.01,10.61
+		c-23.14,32.46-16.36,80.65,0.11,103.41c12.16,16.81,29.02,26.2,48.63,31.06c5.49,1.36,11.11,2.15,17.1,3.28
+		c-0.24,0.79-0.43,1.81-0.85,2.71c-3.15,6.91-6.27,13.83-9.6,20.65c-0.67,1.38-2.06,3.14-3.34,3.35c-7.3,1.18-14.65,2.46-22.02,2.7
+		c-6.5,0.21-12.09-3.32-16.37-8c-5.11-5.59-9.65-11.73-14.19-17.83c-6.07-8.15-14.2-12.42-24.05-13.3c-2.22-0.2-4.58,1.12-7.99,2.05
+		c2.69,3.23,4.46,6.17,6.96,8.2c10.32,8.35,17.42,19.1,22.26,31.24c4.72,11.85,14.1,16.55,25.31,19.27
+		c9.24,2.24,18.51,1.5,27.83,0.78c6.07-0.47,6.09-0.22,6.09,5.98c0.01,5.58,0,11.17,0,16.81C86.7,368.22,7.24,301.96,4.67,198.28
+		C2.19,97.88,77.65,13.69,175.94,5.34c103.69-8.81,192.95,68.52,202.14,171.23C387.22,278.72,317.17,355.78,244.51,375.15z"/>
+</g>
+</svg>

browser_extension/extension/images/social/reddit.svg

@@ -0,0 +1,35 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!-- Generator: Adobe Illustrator 26.0.2, SVG Export Plug-In . SVG Version: 6.00 Build 0)  -->
+<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
+	 viewBox="0 0 384 384" style="enable-background:new 0 0 384 384;" xml:space="preserve">
+<style type="text/css">
+	.st0{fill:#FFFFFF;}
+</style>
+<g>
+	<path class="st0" d="M240.82,274c-3.86,3.78-8.45,6.33-13.43,8.26c-9.77,3.79-19.98,5.34-30.38,5.85
+		c-5.42,0.26-10.85,0.12-16.26-0.44c-5.89-0.61-11.72-1.56-17.42-3.18c-7.65-2.17-14.86-5.18-20.68-10.86
+		c-0.98-0.96-2.26-1.44-3.64-1.48c-2.72-0.08-4.71,1.17-5.94,3.55c-1.14,2.2-0.8,4.87,0.94,6.7c1.28,1.35,2.7,2.59,4.17,3.73
+		c5.88,4.57,12.58,7.49,19.65,9.63c11.11,3.37,22.52,4.66,36.04,4.85c2.63-0.18,7.2-0.42,11.76-0.82c6.75-0.6,13.38-1.86,19.88-3.8
+		c6.56-1.95,12.83-4.56,18.42-8.57c1.94-1.4,3.82-2.93,5.52-4.61c2.44-2.4,2.35-6.62,0.01-8.95
+		C247.1,271.52,243.37,271.51,240.82,274z"/>
+	<path class="st0" d="M192.39,4.72C89.03,4.5,5.07,88.18,4.86,191.61c-0.21,103.43,83.4,187.46,186.75,187.67
+		c103.35,0.21,187.31-83.46,187.53-186.89C379.36,88.96,295.74,4.93,192.39,4.72z M327.66,225.47c0.14,1.32,0.27,2.64,0.41,3.96
+		c0.53,4.98,0.43,9.96-0.23,14.91c-2.24,16.85-10.03,30.98-21.45,43.28c-8.94,9.62-19.42,17.22-30.95,23.42
+		c-13.96,7.52-28.79,12.64-44.3,15.87c-8.46,1.76-17,2.91-25.62,3.51c-6.58,0.46-13.16,0.6-19.75,0.39
+		c-6.8-0.22-13.58-0.8-20.34-1.72c-11.93-1.64-23.6-4.38-34.96-8.39c-14.15-4.99-27.38-11.73-39.34-20.86
+		c-9.74-7.44-18.15-16.12-24.63-26.59c-5.11-8.26-8.64-17.14-10.2-26.75c-1.1-6.76-1.23-13.55-0.14-20.33c0.03-0.21,0-0.44,0-0.7
+		c-10.51-5.25-17.19-13.54-19.32-25.14c-1.71-9.32,0.32-17.97,5.8-25.72c11.73-16.6,35.97-19.53,51.41-4.96
+		c27.77-19.23,59-27.5,92.46-28.92c0.46-2.12,0.92-4.2,1.36-6.28c3.46-16.26,6.92-32.53,10.37-48.79
+		c1.82-8.57,3.63-17.14,5.43-25.71c0.26-1.23,0.59-2.42,1.44-3.4c1.68-1.92,3.83-2.34,6.2-1.85c6.31,1.3,12.62,2.66,18.92,3.99
+		c10.35,2.19,20.7,4.39,31.05,6.59c2,0.42,4,0.84,6.09,1.28c3.7-6.86,9.29-11.3,16.92-12.93c5.54-1.19,10.89-0.41,15.96,2.14
+		c10.03,5.06,15.32,16.56,12.65,27.42c-2.74,11.12-12.68,18.73-24.01,18.65c-11.69-0.08-23.34-9.38-24.16-23.21
+		c-1.43-0.6-49.41-10.81-50.5-10.7c-5.15,24.12-10.32,48.27-15.51,72.57c0.64,0.04,1.07,0.07,1.51,0.09
+		c19.74,0.93,38.91,4.68,57.37,11.79c11.05,4.25,21.5,9.64,31.21,16.44c0.29,0.2,0.6,0.38,0.94,0.59
+		c8.62-7.75,18.62-11.01,30.07-8.89c9.47,1.76,16.97,6.78,22.22,14.87c5.67,8.73,7.06,18.26,4.21,28.25
+		C343.42,213.65,337.03,220.82,327.66,225.47z"/>
+	<path class="st0" d="M162.26,218.68c0.08-13.49-10.62-24.33-24.09-24.41c-13.86,0-24.09,11.31-24.22,23.89
+		c-0.13,12.52,9.75,24.24,23.96,24.42C151.22,242.73,162.19,231.9,162.26,218.68z"/>
+	<path class="st0" d="M245.16,194.56c-13.42,0.03-24.09,10.83-24.11,24.24c-0.02,12.58,10.14,24.01,24.17,24.05
+		c13.33,0.04,24.16-10.89,24.12-24.26C269.32,205.3,258.48,194.52,245.16,194.56z"/>
+</g>
+</svg>

browser_extension/extension/index.html

@@ -0,0 +1,93 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta http-equiv="X-UA-Compatible" content="IE=edge">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>TubeArchivist Companion</title>
+    <link rel="stylesheet" href="https://fonts.googleapi.com/css?family=Sen" type="text/css">
+    <style>
+        body {
+            font-family: Sen-Regular, sans-serif;
+            background-color: #00202f;
+            color: #97d4c8;
+        }
+        .container {
+            padding: 10px;
+            min-width: 300px;
+        }
+        hr {
+            height: 1px;
+            color: white;
+            background-color: white;
+        }
+        .login-form {
+            display: grid;
+            grid-template-columns: 1fr 1fr;
+        }
+        .login-form label,
+        .login-form input {
+            margin: 3px 0;
+        }
+        .submit button {
+            margin: 10px auto 15px auto;
+            display: block;
+            border-radius: 0;
+            padding: 5px 13px;
+            border: none;
+            cursor: pointer;
+            background-color: #259485;
+            color: #ffffff;
+        }
+        .submit button:hover {
+            background-color: #97d4c8;
+            transform: scale(1.05);
+            color: #00202f;
+        }
+        .icons {
+            display: flex;
+            grid-template-columns: 1fr 1fr;
+            justify-content: space-between;
+        }
+        .icons img {
+            width: 25px;
+        }
+    </style>
+</head>
+<body>
+    <div class="container">
+        <img src="/images/logo.svg" alt="ta-logo">
+        <hr>
+        <form class="login-form">
+            <label for="url">Tube Archivist Url:</label>
+            <input type="text" id="url" name="url">
+            <label for="port">Tube Archivist Port:</label>
+            <input type="text" id="port" name="port">
+            <label for="api-key">API key:</label>
+            <input type="text" id="api-key" name="api-key">
+        </form>
+        <div class="submit">
+            <button onclick="storeLogin()" id="save-login">Save</button>
+        </div>
+        <div class="icons">
+            <div>
+                <a href="https://www.reddit.com/r/TubeArchivist/" target="_blank">
+                    <img src="/images/social/reddit.svg" alt="reddit-icon">
+                </a>
+                <a href="https://discord.gg/AFwz8nE7BK" target="_blank">
+                    <img src="/images/social/discord.svg" alt="discord-icon">
+                </a>
+                <a href="https://github.com/bbilly1/tubearchivist" target="_blank">
+                    <img src="/images/social/github.svg" alt="github-icon">
+                </a>
+            </div>
+            <div>
+                <a href="#">
+                    <img src="/images/question.svg" alt="question-icon">
+                </a>
+            </div>
+        </div>
+    </div>
+    <script type="text/javascript" src="popup.js"></script>
+</body>
+</html>

browser_extension/extension/manifest.json

@@ -0,0 +1,25 @@
+{
+    "manifest_version": 2,
+    "name": "TubeArchivist Companion",
+    "description": "Interact with your selhosted TA server.",
+    "version": "0.0.1",
+    "icons": {
+        "128": "/images/icon128.png"
+    },
+    "browser_action": {
+        "default_icon": "/images/icon.png",
+        "default_popup": "index.html"
+    },
+    "permissions": [
+        "storage"
+    ],
+    "content_scripts": [
+        {
+            "matches": ["https://www.youtube.com/results*"],
+            "js": ["script.js"]
+        }
+    ],
+    "background": {
+        "scripts": ["background.js"]
+    }
+}

browser_extension/extension/popup.js

@@ -0,0 +1,63 @@
+/*
+Loaded into popup index.html
+*/
+
+let browserType = getBrowser();
+
+// boilerplate to dedect browser type api
+function getBrowser() {
+    if (typeof chrome !== "undefined") {
+        if (typeof browser !== "undefined") {
+            console.log("detected firefox");
+            return browser;
+        } else {
+            console.log("detected chrome");
+            return chrome;
+        }
+    } else {
+        console.log("failed to dedect browser");
+        throw "browser detection error"
+    };
+}
+
+// store access details
+document.getElementById("save-login").addEventListener("click", function () {
+    console.log("save form");
+    let toStore = {
+        "access": {
+            "url": document.getElementById("url").value,
+            "port": document.getElementById("port").value,
+            "apiKey": document.getElementById("api-key").value
+        }
+    };
+    console.log(toStore);
+    browserType.storage.local.set(toStore, function() {
+        console.log("Stored connection details: " + JSON.stringify(toStore));
+    });
+})
+
+// fill in form
+document.addEventListener("DOMContentLoaded", async () => {
+
+    console.log("executing dom loader");
+
+    function onGot(item) {
+        if (!item.access) {
+            console.log("no access details found");
+            return
+        }
+        console.log(item.access);
+        document.getElementById("url").value = item.access.url;
+        document.getElementById("port").value = item.access.port;
+        document.getElementById("api-key").value = item.access.apiKey;
+    };
+    
+    function onError(error) {
+        console.log(`Error: ${error}`);
+    };
+
+    browserType.storage.local.get("access", function(result) {
+        onGot(result)
+    });
+
+})

browser_extension/extension/script.js

@@ -0,0 +1,72 @@
+/*
+content script running on youtube.com
+*/
+
+console.log("running script.js");
+
+let browserType = getBrowser();
+
+setTimeout(function(){
+    console.log("running setimeout")
+    linkFinder();
+    return false;
+}, 2000);
+
+
+// boilerplate to dedect browser type api
+function getBrowser() {
+    if (typeof chrome !== "undefined") {
+        if (typeof browser !== "undefined") {
+            console.log("detected firefox");
+            return browser;
+        } else {
+            console.log("detected chrome");
+            return chrome;
+        }
+    } else {
+        console.log("failed to dedect browser");
+        throw "browser detection error"
+    };
+}
+
+
+// event handler for download task
+function addToDownload(videoId) {
+
+    console.log(`downloading ${videoId}`);
+    let payload = {
+        "download": {
+            "videoId": videoId
+        }
+    };
+
+    browserType.runtime.sendMessage(payload);
+
+}
+
+
+// find relevant links to add a button to
+function linkFinder() {
+
+    console.log("running link finder");
+
+    var allLinks = document.links;
+    for (let i = 0; i < allLinks.length; i++) {
+        
+        const linkItem = allLinks[i];
+        const linkDest = linkItem.getAttribute("href");
+
+        if (linkDest.startsWith("/watch?v=") && linkItem.id == "video-title") {
+            var dlButton = document.createElement("button");
+            dlButton.innerText = "download";
+            var videoId = linkDest.split("=")[1];
+            dlButton.setAttribute("data-id", videoId);
+            dlButton.setAttribute("id", "ta-dl-" + videoId);
+            dlButton.onclick = function(event) {
+                var videoId = this.getAttribute("data-id");
+                addToDownload(videoId);
+            };
+            linkItem.parentElement.appendChild(dlButton);
+        }
+    }
+}

deploy.sh

@@ -85,9 +85,7 @@ function validate {
     fi
 
     echo "run validate on $check_path"
-    
-    echo "running bandit"
-    bandit --recursive --skip B105,B108,B404,B603,B607 "$check_path"
+
     echo "running black"
     black --diff --color --check -l 79 "$check_path"
     echo "running codespell"
@@ -103,7 +101,7 @@ function validate {
 
 
 # publish unstable tag to docker
-function sync_unsable {
+function sync_unstable {
 
     if [[ $(systemctl is-active docker) != 'active' ]]; then
         echo "starting docker"
@@ -147,7 +145,6 @@ function sync_docker {
     sudo docker buildx build \
         --platform linux/amd64,linux/arm64 \
         -t bbilly1/tubearchivist:latest \
-        -t bbilly1/tubearchivist:unstable \
         -t bbilly1/tubearchivist:"$VERSION" --push .
 
     # create release tag
@@ -171,8 +168,9 @@ elif [[ $1 == "validate" ]]; then
     validate "$2"
 elif [[ $1 == "docker" ]]; then
     sync_docker
+    sync_unstable
 elif [[ $1 == "unstable" ]]; then
-    sync_unsable
+    sync_unstable
 else
     echo "valid options are: blackhole | test | validate | docker | unstable"
 fi

docker-compose.yml

@@ -8,8 +8,8 @@ services:
     ports:
       - 8000:8000
     volumes:
-      - ./volumes/tubearchivist/media:/youtube
-      - ./volumes/tubearchivist/cache:/cache
+      - media:/youtube
+      - cache:/cache
     environment:
       - ES_URL=http://archivist-es:9200
       - REDIS_HOST=archivist-redis
@@ -23,17 +23,17 @@ services:
       - archivist-es
       - archivist-redis
   archivist-redis:
-    image: redislabs/rejson:latest
+    image: redislabs/rejson:latest      # For arm64 just update this line with bbilly1/rejson:latest
     container_name: archivist-redis
     restart: always
     expose:
       - "6379"
     volumes:
-      - ./volumes/tubearchivist/redis:/data
+      - redis:/data
     depends_on:
       - archivist-es
   archivist-es:
-    image: docker.elastic.co/elasticsearch/elasticsearch:7.16.2
+    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.0
     container_name: archivist-es
     restart: always
     environment:
@@ -46,6 +46,12 @@ services:
         soft: -1
         hard: -1
     volumes:
-      - ./volumes/tubearchivist/es:/usr/share/elasticsearch/data
+      - es:/usr/share/elasticsearch/data
     expose:
       - "9200"
+
+volumes:
+  media:
+  cache:
+  redis:
+  es:

docker_assets/nginx.conf

@@ -16,6 +16,10 @@ server {
 
     location /media/ {
         alias /youtube/;
+
+        types {
+            text/vtt vtt;
+        }
     }
 
     location / {

docker_assets/run.sh

@@ -1,11 +1,6 @@
 #!/bin/bash
 # startup script inside the container for tubearchivist
 
-# check environment
-if [[ -z "$DJANGO_DEBUG" ]]; then
-    export DJANGO_DEBUG=False
-fi
-
 if [[ -z "$ELASTIC_USER" ]]; then
     export ELASTIC_USER=elastic
 fi

docs/FAQ.md

@@ -0,0 +1,31 @@
+# Frequently Asked Questions
+
+## 1. Scope of this project
+Tube Archivist is *Your self hosted YouTube media server*, which also defines the primary scope of what this project tries to do:
+- **Self hosted**: This assumes you have full control over the underlying operating system and hardware and can configure things to work properly with Docker, it's volumes and networks as well as whatever disk storage and filesystem you choose to use.
+- **YouTube**: Downloading, indexing and playing videos from YouTube, there are currently no plans to expand this to any additional platforms.
+- **Media server**: This project tries to be a stand alone media server in it's own web interface.
+
+Additionally to that, progress is also happening on:
+- **API**: Endpoints for additional integrations.
+- **Browser Extension**: To integrate between youtube.com and Tube Archivist.
+
+Defining the scope is important for the success of any project:
+- A scope too broad will result in development effort spreading too thin and will run into danger that his project tries to do too many things and none of them well.
+- A too narrow scope will make this project uninteresting and will exclude audiences that could also benefit from this project.
+- Not defining a scope will easily lead to misunderstandings and false hopes of where this project tries to go.
+
+Of course this is subject to change, as this project continues to grow and more people contribute.
+
+## 2. Emby/Plex/Jellyfin/Kodi integrations
+Although there are similarities between these excellent projects and Tube Archivist, they have a very different use case. Trying to fit the metadata relations and database structure of a YouTube archival project into these media servers that specialize in Movies and TV shows is always going to be limiting.
+
+Part of the scope is to be its own media server, so that's where the focus and effort of this project is. That being said, the nature of self hosted and open source software gives you all the possible freedom to use your media as you wish.
+
+## 3. To Docker or not to Docker
+This project is a classical docker application: There are multiple moving parts that need to be able to interact with each other and need to be compatible with multiple architectures and operating systems. Additionally Docker also drastically reduces development complexity which is highly appreciated.  
+
+So Docker is the only supported installation method. If you don't have any experience with Docker, consider investing the time to learn this very useful technology.
+
+## 4. Finetuning Elasticsearch
+A minimal configuration of Elasticsearch (ES) is provided in the example docker-compose.yml file. ES is highly configurable and very interesting to learn more about. Refer to the [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html) if you want to get into it.

docs/Home.md

@@ -2,6 +2,7 @@
 Welcome to the official Tube Archivist Wiki. This is an up-to-date documentation of user functionality.
 
 Table of contents:
+* [FAQ](FAQ): Frequently asked questions what this project is and tries to do
 * [Channels](Channels): Browse your channels, handle channel subscriptions
 * [Playlists](Playlists): Browse your indexed playlists, handle playlist subscriptions
 * [Downloads](Downloads): Scanning subscriptions, handle download queue
@@ -21,6 +22,7 @@ Table of contents:
 * Clicking on a video title brings you to the dedicated video page and shows additional details.
 * Clicking on a video thumbnail opens the video player and starts streaming the selected video.
 * Clicking on the search icon <img src="assets/icon-search.png?raw=true" alt="gridview icon" width="20px" style="margin:0 5px;"> will open a dedicated search page to search over your complete index.
+* The pagination - if available - builds links for up to 10'000 results, use the search, sort or filter functionality to find what you are looking for.
 
 
 An empty checkbox icon <img src="assets/icon-unseen.png?raw=true" alt="unseen icon" width="20px" style="margin:0 5px;"> will show for videos you haven't marked as watched. Click on it and the icon will change to a filled checkbox <img src="assets/icon-seen.png?raw=true" alt="seen icon" width="20px" style="margin:0 5px;"> indicating it as watched - click again to revert.

docs/Settings.md

@@ -27,9 +27,26 @@ Additional settings passed to yt-dlp.
 - **Embed Metadata**: This saves the available tags directly into the media file by passing `--embed-metadata` to yt-dlp.
 - **Embed Thumbnail**: This will save the thumbnail into the media file by passing `--embed-thumbnail` to yt-dlp.
 
+## Subtitles
+- **Download Setting**: Select the subtitle language you like to download. Add a comma separated list for multiple languages.
+- **Source Settings**: User created subtitles are provided from the uploader and are usually the video script. Auto generated is from YouTube, quality varies, particularly for auto translated tracks.
+- **Index Settings**: Enabling subtitle indexing will add the lines to Elasticsearch and will make subtitles searchable. This will increase the index size and is not recommended on low-end hardware.
+
 ## Integrations
 All third party integrations of TubeArchivist will **always** be *opt in*.
-- **returnyoutubedislike.com**: This will get dislikes and average ratings for each video back by integarting with the API from [returnyoutubedislike.com](https://www.returnyoutubedislike.com/).
+- **API**: Your access token for the Tube Archivist API. 
+- **returnyoutubedislike.com**: This will get return dislikes and average ratings for each video by integrating with the API from [returnyoutubedislike.com](https://www.returnyoutubedislike.com/).
+- **Cast**: Enabling the cast integration in the settings page will load an additional JS library from **Google**.
+    * Requirements
+        - HTTPS
+            * To use the cast integration HTTPS needs to be enabled, which can be done using a reverse proxy. This is a requirement by Google as communication to the cast device is required to be encrypted, but the content itself is not.
+        - Supported Browser
+            * A supported browser is required for this integration such as Google Chrome. Other browsers, especially Chromium-based browsers, may support casting by enabling it in the settings.
+        - Subtitles
+            * Subtitles are supported however they do not work out of the box and require additional configuration. Due to requirements by Google, to use subtitles you need additional headers which will need to be configured in your reverse proxy. See this [page](https://developers.google.com/cast/docs/web_sender/advanced#cors_requirements) for the specific requirements.
+               > You need the following headers: Content-Type, Accept-Encoding, and Range. Note that the last two headers, Accept-Encoding and Range, are additional headers that you may not have needed previously.
+               > Wildcards "*" cannot be used for the Access-Control-Allow-Origin header. If the page has protected media content, it must use a domain instead of a wildcard.
+
 
 # Scheduler Setup
 Schedule settings expect a cron like format, where the first value is minute, second is hour and third is day of the week. Day 0 is Sunday, day 1 is Monday etc.
@@ -69,7 +86,7 @@ Create a zip file of the metadata and select **Max auto backups to keep** to aut
 Additional database functionality.
 
 ## Manual Media Files Import
-So far this depends on the video you are trying to import to be still available on YouTube to get the metadata. Add the files you like to import to the */cache/import* folder. Then start the process from the settings page *Manual Media Files Import*. Make sure to follow one of the two methods below.
+So far this depends on the video you are trying to import to be still available on YouTube to get the metadata. Add the files you'd like to import to the */cache/import* folder. Then start the process from the settings page *Manual Media Files Import*. Make sure to follow one of the two methods below.
 
 ### Method 1:
 Add a matching *.json* file with the media file. Both files need to have the same base name, for example:
@@ -86,6 +103,7 @@ Detect the YouTube ID from filename, this accepts the default yt-dlp naming conv
 
 ### Some notes:
 - This will **consume** the files you put into the import folder: Files will get converted to mp4 if needed (this might take a long time...) and moved to the archive, *.json* files will get deleted upon completion to avoid having duplicates on the next run.
+- There should be no subdirectories added to */cache/import*, only video files. If your existing video library has video files inside subdirectories, you can get all the files into one directory by running `find ./ -mindepth 2 -type f -exec mv '{}' . \;` from the top-level directory of your existing video library. You can also delete any remaining empty subdirectories with `find ./ -mindepth 1 -type d -delete`.
 - Maybe start with a subset of your files to import to make sure everything goes well...
 - Follow the logs to monitor progress and errors: `docker-compose logs -f tubearchivist`.
 

tubearchivist/api/README.md

@@ -0,0 +1,85 @@
+# TubeArchivist API
+Documentation of available API endpoints.  
+**Note: This is very early alpha and will change!**
+
+## Authentication
+API token will get automatically created, accessible on the settings page. Token needs to be passed as an authorization header with every request. Additionally session based authentication is enabled too: When you are logged into your TubeArchivist instance, you'll have access to the api in the browser for testing.
+
+Curl example:
+```shell
+curl -v /api/video/<video-id>/ \
+    -H "Authorization: Token xxxxxxxxxx"
+```
+
+Python requests example:
+```python
+import requests
+
+url = "/api/video/<video-id>/"
+headers = {"Authorization": "Token xxxxxxxxxx"}
+response = requests.get(url, headers=headers)
+```
+
+## Video Item View
+/api/video/\<video_id>/
+
+## Video Progress View
+/api/video/\<video_id>/progress  
+
+Progress is stored for each user.
+
+### Get last player position of a video
+GET /api/video/\<video_id>/progress
+```json
+{
+    "youtube_id": "<video_id>",
+    "user_id": 1,
+    "position": 100
+}
+```
+
+### Post player position of video
+POST /api/video/\<video_id>/progress
+```json
+{
+    "position": 100
+}
+```
+
+### Delete player position of video
+DELETE /api/video/\<video_id>/progress  
+
+## Channel List View
+/api/channel/
+
+### Subscribe to a list of channels
+POST /api/channel/
+```json
+{
+    "data": [
+        {"channel_id": "UC9-y-6csu5WGm29I7JiwpnA", "channel_subscribed": true}
+    ]
+}
+```
+
+## Channel Item View
+/api/channel/\<channel_id>/
+
+## Playlists Item View
+/api/playlist/\<playlist_id>/
+
+## Download Queue List View
+/api/download/
+
+### Add list of videos to download queue
+POST /api/download/
+```json
+{
+    "data": [
+        {"youtube_id": "NYj3DnI81AQ", "status": "pending"}
+    ]
+}
+```
+
+## Download Queue Item View
+/api/download/\<video_id>/

tubearchivist/api/admin.py

@@ -0,0 +1,3 @@
+from django.contrib import admin  # noqa: F401
+
+# Register your models here.

tubearchivist/api/apps.py

@@ -0,0 +1,10 @@
+"""apps file for api package"""
+
+from django.apps import AppConfig
+
+
+class ApiConfig(AppConfig):
+    """app config"""
+
+    default_auto_field = "django.db.models.BigAutoField"
+    name = "api"

tubearchivist/api/models.py

@@ -0,0 +1,3 @@
+"""api models"""
+
+# from django.db import models

tubearchivist/api/tests.py

@@ -0,0 +1,3 @@
+from django.test import TestCase  # noqa: F401
+
+# Create your tests here.

tubearchivist/api/urls.py

@@ -0,0 +1,50 @@
+"""all api urls"""
+
+from api.views import (
+    ChannelApiListView,
+    ChannelApiView,
+    DownloadApiListView,
+    DownloadApiView,
+    PlaylistApiView,
+    VideoApiView,
+    VideoProgressView,
+)
+from django.urls import path
+
+urlpatterns = [
+    path(
+        "video/<slug:video_id>/",
+        VideoApiView.as_view(),
+        name="api-video",
+    ),
+    path(
+        "video/<slug:video_id>/progress/",
+        VideoProgressView.as_view(),
+        name="api-video-progress",
+    ),
+    path(
+        "channel/",
+        ChannelApiListView.as_view(),
+        name="api-channel-list",
+    ),
+    path(
+        "channel/<slug:channel_id>/",
+        ChannelApiView.as_view(),
+        name="api-channel",
+    ),
+    path(
+        "playlist/<slug:playlist_id>/",
+        PlaylistApiView.as_view(),
+        name="api-playlist",
+    ),
+    path(
+        "download/",
+        DownloadApiListView.as_view(),
+        name="api-download-list",
+    ),
+    path(
+        "download/<slug:video_id>/",
+        DownloadApiView.as_view(),
+        name="api-download",
+    ),
+]

tubearchivist/api/views.py

@@ -0,0 +1,262 @@
+"""all API views"""
+
+import requests
+from home.src.download.thumbnails import ThumbManager
+from home.src.ta.config import AppConfig
+from home.src.ta.helper import UrlListParser
+from home.src.ta.ta_redis import RedisArchivist
+from home.tasks import extrac_dl, subscribe_to
+from rest_framework.authentication import (
+    SessionAuthentication,
+    TokenAuthentication,
+)
+from rest_framework.permissions import IsAuthenticated
+from rest_framework.response import Response
+from rest_framework.views import APIView
+
+
+class ApiBaseView(APIView):
+    """base view to inherit from"""
+
+    authentication_classes = [SessionAuthentication, TokenAuthentication]
+    permission_classes = [IsAuthenticated]
+    search_base = False
+
+    def __init__(self):
+        super().__init__()
+        self.response = {"data": False}
+        self.status_code = False
+        self.context = False
+        self.default_conf = AppConfig().config
+
+    def config_builder(self):
+        """build confic context"""
+        self.context = {
+            "es_url": self.default_conf["application"]["es_url"],
+            "es_auth": self.default_conf["application"]["es_auth"],
+        }
+        self.response["config"] = self.default_conf
+
+    def get_document(self, document_id):
+        """get single document from es"""
+        es_url = self.context["es_url"]
+        url = f"{es_url}{self.search_base}{document_id}"
+        print(url)
+        response = requests.get(url, auth=self.context["es_auth"])
+        try:
+            self.response["data"] = response.json()["_source"]
+        except KeyError:
+            print(f"item not found: {document_id}")
+            self.response["data"] = False
+        self.status_code = response.status_code
+
+    def process_keys(self):
+        """process keys for frontend"""
+        all_keys = self.response["data"].keys()
+        if "media_url" in all_keys:
+            media_url = self.response["data"]["media_url"]
+            self.response["data"]["media_url"] = f"/media/{media_url}"
+        if "vid_thumb_url" in all_keys:
+            youtube_id = self.response["data"]["youtube_id"]
+            vid_thumb_url = ThumbManager().vid_thumb_path(youtube_id)
+            cache_dir = self.default_conf["application"]["cache_dir"]
+            new_thumb = f"{cache_dir}/{vid_thumb_url}"
+            self.response["data"]["vid_thumb_url"] = new_thumb
+        if "subtitles" in all_keys:
+            all_subtitles = self.response["data"]["subtitles"]
+            for idx, _ in enumerate(all_subtitles):
+                url = self.response["data"]["subtitles"][idx]["media_url"]
+                new_url = f"/media/{url}"
+                self.response["data"]["subtitles"][idx]["media_url"] = new_url
+
+    def get_paginate(self):
+        """add pagination detail to response"""
+        self.response["paginate"] = False
+
+    def get_document_list(self, data):
+        """get a list of results"""
+        es_url = self.context["es_url"]
+        url = f"{es_url}{self.search_base}"
+        print(url)
+        response = requests.get(url, json=data, auth=self.context["es_auth"])
+        all_hits = response.json()["hits"]["hits"]
+        self.response["data"] = [i["_source"] for i in all_hits]
+        self.status_code = response.status_code
+
+
+class VideoApiView(ApiBaseView):
+    """resolves to /api/video/<video_id>/
+    GET: returns metadata dict of video
+    """
+
+    search_base = "/ta_video/_doc/"
+
+    def get(self, request, video_id):
+        # pylint: disable=unused-argument
+        """get request"""
+        self.config_builder()
+        self.get_document(video_id)
+        self.process_keys()
+        return Response(self.response, status=self.status_code)
+
+
+class VideoProgressView(ApiBaseView):
+    """resolves to /api/video/<video_id>/
+    handle progress status for video
+    """
+
+    def get(self, request, video_id):
+        """get progress for a single video"""
+        user_id = request.user.id
+        key = f"{user_id}:progress:{video_id}"
+        video_progress = RedisArchivist().get_message(key)
+        position = video_progress.get("position", 0)
+
+        self.response = {
+            "youtube_id": video_id,
+            "user_id": user_id,
+            "position": position,
+        }
+        return Response(self.response)
+
+    def post(self, request, video_id):
+        """set progress position in redis"""
+        position = request.data.get("position", 0)
+        key = f"{request.user.id}:progress:{video_id}"
+        message = {"position": position, "youtube_id": video_id}
+        RedisArchivist().set_message(key, message, expire=False)
+        self.response = request.data
+
+        return Response(self.response)
+
+    def delete(self, request, video_id):
+        """delete progress position"""
+        key = f"{request.user.id}:progress:{video_id}"
+        RedisArchivist().del_message(key)
+        self.response = {"progress-reset": video_id}
+
+        return Response(self.response)
+
+
+class ChannelApiView(ApiBaseView):
+    """resolves to /api/channel/<channel_id>/
+    GET: returns metadata dict of channel
+    """
+
+    search_base = "/ta_channel/_doc/"
+
+    def get(self, request, channel_id):
+        # pylint: disable=unused-argument
+        """get request"""
+        self.config_builder()
+        self.get_document(channel_id)
+        return Response(self.response, status=self.status_code)
+
+
+class ChannelApiListView(ApiBaseView):
+    """resolves to /api/channel/
+    GET: returns list of channels
+    POST: edit a list of channels
+    """
+
+    search_base = "/ta_channel/_search/"
+
+    def get(self, request):
+        # pylint: disable=unused-argument
+        """get request"""
+        data = {"query": {"match_all": {}}}
+        self.config_builder()
+        self.get_document_list(data)
+        self.get_paginate()
+
+        return Response(self.response)
+
+    @staticmethod
+    def post(request):
+        """subscribe to list of channels"""
+        data = request.data
+        try:
+            to_add = data["data"]
+        except KeyError:
+            message = "missing expected data key"
+            print(message)
+            return Response({"message": message}, status=400)
+
+        pending = [i["channel_id"] for i in to_add if i["channel_subscribed"]]
+        url_str = " ".join(pending)
+        subscribe_to.delay(url_str)
+
+        return Response(data)
+
+
+class PlaylistApiView(ApiBaseView):
+    """resolves to /api/playlist/<playlist_id>/
+    GET: returns metadata dict of playlist
+    """
+
+    search_base = "/ta_playlist/_doc/"
+
+    def get(self, request, playlist_id):
+        # pylint: disable=unused-argument
+        """get request"""
+        self.config_builder()
+        self.get_document(playlist_id)
+        return Response(self.response, status=self.status_code)
+
+
+class DownloadApiView(ApiBaseView):
+    """resolves to /api/download/<video_id>/
+    GET: returns metadata dict of an item in the download queue
+    """
+
+    search_base = "/ta_download/_doc/"
+
+    def get(self, request, video_id):
+        # pylint: disable=unused-argument
+        """get request"""
+        self.config_builder()
+        self.get_document(video_id)
+        return Response(self.response, status=self.status_code)
+
+
+class DownloadApiListView(ApiBaseView):
+    """resolves to /api/download/
+    GET: returns latest videos in the download queue
+    POST: add a list of videos to download queue
+    """
+
+    search_base = "/ta_download/_search/"
+
+    def get(self, request):
+        # pylint: disable=unused-argument
+        """get request"""
+        data = {"query": {"match_all": {}}}
+        self.config_builder()
+        self.get_document_list(data)
+        self.get_paginate()
+        return Response(self.response)
+
+    @staticmethod
+    def post(request):
+        """add list of videos to download queue"""
+        print(f"request meta data: {request.META}")
+        data = request.data
+        try:
+            to_add = data["data"]
+        except KeyError:
+            message = "missing expected data key"
+            print(message)
+            return Response({"message": message}, status=400)
+
+        pending = [i["youtube_id"] for i in to_add if i["status"] == "pending"]
+        url_str = " ".join(pending)
+        try:
+            youtube_ids = UrlListParser(url_str).process_list()
+        except ValueError:
+            message = f"failed to parse: {url_str}"
+            print(message)
+            return Response({"message": message}, status=400)
+
+        extrac_dl.delay(youtube_ids)
+
+        return Response(data)

tubearchivist/config/settings.py

@@ -14,7 +14,8 @@ import hashlib
 from os import environ, path
 from pathlib import Path
 
-from home.src.config import AppConfig
+from corsheaders.defaults import default_headers
+from home.src.ta.config import AppConfig
 
 # Build paths inside the project like this: BASE_DIR / 'subdir'.
 BASE_DIR = Path(__file__).resolve().parent.parent
@@ -41,14 +42,19 @@ INSTALLED_APPS = [
     "django.contrib.contenttypes",
     "django.contrib.sessions",
     "django.contrib.messages",
+    "corsheaders",
     "whitenoise.runserver_nostatic",
     "django.contrib.staticfiles",
     "django.contrib.humanize",
+    "rest_framework",
+    "rest_framework.authtoken",
+    "api",
 ]
 
 MIDDLEWARE = [
     "django.middleware.security.SecurityMiddleware",
     "django.contrib.sessions.middleware.SessionMiddleware",
+    "corsheaders.middleware.CorsMiddleware",
     "whitenoise.middleware.WhiteNoiseMiddleware",
     "django.middleware.common.CommonMiddleware",
     "django.middleware.csrf.CsrfViewMiddleware",
@@ -137,3 +143,11 @@ DEFAULT_AUTO_FIELD = "django.db.models.BigAutoField"
 
 LOGIN_URL = "/login/"
 LOGOUT_REDIRECT_URL = "/login/"
+
+# Cors needed for browser extension
+# background.js makes the request so HTTP_ORIGIN will be from extension
+CORS_ALLOWED_ORIGIN_REGEXES = [r"moz-extension://*", r"chrome-extension://*"]
+
+CORS_ALLOW_HEADERS = list(default_headers) + [
+    "mode",
+]

tubearchivist/config/urls.py

@@ -18,5 +18,6 @@ from django.urls import include, path
 
 urlpatterns = [
     path("", include("home.urls")),
+    path("api/", include("api.urls")),
     path("admin/", admin.site.urls),
 ]

tubearchivist/home/apps.py

@@ -3,9 +3,9 @@
 import os
 
 from django.apps import AppConfig
-from home.src.config import AppConfig as ArchivistConfig
-from home.src.helper import RedisArchivist
-from home.src.index_management import index_check
+from home.src.es.index_setup import index_check
+from home.src.ta.config import AppConfig as ArchivistConfig
+from home.src.ta.ta_redis import RedisArchivist
 
 
 def sync_redis_state():

tubearchivist/home/config.json

@@ -23,6 +23,9 @@
         "format": false,
         "add_metadata": false,
         "add_thumbnail": false,
+        "subtitle": false,
+        "subtitle_source": false,
+        "subtitle_index": false,
         "throttledratelimit": false,
         "integrate_ryd": false
     },
@@ -31,7 +34,8 @@
         "cache_dir": "/cache",
         "videos": "/youtube",
         "file_template": "%(id)s_%(title)s.mp4",
-        "colors": "dark"
+        "colors": "dark",
+        "enable_cast": false
     },
     "scheduler": {
         "update_subscribed": false,

tubearchivist/home/src/download.py

@@ -1,802 +0,0 @@
-"""
-Functionality:
-- handele the download queue
-- manage subscriptions to channels
-- manage subscriptions to playlists
-- downloading videos
-"""
-
-import json
-import os
-import shutil
-from datetime import datetime
-from time import sleep
-
-import requests
-import yt_dlp
-from home.src.config import AppConfig
-from home.src.helper import (
-    DurationConverter,
-    RedisArchivist,
-    RedisQueue,
-    clean_string,
-    ignore_filelist,
-)
-from home.src.index import (
-    IndexPaginate,
-    YoutubeChannel,
-    YoutubePlaylist,
-    YoutubeVideo,
-    index_new_video,
-)
-
-
-class PendingList:
-    """manage the pending videos list"""
-
-    CONFIG = AppConfig().config
-    ES_URL = CONFIG["application"]["es_url"]
-    ES_AUTH = CONFIG["application"]["es_auth"]
-    VIDEOS = CONFIG["application"]["videos"]
-
-    def __init__(self):
-        self.all_channel_ids = False
-        self.all_downloaded = False
-        self.missing_from_playlists = []
-
-    def parse_url_list(self, youtube_ids):
-        """extract youtube ids from list"""
-        missing_videos = []
-        for entry in youtube_ids:
-            # notify
-            mess_dict = {
-                "status": "message:add",
-                "level": "info",
-                "title": "Adding to download queue.",
-                "message": "Extracting lists",
-            }
-            RedisArchivist().set_message("message:add", mess_dict)
-            # extract
-            url = entry["url"]
-            url_type = entry["type"]
-            if url_type == "video":
-                missing_videos.append(url)
-            elif url_type == "channel":
-                video_results = ChannelSubscription().get_last_youtube_videos(
-                    url, limit=False
-                )
-                youtube_ids = [i[0] for i in video_results]
-                missing_videos = missing_videos + youtube_ids
-            elif url_type == "playlist":
-                self.missing_from_playlists.append(entry)
-                video_results = YoutubePlaylist(url).get_entries()
-                youtube_ids = [i["youtube_id"] for i in video_results]
-                missing_videos = missing_videos + youtube_ids
-
-        return missing_videos
-
-    def add_to_pending(self, missing_videos, ignore=False):
-        """build the bulk json data from pending"""
-        # check if channel is indexed
-        channel_handler = ChannelSubscription()
-        all_indexed = channel_handler.get_channels(subscribed_only=False)
-        self.all_channel_ids = [i["channel_id"] for i in all_indexed]
-        # check if already there
-        self.all_downloaded = self.get_all_downloaded()
-
-        bulk_list, all_videos_added = self.build_bulk(missing_videos, ignore)
-        # add last newline
-        bulk_list.append("\n")
-        query_str = "\n".join(bulk_list)
-        headers = {"Content-type": "application/x-ndjson"}
-        url = self.ES_URL + "/_bulk"
-        request = requests.post(
-            url, data=query_str, headers=headers, auth=self.ES_AUTH
-        )
-        if not request.ok:
-            print(request)
-            raise ValueError("failed to add video to download queue")
-
-        return all_videos_added
-
-    def build_bulk(self, missing_videos, ignore=False):
-        """build the bulk lists"""
-        bulk_list = []
-        all_videos_added = []
-
-        for idx, youtube_id in enumerate(missing_videos):
-            # check if already downloaded
-            if youtube_id in self.all_downloaded:
-                continue
-
-            video = self.get_youtube_details(youtube_id)
-            # skip on download error
-            if not video:
-                continue
-
-            channel_indexed = video["channel_id"] in self.all_channel_ids
-            video["channel_indexed"] = channel_indexed
-            if ignore:
-                video["status"] = "ignore"
-            else:
-                video["status"] = "pending"
-            action = {"create": {"_id": youtube_id, "_index": "ta_download"}}
-            bulk_list.append(json.dumps(action))
-            bulk_list.append(json.dumps(video))
-            all_videos_added.append((youtube_id, video["vid_thumb_url"]))
-            # notify
-            progress = f"{idx + 1}/{len(missing_videos)}"
-            mess_dict = {
-                "status": "message:add",
-                "level": "info",
-                "title": "Adding new videos to download queue.",
-                "message": "Progress: " + progress,
-            }
-            if idx + 1 == len(missing_videos):
-                RedisArchivist().set_message(
-                    "message:add", mess_dict, expire=4
-                )
-            else:
-                RedisArchivist().set_message("message:add", mess_dict)
-            if idx + 1 % 25 == 0:
-                print("adding to queue progress: " + progress)
-
-        return bulk_list, all_videos_added
-
-    @staticmethod
-    def get_youtube_details(youtube_id):
-        """get details from youtubedl for single pending video"""
-        obs = {
-            "default_search": "ytsearch",
-            "quiet": True,
-            "check_formats": "selected",
-            "noplaylist": True,
-            "writethumbnail": True,
-            "simulate": True,
-        }
-        try:
-            vid = yt_dlp.YoutubeDL(obs).extract_info(youtube_id)
-        except yt_dlp.utils.DownloadError:
-            print("failed to extract info for: " + youtube_id)
-            return False
-        # stop if video is streaming live now
-        if vid["is_live"]:
-            return False
-        # parse response
-        seconds = vid["duration"]
-        duration_str = DurationConverter.get_str(seconds)
-        if duration_str == "NA":
-            print(f"skip extracting duration for: {youtube_id}")
-        upload_date = vid["upload_date"]
-        upload_dt = datetime.strptime(upload_date, "%Y%m%d")
-        published = upload_dt.strftime("%Y-%m-%d")
-        # build dict
-        youtube_details = {
-            "youtube_id": youtube_id,
-            "channel_name": vid["channel"],
-            "vid_thumb_url": vid["thumbnail"],
-            "title": vid["title"],
-            "channel_id": vid["channel_id"],
-            "duration": duration_str,
-            "published": published,
-            "timestamp": int(datetime.now().strftime("%s")),
-        }
-        return youtube_details
-
-    @staticmethod
-    def get_all_pending():
-        """get a list of all pending videos in ta_download"""
-        data = {
-            "query": {"match_all": {}},
-            "sort": [{"timestamp": {"order": "asc"}}],
-        }
-        all_results = IndexPaginate("ta_download", data).get_results()
-
-        all_pending = []
-        all_ignore = []
-
-        for result in all_results:
-            if result["status"] == "pending":
-                all_pending.append(result)
-            elif result["status"] == "ignore":
-                all_ignore.append(result)
-
-        return all_pending, all_ignore
-
-    @staticmethod
-    def get_all_indexed():
-        """get a list of all videos indexed"""
-
-        data = {
-            "query": {"match_all": {}},
-            "sort": [{"published": {"order": "desc"}}],
-        }
-        all_indexed = IndexPaginate("ta_video", data).get_results()
-
-        return all_indexed
-
-    def get_all_downloaded(self):
-        """get a list of all videos in archive"""
-        channel_folders = os.listdir(self.VIDEOS)
-        all_channel_folders = ignore_filelist(channel_folders)
-        all_downloaded = []
-        for channel_folder in all_channel_folders:
-            channel_path = os.path.join(self.VIDEOS, channel_folder)
-            videos = os.listdir(channel_path)
-            all_videos = ignore_filelist(videos)
-            youtube_vids = [i[9:20] for i in all_videos]
-            for youtube_id in youtube_vids:
-                all_downloaded.append(youtube_id)
-        return all_downloaded
-
-    def delete_from_pending(self, youtube_id):
-        """delete the youtube_id from ta_download"""
-        url = f"{self.ES_URL}/ta_download/_doc/{youtube_id}"
-        response = requests.delete(url, auth=self.ES_AUTH)
-        if not response.ok:
-            print(response.text)
-
-    def delete_pending(self, status):
-        """delete download queue based on status value"""
-        data = {"query": {"term": {"status": {"value": status}}}}
-        payload = json.dumps(data)
-        url = self.ES_URL + "/ta_download/_delete_by_query"
-        headers = {"Content-type": "application/json"}
-        response = requests.post(
-            url, data=payload, headers=headers, auth=self.ES_AUTH
-        )
-        if not response.ok:
-            print(response.text)
-
-    def ignore_from_pending(self, ignore_list):
-        """build the bulk query string"""
-
-        stamp = int(datetime.now().strftime("%s"))
-        bulk_list = []
-
-        for youtube_id in ignore_list:
-            action = {"update": {"_id": youtube_id, "_index": "ta_download"}}
-            source = {"doc": {"status": "ignore", "timestamp": stamp}}
-            bulk_list.append(json.dumps(action))
-            bulk_list.append(json.dumps(source))
-
-        # add last newline
-        bulk_list.append("\n")
-        query_str = "\n".join(bulk_list)
-
-        headers = {"Content-type": "application/x-ndjson"}
-        url = self.ES_URL + "/_bulk"
-        request = requests.post(
-            url, data=query_str, headers=headers, auth=self.ES_AUTH
-        )
-        if not request.ok:
-            print(request)
-            raise ValueError("failed to set video to ignore")
-
-
-class ChannelSubscription:
-    """manage the list of channels subscribed"""
-
-    def __init__(self):
-        config = AppConfig().config
-        self.es_url = config["application"]["es_url"]
-        self.es_auth = config["application"]["es_auth"]
-        self.channel_size = config["subscriptions"]["channel_size"]
-
-    @staticmethod
-    def get_channels(subscribed_only=True):
-        """get a list of all channels subscribed to"""
-        data = {
-            "sort": [{"channel_name.keyword": {"order": "asc"}}],
-        }
-        if subscribed_only:
-            data["query"] = {"term": {"channel_subscribed": {"value": True}}}
-        else:
-            data["query"] = {"match_all": {}}
-
-        all_channels = IndexPaginate("ta_channel", data).get_results()
-
-        return all_channels
-
-    def get_last_youtube_videos(self, channel_id, limit=True):
-        """get a list of last videos from channel"""
-        url = f"https://www.youtube.com/channel/{channel_id}/videos"
-        obs = {
-            "default_search": "ytsearch",
-            "quiet": True,
-            "skip_download": True,
-            "extract_flat": True,
-        }
-        if limit:
-            obs["playlistend"] = self.channel_size
-        chan = yt_dlp.YoutubeDL(obs).extract_info(url, download=False)
-        last_videos = [(i["id"], i["title"]) for i in chan["entries"]]
-        return last_videos
-
-    def find_missing(self):
-        """add missing videos from subscribed channels to pending"""
-        all_channels = self.get_channels()
-        pending_handler = PendingList()
-        all_pending, all_ignore = pending_handler.get_all_pending()
-        all_ids = [i["youtube_id"] for i in all_ignore + all_pending]
-        all_downloaded = pending_handler.get_all_downloaded()
-        to_ignore = all_ids + all_downloaded
-
-        missing_videos = []
-
-        for idx, channel in enumerate(all_channels):
-            channel_id = channel["channel_id"]
-            last_videos = self.get_last_youtube_videos(channel_id)
-            for video in last_videos:
-                if video[0] not in to_ignore:
-                    missing_videos.append(video[0])
-            # notify
-            message = {
-                "status": "message:rescan",
-                "level": "info",
-                "title": "Scanning channels: Looking for new videos.",
-                "message": f"Progress: {idx + 1}/{len(all_channels)}",
-            }
-            if idx + 1 == len(all_channels):
-                RedisArchivist().set_message(
-                    "message:rescan", message=message, expire=4
-                )
-            else:
-                RedisArchivist().set_message("message:rescan", message=message)
-
-        return missing_videos
-
-    def change_subscribe(self, channel_id, channel_subscribed):
-        """subscribe or unsubscribe from channel and update"""
-        if not isinstance(channel_subscribed, bool):
-            print("invalid status, should be bool")
-            return
-        headers = {"Content-type": "application/json"}
-        channel_handler = YoutubeChannel(channel_id)
-        channel_dict = channel_handler.channel_dict
-        channel_dict["channel_subscribed"] = channel_subscribed
-        if channel_subscribed:
-            # handle subscribe
-            url = self.es_url + "/ta_channel/_doc/" + channel_id
-            payload = json.dumps(channel_dict)
-            print(channel_dict)
-        else:
-            url = self.es_url + "/ta_channel/_update/" + channel_id
-            payload = json.dumps({"doc": channel_dict})
-        # update channel
-        request = requests.post(
-            url, data=payload, headers=headers, auth=self.es_auth
-        )
-        if not request.ok:
-            print(request.text)
-            raise ValueError("failed change subscribe status")
-        # sync to videos
-        channel_handler.sync_to_videos()
-        if channel_handler.source == "scraped":
-            channel_handler.get_channel_art()
-
-
-class PlaylistSubscription:
-    """manage the playlist download functionality"""
-
-    def __init__(self):
-        self.config = AppConfig().config
-
-    @staticmethod
-    def get_playlists(subscribed_only=True):
-        """get a list of all active playlists"""
-        data = {
-            "sort": [{"playlist_channel.keyword": {"order": "desc"}}],
-        }
-        data["query"] = {
-            "bool": {"must": [{"term": {"playlist_active": {"value": True}}}]}
-        }
-        if subscribed_only:
-            data["query"]["bool"]["must"].append(
-                {"term": {"playlist_subscribed": {"value": True}}}
-            )
-
-        all_playlists = IndexPaginate("ta_playlist", data).get_results()
-
-        return all_playlists
-
-    def process_url_str(self, new_playlists, subscribed=True):
-        """process playlist subscribe form url_str"""
-        all_indexed = PendingList().get_all_indexed()
-        all_youtube_ids = [i["youtube_id"] for i in all_indexed]
-
-        new_thumbs = []
-        for idx, playlist in enumerate(new_playlists):
-            url_type = playlist["type"]
-            playlist_id = playlist["url"]
-            if not url_type == "playlist":
-                print(f"{playlist_id} not a playlist, skipping...")
-                continue
-
-            playlist_h = YoutubePlaylist(
-                playlist_id, all_youtube_ids=all_youtube_ids
-            )
-            if not playlist_h.get_es_playlist():
-                playlist_h.get_playlist_dict()
-                playlist_h.playlist_dict["playlist_subscribed"] = subscribed
-                playlist_h.upload_to_es()
-                playlist_h.add_vids_to_playlist()
-                thumb = playlist_h.playlist_dict["playlist_thumbnail"]
-                new_thumbs.append((playlist_id, thumb))
-                self.channel_validate(playlist_h)
-            else:
-                self.change_subscribe(playlist_id, subscribe_status=True)
-
-            # notify
-            message = {
-                "status": "message:subplaylist",
-                "level": "info",
-                "title": "Subscribing to Playlists",
-                "message": f"Processing {idx + 1} of {len(new_playlists)}",
-            }
-            RedisArchivist().set_message(
-                "message:subplaylist", message=message
-            )
-
-        return new_thumbs
-
-    @staticmethod
-    def channel_validate(playlist_handler):
-        """make sure channel of playlist is there"""
-        channel_id = playlist_handler.playlist_dict["playlist_channel_id"]
-        channel_handler = YoutubeChannel(channel_id)
-        if channel_handler.source == "scraped":
-            channel_handler.channel_dict["channel_subscribed"] = False
-            channel_handler.upload_to_es()
-            channel_handler.get_channel_art()
-
-    def change_subscribe(self, playlist_id, subscribe_status):
-        """change the subscribe status of a playlist"""
-        es_url = self.config["application"]["es_url"]
-        es_auth = self.config["application"]["es_auth"]
-        playlist_handler = YoutubePlaylist(playlist_id)
-        playlist_handler.get_playlist_dict()
-        subed_now = playlist_handler.playlist_dict["playlist_subscribed"]
-
-        if subed_now == subscribe_status:
-            # status already as expected, do nothing
-            return False
-
-        # update subscribed status
-        headers = {"Content-type": "application/json"}
-        url = f"{es_url}/ta_playlist/_update/{playlist_id}"
-        payload = json.dumps(
-            {"doc": {"playlist_subscribed": subscribe_status}}
-        )
-        response = requests.post(
-            url, data=payload, headers=headers, auth=es_auth
-        )
-        if not response.ok:
-            print(response.text)
-            raise ValueError("failed to change subscribe status")
-
-        return True
-
-    @staticmethod
-    def get_to_ignore():
-        """get all youtube_ids already downloaded or ignored"""
-        pending_handler = PendingList()
-        all_pending, all_ignore = pending_handler.get_all_pending()
-        all_ids = [i["youtube_id"] for i in all_ignore + all_pending]
-        all_downloaded = pending_handler.get_all_downloaded()
-        to_ignore = all_ids + all_downloaded
-        return to_ignore
-
-    def find_missing(self):
-        """find videos in subscribed playlists not downloaded yet"""
-        all_playlists = [i["playlist_id"] for i in self.get_playlists()]
-        to_ignore = self.get_to_ignore()
-
-        missing_videos = []
-        counter = 1
-        for playlist_id in all_playlists:
-            size_limit = self.config["subscriptions"]["channel_size"]
-            playlist_handler = YoutubePlaylist(playlist_id)
-            playlist = playlist_handler.update_playlist()
-            if not playlist:
-                playlist_handler.deactivate()
-                continue
-
-            if size_limit:
-                playlist_entries = playlist["playlist_entries"][:size_limit]
-            else:
-                playlist_entries = playlist["playlist_entries"]
-            all_missing = [i for i in playlist_entries if not i["downloaded"]]
-
-            message = {
-                "status": "message:rescan",
-                "level": "info",
-                "title": "Scanning playlists: Looking for new videos.",
-                "message": f"Progress: {counter}/{len(all_playlists)}",
-            }
-            RedisArchivist().set_message("message:rescan", message=message)
-
-            for video in all_missing:
-                youtube_id = video["youtube_id"]
-                if youtube_id not in to_ignore:
-                    missing_videos.append(youtube_id)
-            counter = counter + 1
-
-        return missing_videos
-
-
-class VideoDownloader:
-    """
-    handle the video download functionality
-    if not initiated with list, take from queue
-    """
-
-    def __init__(self, youtube_id_list=False):
-        self.youtube_id_list = youtube_id_list
-        self.config = AppConfig().config
-        self.channels = set()
-
-    def run_queue(self):
-        """setup download queue in redis loop until no more items"""
-        queue = RedisQueue("dl_queue")
-
-        limit_queue = self.config["downloads"]["limit_count"]
-        if limit_queue:
-            queue.trim(limit_queue - 1)
-
-        while True:
-            youtube_id = queue.get_next()
-            if not youtube_id:
-                break
-
-            try:
-                self.dl_single_vid(youtube_id)
-            except yt_dlp.utils.DownloadError:
-                print("failed to download " + youtube_id)
-                continue
-            vid_dict = index_new_video(youtube_id)
-            self.channels.add(vid_dict["channel"]["channel_id"])
-            self.move_to_archive(vid_dict)
-            self.delete_from_pending(youtube_id)
-
-        autodelete_days = self.config["downloads"]["autodelete_days"]
-        if autodelete_days:
-            print(f"auto delete older than {autodelete_days} days")
-            self.auto_delete_watched(autodelete_days)
-
-    @staticmethod
-    def add_pending():
-        """add pending videos to download queue"""
-        mess_dict = {
-            "status": "message:download",
-            "level": "info",
-            "title": "Looking for videos to download",
-            "message": "Scanning your download queue.",
-        }
-        RedisArchivist().set_message("message:download", mess_dict)
-        all_pending, _ = PendingList().get_all_pending()
-        to_add = [i["youtube_id"] for i in all_pending]
-        if not to_add:
-            # there is nothing pending
-            print("download queue is empty")
-            mess_dict = {
-                "status": "message:download",
-                "level": "error",
-                "title": "Download queue is empty",
-                "message": "Add some videos to the queue first.",
-            }
-            RedisArchivist().set_message("message:download", mess_dict)
-            return
-
-        queue = RedisQueue("dl_queue")
-        queue.add_list(to_add)
-
-    @staticmethod
-    def progress_hook(response):
-        """process the progress_hooks from yt_dlp"""
-        # title
-        path = os.path.split(response["filename"])[-1][12:]
-        filename = os.path.splitext(os.path.splitext(path)[0])[0]
-        filename_clean = filename.replace("_", " ")
-        title = "Downloading: " + filename_clean
-        # message
-        try:
-            percent = response["_percent_str"]
-            size = response["_total_bytes_str"]
-            speed = response["_speed_str"]
-            eta = response["_eta_str"]
-            message = f"{percent} of {size} at {speed} - time left: {eta}"
-        except KeyError:
-            message = "processing"
-        mess_dict = {
-            "status": "message:download",
-            "level": "info",
-            "title": title,
-            "message": message,
-        }
-        RedisArchivist().set_message("message:download", mess_dict)
-
-    def build_obs(self):
-        """build obs dictionary for yt-dlp"""
-        obs = {
-            "default_search": "ytsearch",
-            "merge_output_format": "mp4",
-            "restrictfilenames": True,
-            "outtmpl": (
-                self.config["application"]["cache_dir"]
-                + "/download/"
-                + self.config["application"]["file_template"]
-            ),
-            "progress_hooks": [self.progress_hook],
-            "noprogress": True,
-            "quiet": True,
-            "continuedl": True,
-            "retries": 3,
-            "writethumbnail": False,
-            "noplaylist": True,
-            "check_formats": "selected",
-        }
-        if self.config["downloads"]["format"]:
-            obs["format"] = self.config["downloads"]["format"]
-        if self.config["downloads"]["limit_speed"]:
-            obs["ratelimit"] = self.config["downloads"]["limit_speed"] * 1024
-
-        throttle = self.config["downloads"]["throttledratelimit"]
-        if throttle:
-            obs["throttledratelimit"] = throttle * 1024
-
-        postprocessors = []
-
-        if self.config["downloads"]["add_metadata"]:
-            postprocessors.append(
-                {
-                    "key": "FFmpegMetadata",
-                    "add_chapters": True,
-                    "add_metadata": True,
-                }
-            )
-
-        if self.config["downloads"]["add_thumbnail"]:
-            postprocessors.append(
-                {
-                    "key": "EmbedThumbnail",
-                    "already_have_thumbnail": True,
-                }
-            )
-            obs["writethumbnail"] = True
-
-        obs["postprocessors"] = postprocessors
-
-        return obs
-
-    def dl_single_vid(self, youtube_id):
-        """download single video"""
-        dl_cache = self.config["application"]["cache_dir"] + "/download/"
-        obs = self.build_obs()
-
-        # check if already in cache to continue from there
-        all_cached = ignore_filelist(os.listdir(dl_cache))
-        for file_name in all_cached:
-            if youtube_id in file_name:
-                obs["outtmpl"] = os.path.join(dl_cache, file_name)
-        with yt_dlp.YoutubeDL(obs) as ydl:
-            try:
-                ydl.download([youtube_id])
-            except yt_dlp.utils.DownloadError:
-                print("retry failed download: " + youtube_id)
-                sleep(10)
-                ydl.download([youtube_id])
-
-        if obs["writethumbnail"]:
-            # webp files don't get cleaned up automatically
-            all_cached = ignore_filelist(os.listdir(dl_cache))
-            to_clean = [i for i in all_cached if not i.endswith(".mp4")]
-            for file_name in to_clean:
-                file_path = os.path.join(dl_cache, file_name)
-                os.remove(file_path)
-
-    def move_to_archive(self, vid_dict):
-        """move downloaded video from cache to archive"""
-        videos = self.config["application"]["videos"]
-        host_uid = self.config["application"]["HOST_UID"]
-        host_gid = self.config["application"]["HOST_GID"]
-        channel_name = clean_string(vid_dict["channel"]["channel_name"])
-        # make archive folder with correct permissions
-        new_folder = os.path.join(videos, channel_name)
-        if not os.path.exists(new_folder):
-            os.makedirs(new_folder)
-            if host_uid and host_gid:
-                os.chown(new_folder, host_uid, host_gid)
-        # find real filename
-        cache_dir = self.config["application"]["cache_dir"]
-        all_cached = ignore_filelist(os.listdir(cache_dir + "/download/"))
-        for file_str in all_cached:
-            if vid_dict["youtube_id"] in file_str:
-                old_file = file_str
-        old_file_path = os.path.join(cache_dir, "download", old_file)
-        new_file_path = os.path.join(videos, vid_dict["media_url"])
-        # move media file and fix permission
-        shutil.move(old_file_path, new_file_path)
-        if host_uid and host_gid:
-            os.chown(new_file_path, host_uid, host_gid)
-
-    def delete_from_pending(self, youtube_id):
-        """delete downloaded video from pending index if its there"""
-        es_url = self.config["application"]["es_url"]
-        es_auth = self.config["application"]["es_auth"]
-        url = f"{es_url}/ta_download/_doc/{youtube_id}"
-        response = requests.delete(url, auth=es_auth)
-        if not response.ok and not response.status_code == 404:
-            print(response.text)
-
-    def add_subscribed_channels(self):
-        """add all channels subscribed to refresh"""
-        all_subscribed = PlaylistSubscription().get_playlists()
-        if not all_subscribed:
-            return
-
-        channel_ids = [i["playlist_channel_id"] for i in all_subscribed]
-        for channel_id in channel_ids:
-            self.channels.add(channel_id)
-
-        return
-
-    def validate_playlists(self):
-        """look for playlist needing to update"""
-        print("sync playlists")
-        self.add_subscribed_channels()
-        all_indexed = PendingList().get_all_indexed()
-        all_youtube_ids = [i["youtube_id"] for i in all_indexed]
-        for id_c, channel_id in enumerate(self.channels):
-            playlists = YoutubeChannel(channel_id).get_indexed_playlists()
-            all_playlist_ids = [i["playlist_id"] for i in playlists]
-            for id_p, playlist_id in enumerate(all_playlist_ids):
-                playlist_handler = YoutubePlaylist(
-                    playlist_id, all_youtube_ids=all_youtube_ids
-                )
-                playlist_dict = playlist_handler.update_playlist()
-                if not playlist_dict:
-                    playlist_handler.deactivate()
-                    continue
-
-                playlist_handler.add_vids_to_playlist()
-                # notify
-                title = (
-                    "Processing playlists for channels: "
-                    + f"{id_c + 1}/{len(self.channels)}"
-                )
-                message = f"Progress: {id_p + 1}/{len(all_playlist_ids)}"
-                mess_dict = {
-                    "status": "message:download",
-                    "level": "info",
-                    "title": title,
-                    "message": message,
-                }
-                if id_p + 1 == len(all_playlist_ids):
-                    RedisArchivist().set_message(
-                        "message:download", mess_dict, expire=4
-                    )
-                else:
-                    RedisArchivist().set_message("message:download", mess_dict)
-
-    @staticmethod
-    def auto_delete_watched(autodelete_days):
-        """delete watched videos after x days"""
-        now = int(datetime.now().strftime("%s"))
-        now_lte = now - autodelete_days * 24 * 60 * 60
-        data = {
-            "query": {"range": {"player.watched_date": {"lte": now_lte}}},
-            "sort": [{"player.watched_date": {"order": "asc"}}],
-        }
-        all_to_delete = IndexPaginate("ta_video", data).get_results()
-        all_youtube_ids = [i["youtube_id"] for i in all_to_delete]
-        if not all_youtube_ids:
-            return
-
-        for youtube_id in all_youtube_ids:
-            print(f"autodelete {youtube_id}")
-            YoutubeVideo(youtube_id).delete_media_file()
-
-        print("add deleted to ignore list")
-        pending_handler = PendingList()
-        pending_handler.add_to_pending(all_youtube_ids, ignore=True)

tubearchivist/home/src/download/queue.py

@@ -0,0 +1,263 @@
+"""
+Functionality:
+- handle download queue
+- linked with ta_dowload index
+"""
+
+import json
+import os
+from datetime import datetime
+
+import requests
+import yt_dlp
+from home.src.download.subscriptions import ChannelSubscription
+from home.src.es.connect import IndexPaginate
+from home.src.index.playlist import YoutubePlaylist
+from home.src.ta.config import AppConfig
+from home.src.ta.helper import DurationConverter, ignore_filelist
+from home.src.ta.ta_redis import RedisArchivist
+
+
+class PendingList:
+    """manage the pending videos list"""
+
+    CONFIG = AppConfig().config
+    ES_URL = CONFIG["application"]["es_url"]
+    ES_AUTH = CONFIG["application"]["es_auth"]
+    VIDEOS = CONFIG["application"]["videos"]
+
+    def __init__(self):
+        self.all_channel_ids = False
+        self.all_downloaded = False
+        self.missing_from_playlists = []
+
+    def parse_url_list(self, youtube_ids):
+        """extract youtube ids from list"""
+        missing_videos = []
+        for entry in youtube_ids:
+            # notify
+            mess_dict = {
+                "status": "message:add",
+                "level": "info",
+                "title": "Adding to download queue.",
+                "message": "Extracting lists",
+            }
+            RedisArchivist().set_message("message:add", mess_dict)
+            # extract
+            url = entry["url"]
+            url_type = entry["type"]
+            if url_type == "video":
+                missing_videos.append(url)
+            elif url_type == "channel":
+                video_results = ChannelSubscription().get_last_youtube_videos(
+                    url, limit=False
+                )
+                youtube_ids = [i[0] for i in video_results]
+                missing_videos = missing_videos + youtube_ids
+            elif url_type == "playlist":
+                self.missing_from_playlists.append(entry)
+                playlist = YoutubePlaylist(url)
+                playlist.build_json()
+                video_results = playlist.json_data.get("playlist_entries")
+                youtube_ids = [i["youtube_id"] for i in video_results]
+                missing_videos = missing_videos + youtube_ids
+
+        return missing_videos
+
+    def add_to_pending(self, missing_videos, ignore=False):
+        """build the bulk json data from pending"""
+        # check if channel is indexed
+        channel_handler = ChannelSubscription()
+        all_indexed = channel_handler.get_channels(subscribed_only=False)
+        self.all_channel_ids = [i["channel_id"] for i in all_indexed]
+        # check if already there
+        self.all_downloaded = self.get_all_downloaded()
+
+        bulk_list, all_videos_added = self.build_bulk(missing_videos, ignore)
+        # add last newline
+        bulk_list.append("\n")
+        query_str = "\n".join(bulk_list)
+        headers = {"Content-type": "application/x-ndjson"}
+        url = self.ES_URL + "/_bulk"
+        request = requests.post(
+            url, data=query_str, headers=headers, auth=self.ES_AUTH
+        )
+        if not request.ok:
+            print(request)
+            raise ValueError("failed to add video to download queue")
+
+        return all_videos_added
+
+    def build_bulk(self, missing_videos, ignore=False):
+        """build the bulk lists"""
+        bulk_list = []
+        all_videos_added = []
+
+        for idx, youtube_id in enumerate(missing_videos):
+            # check if already downloaded
+            if youtube_id in self.all_downloaded:
+                continue
+
+            video = self.get_youtube_details(youtube_id)
+            # skip on download error
+            if not video:
+                continue
+
+            channel_indexed = video["channel_id"] in self.all_channel_ids
+            video["channel_indexed"] = channel_indexed
+            if ignore:
+                video["status"] = "ignore"
+            else:
+                video["status"] = "pending"
+            action = {"create": {"_id": youtube_id, "_index": "ta_download"}}
+            bulk_list.append(json.dumps(action))
+            bulk_list.append(json.dumps(video))
+            all_videos_added.append((youtube_id, video["vid_thumb_url"]))
+            # notify
+            progress = f"{idx + 1}/{len(missing_videos)}"
+            mess_dict = {
+                "status": "message:add",
+                "level": "info",
+                "title": "Adding new videos to download queue.",
+                "message": "Progress: " + progress,
+            }
+            if idx + 1 == len(missing_videos):
+                RedisArchivist().set_message(
+                    "message:add", mess_dict, expire=4
+                )
+            else:
+                RedisArchivist().set_message("message:add", mess_dict)
+            if idx + 1 % 25 == 0:
+                print("adding to queue progress: " + progress)
+
+        return bulk_list, all_videos_added
+
+    @staticmethod
+    def get_youtube_details(youtube_id):
+        """get details from youtubedl for single pending video"""
+        obs = {
+            "default_search": "ytsearch",
+            "quiet": True,
+            "check_formats": "selected",
+            "noplaylist": True,
+            "writethumbnail": True,
+            "simulate": True,
+        }
+        try:
+            vid = yt_dlp.YoutubeDL(obs).extract_info(youtube_id)
+        except yt_dlp.utils.DownloadError:
+            print("failed to extract info for: " + youtube_id)
+            return False
+        # stop if video is streaming live now
+        if vid["is_live"]:
+            return False
+        # parse response
+        seconds = vid["duration"]
+        duration_str = DurationConverter.get_str(seconds)
+        if duration_str == "NA":
+            print(f"skip extracting duration for: {youtube_id}")
+        upload_date = vid["upload_date"]
+        upload_dt = datetime.strptime(upload_date, "%Y%m%d")
+        published = upload_dt.strftime("%Y-%m-%d")
+        # build dict
+        youtube_details = {
+            "youtube_id": youtube_id,
+            "channel_name": vid["channel"],
+            "vid_thumb_url": vid["thumbnail"],
+            "title": vid["title"],
+            "channel_id": vid["channel_id"],
+            "duration": duration_str,
+            "published": published,
+            "timestamp": int(datetime.now().strftime("%s")),
+        }
+        return youtube_details
+
+    @staticmethod
+    def get_all_pending():
+        """get a list of all pending videos in ta_download"""
+        data = {
+            "query": {"match_all": {}},
+            "sort": [{"timestamp": {"order": "asc"}}],
+        }
+        all_results = IndexPaginate("ta_download", data).get_results()
+
+        all_pending = []
+        all_ignore = []
+
+        for result in all_results:
+            if result["status"] == "pending":
+                all_pending.append(result)
+            elif result["status"] == "ignore":
+                all_ignore.append(result)
+
+        return all_pending, all_ignore
+
+    @staticmethod
+    def get_all_indexed():
+        """get a list of all videos indexed"""
+
+        data = {
+            "query": {"match_all": {}},
+            "sort": [{"published": {"order": "desc"}}],
+        }
+        all_indexed = IndexPaginate("ta_video", data).get_results()
+
+        return all_indexed
+
+    def get_all_downloaded(self):
+        """get a list of all videos in archive"""
+        channel_folders = os.listdir(self.VIDEOS)
+        all_channel_folders = ignore_filelist(channel_folders)
+        all_downloaded = []
+        for channel_folder in all_channel_folders:
+            channel_path = os.path.join(self.VIDEOS, channel_folder)
+            videos = os.listdir(channel_path)
+            all_videos = ignore_filelist(videos)
+            youtube_vids = [i[9:20] for i in all_videos]
+            for youtube_id in youtube_vids:
+                all_downloaded.append(youtube_id)
+        return all_downloaded
+
+    def delete_from_pending(self, youtube_id):
+        """delete the youtube_id from ta_download"""
+        url = f"{self.ES_URL}/ta_download/_doc/{youtube_id}"
+        response = requests.delete(url, auth=self.ES_AUTH)
+        if not response.ok:
+            print(response.text)
+
+    def delete_pending(self, status):
+        """delete download queue based on status value"""
+        data = {"query": {"term": {"status": {"value": status}}}}
+        payload = json.dumps(data)
+        url = self.ES_URL + "/ta_download/_delete_by_query"
+        headers = {"Content-type": "application/json"}
+        response = requests.post(
+            url, data=payload, headers=headers, auth=self.ES_AUTH
+        )
+        if not response.ok:
+            print(response.text)
+
+    def ignore_from_pending(self, ignore_list):
+        """build the bulk query string"""
+
+        stamp = int(datetime.now().strftime("%s"))
+        bulk_list = []
+
+        for youtube_id in ignore_list:
+            action = {"update": {"_id": youtube_id, "_index": "ta_download"}}
+            source = {"doc": {"status": "ignore", "timestamp": stamp}}
+            bulk_list.append(json.dumps(action))
+            bulk_list.append(json.dumps(source))
+
+        # add last newline
+        bulk_list.append("\n")
+        query_str = "\n".join(bulk_list)
+
+        headers = {"Content-type": "application/x-ndjson"}
+        url = self.ES_URL + "/_bulk"
+        request = requests.post(
+            url, data=query_str, headers=headers, auth=self.ES_AUTH
+        )
+        if not request.ok:
+            print(request)
+            raise ValueError("failed to set video to ignore")

tubearchivist/home/src/download/subscriptions.py

@@ -0,0 +1,222 @@
+"""
+Functionality:
+- handle channel subscriptions
+- handle playlist subscriptions
+"""
+
+import yt_dlp
+from home.src.download import queue  # partial import
+from home.src.es.connect import IndexPaginate
+from home.src.index.channel import YoutubeChannel
+from home.src.index.playlist import YoutubePlaylist
+from home.src.ta.config import AppConfig
+from home.src.ta.ta_redis import RedisArchivist
+
+
+class ChannelSubscription:
+    """manage the list of channels subscribed"""
+
+    def __init__(self):
+        config = AppConfig().config
+        self.es_url = config["application"]["es_url"]
+        self.es_auth = config["application"]["es_auth"]
+        self.channel_size = config["subscriptions"]["channel_size"]
+
+    @staticmethod
+    def get_channels(subscribed_only=True):
+        """get a list of all channels subscribed to"""
+        data = {
+            "sort": [{"channel_name.keyword": {"order": "asc"}}],
+        }
+        if subscribed_only:
+            data["query"] = {"term": {"channel_subscribed": {"value": True}}}
+        else:
+            data["query"] = {"match_all": {}}
+
+        all_channels = IndexPaginate("ta_channel", data).get_results()
+
+        return all_channels
+
+    def get_last_youtube_videos(self, channel_id, limit=True):
+        """get a list of last videos from channel"""
+        url = f"https://www.youtube.com/channel/{channel_id}/videos"
+        obs = {
+            "default_search": "ytsearch",
+            "quiet": True,
+            "skip_download": True,
+            "extract_flat": True,
+        }
+        if limit:
+            obs["playlistend"] = self.channel_size
+
+        try:
+            chan = yt_dlp.YoutubeDL(obs).extract_info(url, download=False)
+        except yt_dlp.utils.DownloadError:
+            print(f"{channel_id}: failed to extract videos, skipping.")
+            return False
+
+        last_videos = [(i["id"], i["title"]) for i in chan["entries"]]
+        return last_videos
+
+    def find_missing(self):
+        """add missing videos from subscribed channels to pending"""
+        all_channels = self.get_channels()
+        pending_handler = queue.PendingList()
+        all_pending, all_ignore = pending_handler.get_all_pending()
+        all_ids = [i["youtube_id"] for i in all_ignore + all_pending]
+        all_downloaded = pending_handler.get_all_downloaded()
+        to_ignore = all_ids + all_downloaded
+
+        missing_videos = []
+
+        for idx, channel in enumerate(all_channels):
+            channel_id = channel["channel_id"]
+            last_videos = self.get_last_youtube_videos(channel_id)
+
+            if last_videos:
+                for video in last_videos:
+                    if video[0] not in to_ignore:
+                        missing_videos.append(video[0])
+            # notify
+            message = {
+                "status": "message:rescan",
+                "level": "info",
+                "title": "Scanning channels: Looking for new videos.",
+                "message": f"Progress: {idx + 1}/{len(all_channels)}",
+            }
+            if idx + 1 == len(all_channels):
+                RedisArchivist().set_message(
+                    "message:rescan", message=message, expire=4
+                )
+            else:
+                RedisArchivist().set_message("message:rescan", message=message)
+
+        return missing_videos
+
+    @staticmethod
+    def change_subscribe(channel_id, channel_subscribed):
+        """subscribe or unsubscribe from channel and update"""
+        channel = YoutubeChannel(channel_id)
+        channel.build_json()
+        channel.json_data["channel_subscribed"] = channel_subscribed
+        channel.upload_to_es()
+        channel.sync_to_videos()
+
+
+class PlaylistSubscription:
+    """manage the playlist download functionality"""
+
+    def __init__(self):
+        self.config = AppConfig().config
+
+    @staticmethod
+    def get_playlists(subscribed_only=True):
+        """get a list of all active playlists"""
+        data = {
+            "sort": [{"playlist_channel.keyword": {"order": "desc"}}],
+        }
+        data["query"] = {
+            "bool": {"must": [{"term": {"playlist_active": {"value": True}}}]}
+        }
+        if subscribed_only:
+            data["query"]["bool"]["must"].append(
+                {"term": {"playlist_subscribed": {"value": True}}}
+            )
+
+        all_playlists = IndexPaginate("ta_playlist", data).get_results()
+
+        return all_playlists
+
+    def process_url_str(self, new_playlists, subscribed=True):
+        """process playlist subscribe form url_str"""
+        all_indexed = queue.PendingList().get_all_indexed()
+        all_youtube_ids = [i["youtube_id"] for i in all_indexed]
+
+        new_thumbs = []
+        for idx, playlist in enumerate(new_playlists):
+            url_type = playlist["type"]
+            playlist_id = playlist["url"]
+            if not url_type == "playlist":
+                print(f"{playlist_id} not a playlist, skipping...")
+                continue
+
+            playlist_h = YoutubePlaylist(playlist_id)
+            playlist_h.all_youtube_ids = all_youtube_ids
+            playlist_h.build_json()
+            playlist_h.json_data["playlist_subscribed"] = subscribed
+            playlist_h.upload_to_es()
+            playlist_h.add_vids_to_playlist()
+            self.channel_validate(playlist_h.json_data["playlist_channel_id"])
+            thumb = playlist_h.json_data["playlist_thumbnail"]
+            new_thumbs.append((playlist_id, thumb))
+            # notify
+            message = {
+                "status": "message:subplaylist",
+                "level": "info",
+                "title": "Subscribing to Playlists",
+                "message": f"Processing {idx + 1} of {len(new_playlists)}",
+            }
+            RedisArchivist().set_message(
+                "message:subplaylist", message=message
+            )
+
+        return new_thumbs
+
+    @staticmethod
+    def channel_validate(channel_id):
+        """make sure channel of playlist is there"""
+        channel = YoutubeChannel(channel_id)
+        channel.build_json()
+
+    @staticmethod
+    def change_subscribe(playlist_id, subscribe_status):
+        """change the subscribe status of a playlist"""
+        playlist = YoutubePlaylist(playlist_id)
+        playlist.build_json()
+        playlist.json_data["playlist_subscribed"] = subscribe_status
+        playlist.upload_to_es()
+
+    @staticmethod
+    def get_to_ignore():
+        """get all youtube_ids already downloaded or ignored"""
+        pending_handler = queue.PendingList()
+        all_pending, all_ignore = pending_handler.get_all_pending()
+        all_ids = [i["youtube_id"] for i in all_ignore + all_pending]
+        all_downloaded = pending_handler.get_all_downloaded()
+        to_ignore = all_ids + all_downloaded
+        return to_ignore
+
+    def find_missing(self):
+        """find videos in subscribed playlists not downloaded yet"""
+        all_playlists = [i["playlist_id"] for i in self.get_playlists()]
+        to_ignore = self.get_to_ignore()
+
+        missing_videos = []
+        for idx, playlist_id in enumerate(all_playlists):
+            size_limit = self.config["subscriptions"]["channel_size"]
+            playlist = YoutubePlaylist(playlist_id)
+            playlist.update_playlist()
+            if not playlist:
+                playlist.deactivate()
+                continue
+
+            playlist_entries = playlist.json_data["playlist_entries"]
+            if size_limit:
+                del playlist_entries[size_limit:]
+
+            all_missing = [i for i in playlist_entries if not i["downloaded"]]
+
+            message = {
+                "status": "message:rescan",
+                "level": "info",
+                "title": "Scanning playlists: Looking for new videos.",
+                "message": f"Progress: {idx + 1}/{len(all_playlists)}",
+            }
+            RedisArchivist().set_message("message:rescan", message=message)
+
+            for video in all_missing:
+                youtube_id = video["youtube_id"]
+                if youtube_id not in to_ignore:
+                    missing_videos.append(youtube_id)
+
+        return missing_videos

tubearchivist/home/src/download/thumbnails.py

@@ -1,16 +1,19 @@
 """
 functionality:
 - handle download and caching for thumbnails
+- check for missing thumbnails
 """
 
 import os
 from collections import Counter
 from time import sleep
 
-import home.src.download as download
 import requests
-from home.src.config import AppConfig
-from home.src.helper import RedisArchivist, ignore_filelist
+from home.src.download import queue  # partial import
+from home.src.download import subscriptions  # partial import
+from home.src.ta.config import AppConfig
+from home.src.ta.helper import ignore_filelist
+from home.src.ta.ta_redis import RedisArchivist
 from mutagen.mp4 import MP4, MP4Cover
 from PIL import Image
 
@@ -55,8 +58,8 @@ class ThumbManager:
     def get_needed_thumbs(self, missing_only=False):
         """get a list of all missing thumbnails"""
         all_thumbs = self.get_all_thumbs()
-        all_indexed = download.PendingList().get_all_indexed()
-        all_in_queue, all_ignored = download.PendingList().get_all_pending()
+        all_indexed = queue.PendingList().get_all_indexed()
+        all_in_queue, all_ignored = queue.PendingList().get_all_pending()
 
         needed_thumbs = []
         for video in all_indexed:
@@ -84,9 +87,8 @@ class ThumbManager:
         all_channel_art = os.listdir(self.CHANNEL_DIR)
         files = [i[0:24] for i in all_channel_art]
         cached_channel_ids = [k for (k, v) in Counter(files).items() if v > 1]
-        channels = download.ChannelSubscription().get_channels(
-            subscribed_only=False
-        )
+        channel_sub = subscriptions.ChannelSubscription()
+        channels = channel_sub.get_channels(subscribed_only=False)
 
         missing_channels = []
         for channel in channels:
@@ -104,10 +106,8 @@ class ThumbManager:
         """get all missing playlist artwork"""
         all_downloaded = ignore_filelist(os.listdir(self.PLAYLIST_DIR))
         all_ids_downloaded = [i.replace(".jpg", "") for i in all_downloaded]
-
-        playlists = download.PlaylistSubscription().get_playlists(
-            subscribed_only=False
-        )
+        playlist_sub = subscriptions.PlaylistSubscription()
+        playlists = playlist_sub.get_playlists(subscribed_only=False)
 
         missing_playlists = []
         for playlist in playlists:
@@ -276,7 +276,7 @@ class ThumbManager:
 
     def get_thumb_list(self):
         """get list of mediafiles and matching thumbnails"""
-        all_indexed = download.PendingList().get_all_indexed()
+        all_indexed = queue.PendingList().get_all_indexed()
         video_list = []
         for video in all_indexed:
             youtube_id = video["youtube_id"]

tubearchivist/home/src/download/yt_dlp_handler.py

@@ -0,0 +1,316 @@
+"""
+functionality:
+- handle yt_dlp
+- build options and post processor
+- download video files
+- move to archive
+"""
+
+import os
+import shutil
+from datetime import datetime
+from time import sleep
+
+import requests
+import yt_dlp
+from home.src.download.queue import PendingList
+from home.src.download.subscriptions import PlaylistSubscription
+from home.src.es.connect import IndexPaginate
+from home.src.index.channel import YoutubeChannel
+from home.src.index.playlist import YoutubePlaylist
+from home.src.index.video import YoutubeVideo, index_new_video
+from home.src.ta.config import AppConfig
+from home.src.ta.helper import clean_string, ignore_filelist
+from home.src.ta.ta_redis import RedisArchivist, RedisQueue
+
+
+class VideoDownloader:
+    """
+    handle the video download functionality
+    if not initiated with list, take from queue
+    """
+
+    def __init__(self, youtube_id_list=False):
+        self.obs = False
+        self.youtube_id_list = youtube_id_list
+        self.config = AppConfig().config
+        self._build_obs()
+        self.channels = set()
+
+    def run_queue(self):
+        """setup download queue in redis loop until no more items"""
+        queue = RedisQueue("dl_queue")
+
+        limit_queue = self.config["downloads"]["limit_count"]
+        if limit_queue:
+            queue.trim(limit_queue - 1)
+
+        while True:
+            youtube_id = queue.get_next()
+            if not youtube_id:
+                break
+
+            try:
+                self._dl_single_vid(youtube_id)
+            except yt_dlp.utils.DownloadError:
+                print("failed to download " + youtube_id)
+                continue
+            vid_dict = index_new_video(youtube_id)
+            self.channels.add(vid_dict["channel"]["channel_id"])
+            self.move_to_archive(vid_dict)
+            self._delete_from_pending(youtube_id)
+
+        autodelete_days = self.config["downloads"]["autodelete_days"]
+        if autodelete_days:
+            print(f"auto delete older than {autodelete_days} days")
+            self.auto_delete_watched(autodelete_days)
+
+    @staticmethod
+    def add_pending():
+        """add pending videos to download queue"""
+        mess_dict = {
+            "status": "message:download",
+            "level": "info",
+            "title": "Looking for videos to download",
+            "message": "Scanning your download queue.",
+        }
+        RedisArchivist().set_message("message:download", mess_dict)
+        all_pending, _ = PendingList().get_all_pending()
+        to_add = [i["youtube_id"] for i in all_pending]
+        if not to_add:
+            # there is nothing pending
+            print("download queue is empty")
+            mess_dict = {
+                "status": "message:download",
+                "level": "error",
+                "title": "Download queue is empty",
+                "message": "Add some videos to the queue first.",
+            }
+            RedisArchivist().set_message("message:download", mess_dict)
+            return
+
+        queue = RedisQueue("dl_queue")
+        queue.add_list(to_add)
+
+    @staticmethod
+    def _progress_hook(response):
+        """process the progress_hooks from yt_dlp"""
+        # title
+        path = os.path.split(response["filename"])[-1][12:]
+        filename = os.path.splitext(os.path.splitext(path)[0])[0]
+        filename_clean = filename.replace("_", " ")
+        title = "Downloading: " + filename_clean
+        # message
+        try:
+            percent = response["_percent_str"]
+            size = response["_total_bytes_str"]
+            speed = response["_speed_str"]
+            eta = response["_eta_str"]
+            message = f"{percent} of {size} at {speed} - time left: {eta}"
+        except KeyError:
+            message = "processing"
+        mess_dict = {
+            "status": "message:download",
+            "level": "info",
+            "title": title,
+            "message": message,
+        }
+        RedisArchivist().set_message("message:download", mess_dict)
+
+    def _build_obs(self):
+        """collection to build all obs passed to yt-dlp"""
+        self._build_obs_basic()
+        self._build_obs_user()
+        self._build_obs_postprocessors()
+
+    def _build_obs_basic(self):
+        """initial obs"""
+        self.obs = {
+            "default_search": "ytsearch",
+            "merge_output_format": "mp4",
+            "restrictfilenames": True,
+            "outtmpl": (
+                self.config["application"]["cache_dir"]
+                + "/download/"
+                + self.config["application"]["file_template"]
+            ),
+            "progress_hooks": [self._progress_hook],
+            "noprogress": True,
+            "quiet": True,
+            "continuedl": True,
+            "retries": 3,
+            "writethumbnail": False,
+            "noplaylist": True,
+            "check_formats": "selected",
+        }
+
+    def _build_obs_user(self):
+        """build user customized options"""
+        if self.config["downloads"]["format"]:
+            self.obs["format"] = self.config["downloads"]["format"]
+        if self.config["downloads"]["limit_speed"]:
+            self.obs["ratelimit"] = (
+                self.config["downloads"]["limit_speed"] * 1024
+            )
+
+        throttle = self.config["downloads"]["throttledratelimit"]
+        if throttle:
+            self.obs["throttledratelimit"] = throttle * 1024
+
+    def _build_obs_postprocessors(self):
+        """add postprocessor to obs"""
+        postprocessors = []
+
+        if self.config["downloads"]["add_metadata"]:
+            postprocessors.append(
+                {
+                    "key": "FFmpegMetadata",
+                    "add_chapters": True,
+                    "add_metadata": True,
+                }
+            )
+
+        if self.config["downloads"]["add_thumbnail"]:
+            postprocessors.append(
+                {
+                    "key": "EmbedThumbnail",
+                    "already_have_thumbnail": True,
+                }
+            )
+            self.obs["writethumbnail"] = True
+
+        self.obs["postprocessors"] = postprocessors
+
+    def _dl_single_vid(self, youtube_id):
+        """download single video"""
+        dl_cache = self.config["application"]["cache_dir"] + "/download/"
+
+        # check if already in cache to continue from there
+        all_cached = ignore_filelist(os.listdir(dl_cache))
+        for file_name in all_cached:
+            if youtube_id in file_name:
+                self.obs["outtmpl"] = os.path.join(dl_cache, file_name)
+        with yt_dlp.YoutubeDL(self.obs) as ydl:
+            try:
+                ydl.download([youtube_id])
+            except yt_dlp.utils.DownloadError:
+                print("retry failed download: " + youtube_id)
+                sleep(10)
+                ydl.download([youtube_id])
+
+        if self.obs["writethumbnail"]:
+            # webp files don't get cleaned up automatically
+            all_cached = ignore_filelist(os.listdir(dl_cache))
+            to_clean = [i for i in all_cached if not i.endswith(".mp4")]
+            for file_name in to_clean:
+                file_path = os.path.join(dl_cache, file_name)
+                os.remove(file_path)
+
+    def move_to_archive(self, vid_dict):
+        """move downloaded video from cache to archive"""
+        videos = self.config["application"]["videos"]
+        host_uid = self.config["application"]["HOST_UID"]
+        host_gid = self.config["application"]["HOST_GID"]
+        channel_name = clean_string(vid_dict["channel"]["channel_name"])
+        if len(channel_name) <= 3:
+            # fall back to channel id
+            channel_name = vid_dict["channel"]["channel_id"]
+        # make archive folder with correct permissions
+        new_folder = os.path.join(videos, channel_name)
+        if not os.path.exists(new_folder):
+            os.makedirs(new_folder)
+            if host_uid and host_gid:
+                os.chown(new_folder, host_uid, host_gid)
+        # find real filename
+        cache_dir = self.config["application"]["cache_dir"]
+        all_cached = ignore_filelist(os.listdir(cache_dir + "/download/"))
+        for file_str in all_cached:
+            if vid_dict["youtube_id"] in file_str:
+                old_file = file_str
+        old_file_path = os.path.join(cache_dir, "download", old_file)
+        new_file_path = os.path.join(videos, vid_dict["media_url"])
+        # move media file and fix permission
+        shutil.move(old_file_path, new_file_path)
+        if host_uid and host_gid:
+            os.chown(new_file_path, host_uid, host_gid)
+
+    def _delete_from_pending(self, youtube_id):
+        """delete downloaded video from pending index if its there"""
+        es_url = self.config["application"]["es_url"]
+        es_auth = self.config["application"]["es_auth"]
+        url = f"{es_url}/ta_download/_doc/{youtube_id}"
+        response = requests.delete(url, auth=es_auth)
+        if not response.ok and not response.status_code == 404:
+            print(response.text)
+
+    def _add_subscribed_channels(self):
+        """add all channels subscribed to refresh"""
+        all_subscribed = PlaylistSubscription().get_playlists()
+        if not all_subscribed:
+            return
+
+        channel_ids = [i["playlist_channel_id"] for i in all_subscribed]
+        for channel_id in channel_ids:
+            self.channels.add(channel_id)
+
+        return
+
+    def validate_playlists(self):
+        """look for playlist needing to update"""
+        print("sync playlists")
+        self._add_subscribed_channels()
+        all_indexed = PendingList().get_all_indexed()
+        all_youtube_ids = [i["youtube_id"] for i in all_indexed]
+        for id_c, channel_id in enumerate(self.channels):
+            playlists = YoutubeChannel(channel_id).get_indexed_playlists()
+            all_playlist_ids = [i["playlist_id"] for i in playlists]
+            for id_p, playlist_id in enumerate(all_playlist_ids):
+                playlist = YoutubePlaylist(playlist_id)
+                playlist.all_youtube_ids = all_youtube_ids
+                playlist.build_json(scrape=True)
+                if not playlist.json_data:
+                    playlist.deactivate()
+
+                playlist.add_vids_to_playlist()
+                playlist.upload_to_es()
+
+                # notify
+                title = (
+                    "Processing playlists for channels: "
+                    + f"{id_c + 1}/{len(self.channels)}"
+                )
+                message = f"Progress: {id_p + 1}/{len(all_playlist_ids)}"
+                mess_dict = {
+                    "status": "message:download",
+                    "level": "info",
+                    "title": title,
+                    "message": message,
+                }
+                if id_p + 1 == len(all_playlist_ids):
+                    RedisArchivist().set_message(
+                        "message:download", mess_dict, expire=4
+                    )
+                else:
+                    RedisArchivist().set_message("message:download", mess_dict)
+
+    @staticmethod
+    def auto_delete_watched(autodelete_days):
+        """delete watched videos after x days"""
+        now = int(datetime.now().strftime("%s"))
+        now_lte = now - autodelete_days * 24 * 60 * 60
+        data = {
+            "query": {"range": {"player.watched_date": {"lte": now_lte}}},
+            "sort": [{"player.watched_date": {"order": "asc"}}],
+        }
+        all_to_delete = IndexPaginate("ta_video", data).get_results()
+        all_youtube_ids = [i["youtube_id"] for i in all_to_delete]
+        if not all_youtube_ids:
+            return
+
+        for youtube_id in all_youtube_ids:
+            print(f"autodelete {youtube_id}")
+            YoutubeVideo(youtube_id).delete_media_file()
+
+        print("add deleted to ignore list")
+        pending_handler = PendingList()
+        pending_handler.add_to_pending(all_youtube_ids, ignore=True)

tubearchivist/home/src/es/connect.py

@@ -0,0 +1,148 @@
+"""
+functionality:
+- wrapper around requests to call elastic search
+- reusable search_after to extract total index
+"""
+
+import json
+
+import requests
+from home.src.ta.config import AppConfig
+
+
+class ElasticWrap:
+    """makes all calls to elastic search
+    returns response json and status code tuple
+    """
+
+    def __init__(self, path, config=False):
+        self.url = False
+        self.auth = False
+        self.path = path
+        self.config = config
+        self._get_config()
+
+    def _get_config(self):
+        """add config if not passed"""
+        if not self.config:
+            self.config = AppConfig().config
+
+        es_url = self.config["application"]["es_url"]
+        self.auth = self.config["application"]["es_auth"]
+        self.url = f"{es_url}/{self.path}"
+
+    def get(self, data=False):
+        """get data from es"""
+        if data:
+            response = requests.get(self.url, json=data, auth=self.auth)
+        else:
+            response = requests.get(self.url, auth=self.auth)
+        if not response.ok:
+            print(response.text)
+
+        return response.json(), response.status_code
+
+    def post(self, data=False, ndjson=False):
+        """post data to es"""
+        if ndjson:
+            headers = {"Content-type": "application/x-ndjson"}
+            payload = data
+        else:
+            headers = {"Content-type": "application/json"}
+            payload = json.dumps(data)
+
+        if data:
+            response = requests.post(
+                self.url, data=payload, headers=headers, auth=self.auth
+            )
+        else:
+            response = requests.post(self.url, headers=headers, auth=self.auth)
+
+        if not response.ok:
+            print(response.text)
+
+        return response.json(), response.status_code
+
+    def put(self, data, refresh=False):
+        """put data to es"""
+        if refresh:
+            self.url = f"{self.url}/?refresh=true"
+        response = requests.put(f"{self.url}", json=data, auth=self.auth)
+        if not response.ok:
+            print(response.text)
+            print(data)
+            raise ValueError("failed to add item to index")
+
+        return response.json(), response.status_code
+
+    def delete(self, data=False):
+        """delete document from es"""
+        if data:
+            response = requests.delete(self.url, json=data, auth=self.auth)
+        else:
+            response = requests.delete(self.url, auth=self.auth)
+
+        if not response.ok:
+            print(response.text)
+
+        return response.json(), response.status_code
+
+
+class IndexPaginate:
+    """use search_after to go through whole index"""
+
+    DEFAULT_SIZE = 500
+
+    def __init__(self, index_name, data, size=False):
+        self.index_name = index_name
+        self.data = data
+        self.pit_id = False
+        self.size = size
+
+    def get_results(self):
+        """get all results"""
+        self.get_pit()
+        self.validate_data()
+        all_results = self.run_loop()
+        self.clean_pit()
+        return all_results
+
+    def get_pit(self):
+        """get pit for index"""
+        path = f"{self.index_name}/_pit?keep_alive=10m"
+        response, _ = ElasticWrap(path).post()
+        self.pit_id = response["id"]
+
+    def validate_data(self):
+        """add pit and size to data"""
+        if "sort" not in self.data.keys():
+            print(self.data)
+            raise ValueError("missing sort key in data")
+
+        size = self.size or self.DEFAULT_SIZE
+
+        self.data["size"] = size
+        self.data["pit"] = {"id": self.pit_id, "keep_alive": "10m"}
+
+    def run_loop(self):
+        """loop through results until last hit"""
+        all_results = []
+        while True:
+            response, _ = ElasticWrap("_search").get(data=self.data)
+            all_hits = response["hits"]["hits"]
+            if all_hits:
+                for hit in all_hits:
+                    source = hit["_source"]
+                    search_after = hit["sort"]
+                    all_results.append(source)
+                # update search_after with last hit data
+                self.data["search_after"] = search_after
+            else:
+                break
+
+        return all_results
+
+    def clean_pit(self):
+        """delete pit from elastic search"""
+        data = {"id": self.pit_id}
+        ElasticWrap("_pit").delete(data=data)

tubearchivist/home/src/es/index_mapping.json

@@ -0,0 +1,375 @@
+{
+    "index_config": [{
+            "index_name": "channel",
+            "expected_map": {
+                "channel_id": {
+                    "type": "keyword"
+                },
+                "channel_name": {
+                    "type": "text",
+                    "analyzer": "english",
+                    "fields": {
+                        "keyword": {
+                            "type": "keyword",
+                            "ignore_above": 256,
+                            "normalizer": "to_lower"
+                        },
+                        "search_as_you_type": {
+                            "type": "search_as_you_type",
+                            "doc_values": false,
+                            "max_shingle_size": 3
+                        }
+                    }
+                },
+                "channel_banner_url": {
+                    "type": "keyword",
+                    "index": false
+                },
+                "channel_tvart_url": {
+                    "type": "keyword",
+                    "index": false
+                },
+                "channel_thumb_url": {
+                    "type": "keyword",
+                    "index": false
+                },
+                "channel_description": {
+                    "type": "text"
+                },
+                "channel_last_refresh": {
+                    "type": "date",
+                    "format": "epoch_second"
+                }
+            },
+            "expected_set": {
+                "analysis": {
+                    "normalizer": {
+                        "to_lower": {
+                            "type": "custom",
+                            "filter": ["lowercase"]
+                        }
+                    }
+                },
+                "number_of_replicas": "0"
+            }
+        },
+        {
+            "index_name": "video",
+            "expected_map": {
+                "vid_thumb_url": {
+                    "type": "text",
+                    "index": false
+                },
+                "date_downloaded": {
+                    "type": "date"
+                },
+                "channel": {
+                    "properties": {
+                        "channel_id": {
+                            "type": "keyword"
+                        },
+                        "channel_name": {
+                            "type": "text",
+                            "analyzer": "english",
+                            "fields": {
+                                "keyword": {
+                                    "type": "keyword",
+                                    "ignore_above": 256,
+                                    "normalizer": "to_lower"
+                                },
+                                "search_as_you_type": {
+                                    "type": "search_as_you_type",
+                                    "doc_values": false,
+                                    "max_shingle_size": 3
+                                }
+                            }
+                        },
+                        "channel_banner_url": {
+                            "type": "keyword",
+                            "index": false
+                        },
+                        "channel_tvart_url": {
+                            "type": "keyword",
+                            "index": false
+                        },
+                        "channel_thumb_url": {
+                            "type": "keyword",
+                            "index": false
+                        },
+                        "channel_description": {
+                            "type": "text"
+                        },
+                        "channel_last_refresh": {
+                            "type": "date",
+                            "format": "epoch_second"
+                        }
+                    }
+                },
+                "description": {
+                    "type": "text"
+                },
+                "media_url": {
+                    "type": "keyword",
+                    "index": false
+                },
+                "tags": {
+                    "type": "text",
+                    "analyzer": "english",
+                    "fields": {
+                        "keyword": {
+                            "type": "keyword",
+                            "ignore_above": 256
+                        }
+                    }
+                },
+                "title": {
+                    "type": "text",
+                    "analyzer": "english",
+                    "fields": {
+                        "keyword": {
+                            "type": "keyword",
+                            "ignore_above": 256,
+                            "normalizer": "to_lower"
+                        },
+                        "search_as_you_type": {
+                            "type": "search_as_you_type",
+                            "doc_values": false,
+                            "max_shingle_size": 3
+                        }
+                    }
+                },
+                "vid_last_refresh": {
+                    "type": "date"
+                },
+                "youtube_id": {
+                    "type": "keyword"
+                },
+                "published": {
+                    "type": "date"
+                },
+                "playlist": {
+                    "type": "text",
+                    "fields": {
+                        "keyword": {
+                            "type": "keyword",
+                            "ignore_above": 256,
+                            "normalizer": "to_lower"
+                        }
+                    }
+                },
+                "subtitles": {
+                    "properties": {
+                        "ext": {
+                            "type": "keyword",
+                            "index": false
+                        },
+                        "lang": {
+                            "type": "keyword",
+                            "index": false
+                        },
+                        "media_url": {
+                            "type": "keyword",
+                            "index": false
+                        },
+                        "name": {
+                            "type": "keyword"
+                        },
+                        "source": {
+                            "type": "keyword"
+                        },
+                        "url": {
+                            "type": "keyword",
+                            "index": false
+                        }
+                    }
+                }
+            },
+            "expected_set": {
+                "analysis": {
+                    "normalizer": {
+                        "to_lower": {
+                            "type": "custom",
+                            "filter": ["lowercase"]
+                        }
+                    }
+                },
+                "number_of_replicas": "0"
+            }
+        },
+        {
+            "index_name": "download",
+            "expected_map": {
+                "timestamp": {
+                    "type": "date"
+                },
+                "channel_id": {
+                    "type": "keyword"
+                },
+                "channel_name": {
+                    "type": "text",
+                    "fields": {
+                        "keyword": {
+                            "type": "keyword",
+                            "ignore_above": 256,
+                            "normalizer": "to_lower"
+                        }
+                    }
+                },
+                "status": {
+                    "type": "keyword"
+                },
+                "title": {
+                    "type": "text",
+                    "fields": {
+                        "keyword": {
+                            "type": "keyword",
+                            "ignore_above": 256,
+                            "normalizer": "to_lower"
+                        }
+                    }
+                },
+                "vid_thumb_url": {
+                    "type": "keyword"
+                },
+                "youtube_id": {
+                    "type": "keyword"
+                }
+            },
+            "expected_set": {
+                "analysis": {
+                    "normalizer": {
+                        "to_lower": {
+                            "type": "custom",
+                            "filter": ["lowercase"]
+                        }
+                    }
+                },
+                "number_of_replicas": "0"
+            }
+        },
+        {
+            "index_name": "playlist",
+            "expected_map": {
+                "playlist_id": {
+                    "type": "keyword"
+                },
+                "playlist_description": {
+                    "type": "text"
+                },
+                "playlist_name": {
+                    "type": "text",
+                    "analyzer": "english",
+                    "fields": {
+                        "keyword": {
+                            "type": "keyword",
+                            "ignore_above": 256,
+                            "normalizer": "to_lower"
+                        },
+                        "search_as_you_type": {
+                            "type": "search_as_you_type",
+                            "doc_values": false,
+                            "max_shingle_size": 3
+                        }
+                    }
+                },
+                "playlist_channel": {
+                    "type": "text",
+                    "fields": {
+                        "keyword": {
+                            "type": "keyword",
+                            "ignore_above": 256,
+                            "normalizer": "to_lower"
+                        }
+                    }
+                },
+                "playlist_channel_id": {
+                    "type": "keyword"
+                },
+                "playlist_thumbnail": {
+                    "type": "keyword"
+                },
+                "playlist_last_refresh": {
+                    "type": "date"
+                }
+            },
+            "expected_set": {
+                "analysis": {
+                    "normalizer": {
+                        "to_lower": {
+                            "type": "custom",
+                            "filter": ["lowercase"]
+                        }
+                    }
+                },
+                "number_of_replicas": "0"
+            }
+        },
+        {
+            "index_name": "subtitle",
+            "expected_map": {
+                "youtube_id": {
+                    "type": "keyword"
+                },
+                "title": {
+                    "type": "text",
+                    "fields": {
+                        "keyword": {
+                            "type": "keyword",
+                            "ignore_above": 256,
+                            "normalizer": "to_lower"
+                        }
+                    }
+                },
+                "subtitle_fragment_id": {
+                    "type": "keyword"
+                },
+                "subtitle_channel": {
+                    "type": "text",
+                    "fields": {
+                        "keyword": {
+                            "type": "keyword",
+                            "ignore_above": 256,
+                            "normalizer": "to_lower"
+                        }
+                    }
+                },
+                "subtitle_channel_id": {
+                    "type": "keyword"
+                },
+                "subtitle_start": {
+                    "type": "text"
+                },
+                "subtitle_end": {
+                    "type": "text"
+                },
+                "subtitle_last_refresh": {
+                    "type": "date"
+                },
+                "subtitle_index": {
+                    "type" : "long"
+                },
+                "subtitle_lang": {
+                    "type": "keyword"
+                },
+                "subtitle_source": {
+                    "type": "keyword"
+                },
+                "subtitle_line": {
+                    "type" : "text",
+                    "analyzer": "english"
+                }
+            },
+            "expected_set": {
+                "analysis": {
+                    "normalizer": {
+                        "to_lower": {
+                            "type": "custom",
+                            "filter": ["lowercase"]
+                        }
+                    }
+                },
+                "number_of_replicas": "0"
+            }
+        }
+    ]
+}

tubearchivist/home/src/es/index_setup.py

@@ -1,9 +1,8 @@
 """
-Functionality:
-- initial elastic search setup
-- index configuration is represented in INDEX_CONFIG
-- index mapping and settings validation
-- backup and restore
+functionality:
+- setup elastic index at first start
+- verify and update index mapping and settings if needed
+- backup and restore metadata
 """
 
 import json
@@ -12,213 +11,8 @@ import zipfile
 from datetime import datetime
 
 import requests
-from home.src.config import AppConfig
-from home.src.helper import ignore_filelist
-
-# expected mapping and settings
-INDEX_CONFIG = [
-    {
-        "index_name": "channel",
-        "expected_map": {
-            "channel_id": {
-                "type": "keyword",
-            },
-            "channel_name": {
-                "type": "text",
-                "analyzer": "english",
-                "fields": {
-                    "keyword": {
-                        "type": "keyword",
-                        "ignore_above": 256,
-                        "normalizer": "to_lower",
-                    },
-                    "search_as_you_type": {
-                        "type": "search_as_you_type",
-                        "doc_values": False,
-                        "max_shingle_size": 3,
-                    },
-                },
-            },
-            "channel_banner_url": {"type": "keyword", "index": False},
-            "channel_thumb_url": {"type": "keyword", "index": False},
-            "channel_description": {"type": "text"},
-            "channel_last_refresh": {"type": "date", "format": "epoch_second"},
-        },
-        "expected_set": {
-            "analysis": {
-                "normalizer": {
-                    "to_lower": {"type": "custom", "filter": ["lowercase"]}
-                }
-            },
-            "number_of_replicas": "0",
-        },
-    },
-    {
-        "index_name": "video",
-        "expected_map": {
-            "vid_thumb_url": {"type": "text", "index": False},
-            "date_downloaded": {"type": "date"},
-            "channel": {
-                "properties": {
-                    "channel_id": {
-                        "type": "keyword",
-                    },
-                    "channel_name": {
-                        "type": "text",
-                        "analyzer": "english",
-                        "fields": {
-                            "keyword": {
-                                "type": "keyword",
-                                "ignore_above": 256,
-                                "normalizer": "to_lower",
-                            },
-                            "search_as_you_type": {
-                                "type": "search_as_you_type",
-                                "doc_values": False,
-                                "max_shingle_size": 3,
-                            },
-                        },
-                    },
-                    "channel_banner_url": {"type": "keyword", "index": False},
-                    "channel_thumb_url": {"type": "keyword", "index": False},
-                    "channel_description": {"type": "text"},
-                    "channel_last_refresh": {
-                        "type": "date",
-                        "format": "epoch_second",
-                    },
-                }
-            },
-            "description": {"type": "text"},
-            "media_url": {"type": "keyword", "index": False},
-            "tags": {
-                "type": "text",
-                "analyzer": "english",
-                "fields": {
-                    "keyword": {"type": "keyword", "ignore_above": 256}
-                },
-            },
-            "title": {
-                "type": "text",
-                "analyzer": "english",
-                "fields": {
-                    "keyword": {
-                        "type": "keyword",
-                        "ignore_above": 256,
-                        "normalizer": "to_lower",
-                    },
-                    "search_as_you_type": {
-                        "type": "search_as_you_type",
-                        "doc_values": False,
-                        "max_shingle_size": 3,
-                    },
-                },
-            },
-            "vid_last_refresh": {"type": "date"},
-            "youtube_id": {"type": "keyword"},
-            "published": {"type": "date"},
-            "playlist": {
-                "type": "text",
-                "fields": {
-                    "keyword": {
-                        "type": "keyword",
-                        "ignore_above": 256,
-                        "normalizer": "to_lower",
-                    }
-                },
-            },
-        },
-        "expected_set": {
-            "analysis": {
-                "normalizer": {
-                    "to_lower": {"type": "custom", "filter": ["lowercase"]}
-                }
-            },
-            "number_of_replicas": "0",
-        },
-    },
-    {
-        "index_name": "download",
-        "expected_map": {
-            "timestamp": {"type": "date"},
-            "channel_id": {"type": "keyword"},
-            "channel_name": {
-                "type": "text",
-                "fields": {
-                    "keyword": {
-                        "type": "keyword",
-                        "ignore_above": 256,
-                        "normalizer": "to_lower",
-                    }
-                },
-            },
-            "status": {"type": "keyword"},
-            "title": {
-                "type": "text",
-                "fields": {
-                    "keyword": {
-                        "type": "keyword",
-                        "ignore_above": 256,
-                        "normalizer": "to_lower",
-                    }
-                },
-            },
-            "vid_thumb_url": {"type": "keyword"},
-            "youtube_id": {"type": "keyword"},
-        },
-        "expected_set": {
-            "analysis": {
-                "normalizer": {
-                    "to_lower": {"type": "custom", "filter": ["lowercase"]}
-                }
-            },
-            "number_of_replicas": "0",
-        },
-    },
-    {
-        "index_name": "playlist",
-        "expected_map": {
-            "playlist_id": {"type": "keyword"},
-            "playlist_description": {"type": "text"},
-            "playlist_name": {
-                "type": "text",
-                "analyzer": "english",
-                "fields": {
-                    "keyword": {
-                        "type": "keyword",
-                        "ignore_above": 256,
-                        "normalizer": "to_lower",
-                    },
-                    "search_as_you_type": {
-                        "type": "search_as_you_type",
-                        "doc_values": False,
-                        "max_shingle_size": 3,
-                    },
-                },
-            },
-            "playlist_channel": {
-                "type": "text",
-                "fields": {
-                    "keyword": {
-                        "type": "keyword",
-                        "ignore_above": 256,
-                        "normalizer": "to_lower",
-                    }
-                },
-            },
-            "playlist_channel_id": {"type": "keyword"},
-            "playlist_thumbnail": {"type": "keyword"},
-            "playlist_last_refresh": {"type": "date"},
-        },
-        "expected_set": {
-            "analysis": {
-                "normalizer": {
-                    "to_lower": {"type": "custom", "filter": ["lowercase"]}
-                }
-            },
-            "number_of_replicas": "0",
-        },
-    },
-]
+from home.src.ta.config import AppConfig
+from home.src.ta.helper import ignore_filelist
 
 
 class ElasticIndex:
@@ -602,48 +396,21 @@ class ElasticBackup:
             os.remove(file_path)
 
 
-def get_available_backups():
-    """return dict of available backups for settings view"""
-    backup_handler = ElasticBackup(INDEX_CONFIG, reason=False)
-    all_backup_files = backup_handler.get_all_backup_files()
-    return all_backup_files
+def get_mapping():
+    """read index_mapping.json and get expected mapping and settings"""
+    with open("home/src/es/index_mapping.json", "r", encoding="utf-8") as f:
+        index_config = json.load(f).get("index_config")
 
-
-def backup_all_indexes(reason):
-    """backup all es indexes to disk"""
-    backup_handler = ElasticBackup(INDEX_CONFIG, reason)
-
-    for index in backup_handler.index_config:
-        index_name = index["index_name"]
-        if not backup_handler.index_exists(index_name):
-            continue
-        all_results = backup_handler.get_all_documents(index_name)
-        file_content = backup_handler.build_bulk(all_results)
-        backup_handler.write_es_json(file_content, index_name)
-        backup_handler.write_ta_json(all_results, index_name)
-
-    backup_handler.zip_it()
-
-    if reason == "auto":
-        backup_handler.rotate_backup()
-
-
-def restore_from_backup(filename):
-    """restore indexes from backup file"""
-    # delete
-    index_check(force_restore=True)
-    # recreate
-    backup_handler = ElasticBackup(INDEX_CONFIG, reason=False)
-    zip_content = backup_handler.unpack_zip_backup(filename)
-    backup_handler.restore_json_files(zip_content)
+    return index_config
 
 
 def index_check(force_restore=False):
     """check if all indexes are created and have correct mapping"""
 
     backed_up = False
+    index_config = get_mapping()
 
-    for index in INDEX_CONFIG:
+    for index in index_config:
         index_name = index["index_name"]
         expected_map = index["expected_map"]
         expected_set = index["expected_set"]
@@ -675,3 +442,42 @@ def index_check(force_restore=False):
 
         # else all good
         print(f"ta_{index_name} index is created and up to date...")
+
+
+def get_available_backups():
+    """return dict of available backups for settings view"""
+    index_config = get_mapping()
+    backup_handler = ElasticBackup(index_config, reason=False)
+    all_backup_files = backup_handler.get_all_backup_files()
+    return all_backup_files
+
+
+def backup_all_indexes(reason):
+    """backup all es indexes to disk"""
+    index_config = get_mapping()
+    backup_handler = ElasticBackup(index_config, reason)
+
+    for index in backup_handler.index_config:
+        index_name = index["index_name"]
+        if not backup_handler.index_exists(index_name):
+            continue
+        all_results = backup_handler.get_all_documents(index_name)
+        file_content = backup_handler.build_bulk(all_results)
+        backup_handler.write_es_json(file_content, index_name)
+        backup_handler.write_ta_json(all_results, index_name)
+
+    backup_handler.zip_it()
+
+    if reason == "auto":
+        backup_handler.rotate_backup()
+
+
+def restore_from_backup(filename):
+    """restore indexes from backup file"""
+    # delete
+    index_check(force_restore=True)
+    # recreate
+    index_config = get_mapping()
+    backup_handler = ElasticBackup(index_config, reason=False)
+    zip_content = backup_handler.unpack_zip_backup(filename)
+    backup_handler.restore_json_files(zip_content)

tubearchivist/home/src/frontend/api_calls.py

@@ -4,19 +4,18 @@ Functionality:
 - called via user input
 """
 
-from home.src.download import (
+from home.src.download.queue import PendingList
+from home.src.download.subscriptions import (
     ChannelSubscription,
-    PendingList,
     PlaylistSubscription,
 )
-from home.src.helper import RedisArchivist, RedisQueue, UrlListParser
-from home.src.index import (
-    WatchState,
-    YoutubeChannel,
-    YoutubePlaylist,
-    YoutubeVideo,
-)
-from home.src.searching import SearchForm
+from home.src.frontend.searching import SearchForm
+from home.src.frontend.watched import WatchState
+from home.src.index.channel import YoutubeChannel
+from home.src.index.playlist import YoutubePlaylist
+from home.src.index.video import YoutubeVideo
+from home.src.ta.helper import UrlListParser
+from home.src.ta.ta_redis import RedisArchivist, RedisQueue
 from home.tasks import (
     download_pending,
     download_single,
@@ -306,7 +305,7 @@ class PostData:
         playlist_dict = self.exec_val
         playlist_id = playlist_dict["playlist-id"]
         playlist_action = playlist_dict["playlist-action"]
-        print(f"delete {playlist_action} from playlist {playlist_id}")
+        print(f"{playlist_id}: delete playlist {playlist_action}")
         if playlist_action == "metadata":
             YoutubePlaylist(playlist_id).delete_metadata()
         elif playlist_action == "all":

tubearchivist/home/src/frontend/forms.py

@@ -62,6 +62,24 @@ class ApplicationSettingsForm(forms.Form):
         ("1", "enable ryd integration"),
     ]
 
+    CAST_CHOICES = [
+        ("", "-- change Cast integration --"),
+        ("0", "disable Cast"),
+        ("1", "enable Cast"),
+    ]
+
+    SUBTITLE_SOURCE_CHOICES = [
+        ("", "-- change subtitle source settings"),
+        ("user", "only download user created"),
+        ("auto", "also download auto generated"),
+    ]
+
+    SUBTITLE_INDEX_CHOICES = [
+        ("", "-- change subtitle index settings --"),
+        ("0", "disable subtitle index"),
+        ("1", "enable subtitle index"),
+    ]
+
     subscriptions_channel_size = forms.IntegerField(required=False)
     downloads_limit_count = forms.IntegerField(required=False)
     downloads_limit_speed = forms.IntegerField(required=False)
@@ -75,9 +93,19 @@ class ApplicationSettingsForm(forms.Form):
     downloads_add_thumbnail = forms.ChoiceField(
         widget=forms.Select, choices=THUMBNAIL_CHOICES, required=False
     )
+    downloads_subtitle = forms.CharField(required=False)
+    downloads_subtitle_source = forms.ChoiceField(
+        widget=forms.Select, choices=SUBTITLE_SOURCE_CHOICES, required=False
+    )
+    downloads_subtitle_index = forms.ChoiceField(
+        widget=forms.Select, choices=SUBTITLE_INDEX_CHOICES, required=False
+    )
     downloads_integrate_ryd = forms.ChoiceField(
         widget=forms.Select, choices=RYD_CHOICES, required=False
     )
+    application_enable_cast = forms.ChoiceField(
+        widget=forms.Select, choices=CAST_CHOICES, required=False
+    )
 
 
 class SchedulerSettingsForm(forms.Form):

tubearchivist/home/src/frontend/searching.py

@@ -6,36 +6,26 @@ Functionality:
 - calculate pagination values
 """
 
-import math
 import urllib.parse
 from datetime import datetime
 
-import requests
-from home.src.config import AppConfig
-from home.src.helper import RedisArchivist
-from home.src.thumbnails import ThumbManager
+from home.src.download.thumbnails import ThumbManager
+from home.src.es.connect import ElasticWrap
+from home.src.ta.config import AppConfig
 
 
 class SearchHandler:
     """search elastic search"""
 
-    CONFIG = AppConfig().config
-    CACHE_DIR = CONFIG["application"]["cache_dir"]
-    ES_AUTH = CONFIG["application"]["es_auth"]
-
-    def __init__(self, url, data):
+    def __init__(self, path, config, data=False):
         self.max_hits = None
-        self.url = url
+        self.path = path
+        self.config = config
         self.data = data
 
     def get_data(self):
         """get the data"""
-        if self.data:
-            response = requests.get(
-                self.url, json=self.data, auth=self.ES_AUTH
-            ).json()
-        else:
-            response = requests.get(self.url, auth=self.ES_AUTH).json()
+        response, _ = ElasticWrap(self.path, config=self.config).get(self.data)
 
         if "hits" in response.keys():
             self.max_hits = response["hits"]["total"]["value"]
@@ -153,11 +143,10 @@ class SearchForm:
     """build query from search form data"""
 
     CONFIG = AppConfig().config
-    ES_URL = CONFIG["application"]["es_url"]
 
     def multi_search(self, search_query):
         """searching through index"""
-        url = self.ES_URL + "/ta_video,ta_channel,ta_playlist/_search"
+        path = "ta_video,ta_channel,ta_playlist/_search"
         data = {
             "size": 30,
             "query": {
@@ -184,7 +173,7 @@ class SearchForm:
                 }
             },
         }
-        look_up = SearchHandler(url, data)
+        look_up = SearchHandler(path, config=self.CONFIG, data=data)
         search_results = look_up.get_data()
         all_results = self.build_results(search_results)
 
@@ -212,62 +201,3 @@ class SearchForm:
         }
 
         return all_results
-
-
-class Pagination:
-    """
-    figure out the pagination based on page size and total_hits
-    """
-
-    def __init__(self, page_get, user_id, search_get=False):
-        self.user_id = user_id
-        self.page_size = self.get_page_size()
-        self.page_get = page_get
-        self.search_get = search_get
-        self.pagination = self.first_guess()
-
-    def get_page_size(self):
-        """get default or user modified page_size"""
-        key = f"{self.user_id}:page_size"
-        page_size = RedisArchivist().get_message(key)["status"]
-        if not page_size:
-            config = AppConfig().config
-            page_size = config["archive"]["page_size"]
-
-        return page_size
-
-    def first_guess(self):
-        """build first guess before api call"""
-        page_get = self.page_get
-        if page_get in [0, 1]:
-            page_from = 0
-            prev_pages = False
-        elif page_get > 1:
-            page_from = (page_get - 1) * self.page_size
-            prev_pages = [
-                i for i in range(page_get - 1, page_get - 6, -1) if i > 1
-            ]
-            prev_pages.reverse()
-        pagination = {
-            "page_size": self.page_size,
-            "page_from": page_from,
-            "prev_pages": prev_pages,
-            "current_page": page_get,
-        }
-        if self.search_get:
-            pagination.update({"search_get": self.search_get})
-        return pagination
-
-    def validate(self, total_hits):
-        """validate pagination with total_hits after making api call"""
-        page_get = self.page_get
-        max_pages = math.ceil(total_hits / self.page_size)
-        if page_get < max_pages and max_pages > 1:
-            self.pagination["last_page"] = max_pages
-        else:
-            self.pagination["last_page"] = False
-        next_pages = [
-            i for i in range(page_get + 1, page_get + 6) if 1 < i < max_pages
-        ]
-
-        self.pagination["next_pages"] = next_pages

tubearchivist/home/src/frontend/watched.py

@@ -0,0 +1,128 @@
+"""
+functionality:
+- handle watched state for videos, channels and playlists
+"""
+
+import json
+from datetime import datetime
+
+import requests
+from home.src.ta.config import AppConfig
+from home.src.ta.helper import UrlListParser
+
+
+class WatchState:
+    """handle watched checkbox for videos and channels"""
+
+    CONFIG = AppConfig().config
+    ES_URL = CONFIG["application"]["es_url"]
+    ES_AUTH = CONFIG["application"]["es_auth"]
+    HEADERS = {"Content-type": "application/json"}
+
+    def __init__(self, youtube_id):
+        self.youtube_id = youtube_id
+        self.stamp = int(datetime.now().strftime("%s"))
+
+    def mark_as_watched(self):
+        """update es with new watched value"""
+        url_type = self.dedect_type()
+        if url_type == "video":
+            self.mark_vid_watched()
+        elif url_type == "channel":
+            self.mark_channel_watched()
+        elif url_type == "playlist":
+            self.mark_playlist_watched()
+
+        print(f"marked {self.youtube_id} as watched")
+
+    def mark_as_unwatched(self):
+        """revert watched state to false"""
+        url_type = self.dedect_type()
+        if url_type == "video":
+            self.mark_vid_watched(revert=True)
+
+        print(f"revert {self.youtube_id} as unwatched")
+
+    def dedect_type(self):
+        """find youtube id type"""
+        print(self.youtube_id)
+        url_process = UrlListParser(self.youtube_id).process_list()
+        url_type = url_process[0]["type"]
+        return url_type
+
+    def mark_vid_watched(self, revert=False):
+        """change watched status of single video"""
+        url = self.ES_URL + "/ta_video/_update/" + self.youtube_id
+        data = {
+            "doc": {"player": {"watched": True, "watched_date": self.stamp}}
+        }
+        if revert:
+            data["doc"]["player"]["watched"] = False
+
+        payload = json.dumps(data)
+        request = requests.post(
+            url, data=payload, headers=self.HEADERS, auth=self.ES_AUTH
+        )
+        if not request.ok:
+            print(request.text)
+            raise ValueError("failed to mark video as watched")
+
+    def mark_channel_watched(self):
+        """change watched status of every video in channel"""
+        data = {
+            "query": {
+                "bool": {
+                    "must": [
+                        {
+                            "term": {
+                                "channel.channel_id": {
+                                    "value": self.youtube_id
+                                }
+                            }
+                        },
+                        {"term": {"player.watched": {"value": False}}},
+                    ]
+                }
+            },
+            "script": {
+                "source": "ctx._source.player['watched'] = true",
+                "lang": "painless",
+            },
+        }
+        payload = json.dumps(data)
+        url = f"{self.ES_URL}/ta_video/_update_by_query"
+        request = requests.post(
+            url, data=payload, headers=self.HEADERS, auth=self.ES_AUTH
+        )
+        if not request.ok:
+            print(request.text)
+            raise ValueError("failed mark channel as watched")
+
+    def mark_playlist_watched(self):
+        """change watched state of all videos in playlist"""
+        data = {
+            "query": {
+                "bool": {
+                    "must": [
+                        {
+                            "term": {
+                                "playlist.keyword": {"value": self.youtube_id}
+                            }
+                        },
+                        {"term": {"player.watched": {"value": False}}},
+                    ]
+                }
+            },
+            "script": {
+                "source": "ctx._source.player['watched'] = true",
+                "lang": "painless",
+            },
+        }
+        payload = json.dumps(data)
+        url = f"{self.ES_URL}/ta_video/_update_by_query"
+        request = requests.post(
+            url, data=payload, headers=self.HEADERS, auth=self.ES_AUTH
+        )
+        if not request.ok:
+            print(request.text)
+            raise ValueError("failed mark playlist as watched")

tubearchivist/home/src/index.py

@@ -1,967 +0,0 @@
-"""
-Functionality:
-- index new videos into elastisearch
-- extract video info with yt_dlp
-- scrape youtube channel page if needed
-"""
-
-import json
-import os
-import re
-from datetime import datetime
-from time import sleep
-
-import requests
-import yt_dlp
-from bs4 import BeautifulSoup
-from home.src.config import AppConfig
-from home.src.helper import DurationConverter, UrlListParser, clean_string
-from home.src.thumbnails import ThumbManager
-from ryd_client import ryd_client
-
-
-class YoutubeChannel:
-    """represents a single youtube channel"""
-
-    CONFIG = AppConfig().config
-    ES_URL = CONFIG["application"]["es_url"]
-    ES_AUTH = CONFIG["application"]["es_auth"]
-    CACHE_DIR = CONFIG["application"]["cache_dir"]
-    VIDEOS = CONFIG["application"]["videos"]
-
-    def __init__(self, channel_id):
-        self.channel_id = channel_id
-        self.json_data = None
-        self.source = None
-        self.channel_dict = self.build_channel_dict()
-
-    def build_channel_dict(self, scrape=False):
-        """combine the dicts build from extracted json payload"""
-        if scrape:
-            channel_dict = False
-        else:
-            channel_dict = self.get_es_channel()
-        if not channel_dict:
-            print("scrape data from youtube")
-            self.scrape_channel()
-            channel_dict = self.parse_channel_main()
-            channel_dict.update(self.parse_channel_meta())
-            self.source = "scraped"
-        return channel_dict
-
-    def get_es_channel(self):
-        """get from elastic search first if possible"""
-        channel_id = self.channel_id
-        url = f"{self.ES_URL}/ta_channel/_doc/{channel_id}"
-        response = requests.get(url, auth=self.ES_AUTH)
-        if response.ok:
-            channel_source = response.json()["_source"]
-            self.source = "elastic"
-            return channel_source
-        return False
-
-    def scrape_channel(self):
-        """scrape channel page for additional infos"""
-        channel_id = self.channel_id
-        url = f"https://www.youtube.com/channel/{channel_id}/about?hl=en"
-        cookies = {"CONSENT": "YES+xxxxxxxxxxxxxxxxxxxxxxxxxxx"}
-        response = requests.get(url, cookies=cookies, auth=self.ES_AUTH)
-        if response.ok:
-            channel_page = response.text
-        else:
-            print(f"failed to extract channel info for: {channel_id}")
-            raise ConnectionError
-        soup = BeautifulSoup(channel_page, "html.parser")
-        # load script into json
-        all_scripts = soup.find("body").find_all("script")
-        for script in all_scripts:
-            if "var ytInitialData = " in str(script):
-                script_content = str(script)
-                break
-        # extract payload
-        script_content = script_content.split("var ytInitialData = ")[1]
-        json_raw = script_content.rstrip(";</script>")
-        json_data = json.loads(json_raw)
-        # add to self
-        self.json_data = json_data
-
-    def parse_channel_main(self):
-        """extract maintab values from scraped channel json data"""
-        main_tab = self.json_data["header"]["c4TabbedHeaderRenderer"]
-        channel_name = main_tab["title"]
-        last_refresh = int(datetime.now().strftime("%s"))
-        # channel_subs
-        try:
-            sub_text_simple = main_tab["subscriberCountText"]["simpleText"]
-            sub_text = sub_text_simple.split(" ")[0]
-            if sub_text[-1] == "K":
-                channel_subs = int(float(sub_text.replace("K", "")) * 1000)
-            elif sub_text[-1] == "M":
-                channel_subs = int(float(sub_text.replace("M", "")) * 1000000)
-            elif int(sub_text) >= 0:
-                channel_subs = int(sub_text)
-            else:
-                message = f"{sub_text} not dealt with"
-                print(message)
-        except KeyError:
-            channel_subs = 0
-        # banner
-        try:
-            all_banners = main_tab["banner"]["thumbnails"]
-            banner = sorted(all_banners, key=lambda k: k["width"])[-1]["url"]
-        except KeyError:
-            banner = False
-        # build and return dict
-        main_channel_dict = {
-            "channel_active": True,
-            "channel_last_refresh": last_refresh,
-            "channel_subs": channel_subs,
-            "channel_banner_url": banner,
-            "channel_name": channel_name,
-            "channel_id": self.channel_id,
-        }
-        return main_channel_dict
-
-    def parse_channel_meta(self):
-        """extract meta tab values from channel payload"""
-        # meta tab
-        json_data = self.json_data
-        meta_tab = json_data["metadata"]["channelMetadataRenderer"]
-        description = meta_tab["description"]
-        all_thumbs = meta_tab["avatar"]["thumbnails"]
-        thumb_url = sorted(all_thumbs, key=lambda k: k["width"])[-1]["url"]
-        # stats tab
-        renderer = "twoColumnBrowseResultsRenderer"
-        all_tabs = json_data["contents"][renderer]["tabs"]
-        for tab in all_tabs:
-            if "tabRenderer" in tab.keys():
-                if tab["tabRenderer"]["title"] == "About":
-                    about_tab = tab["tabRenderer"]["content"][
-                        "sectionListRenderer"
-                    ]["contents"][0]["itemSectionRenderer"]["contents"][0][
-                        "channelAboutFullMetadataRenderer"
-                    ]
-                    break
-        try:
-            channel_views_text = about_tab["viewCountText"]["simpleText"]
-            channel_views = int(re.sub(r"\D", "", channel_views_text))
-        except KeyError:
-            channel_views = 0
-
-        meta_channel_dict = {
-            "channel_description": description,
-            "channel_thumb_url": thumb_url,
-            "channel_views": channel_views,
-        }
-
-        return meta_channel_dict
-
-    def get_channel_art(self):
-        """download channel art for new channels"""
-        channel_id = self.channel_id
-        channel_thumb = self.channel_dict["channel_thumb_url"]
-        channel_banner = self.channel_dict["channel_banner_url"]
-        ThumbManager().download_chan(
-            [(channel_id, channel_thumb, channel_banner)]
-        )
-
-    def upload_to_es(self):
-        """upload channel data to elastic search"""
-        url = f"{self.ES_URL}/ta_channel/_doc/{self.channel_id}"
-        response = requests.put(url, json=self.channel_dict, auth=self.ES_AUTH)
-        print(f"added {self.channel_id} to es")
-        if not response.ok:
-            print(response.text)
-            raise ValueError("failed to add channel to index")
-
-    def sync_to_videos(self):
-        """sync new channel_dict to all videos of channel"""
-        headers = {"Content-type": "application/json"}
-        channel_id = self.channel_id
-        # add ingest pipeline
-        processors = []
-        for field, value in self.channel_dict.items():
-            line = {"set": {"field": "channel." + field, "value": value}}
-            processors.append(line)
-        data = {"description": channel_id, "processors": processors}
-        payload = json.dumps(data)
-        url = self.ES_URL + "/_ingest/pipeline/" + channel_id
-        request = requests.put(
-            url, data=payload, headers=headers, auth=self.ES_AUTH
-        )
-        if not request.ok:
-            print(request.text)
-        # apply pipeline
-        data = {"query": {"match": {"channel.channel_id": channel_id}}}
-        payload = json.dumps(data)
-        url = self.ES_URL + "/ta_video/_update_by_query?pipeline=" + channel_id
-        request = requests.post(
-            url, data=payload, headers=headers, auth=self.ES_AUTH
-        )
-        if not request.ok:
-            print(request.text)
-
-    def get_folder_path(self):
-        """get folder where media files get stored"""
-        channel_name = self.channel_dict["channel_name"]
-        folder_name = clean_string(channel_name)
-        folder_path = os.path.join(self.VIDEOS, folder_name)
-        return folder_path
-
-    def delete_es_videos(self):
-        """delete all channel documents from elasticsearch"""
-        headers = {"Content-type": "application/json"}
-        data = {
-            "query": {
-                "term": {"channel.channel_id": {"value": self.channel_id}}
-            }
-        }
-        payload = json.dumps(data)
-        url = self.ES_URL + "/ta_video/_delete_by_query"
-        response = requests.post(
-            url, data=payload, headers=headers, auth=self.ES_AUTH
-        )
-        if not response.ok:
-            print(response.text)
-
-    def delete_playlists(self):
-        """delete all indexed playlist from es"""
-        all_playlists = self.get_indexed_playlists()
-        for playlist in all_playlists:
-            playlist_id = playlist["playlist_id"]
-            YoutubePlaylist(playlist_id).delete_metadata()
-
-    def delete_channel(self):
-        """delete channel and all videos"""
-        print(f"deleting {self.channel_id} and all matching media files")
-        folder_path = self.get_folder_path()
-        print("delete all media files")
-        try:
-            all_videos = os.listdir(folder_path)
-            for video in all_videos:
-                video_path = os.path.join(folder_path, video)
-                os.remove(video_path)
-            os.rmdir(folder_path)
-        except FileNotFoundError:
-            print(f"no videos found for {folder_path}")
-
-        ThumbManager().delete_chan_thumb(self.channel_id)
-        print("delete indexed playlists")
-        self.delete_playlists()
-        print("delete indexed videos")
-        self.delete_es_videos()
-        url = self.ES_URL + "/ta_channel/_doc/" + self.channel_id
-        response = requests.delete(url, auth=self.ES_AUTH)
-        if not response.ok:
-            print(response.text)
-
-    def get_all_playlists(self):
-        """get all playlists owned by this channel"""
-        url = (
-            f"https://www.youtube.com/channel/{self.channel_id}"
-            + "/playlists?view=1&sort=dd&shelf_id=0"
-        )
-        obs = {
-            "quiet": True,
-            "skip_download": True,
-            "extract_flat": True,
-        }
-        playlists = yt_dlp.YoutubeDL(obs).extract_info(url)
-        all_entries = [(i["id"], i["title"]) for i in playlists["entries"]]
-
-        return all_entries
-
-    def get_indexed_playlists(self):
-        """get all indexed playlists from channel"""
-        data = {
-            "query": {
-                "term": {"playlist_channel_id": {"value": self.channel_id}}
-            },
-            "sort": [{"playlist_channel.keyword": {"order": "desc"}}],
-        }
-        all_playlists = IndexPaginate("ta_playlist", data).get_results()
-        return all_playlists
-
-
-class YoutubeVideo:
-    """represents a single youtube video"""
-
-    CONFIG = AppConfig().config
-    ES_URL = CONFIG["application"]["es_url"]
-    ES_AUTH = CONFIG["application"]["es_auth"]
-    CACHE_DIR = CONFIG["application"]["cache_dir"]
-    VIDEOS = CONFIG["application"]["videos"]
-
-    def __init__(self, youtube_id):
-        self.youtube_id = youtube_id
-        self.channel_id = None
-        self.vid_dict = None
-
-    def get_vid_dict(self):
-        """wrapper to loop around yt_dlp to retry on failure"""
-        print(f"get video data for {self.youtube_id}")
-        vid_dict = False
-        for i in range(3):
-            try:
-                vid_dict = self.get_youtubedl_vid_data()
-            except KeyError as e:
-                print(e)
-                sleep((i + 1) ** 2)
-                continue
-            else:
-                break
-
-        self.vid_dict = vid_dict
-        if self.CONFIG["downloads"]["integrate_ryd"]:
-            self.get_ryd_stats()
-
-    def get_youtubedl_vid_data(self):
-        """parse youtubedl extract info"""
-        youtube_id = self.youtube_id
-        obs = {
-            "quiet": True,
-            "default_search": "ytsearch",
-            "skip_download": True,
-            "check_formats": "selected",
-            "noplaylist": True,
-        }
-        try:
-            vid = yt_dlp.YoutubeDL(obs).extract_info(youtube_id)
-        except (
-            yt_dlp.utils.ExtractorError,
-            yt_dlp.utils.DownloadError,
-        ):
-            print("failed to get info for " + youtube_id)
-            return False
-        # extract
-        self.channel_id = vid["channel_id"]
-        upload_date = vid["upload_date"]
-        upload_date_time = datetime.strptime(upload_date, "%Y%m%d")
-        published = upload_date_time.strftime("%Y-%m-%d")
-        last_refresh = int(datetime.now().strftime("%s"))
-        # likes
-        try:
-            like_count = vid["like_count"]
-        except KeyError:
-            like_count = 0
-        try:
-            dislike_count = vid["dislike_count"]
-        except KeyError:
-            dislike_count = 0
-        # build dicts
-        stats = {
-            "view_count": vid["view_count"],
-            "like_count": like_count,
-            "dislike_count": dislike_count,
-            "average_rating": vid["average_rating"],
-        }
-        vid_basic = {
-            "title": vid["title"],
-            "description": vid["description"],
-            "category": vid["categories"],
-            "vid_thumb_url": vid["thumbnail"],
-            "tags": vid["tags"],
-            "published": published,
-            "stats": stats,
-            "vid_last_refresh": last_refresh,
-            "date_downloaded": last_refresh,
-            "youtube_id": youtube_id,
-            "active": True,
-            "channel": False,
-        }
-
-        return vid_basic
-
-    def add_player(self, missing_vid):
-        """add player information for new videos"""
-        cache_path = self.CACHE_DIR + "/download/"
-        videos = self.VIDEOS
-
-        if missing_vid:
-            # coming from scan_filesystem
-            channel_name, file_name, _ = missing_vid
-            vid_path = os.path.join(videos, channel_name, file_name)
-        else:
-            # coming from VideoDownload
-            all_cached = os.listdir(cache_path)
-            for file_cached in all_cached:
-                if self.youtube_id in file_cached:
-                    vid_path = os.path.join(cache_path, file_cached)
-                    break
-
-        duration_handler = DurationConverter()
-        duration = duration_handler.get_sec(vid_path)
-        duration_str = duration_handler.get_str(duration)
-        player = {
-            "watched": False,
-            "duration": duration,
-            "duration_str": duration_str,
-        }
-        self.vid_dict["player"] = player
-
-    def build_file_path(self, channel_name):
-        """build media_url from where file will be located"""
-        clean_channel_name = clean_string(channel_name)
-        timestamp = self.vid_dict["published"].replace("-", "")
-        youtube_id = self.vid_dict["youtube_id"]
-        title = self.vid_dict["title"]
-        clean_title = clean_string(title)
-        filename = f"{timestamp}_{youtube_id}_{clean_title}.mp4"
-        media_url = os.path.join(clean_channel_name, filename)
-        self.vid_dict["media_url"] = media_url
-
-    def get_es_data(self):
-        """get current data from elastic search"""
-        url = self.ES_URL + "/ta_video/_doc/" + self.youtube_id
-        response = requests.get(url, auth=self.ES_AUTH)
-        if not response.ok:
-            print(response.text)
-        es_vid_dict = json.loads(response.text)
-        return es_vid_dict
-
-    def upload_to_es(self):
-        """upload video data to elastic search"""
-        url = f"{self.ES_URL}/ta_video/_doc/{self.youtube_id}/?refresh=true"
-        response = requests.put(url, json=self.vid_dict, auth=self.ES_AUTH)
-        if not response.ok:
-            print(response.text)
-            raise ValueError("failed to add video to index")
-
-    def deactivate(self):
-        """deactivate document on extractor error"""
-        youtube_id = self.youtube_id
-        headers = {"Content-type": "application/json"}
-        url = f"{self.ES_URL}/ta_video/_update/{youtube_id}"
-        data = {"script": "ctx._source.active = false"}
-        json_str = json.dumps(data)
-        response = requests.post(
-            url, data=json_str, headers=headers, auth=self.ES_AUTH
-        )
-        print(f"deactivated {youtube_id}")
-        if not response.ok:
-            print(response.text)
-
-    def delete_media_file(self):
-        """delete video file, meta data, thumbnails"""
-        # delete media file
-        es_vid_dict = self.get_es_data()
-        media_url = es_vid_dict["_source"]["media_url"]
-        print(f"delete {media_url} from file system")
-        to_delete = os.path.join(self.VIDEOS, media_url)
-        os.remove(to_delete)
-        # delete from index
-        url = f"{self.ES_URL}/ta_video/_doc/{self.youtube_id}"
-        response = requests.delete(url, auth=self.ES_AUTH)
-        if not response.ok:
-            print(response.text)
-        # delete thumbs from cache
-        ThumbManager().delete_vid_thumb(self.youtube_id)
-
-    def get_ryd_stats(self):
-        """get optional stats from returnyoutubedislikeapi.com"""
-        try:
-            print(f"get ryd stats for: {self.youtube_id}")
-            result = ryd_client.get(self.youtube_id)
-        except requests.exceptions.ConnectionError:
-            print(f"failed to query ryd api, skipping {self.youtube_id}")
-            return False
-
-        dislikes = {
-            "dislike_count": result["dislikes"],
-            "average_rating": result["rating"],
-        }
-        self.vid_dict["stats"].update(dislikes)
-
-        return True
-
-
-class YoutubePlaylist:
-    """represent a single playlist on YouTube"""
-
-    CONFIG = AppConfig().config
-    ES_URL = CONFIG["application"]["es_url"]
-    ES_AUTH = CONFIG["application"]["es_auth"]
-
-    def __init__(self, playlist_id, all_youtube_ids=False):
-        self.playlist_id = playlist_id
-        self.stamp = int(datetime.now().strftime("%s"))
-        self.all_youtube_ids = all_youtube_ids
-        self.playlist_dict = False
-
-    def get_playlist_dict(self, scrape=False):
-        """get data from es or youtube"""
-        print(f"get playlist with id {self.playlist_id}")
-
-        if scrape:
-            playlist_dict = self.get_youtube_playlist()
-            if not playlist_dict:
-                return False
-            playlist_dict["playlist_entries"] = self.get_entries()
-        else:
-            playlist_dict = self.get_es_playlist()
-            if not playlist_dict:
-                playlist_dict = self.get_youtube_playlist()
-                playlist_dict["playlist_entries"] = self.get_entries()
-
-        self.playlist_dict = playlist_dict
-        return True
-
-    def get_youtube_playlist(self):
-        """get meta data dict from youtube"""
-        url = "https://www.youtube.com/playlist?list=" + self.playlist_id
-        obs = {
-            "default_search": "ytsearch",
-            "quiet": True,
-            "skip_download": True,
-            "extract_flat": True,
-            "playlistend": 0,
-        }
-        try:
-            playlist = yt_dlp.YoutubeDL(obs).extract_info(url, download=False)
-        except (
-            yt_dlp.utils.ExtractorError,
-            yt_dlp.utils.DownloadError,
-        ):
-            print("failed to get info for " + self.playlist_id)
-            return False
-
-        playlist_es = {
-            "playlist_id": self.playlist_id,
-            "playlist_active": True,
-            "playlist_subscribed": False,
-            "playlist_name": playlist["title"],
-            "playlist_channel": playlist["channel"],
-            "playlist_channel_id": playlist["channel_id"],
-            "playlist_thumbnail": playlist["thumbnails"][-1]["url"],
-            "playlist_description": playlist["description"] or False,
-            "playlist_last_refresh": self.stamp,
-        }
-
-        return playlist_es
-
-    def get_es_playlist(self):
-        """get indexed data from es"""
-        url = f"{self.ES_URL}/ta_playlist/_doc/{self.playlist_id}"
-        response = requests.get(url, auth=self.ES_AUTH)
-        if response.ok:
-            return json.loads(response.text)["_source"]
-
-        return False
-
-    def get_entries(self, playlistend=False):
-        """get all videos in playlist"""
-        url = "https://www.youtube.com/playlist?list=" + self.playlist_id
-        obs = {
-            "default_search": "ytsearch",
-            "quiet": True,
-            "skip_download": True,
-            "extract_flat": True,
-        }
-        if playlistend:
-            obs["playlistend"] = playlistend
-
-        try:
-            playlist = yt_dlp.YoutubeDL(obs).extract_info(url, download=False)
-        except (
-            yt_dlp.utils.ExtractorError,
-            yt_dlp.utils.DownloadError,
-        ):
-            print("failed to get plealist entries for " + self.playlist_id)
-            return False
-
-        all_members = []
-        for idx, entry in enumerate(playlist["entries"]):
-            uploader = entry["uploader"]
-            youtube_id = entry["id"]
-            if self.all_youtube_ids:
-                downloaded = youtube_id in self.all_youtube_ids
-            else:
-                downloaded = False
-            if not uploader:
-                continue
-            to_append = {
-                "youtube_id": youtube_id,
-                "title": entry["title"],
-                "uploader": uploader,
-                "idx": idx,
-                "downloaded": downloaded,
-            }
-            all_members.append(to_append)
-
-        return all_members
-
-    def upload_to_es(self):
-        """add playlist to es with its entries"""
-        playlist = self.playlist_dict
-        url = f"{self.ES_URL}/ta_playlist/_doc/{self.playlist_id}"
-        response = requests.put(url, json=playlist, auth=self.ES_AUTH)
-        if not response.ok:
-            print(response.text)
-            raise ValueError("failed to add playlist to index")
-
-    def add_vids_to_playlist(self):
-        """sync the playlist id to videos"""
-        playlist_dict = self.playlist_dict
-        script = (
-            'if (!ctx._source.containsKey("playlist")) '
-            + "{ctx._source.playlist = [params.playlist]} "
-            + "else if (!ctx._source.playlist.contains(params.playlist)) "
-            + "{ctx._source.playlist.add(params.playlist)} "
-            + "else {ctx.op = 'none'}"
-        )
-
-        bulk_list = []
-        for entry in playlist_dict["playlist_entries"]:
-            youtube_id = entry["youtube_id"]
-            action = {"update": {"_id": youtube_id, "_index": "ta_video"}}
-            source = {
-                "script": {
-                    "source": script,
-                    "lang": "painless",
-                    "params": {"playlist": self.playlist_id},
-                }
-            }
-            bulk_list.append(json.dumps(action))
-            bulk_list.append(json.dumps(source))
-
-        # add last newline
-        bulk_list.append("\n")
-        query_str = "\n".join(bulk_list)
-        headers = {"Content-type": "application/x-ndjson"}
-        url = self.ES_URL + "/_bulk"
-        response = requests.post(
-            url, data=query_str, headers=headers, auth=self.ES_AUTH
-        )
-        if not response.ok:
-            print(response.text)
-
-    def update_playlist(self):
-        """update metadata for playlist with data from YouTube"""
-        subscribed = self.get_es_playlist()["playlist_subscribed"]
-        self.get_playlist_dict(scrape=True)
-        if not self.playlist_dict:
-            # return false to deactivate
-            return False
-
-        self.playlist_dict["playlist_subscribed"] = subscribed
-        self.upload_to_es()
-        return self.playlist_dict
-
-    def build_nav(self, youtube_id):
-        """find next and previous in playlist of a given youtube_id"""
-        all_entries_available = self.playlist_dict["playlist_entries"]
-        all_entries = [i for i in all_entries_available if i["downloaded"]]
-        current = [i for i in all_entries if i["youtube_id"] == youtube_id]
-        # stop if not found or playlist of 1
-        if not current or not len(all_entries) > 1:
-            return False
-
-        current_idx = all_entries.index(current[0])
-        if current_idx == 0:
-            previous_item = False
-        else:
-            previous_item = all_entries[current_idx - 1]
-            prev_thumb = ThumbManager().vid_thumb_path(
-                previous_item["youtube_id"]
-            )
-            previous_item["vid_thumb"] = prev_thumb
-
-        if current_idx == len(all_entries) - 1:
-            next_item = False
-        else:
-            next_item = all_entries[current_idx + 1]
-            next_thumb = ThumbManager().vid_thumb_path(next_item["youtube_id"])
-            next_item["vid_thumb"] = next_thumb
-
-        nav = {
-            "playlist_meta": {
-                "current_idx": current[0]["idx"],
-                "playlist_id": self.playlist_id,
-                "playlist_name": self.playlist_dict["playlist_name"],
-                "playlist_channel": self.playlist_dict["playlist_channel"],
-            },
-            "playlist_previous": previous_item,
-            "playlist_next": next_item,
-        }
-        return nav
-
-    def delete_metadata(self):
-        """delete metadata for playlist"""
-        script = (
-            "ctx._source.playlist.removeAll("
-            + "Collections.singleton(params.playlist)) "
-        )
-        data = {
-            "query": {
-                "term": {"playlist.keyword": {"value": self.playlist_id}}
-            },
-            "script": {
-                "source": script,
-                "lang": "painless",
-                "params": {"playlist": self.playlist_id},
-            },
-        }
-        payload = json.dumps(data)
-        url = f"{self.ES_URL}/ta_video/_update_by_query"
-        headers = {"Content-type": "application/json"}
-        response = requests.post(
-            url, data=payload, headers=headers, auth=self.ES_AUTH
-        )
-        if not response.ok:
-            print(response.text)
-
-        self.delete_playlist()
-
-    def delete_videos_playlist(self):
-        """delete playlist with all videos"""
-        print(f"delete playlist {self.playlist_id} with all videos")
-        self.get_playlist_dict()
-        all_youtube_id = [
-            i["youtube_id"]
-            for i in self.playlist_dict["playlist_entries"]
-            if i["downloaded"]
-        ]
-        for youtube_id in all_youtube_id:
-            YoutubeVideo(youtube_id).delete_media_file()
-
-        self.delete_playlist()
-
-    def delete_playlist(self):
-        """delete only playlist document"""
-        url = f"{self.ES_URL}/ta_playlist/_doc/{self.playlist_id}"
-        response = requests.delete(url, auth=self.ES_AUTH)
-        if not response.ok:
-            print(response.text)
-
-    def deactivate(self):
-        """deactivate document on extractor error"""
-        headers = {"Content-type": "application/json"}
-        url = f"{self.ES_URL}/ta_playlist/_update/{self.playlist_id}"
-        data = {"script": "ctx._source.playlist_active = false"}
-        json_str = json.dumps(data)
-        response = requests.post(
-            url, data=json_str, headers=headers, auth=self.ES_AUTH
-        )
-        print(f"deactivated {self.playlist_id}")
-        if not response.ok:
-            print(response.text)
-
-
-class WatchState:
-    """handle watched checkbox for videos and channels"""
-
-    CONFIG = AppConfig().config
-    ES_URL = CONFIG["application"]["es_url"]
-    ES_AUTH = CONFIG["application"]["es_auth"]
-    HEADERS = {"Content-type": "application/json"}
-
-    def __init__(self, youtube_id):
-        self.youtube_id = youtube_id
-        self.stamp = int(datetime.now().strftime("%s"))
-
-    def mark_as_watched(self):
-        """update es with new watched value"""
-        url_type = self.dedect_type()
-        if url_type == "video":
-            self.mark_vid_watched()
-        elif url_type == "channel":
-            self.mark_channel_watched()
-        elif url_type == "playlist":
-            self.mark_playlist_watched()
-
-        print(f"marked {self.youtube_id} as watched")
-
-    def mark_as_unwatched(self):
-        """revert watched state to false"""
-        url_type = self.dedect_type()
-        if url_type == "video":
-            self.mark_vid_watched(revert=True)
-
-        print(f"revert {self.youtube_id} as unwatched")
-
-    def dedect_type(self):
-        """find youtube id type"""
-        print(self.youtube_id)
-        url_process = UrlListParser(self.youtube_id).process_list()
-        url_type = url_process[0]["type"]
-        return url_type
-
-    def mark_vid_watched(self, revert=False):
-        """change watched status of single video"""
-        url = self.ES_URL + "/ta_video/_update/" + self.youtube_id
-        data = {
-            "doc": {"player": {"watched": True, "watched_date": self.stamp}}
-        }
-        if revert:
-            data["doc"]["player"]["watched"] = False
-
-        payload = json.dumps(data)
-        request = requests.post(
-            url, data=payload, headers=self.HEADERS, auth=self.ES_AUTH
-        )
-        if not request.ok:
-            print(request.text)
-            raise ValueError("failed to mark video as watched")
-
-    def mark_channel_watched(self):
-        """change watched status of every video in channel"""
-        data = {
-            "query": {
-                "bool": {
-                    "must": [
-                        {
-                            "term": {
-                                "channel.channel_id": {
-                                    "value": self.youtube_id
-                                }
-                            }
-                        },
-                        {"term": {"player.watched": {"value": False}}},
-                    ]
-                }
-            },
-            "script": {
-                "source": "ctx._source.player['watched'] = true",
-                "lang": "painless",
-            },
-        }
-        payload = json.dumps(data)
-        url = f"{self.ES_URL}/ta_video/_update_by_query"
-        request = requests.post(
-            url, data=payload, headers=self.HEADERS, auth=self.ES_AUTH
-        )
-        if not request.ok:
-            print(request.text)
-            raise ValueError("failed mark channel as watched")
-
-    def mark_playlist_watched(self):
-        """change watched state of all videos in playlist"""
-        data = {
-            "query": {
-                "bool": {
-                    "must": [
-                        {
-                            "term": {
-                                "playlist.keyword": {"value": self.youtube_id}
-                            }
-                        },
-                        {"term": {"player.watched": {"value": False}}},
-                    ]
-                }
-            },
-            "script": {
-                "source": "ctx._source.player['watched'] = true",
-                "lang": "painless",
-            },
-        }
-        payload = json.dumps(data)
-        url = f"{self.ES_URL}/ta_video/_update_by_query"
-        request = requests.post(
-            url, data=payload, headers=self.HEADERS, auth=self.ES_AUTH
-        )
-        if not request.ok:
-            print(request.text)
-            raise ValueError("failed mark playlist as watched")
-
-
-class IndexPaginate:
-    """use search_after to go through whole index"""
-
-    CONFIG = AppConfig().config
-    ES_URL = CONFIG["application"]["es_url"]
-    ES_AUTH = CONFIG["application"]["es_auth"]
-    HEADERS = {"Content-type": "application/json"}
-    DEFAULT_SIZE = 500
-
-    def __init__(self, index_name, data, size=False):
-        self.index_name = index_name
-        self.data = data
-        self.pit_id = False
-        self.size = size
-
-    def get_results(self):
-        """get all results"""
-        self.get_pit()
-        self.validate_data()
-        all_results = self.run_loop()
-        self.clean_pit()
-        return all_results
-
-    def get_pit(self):
-        """get pit for index"""
-        url = f"{self.ES_URL}/{self.index_name}/_pit?keep_alive=10m"
-        response = requests.post(url, auth=self.ES_AUTH)
-        json_data = json.loads(response.text)
-
-        self.pit_id = json_data["id"]
-
-    def validate_data(self):
-        """add pit and size to data"""
-        if "sort" not in self.data.keys():
-            print(self.data)
-            raise ValueError("missing sort key in data")
-
-        size = self.size or self.DEFAULT_SIZE
-
-        self.data["size"] = size
-        self.data["pit"] = {"id": self.pit_id, "keep_alive": "10m"}
-
-    def run_loop(self):
-        """loop through results until last hit"""
-        query_str = json.dumps(self.data)
-        url = self.ES_URL + "/_search"
-
-        all_results = []
-        while True:
-            response = requests.get(
-                url, data=query_str, headers=self.HEADERS, auth=self.ES_AUTH
-            )
-            json_data = json.loads(response.text)
-            all_hits = json_data["hits"]["hits"]
-            if all_hits:
-                for hit in all_hits:
-                    source = hit["_source"]
-                    search_after = hit["sort"]
-                    all_results.append(source)
-                # update search_after with last hit data
-                self.data["search_after"] = search_after
-                query_str = json.dumps(self.data)
-            else:
-                break
-
-        return all_results
-
-    def clean_pit(self):
-        """delete pit from elastic search"""
-        query_str = json.dumps({"id": self.pit_id})
-        requests.delete(
-            self.ES_URL + "/_pit",
-            data=query_str,
-            headers=self.HEADERS,
-            auth=self.ES_AUTH,
-        )
-
-
-def index_new_video(youtube_id, missing_vid=False):
-    """combine video and channel classes for new video index"""
-    vid_handler = YoutubeVideo(youtube_id)
-    vid_handler.get_vid_dict()
-    if not vid_handler.vid_dict:
-        raise ValueError("failed to get metadata for " + youtube_id)
-
-    channel_handler = YoutubeChannel(vid_handler.channel_id)
-    # add filepath to vid_dict
-    channel_name = channel_handler.channel_dict["channel_name"]
-    vid_handler.build_file_path(channel_name)
-    # add channel and player to video
-    vid_handler.add_player(missing_vid)
-    vid_handler.vid_dict["channel"] = channel_handler.channel_dict
-    # add new channel to es
-    if channel_handler.source == "scraped":
-        channel_handler.channel_dict["channel_subscribed"] = False
-        channel_handler.upload_to_es()
-        channel_handler.get_channel_art()
-    # upload video to es
-    vid_handler.upload_to_es()
-    # return vid_dict for further processing
-    return vid_handler.vid_dict

tubearchivist/home/src/index/channel.py

@@ -0,0 +1,269 @@
+"""
+functionality:
+- get metadata from youtube for a channel
+- index and update in es
+"""
+
+import json
+import os
+import re
+from datetime import datetime
+
+import requests
+import yt_dlp
+from bs4 import BeautifulSoup
+from home.src.download.thumbnails import ThumbManager
+from home.src.es.connect import ElasticWrap, IndexPaginate
+from home.src.index.generic import YouTubeItem
+from home.src.index.playlist import YoutubePlaylist
+from home.src.ta.helper import clean_string
+
+
+class ChannelScraper:
+    """custom scraper using bs4 to scrape channel about page
+    will be able to be integrated into yt-dlp
+    once #2237 and #2350 are merged upstream
+    """
+
+    def __init__(self, channel_id):
+        self.channel_id = channel_id
+        self.soup = False
+        self.yt_json = False
+        self.json_data = False
+
+    def get_json(self):
+        """main method to return channel dict"""
+        self.get_soup()
+        self._extract_yt_json()
+        self._parse_channel_main()
+        self._parse_channel_meta()
+        return self.json_data
+
+    def get_soup(self):
+        """return soup from youtube"""
+        print(f"{self.channel_id}: scrape channel data from youtube")
+        url = f"https://www.youtube.com/channel/{self.channel_id}/about?hl=en"
+        cookies = {"CONSENT": "YES+xxxxxxxxxxxxxxxxxxxxxxxxxxx"}
+        response = requests.get(url, cookies=cookies)
+        if response.ok:
+            channel_page = response.text
+        else:
+            print(f"{self.channel_id}: failed to extract channel info")
+            raise ConnectionError
+        self.soup = BeautifulSoup(channel_page, "html.parser")
+
+    def _extract_yt_json(self):
+        """parse soup and get ytInitialData json"""
+        all_scripts = self.soup.find("body").find_all("script")
+        for script in all_scripts:
+            if "var ytInitialData = " in str(script):
+                script_content = str(script)
+                break
+        # extract payload
+        script_content = script_content.split("var ytInitialData = ")[1]
+        json_raw = script_content.rstrip(";</script>")
+        self.yt_json = json.loads(json_raw)
+
+    def _parse_channel_main(self):
+        """extract maintab values from scraped channel json data"""
+        main_tab = self.yt_json["header"]["c4TabbedHeaderRenderer"]
+        # build and return dict
+        self.json_data = {
+            "channel_active": True,
+            "channel_last_refresh": int(datetime.now().strftime("%s")),
+            "channel_subs": self._get_channel_subs(main_tab),
+            "channel_name": main_tab["title"],
+            "channel_banner_url": self._get_thumbnails(main_tab, "banner"),
+            "channel_tvart_url": self._get_thumbnails(main_tab, "tvBanner"),
+            "channel_id": self.channel_id,
+            "channel_subscribed": False,
+        }
+
+    @staticmethod
+    def _get_thumbnails(main_tab, thumb_name):
+        """extract banner url from main_tab"""
+        try:
+            all_banners = main_tab[thumb_name]["thumbnails"]
+            banner = sorted(all_banners, key=lambda k: k["width"])[-1]["url"]
+        except KeyError:
+            banner = False
+
+        return banner
+
+    @staticmethod
+    def _get_channel_subs(main_tab):
+        """process main_tab to get channel subs as int"""
+        try:
+            sub_text_simple = main_tab["subscriberCountText"]["simpleText"]
+            sub_text = sub_text_simple.split(" ")[0]
+            if sub_text[-1] == "K":
+                channel_subs = int(float(sub_text.replace("K", "")) * 1000)
+            elif sub_text[-1] == "M":
+                channel_subs = int(float(sub_text.replace("M", "")) * 1000000)
+            elif int(sub_text) >= 0:
+                channel_subs = int(sub_text)
+            else:
+                message = f"{sub_text} not dealt with"
+                print(message)
+        except KeyError:
+            channel_subs = 0
+
+        return channel_subs
+
+    def _parse_channel_meta(self):
+        """extract meta tab values from channel payload"""
+        # meta tab
+        meta_tab = self.yt_json["metadata"]["channelMetadataRenderer"]
+        all_thumbs = meta_tab["avatar"]["thumbnails"]
+        thumb_url = sorted(all_thumbs, key=lambda k: k["width"])[-1]["url"]
+        # stats tab
+        renderer = "twoColumnBrowseResultsRenderer"
+        all_tabs = self.yt_json["contents"][renderer]["tabs"]
+        for tab in all_tabs:
+            if "tabRenderer" in tab.keys():
+                if tab["tabRenderer"]["title"] == "About":
+                    about_tab = tab["tabRenderer"]["content"][
+                        "sectionListRenderer"
+                    ]["contents"][0]["itemSectionRenderer"]["contents"][0][
+                        "channelAboutFullMetadataRenderer"
+                    ]
+                    break
+        try:
+            channel_views_text = about_tab["viewCountText"]["simpleText"]
+            channel_views = int(re.sub(r"\D", "", channel_views_text))
+        except KeyError:
+            channel_views = 0
+
+        self.json_data.update(
+            {
+                "channel_description": meta_tab["description"],
+                "channel_thumb_url": thumb_url,
+                "channel_views": channel_views,
+            }
+        )
+
+
+class YoutubeChannel(YouTubeItem):
+    """represents a single youtube channel"""
+
+    es_path = False
+    index_name = "ta_channel"
+    yt_base = "https://www.youtube.com/channel/"
+
+    def __init__(self, youtube_id):
+        super().__init__(youtube_id)
+        self.es_path = f"{self.index_name}/_doc/{youtube_id}"
+
+    def build_json(self, upload=False):
+        """get from es or from youtube"""
+        self.get_from_es()
+        if self.json_data:
+            return
+
+        self.get_from_youtube()
+        if upload:
+            self.upload_to_es()
+        return
+
+    def get_from_youtube(self):
+        """use bs4 to scrape channel about page"""
+        self.json_data = ChannelScraper(self.youtube_id).get_json()
+        self.get_channel_art()
+
+    def get_channel_art(self):
+        """download channel art for new channels"""
+        channel_id = self.youtube_id
+        channel_thumb = self.json_data["channel_thumb_url"]
+        channel_banner = self.json_data["channel_banner_url"]
+        ThumbManager().download_chan(
+            [(channel_id, channel_thumb, channel_banner)]
+        )
+
+    def sync_to_videos(self):
+        """sync new channel_dict to all videos of channel"""
+        # add ingest pipeline
+        processors = []
+        for field, value in self.json_data.items():
+            line = {"set": {"field": "channel." + field, "value": value}}
+            processors.append(line)
+        data = {"description": self.youtube_id, "processors": processors}
+        ingest_path = f"_ingest/pipeline/{self.youtube_id}"
+        _, _ = ElasticWrap(ingest_path).put(data)
+        # apply pipeline
+        data = {"query": {"match": {"channel.channel_id": self.youtube_id}}}
+        update_path = f"ta_video/_update_by_query?pipeline={self.youtube_id}"
+        _, _ = ElasticWrap(update_path).post(data)
+
+    def get_folder_path(self):
+        """get folder where media files get stored"""
+        channel_name = self.json_data["channel_name"]
+        folder_name = clean_string(channel_name)
+        if len(folder_name) <= 3:
+            # fall back to channel id
+            folder_name = self.json_data["channel_id"]
+        folder_path = os.path.join(self.app_conf["videos"], folder_name)
+        return folder_path
+
+    def delete_es_videos(self):
+        """delete all channel documents from elasticsearch"""
+        data = {
+            "query": {
+                "term": {"channel.channel_id": {"value": self.youtube_id}}
+            }
+        }
+        _, _ = ElasticWrap("ta_video/_delete_by_query").post(data)
+
+    def delete_playlists(self):
+        """delete all indexed playlist from es"""
+        all_playlists = self.get_indexed_playlists()
+        for playlist in all_playlists:
+            playlist_id = playlist["playlist_id"]
+            YoutubePlaylist(playlist_id).delete_metadata()
+
+    def delete_channel(self):
+        """delete channel and all videos"""
+        print(f"{self.youtube_id}: delete channel")
+        self.get_from_es()
+        folder_path = self.get_folder_path()
+        print(f"{self.youtube_id}: delete all media files")
+        try:
+            all_videos = os.listdir(folder_path)
+            for video in all_videos:
+                video_path = os.path.join(folder_path, video)
+                os.remove(video_path)
+            os.rmdir(folder_path)
+        except FileNotFoundError:
+            print(f"no videos found for {folder_path}")
+
+        print(f"{self.youtube_id}: delete indexed playlists")
+        self.delete_playlists()
+        print(f"{self.youtube_id}: delete indexed videos")
+        self.delete_es_videos()
+        self.del_in_es()
+
+    def get_all_playlists(self):
+        """get all playlists owned by this channel"""
+        url = (
+            f"https://www.youtube.com/channel/{self.youtube_id}"
+            + "/playlists?view=1&sort=dd&shelf_id=0"
+        )
+        obs = {
+            "quiet": True,
+            "skip_download": True,
+            "extract_flat": True,
+        }
+        playlists = yt_dlp.YoutubeDL(obs).extract_info(url)
+        all_entries = [(i["id"], i["title"]) for i in playlists["entries"]]
+
+        return all_entries
+
+    def get_indexed_playlists(self):
+        """get all indexed playlists from channel"""
+        data = {
+            "query": {
+                "term": {"playlist_channel_id": {"value": self.youtube_id}}
+            },
+            "sort": [{"playlist_channel.keyword": {"order": "desc"}}],
+        }
+        all_playlists = IndexPaginate("ta_playlist", data).get_results()
+        return all_playlists

tubearchivist/home/src/index/filesystem.py

@@ -0,0 +1,326 @@
+"""
+Functionality:
+- reindexing old documents
+- syncing updated values between indexes
+- scan the filesystem to delete or index
+"""
+
+import json
+import os
+import re
+import shutil
+import subprocess
+from datetime import datetime
+
+import requests
+from home.src.download.queue import PendingList
+from home.src.download.yt_dlp_handler import VideoDownloader
+from home.src.index.reindex import Reindex
+from home.src.index.video import index_new_video
+from home.src.ta.config import AppConfig
+from home.src.ta.helper import clean_string, ignore_filelist
+from home.src.ta.ta_redis import RedisArchivist
+
+
+class FilesystemScanner:
+    """handle scanning and fixing from filesystem"""
+
+    CONFIG = AppConfig().config
+    ES_URL = CONFIG["application"]["es_url"]
+    ES_AUTH = CONFIG["application"]["es_auth"]
+    VIDEOS = CONFIG["application"]["videos"]
+
+    def __init__(self):
+        self.all_downloaded = self.get_all_downloaded()
+        self.all_indexed = self.get_all_indexed()
+        self.mismatch = None
+        self.to_rename = None
+        self.to_index = None
+        self.to_delete = None
+
+    def get_all_downloaded(self):
+        """get a list of all video files downloaded"""
+        channels = os.listdir(self.VIDEOS)
+        all_channels = ignore_filelist(channels)
+        all_channels.sort()
+        all_downloaded = []
+        for channel_name in all_channels:
+            channel_path = os.path.join(self.VIDEOS, channel_name)
+            channel_files = os.listdir(channel_path)
+            channel_files_clean = ignore_filelist(channel_files)
+            all_videos = [i for i in channel_files_clean if i.endswith(".mp4")]
+            for video in all_videos:
+                youtube_id = video[9:20]
+                all_downloaded.append((channel_name, video, youtube_id))
+
+        return all_downloaded
+
+    @staticmethod
+    def get_all_indexed():
+        """get a list of all indexed videos"""
+        index_handler = PendingList()
+        all_indexed_raw = index_handler.get_all_indexed()
+        all_indexed = []
+        for video in all_indexed_raw:
+            youtube_id = video["youtube_id"]
+            media_url = video["media_url"]
+            published = video["published"]
+            title = video["title"]
+            all_indexed.append((youtube_id, media_url, published, title))
+        return all_indexed
+
+    def list_comarison(self):
+        """compare the lists to figure out what to do"""
+        self.find_unindexed()
+        self.find_missing()
+        self.find_bad_media_url()
+
+    def find_unindexed(self):
+        """find video files without a matching document indexed"""
+        all_indexed_ids = [i[0] for i in self.all_indexed]
+        to_index = []
+        for downloaded in self.all_downloaded:
+            if downloaded[2] not in all_indexed_ids:
+                to_index.append(downloaded)
+
+        self.to_index = to_index
+
+    def find_missing(self):
+        """find indexed videos without matching media file"""
+        all_downloaded_ids = [i[2] for i in self.all_downloaded]
+        to_delete = []
+        for video in self.all_indexed:
+            youtube_id = video[0]
+            if youtube_id not in all_downloaded_ids:
+                to_delete.append(video)
+
+        self.to_delete = to_delete
+
+    def find_bad_media_url(self):
+        """rename media files not matching the indexed title"""
+        to_fix = []
+        to_rename = []
+        for downloaded in self.all_downloaded:
+            channel, filename, downloaded_id = downloaded
+            # find in indexed
+            for indexed in self.all_indexed:
+                indexed_id, media_url, published, title = indexed
+                if indexed_id == downloaded_id:
+                    # found it
+                    title_c = clean_string(title)
+                    pub = published.replace("-", "")
+                    expected_filename = f"{pub}_{indexed_id}_{title_c}.mp4"
+                    new_url = os.path.join(channel, expected_filename)
+                    if expected_filename != filename:
+                        # file to rename
+                        to_rename.append(
+                            (channel, filename, expected_filename)
+                        )
+                    if media_url != new_url:
+                        # media_url to update in es
+                        to_fix.append((indexed_id, new_url))
+
+                    break
+
+        self.mismatch = to_fix
+        self.to_rename = to_rename
+
+    def rename_files(self):
+        """rename media files as identified by find_bad_media_url"""
+        for bad_filename in self.to_rename:
+            channel, filename, expected_filename = bad_filename
+            print(f"renaming [{filename}] to [{expected_filename}]")
+            old_path = os.path.join(self.VIDEOS, channel, filename)
+            new_path = os.path.join(self.VIDEOS, channel, expected_filename)
+            os.rename(old_path, new_path)
+
+    def send_mismatch_bulk(self):
+        """build bulk update"""
+        bulk_list = []
+        for video_mismatch in self.mismatch:
+            youtube_id, media_url = video_mismatch
+            print(f"{youtube_id}: fixing media url {media_url}")
+            action = {"update": {"_id": youtube_id, "_index": "ta_video"}}
+            source = {"doc": {"media_url": media_url}}
+            bulk_list.append(json.dumps(action))
+            bulk_list.append(json.dumps(source))
+        # add last newline
+        bulk_list.append("\n")
+        query_str = "\n".join(bulk_list)
+        # make the call
+        headers = {"Content-type": "application/x-ndjson"}
+        url = self.ES_URL + "/_bulk"
+        request = requests.post(
+            url, data=query_str, headers=headers, auth=self.ES_AUTH
+        )
+        if not request.ok:
+            print(request.text)
+
+    def delete_from_index(self):
+        """find indexed but deleted mediafile"""
+        for indexed in self.to_delete:
+            youtube_id = indexed[0]
+            print(f"deleting {youtube_id} from index")
+            url = self.ES_URL + "/ta_video/_doc/" + youtube_id
+            request = requests.delete(url, auth=self.ES_AUTH)
+            if not request.ok:
+                print(request.text)
+
+
+class ManualImport:
+    """import and indexing existing video files"""
+
+    CONFIG = AppConfig().config
+    CACHE_DIR = CONFIG["application"]["cache_dir"]
+    IMPORT_DIR = os.path.join(CACHE_DIR, "import")
+
+    def __init__(self):
+        self.identified = self.import_folder_parser()
+
+    def import_folder_parser(self):
+        """detect files in import folder"""
+        import_files = os.listdir(self.IMPORT_DIR)
+        to_import = ignore_filelist(import_files)
+        to_import.sort()
+        video_files = [i for i in to_import if not i.endswith(".json")]
+
+        identified = []
+
+        for file_path in video_files:
+
+            file_dict = {"video_file": file_path}
+            file_name, _ = os.path.splitext(file_path)
+
+            matching_json = [
+                i
+                for i in to_import
+                if i.startswith(file_name) and i.endswith(".json")
+            ]
+            if matching_json:
+                json_file = matching_json[0]
+                youtube_id = self.extract_id_from_json(json_file)
+                file_dict.update({"json_file": json_file})
+            else:
+                youtube_id = self.extract_id_from_filename(file_name)
+                file_dict.update({"json_file": False})
+
+            file_dict.update({"youtube_id": youtube_id})
+            identified.append(file_dict)
+
+        return identified
+
+    @staticmethod
+    def extract_id_from_filename(file_name):
+        """
+        look at the file name for the youtube id
+        expects filename ending in [<youtube_id>].<ext>
+        """
+        id_search = re.search(r"\[([a-zA-Z0-9_-]{11})\]$", file_name)
+        if id_search:
+            youtube_id = id_search.group(1)
+            return youtube_id
+
+        print("failed to extract youtube id for: " + file_name)
+        raise Exception
+
+    def extract_id_from_json(self, json_file):
+        """open json file and extract id"""
+        json_path = os.path.join(self.CACHE_DIR, "import", json_file)
+        with open(json_path, "r", encoding="utf-8") as f:
+            json_content = f.read()
+
+        youtube_id = json.loads(json_content)["id"]
+
+        return youtube_id
+
+    def process_import(self):
+        """go through identified media files"""
+
+        all_videos_added = []
+
+        for media_file in self.identified:
+            json_file = media_file["json_file"]
+            video_file = media_file["video_file"]
+            youtube_id = media_file["youtube_id"]
+
+            video_path = os.path.join(self.CACHE_DIR, "import", video_file)
+
+            self.move_to_cache(video_path, youtube_id)
+
+            # identify and archive
+            vid_dict = index_new_video(youtube_id)
+            VideoDownloader([youtube_id]).move_to_archive(vid_dict)
+            youtube_id = vid_dict["youtube_id"]
+            thumb_url = vid_dict["vid_thumb_url"]
+            all_videos_added.append((youtube_id, thumb_url))
+
+            # cleanup
+            if os.path.exists(video_path):
+                os.remove(video_path)
+            if json_file:
+                json_path = os.path.join(self.CACHE_DIR, "import", json_file)
+                os.remove(json_path)
+
+        return all_videos_added
+
+    def move_to_cache(self, video_path, youtube_id):
+        """move identified video file to cache, convert to mp4"""
+        file_name = os.path.split(video_path)[-1]
+        video_file, ext = os.path.splitext(file_name)
+
+        # make sure youtube_id is in filename
+        if youtube_id not in video_file:
+            video_file = f"{video_file}_{youtube_id}"
+
+        # move, convert if needed
+        if ext == ".mp4":
+            new_file = video_file + ext
+            dest_path = os.path.join(self.CACHE_DIR, "download", new_file)
+            shutil.move(video_path, dest_path)
+        else:
+            print(f"processing with ffmpeg: {video_file}")
+            new_file = video_file + ".mp4"
+            dest_path = os.path.join(self.CACHE_DIR, "download", new_file)
+            subprocess.run(
+                [
+                    "ffmpeg",
+                    "-i",
+                    video_path,
+                    dest_path,
+                    "-loglevel",
+                    "warning",
+                    "-stats",
+                ],
+                check=True,
+            )
+
+
+def scan_filesystem():
+    """grouped function to delete and update index"""
+    filesystem_handler = FilesystemScanner()
+    filesystem_handler.list_comarison()
+    if filesystem_handler.to_rename:
+        print("renaming files")
+        filesystem_handler.rename_files()
+    if filesystem_handler.mismatch:
+        print("fixing media urls in index")
+        filesystem_handler.send_mismatch_bulk()
+    if filesystem_handler.to_delete:
+        print("delete metadata from index")
+        filesystem_handler.delete_from_index()
+    if filesystem_handler.to_index:
+        print("index new videos")
+        for missing_vid in filesystem_handler.to_index:
+            youtube_id = missing_vid[2]
+            index_new_video(youtube_id)
+
+
+def reindex_old_documents():
+    """daily refresh of old documents"""
+    # continue if needed
+    reindex_handler = Reindex()
+    reindex_handler.check_outdated()
+    reindex_handler.reindex()
+    # set timestamp
+    now = int(datetime.now().strftime("%s"))
+    RedisArchivist().set_message("last_reindex", now, expire=False)

tubearchivist/home/src/index/generic.py

@@ -0,0 +1,149 @@
+"""
+functionality:
+- generic base class to inherit from for video, channel and playlist
+"""
+
+import math
+
+import yt_dlp
+from home.src.es.connect import ElasticWrap
+from home.src.ta.config import AppConfig
+from home.src.ta.ta_redis import RedisArchivist
+
+
+class YouTubeItem:
+    """base class for youtube"""
+
+    es_path = False
+    index_name = False
+    yt_base = False
+    yt_obs = {
+        "quiet": True,
+        "default_search": "ytsearch",
+        "skip_download": True,
+        "check_formats": "selected",
+        "noplaylist": True,
+    }
+
+    def __init__(self, youtube_id):
+        self.youtube_id = youtube_id
+        self.config = False
+        self.app_conf = False
+        self.youtube_meta = False
+        self.json_data = False
+        self._get_conf()
+
+    def _get_conf(self):
+        """read user conf"""
+        self.config = AppConfig().config
+        self.app_conf = self.config["application"]
+
+    def get_from_youtube(self):
+        """use yt-dlp to get meta data from youtube"""
+        print(f"{self.youtube_id}: get metadata from youtube")
+        try:
+            yt_item = yt_dlp.YoutubeDL(self.yt_obs)
+            response = yt_item.extract_info(self.yt_base + self.youtube_id)
+        except (
+            yt_dlp.utils.ExtractorError,
+            yt_dlp.utils.DownloadError,
+        ):
+            print(f"{self.youtube_id}: failed to get info from youtube")
+            response = False
+
+        self.youtube_meta = response
+
+    def get_from_es(self):
+        """get indexed data from elastic search"""
+        print(f"{self.youtube_id}: get metadata from es")
+        response, _ = ElasticWrap(f"{self.es_path}").get()
+        source = response.get("_source")
+        self.json_data = source
+
+    def upload_to_es(self):
+        """add json_data to elastic"""
+        _, _ = ElasticWrap(self.es_path).put(self.json_data, refresh=True)
+
+    def deactivate(self):
+        """deactivate document in es"""
+        print(f"{self.youtube_id}: deactivate document")
+        key_match = {
+            "ta_video": "active",
+            "ta_channel": "channel_active",
+            "ta_playlist": "playlist_active",
+        }
+        update_path = f"{self.index_name}/_update/{self.youtube_id}"
+        data = {
+            "script": f"ctx._source.{key_match.get(self.index_name)} = false"
+        }
+        _, _ = ElasticWrap(update_path).post(data)
+
+    def del_in_es(self):
+        """delete item from elastic search"""
+        print(f"{self.youtube_id}: delete from es")
+        _, _ = ElasticWrap(self.es_path).delete()
+
+
+class Pagination:
+    """
+    figure out the pagination based on page size and total_hits
+    """
+
+    def __init__(self, page_get, user_id, search_get=False):
+        self.user_id = user_id
+        self.page_size = self.get_page_size()
+        self.page_get = page_get
+        self.search_get = search_get
+        self.pagination = self.first_guess()
+
+    def get_page_size(self):
+        """get default or user modified page_size"""
+        key = f"{self.user_id}:page_size"
+        page_size = RedisArchivist().get_message(key)["status"]
+        if not page_size:
+            config = AppConfig().config
+            page_size = config["archive"]["page_size"]
+
+        return page_size
+
+    def first_guess(self):
+        """build first guess before api call"""
+        page_get = self.page_get
+        if page_get in [0, 1]:
+            page_from = 0
+            prev_pages = False
+        elif page_get > 1:
+            page_from = (page_get - 1) * self.page_size
+            prev_pages = [
+                i for i in range(page_get - 1, page_get - 6, -1) if i > 1
+            ]
+            prev_pages.reverse()
+        pagination = {
+            "page_size": self.page_size,
+            "page_from": page_from,
+            "prev_pages": prev_pages,
+            "current_page": page_get,
+            "max_hits": False,
+        }
+        if self.search_get:
+            pagination.update({"search_get": self.search_get})
+        return pagination
+
+    def validate(self, total_hits):
+        """validate pagination with total_hits after making api call"""
+        page_get = self.page_get
+        max_pages = math.ceil(total_hits / self.page_size)
+        if total_hits > 10000:
+            # es returns maximal 10000 results
+            self.pagination["max_hits"] = True
+            max_pages = max_pages - 1
+
+        if page_get < max_pages and max_pages > 1:
+            self.pagination["last_page"] = max_pages
+        else:
+            self.pagination["last_page"] = False
+        next_pages = [
+            i for i in range(page_get + 1, page_get + 6) if 1 < i < max_pages
+        ]
+
+        self.pagination["next_pages"] = next_pages

tubearchivist/home/src/index/playlist.py

@@ -0,0 +1,208 @@
+"""
+functionality:
+- get metadata from youtube for a playlist
+- index and update in es
+"""
+
+import json
+from datetime import datetime
+
+from home.src.download.thumbnails import ThumbManager
+from home.src.es.connect import ElasticWrap
+from home.src.index.generic import YouTubeItem
+from home.src.index.video import YoutubeVideo
+
+
+class YoutubePlaylist(YouTubeItem):
+    """represents a single youtube playlist"""
+
+    es_path = False
+    index_name = "ta_playlist"
+    yt_obs = {
+        "default_search": "ytsearch",
+        "quiet": True,
+        "skip_download": True,
+        "extract_flat": True,
+    }
+    yt_base = "https://www.youtube.com/playlist?list="
+
+    def __init__(self, youtube_id):
+        super().__init__(youtube_id)
+        self.es_path = f"{self.index_name}/_doc/{youtube_id}"
+        self.all_members = False
+        self.nav = False
+        self.all_youtube_ids = []
+
+    def build_json(self, scrape=False):
+        """collection to create json_data"""
+        self.get_from_es()
+        if self.json_data:
+            subscribed = self.json_data.get("playlist_subscribed")
+        else:
+            subscribed = False
+
+        if scrape or not self.json_data:
+            self.get_from_youtube()
+            self.process_youtube_meta()
+            self.get_entries()
+            self.json_data["playlist_entries"] = self.all_members
+            self.get_playlist_art()
+            self.json_data["playlist_subscribed"] = subscribed
+
+    def process_youtube_meta(self):
+        """extract relevant fields from youtube"""
+        self.json_data = {
+            "playlist_id": self.youtube_id,
+            "playlist_active": True,
+            "playlist_name": self.youtube_meta["title"],
+            "playlist_channel": self.youtube_meta["channel"],
+            "playlist_channel_id": self.youtube_meta["channel_id"],
+            "playlist_thumbnail": self.youtube_meta["thumbnails"][-1]["url"],
+            "playlist_description": self.youtube_meta["description"] or False,
+            "playlist_last_refresh": int(datetime.now().strftime("%s")),
+        }
+
+    def get_entries(self, playlistend=False):
+        """get all videos in playlist"""
+        if playlistend:
+            # implement playlist end
+            print(playlistend)
+        all_members = []
+        for idx, entry in enumerate(self.youtube_meta["entries"]):
+            if self.all_youtube_ids:
+                downloaded = entry["id"] in self.all_youtube_ids
+            else:
+                downloaded = False
+            if not entry["uploader"]:
+                continue
+            to_append = {
+                "youtube_id": entry["id"],
+                "title": entry["title"],
+                "uploader": entry["uploader"],
+                "idx": idx,
+                "downloaded": downloaded,
+            }
+            all_members.append(to_append)
+
+        self.all_members = all_members
+
+    @staticmethod
+    def get_playlist_art():
+        """download artwork of playlist"""
+        thumbnails = ThumbManager()
+        missing_playlists = thumbnails.get_missing_playlists()
+        thumbnails.download_playlist(missing_playlists)
+
+    def add_vids_to_playlist(self):
+        """sync the playlist id to videos"""
+        script = (
+            'if (!ctx._source.containsKey("playlist")) '
+            + "{ctx._source.playlist = [params.playlist]} "
+            + "else if (!ctx._source.playlist.contains(params.playlist)) "
+            + "{ctx._source.playlist.add(params.playlist)} "
+            + "else {ctx.op = 'none'}"
+        )
+
+        bulk_list = []
+        for entry in self.json_data["playlist_entries"]:
+            video_id = entry["youtube_id"]
+            action = {"update": {"_id": video_id, "_index": "ta_video"}}
+            source = {
+                "script": {
+                    "source": script,
+                    "lang": "painless",
+                    "params": {"playlist": self.youtube_id},
+                }
+            }
+            bulk_list.append(json.dumps(action))
+            bulk_list.append(json.dumps(source))
+
+        # add last newline
+        bulk_list.append("\n")
+        query_str = "\n".join(bulk_list)
+
+        ElasticWrap("_bulk").post(query_str, ndjson=True)
+
+    def update_playlist(self):
+        """update metadata for playlist with data from YouTube"""
+        self.get_from_es()
+        subscribed = self.json_data["playlist_subscribed"]
+        self.get_from_youtube()
+        if not self.json_data:
+            # return false to deactivate
+            return False
+
+        self.json_data["playlist_subscribed"] = subscribed
+        self.upload_to_es()
+        return True
+
+    def build_nav(self, youtube_id):
+        """find next and previous in playlist of a given youtube_id"""
+        all_entries_available = self.json_data["playlist_entries"]
+        all_entries = [i for i in all_entries_available if i["downloaded"]]
+        current = [i for i in all_entries if i["youtube_id"] == youtube_id]
+        # stop if not found or playlist of 1
+        if not current or not len(all_entries) > 1:
+            return
+
+        current_idx = all_entries.index(current[0])
+        if current_idx == 0:
+            previous_item = False
+        else:
+            previous_item = all_entries[current_idx - 1]
+            prev_thumb = ThumbManager().vid_thumb_path(
+                previous_item["youtube_id"]
+            )
+            previous_item["vid_thumb"] = prev_thumb
+
+        if current_idx == len(all_entries) - 1:
+            next_item = False
+        else:
+            next_item = all_entries[current_idx + 1]
+            next_thumb = ThumbManager().vid_thumb_path(next_item["youtube_id"])
+            next_item["vid_thumb"] = next_thumb
+
+        self.nav = {
+            "playlist_meta": {
+                "current_idx": current[0]["idx"],
+                "playlist_id": self.youtube_id,
+                "playlist_name": self.json_data["playlist_name"],
+                "playlist_channel": self.json_data["playlist_channel"],
+            },
+            "playlist_previous": previous_item,
+            "playlist_next": next_item,
+        }
+        return
+
+    def delete_metadata(self):
+        """delete metadata for playlist"""
+        script = (
+            "ctx._source.playlist.removeAll("
+            + "Collections.singleton(params.playlist)) "
+        )
+        data = {
+            "query": {
+                "term": {"playlist.keyword": {"value": self.youtube_id}}
+            },
+            "script": {
+                "source": script,
+                "lang": "painless",
+                "params": {"playlist": self.youtube_id},
+            },
+        }
+        _, _ = ElasticWrap("ta_video/_update_by_query").post(data)
+        self.del_in_es()
+
+    def delete_videos_playlist(self):
+        """delete playlist with all videos"""
+        print(f"{self.youtube_id}: delete playlist")
+        self.get_from_es()
+        all_youtube_id = [
+            i["youtube_id"]
+            for i in self.json_data["playlist_entries"]
+            if i["downloaded"]
+        ]
+        for youtube_id in all_youtube_id:
+            YoutubeVideo(youtube_id).delete_media_file()
+
+        self.delete_metadata()

tubearchivist/home/src/index/reindex.py

@@ -0,0 +1,274 @@
+"""
+functionality:
+- periodically refresh documents
+- index and update in es
+"""
+
+import json
+from datetime import datetime
+from math import ceil
+from time import sleep
+
+import requests
+from home.src.download.queue import PendingList
+from home.src.download.subscriptions import ChannelSubscription
+from home.src.download.thumbnails import ThumbManager
+from home.src.index.channel import YoutubeChannel
+from home.src.index.playlist import YoutubePlaylist
+from home.src.index.video import YoutubeVideo
+from home.src.ta.config import AppConfig
+from home.src.ta.helper import get_total_hits
+
+
+class Reindex:
+    """check for outdated documents and refresh data from youtube"""
+
+    def __init__(self):
+        # config
+        config = AppConfig().config
+        self.sleep_interval = config["downloads"]["sleep_interval"]
+        self.es_url = config["application"]["es_url"]
+        self.es_auth = config["application"]["es_auth"]
+        self.refresh_interval = config["scheduler"]["check_reindex_days"]
+        self.integrate_ryd = config["downloads"]["integrate_ryd"]
+        # scan
+        self.all_youtube_ids = False
+        self.all_channel_ids = False
+        self.all_playlist_ids = False
+
+    def get_daily(self):
+        """get daily refresh values"""
+        total_videos = get_total_hits(
+            "ta_video", self.es_url, self.es_auth, "active"
+        )
+        video_daily = ceil(total_videos / self.refresh_interval * 1.2)
+        total_channels = get_total_hits(
+            "ta_channel", self.es_url, self.es_auth, "channel_active"
+        )
+        channel_daily = ceil(total_channels / self.refresh_interval * 1.2)
+        total_playlists = get_total_hits(
+            "ta_playlist", self.es_url, self.es_auth, "playlist_active"
+        )
+        playlist_daily = ceil(total_playlists / self.refresh_interval * 1.2)
+        return (video_daily, channel_daily, playlist_daily)
+
+    def get_outdated_vids(self, size):
+        """get daily videos to refresh"""
+        headers = {"Content-type": "application/json"}
+        now = int(datetime.now().strftime("%s"))
+        now_lte = now - self.refresh_interval * 24 * 60 * 60
+        data = {
+            "size": size,
+            "query": {
+                "bool": {
+                    "must": [
+                        {"match": {"active": True}},
+                        {"range": {"vid_last_refresh": {"lte": now_lte}}},
+                    ]
+                }
+            },
+            "sort": [{"vid_last_refresh": {"order": "asc"}}],
+            "_source": False,
+        }
+        query_str = json.dumps(data)
+        url = self.es_url + "/ta_video/_search"
+        response = requests.get(
+            url, data=query_str, headers=headers, auth=self.es_auth
+        )
+        if not response.ok:
+            print(response.text)
+        response_dict = json.loads(response.text)
+        all_youtube_ids = [i["_id"] for i in response_dict["hits"]["hits"]]
+        return all_youtube_ids
+
+    def get_unrated_vids(self):
+        """get all videos without rating if ryd integration is enabled"""
+        headers = {"Content-type": "application/json"}
+        data = {
+            "size": 200,
+            "query": {
+                "bool": {
+                    "must_not": [{"exists": {"field": "stats.average_rating"}}]
+                }
+            },
+        }
+        query_str = json.dumps(data)
+        url = self.es_url + "/ta_video/_search"
+        response = requests.get(
+            url, data=query_str, headers=headers, auth=self.es_auth
+        )
+        if not response.ok:
+            print(response.text)
+        response_dict = json.loads(response.text)
+        missing_rating = [i["_id"] for i in response_dict["hits"]["hits"]]
+        self.all_youtube_ids = self.all_youtube_ids + missing_rating
+
+    def get_outdated_channels(self, size):
+        """get daily channels to refresh"""
+        headers = {"Content-type": "application/json"}
+        now = int(datetime.now().strftime("%s"))
+        now_lte = now - self.refresh_interval * 24 * 60 * 60
+        data = {
+            "size": size,
+            "query": {
+                "bool": {
+                    "must": [
+                        {"match": {"channel_active": True}},
+                        {"range": {"channel_last_refresh": {"lte": now_lte}}},
+                    ]
+                }
+            },
+            "sort": [{"channel_last_refresh": {"order": "asc"}}],
+            "_source": False,
+        }
+        query_str = json.dumps(data)
+        url = self.es_url + "/ta_channel/_search"
+        response = requests.get(
+            url, data=query_str, headers=headers, auth=self.es_auth
+        )
+        if not response.ok:
+            print(response.text)
+        response_dict = json.loads(response.text)
+        all_channel_ids = [i["_id"] for i in response_dict["hits"]["hits"]]
+        return all_channel_ids
+
+    def get_outdated_playlists(self, size):
+        """get daily outdated playlists to refresh"""
+        headers = {"Content-type": "application/json"}
+        now = int(datetime.now().strftime("%s"))
+        now_lte = now - self.refresh_interval * 24 * 60 * 60
+        data = {
+            "size": size,
+            "query": {
+                "bool": {
+                    "must": [
+                        {"match": {"playlist_active": True}},
+                        {"range": {"playlist_last_refresh": {"lte": now_lte}}},
+                    ]
+                }
+            },
+            "sort": [{"playlist_last_refresh": {"order": "asc"}}],
+            "_source": False,
+        }
+        query_str = json.dumps(data)
+        url = self.es_url + "/ta_playlist/_search"
+        response = requests.get(
+            url, data=query_str, headers=headers, auth=self.es_auth
+        )
+        if not response.ok:
+            print(response.text)
+        response_dict = json.loads(response.text)
+        all_playlist_ids = [i["_id"] for i in response_dict["hits"]["hits"]]
+        return all_playlist_ids
+
+    def check_outdated(self):
+        """add missing vids and channels"""
+        video_daily, channel_daily, playlist_daily = self.get_daily()
+        self.all_youtube_ids = self.get_outdated_vids(video_daily)
+        self.all_channel_ids = self.get_outdated_channels(channel_daily)
+        self.all_playlist_ids = self.get_outdated_playlists(playlist_daily)
+        if self.integrate_ryd:
+            self.get_unrated_vids()
+
+    def rescrape_all_channels(self):
+        """sync new data from channel to all matching videos"""
+        sleep_interval = self.sleep_interval
+        channel_sub_handler = ChannelSubscription()
+        all_channels = channel_sub_handler.get_channels(subscribed_only=False)
+        all_channel_ids = [i["channel_id"] for i in all_channels]
+
+        for channel_id in all_channel_ids:
+            channel = YoutubeChannel(channel_id)
+            subscribed = channel.json_data["channel_subscribed"]
+            channel.get_from_youtube()
+            channel.json_data["channel_subscribed"] = subscribed
+            channel.upload_to_es()
+            channel.sync_to_videos()
+
+            if sleep_interval:
+                sleep(sleep_interval)
+
+    @staticmethod
+    def reindex_single_video(youtube_id):
+        """refresh data for single video"""
+        video = YoutubeVideo(youtube_id)
+
+        # read current state
+        video.get_from_es()
+        player = video.json_data["player"]
+        date_downloaded = video.json_data["date_downloaded"]
+        channel_dict = video.json_data["channel"]
+        playlist = video.json_data.get("playlist")
+
+        # get new
+        video.build_json()
+        if not video.youtube_meta:
+            video.deactivate()
+            return
+
+        video.delete_subtitles()
+        # add back
+        video.json_data["player"] = player
+        video.json_data["date_downloaded"] = date_downloaded
+        video.json_data["channel"] = channel_dict
+        if playlist:
+            video.json_data["playlist"] = playlist
+
+        video.upload_to_es()
+
+        thumb_handler = ThumbManager()
+        thumb_handler.delete_vid_thumb(youtube_id)
+        to_download = (youtube_id, video.json_data["vid_thumb_url"])
+        thumb_handler.download_vid([to_download], notify=False)
+        return
+
+    @staticmethod
+    def reindex_single_channel(channel_id):
+        """refresh channel data and sync to videos"""
+        channel = YoutubeChannel(channel_id)
+        channel.get_from_es()
+        subscribed = channel.json_data["channel_subscribed"]
+        channel.get_from_youtube()
+        channel.json_data["channel_subscribed"] = subscribed
+        channel.upload_to_es()
+        channel.sync_to_videos()
+
+    @staticmethod
+    def reindex_single_playlist(playlist_id, all_indexed_ids):
+        """refresh playlist data"""
+        playlist = YoutubePlaylist(playlist_id)
+        playlist.get_from_es()
+        subscribed = playlist.json_data["playlist_subscribed"]
+        playlist.all_youtube_ids = all_indexed_ids
+        playlist.build_json(scrape=True)
+        if not playlist.json_data:
+            playlist.deactivate()
+            return
+
+        playlist.json_data["playlist_subscribed"] = subscribed
+        playlist.upload_to_es()
+        return
+
+    def reindex(self):
+        """reindex what's needed"""
+        # videos
+        print(f"reindexing {len(self.all_youtube_ids)} videos")
+        for youtube_id in self.all_youtube_ids:
+            self.reindex_single_video(youtube_id)
+            if self.sleep_interval:
+                sleep(self.sleep_interval)
+        # channels
+        print(f"reindexing {len(self.all_channel_ids)} channels")
+        for channel_id in self.all_channel_ids:
+            self.reindex_single_channel(channel_id)
+            if self.sleep_interval:
+                sleep(self.sleep_interval)
+        # playlist
+        print(f"reindexing {len(self.all_playlist_ids)} playlists")
+        if self.all_playlist_ids:
+            all_indexed = PendingList().get_all_indexed()
+            all_indexed_ids = [i["youtube_id"] for i in all_indexed]
+            for playlist_id in self.all_playlist_ids:
+                self.reindex_single_playlist(playlist_id, all_indexed_ids)
+                if self.sleep_interval:
+                    sleep(self.sleep_interval)

tubearchivist/home/src/index/video.py

@@ -0,0 +1,472 @@
+"""
+functionality:
+- get metadata from youtube for a video
+- index and update in es
+"""
+
+import json
+import os
+import re
+from datetime import datetime
+
+import requests
+from home.src.es.connect import ElasticWrap
+from home.src.index import channel as ta_channel
+from home.src.index.generic import YouTubeItem
+from home.src.ta.helper import DurationConverter, clean_string
+from ryd_client import ryd_client
+
+
+class YoutubeSubtitle:
+    """handle video subtitle functionality"""
+
+    def __init__(self, video):
+        self.video = video
+        self.languages = False
+
+    def sub_conf_parse(self):
+        """add additional conf values to self"""
+        languages_raw = self.video.config["downloads"]["subtitle"]
+        if languages_raw:
+            self.languages = [i.strip() for i in languages_raw.split(",")]
+
+    def get_subtitles(self):
+        """check what to do"""
+        self.sub_conf_parse()
+        if not self.languages:
+            # no subtitles
+            return False
+
+        relevant_subtitles = []
+        for lang in self.languages:
+            user_sub = self.get_user_subtitles(lang)
+            if user_sub:
+                relevant_subtitles.append(user_sub)
+                continue
+
+            if self.video.config["downloads"]["subtitle_source"] == "auto":
+                auto_cap = self.get_auto_caption(lang)
+                if auto_cap:
+                    relevant_subtitles.append(auto_cap)
+
+        return relevant_subtitles
+
+    def get_auto_caption(self, lang):
+        """get auto_caption subtitles"""
+        print(f"{self.video.youtube_id}-{lang}: get auto generated subtitles")
+        all_subtitles = self.video.youtube_meta.get("automatic_captions")
+
+        if not all_subtitles:
+            return False
+
+        video_media_url = self.video.json_data["media_url"]
+        media_url = video_media_url.replace(".mp4", f"-{lang}.vtt")
+        all_formats = all_subtitles.get(lang)
+        if not all_formats:
+            return False
+
+        subtitle = [i for i in all_formats if i["ext"] == "vtt"][0]
+        subtitle.update(
+            {"lang": lang, "source": "auto", "media_url": media_url}
+        )
+
+        return subtitle
+
+    def _normalize_lang(self):
+        """normalize country specific language keys"""
+        all_subtitles = self.video.youtube_meta.get("subtitles")
+        if not all_subtitles:
+            return False
+
+        all_keys = list(all_subtitles.keys())
+        for key in all_keys:
+            lang = key.split("-")[0]
+            old = all_subtitles.pop(key)
+            if lang == "live_chat":
+                continue
+            all_subtitles[lang] = old
+
+        return all_subtitles
+
+    def get_user_subtitles(self, lang):
+        """get subtitles uploaded from channel owner"""
+        print(f"{self.video.youtube_id}-{lang}: get user uploaded subtitles")
+        all_subtitles = self._normalize_lang()
+        if not all_subtitles:
+            return False
+
+        video_media_url = self.video.json_data["media_url"]
+        media_url = video_media_url.replace(".mp4", f"-{lang}.vtt")
+        all_formats = all_subtitles.get(lang)
+        if not all_formats:
+            # no user subtitles found
+            return False
+
+        subtitle = [i for i in all_formats if i["ext"] == "vtt"][0]
+        subtitle.update(
+            {"lang": lang, "source": "user", "media_url": media_url}
+        )
+
+        return subtitle
+
+    def download_subtitles(self, relevant_subtitles):
+        """download subtitle files to archive"""
+        videos_base = self.video.config["application"]["videos"]
+        for subtitle in relevant_subtitles:
+            dest_path = os.path.join(videos_base, subtitle["media_url"])
+            source = subtitle["source"]
+            response = requests.get(subtitle["url"])
+            if not response.ok:
+                print(f"{self.video.youtube_id}: failed to download subtitle")
+                continue
+
+            parser = SubtitleParser(response.text, subtitle.get("lang"))
+            parser.process()
+            subtitle_str = parser.get_subtitle_str()
+            self._write_subtitle_file(dest_path, subtitle_str)
+            if self.video.config["downloads"]["subtitle_index"]:
+                query_str = parser.create_bulk_import(self.video, source)
+                self._index_subtitle(query_str)
+
+    @staticmethod
+    def _write_subtitle_file(dest_path, subtitle_str):
+        """write subtitle file to disk"""
+        # create folder here for first video of channel
+        os.makedirs(os.path.split(dest_path)[0], exist_ok=True)
+        with open(dest_path, "w", encoding="utf-8") as subfile:
+            subfile.write(subtitle_str)
+
+    @staticmethod
+    def _index_subtitle(query_str):
+        """send subtitle to es for indexing"""
+        _, _ = ElasticWrap("_bulk").post(data=query_str, ndjson=True)
+
+
+class SubtitleParser:
+    """parse subtitle str from youtube"""
+
+    time_reg = r"^([0-9]{2}:?){3}\.[0-9]{3} --> ([0-9]{2}:?){3}\.[0-9]{3}"
+    stamp_reg = r"<([0-9]{2}:?){3}\.[0-9]{3}>"
+    tag_reg = r"</?c>"
+
+    def __init__(self, subtitle_str, lang):
+        self.subtitle_str = subtitle_str
+        self.lang = lang
+        self.header = False
+        self.parsed_cue_list = False
+        self.all_text_lines = False
+        self.matched = False
+
+    def process(self):
+        """collection to process subtitle string"""
+        self._parse_cues()
+        self._match_text_lines()
+        self._add_id()
+        self._timestamp_check()
+
+    def _parse_cues(self):
+        """split into cues"""
+        all_cues = self.subtitle_str.replace("\n \n", "\n").split("\n\n")
+        self.header = all_cues[0]
+        self.all_text_lines = []
+        self.parsed_cue_list = [self._cue_cleaner(i) for i in all_cues[1:]]
+
+    def _cue_cleaner(self, cue):
+        """parse single cue"""
+        all_lines = cue.split("\n")
+        cue_dict = {"lines": []}
+
+        for line in all_lines:
+            if re.match(self.time_reg, line):
+                clean = re.search(self.time_reg, line).group()
+                start, end = clean.split(" --> ")
+                cue_dict.update({"start": start, "end": end})
+            else:
+                clean = re.sub(self.stamp_reg, "", line)
+                clean = re.sub(self.tag_reg, "", clean)
+                cue_dict["lines"].append(clean)
+                if clean.strip() and clean not in self.all_text_lines[-4:]:
+                    # remove immediate duplicates
+                    self.all_text_lines.append(clean)
+
+        return cue_dict
+
+    def _match_text_lines(self):
+        """match unique text lines with timestamps"""
+
+        self.matched = []
+
+        while self.all_text_lines:
+            check = self.all_text_lines[0]
+            matches = [i for i in self.parsed_cue_list if check in i["lines"]]
+            new_cue = matches[-1]
+            new_cue["start"] = matches[0]["start"]
+
+            for line in new_cue["lines"]:
+                try:
+                    self.all_text_lines.remove(line)
+                except ValueError:
+                    continue
+
+            self.matched.append(new_cue)
+
+    def _timestamp_check(self):
+        """check if end timestamp is bigger than start timestamp"""
+        for idx, cue in enumerate(self.matched):
+            # this
+            end = int(re.sub("[^0-9]", "", cue.get("end")))
+            # next
+            try:
+                next_cue = self.matched[idx + 1]
+            except IndexError:
+                continue
+
+            start_next = int(re.sub("[^0-9]", "", next_cue.get("start")))
+            if end > start_next:
+                self.matched[idx]["end"] = next_cue.get("start")
+
+    def _add_id(self):
+        """add id to matched cues"""
+        for idx, _ in enumerate(self.matched):
+            self.matched[idx]["id"] = idx + 1
+
+    def get_subtitle_str(self):
+        """stitch cues and return processed new string"""
+        new_subtitle_str = self.header + "\n\n"
+
+        for cue in self.matched:
+            timestamp = f"{cue.get('start')} --> {cue.get('end')}"
+            lines = "\n".join(cue.get("lines"))
+            cue_text = f"{cue.get('id')}\n{timestamp}\n{lines}\n\n"
+            new_subtitle_str = new_subtitle_str + cue_text
+
+        return new_subtitle_str
+
+    def create_bulk_import(self, video, source):
+        """process matched for es import"""
+        bulk_list = []
+        channel = video.json_data.get("channel")
+
+        document = {
+            "youtube_id": video.youtube_id,
+            "title": video.json_data.get("title"),
+            "subtitle_channel": channel.get("channel_name"),
+            "subtitle_channel_id": channel.get("channel_id"),
+            "subtitle_last_refresh": int(datetime.now().strftime("%s")),
+            "subtitle_lang": self.lang,
+            "subtitle_source": source,
+        }
+
+        for match in self.matched:
+            match_id = match.get("id")
+            document_id = f"{video.youtube_id}-{self.lang}-{match_id}"
+            action = {"index": {"_index": "ta_subtitle", "_id": document_id}}
+            document.update(
+                {
+                    "subtitle_fragment_id": document_id,
+                    "subtitle_start": match.get("start"),
+                    "subtitle_end": match.get("end"),
+                    "subtitle_index": match_id,
+                    "subtitle_line": " ".join(match.get("lines")),
+                }
+            )
+            bulk_list.append(json.dumps(action))
+            bulk_list.append(json.dumps(document))
+
+        bulk_list.append("\n")
+        query_str = "\n".join(bulk_list)
+
+        return query_str
+
+
+class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
+    """represents a single youtube video"""
+
+    es_path = False
+    index_name = "ta_video"
+    yt_base = "https://www.youtube.com/watch?v="
+
+    def __init__(self, youtube_id):
+        super().__init__(youtube_id)
+        self.channel_id = False
+        self.es_path = f"{self.index_name}/_doc/{youtube_id}"
+
+    def build_json(self):
+        """build json dict of video"""
+        self.get_from_youtube()
+        if not self.youtube_meta:
+            return
+
+        self._process_youtube_meta()
+        self._add_channel()
+        self._add_stats()
+        self.add_file_path()
+        self.add_player()
+        self._check_subtitles()
+        if self.config["downloads"]["integrate_ryd"]:
+            self._get_ryd_stats()
+
+        return
+
+    def _process_youtube_meta(self):
+        """extract relevant fields from youtube"""
+        # extract
+        self.channel_id = self.youtube_meta["channel_id"]
+        upload_date = self.youtube_meta["upload_date"]
+        upload_date_time = datetime.strptime(upload_date, "%Y%m%d")
+        published = upload_date_time.strftime("%Y-%m-%d")
+        last_refresh = int(datetime.now().strftime("%s"))
+        # build json_data basics
+        self.json_data = {
+            "title": self.youtube_meta["title"],
+            "description": self.youtube_meta["description"],
+            "category": self.youtube_meta["categories"],
+            "vid_thumb_url": self.youtube_meta["thumbnail"],
+            "tags": self.youtube_meta["tags"],
+            "published": published,
+            "vid_last_refresh": last_refresh,
+            "date_downloaded": last_refresh,
+            "youtube_id": self.youtube_id,
+            "active": True,
+        }
+
+    def _add_channel(self):
+        """add channel dict to video json_data"""
+        channel = ta_channel.YoutubeChannel(self.channel_id)
+        channel.build_json(upload=True)
+        self.json_data.update({"channel": channel.json_data})
+
+    def _add_stats(self):
+        """add stats dicst to json_data"""
+        # likes
+        like_count = self.youtube_meta.get("like_count", 0)
+        dislike_count = self.youtube_meta.get("dislike_count", 0)
+        self.json_data.update(
+            {
+                "stats": {
+                    "view_count": self.youtube_meta["view_count"],
+                    "like_count": like_count,
+                    "dislike_count": dislike_count,
+                    "average_rating": self.youtube_meta["average_rating"],
+                }
+            }
+        )
+
+    def build_dl_cache_path(self):
+        """find video path in dl cache"""
+        cache_dir = self.app_conf["cache_dir"]
+        cache_path = f"{cache_dir}/download/"
+        all_cached = os.listdir(cache_path)
+        for file_cached in all_cached:
+            if self.youtube_id in file_cached:
+                vid_path = os.path.join(cache_path, file_cached)
+                return vid_path
+
+        raise FileNotFoundError
+
+    def add_player(self):
+        """add player information for new videos"""
+        try:
+            # when indexing from download task
+            vid_path = self.build_dl_cache_path()
+        except FileNotFoundError as err:
+            # when reindexing needs to handle title rename
+            channel = os.path.split(self.json_data["media_url"])[0]
+            channel_dir = os.path.join(self.app_conf["videos"], channel)
+            all_files = os.listdir(channel_dir)
+            for file in all_files:
+                if self.youtube_id in file:
+                    vid_path = os.path.join(channel_dir, file)
+                    break
+            else:
+                raise FileNotFoundError("could not find video file") from err
+
+        duration_handler = DurationConverter()
+        duration = duration_handler.get_sec(vid_path)
+        duration_str = duration_handler.get_str(duration)
+        self.json_data.update(
+            {
+                "player": {
+                    "watched": False,
+                    "duration": duration,
+                    "duration_str": duration_str,
+                }
+            }
+        )
+
+    def add_file_path(self):
+        """build media_url for where file will be located"""
+        channel_name = self.json_data["channel"]["channel_name"]
+        clean_channel_name = clean_string(channel_name)
+        if len(clean_channel_name) <= 3:
+            # fall back to channel id
+            clean_channel_name = self.json_data["channel"]["channel_id"]
+
+        timestamp = self.json_data["published"].replace("-", "")
+        youtube_id = self.json_data["youtube_id"]
+        title = self.json_data["title"]
+        clean_title = clean_string(title)
+        filename = f"{timestamp}_{youtube_id}_{clean_title}.mp4"
+        media_url = os.path.join(clean_channel_name, filename)
+        self.json_data["media_url"] = media_url
+
+    def delete_media_file(self):
+        """delete video file, meta data"""
+        self.get_from_es()
+        video_base = self.app_conf["videos"]
+        to_del = [self.json_data.get("media_url")]
+
+        all_subtitles = self.json_data.get("subtitles")
+        if all_subtitles:
+            to_del = to_del + [i.get("media_url") for i in all_subtitles]
+
+        for media_url in to_del:
+            file_path = os.path.join(video_base, media_url)
+            os.remove(file_path)
+
+        self.del_in_es()
+        self.delete_subtitles()
+
+    def _get_ryd_stats(self):
+        """get optional stats from returnyoutubedislikeapi.com"""
+        try:
+            print(f"{self.youtube_id}: get ryd stats")
+            result = ryd_client.get(self.youtube_id)
+        except requests.exceptions.ConnectionError:
+            print(f"{self.youtube_id}: failed to query ryd api, skipping")
+            return False
+
+        if result["status"] == 404:
+            return False
+
+        dislikes = {
+            "dislike_count": result["dislikes"],
+            "average_rating": result["rating"],
+        }
+        self.json_data["stats"].update(dislikes)
+
+        return True
+
+    def _check_subtitles(self):
+        """optionally add subtitles"""
+        handler = YoutubeSubtitle(self)
+        subtitles = handler.get_subtitles()
+        if subtitles:
+            self.json_data["subtitles"] = subtitles
+            handler.download_subtitles(relevant_subtitles=subtitles)
+
+    def delete_subtitles(self):
+        """delete indexed subtitles"""
+        data = {"query": {"term": {"youtube_id": {"value": self.youtube_id}}}}
+        _, _ = ElasticWrap("ta_subtitle/_delete_by_query").post(data=data)
+
+
+def index_new_video(youtube_id):
+    """combined classes to create new video in index"""
+    video = YoutubeVideo(youtube_id)
+    video.build_json()
+    if not video.json_data:
+        raise ValueError("failed to get metadata for " + youtube_id)
+
+    video.upload_to_es()
+    return video.json_data

tubearchivist/home/src/reindex.py

@@ -1,600 +0,0 @@
-"""
-Functionality:
-- reindexing old documents
-- syncing updated values between indexes
-- scan the filesystem to delete or index
-"""
-
-import json
-import os
-import re
-import shutil
-import subprocess
-from datetime import datetime
-from math import ceil
-from time import sleep
-
-import requests
-from home.src.config import AppConfig
-from home.src.download import ChannelSubscription, PendingList, VideoDownloader
-from home.src.helper import (
-    RedisArchivist,
-    clean_string,
-    get_total_hits,
-    ignore_filelist,
-)
-from home.src.index import (
-    YoutubeChannel,
-    YoutubePlaylist,
-    YoutubeVideo,
-    index_new_video,
-)
-from home.src.thumbnails import ThumbManager
-
-
-class Reindex:
-    """check for outdated documents and refresh data from youtube"""
-
-    def __init__(self):
-        # config
-        config = AppConfig().config
-        self.sleep_interval = config["downloads"]["sleep_interval"]
-        self.es_url = config["application"]["es_url"]
-        self.es_auth = config["application"]["es_auth"]
-        self.refresh_interval = config["scheduler"]["check_reindex_days"]
-        self.integrate_ryd = config["downloads"]["integrate_ryd"]
-        # scan
-        self.all_youtube_ids = False
-        self.all_channel_ids = False
-        self.all_playlist_ids = False
-
-    def get_daily(self):
-        """get daily refresh values"""
-        total_videos = get_total_hits(
-            "ta_video", self.es_url, self.es_auth, "active"
-        )
-        video_daily = ceil(total_videos / self.refresh_interval * 1.2)
-        total_channels = get_total_hits(
-            "ta_channel", self.es_url, self.es_auth, "channel_active"
-        )
-        channel_daily = ceil(total_channels / self.refresh_interval * 1.2)
-        total_playlists = get_total_hits(
-            "ta_playlist", self.es_url, self.es_auth, "playlist_active"
-        )
-        playlist_daily = ceil(total_playlists / self.refresh_interval * 1.2)
-        return (video_daily, channel_daily, playlist_daily)
-
-    def get_outdated_vids(self, size):
-        """get daily videos to refresh"""
-        headers = {"Content-type": "application/json"}
-        now = int(datetime.now().strftime("%s"))
-        now_lte = now - self.refresh_interval * 24 * 60 * 60
-        data = {
-            "size": size,
-            "query": {
-                "bool": {
-                    "must": [
-                        {"match": {"active": True}},
-                        {"range": {"vid_last_refresh": {"lte": now_lte}}},
-                    ]
-                }
-            },
-            "sort": [{"vid_last_refresh": {"order": "asc"}}],
-            "_source": False,
-        }
-        query_str = json.dumps(data)
-        url = self.es_url + "/ta_video/_search"
-        response = requests.get(
-            url, data=query_str, headers=headers, auth=self.es_auth
-        )
-        if not response.ok:
-            print(response.text)
-        response_dict = json.loads(response.text)
-        all_youtube_ids = [i["_id"] for i in response_dict["hits"]["hits"]]
-        return all_youtube_ids
-
-    def get_unrated_vids(self):
-        """get all videos without rating if ryd integration is enabled"""
-        headers = {"Content-type": "application/json"}
-        data = {
-            "size": 200,
-            "query": {
-                "bool": {
-                    "must_not": [{"exists": {"field": "stats.average_rating"}}]
-                }
-            },
-        }
-        query_str = json.dumps(data)
-        url = self.es_url + "/ta_video/_search"
-        response = requests.get(
-            url, data=query_str, headers=headers, auth=self.es_auth
-        )
-        if not response.ok:
-            print(response.text)
-        response_dict = json.loads(response.text)
-        missing_rating = [i["_id"] for i in response_dict["hits"]["hits"]]
-        self.all_youtube_ids = self.all_youtube_ids + missing_rating
-
-    def get_outdated_channels(self, size):
-        """get daily channels to refresh"""
-        headers = {"Content-type": "application/json"}
-        now = int(datetime.now().strftime("%s"))
-        now_lte = now - self.refresh_interval * 24 * 60 * 60
-        data = {
-            "size": size,
-            "query": {
-                "bool": {
-                    "must": [
-                        {"match": {"channel_active": True}},
-                        {"range": {"channel_last_refresh": {"lte": now_lte}}},
-                    ]
-                }
-            },
-            "sort": [{"channel_last_refresh": {"order": "asc"}}],
-            "_source": False,
-        }
-        query_str = json.dumps(data)
-        url = self.es_url + "/ta_channel/_search"
-        response = requests.get(
-            url, data=query_str, headers=headers, auth=self.es_auth
-        )
-        if not response.ok:
-            print(response.text)
-        response_dict = json.loads(response.text)
-        all_channel_ids = [i["_id"] for i in response_dict["hits"]["hits"]]
-        return all_channel_ids
-
-    def get_outdated_playlists(self, size):
-        """get daily outdated playlists to refresh"""
-        headers = {"Content-type": "application/json"}
-        now = int(datetime.now().strftime("%s"))
-        now_lte = now - self.refresh_interval * 24 * 60 * 60
-        data = {
-            "size": size,
-            "query": {
-                "bool": {
-                    "must": [
-                        {"match": {"playlist_active": True}},
-                        {"range": {"playlist_last_refresh": {"lte": now_lte}}},
-                    ]
-                }
-            },
-            "sort": [{"playlist_last_refresh": {"order": "asc"}}],
-            "_source": False,
-        }
-        query_str = json.dumps(data)
-        url = self.es_url + "/ta_playlist/_search"
-        response = requests.get(
-            url, data=query_str, headers=headers, auth=self.es_auth
-        )
-        if not response.ok:
-            print(response.text)
-        response_dict = json.loads(response.text)
-        all_playlist_ids = [i["_id"] for i in response_dict["hits"]["hits"]]
-        return all_playlist_ids
-
-    def check_outdated(self):
-        """add missing vids and channels"""
-        video_daily, channel_daily, playlist_daily = self.get_daily()
-        self.all_youtube_ids = self.get_outdated_vids(video_daily)
-        self.all_channel_ids = self.get_outdated_channels(channel_daily)
-        self.all_playlist_ids = self.get_outdated_playlists(playlist_daily)
-        if self.integrate_ryd:
-            self.get_unrated_vids()
-
-    def rescrape_all_channels(self):
-        """sync new data from channel to all matching videos"""
-        sleep_interval = self.sleep_interval
-        channel_sub_handler = ChannelSubscription()
-        all_channels = channel_sub_handler.get_channels(subscribed_only=False)
-        all_channel_ids = [i["channel_id"] for i in all_channels]
-
-        counter = 1
-        for channel_id in all_channel_ids:
-            channel_index = YoutubeChannel(channel_id)
-            subscribed = channel_index.channel_dict["channel_subscribed"]
-            channel_index.channel_dict = channel_index.build_channel_dict(
-                scrape=True
-            )
-            channel_index.channel_dict["channel_subscribed"] = subscribed
-            channel_index.upload_to_es()
-            channel_index.sync_to_videos()
-            counter = counter + 1
-            if sleep_interval:
-                sleep(sleep_interval)
-
-    @staticmethod
-    def reindex_single_video(youtube_id):
-        """refresh data for single video"""
-        vid_handler = YoutubeVideo(youtube_id)
-        vid_handler.get_vid_dict()
-        if not vid_handler.vid_dict:
-            # stop if deactivated
-            vid_handler.deactivate()
-            return
-
-        es_vid_dict = vid_handler.get_es_data()
-        player = es_vid_dict["_source"]["player"]
-        date_downloaded = es_vid_dict["_source"]["date_downloaded"]
-        channel_dict = es_vid_dict["_source"]["channel"]
-        channel_name = channel_dict["channel_name"]
-        try:
-            playlist = es_vid_dict["_source"]["playlist"]
-        except KeyError:
-            playlist = False
-
-        vid_handler.build_file_path(channel_name)
-        # add to vid_dict
-        vid_handler.vid_dict["player"] = player
-        vid_handler.vid_dict["date_downloaded"] = date_downloaded
-        vid_handler.vid_dict["channel"] = channel_dict
-        if playlist:
-            vid_handler.vid_dict["playlist"] = playlist
-        # update
-        vid_handler.upload_to_es()
-        thumb_handler = ThumbManager()
-        thumb_handler.delete_vid_thumb(youtube_id)
-        to_download = (youtube_id, vid_handler.vid_dict["vid_thumb_url"])
-        thumb_handler.download_vid([to_download], notify=False)
-
-    @staticmethod
-    def reindex_single_channel(channel_id):
-        """refresh channel data and sync to videos"""
-        channel_handler = YoutubeChannel(channel_id)
-        subscribed = channel_handler.channel_dict["channel_subscribed"]
-        channel_handler.channel_dict = channel_handler.build_channel_dict(
-            scrape=True
-        )
-        channel_handler.channel_dict["channel_subscribed"] = subscribed
-        # update
-        channel_handler.upload_to_es()
-        channel_handler.sync_to_videos()
-        thumb_handler = ThumbManager()
-        thumb_handler.delete_chan_thumb(channel_id)
-        channel_thumb = channel_handler.channel_dict["channel_thumb_url"]
-        channel_banner = channel_handler.channel_dict["channel_banner_url"]
-        to_download = (channel_id, channel_thumb, channel_banner)
-        thumb_handler.download_chan([to_download])
-
-    @staticmethod
-    def reindex_single_playlist(playlist_id, all_indexed_ids):
-        """refresh playlist data"""
-        playlist_handler = YoutubePlaylist(
-            playlist_id, all_youtube_ids=all_indexed_ids
-        )
-        playlist = playlist_handler.update_playlist()
-        if not playlist:
-            playlist_handler.deactivate()
-            return
-
-        playlist_thumbnail = (playlist_id, playlist["playlist_thumbnail"])
-        thumb_handler = ThumbManager()
-        thumb_handler.download_playlist([playlist_thumbnail])
-        return
-
-    def reindex(self):
-        """reindex what's needed"""
-        # videos
-        print(f"reindexing {len(self.all_youtube_ids)} videos")
-        for youtube_id in self.all_youtube_ids:
-            self.reindex_single_video(youtube_id)
-            if self.sleep_interval:
-                sleep(self.sleep_interval)
-        # channels
-        print(f"reindexing {len(self.all_channel_ids)} channels")
-        for channel_id in self.all_channel_ids:
-            self.reindex_single_channel(channel_id)
-            if self.sleep_interval:
-                sleep(self.sleep_interval)
-        # playlist
-        print(f"reindexing {len(self.all_playlist_ids)} playlists")
-        if self.all_playlist_ids:
-            all_indexed = PendingList().get_all_indexed()
-            all_indexed_ids = [i["youtube_id"] for i in all_indexed]
-            for playlist_id in self.all_playlist_ids:
-                self.reindex_single_playlist(playlist_id, all_indexed_ids)
-                if self.sleep_interval:
-                    sleep(self.sleep_interval)
-
-
-class FilesystemScanner:
-    """handle scanning and fixing from filesystem"""
-
-    CONFIG = AppConfig().config
-    ES_URL = CONFIG["application"]["es_url"]
-    ES_AUTH = CONFIG["application"]["es_auth"]
-    VIDEOS = CONFIG["application"]["videos"]
-
-    def __init__(self):
-        self.all_downloaded = self.get_all_downloaded()
-        self.all_indexed = self.get_all_indexed()
-        self.mismatch = None
-        self.to_rename = None
-        self.to_index = None
-        self.to_delete = None
-
-    def get_all_downloaded(self):
-        """get a list of all video files downloaded"""
-        channels = os.listdir(self.VIDEOS)
-        all_channels = ignore_filelist(channels)
-        all_channels.sort()
-        all_downloaded = []
-        for channel_name in all_channels:
-            channel_path = os.path.join(self.VIDEOS, channel_name)
-            videos = os.listdir(channel_path)
-            all_videos = ignore_filelist(videos)
-            for video in all_videos:
-                youtube_id = video[9:20]
-                all_downloaded.append((channel_name, video, youtube_id))
-
-        return all_downloaded
-
-    @staticmethod
-    def get_all_indexed():
-        """get a list of all indexed videos"""
-        index_handler = PendingList()
-        all_indexed_raw = index_handler.get_all_indexed()
-        all_indexed = []
-        for video in all_indexed_raw:
-            youtube_id = video["youtube_id"]
-            media_url = video["media_url"]
-            published = video["published"]
-            title = video["title"]
-            all_indexed.append((youtube_id, media_url, published, title))
-        return all_indexed
-
-    def list_comarison(self):
-        """compare the lists to figure out what to do"""
-        self.find_unindexed()
-        self.find_missing()
-        self.find_bad_media_url()
-
-    def find_unindexed(self):
-        """find video files without a matching document indexed"""
-        all_indexed_ids = [i[0] for i in self.all_indexed]
-        to_index = []
-        for downloaded in self.all_downloaded:
-            if downloaded[2] not in all_indexed_ids:
-                to_index.append(downloaded)
-
-        self.to_index = to_index
-
-    def find_missing(self):
-        """find indexed videos without matching media file"""
-        all_downloaded_ids = [i[2] for i in self.all_downloaded]
-        to_delete = []
-        for video in self.all_indexed:
-            youtube_id = video[0]
-            if youtube_id not in all_downloaded_ids:
-                to_delete.append(video)
-
-        self.to_delete = to_delete
-
-    def find_bad_media_url(self):
-        """rename media files not matching the indexed title"""
-        to_fix = []
-        to_rename = []
-        for downloaded in self.all_downloaded:
-            channel, filename, downloaded_id = downloaded
-            # find in indexed
-            for indexed in self.all_indexed:
-                indexed_id, media_url, published, title = indexed
-                if indexed_id == downloaded_id:
-                    # found it
-                    title_c = clean_string(title)
-                    pub = published.replace("-", "")
-                    expected_filename = f"{pub}_{indexed_id}_{title_c}.mp4"
-                    new_url = os.path.join(channel, expected_filename)
-                    if expected_filename != filename:
-                        # file to rename
-                        to_rename.append(
-                            (channel, filename, expected_filename)
-                        )
-                    if media_url != new_url:
-                        # media_url to update in es
-                        to_fix.append((indexed_id, new_url))
-
-                    break
-
-        self.mismatch = to_fix
-        self.to_rename = to_rename
-
-    def rename_files(self):
-        """rename media files as identified by find_bad_media_url"""
-        for bad_filename in self.to_rename:
-            channel, filename, expected_filename = bad_filename
-            print(f"renaming [{filename}] to [{expected_filename}]")
-            old_path = os.path.join(self.VIDEOS, channel, filename)
-            new_path = os.path.join(self.VIDEOS, channel, expected_filename)
-            os.rename(old_path, new_path)
-
-    def send_mismatch_bulk(self):
-        """build bulk update"""
-        bulk_list = []
-        for video_mismatch in self.mismatch:
-            youtube_id, media_url = video_mismatch
-            print(f"{youtube_id}: fixing media url {media_url}")
-            action = {"update": {"_id": youtube_id, "_index": "ta_video"}}
-            source = {"doc": {"media_url": media_url}}
-            bulk_list.append(json.dumps(action))
-            bulk_list.append(json.dumps(source))
-        # add last newline
-        bulk_list.append("\n")
-        query_str = "\n".join(bulk_list)
-        # make the call
-        headers = {"Content-type": "application/x-ndjson"}
-        url = self.ES_URL + "/_bulk"
-        request = requests.post(
-            url, data=query_str, headers=headers, auth=self.ES_AUTH
-        )
-        if not request.ok:
-            print(request.text)
-
-    def delete_from_index(self):
-        """find indexed but deleted mediafile"""
-        for indexed in self.to_delete:
-            youtube_id = indexed[0]
-            print(f"deleting {youtube_id} from index")
-            url = self.ES_URL + "/ta_video/_doc/" + youtube_id
-            request = requests.delete(url, auth=self.ES_AUTH)
-            if not request.ok:
-                print(request.text)
-
-
-class ManualImport:
-    """import and indexing existing video files"""
-
-    CONFIG = AppConfig().config
-    CACHE_DIR = CONFIG["application"]["cache_dir"]
-    IMPORT_DIR = os.path.join(CACHE_DIR, "import")
-
-    def __init__(self):
-        self.identified = self.import_folder_parser()
-
-    def import_folder_parser(self):
-        """detect files in import folder"""
-        import_files = os.listdir(self.IMPORT_DIR)
-        to_import = ignore_filelist(import_files)
-        to_import.sort()
-        video_files = [i for i in to_import if not i.endswith(".json")]
-
-        identified = []
-
-        for file_path in video_files:
-
-            file_dict = {"video_file": file_path}
-            file_name, _ = os.path.splitext(file_path)
-
-            matching_json = [
-                i
-                for i in to_import
-                if i.startswith(file_name) and i.endswith(".json")
-            ]
-            if matching_json:
-                json_file = matching_json[0]
-                youtube_id = self.extract_id_from_json(json_file)
-                file_dict.update({"json_file": json_file})
-            else:
-                youtube_id = self.extract_id_from_filename(file_name)
-                file_dict.update({"json_file": False})
-
-            file_dict.update({"youtube_id": youtube_id})
-            identified.append(file_dict)
-
-        return identified
-
-    @staticmethod
-    def extract_id_from_filename(file_name):
-        """
-        look at the file name for the youtube id
-        expects filename ending in [<youtube_id>].<ext>
-        """
-        id_search = re.search(r"\[([a-zA-Z0-9_-]{11})\]$", file_name)
-        if id_search:
-            youtube_id = id_search.group(1)
-            return youtube_id
-
-        print("failed to extract youtube id for: " + file_name)
-        raise Exception
-
-    def extract_id_from_json(self, json_file):
-        """open json file and extract id"""
-        json_path = os.path.join(self.CACHE_DIR, "import", json_file)
-        with open(json_path, "r", encoding="utf-8") as f:
-            json_content = f.read()
-
-        youtube_id = json.loads(json_content)["id"]
-
-        return youtube_id
-
-    def process_import(self):
-        """go through identified media files"""
-
-        all_videos_added = []
-
-        for media_file in self.identified:
-            json_file = media_file["json_file"]
-            video_file = media_file["video_file"]
-            youtube_id = media_file["youtube_id"]
-
-            video_path = os.path.join(self.CACHE_DIR, "import", video_file)
-
-            self.move_to_cache(video_path, youtube_id)
-
-            # identify and archive
-            vid_dict = index_new_video(youtube_id)
-            VideoDownloader([youtube_id]).move_to_archive(vid_dict)
-            youtube_id = vid_dict["youtube_id"]
-            thumb_url = vid_dict["vid_thumb_url"]
-            all_videos_added.append((youtube_id, thumb_url))
-
-            # cleanup
-            if os.path.exists(video_path):
-                os.remove(video_path)
-            if json_file:
-                json_path = os.path.join(self.CACHE_DIR, "import", json_file)
-                os.remove(json_path)
-
-        return all_videos_added
-
-    def move_to_cache(self, video_path, youtube_id):
-        """move identified video file to cache, convert to mp4"""
-        file_name = os.path.split(video_path)[-1]
-        video_file, ext = os.path.splitext(file_name)
-
-        # make sure youtube_id is in filename
-        if youtube_id not in video_file:
-            video_file = f"{video_file}_{youtube_id}"
-
-        # move, convert if needed
-        if ext == ".mp4":
-            new_file = video_file + ext
-            dest_path = os.path.join(self.CACHE_DIR, "download", new_file)
-            shutil.move(video_path, dest_path)
-        else:
-            print(f"processing with ffmpeg: {video_file}")
-            new_file = video_file + ".mp4"
-            dest_path = os.path.join(self.CACHE_DIR, "download", new_file)
-            subprocess.run(
-                [
-                    "ffmpeg",
-                    "-i",
-                    video_path,
-                    dest_path,
-                    "-loglevel",
-                    "warning",
-                    "-stats",
-                ],
-                check=True,
-            )
-
-
-def scan_filesystem():
-    """grouped function to delete and update index"""
-    filesystem_handler = FilesystemScanner()
-    filesystem_handler.list_comarison()
-    if filesystem_handler.to_rename:
-        print("renaming files")
-        filesystem_handler.rename_files()
-    if filesystem_handler.mismatch:
-        print("fixing media urls in index")
-        filesystem_handler.send_mismatch_bulk()
-    if filesystem_handler.to_delete:
-        print("delete metadata from index")
-        filesystem_handler.delete_from_index()
-    if filesystem_handler.to_index:
-        print("index new videos")
-        for missing_vid in filesystem_handler.to_index:
-            youtube_id = missing_vid[2]
-            index_new_video(youtube_id, missing_vid=missing_vid)
-
-
-def reindex_old_documents():
-    """daily refresh of old documents"""
-    # continue if needed
-    reindex_handler = Reindex()
-    reindex_handler.check_outdated()
-    reindex_handler.reindex()
-    # set timestamp
-    now = int(datetime.now().strftime("%s"))
-    RedisArchivist().set_message("last_reindex", now, expire=False)

tubearchivist/home/src/ta/config.py

@@ -2,7 +2,6 @@
 Functionality:
 - read and write config
 - load config variables into redis
-- needs to be a separate module to avoid circular import
 """
 
 import json
@@ -10,7 +9,7 @@ import os
 import re
 
 from celery.schedules import crontab
-from home.src.helper import RedisArchivist
+from home.src.ta.ta_redis import RedisArchivist
 
 
 class AppConfig:
@@ -39,8 +38,7 @@ class AppConfig:
     def get_config_file(self):
         """read the defaults from config.json"""
         with open("home/config.json", "r", encoding="utf-8") as f:
-            config_str = f.read()
-            config_file = json.loads(config_str)
+            config_file = json.load(f)
 
         config_file["application"].update(self.get_config_env())
 

tubearchivist/home/src/ta/helper.py

@@ -4,14 +4,12 @@ Loose collection of helper functions
 """
 
 import json
-import os
 import re
 import string
 import subprocess
 import unicodedata
 from urllib.parse import parse_qs, urlparse
 
-import redis
 import requests
 import yt_dlp
 
@@ -149,153 +147,6 @@ class UrlListParser:
         return channel_id
 
 
-class RedisArchivist:
-    """collection of methods to interact with redis"""
-
-    REDIS_HOST = os.environ.get("REDIS_HOST")
-    REDIS_PORT = os.environ.get("REDIS_PORT") or 6379
-    NAME_SPACE = "ta:"
-    CHANNELS = [
-        "download",
-        "add",
-        "rescan",
-        "subchannel",
-        "subplaylist",
-        "playlistscan",
-        "setting",
-    ]
-
-    def __init__(self):
-        self.redis_connection = redis.Redis(
-            host=self.REDIS_HOST, port=self.REDIS_PORT
-        )
-
-    def set_message(self, key, message, expire=True):
-        """write new message to redis"""
-        self.redis_connection.execute_command(
-            "JSON.SET", self.NAME_SPACE + key, ".", json.dumps(message)
-        )
-
-        if expire:
-            if isinstance(expire, bool):
-                secs = 20
-            else:
-                secs = expire
-            self.redis_connection.execute_command(
-                "EXPIRE", self.NAME_SPACE + key, secs
-            )
-
-    def get_message(self, key):
-        """get message dict from redis"""
-        reply = self.redis_connection.execute_command(
-            "JSON.GET", self.NAME_SPACE + key
-        )
-        if reply:
-            json_str = json.loads(reply)
-        else:
-            json_str = {"status": False}
-
-        return json_str
-
-    def del_message(self, key):
-        """delete key from redis"""
-        response = self.redis_connection.execute_command(
-            "DEL", self.NAME_SPACE + key
-        )
-        return response
-
-    def get_lock(self, lock_key):
-        """handle lock for task management"""
-        redis_lock = self.redis_connection.lock(self.NAME_SPACE + lock_key)
-        return redis_lock
-
-    def get_progress(self):
-        """get a list of all progress messages"""
-        all_messages = []
-        for channel in self.CHANNELS:
-            key = "message:" + channel
-            reply = self.redis_connection.execute_command(
-                "JSON.GET", self.NAME_SPACE + key
-            )
-            if reply:
-                json_str = json.loads(reply)
-                all_messages.append(json_str)
-
-        return all_messages
-
-    @staticmethod
-    def monitor_cache_dir(cache_dir):
-        """
-        look at download cache dir directly as alternative progress info
-        """
-        dl_cache = os.path.join(cache_dir, "download")
-        all_cache_file = os.listdir(dl_cache)
-        cache_file = ignore_filelist(all_cache_file)
-        if cache_file:
-            filename = cache_file[0][12:].replace("_", " ").split(".")[0]
-            mess_dict = {
-                "status": "message:download",
-                "level": "info",
-                "title": "Downloading: " + filename,
-                "message": "",
-            }
-        else:
-            return False
-
-        return mess_dict
-
-
-class RedisQueue:
-    """dynamically interact with the download queue in redis"""
-
-    REDIS_HOST = os.environ.get("REDIS_HOST")
-    REDIS_PORT = os.environ.get("REDIS_PORT")
-    NAME_SPACE = "ta:"
-
-    if not REDIS_PORT:
-        REDIS_PORT = 6379
-
-    def __init__(self, key):
-        self.key = self.NAME_SPACE + key
-        self.conn = redis.Redis(host=self.REDIS_HOST, port=self.REDIS_PORT)
-
-    def get_all(self):
-        """return all elements in list"""
-        result = self.conn.execute_command("LRANGE", self.key, 0, -1)
-        all_elements = [i.decode() for i in result]
-        return all_elements
-
-    def add_list(self, to_add):
-        """add list to queue"""
-        self.conn.execute_command("RPUSH", self.key, *to_add)
-
-    def add_priority(self, to_add):
-        """add single video to front of queue"""
-        self.clear_item(to_add)
-        self.conn.execute_command("LPUSH", self.key, to_add)
-
-    def get_next(self):
-        """return next element in the queue, False if none"""
-        result = self.conn.execute_command("LPOP", self.key)
-        if not result:
-            return False
-
-        next_element = result.decode()
-        return next_element
-
-    def clear(self):
-        """delete list from redis"""
-        self.conn.execute_command("DEL", self.key)
-
-    def clear_item(self, to_clear):
-        """remove single item from list if it's there"""
-        self.conn.execute_command("LREM", self.key, 0, to_clear)
-
-    def trim(self, size):
-        """trim the queue based on settings amount"""
-        self.conn.execute_command("LTRIM", self.key, 0, size)
-
-
 class DurationConverter:
     """
     using ffmpeg to get and parse duration from filepath
@@ -318,7 +169,11 @@ class DurationConverter:
             capture_output=True,
             check=True,
         )
-        duration_sec = int(float(duration.stdout.decode().strip()))
+        duration_raw = duration.stdout.decode().strip()
+        if duration_raw == "N/A":
+            return 0
+
+        duration_sec = int(float(duration_raw))
         return duration_sec
 
     @staticmethod

tubearchivist/home/src/ta/ta_redis.py

@@ -0,0 +1,171 @@
+"""
+functionality:
+- interact with redis
+- hold temporary download queue in redis
+"""
+
+import json
+import os
+
+import redis
+from home.src.ta.helper import ignore_filelist
+
+
+class RedisArchivist:
+    """collection of methods to interact with redis"""
+
+    REDIS_HOST = os.environ.get("REDIS_HOST")
+    REDIS_PORT = os.environ.get("REDIS_PORT") or 6379
+    NAME_SPACE = "ta:"
+    CHANNELS = [
+        "download",
+        "add",
+        "rescan",
+        "subchannel",
+        "subplaylist",
+        "playlistscan",
+        "setting",
+    ]
+
+    def __init__(self):
+        self.redis_connection = redis.Redis(
+            host=self.REDIS_HOST, port=self.REDIS_PORT
+        )
+
+    def set_message(self, key, message, expire=True):
+        """write new message to redis"""
+        self.redis_connection.execute_command(
+            "JSON.SET", self.NAME_SPACE + key, ".", json.dumps(message)
+        )
+
+        if expire:
+            if isinstance(expire, bool):
+                secs = 20
+            else:
+                secs = expire
+            self.redis_connection.execute_command(
+                "EXPIRE", self.NAME_SPACE + key, secs
+            )
+
+    def get_message(self, key):
+        """get message dict from redis"""
+        reply = self.redis_connection.execute_command(
+            "JSON.GET", self.NAME_SPACE + key
+        )
+        if reply:
+            json_str = json.loads(reply)
+        else:
+            json_str = {"status": False}
+
+        return json_str
+
+    def list_items(self, query):
+        """list all matches"""
+        reply = self.redis_connection.execute_command(
+            "KEYS", self.NAME_SPACE + query + "*"
+        )
+        all_matches = [i.decode().lstrip(self.NAME_SPACE) for i in reply]
+        all_results = []
+        for match in all_matches:
+            json_str = self.get_message(match)
+            all_results.append(json_str)
+
+        return all_results
+
+    def del_message(self, key):
+        """delete key from redis"""
+        response = self.redis_connection.execute_command(
+            "DEL", self.NAME_SPACE + key
+        )
+        return response
+
+    def get_lock(self, lock_key):
+        """handle lock for task management"""
+        redis_lock = self.redis_connection.lock(self.NAME_SPACE + lock_key)
+        return redis_lock
+
+    def get_progress(self):
+        """get a list of all progress messages"""
+        all_messages = []
+        for channel in self.CHANNELS:
+            key = "message:" + channel
+            reply = self.redis_connection.execute_command(
+                "JSON.GET", self.NAME_SPACE + key
+            )
+            if reply:
+                json_str = json.loads(reply)
+                all_messages.append(json_str)
+
+        return all_messages
+
+    @staticmethod
+    def monitor_cache_dir(cache_dir):
+        """
+        look at download cache dir directly as alternative progress info
+        """
+        dl_cache = os.path.join(cache_dir, "download")
+        all_cache_file = os.listdir(dl_cache)
+        cache_file = ignore_filelist(all_cache_file)
+        if cache_file:
+            filename = cache_file[0][12:].replace("_", " ").split(".")[0]
+            mess_dict = {
+                "status": "message:download",
+                "level": "info",
+                "title": "Downloading: " + filename,
+                "message": "",
+            }
+        else:
+            return False
+
+        return mess_dict
+
+
+class RedisQueue:
+    """dynamically interact with the download queue in redis"""
+
+    REDIS_HOST = os.environ.get("REDIS_HOST")
+    REDIS_PORT = os.environ.get("REDIS_PORT")
+    NAME_SPACE = "ta:"
+
+    if not REDIS_PORT:
+        REDIS_PORT = 6379
+
+    def __init__(self, key):
+        self.key = self.NAME_SPACE + key
+        self.conn = redis.Redis(host=self.REDIS_HOST, port=self.REDIS_PORT)
+
+    def get_all(self):
+        """return all elements in list"""
+        result = self.conn.execute_command("LRANGE", self.key, 0, -1)
+        all_elements = [i.decode() for i in result]
+        return all_elements
+
+    def add_list(self, to_add):
+        """add list to queue"""
+        self.conn.execute_command("RPUSH", self.key, *to_add)
+
+    def add_priority(self, to_add):
+        """add single video to front of queue"""
+        self.clear_item(to_add)
+        self.conn.execute_command("LPUSH", self.key, to_add)
+
+    def get_next(self):
+        """return next element in the queue, False if none"""
+        result = self.conn.execute_command("LPOP", self.key)
+        if not result:
+            return False
+
+        next_element = result.decode()
+        return next_element
+
+    def clear(self):
+        """delete list from redis"""
+        self.conn.execute_command("DEL", self.key)
+
+    def clear_item(self, to_clear):
+        """remove single item from list if it's there"""
+        self.conn.execute_command("LREM", self.key, 0, to_clear)
+
+    def trim(self, size):
+        """trim the queue based on settings amount"""
+        self.conn.execute_command("LTRIM", self.key, 0, size)

tubearchivist/home/tasks.py

@@ -10,22 +10,24 @@ import os
 
 import home.apps as startup_apps
 from celery import Celery, shared_task
-from home.src.config import AppConfig, ScheduleBuilder
-from home.src.download import (
+from home.src.download.queue import PendingList
+from home.src.download.subscriptions import (
     ChannelSubscription,
-    PendingList,
     PlaylistSubscription,
-    VideoDownloader,
 )
-from home.src.helper import RedisArchivist, RedisQueue, UrlListParser
-from home.src.index import YoutubeChannel, YoutubePlaylist
-from home.src.index_management import backup_all_indexes, restore_from_backup
-from home.src.reindex import (
+from home.src.download.thumbnails import ThumbManager, validate_thumbnails
+from home.src.download.yt_dlp_handler import VideoDownloader
+from home.src.es.index_setup import backup_all_indexes, restore_from_backup
+from home.src.index.channel import YoutubeChannel
+from home.src.index.filesystem import (
     ManualImport,
     reindex_old_documents,
     scan_filesystem,
 )
-from home.src.thumbnails import ThumbManager, validate_thumbnails
+from home.src.index.playlist import YoutubePlaylist
+from home.src.ta.config import AppConfig, ScheduleBuilder
+from home.src.ta.helper import UrlListParser
+from home.src.ta.ta_redis import RedisArchivist, RedisQueue
 
 CONFIG = AppConfig().config
 REDIS_HOST = os.environ.get("REDIS_HOST")
@@ -266,17 +268,16 @@ def subscribe_to(url_str):
 @shared_task
 def index_channel_playlists(channel_id):
     """add all playlists of channel to index"""
-    channel_handler = YoutubeChannel(channel_id)
-    channel_name = channel_handler.channel_dict["channel_name"]
+    channel = YoutubeChannel(channel_id)
     # notify
     mess_dict = {
         "status": "message:playlistscan",
         "level": "info",
         "title": "Looking for playlists",
-        "message": f'Scanning channel "{channel_name}" in progress',
+        "message": f'Scanning channel "{channel.youtube_id}" in progress',
     }
     RedisArchivist().set_message("message:playlistscan", mess_dict)
-    all_playlists = channel_handler.get_all_playlists()
+    all_playlists = channel.get_all_playlists()
 
     if not all_playlists:
         print(f"no playlists found for channel {channel_id}")
@@ -295,28 +296,29 @@ def index_channel_playlists(channel_id):
         }
         RedisArchivist().set_message("message:playlistscan", mess_dict)
         print("add playlist: " + playlist_title)
-        playlist_handler = YoutubePlaylist(
-            playlist_id, all_youtube_ids=all_youtube_ids
-        )
-        playlist_handler.get_playlist_dict()
-        if not playlist_handler.playlist_dict:
+
+        playlist = YoutubePlaylist(playlist_id)
+        playlist.all_youtube_ids = all_youtube_ids
+        playlist.build_json()
+
+        if not playlist.json_data:
             # skip if not available
             continue
+
         # don't add if no videos downloaded
         downloaded = [
             i
-            for i in playlist_handler.playlist_dict["playlist_entries"]
+            for i in playlist.json_data["playlist_entries"]
             if i["downloaded"]
         ]
         if not downloaded:
             continue
-        playlist_handler.upload_to_es()
-        playlist_handler.add_vids_to_playlist()
+
+        playlist.upload_to_es()
+        playlist.add_vids_to_playlist()
 
     if all_playlists:
-        handler = ThumbManager()
-        missing_playlists = handler.get_missing_playlists()
-        handler.download_playlist(missing_playlists)
+        playlist.get_playlist_art()
 
     return
 

tubearchivist/home/templates/home/base.html

@@ -28,6 +28,10 @@
         <link rel="stylesheet" href="{% static 'css/light.css' %}">
     {% endif %}
     <script type="text/javascript" src="{% static 'script.js' %}"></script>
+    {% if cast %}
+        <script type="text/javascript" src="https://www.gstatic.com/cv/js/sender/v1/cast_sender.js?loadCastFramework=1"></script>
+        <script id="cast-script" type="text/javascript" src="{% static 'cast-videos.js' %}"></script>
+    {% endif %}
 </head>
 <body>
     <div class="main-content">
@@ -65,7 +69,7 @@
                         <img src="{% static 'img/icon-gear.svg' %}" alt="gear-icon" title="Settings">
                     </a>
                     <a href="{% url 'logout' %}">
-                        <img src="{% static 'img/icon-exit.svg' %}" alt="exit-icon" title="Logout">
+                        <img class="alert-hover" src="{% static 'img/icon-exit.svg' %}" alt="exit-icon" title="Logout">
                     </a>
                 </div>
             </div>
@@ -105,9 +109,21 @@
                     {% endif %}
                     {% if pagination.last_page > 0 %}
                         {% if pagination.search_get %}
-                        <a class="pagination-item" href="?page={{ pagination.last_page }}&search={{ pagination.search_get }}">Last ({{ pagination.last_page }})</a>
+                        <a class="pagination-item" href="?page={{ pagination.last_page }}&search={{ pagination.search_get }}">
+                            {% if pagination.max_hits %}
+                                Max ({{ pagination.last_page }})
+                            {% else %}
+                                Last ({{ pagination.last_page }})
+                            {% endif %}
+                        </a>
                         {% else %}
-                        <a class="pagination-item" href="?page={{ pagination.last_page }}">Last ({{ pagination.last_page }})</a>
+                        <a class="pagination-item" href="?page={{ pagination.last_page }}">
+                            {% if pagination.max_hits %}
+                                Max ({{ pagination.last_page }})
+                            {% else %}
+                                Last ({{ pagination.last_page }})
+                            {% endif %}
+                        </a>
                         {% endif %}
                     {% endif %}
                 {% endif %}
@@ -116,7 +132,7 @@
     </div>
     <div class="footer">
         <div class="boxed-content">
-            <span>© 2021 - <script type="text/javascript">document.write(new Date().getFullYear());</script> TubeArchivist v0.1.0 </span><span><a href="{% url 'about' %}">About</a> | <a href="https://github.com/bbilly1/tubearchivist" target="_blank">GitHub</a> | <a href="https://hub.docker.com/r/bbilly1/tubearchivist" target="_blank">Docker Hub</a> | <a href="https://discord.gg/AFwz8nE7BK" target="_blank">Discord</a> | <a href="https://www.reddit.com/r/TubeArchivist/">Reddit</a></span>
+            <span>© 2021 - <script type="text/javascript">document.write(new Date().getFullYear());</script> TubeArchivist v0.1.2 </span><span><a href="{% url 'about' %}">About</a> | <a href="https://github.com/bbilly1/tubearchivist" target="_blank">GitHub</a> | <a href="https://hub.docker.com/r/bbilly1/tubearchivist" target="_blank">Docker Hub</a> | <a href="https://discord.gg/AFwz8nE7BK" target="_blank">Discord</a> | <a href="https://www.reddit.com/r/TubeArchivist/">Reddit</a></span>
         </div>
     </div>
 </body>

tubearchivist/home/templates/home/channel.html

@@ -24,7 +24,7 @@
             <span>Show subscribed only:</span>
             <div class="toggleBox">
                 <input id="show_subed_only" onclick="toggleCheckbox(this)" type="checkbox" {% if show_subed_only %}checked{% endif %}>
-                {% if show_subed_only %}
+                {% if not show_subed_only %}
                     <label for="" class="ofbtn">Off</label>
                 {% else %}
                     <label for="" class="onbtn">On</label>

tubearchivist/home/templates/home/channel_id.html

@@ -71,7 +71,7 @@
             <span>Hide watched videos:</span>
             <div class="toggleBox">
                 <input id="hide_watched" onclick="toggleCheckbox(this)" type="checkbox" {% if hide_watched %}checked{% endif %}>
-                {% if hide_watched %}
+                {% if not hide_watched %}
                     <label for="" class="ofbtn">Off</label>
                 {% else %}
                     <label for="" class="onbtn">On</label>
@@ -106,10 +106,15 @@
         {% if results %}
             {% for video in results %}
                 <div class="video-item {{ view_style }}">
-                    <a href="#player" data-src="/media/{{ video.source.media_url }}" data-thumb="/cache/{{ video.source.vid_thumb_url }}" data-title="{{ video.source.title }}" data-channel="{{ video.source.channel.channel_name }}" data-channel-id="{{ video.source.channel.channel_id }}" data-id="{{ video.source.youtube_id }}" onclick="createPlayer(this)">
+                    <a href="#player" data-id="{{ video.source.youtube_id }}" onclick="createPlayer(this)">
                         <div class="video-thumb-wrap {{ view_style }}">
                             <div class="video-thumb">
                                 <img src="/cache/{{ video.source.vid_thumb_url }}" alt="video-thumb">
+                                {% if video.source.player.progress %}
+                                <div class="video-progress-bar" id="progress-{{ video.source.youtube_id }}" style="width: {{video.source.player.progress}}%;"></div>
+                                {% else %}
+                                <div class="video-progress-bar" id="progress-{{ video.source.youtube_id }}" style="width: 0%;"></div>
+                                {% endif %}
                             </div>
                             <div class="video-play">
                                 <img src="{% static 'img/icon-play.svg' %}" alt="play-icon">
@@ -133,6 +138,7 @@
             {% endfor %}
         {% else %}
             <h2>No videos found...</h2>
+            <p>Try going to the <a href="{% url 'downloads' %}">downloads page</a> to start the scan and download tasks.</p>
         {% endif %}
     </div>
 </div>

tubearchivist/home/templates/home/downloads.html

@@ -33,7 +33,7 @@
             <span>Show only ignored videos:</span>
             <div class="toggleBox">
                 <input id="show_ignored_only" onclick="toggleCheckbox(this)" type="checkbox" {% if show_ignored_only %}checked{% endif %}>
-                {% if show_ignored_only %}
+                {% if not show_ignored_only %}
                     <label for="" class="ofbtn">Off</label>
                 {% else %}
                     <label for="" class="onbtn">On</label>
@@ -54,7 +54,7 @@
             <button onclick="deleteQueue(this)" data-id="pending" title="Delete all pending videos from the queue">Delete all queued</button>
         {% endif %}
     </div>
-    <h3>Total videos: {{ max_hits }}</h3>
+    <h3>Total videos: {{ max_hits }}{% if max_hits == 10000 %}+{% endif %}</h3>
     <div class="dl-list {{ view_style }}">
         {% if results %}
             {% for video in results %}

tubearchivist/home/templates/home/home.html

@@ -10,7 +10,7 @@
             <span>Hide watched:</span>
             <div class="toggleBox">
                 <input id="hide_watched" onclick="toggleCheckbox(this)" type="checkbox" {% if hide_watched %}checked{% endif %}>
-                {% if hide_watched %}
+                {% if not hide_watched %}
                     <label for="" class="ofbtn">Off</label>
                 {% else %}
                     <label for="" class="onbtn">On</label>
@@ -45,10 +45,15 @@
         {% if results %}
             {% for video in results %}
                 <div class="video-item {{ view_style }}">
-                    <a href="#player" data-src="/media/{{ video.source.media_url }}" data-thumb="/cache/{{ video.source.vid_thumb_url }}" data-title="{{ video.source.title }}" data-channel="{{ video.source.channel.channel_name }}" data-channel-id="{{ video.source.channel.channel_id }}" data-id="{{ video.source.youtube_id }}" onclick="createPlayer(this)">
+                    <a href="#player" data-id="{{ video.source.youtube_id }}" onclick="createPlayer(this)">
                         <div class="video-thumb-wrap {{ view_style }}">
                             <div class="video-thumb">
                                 <img src="/cache/{{ video.source.vid_thumb_url }}" alt="video-thumb">
+                                {% if video.source.player.progress %}
+                                <div class="video-progress-bar" id="progress-{{ video.source.youtube_id }}" style="width: {{video.source.player.progress}}%;"></div>
+                                {% else %}
+                                <div class="video-progress-bar" id="progress-{{ video.source.youtube_id }}" style="width: 0%;"></div>
+                                {% endif %}
                             </div>
                             <div class="video-play">
                                 <img src="{% static 'img/icon-play.svg' %}" alt="play-icon">
@@ -73,6 +78,7 @@
             {% endfor %}
         {% else %}
             <h2>No videos found...</h2>
+            <p>If you've already added a channel or playlist, try going to the <a href="{% url 'downloads' %}">downloads page</a> to start the scan and download tasks.</p>
         {% endif %}
     </div>
 </div>

tubearchivist/home/templates/home/playlist.html

@@ -23,7 +23,7 @@
             <span>Show subscribed only:</span>
             <div class="toggleBox">
                 <input id="show_subed_only" onclick="toggleCheckbox(this)" type="checkbox" {% if show_subed_only %}checked{% endif %}>
-                {% if show_subed_only %}
+                {% if not show_subed_only %}
                     <label for="" class="ofbtn">Off</label>
                 {% else %}
                     <label for="" class="onbtn">On</label>

tubearchivist/home/templates/home/playlist_id.html

@@ -68,7 +68,7 @@
             <span>Hide watched videos:</span>
             <div class="toggleBox">
                 <input id="hide_watched" onclick="toggleCheckbox(this)" type="checkbox" {% if hide_watched %}checked{% endif %}>
-                {% if hide_watched %}
+                {% if not hide_watched %}
                     <label for="" class="ofbtn">Off</label>
                 {% else %}
                     <label for="" class="onbtn">On</label>
@@ -87,10 +87,15 @@
         {% if results %}
             {% for video in results %}
                 <div class="video-item {{ view_style }}">
-                    <a href="#player" data-src="/media/{{ video.source.media_url }}" data-thumb="/cache/{{ video.source.vid_thumb_url }}" data-title="{{ video.source.title }}" data-channel="{{ video.source.channel.channel_name }}" data-channel-id="{{ video.source.channel.channel_id }}" data-id="{{ video.source.youtube_id }}" onclick="createPlayer(this)">
+                    <a href="#player" data-id="{{ video.source.youtube_id }}" onclick="createPlayer(this)">
                         <div class="video-thumb-wrap {{ view_style }}">
                             <div class="video-thumb">
                                 <img src="/cache/{{ video.source.vid_thumb_url }}" alt="video-thumb">
+                                {% if video.source.player.progress %}
+                                <div class="video-progress-bar" id="progress-{{ video.source.youtube_id }}" style="width: {{video.source.player.progress}}%;"></div>
+                                {% else %}
+                                <div class="video-progress-bar" id="progress-{{ video.source.youtube_id }}" style="width: 0%;"></div>
+                                {% endif %}
                             </div>
                             <div class="video-play">
                                 <img src="{% static 'img/icon-play.svg' %}" alt="play-icon">
@@ -114,6 +119,7 @@
             {% endfor %}
         {% else %}
             <h2>No videos found...</h2>
+            <p>Try going to the <a href="{% url 'downloads' %}">downloads page</a> to start the scan and download tasks.</p>
         {% endif %}
     </div>
 </div>

tubearchivist/home/templates/home/settings.html

@@ -95,13 +95,44 @@
                 {{ app_form.downloads_add_thumbnail }}
             </div>
         </div>
+        <div class="settings-group">
+            <h2 id="format">Subtitles</h2>
+            <div class="settings-item">
+                <p>Subtitles download setting: <span class="settings-current">{{ config.downloads.subtitle }}</span><br>
+                <i>Choose which subtitles to download, add comma separated two letter language ISO code,<br>
+                    e.g. <span class="settings-current">en, de</span></i><br>
+                {{ app_form.downloads_subtitle }}</p>
+            </div>
+            <div class="settings-item">
+                <p>Subtitle source settings: <span class="settings-current">{{ config.downloads.subtitle_source }}</span></p>
+                <i>Download only user generated, or also less accurate auto generated subtitles.</i><br>
+                {{ app_form.downloads_subtitle_source }}
+            </div>
+            <div class="settings-item">
+                <p>Index and make subtitles searchable: <span class="settings-current">{{ config.downloads.subtitle_index }}</span></p>
+                <i>Store subtitle lines in Elasticsearch. Not recommended for low-end hardware.</i><br>
+                {{ app_form.downloads_subtitle_index }}
+            </div>
+        </div>
         <div class="settings-group">
             <h2 id="integrations">Integrations</h2>
+            <div class="settings-item">
+                <p>API token: <button type="button" onclick="textReveal()" id="text-reveal-button">Show</button></p>
+                <div id="text-reveal" class="description-text">
+                    <p>{{ api_token }}</p>
+                    <button class="danger-button" type="button" onclick="resetToken()">Revoke</button>
+                </div>
+            </div>
             <div class="settings-item">
                 <p>Integrate with <a href="https://returnyoutubedislike.com/">returnyoutubedislike.com</a> to get dislikes and average ratings back: <span class="settings-current">{{ config.downloads.integrate_ryd }}</span></p>
                 <i>Before activating that, make sure you have a scraping sleep interval of at least 3 secs set to avoid ratelimiting issues.</i><br>
                 {{ app_form.downloads_integrate_ryd }}
             </div>
+            <div class="settings-item">
+                <p>Current Cast integration: <span class="settings-current">{{ config.application.enable_cast }}</span></p>
+                <i>Enabling Cast will load an additional JS library from Google. HTTPS and a supported browser are required for this integration.</i><br>
+                {{ app_form.application_enable_cast }}
+            </div>
         </div>
         <button type="submit" name="application-settings">Update Application Configurations</button>
     </form>
@@ -284,4 +315,4 @@
     {% endif %}
 </div>
 <script type="text/javascript" src="{% static 'progress.js' %}"></script>
-{% endblock content %}
+{% endblock content %}

tubearchivist/home/templates/home/video.html

@@ -2,16 +2,13 @@
 {% block content %}
 {% load static %}
 {% load humanize %}
-<div class="video-main">
-    <video 
-        src="/media/{{ video.media_url }}" 
-        poster="/cache/{{ video.vid_thumb_url }}" controls preload="false" 
-        type='video/mp4' width="100%" playsinline>
-    </video>
-</div>
+<div class="video-main"></div>
 <div class="boxed-content">
     <div class="title-bar">
-        <h1>{{ video.title }}</h1>
+        {% if cast %}
+            <google-cast-launcher id="castbutton"></google-cast-launcher>
+        {% endif %}
+        <h1 id="video-title">{{ video.title }}</h1>
     </div>
     <div class="info-box info-box-3">
         <div class="info-box-item">
@@ -45,6 +42,7 @@
                 {% else %}
                     <p>Youtube: Deactivated</p>
                 {% endif %}
+                <a download="" href="/media/{{ video.media_url }}"><button id="download-item">Download File</button></a>
                 <button onclick="deleteConfirm()" id="delete-item">Delete Video</button>
                 <div class="delete-confirm" id="delete-button">
                     <span>Are you sure? </span><button class="danger-button" onclick="deleteVideo(this)" data-id="{{ video.youtube_id }}" data-redirect = "{{ video.channel.channel_id }}">Delete</button> <button onclick="cancelDelete()">Cancel</button>
@@ -53,10 +51,10 @@
         </div>
         <div class="info-box-item">
             <div>
-                <p>Views: {{ video.stats.view_count|intcomma }}</p>
+                <p class="thumb-icon"><img src="{% static 'img/icon-eye.svg' %}" alt="views">: {{ video.stats.view_count|intcomma }}</p>
                 <p class="thumb-icon like"><img src="{% static 'img/icon-thumb.svg' %}" alt="thumbs-up">: {{ video.stats.like_count|intcomma }}</p>
                 {% if video.stats.dislike_count %}
-                    <p class="thumb-icon dislike"><img src="{% static 'img/icon-thumb.svg' %}" alt="thumbs-down">: {{ video.stats.dislike_count|intcomma }}</p>
+                    <p class="thumb-icon"><img class="dislike" src="{% static 'img/icon-thumb.svg' %}" alt="thumbs-down">: {{ video.stats.dislike_count|intcomma }}</p>
                 {% endif %}
                 {% if video.stats.average_rating %}
                     <p class="rating-stars">Rating: 
@@ -114,4 +112,9 @@
         {% endfor %}
     {% endif %}
 </div>
-{% endblock content %}
+<script>
+    var videoData = getVideoData('{{ video.youtube_id }}');
+    var videoProgress = getVideoProgress('{{ video.youtube_id }}').position;
+    window.onload = insertVideoTag(videoData, videoProgress);
+</script>
+{% endblock content %}

tubearchivist/home/views.py

@@ -1,7 +1,7 @@
 """
 Functionality:
 - all views for home app
-- process post data received from frontend via ajax
+- holds base classes to inherit from
 """
 
 import json
@@ -14,7 +14,9 @@ from django.contrib.auth.forms import AuthenticationForm
 from django.http import JsonResponse
 from django.shortcuts import redirect, render
 from django.views import View
-from home.forms import (
+from home.src.es.index_setup import get_available_backups
+from home.src.frontend.api_calls import PostData
+from home.src.frontend.forms import (
     AddToQueueForm,
     ApplicationSettingsForm,
     CustomAuthForm,
@@ -24,13 +26,14 @@ from home.forms import (
     SubscribeToPlaylistForm,
     UserSettingsForm,
 )
-from home.src.config import AppConfig, ScheduleBuilder
-from home.src.frontend import PostData
-from home.src.helper import RedisArchivist, UrlListParser
-from home.src.index import YoutubePlaylist
-from home.src.index_management import get_available_backups
-from home.src.searching import Pagination, SearchHandler
+from home.src.frontend.searching import SearchHandler
+from home.src.index.generic import Pagination
+from home.src.index.playlist import YoutubePlaylist
+from home.src.ta.config import AppConfig, ScheduleBuilder
+from home.src.ta.helper import UrlListParser
+from home.src.ta.ta_redis import RedisArchivist
 from home.tasks import extrac_dl, subscribe_to
+from rest_framework.authtoken.models import Token
 
 
 class ArchivistViewConfig(View):
@@ -110,6 +113,7 @@ class ArchivistViewConfig(View):
 
         self.context = {
             "colors": self.default_conf["application"]["colors"],
+            "cast": self.default_conf["application"]["enable_cast"],
             "sort_by": self._get_sort_by(),
             "sort_order": self._get_sort_order(),
             "view_style": self._get_view_style(),
@@ -165,10 +169,23 @@ class ArchivistResultsView(ArchivistViewConfig):
         }
         self.data = data
 
+    def match_progress(self):
+        """add video progress to result context"""
+        results = RedisArchivist().list_items(f"{self.user_id}:progress:")
+        if not results or not self.context["results"]:
+            return
+
+        progress = {i["youtube_id"]: i["position"] for i in results}
+        for hit in self.context["results"]:
+            video = hit["source"]
+            if video["youtube_id"] in progress:
+                played_sec = progress.get(video["youtube_id"])
+                total = video["player"]["duration"]
+                video["player"]["progress"] = 100 * (played_sec / total)
+
     def single_lookup(self, es_path):
         """retrieve a single item from url"""
-        es_url = self.default_conf["application"]["es_url"]
-        search = SearchHandler(f"{es_url}/{es_path}", data=False)
+        search = SearchHandler(es_path, config=self.default_conf)
         result = search.get_data()[0]["source"]
         return result
 
@@ -187,8 +204,9 @@ class ArchivistResultsView(ArchivistViewConfig):
 
     def find_results(self):
         """add results and pagination to context"""
-        url = self.default_conf["application"]["es_url"] + self.es_search
-        search = SearchHandler(url, self.data)
+        search = SearchHandler(
+            self.es_search, config=self.default_conf, data=self.data
+        )
         self.context["results"] = search.get_data()
         self.pagination_handler.validate(search.max_hits)
         self.context["max_hits"] = search.max_hits
@@ -201,13 +219,14 @@ class HomeView(ArchivistResultsView):
     """
 
     view_origin = "home"
-    es_search = "/ta_video/_search"
+    es_search = "ta_video/_search"
 
     def get(self, request):
         """handle get requests"""
         self.initiate_vars(request)
         self._update_view_data()
         self.find_results()
+        self.match_progress()
 
         return render(request, "home/home.html", self.context)
 
@@ -282,7 +301,7 @@ class DownloadView(ArchivistResultsView):
     """
 
     view_origin = "downloads"
-    es_search = "/ta_download/_search"
+    es_search = "ta_download/_search"
 
     def get(self, request):
         """handle get request"""
@@ -344,13 +363,14 @@ class ChannelIdView(ArchivistResultsView):
     """
 
     view_origin = "home"
-    es_search = "/ta_video/_search"
+    es_search = "ta_video/_search"
 
     def get(self, request, channel_id):
         """get request"""
         self.initiate_vars(request)
         self._update_view_data(channel_id)
         self.find_results()
+        self.match_progress()
 
         if self.context["results"]:
             channel_info = self.context["results"][0]["source"]["channel"]
@@ -393,7 +413,7 @@ class ChannelView(ArchivistResultsView):
     """
 
     view_origin = "channel"
-    es_search = "/ta_channel/_search"
+    es_search = "ta_channel/_search"
 
     def get(self, request):
         """handle get request"""
@@ -443,7 +463,7 @@ class PlaylistIdView(ArchivistResultsView):
     """
 
     view_origin = "home"
-    es_search = "/ta_video/_search"
+    es_search = "ta_video/_search"
 
     def get(self, request, playlist_id):
         """handle get request"""
@@ -452,6 +472,7 @@ class PlaylistIdView(ArchivistResultsView):
         playlist_name = playlist_info["playlist_name"]
         self._update_view_data(playlist_id, playlist_info)
         self.find_results()
+        self.match_progress()
         self.context.update(
             {
                 "title": "Playlist: " + playlist_name,
@@ -519,7 +540,7 @@ class PlaylistView(ArchivistResultsView):
     """
 
     view_origin = "playlist"
-    es_search = "/ta_playlist/_search"
+    es_search = "ta_playlist/_search"
 
     def get(self, request):
         """handle get request"""
@@ -590,10 +611,9 @@ class VideoView(View):
 
     def get(self, request, video_id):
         """get single video"""
-        es_url, colors = self.read_config(user_id=request.user.id)
-        url = f"{es_url}/ta_video/_doc/{video_id}"
-        data = None
-        look_up = SearchHandler(url, data)
+        colors, cast = self.read_config(user_id=request.user.id)
+        path = f"ta_video/_doc/{video_id}"
+        look_up = SearchHandler(path, config=False)
         video_hit = look_up.get_data()
         video_data = video_hit[0]["source"]
         try:
@@ -614,6 +634,7 @@ class VideoView(View):
             "playlist_nav": playlist_nav,
             "title": video_title,
             "colors": colors,
+            "cast": cast,
         }
         return render(request, "home/video.html", context)
 
@@ -622,11 +643,11 @@ class VideoView(View):
         """build playlist nav if available"""
         all_navs = []
         for playlist_id in playlists:
-            handler = YoutubePlaylist(playlist_id)
-            handler.get_playlist_dict()
-            nav = handler.build_nav(video_id)
-            if nav:
-                all_navs.append(nav)
+            playlist = YoutubePlaylist(playlist_id)
+            playlist.get_from_es()
+            playlist.build_nav(video_id)
+            if playlist.nav:
+                all_navs.append(playlist.nav)
 
         return all_navs
 
@@ -634,9 +655,9 @@ class VideoView(View):
     def read_config(user_id):
         """read config file"""
         config_handler = AppConfig(user_id)
-        es_url = config_handler.config["application"]["es_url"]
+        cast = config_handler.config["application"]["enable_cast"]
         colors = config_handler.colors
-        return es_url, colors
+        return colors, cast
 
     @staticmethod
     def star_creator(rating):
@@ -682,8 +703,7 @@ class SettingsView(View):
     take post request from the form to update settings
     """
 
-    @staticmethod
-    def get(request):
+    def get(self, request):
         """read and display current settings"""
         config_handler = AppConfig(request.user.id)
         colors = config_handler.colors
@@ -692,10 +712,12 @@ class SettingsView(View):
         user_form = UserSettingsForm()
         app_form = ApplicationSettingsForm()
         scheduler_form = SchedulerSettingsForm()
+        token = self.get_token(request)
 
         context = {
             "title": "Settings",
             "config": config_handler.config,
+            "api_token": token,
             "colors": colors,
             "available_backups": available_backups,
             "user_form": user_form,
@@ -705,6 +727,13 @@ class SettingsView(View):
 
         return render(request, "home/settings.html", context)
 
+    @staticmethod
+    def get_token(request):
+        """get existing or create new token of user"""
+        # pylint: disable=no-member
+        token = Token.objects.get_or_create(user=request.user)[0]
+        return token
+
     @staticmethod
     def post(request):
         """handle form post to update settings"""
@@ -745,6 +774,11 @@ def process(request):
     if request.method == "POST":
         current_user = request.user.id
         post_dict = json.loads(request.body.decode())
+        if post_dict.get("reset-token"):
+            print("revoke API token")
+            request.user.auth_token.delete()
+            return JsonResponse({"success": True})
+
         post_handler = PostData(post_dict, current_user)
         if post_handler.to_exec:
             task_result = post_handler.run_task()

tubearchivist/requirements.txt

@@ -1,10 +1,12 @@
 beautifulsoup4==4.10.0
 celery==5.2.3
-Django==4.0.1
-Pillow==9.0.0
-redis==4.1.0
+Django==4.0.2
+django-cors-headers==3.11.0
+djangorestframework==3.13.1
+Pillow==9.0.1
+redis==4.1.4
 requests==2.27.1
 ryd-client==0.0.3
 uWSGI==2.0.20
-whitenoise==5.3.0
-yt_dlp==2021.12.27
+whitenoise==6.0.0
+yt_dlp==2022.2.4

tubearchivist/static/.jshintrc

@@ -0,0 +1,3 @@
+{
+    "esversion": 6
+}

tubearchivist/static/cast-videos.js

@@ -0,0 +1,148 @@
+function initializeCastApi() {
+    cast.framework.CastContext.getInstance().setOptions({
+        receiverApplicationId: chrome.cast.media.DEFAULT_MEDIA_RECEIVER_APP_ID, // Use built in reciver app on cast device, see https://developers.google.com/cast/docs/styled_receiver if you want to be able to add a theme, splash screen or watermark. Has a $5 one time fee.
+        autoJoinPolicy: chrome.cast.AutoJoinPolicy.ORIGIN_SCOPED
+    });
+
+    var player = new cast.framework.RemotePlayer();
+    var playerController = new cast.framework.RemotePlayerController(player);
+
+    // Add event listerner to check if a connection to a cast device is initiated
+    playerController.addEventListener(
+        cast.framework.RemotePlayerEventType.IS_CONNECTED_CHANGED, function() { 
+            castConnectionChange(player) 
+        }
+    );
+    playerController.addEventListener(
+        cast.framework.RemotePlayerEventType.CURRENT_TIME_CHANGED, function() { 
+            castVideoProgress(player) 
+        }
+    );
+    playerController.addEventListener(
+        cast.framework.RemotePlayerEventType.IS_PAUSED_CHANGED, function() { 
+            castVideoPaused(player) 
+        }
+    );
+}
+
+
+function castConnectionChange(player) {
+    // If cast connection is initialized start cast
+    if (player.isConnected) {
+        // console.log("Cast Connected.");
+        castStart();
+    } else if (!player.isConnected) {
+        // console.log("Cast Disconnected.");
+    }
+}
+
+function castVideoProgress(player) {
+    var videoId = getVideoPlayerVideoId();
+    if (player.mediaInfo.contentId.includes(videoId)) {
+        var currentTime = player.currentTime;
+        var duration = player.duration;
+        if ((currentTime % 10) <= 1.0 && currentTime != 0 && duration != 0) { // Check progress every 10 seconds or else progress is checked a few times a second
+            postVideoProgress(videoId, currentTime);
+            setProgressBar(videoId, currentTime, duration);
+            if (!getVideoPlayerWatchStatus()) { // Check if video is already marked as watched
+                if (watchedThreshold(currentTime, duration)) {
+                    isWatched(videoId);
+                }
+            }
+        }
+    }
+}
+
+function castVideoPaused(player) {
+    var videoId = getVideoPlayerVideoId();
+    var currentTime = player.currentTime;
+    var duration = player.duration;
+    if (player.mediaInfo != null) {
+        if (player.mediaInfo.contentId.includes(videoId)) {
+            if (currentTime != 0 && duration != 0) {
+                postVideoProgress(videoId, currentTime);
+            }
+        }
+    }
+}
+
+function castStart() {
+    var castSession = cast.framework.CastContext.getInstance().getCurrentSession();
+    // Check if there is already media playing on the cast target to prevent recasting on page reload or switching to another video page
+    if (!castSession.getMediaSession()) { 
+        var videoId = getVideoPlayerVideoId();
+        var videoData = getVideoData(videoId);
+        var contentId = getURL() + videoData.data.media_url;
+        var contentTitle = videoData.data.title;
+        var contentImage = getURL() + videoData.data.vid_thumb_url;
+
+        contentType = 'video/mp4'; // Set content type, only videos right now so it is hard coded
+        contentCurrentTime = getVideoPlayerCurrentTime(); // Get video's current position
+        contentActiveSubtitle = [];
+        // Check if a subtitle is turned on.
+        for (var i = 0; i < getVideoPlayer().textTracks.length; i++) {
+            if (getVideoPlayer().textTracks[i].mode == "showing") {
+                contentActiveSubtitle =[i + 1];
+            }
+        }
+        contentSubtitles = [];
+        var videoSubtitles = videoData.data.subtitles; // Array of subtitles
+        if (typeof(videoSubtitles) != 'undefined' && videoData.config.downloads.subtitle) {
+            for (var i = 0; i < videoSubtitles.length; i++) {
+                subtitle = new chrome.cast.media.Track(i, chrome.cast.media.TrackType.TEXT);
+                subtitle.trackContentId = videoSubtitles[i].media_url;
+                subtitle.trackContentType = 'text/vtt';
+                subtitle.subtype = chrome.cast.media.TextTrackType.SUBTITLES;
+                subtitle.name = videoSubtitles[i].name;
+                subtitle.language = videoSubtitles[i].lang;
+                subtitle.customData = null;
+                contentSubtitles.push(subtitle);
+            }
+        }
+
+        mediaInfo = new chrome.cast.media.MediaInfo(contentId, contentType); // Create MediaInfo var that contains url and content type
+        // mediaInfo.streamType = chrome.cast.media.StreamType.BUFFERED; // Set type of stream, BUFFERED, LIVE, OTHER
+        mediaInfo.metadata = new chrome.cast.media.GenericMediaMetadata(); // Create metadata var and add it to MediaInfo
+        mediaInfo.metadata.title = contentTitle.replace("&amp;", "&"); // Set the video title
+        mediaInfo.metadata.images = [new chrome.cast.Image(contentImage)]; // Set the video thumbnail
+        // mediaInfo.textTrackStyle = new chrome.cast.media.TextTrackStyle();
+        mediaInfo.tracks = contentSubtitles;
+
+        var request = new chrome.cast.media.LoadRequest(mediaInfo); // Create request with the previously set MediaInfo. 
+        // request.queueData = new chrome.cast.media.QueueData(); // See https://developers.google.com/cast/docs/reference/web_sender/chrome.cast.media.QueueData for playlist support.
+        request.currentTime = shiftCurrentTime(contentCurrentTime); // Set video start position based on the browser video position
+        request.activeTrackIds = contentActiveSubtitle; // Set active subtitle based on video player
+        // request.autoplay = false; // Set content to auto play, true by default
+        castSession.loadMedia(request).then(
+            function() { 
+                castSuccessful();
+            }, 
+            function() { 
+                castFailed(errorCode); 
+            }
+        ); // Send request to cast device
+    }   
+}
+
+function shiftCurrentTime(contentCurrentTime) { // Shift media back 3 seconds to prevent missing some of the content
+    if (contentCurrentTime > 5) {
+        return(contentCurrentTime - 3);
+    } else {
+        return(0);
+    }
+}
+
+function castSuccessful() {
+    // console.log('Cast Successful.');
+    getVideoPlayer().pause(); // Pause browser video on successful cast
+}
+
+function castFailed(errorCode) {
+    console.log('Error code: ' + errorCode);
+}
+
+window['__onGCastApiAvailable'] = function(isAvailable) {
+    if (isAvailable) {
+        initializeCastApi();
+    }
+}

tubearchivist/static/css/style.css

@@ -17,6 +17,16 @@ font-family: 'Sen-Regular';
 
 html {
     height: 100%;
+    scrollbar-color: var(--accent-font-dark) #0000;
+}
+
+::-webkit-scrollbar {
+    width: 7px;
+}
+
+::-webkit-scrollbar-thumb {
+    background: var(--accent-font-dark);
+    border-radius: 10px;
 }
 
 body {
@@ -155,8 +165,8 @@ button:hover {
     position: relative;
     width: 70px;
     height: 30px;
-    background-color: var(--accent-font-dark);
-    border-color: var(--accent-font-dark);
+    background-color: var(--accent-font-light);
+    border-color: var(--accent-font-light);
     appearance: none;
     border-radius: 15px;
     transition: 0.4s;
@@ -165,8 +175,8 @@ button:hover {
 }
 
 .toggleBox > input:checked[type="checkbox"] {
-    background-color: var(--accent-font-light);
-    border-color: var(--accent-font-light);
+    background-color: var(--accent-font-dark);
+    border-color: var(--accent-font-dark);
 }
 
 .toggleBox > input[type="checkbox"]::before {
@@ -200,14 +210,15 @@ button:hover {
 }
 
 .toggleBox > .onbtn {
-    right: 35%;
+    right: 70%;
     top: 45%;
     transform: translate(50%,-50%);
     font-family: Sen-Regular, sans-serif;
+
 }
 
 .toggleBox > .ofbtn {
-    left: 0;
+    left: 37%;
     top: 45%;
     transform: translate(50%,-50%);
     font-family: Sen-Regular, sans-serif;
@@ -267,6 +278,18 @@ button:hover {
     filter: var(--img-filter);
 }
 
+#castbutton {
+    float: right;
+    width: 40px;
+    padding: 0 5px;
+    --disconnected-color: var(--accent-font-dark);
+    --connected-color: var(--accent-font-light);
+}
+
+.alert-hover:hover {
+    filter: var(--img-filter-error);
+}
+
 /* top of page */
 .title-bar {
     padding-top: 30px;
@@ -368,8 +391,17 @@ button:hover {
     grid-template-columns: 25% auto;
 }
 
+.video-progress-bar {
+    position: absolute;
+    background-color: var(--accent-font-dark);
+    height: 7px;
+    left: 0;
+    bottom: 3px;
+}
+
 .video-thumb img {
     width: 100%;
+    position: relative;
 }
 
 .video-play img {
@@ -433,6 +465,17 @@ button:hover {
     text-transform: uppercase;
 }
 
+.player-stats {
+    float: right;
+    display: flex;
+    align-items: center;
+    margin-top: 10px;
+}
+
+.player-stats span {
+    margin: 0 5px;
+}
+
 .video-desc-player {
     margin-bottom: 8px;
     display: flex;
@@ -616,7 +659,7 @@ button:hover {
     filter: var(--img-filter);
 }
 
-.thumb-icon.dislike img {
+.dislike {
     transform: rotate(180deg);
 }
 

tubearchivist/static/img/icon-exit.svg

@@ -9,15 +9,15 @@
    xmlns="http://www.w3.org/2000/svg"
    xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
    xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
-   width="500"
-   height="500"
-   viewBox="0 0 132.29197 132.29167"
+   width="210mm"
+   height="210mm"
+   viewBox="0 0 210 210"
    version="1.1"
-   id="svg1303"
+   id="svg1566"
    inkscape:version="0.92.4 (5da689c313, 2019-01-14)"
-   sodipodi:docname="Icons_exit.svg">
+   sodipodi:docname="Icons_exit 05.svg">
   <defs
-     id="defs1297" />
+     id="defs1560" />
   <sodipodi:namedview
      id="base"
      pagecolor="#ffffff"
@@ -25,20 +25,19 @@
      borderopacity="1.0"
      inkscape:pageopacity="0.0"
      inkscape:pageshadow="2"
-     inkscape:zoom="0.66442107"
-     inkscape:cx="161.45413"
-     inkscape:cy="207.61753"
+     inkscape:zoom="0.35355339"
+     inkscape:cx="963.7258"
+     inkscape:cy="291.01609"
      inkscape:document-units="mm"
      inkscape:current-layer="layer1"
      showgrid="false"
-     units="px"
-     inkscape:window-width="1169"
-     inkscape:window-height="893"
-     inkscape:window-x="729"
-     inkscape:window-y="13"
-     inkscape:window-maximized="0" />
+     inkscape:window-width="1920"
+     inkscape:window-height="1009"
+     inkscape:window-x="-8"
+     inkscape:window-y="-8"
+     inkscape:window-maximized="1" />
   <metadata
-     id="metadata1300">
+     id="metadata1563">
     <rdf:RDF>
       <cc:Work
          rdf:about="">
@@ -53,15 +52,24 @@
      inkscape:label="Ebene 1"
      inkscape:groupmode="layer"
      id="layer1"
-     transform="translate(0,-164.70764)">
-    <g
-       id="g855"
-       transform="matrix(1.9016362,0,0,1.9016362,-197.93838,-58.9418)">
-      <path
-         inkscape:connector-curvature="0"
-         id="rect1208"
-         style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;paint-order:markers fill stroke"
-         d="m 124.57603,151.92962 c -0.0433,2.30016 2.0751,4.19245 4.75007,4.24278 l 30.26401,0.43007 -6.00195,5.78023 3.43246,3.56154 10.2778,-9.9006 0.002,0.002 3.5183,-3.3908 -3.42991,-3.564 -9.8737,-10.24989 -3.51834,3.39083 5.84388,6.06803 -30.35875,-0.43185 c -2.67494,-0.0503 -4.86301,1.76094 -4.90629,4.06112 z m -17.65039,-32.01644 v 64.95883 h 7.44347 v -58.27707 h 26.3896 v 18.5229 h 7.44296 v -25.20466 z m 33.83307,39.75416 v 25.20467 h 7.44296 v -25.20467 z" />
-    </g>
+     transform="translate(0,-87)">
+    <path
+       style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:2.35654187;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;paint-order:markers fill stroke"
+       d="M 106.49932,87.901069 C 49.504302,87.900974 3.3006913,134.10459 3.3007713,191.0996 c 0,0.30098 0.003,0.60131 0.005,0.90167 v 0 c -0.003,0.29952 -0.006,0.59901 -0.006,0.89912 -8e-5,56.99502 46.2035307,103.19865 103.1985287,103.19854 23.01714,-0.0773 45.34783,-7.84709 63.44155,-22.07425 0,0 9.01874,-8.71006 2.40579,-16.41737 -6.61297,-7.70731 -19.11222,0.3185 -19.11222,0.3185 -13.60985,9.81394 -29.95596,15.11012 -46.73512,15.14236 -44.275428,0 -80.167758,-35.89234 -80.167758,-80.16778 0,-0.30097 0.003,-0.60148 0.006,-0.90166 h -5.2e-4 c -0.003,-0.29934 -0.006,-0.59901 -0.006,-0.89913 0,-44.27545 35.89234,-80.16777 80.167778,-80.16777 16.77916,0.0322 33.12527,5.32843 46.73512,15.14236 0,0 12.49925,8.02581 19.11222,0.3185 6.61295,-7.70732 -2.4058,-16.41739 -2.4058,-16.41739 C 151.84561,95.74815 129.51494,87.97828 106.4978,87.901069 Z m 54.30959,56.450221 -12.13663,11.69622 20.15864,20.93332 -93.932488,-1.4899 c -9.22763,-0.17349 -16.77655,6.07423 -16.92587,14.00904 l 0.002,0.002 c -0.0149,1.82673 -0.0235,3.40102 0,4.99598 l -0.002,0.002 c 0.14932,7.93483 7.69824,14.18254 16.92587,14.00905 l 93.932488,-1.48991 -20.15864,20.93333 12.13663,11.69622 34.0585,-35.35536 11.82982,-12.29208 h 0.003 l -9.9e-4,-0.002 9.9e-4,-9.9e-4 h -0.003 l -11.82982,-12.29208 z"
+       id="path1405"
+       inkscape:connector-curvature="0"
+       sodipodi:nodetypes="cccccccsccsccsccscccccccccccccccccccccc" />
+    <path
+       style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:2.39729571;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;paint-order:markers fill stroke"
+       d="m 506.57967,92.503023 c -57.98068,-1e-4 -104.98336,47.002567 -104.98326,104.983257 1.9e-4,57.98049 47.00276,104.98284 104.98326,104.98273 23.42489,-0.0758 46.15146,-7.98387 57.83458,-18.08923 11.68313,-10.10537 12.15613,-18.62993 7.38675,-23.04107 v -0.002 c -4.7711,-4.41269 -12.38099,-1.9587 -17.69245,2.25103 -13.83538,9.99805 -30.45915,15.40285 -47.52888,15.4528 -45.04116,0 -81.55421,-36.51305 -81.5542,-81.55419 0,-45.04114 36.51307,-81.5542 81.5542,-81.5542 17.06933,0.0328 33.21884,5.19482 43.16812,12.86758 9.94929,7.67275 17.33418,9.17607 22.1053,4.76338 v -0.002 c 4.77116,-4.41278 5.55882,-12.9887 -0.73482,-18.60197 -18.40654,-14.47308 -41.1234,-22.377337 -64.5386,-22.455877 z m 55.24881,57.426467 -12.34652,11.8985 20.50728,21.29534 -95.55697,-1.51567 c -9.38721,-0.17649 -17.06669,6.17929 -17.21858,14.25133 l 0.003,0.002 c -0.15192,8.07203 7.28245,14.71295 16.66978,14.88953 l 95.22519,1.50947 -21.06332,20.28455 12.04579,12.49846 36.06808,-34.74464 0.005,0.005 12.34654,-11.89954 -12.03701,-12.50724 z m 35.17874,98.71801 0.69918,0.67386 c 0.13539,-0.22412 0.26991,-0.44874 0.4036,-0.67386 z"
+       id="path1405-6"
+       inkscape:connector-curvature="0"
+       sodipodi:nodetypes="ccccccccsczccccccccccccccccccccccc" />
+    <path
+       style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:2.39729571;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;paint-order:markers fill stroke"
+       d="m 740.89945,94.730897 c -57.98068,-9.6e-5 -104.98334,47.002563 -104.98325,104.983253 1.9e-4,57.98049 47.00276,104.98284 104.98325,104.98274 23.42488,-0.0758 46.15145,-7.98387 64.5635,-22.46581 l -17.03461,-16.41553 c -13.83537,9.99805 -30.45916,15.40285 -47.52889,15.4528 -45.04113,0 -81.55419,-36.51306 -81.55419,-81.5542 0,-45.04114 36.51306,-81.55419 81.55419,-81.55419 17.06934,0.0328 33.69814,5.42058 47.54336,15.40423 l 16.99534,-16.3773 c -18.40663,-14.4732 -41.12349,-22.377447 -64.5387,-22.455993 z m 55.24882,57.426473 -12.34653,11.8985 20.50728,21.29534 -95.55696,-1.51567 c -9.38721,-0.17649 -17.06668,6.17928 -17.21858,14.25132 l 0.002,0.002 c -0.1519,8.07203 7.28245,14.71295 16.66978,14.88953 l 95.22519,1.50947 -21.06332,20.28455 12.04578,12.49846 36.06808,-34.74465 0.005,0.005 12.34653,-11.89953 -12.03699,-12.50725 z m 35.17873,98.718 0.69919,0.67386 c 0.13538,-0.22412 0.26991,-0.44874 0.40359,-0.67386 z"
+       id="path1405-9"
+       inkscape:connector-curvature="0"
+       sodipodi:nodetypes="ccccccsccccccccccccccccccccccc" />
   </g>
 </svg>

tubearchivist/static/img/icon-eye.svg

@@ -0,0 +1,63 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   xmlns:dc="http://purl.org/dc/elements/1.1/"
+   xmlns:cc="http://creativecommons.org/ns#"
+   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+   xmlns:svg="http://www.w3.org/2000/svg"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   width="500"
+   height="500"
+   viewBox="0 0 132.29197 132.29167"
+   version="1.1"
+   id="svg1303"
+   inkscape:version="0.92.4 (5da689c313, 2019-01-14)"
+   sodipodi:docname="Icons_seen.svg">
+  <defs
+     id="defs1297" />
+  <sodipodi:namedview
+     id="base"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageopacity="0.0"
+     inkscape:pageshadow="2"
+     inkscape:zoom="1.0105705"
+     inkscape:cx="84.208758"
+     inkscape:cy="136.94831"
+     inkscape:document-units="mm"
+     inkscape:current-layer="layer1"
+     showgrid="false"
+     units="px"
+     inkscape:window-width="1920"
+     inkscape:window-height="1017"
+     inkscape:window-x="-8"
+     inkscape:window-y="-8"
+     inkscape:window-maximized="1" />
+  <metadata
+     id="metadata1300">
+    <rdf:RDF>
+      <cc:Work
+         rdf:about="">
+        <dc:format>image/svg+xml</dc:format>
+        <dc:type
+           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+        <dc:title />
+      </cc:Work>
+    </rdf:RDF>
+  </metadata>
+  <g
+     inkscape:label="Ebene 1"
+     inkscape:groupmode="layer"
+     id="layer1"
+     transform="translate(0,-164.70764)">
+    <path
+       style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;paint-order:markers fill stroke"
+       d="M 66.145984,191.3255 A 71.797122,73.404487 0 0 0 2.5025987,230.88314 71.797122,73.404487 0 0 0 66.145984,270.38145 71.797122,73.404487 0 0 0 129.78937,230.82387 71.797122,73.404487 0 0 0 66.145984,191.3255 Z m -4.921549,7.22394 a 32.724755,32.724755 0 0 0 -13.334395,5.17759 12.828107,12.828107 0 0 1 2.180958,-0.18652 12.828107,12.828107 0 0 1 12.828141,12.82816 12.828107,12.828107 0 0 1 -12.828141,12.82822 12.828107,12.828107 0 0 1 -12.828192,-12.82822 12.828107,12.828107 0 0 1 0.04509,-0.91548 32.724755,32.724755 0 0 0 -3.86581,15.39964 32.724755,32.724755 0 0 0 27.161922,32.24981 A 59.09757,60.420619 0 0 1 13.7604,230.8774 59.09757,60.420619 0 0 1 61.224664,198.54948 Z m 10.482345,0.0563 a 59.09757,60.420619 0 0 1 46.82502,32.22523 59.09757,60.420619 0 0 1 -47.393377,32.32497 32.724755,32.724755 0 0 0 27.73169,-32.30187 32.724755,32.724755 0 0 0 -27.163333,-32.24833 z"
+       id="path1091"
+       inkscape:connector-curvature="0" />
+  </g>
+</svg>

tubearchivist/static/img/icon-thumb.svg

@@ -15,7 +15,7 @@
    version="1.1"
    id="svg1303"
    inkscape:version="0.92.4 (5da689c313, 2019-01-14)"
-   sodipodi:docname="Icons_thumbs.svg">
+   sodipodi:docname="icon_thumb.svg">
   <defs
      id="defs1297" />
   <sodipodi:namedview
@@ -25,18 +25,31 @@
      borderopacity="1.0"
      inkscape:pageopacity="0.0"
      inkscape:pageshadow="2"
-     inkscape:zoom="0.71458126"
-     inkscape:cx="-133.80736"
-     inkscape:cy="163.34297"
+     inkscape:zoom="1.5046797"
+     inkscape:cx="35.718548"
+     inkscape:cy="203.39339"
      inkscape:document-units="mm"
      inkscape:current-layer="layer1"
      showgrid="false"
      units="px"
      inkscape:window-width="1920"
-     inkscape:window-height="1017"
+     inkscape:window-height="1009"
      inkscape:window-x="-8"
      inkscape:window-y="-8"
-     inkscape:window-maximized="1" />
+     inkscape:window-maximized="1"
+     showguides="true"
+     inkscape:guide-bbox="true">
+    <sodipodi:guide
+       position="80.99058,65.965029"
+       orientation="0,1"
+       id="guide816"
+       inkscape:locked="false" />
+    <sodipodi:guide
+       position="65.965178,49.107299"
+       orientation="1,0"
+       id="guide818"
+       inkscape:locked="false" />
+  </sodipodi:namedview>
   <metadata
      id="metadata1300">
     <rdf:RDF>
@@ -55,8 +68,8 @@
      id="layer1"
      transform="translate(0,-164.70764)">
     <path
-       style="opacity:1;fill:#1a1a1a;fill-opacity:1;stroke:none;stroke-width:0;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;paint-order:markers fill stroke"
-       d="m 74.762046,169.84647 c -0.757275,-0.0266 -1.556205,0.0266 -2.398918,0.17006 -17.977757,2.9811 5.863528,29.19697 -15.075776,44.15933 -17.503229,14.60104 -23.870308,10.01972 -24.044759,14.80451 v 47.00292 c 2.623747,6.94761 20.093043,14.05087 38.660635,15.56434 18.567588,1.51346 44.529822,-2.41107 44.816272,-10.49357 0.15245,-7.31882 -4.01785,-7.48655 -4.01785,-7.48655 0,0 9.14077,-0.56532 9.44575,-8.4939 0.15212,-7.77626 -8.85445,-8.88809 -8.85445,-8.88809 0,0 11.53636,-0.3275 11.53636,-8.34902 0,-8.20546 -8.87381,-8.85175 -11.6189,-8.99092 1.70324,-0.10363 8.42562,-2.42967 8.3861,-10.53741 -0.0399,-8.37053 -12.87508,-7.67987 -35.956784,-9.20144 2.66326,-15.52873 4.52035,-48.76031 -10.87768,-49.26032 z m -63.387663,53.22425 c -2.167221,0 -3.911718,1.45375 -3.911718,3.25977 v 53.07599 c 0,1.80609 1.744497,3.25983 3.911718,3.25983 h 14.602385 c 2.167228,0 3.911725,-1.45374 3.911725,-3.25983 v -53.07599 c 0,-1.80602 -1.744497,-3.25977 -3.911725,-3.25977 z"
+       style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;paint-order:markers fill stroke"
+       d="m 72.35681,178.12175 c -0.545876,-0.0192 -1.121779,0.0192 -1.729242,0.12259 -12.959134,2.14891 4.226681,21.04642 -10.867262,31.83193 -12.617074,10.52505 -17.206737,7.22264 -17.332488,10.67172 v 33.88171 c 1.891308,5.00813 14.483922,10.12847 27.868234,11.21944 13.384309,1.09096 32.098988,-1.73801 32.305468,-7.56421 0.10989,-5.27572 -2.896223,-5.39662 -2.896223,-5.39662 0,0 6.589043,-0.40752 6.808883,-6.12278 0.10966,-5.60545 -6.38265,-6.40691 -6.38265,-6.40691 0,0 8.31589,-0.23607 8.31589,-6.01832 0,-5.91485 -6.39662,-6.38072 -8.37539,-6.48104 1.22776,-0.0747 6.07353,-1.75141 6.04505,-7.59582 -0.0287,-6.03384 -9.280896,-5.53597 -25.919173,-6.63279 1.919791,-11.19377 3.258461,-35.14851 -7.841097,-35.50894 z m -45.69253,38.36631 c -1.562226,0 -2.819733,1.04793 -2.819733,2.34979 v 38.25943 c 0,1.30191 1.257507,2.34983 2.819733,2.34983 h 10.526021 c 1.562231,0 2.819739,-1.04792 2.819739,-2.34983 v -38.25943 c 0,-1.30186 -1.257508,-2.34979 -2.819739,-2.34979 z"
        id="rect1278"
        inkscape:connector-curvature="0" />
   </g>

tubearchivist/static/script.js

@@ -9,6 +9,8 @@ function sortChange(sortValue) {
 }
 
 function isWatched(youtube_id) {
+    postVideoProgress(youtube_id, 0); // Reset video progress on watched;
+    removeProgressBar(youtube_id);
     var payload = JSON.stringify({'watched': youtube_id});
     sendPost(payload);
     var seenIcon = document.createElement('img');
@@ -21,6 +23,11 @@ function isWatched(youtube_id) {
     document.getElementById(youtube_id).replaceWith(seenIcon);
 }
 
+// Removes the progress bar when passed a video id
+function removeProgressBar(videoId) {
+    setProgressBar(videoId, 0, 1);
+}
+
 function isWatchedButton(button) {
     youtube_id = button.getAttribute("data-id");
     var payload = JSON.stringify({'watched': youtube_id});
@@ -33,6 +40,7 @@ function isWatchedButton(button) {
 }
 
 function isUnwatched(youtube_id) {
+    postVideoProgress(youtube_id, 0); // Reset video progress on unwatched;
     var payload = JSON.stringify({'un_watched': youtube_id});
     sendPost(payload);
     var unseenIcon = document.createElement('img');
@@ -188,7 +196,7 @@ function reEmbed() {
 
 function dbBackup() {
     var payload = JSON.stringify({'db-backup': true});
-    sendPost(payload)
+    sendPost(payload);
     // clear button
     var message = document.createElement('p');
     message.innerText = 'backing up archive';
@@ -233,6 +241,14 @@ function findPlaylists(button) {
     }, 500);
 }
 
+function resetToken() {
+    var payload = JSON.stringify({'reset-token': true});
+    sendPost(payload);
+    var message = document.createElement("p");
+    message.innerText = "Token revoked";
+    document.getElementById("text-reveal").replaceWith(message);
+}
+
 // delete from file system
 function deleteConfirm() {
     to_show = document.getElementById("delete-button");
@@ -286,153 +302,389 @@ function cancelDelete() {
 
 // player
 function createPlayer(button) {
-    var mediaUrl = button.getAttribute('data-src');
-    var mediaThumb = button.getAttribute('data-thumb');
-    var mediaTitle = button.getAttribute('data-title');
-    var mediaChannel = button.getAttribute('data-channel');
-    var mediaChannelId = button.getAttribute('data-channel-id');
-    var dataId = button.getAttribute('data-id');
-    // get watched status
-    var playedStatus = document.createDocumentFragment();
-    playedStatus.appendChild(document.getElementById(dataId));
-    // create player
+    var videoId = button.getAttribute('data-id');
+    var videoData = getVideoData(videoId);
+    var videoProgress = getVideoProgress(videoId).position;
+    var videoName = videoData.data.title;
+
+    var videoTag = createVideoTag(videoData, videoProgress);
+
+    var playlist = '';
+    var videoPlaylists = videoData.data.playlist; // Array of playlists the video is in
+    if (typeof(videoPlaylists) != 'undefined') {
+        var subbedPlaylists = getSubbedPlaylists(videoPlaylists); // Array of playlist the video is in that are subscribed
+        if (subbedPlaylists.length != 0) {
+            var playlistData = getPlaylistData(subbedPlaylists[0]); // Playlist data for first subscribed playlist
+            var playlistId = playlistData.playlist_id;
+            var playlistName = playlistData.playlist_name;
+            var playlist = `<h5><a href="/playlist/${playlistId}/">${playlistName}</a></h5>`;
+        }
+    }
+
+    var videoViews = formatNumbers(videoData.data.stats.view_count);
+
+    var channelId = videoData.data.channel.channel_id;
+    var channelName = videoData.data.channel.channel_name;
+
     removePlayer();
-    var playerElement = document.createElement('div');
-    playerElement.classList.add("video-player");
-    // var playerElement = document.getElementById('player');
-    playerElement.setAttribute('data-id', dataId);
-    // playerElement.innerHTML = '';
-    var videoPlayer = document.createElement('video');
-    videoPlayer.setAttribute('src', mediaUrl);
-    videoPlayer.setAttribute('controls', true);
-    videoPlayer.setAttribute('autoplay', true);
-    videoPlayer.setAttribute('width', '100%');
-    videoPlayer.setAttribute('playsinline', true);
-    videoPlayer.setAttribute('poster', mediaThumb);
-    playerElement.appendChild(videoPlayer);
-    // title bar
-    var titleBar = document.createElement('div');
-    titleBar.classList.add('player-title', 'boxed-content');
-    // close
-    var closeButton = document.createElement('img');
-    closeButton.className = 'close-button';
-    closeButton.setAttribute('src', "/static/img/icon-close.svg");
-    closeButton.setAttribute('alt', 'close-icon');
-    closeButton.setAttribute('data', dataId);
-    closeButton.setAttribute('onclick', "removePlayer()");
-    closeButton.setAttribute('title', 'Close player');
-    titleBar.appendChild(closeButton);
-    // played
-    titleBar.appendChild(playedStatus);
-    // channel title
-    var channelTitleLink = document.createElement('a');
-    channelTitleLink.setAttribute('href', '/channel/' + mediaChannelId + '/');
-    var channelTitle = document.createElement('h3');
-    channelTitle.innerText = mediaChannel;
-    channelTitleLink.appendChild(channelTitle);
-    titleBar.appendChild(channelTitleLink);
-    // video title
-    var videoTitleLink = document.createElement('a');
-    videoTitleLink.setAttribute('href', '/video/' + dataId + '/');
-    var videoTitle = document.createElement('h2');
-    videoTitle.innerText = mediaTitle;
-    videoTitleLink.appendChild(videoTitle);
-    titleBar.appendChild(videoTitleLink);
-    // add titlebar
-    playerElement.appendChild(titleBar);
-    // add whole
-    document.getElementById("player").appendChild(playerElement);
+    document.getElementById(videoId).outerHTML = ''; // Remove watch indicator from video info
+
+    // If cast integration is enabled create cast button
+    var castButton = '';
+    if (videoData.config.application.enable_cast) {
+        var castButton = `<google-cast-launcher id="castbutton"></google-cast-launcher>`;
+    }
+
+    // Watched indicator
+    if (videoData.data.player.watched) {
+        var playerState = "seen";
+        var watchedFunction = "Unwatched";
+    } else {
+        var playerState = "unseen";
+        var watchedFunction = "Watched";  
+    }
+
+    var playerStats = `<div class="thumb-icon player-stats"><img src="/static/img/icon-eye.svg" alt="views icon"><span>${videoViews}</span>`;
+    if (videoData.data.stats.like_count) {
+        var likes = formatNumbers(videoData.data.stats.like_count);
+        playerStats += `<span>|</span><img src="/static/img/icon-thumb.svg" alt="thumbs-up"><span>${likes}</span>`;
+    }
+    if (videoData.data.stats.dislike_count && videoData.config.downloads.integrate_ryd) {
+        var dislikes = formatNumbers(videoData.data.stats.dislike_count);
+        playerStats += `<span>|</span><img class="dislike" src="/static/img/icon-thumb.svg" alt="thumbs-down"><span>${dislikes}</span>`;
+    }
+    playerStats += "</div>";
+
+    const markup = `
+    <div class="video-player" data-id="${videoId}">
+        ${videoTag}
+        <div class="player-title boxed-content">
+            <img class="close-button" src="/static/img/icon-close.svg" alt="close-icon" data="${videoId}" onclick="removePlayer()" title="Close player">
+            <img src="/static/img/icon-${playerState}.svg" alt="${playerState}-icon" id="${videoId}" onclick="is${watchedFunction}(this.id)" class="${playerState}-icon" title="Mark as ${watchedFunction}">
+            ${castButton}
+            ${playerStats}
+            <div class="player-channel-playlist">
+                <h3><a href="/channel/${channelId}/">${channelName}</a></h3>
+                ${playlist}
+            </div>
+            <a href="/video/${videoId}/"><h2 id="video-title">${videoName}</h2></a>
+        </div>
+    </div>
+    `;
+    const divPlayer = document.getElementById("player");
+    divPlayer.innerHTML = markup;
+}
+
+// Add video tag to video page when passed a video id, function loaded on page load `video.html (115-117)`
+function insertVideoTag(videoData, videoProgress) {
+    var videoTag = createVideoTag(videoData, videoProgress);
+    var videoMain = document.getElementsByClassName("video-main");
+    videoMain[0].innerHTML = videoTag;
+}
+
+// Generates a video tag with subtitles when passed videoData and videoProgress.
+function createVideoTag(videoData, videoProgress) {
+    var videoId = videoData.data.youtube_id;
+    var videoUrl = videoData.data.media_url;
+    var videoThumbUrl = videoData.data.vid_thumb_url;
+    var subtitles = '';
+    var videoSubtitles = videoData.data.subtitles; // Array of subtitles
+    if (typeof(videoSubtitles) != 'undefined' && videoData.config.downloads.subtitle) {
+        for (var i = 0; i < videoSubtitles.length; i++) {
+            subtitles += `<track label="${videoSubtitles[i].name}" kind="subtitles" srclang="${videoSubtitles[i].lang}" src="${videoSubtitles[i].media_url}">`;
+        }
+    }
+
+    var videoTag = `
+    <video poster="${videoThumbUrl}" ontimeupdate="onVideoProgress()" onpause="onVideoPause()" onended="onVideoEnded()" controls autoplay width="100%" playsinline id="video-item">
+        <source src="${videoUrl}#t=${videoProgress}" type="video/mp4" id="video-source" videoid="${videoId}">
+        ${subtitles}
+    </video>
+    `;
+    return videoTag;
+}
+
+// Gets video tag
+function getVideoPlayer() {
+    var videoElement = document.getElementById("video-item");
+    return videoElement;
+}
+
+// Gets the video source tag
+function getVideoPlayerVideoSource() {
+    var videoPlayerVideoSource = document.getElementById("video-source");
+    return videoPlayerVideoSource;
+}
+
+// Gets the current progress of the video currently in the player
+function getVideoPlayerCurrentTime() {
+    var videoElement = getVideoPlayer();
+    if (videoElement != null) {
+        return videoElement.currentTime;
+    }
+}
+
+// Gets the video id of the video currently in the player
+function getVideoPlayerVideoId() {
+    var videoPlayerVideoSource = getVideoPlayerVideoSource();
+    if (videoPlayerVideoSource != null) {
+        return videoPlayerVideoSource.getAttribute("videoid");
+    }
+}
+
+// Gets the duration of the video currently in the player
+function getVideoPlayerDuration() {
+    var videoElement = getVideoPlayer();
+    if (videoElement != null) {
+        return videoElement.duration;
+    }
+}
+
+// Gets current watch status of video based on watch button
+function getVideoPlayerWatchStatus() {
+    var videoId = getVideoPlayerVideoId();
+    var watched = false;
+    if(document.getElementById(videoId).className != "unseen-icon") {
+        watched = true;
+    }
+    return watched;
+}
+
+// Runs on video playback, marks video as watched if video gets to 90% or higher, sends position to api
+function onVideoProgress() {
+    var videoId = getVideoPlayerVideoId();
+    var currentTime = getVideoPlayerCurrentTime();
+    var duration = getVideoPlayerDuration();
+    if ((currentTime % 10).toFixed(1) <= 0.2) { // Check progress every 10 seconds or else progress is checked a few times a second
+        postVideoProgress(videoId, currentTime);
+        if (!getVideoPlayerWatchStatus()) { // Check if video is already marked as watched
+            if (watchedThreshold(currentTime, duration)) {
+                isWatched(videoId);
+            }
+        }
+    }
+}
+
+// Runs on video end, marks video as watched
+function onVideoEnded() {
+    var videoId = getVideoPlayerVideoId();
+    if (!getVideoPlayerWatchStatus()) { // Check if video is already marked as watched
+        isWatched(videoId);
+    }
+}
+
+function watchedThreshold(currentTime, duration) {
+    var watched = false;
+    if (duration <= 1800){ // If video is less than 30 min
+        if ((currentTime / duration) >= 0.90) { // Mark as watched at 90%
+            var watched = true;
+        }
+    } else { // If video is more than 30 min
+        if (currentTime >= (duration - 120)) { // Mark as watched if there is two minutes left
+            var watched = true;
+        }
+    }
+    return watched;
+}
+
+// Runs on video pause. Sends current position.
+function onVideoPause() {
+    var videoId = getVideoPlayerVideoId();
+    var currentTime = getVideoPlayerCurrentTime();
+    postVideoProgress(videoId, currentTime);
+}
+
+// Format numbers for frontend
+function formatNumbers(number) {
+    var numberUnformatted = parseFloat(number);
+    if (numberUnformatted > 999999999) {
+        var numberFormatted = (numberUnformatted / 1000000000).toFixed(1).toString() + "B";
+    } else if (numberUnformatted > 999999) {
+        var numberFormatted = (numberUnformatted / 1000000).toFixed(1).toString() + "M";
+    } else if (numberUnformatted > 999) {
+        var numberFormatted = (numberUnformatted / 1000).toFixed(1).toString() + "K";
+    } else {
+        var numberFormatted = numberUnformatted;
+    }
+    return numberFormatted;
+}
+
+// Gets video data when passed video ID
+function getVideoData(videoId) {
+    var apiEndpoint = "/api/video/" + videoId + "/";
+    var videoData = apiRequest(apiEndpoint, "GET");
+    return videoData;
+}
+
+// Gets channel data when passed channel ID
+function getChannelData(channelId) {
+    var apiEndpoint = "/api/channel/" + channelId + "/";
+    var channelData = apiRequest(apiEndpoint, "GET");
+    return channelData.data;
+}
+
+// Gets playlist data when passed playlist ID
+function getPlaylistData(playlistId) {
+    var apiEndpoint = "/api/playlist/" + playlistId + "/";
+    var playlistData = apiRequest(apiEndpoint, "GET");
+    return playlistData.data;
+}
+
+// Get video progress data when passed video ID
+function getVideoProgress(videoId) {
+    var apiEndpoint = "/api/video/" + videoId + "/progress/";
+    var videoProgress = apiRequest(apiEndpoint, "GET");
+    return videoProgress;
+}
+
+// Given an array of playlist ids it returns an array of subbed playlist ids from that list
+function getSubbedPlaylists(videoPlaylists) {
+    var subbedPlaylists = [];
+    for (var i = 0; i < videoPlaylists.length; i++) {
+        if(getPlaylistData(videoPlaylists[i]).playlist_subscribed) {
+            subbedPlaylists.push(videoPlaylists[i]);
+        }
+    }
+    return subbedPlaylists;
+}
+
+// Send video position when given video id and progress in seconds
+function postVideoProgress(videoId, videoProgress) {
+    var apiEndpoint = "/api/video/" + videoId + "/progress/";
+    var duartion = getVideoPlayerDuration();
+    if (!isNaN(videoProgress) && duartion != 'undefined') {
+        var data = {
+            "position": videoProgress
+        };
+        if (videoProgress == 0) {
+            apiRequest(apiEndpoint, "DELETE");
+            // console.log("Deleting Video Progress for Video ID: " + videoId + ", Progress: " + videoProgress);
+        } else if (!getVideoPlayerWatchStatus()) {
+            apiRequest(apiEndpoint, "POST", data);
+            // console.log("Saving Video Progress for Video ID: " + videoId + ", Progress: " + videoProgress);
+        }
+    }
+}
+
+// Makes api requests when passed an endpoint and method ("GET", "POST", "DELETE")
+function apiRequest(apiEndpoint, method, data) {
+    const xhttp = new XMLHttpRequest();
+    var sessionToken = getCookie("sessionid");
+    xhttp.open(method, apiEndpoint, false);
+    xhttp.setRequestHeader("X-CSRFToken", getCookie("csrftoken")); // Used for video progress POST requests
+    xhttp.setRequestHeader("Authorization", "Token " + sessionToken);
+    xhttp.setRequestHeader("Content-Type", "application/json");
+    xhttp.send(JSON.stringify(data));
+    return JSON.parse(xhttp.responseText);
+}
+
+// Gets origin URL
+function getURL() {
+    return window.location.origin;
 }
 
 function removePlayer() {
+    var currentTime = getVideoPlayerCurrentTime();
+    var duration = getVideoPlayerDuration();
+    var videoId = getVideoPlayerVideoId();
+    postVideoProgress(videoId, currentTime);
+    setProgressBar(videoId, currentTime, duration);
     var playerElement = document.getElementById('player');
     if (playerElement.hasChildNodes()) {
-        var youtubeId = playerElement.childNodes[0].getAttribute("data-id");
+        var youtubeId = playerElement.childNodes[1].getAttribute("data-id");
         var playedStatus = document.createDocumentFragment();
         var playedBox = document.getElementById(youtubeId);
         if (playedBox) {
             playedStatus.appendChild(playedBox);
-        };
+        }
         playerElement.innerHTML = '';
         // append played status
         var videoInfo = document.getElementById('video-info-' + youtubeId);
         videoInfo.insertBefore(playedStatus, videoInfo.firstChild);
-    };
+    }
 }
 
+// Sets the progress bar when passed a video id, video progress and video duration
+function setProgressBar(videoId, currentTime, duration) {
+    progressBar = document.getElementById("progress-" + videoId);
+    progressBarWidth = (currentTime / duration) * 100 + "%";
+    if (progressBar) {
+        progressBar.style.width = progressBarWidth;
+    }
+}
 
 // multi search form
 function searchMulti(query) {
     if (query.length > 1) {
-        var payload = JSON.stringify({'multi_search': query})
+        var payload = JSON.stringify({'multi_search': query});
         var http = new XMLHttpRequest();
         http.onreadystatechange = function() {
             if (http.readyState === 4) {
-                allResults = JSON.parse(http.response)['results'];
+                allResults = JSON.parse(http.response).results;
                 populateMultiSearchResults(allResults);
-            };
+            }
         };
         http.open("POST", "/process/", true);
         http.setRequestHeader("X-CSRFToken", getCookie("csrftoken"));
         http.setRequestHeader("Content-type", "application/json");
         http.send(payload);
-    };
+    }
 }
 
 function getViewDefaults(view) {
     var defaultView = document.getElementById("id_" + view).value;
-    return defaultView
+    return defaultView;
 }
 
 function populateMultiSearchResults(allResults) {
     // videos
     var defaultVideo = getViewDefaults("home");
-    var allVideos = allResults["video_results"];
+    var allVideos = allResults.video_results;
     var videoBox = document.getElementById("video-results");
     videoBox.innerHTML = "";
     for (let index = 0; index < allVideos.length; index++) {
-        const video = allVideos[index]["source"];
+        const video = allVideos[index].source;
         const videoDiv = createVideo(video, defaultVideo);
         videoBox.appendChild(videoDiv);
-    };
+    }
     // channels
     var defaultChannel = getViewDefaults("channel");
-    var allChannels = allResults["channel_results"];
+    var allChannels = allResults.channel_results;
     var channelBox = document.getElementById("channel-results");
     channelBox.innerHTML = "";
     for (let index = 0; index < allChannels.length; index++) {
-        const channel = allChannels[index]["source"];
+        const channel = allChannels[index].source;
         const channelDiv = createChannel(channel, defaultChannel);
         channelBox.appendChild(channelDiv);
-    };
+    }
     // playlists
     var defaultPlaylist = getViewDefaults("playlist");
-    var allPlaylists = allResults["playlist_results"];
+    var allPlaylists = allResults.playlist_results;
     var playlistBox = document.getElementById("playlist-results");
     playlistBox.innerHTML = "";
     for (let index = 0; index < allPlaylists.length; index++) {
-        const playlist = allPlaylists[index]["source"];
+        const playlist = allPlaylists[index].source;
         const playlistDiv = createPlaylist(playlist, defaultPlaylist);
         playlistBox.appendChild(playlistDiv);
-    };
+    }
 }
 
 
 function createVideo(video, viewStyle) {
     // create video item div from template
-    const videoId = video["youtube_id"];
-    const mediaUrl = video["media_url"];
-    const thumbUrl = "/cache/" + video["vid_thumb_url"];
-    const videoTitle = video["title"];
-    const videoPublished = video["published"];
-    const videoDuration = video["player"]["duration_str"];
-    if (video["player"]["watched"]) {
+    const videoId = video.youtube_id;
+    const mediaUrl = video.media_url;
+    const thumbUrl = "/cache/" + video.vid_thumb_url;
+    const videoTitle = video.title;
+    const videoPublished = video.published;
+    const videoDuration = video.player.duration_str;
+    if (video.player.watched) {
         var playerState = "seen";
     } else {
         var playerState = "unseen";
     };
-    const channelId = video["channel"]["channel_id"];
-    const channelName = video["channel"]["channel_name"];
+    const channelId = video.channel.channel_id;
+    const channelName = video.channel.channel_name;
     // build markup
     const markup = `
     <a href="#player" data-src="/media/${mediaUrl}" data-thumb="${thumbUrl}" data-title="${videoTitle}" data-channel="${channelName}" data-channel-id="${channelId}" data-id="${videoId}" onclick="createPlayer(this)">
@@ -455,25 +707,25 @@ function createVideo(video, viewStyle) {
             <a class="video-more" href="/video/${videoId}/"><h2>${videoTitle}</h2></a>
         </div>
     </div>
-    `
+    `;
     const videoDiv = document.createElement("div");
     videoDiv.setAttribute("class", "video-item " + viewStyle);
-    videoDiv.innerHTML = markup
-    return videoDiv
+    videoDiv.innerHTML = markup;
+    return videoDiv;
 }
 
 
 function createChannel(channel, viewStyle) {
     // create channel item div from template
-    const channelId = channel["channel_id"];
-    const channelName = channel["channel_name"];
-    const channelSubs = channel["channel_subs"];
-    const channelLastRefresh = channel["channel_last_refresh"];
-    if (channel["channel_subscribed"]) {
-        var button = `<button class="unsubscribe" type="button" id="${channelId}" onclick="unsubscribe(this.id)" title="Unsubscribe from ${channelName}">Unsubscribe</button>`
+    const channelId = channel.channel_id;
+    const channelName = channel.channel_name;
+    const channelSubs = channel.channel_subs;
+    const channelLastRefresh = channel.channel_last_refresh;
+    if (channel.channel_subscribed) {
+        var button = `<button class="unsubscribe" type="button" id="${channelId}" onclick="unsubscribe(this.id)" title="Unsubscribe from ${channelName}">Unsubscribe</button>`;
     } else {
-        var button = `<button type="button" id="${channelId}" onclick="subscribe(this.id)" title="Subscribe to ${channelName}">Subscribe</button>`
-    };
+        var button = `<button type="button" id="${channelId}" onclick="subscribe(this.id)" title="Subscribe to ${channelName}">Subscribe</button>`;
+    }
     // build markup
     const markup = `
     <div class="channel-banner ${viewStyle}">
@@ -500,25 +752,25 @@ function createChannel(channel, viewStyle) {
             </div>
         </div>
     </div>
-    `
+    `;
     const channelDiv = document.createElement("div");
     channelDiv.setAttribute("class", "channel-item " + viewStyle);
     channelDiv.innerHTML = markup;
-    return channelDiv
+    return channelDiv;
 }
 
 function createPlaylist(playlist, viewStyle) {
     // create playlist item div from template
-    const playlistId = playlist["playlist_id"];
-    const playlistName = playlist["playlist_name"];
-    const playlistChannelId = playlist["playlist_channel_id"];
-    const playlistChannel = playlist["playlist_channel"];
-    const playlistLastRefresh = playlist["playlist_last_refresh"];
-    if (playlist["playlist_subscribed"]) {
-        var button = `<button class="unsubscribe" type="button" id="${playlistId}" onclick="unsubscribe(this.id)" title="Unsubscribe from ${playlistName}">Unsubscribe</button>`
+    const playlistId = playlist.playlist_id;
+    const playlistName = playlist.playlist_name;
+    const playlistChannelId = playlist.playlist_channel_id;
+    const playlistChannel = playlist.playlist_channel;
+    const playlistLastRefresh = playlist.playlist_last_refresh;
+    if (playlist.playlist_subscribed) {
+        var button = `<button class="unsubscribe" type="button" id="${playlistId}" onclick="unsubscribe(this.id)" title="Unsubscribe from ${playlistName}">Unsubscribe</button>`;
     } else {
-        var button = `<button type="button" id="${playlistId}" onclick="subscribe(this.id)" title="Subscribe to ${playlistName}">Subscribe</button>`
-    };
+        var button = `<button type="button" id="${playlistId}" onclick="subscribe(this.id)" title="Subscribe to ${playlistName}">Subscribe</button>`;
+    }
     const markup = `
     <div class="playlist-thumbnail">
         <a href="/playlist/${playlistId}/">
@@ -531,11 +783,11 @@ function createPlaylist(playlist, viewStyle) {
         <p>Last refreshed: ${playlistLastRefresh}</p>
         ${button}
     </div>
-    `
+    `;
     const playlistDiv = document.createElement("div");
     playlistDiv.setAttribute("class", "playlist-item " + viewStyle);
     playlistDiv.innerHTML = markup;
-    return playlistDiv
+    return playlistDiv;
 }
 
 
@@ -557,8 +809,8 @@ function getCookie(c_name) {
             c_end = document.cookie.indexOf(";", c_start);
             if (c_end == -1) c_end = document.cookie.length;
             return unescape(document.cookie.substring(c_start,c_end));
-        };
-    };
+        }
+    }
     return "";
 }
 
@@ -574,17 +826,17 @@ function textReveal() {
     } else {
         textBox.style.height = 'unset';
         button.innerText = 'Hide';
-    };
+    }
 }
 
 function showForm() {
     var formElement = document.getElementById('hidden-form');
-    var displayStyle = formElement.style.display
+    var displayStyle = formElement.style.display;
     if (displayStyle === "") {
         formElement.style.display = 'block';
     } else {
         formElement.style.display = "";
-    };
+    }
     animate('animate-icon', 'pulse-img');
 }
 
@@ -594,5 +846,5 @@ function animate(elementId, animationClass) {
         toAnimate.className = animationClass;
     } else {
         toAnimate.classList.remove(animationClass);
-    };
+    }
 }