merge for v0.1.5 build

Changed:
- Added cookie validation button
- Fixed all the open bug reports! :-)

CONTRIBUTING.md

@@ -19,7 +19,7 @@ I have learned the hard way, that working on a dockerized application outside of
 
 This is my setup I have landed on, YMMV:
 - Clone the repo, work on it with your favorite code editor in your local filesystem. *testing* branch is the where all the changes are happening, might be unstable and is WIP.
-- Then I have a VM on KVM hypervisor running standard Ubuntu Server LTS with docker installed. The VM keeps my projects separate and offers convenient snapshot functionality. The VM also offers ways to simulate lowend environments by limiting CPU cores and memory. But you could also just run docker on your host system.
+- Then I have a VM on KVM hypervisor running standard Ubuntu Server LTS with docker installed. The VM keeps my projects separate and offers convenient snapshot functionality. The VM also offers ways to simulate lowend environments by limiting CPU cores and memory. You can use this [Ansible Docker Ubuntu](https://github.com/bbilly1/ansible-playbooks) playbook to get started quickly. But you could also just run docker on your host system.
 - The `Dockerfile` is structured in a way that the actual application code is in the last layer so rebuilding the image with only code changes utilizes the build cache for everything else and will just take a few seconds.
 - Take a look at the `deploy.sh` file. I have my local DNS resolve `tubearchivist.local` to the IP of the VM for convenience. To deploy the latest changes and rebuild the application to the testing VM run:
 ```bash
@@ -32,11 +32,11 @@ This is my setup I have landed on, YMMV:
 ## Working with Elasticsearch
 Additionally to the required services as listed in the example docker-compose file, the **Dev Tools** of [Kibana](https://www.elastic.co/guide/en/kibana/current/docker.html) are invaluable for running and testing Elasticsearch queries.  
 
-If you want to run queries in on the Elasticsearch container directly from your host with for example `curl` or something like *postman*, you might want to **publish** the port 9200 instead of just **exposing** it.
+If you want to run queries on the Elasticsearch container directly from your host with for example `curl` or something like *postman*, you might want to **publish** the port 9200 instead of just **exposing** it.
 
 ## Implementing a new feature
 
-Do you see anything on the roadmap that you would like to take a closer look at but you are not sure, what's the best way to tackle that? Or anything not on there yet you'd like to implement but are not sure how? Open up an issue and we try to find a solution together.
+Do you see anything on the roadmap that you would like to take a closer look at but you are not sure, what's the best way to tackle that? Or anything not on there yet you'd like to implement but are not sure how? Reach out on Discord and we'll look into it together.
 
 ## Making changes
 
@@ -53,10 +53,10 @@ If you want to see what's in your container, checkout the matching release tag.
 
 ## Code formatting and linting
 
-To keep things clean and consistent for everybody, there is a github action setup to lint and check the changes. You can test your code locally first if you want. For example if you made changes in the **download** module, run
+To keep things clean and consistent for everybody, there is a github action setup to lint and check the changes. You can test your code locally first if you want. For example if you made changes in the **video** module, run
 
 ```shell
-./deploy.sh validate tubearchivist/home/src/download.py
+./deploy.sh validate tubearchivist/home/src/index/video.py
 ```
 
-to validate your changes. If you omit the path, all the project files will get checked. This is subject to change as the codebase improves. 
+to validate your changes. If you omit the path, all the project files will get checked. This is subject to change as the codebase improves. 

Dockerfile

@@ -7,21 +7,7 @@ FROM python:3.10.4-slim-bullseye AS builder
 ARG TARGETPLATFORM
 
 RUN apt-get update
-RUN apt-get install -y --no-install-recommends build-essential gcc curl
+RUN apt-get install -y --no-install-recommends build-essential gcc
-
-# get newest patched ffmpeg and ffprobe builds for amd64 fall back to repo ffmpeg for arm64
-RUN if [ "$TARGETPLATFORM" = "linux/amd64" ] ; then \
-    curl -s https://api.github.com/repos/yt-dlp/FFmpeg-Builds/releases/latest \
-        | grep browser_download_url \
-        | grep ".*master.*linux64.*tar.xz" \
-        | cut -d '"' -f 4 \
-        | xargs curl -L --output ffmpeg.tar.xz && \
-        tar -xf ffmpeg.tar.xz --strip-components=2 --no-anchored -C /usr/bin/ "ffmpeg" && \
-        tar -xf ffmpeg.tar.xz --strip-components=2 --no-anchored -C /usr/bin/ "ffprobe" && \
-        rm ffmpeg.tar.xz \
-    ; elif [ "$TARGETPLATFORM" = "linux/arm64" ] ; then \
-        apt-get -y update && apt-get -y install --no-install-recommends ffmpeg && rm -rf /var/lib/apt/lists/* \
-    ; fi
 
 # install requirements
 COPY ./tubearchivist/requirements.txt /requirements.txt
@@ -37,15 +23,28 @@ ENV PYTHONUNBUFFERED 1
 
 # copy build requirements
 COPY --from=builder /root/.local /root/.local
-COPY --from=builder /usr/bin/ffmpeg /usr/bin/ffmpeg
-COPY --from=builder /usr/bin/ffprobe /usr/bin/ffprobe
 ENV PATH=/root/.local/bin:$PATH
 
 # install distro packages needed
 RUN apt-get clean && apt-get -y update && apt-get -y install --no-install-recommends \
     nginx \
     atomicparsley \
-    curl && rm -rf /var/lib/apt/lists/*
+    curl \
+    xz-utils && rm -rf /var/lib/apt/lists/*
+
+# get newest patched ffmpeg and ffprobe builds for amd64 fall back to repo ffmpeg for arm64
+RUN if [ "$TARGETPLATFORM" = "linux/amd64" ] ; then \
+    curl -s https://api.github.com/repos/yt-dlp/FFmpeg-Builds/releases/latest \
+        | grep browser_download_url \
+        | grep ".*master.*linux64.*tar.xz" \
+        | cut -d '"' -f 4 \
+        | xargs curl -L --output ffmpeg.tar.xz && \
+        tar -xf ffmpeg.tar.xz --strip-components=2 --no-anchored -C /usr/bin/ "ffmpeg" && \
+        tar -xf ffmpeg.tar.xz --strip-components=2 --no-anchored -C /usr/bin/ "ffprobe" && \
+        rm ffmpeg.tar.xz \
+    ; elif [ "$TARGETPLATFORM" = "linux/arm64" ] ; then \
+        apt-get -y update && apt-get -y install --no-install-recommends ffmpeg && rm -rf /var/lib/apt/lists/* \
+    ; fi
 
 # install debug tools for testing environment
 RUN if [ "$INSTALL_DEBUG" ] ; then \

README.md

@@ -5,11 +5,12 @@
 Tube Archivist has a new home: https://github.com/tubearchivist/tubearchivist
 
 ## Table of contents:
-* [Wiki](https://github.com/tubearchivist/tubearchivist/wiki) for a detailed documentation, with [FAQ](https://github.com/tubearchivist/tubearchivist/wiki/FAQ)
+* [Wiki](https://github.com/tubearchivist/tubearchivist/wiki) with [FAQ](https://github.com/tubearchivist/tubearchivist/wiki/FAQ)
 * [Core functionality](#core-functionality)
 * [Screenshots](#screenshots)
 * [Problem Tube Archivist tries to solve](#problem-tube-archivist-tries-to-solve)
 * [Connect](#connect)
+* [Extended Universe](#extended-universe)
 * [Installing and updating](#installing-and-updating)
 * [Getting Started](#getting-started)
 * [Potential pitfalls](#potential-pitfalls)
@@ -52,6 +53,10 @@ Once your YouTube video collection grows, it becomes hard to search and find a s
 - [Discord](https://discord.gg/AFwz8nE7BK): Connect with us on our Discord server.
 - [r/TubeArchivist](https://www.reddit.com/r/TubeArchivist/): Join our Subreddit.
 
+## Extended Universe
+- [Browser Extension](https://github.com/tubearchivist/browser-extension) Tube Archivist Companion, for [Firefox](https://addons.mozilla.org/addon/tubearchivist-companion/) and [Chrome](https://chrome.google.com/webstore/detail/tubearchivist-companion/jjnkmicfnfojkkgobdfeieblocadmcie)
+- [Tube Archivist Metrics](https://github.com/tubearchivist/tubearchivist-metrics) to create statistics in Prometheus/OpenMetrics format.
+
 ## Installing and updating
 Take a look at the example `docker-compose.yml` file provided. Use the *latest* or the named semantic version tag. The *unstable* tag is for intermediate testing and as the name implies, is **unstable** and not be used on your main installation but in a [testing environment](CONTRIBUTING.md).  
 
@@ -134,7 +139,9 @@ chown 1000:0 /path/to/mount/point
 This will match the permissions with the **UID** and **GID** of elasticsearch within the container and should fix the issue.
 
 ### Disk usage
-The Elasticsearch index will turn to *read only* if the disk usage of the container goes above 95% until the usage drops below 90% again. Similar to that, TubeArchivist will become all sorts of messed up when running out of disk space. There are some error messages in the logs when that happens, but it's best to make sure to have enough disk space before starting to download.
+The Elasticsearch index will turn to *read only* if the disk usage of the container goes above 95% until the usage drops below 90% again, you will see error messages like `disk usage exceeded flood-stage watermark`, [link](https://github.com/tubearchivist/tubearchivist#disk-usage).  
+
+Similar to that, TubeArchivist will become all sorts of messed up when running out of disk space. There are some error messages in the logs when that happens, but it's best to make sure to have enough disk space before starting to download.
 
 ## Getting Started
 1. Go through the **settings** page and look at the available options. Particularly set *Download Format* to your desired video quality before downloading. **Tube Archivist** downloads the best available quality by default. To support iOS or MacOS and some other browsers a compatible format must be specified. For example:
@@ -153,7 +160,6 @@ We have come far, nonetheless we are not short of ideas on how to improve and ex
 - [ ] Podcast mode to serve channel as mp3
 - [ ] Implement [PyFilesystem](https://github.com/PyFilesystem/pyfilesystem2) for flexible video storage
 - [ ] Implement [Apprise](https://github.com/caronc/apprise) for notifications ([#97](https://github.com/tubearchivist/tubearchivist/issues/97))
-- [ ] Add passing browser cookies to yt-dlp ([#199](https://github.com/tubearchivist/tubearchivist/issues/199))
 - [ ] User created playlists, random and repeat controls ([#108](https://github.com/tubearchivist/tubearchivist/issues/108), [#220](https://github.com/tubearchivist/tubearchivist/issues/220))
 - [ ] Auto play or play next link ([#226](https://github.com/tubearchivist/tubearchivist/issues/226))
 - [ ] Show similar videos on video page
@@ -168,6 +174,7 @@ We have come far, nonetheless we are not short of ideas on how to improve and ex
 - [ ] Download video comments
 
 Implemented:
+- [X] Add passing browser cookies to yt-dlp [2022-05-08]
 - [X] Add [SponsorBlock](https://sponsor.ajay.app/) integration [2022-04-16]
 - [X] Implement per channel settings [2022-03-26]
 - [X] Subtitle download & indexing [2022-02-13]

deploy.sh

@@ -33,7 +33,7 @@ function sync_blackhole {
         . -e ssh "$host":tubearchivist
 
     echo "$PASS" | ssh "$host" 'sudo -S docker buildx build --platform linux/amd64 -t bbilly1/tubearchivist:latest tubearchivist --load 2>/dev/null'
-    echo "$PASS" | ssh "$host" 'sudo -S docker-compose up -d 2>/dev/null'
+    echo "$PASS" | ssh "$host" 'sudo -S docker compose up -d 2>/dev/null'
 
 }
 
@@ -69,7 +69,7 @@ function sync_test {
     fi
 
     ssh "$host" "docker buildx build --build-arg INSTALL_DEBUG=1 --platform $platform -t bbilly1/tubearchivist:latest tubearchivist --load"
-    ssh "$host" 'docker-compose -f docker/docker-compose.yml up -d'
+    ssh "$host" 'docker compose -f docker/docker-compose.yml up -d'
 
 }
 

docker-compose.yml

@@ -33,7 +33,7 @@ services:
     depends_on:
       - archivist-es
   archivist-es:
-    image: bbilly1/tubearchivist-es         # only for amd64, or use official es 7.17.2
+    image: bbilly1/tubearchivist-es         # only for amd64, or use official es 7.17.3
     container_name: archivist-es
     restart: always
     environment:

docs/Settings.md

@@ -32,6 +32,29 @@ Additional settings passed to yt-dlp.
 - **Source Settings**: User created subtitles are provided from the uploader and are usually the video script. Auto generated is from YouTube, quality varies, particularly for auto translated tracks.
 - **Index Settings**: Enabling subtitle indexing will add the lines to Elasticsearch and will make subtitles searchable. This will increase the index size and is not recommended on low-end hardware.
 
+## Cookie
+Importing your YouTube Cookie into Tube Archivist allows yt-dlp to bypass age restrictions, gives access to private videos and your *watch later* or *liked videos*.
+
+### Security concerns
+Cookies are used to store your session and contain your access token to your google account, this file can be used to take over your account. Treat that file with utmost care as you would any other password or credential. *Tube Archivist* stores your cookie in Redis and will export the file container internally if it's needed.
+
+### Export your cookie
+- Install **Cookies.txt** addon for [chrome](https://chrome.google.com/webstore/detail/get-cookiestxt/bgaddhkoddajcdgocldbbfleckgcbcid) or [firefox](https://addons.mozilla.org/firefox/addon/cookies-txt).
+- Visit YouTube and login with whichever YouTube account you wish to use to generate the cookies.
+- Click on the extension icon in the toolbar - it will drop down showing the active cookies for YT.
+- Click Export to export the cookies, filename is by default *cookies.google.txt*.
+
+### Import your cookie
+Place the file *cookies.google.txt* into the *cache/import* folder of Tube Archivist and enable the cookie import. Once you click on *Update Application Configurations* to save your changes, your cookie will get imported and stored internally.
+
+Once imported, a **Validate Cookie File** button will show, where you can confirm if your cookie is working or not.
+
+### Use your cookie
+Once imported, additionally to the advantages above, your [Watch Later](https://www.youtube.com/playlist?list=WL) and [Liked Videos](https://www.youtube.com/playlist?list=LL) become a regular playlist you can download and subscribe to as any other [playlist](Playlists).
+
+### Limitation
+There is only one cookie per Tube Archivist instance, this will be shared between all users.
+
 ## Integrations
 All third party integrations of TubeArchivist will **always** be *opt in*.
 - **API**: Your access token for the Tube Archivist API.

tubearchivist/api/README.md

@@ -20,6 +20,19 @@ headers = {"Authorization": "Token xxxxxxxxxx"}
 response = requests.get(url, headers=headers)
 ```
 
+## Pagination
+The list views return a paginate object with the following keys:
+- page_size: int current page size set in config
+- page_from: int first result idx
+- prev_pages: array of ints of previous pages, if available
+- current_page: int current page from query
+- max_hits: reached: bool if max of 10k results is reached
+- last_page: int of last page link
+- next_pages: array of ints of next pages
+- total_hits: int total results
+
+Pass page number as a query parameter: `page=2`. Defaults to *0*, `page=1` is redundant and falls back to *0*. If a page query doesn't return any results, you'll get `HTTP 404 Not Found`.
+
 ## Login View
 Return token and user ID for username and password:  
 POST /api/login
@@ -123,14 +136,23 @@ POST /api/channel/
 ## Channel Item View
 /api/channel/\<channel_id>/
 
+## Channel Videos View
+/api/channel/\<channel_id>/video/
+
 ## Playlist List View
 /api/playlist/
 
 ## Playlists Item View
 /api/playlist/\<playlist_id>/
 
+## Playlist Videos View
+/api/playlist/\<playlist_id>/video/
+
 ## Download Queue List View
-/api/download/
+GET /api/download/
+
+Parameter:
+- filter: pending, ignore
 
 ### Add list of videos to download queue
 POST /api/download/
@@ -142,9 +164,30 @@ POST /api/download/
 }
 ```
 
-## Download Queue Item View
+### Delete download queue items by filter
-/api/download/\<video_id>/
+DELETE /api/download/?filter=ignore  
+DELETE /api/download/?filter=pending
 
+## Download Queue Item View
+GET /api/download/\<video_id>/  
+POST /api/download/\<video_id>/
+
+Ignore video in download queue:
+```json
+{
+    "status": "ignore"
+}
+```
+
+Add to queue previously ignored video:
+```json
+{
+    "status": "pending"
+}
+```
+
+DELETE /api/download/\<video_id>/  
+Forget or delete from download queue
 
 ## Ping View
 Validate your connection with the API  
@@ -157,3 +200,34 @@ When valid returns message with user id:
     "user": 1
 }
 ```
+
+## Task View
+Start a background task
+POST /api/task/
+```json
+{
+    "run": "task_name"
+}
+```
+
+List of valid task names:
+- **download_pending**: Start the download queue
+- **rescan_pending**: Rescan your subscriptions
+
+
+## Cookie View
+Check your youtube cookie settings  
+GET /api/cookie/
+```json
+{
+    "cookie_enabled": true
+}
+```
+
+POST /api/cookie/  
+Send empty post request to validate cookie.
+```json
+{
+    "cookie_validated": true
+}
+```

tubearchivist/api/src/search_processor.py

@@ -7,12 +7,16 @@ Functionality:
 import urllib.parse
 
 from home.src.download.thumbnails import ThumbManager
+from home.src.ta.config import AppConfig
 from home.src.ta.helper import date_praser
 
 
 class SearchProcess:
     """process search results"""
 
+    CONFIG = AppConfig().config
+    CACHE_DIR = CONFIG["application"]["cache_dir"]
+
     def __init__(self, response):
         self.response = response
         self.processed = False
@@ -42,6 +46,8 @@ class SearchProcess:
             processed = self._process_channel(result["_source"])
         if index == "ta_playlist":
             processed = self._process_playlist(result["_source"])
+        if index == "ta_download":
+            processed = self._process_download(result["_source"])
 
         return processed
 
@@ -71,13 +77,18 @@ class SearchProcess:
         vid_thumb_url = ThumbManager().vid_thumb_path(video_id)
         channel = self._process_channel(video_dict["channel"])
 
+        if "subtitles" in video_dict:
+            for idx, _ in enumerate(video_dict["subtitles"]):
+                url = video_dict["subtitles"][idx]["media_url"]
+                video_dict["subtitles"][idx]["media_url"] = f"/media/{url}"
+
         video_dict.update(
             {
                 "channel": channel,
-                "media_url": media_url,
+                "media_url": f"/media/{media_url}",
                 "vid_last_refresh": vid_last_refresh,
                 "published": published,
-                "vid_thumb_url": vid_thumb_url,
+                "vid_thumb_url": f"{self.CACHE_DIR}/{vid_thumb_url}",
             }
         )
 
@@ -98,3 +109,17 @@ class SearchProcess:
         )
 
         return dict(sorted(playlist_dict.items()))
+
+    def _process_download(self, download_dict):
+        """run on single download item"""
+        video_id = download_dict["youtube_id"]
+        vid_thumb_url = ThumbManager().vid_thumb_path(video_id)
+        published = date_praser(download_dict["published"])
+
+        download_dict.update(
+            {
+                "vid_thumb_url": f"{self.CACHE_DIR}/{vid_thumb_url}",
+                "published": published,
+            }
+        )
+        return dict(sorted(download_dict.items()))

tubearchivist/api/src/task_processor.py

@@ -0,0 +1,54 @@
+"""
+Functionality:
+- process tasks from API
+- validate
+- handover to celery
+"""
+
+from home.src.ta.ta_redis import RedisArchivist
+from home.tasks import download_pending, update_subscribed
+
+
+class TaskHandler:
+    """handle tasks from api"""
+
+    def __init__(self, data):
+        self.data = data
+
+    def run_task(self):
+        """map data and run"""
+        task_name = self.data["run"]
+        try:
+            to_run = self.exec_map(task_name)
+        except KeyError as err:
+            print(f"invalid task name {task_name}")
+            raise ValueError from err
+
+        response = to_run()
+        response.update({"task": task_name})
+        return response
+
+    def exec_map(self, task_name):
+        """map dict key and return function to execute"""
+        exec_map = {
+            "download_pending": self._download_pending,
+            "rescan_pending": self._rescan_pending,
+        }
+
+        return exec_map[task_name]
+
+    @staticmethod
+    def _rescan_pending():
+        """look for new items in subscribed channels"""
+        print("rescan subscribed channels")
+        update_subscribed.delay()
+        return {"success": True}
+
+    @staticmethod
+    def _download_pending():
+        """start the download queue"""
+        print("download pending")
+        running = download_pending.delay()
+        print("set task id: " + running.id)
+        RedisArchivist().set_message("dl_queue_id", running.id, expire=False)
+        return {"success": True}

tubearchivist/api/urls.py

@@ -2,13 +2,17 @@
 
 from api.views import (
     ChannelApiListView,
+    ChannelApiVideoView,
     ChannelApiView,
+    CookieView,
     DownloadApiListView,
     DownloadApiView,
     LoginApiView,
     PingView,
     PlaylistApiListView,
+    PlaylistApiVideoView,
     PlaylistApiView,
+    TaskApiView,
     VideoApiListView,
     VideoApiView,
     VideoProgressView,
@@ -50,15 +54,25 @@ urlpatterns = [
         name="api-channel",
     ),
     path(
-        "playlist/<slug:playlist_id>/",
+        "channel/<slug:channel_id>/video/",
-        PlaylistApiView.as_view(),
+        ChannelApiVideoView.as_view(),
-        name="api-playlist",
+        name="api-channel-video",
     ),
     path(
         "playlist/",
         PlaylistApiListView.as_view(),
         name="api-playlist-list",
     ),
+    path(
+        "playlist/<slug:playlist_id>/",
+        PlaylistApiView.as_view(),
+        name="api-playlist",
+    ),
+    path(
+        "playlist/<slug:playlist_id>/video/",
+        PlaylistApiVideoView.as_view(),
+        name="api-playlist-video",
+    ),
     path(
         "download/",
         DownloadApiListView.as_view(),
@@ -69,4 +83,14 @@ urlpatterns = [
         DownloadApiView.as_view(),
         name="api-download",
     ),
+    path(
+        "task/",
+        TaskApiView.as_view(),
+        name="api-task",
+    ),
+    path(
+        "cookie/",
+        CookieView.as_view(),
+        name="api-cookie",
+    ),
 ]

tubearchivist/api/views.py

@@ -1,12 +1,15 @@
 """all API views"""
 
 from api.src.search_processor import SearchProcess
-from home.src.download.thumbnails import ThumbManager
+from api.src.task_processor import TaskHandler
+from home.src.download.queue import PendingInteract
+from home.src.download.yt_cookie import CookieHandler
 from home.src.es.connect import ElasticWrap
+from home.src.index.generic import Pagination
 from home.src.index.video import SponsorBlock
 from home.src.ta.config import AppConfig
 from home.src.ta.helper import UrlListParser
-from home.src.ta.ta_redis import RedisArchivist
+from home.src.ta.ta_redis import RedisArchivist, RedisQueue
 from home.tasks import extrac_dl, subscribe_to
 from rest_framework.authentication import (
     SessionAuthentication,
@@ -25,12 +28,15 @@ class ApiBaseView(APIView):
     authentication_classes = [SessionAuthentication, TokenAuthentication]
     permission_classes = [IsAuthenticated]
     search_base = False
+    data = False
 
     def __init__(self):
         super().__init__()
         self.response = {"data": False, "config": AppConfig().config}
+        self.data = {"query": {"match_all": {}}}
         self.status_code = False
         self.context = False
+        self.pagination_handler = False
 
     def get_document(self, document_id):
         """get single document from es"""
@@ -44,35 +50,32 @@ class ApiBaseView(APIView):
             self.response["data"] = False
         self.status_code = status_code
 
-    def process_keys(self):
+    def initiate_pagination(self, request):
-        """process keys for frontend"""
+        """set initial pagination values"""
-        all_keys = self.response["data"].keys()
+        user_id = request.user.id
-        if "media_url" in all_keys:
+        page_get = int(request.GET.get("page", 0))
-            media_url = self.response["data"]["media_url"]
+        self.pagination_handler = Pagination(page_get, user_id)
-            self.response["data"]["media_url"] = f"/media/{media_url}"
+        self.data.update(
-        if "vid_thumb_url" in all_keys:
+            {
-            youtube_id = self.response["data"]["youtube_id"]
+                "size": self.pagination_handler.pagination["page_size"],
-            vid_thumb_url = ThumbManager().vid_thumb_path(youtube_id)
+                "from": self.pagination_handler.pagination["page_from"],
-            cache_dir = self.response["config"]["application"]["cache_dir"]
+            }
-            new_thumb = f"{cache_dir}/{vid_thumb_url}"
+        )
-            self.response["data"]["vid_thumb_url"] = new_thumb
-        if "subtitles" in all_keys:
-            all_subtitles = self.response["data"]["subtitles"]
-            for idx, _ in enumerate(all_subtitles):
-                url = self.response["data"]["subtitles"][idx]["media_url"]
-                new_url = f"/media/{url}"
-                self.response["data"]["subtitles"][idx]["media_url"] = new_url
 
-    def get_paginate(self):
+    def get_document_list(self, request):
-        """add pagination detail to response"""
-        self.response["paginate"] = False
-
-    def get_document_list(self, data):
         """get a list of results"""
         print(self.search_base)
-        response, status_code = ElasticWrap(self.search_base).get(data=data)
+        self.initiate_pagination(request)
+        es_handler = ElasticWrap(self.search_base)
+        response, status_code = es_handler.get(data=self.data)
         self.response["data"] = SearchProcess(response).process()
-        self.status_code = status_code
+        if self.response["data"]:
+            self.status_code = status_code
+        else:
+            self.status_code = 404
+
+        self.pagination_handler.validate(response["hits"]["total"]["value"])
+        self.response["paginate"] = self.pagination_handler.pagination
 
 
 class VideoApiView(ApiBaseView):
@@ -86,8 +89,6 @@ class VideoApiView(ApiBaseView):
         # pylint: disable=unused-argument
         """get request"""
         self.get_document(video_id)
-        if self.response.get("data"):
-            self.process_keys()
         return Response(self.response, status=self.status_code)
 
 
@@ -99,11 +100,9 @@ class VideoApiListView(ApiBaseView):
     search_base = "ta_video/_search/"
 
     def get(self, request):
-        # pylint: disable=unused-argument
         """get request"""
-        data = {"query": {"match_all": {}}}
+        self.data.update({"sort": [{"published": {"order": "desc"}}]})
-        self.get_document_list(data)
+        self.get_document_list(request)
-        self.get_paginate()
 
         return Response(self.response)
 
@@ -218,11 +217,11 @@ class ChannelApiListView(ApiBaseView):
     search_base = "ta_channel/_search/"
 
     def get(self, request):
-        # pylint: disable=unused-argument
         """get request"""
-        data = {"query": {"match_all": {}}}
+        self.get_document_list(request)
-        self.get_document_list(data)
+        self.data.update(
-        self.get_paginate()
+            {"sort": [{"channel_name.keyword": {"order": "asc"}}]}
+        )
 
         return Response(self.response)
 
@@ -244,6 +243,44 @@ class ChannelApiListView(ApiBaseView):
         return Response(data)
 
 
+class ChannelApiVideoView(ApiBaseView):
+    """resolves to /api/channel/<channel-id>/video
+    GET: returns a list of videos of channel
+    """
+
+    search_base = "ta_video/_search/"
+
+    def get(self, request, channel_id):
+        """handle get request"""
+        self.data.update(
+            {
+                "query": {
+                    "term": {"channel.channel_id": {"value": channel_id}}
+                },
+                "sort": [{"published": {"order": "desc"}}],
+            }
+        )
+        self.get_document_list(request)
+
+        return Response(self.response, status=self.status_code)
+
+
+class PlaylistApiListView(ApiBaseView):
+    """resolves to /api/playlist/
+    GET: returns list of indexed playlists
+    """
+
+    search_base = "ta_playlist/_search/"
+
+    def get(self, request):
+        """handle get request"""
+        self.data.update(
+            {"sort": [{"playlist_name.keyword": {"order": "asc"}}]}
+        )
+        self.get_document_list(request)
+        return Response(self.response)
+
+
 class PlaylistApiView(ApiBaseView):
     """resolves to /api/playlist/<playlist_id>/
     GET: returns metadata dict of playlist
@@ -258,28 +295,33 @@ class PlaylistApiView(ApiBaseView):
         return Response(self.response, status=self.status_code)
 
 
-class PlaylistApiListView(ApiBaseView):
+class PlaylistApiVideoView(ApiBaseView):
-    """resolves to /api/playlist/
+    """resolves to /api/playlist/<playlist_id>/video
-    GET: returns list of indexed playlists
+    GET: returns list of videos in playlist
     """
 
-    search_base = "ta_playlist/_search/"
+    search_base = "ta_video/_search/"
 
-    def get(self, request):
+    def get(self, request, playlist_id):
-        # pylint: disable=unused-argument
         """handle get request"""
-        data = {"query": {"match_all": {}}}
+        self.data["query"] = {
-        self.get_document_list(data)
+            "term": {"playlist.keyword": {"value": playlist_id}}
-        self.get_paginate()
+        }
-        return Response(self.response)
+        self.data.update({"sort": [{"published": {"order": "desc"}}]})
+
+        self.get_document_list(request)
+        return Response(self.response, status=self.status_code)
 
 
 class DownloadApiView(ApiBaseView):
     """resolves to /api/download/<video_id>/
     GET: returns metadata dict of an item in the download queue
+    POST: update status of item to pending or ignore
+    DELETE: forget from download queue
     """
 
     search_base = "ta_download/_doc/"
+    valid_status = ["pending", "ignore"]
 
     def get(self, request, video_id):
         # pylint: disable=unused-argument
@@ -287,21 +329,53 @@ class DownloadApiView(ApiBaseView):
         self.get_document(video_id)
         return Response(self.response, status=self.status_code)
 
+    def post(self, request, video_id):
+        """post to video to change status"""
+        item_status = request.data["status"]
+        if item_status not in self.valid_status:
+            message = f"{video_id}: invalid status {item_status}"
+            print(message)
+            return Response({"message": message}, status=400)
+
+        print(f"{video_id}: change status to {item_status}")
+        PendingInteract(video_id=video_id, status=item_status).update_status()
+        RedisQueue().clear_item(video_id)
+
+        return Response(request.data)
+
+    @staticmethod
+    def delete(request, video_id):
+        # pylint: disable=unused-argument
+        """delete single video from queue"""
+        print(f"{video_id}: delete from queue")
+        PendingInteract(video_id=video_id).delete_item()
+
+        return Response({"success": True})
+
 
 class DownloadApiListView(ApiBaseView):
     """resolves to /api/download/
     GET: returns latest videos in the download queue
     POST: add a list of videos to download queue
+    DELETE: remove items based on query filter
     """
 
     search_base = "ta_download/_search/"
+    valid_filter = ["pending", "ignore"]
 
     def get(self, request):
-        # pylint: disable=unused-argument
         """get request"""
-        data = {"query": {"match_all": {}}}
+        query_filter = request.GET.get("filter", False)
-        self.get_document_list(data)
+        self.data.update({"sort": [{"timestamp": {"order": "asc"}}]})
-        self.get_paginate()
+        if query_filter:
+            if query_filter not in self.valid_filter:
+                message = f"invalid url query filder: {query_filter}"
+                print(message)
+                return Response({"message": message}, status=400)
+
+            self.data["query"] = {"term": {"status": {"value": query_filter}}}
+
+        self.get_document_list(request)
         return Response(self.response)
 
     @staticmethod
@@ -329,6 +403,20 @@ class DownloadApiListView(ApiBaseView):
 
         return Response(data)
 
+    def delete(self, request):
+        """delete download queue"""
+        query_filter = request.GET.get("filter", False)
+        if query_filter not in self.valid_filter:
+            message = f"invalid url query filter: {query_filter}"
+            print(message)
+            return Response({"message": message}, status=400)
+
+        message = f"delete queue by status: {query_filter}"
+        print(message)
+        PendingInteract(status=query_filter).delete_by_status()
+
+        return Response({"message": message})
+
 
 class PingView(ApiBaseView):
     """resolves to /api/ping/
@@ -360,3 +448,42 @@ class LoginApiView(ObtainAuthToken):
         print(f"returning token for user with id {user.pk}")
 
         return Response({"token": token.key, "user_id": user.pk})
+
+
+class TaskApiView(ApiBaseView):
+    """resolves to /api/task/
+    POST: start a new background task
+    """
+
+    def post(self, request):
+        """handle post request"""
+
+        data = request.data
+        print(data)
+        response = TaskHandler(data).run_task()
+
+        return Response(response)
+
+
+class CookieView(ApiBaseView):
+    """resolves to /api/cookie/
+    GET: check if cookie is enabled
+    POST: verify validity of cookie
+    """
+
+    @staticmethod
+    def get(request):
+        """handle get request"""
+        # pylint: disable=unused-argument
+        config = AppConfig().config
+        cookie_enabled = config["downloads"]["cookie_import"]
+
+        return Response({"cookie_enabled": cookie_enabled})
+
+    @staticmethod
+    def post(request):
+        """handle post request"""
+        # pylint: disable=unused-argument
+        validated = CookieHandler().validate()
+
+        return Response({"cookie_validated": validated})

tubearchivist/config/settings.py

@@ -146,7 +146,16 @@ LOGOUT_REDIRECT_URL = "/login/"
 
 # Cors needed for browser extension
 # background.js makes the request so HTTP_ORIGIN will be from extension
-CORS_ALLOWED_ORIGIN_REGEXES = [r"moz-extension://*", r"chrome-extension://*"]
+if environ.get("DISABLE_CORS"):
+    # disable cors
+    CORS_ORIGIN_ALLOW_ALL = True
+else:
+    CORS_ALLOWED_ORIGIN_REGEXES = [
+        r"moz-extension://*",
+        r"chrome-extension://*",
+    ]
+    CORS_ALLOWED_ORIGINS = ["http://localhost:3000"]
+
 
 CORS_ALLOW_HEADERS = list(default_headers) + [
     "mode",
@@ -154,4 +163,4 @@ CORS_ALLOW_HEADERS = list(default_headers) + [
 
 # TA application settings
 TA_UPSTREAM = "https://github.com/tubearchivist/tubearchivist"
-TA_VERSION = "v0.1.4"
+TA_VERSION = "v0.1.5"

tubearchivist/home/config.json

@@ -26,6 +26,7 @@
         "subtitle": false,
         "subtitle_source": false,
         "subtitle_index": false,
+        "cookie_import": false,
         "throttledratelimit": false,
         "integrate_ryd": false,
         "integrate_sponsorblock": false

tubearchivist/home/src/download/queue.py

@@ -13,8 +13,10 @@ from home.src.download.subscriptions import (
     PlaylistSubscription,
 )
 from home.src.download.thumbnails import ThumbManager
+from home.src.download.yt_cookie import CookieHandler
 from home.src.es.connect import ElasticWrap, IndexPaginate
 from home.src.index.playlist import YoutubePlaylist
+from home.src.ta.config import AppConfig
 from home.src.ta.helper import DurationConverter
 from home.src.ta.ta_redis import RedisArchivist
 
@@ -101,7 +103,7 @@ class PendingInteract:
     def delete_item(self):
         """delete single item from pending"""
         path = f"ta_download/_doc/{self.video_id}"
-        _, _ = ElasticWrap(path).delete()
+        _, _ = ElasticWrap(path).delete(refresh=True)
 
     def delete_by_status(self):
         """delete all matching item by status"""
@@ -119,12 +121,40 @@ class PendingInteract:
 class PendingList(PendingIndex):
     """manage the pending videos list"""
 
+    yt_obs = {
+        "default_search": "ytsearch",
+        "quiet": True,
+        "check_formats": "selected",
+        "noplaylist": True,
+        "writethumbnail": True,
+        "simulate": True,
+        "socket_timeout": 3,
+    }
+
     def __init__(self, youtube_ids=False):
         super().__init__()
+        self.process_config()
         self.youtube_ids = youtube_ids
         self.to_skip = False
         self.missing_videos = False
 
+    def process_config(self):
+        """add user config to yt_obs"""
+        config = AppConfig().config
+        if config["downloads"]["cookie_import"]:
+            cookie_path = CookieHandler().use()
+            self.yt_obs.update({"cookiefile": cookie_path})
+
+    def close_config(self):
+        """remove config after task finished"""
+        config = AppConfig().config
+        if config["downloads"]["cookie_import"]:
+            CookieHandler().hide()
+            try:
+                del self.yt_obs["cookiefile"]
+            except KeyError:
+                pass
+
     def parse_url_list(self):
         """extract youtube ids from list"""
         self.missing_videos = []
@@ -199,10 +229,13 @@ class PendingList(PendingIndex):
             thumb_handler.download_vid(thumb_needed)
             self._notify_add(idx)
 
-        # add last newline
+        if bulk_list:
-        bulk_list.append("\n")
+            # add last newline
-        query_str = "\n".join(bulk_list)
+            bulk_list.append("\n")
-        _, _ = ElasticWrap("_bulk").post(query_str, ndjson=True)
+            query_str = "\n".join(bulk_list)
+            _, _ = ElasticWrap("_bulk").post(query_str, ndjson=True)
+
+        self.close_config()
 
     def _notify_add(self, idx):
         """send notification for adding videos to download queue"""
@@ -223,18 +256,14 @@ class PendingList(PendingIndex):
 
     def get_youtube_details(self, youtube_id):
         """get details from youtubedl for single pending video"""
-        obs = {
-            "default_search": "ytsearch",
-            "quiet": True,
-            "check_formats": "selected",
-            "noplaylist": True,
-            "writethumbnail": True,
-            "simulate": True,
-        }
         try:
-            vid = yt_dlp.YoutubeDL(obs).extract_info(youtube_id)
+            vid = yt_dlp.YoutubeDL(self.yt_obs).extract_info(youtube_id)
         except yt_dlp.utils.DownloadError:
-            print("failed to extract info for: " + youtube_id)
+            print(f"{youtube_id}: failed to extract info")
+            return False
+        if vid.get("id") != youtube_id:
+            # skip premium videos with different id
+            print(f"{youtube_id}: skipping premium video, id not matching")
             return False
         # stop if video is streaming live now
         if vid["is_live"]:

tubearchivist/home/src/download/subscriptions.py

@@ -168,7 +168,7 @@ class PlaylistSubscription:
     def channel_validate(channel_id):
         """make sure channel of playlist is there"""
         channel = YoutubeChannel(channel_id)
-        channel.build_json()
+        channel.build_json(upload=True)
 
     @staticmethod
     def change_subscribe(playlist_id, subscribe_status):

tubearchivist/home/src/download/thumbnails.py

@@ -296,6 +296,7 @@ class ThumbManager:
     def get_thumb_list(self):
         """get list of mediafiles and matching thumbnails"""
         pending = queue.PendingList()
+        pending.get_download()
         pending.get_indexed()
 
         video_list = []

tubearchivist/home/src/download/yt_cookie.py

@@ -0,0 +1,88 @@
+"""
+functionality:
+- import yt cookie from filesystem
+- make cookie available for yt-dlp
+"""
+
+import os
+
+import yt_dlp
+from home.src.ta.config import AppConfig
+from home.src.ta.ta_redis import RedisArchivist
+
+
+class CookieHandler:
+    """handle youtube cookie for yt-dlp"""
+
+    CONFIG = AppConfig().config
+    CACHE_PATH = CONFIG["application"]["cache_dir"]
+    COOKIE_FILE_NAME = "cookies.google.txt"
+    COOKIE_KEY = "cookie"
+    COOKIE_PATH = "cookie.txt"
+
+    def import_cookie(self):
+        """import cookie from file"""
+        import_path = os.path.join(
+            self.CACHE_PATH, "import", self.COOKIE_FILE_NAME
+        )
+        with open(import_path, encoding="utf-8") as cookie_file:
+            cookie = cookie_file.read()
+
+        RedisArchivist().set_message(self.COOKIE_KEY, cookie, expire=False)
+
+        os.remove(import_path)
+        print("cookie: import successfully")
+
+    def use(self):
+        """make cookie available in FS"""
+        cookie = RedisArchivist().get_message(self.COOKIE_KEY)
+        if isinstance(cookie, dict):
+            print("no cookie imported")
+            raise FileNotFoundError
+
+        if os.path.exists(self.COOKIE_PATH):
+            return self.COOKIE_PATH
+
+        with open(self.COOKIE_PATH, "w", encoding="utf-8") as cookie_file:
+            cookie_file.write(cookie)
+
+        print("cookie: made available")
+        return self.COOKIE_PATH
+
+    def hide(self):
+        """hide cookie file if not in use"""
+        try:
+            os.remove(self.COOKIE_PATH)
+        except FileNotFoundError:
+            print("cookie: not available")
+            return
+
+        print("cookie: hidden")
+
+    def revoke(self):
+        """revoke cookie"""
+        self.hide()
+        RedisArchivist().del_message(self.COOKIE_KEY)
+        print("cookie: revoked")
+
+    def validate(self):
+        """validate cookie using the liked videos playlist"""
+        try:
+            _ = self.use()
+        except FileNotFoundError:
+            return False
+
+        url = "https://www.youtube.com/playlist?list=LL"
+        yt_obs = {
+            "quiet": True,
+            "skip_download": True,
+            "extract_flat": True,
+            "cookiefile": self.COOKIE_PATH,
+        }
+        try:
+            response = yt_dlp.YoutubeDL(yt_obs).extract_info(url)
+        except yt_dlp.utils.DownloadError:
+            print("failed to validate cookie")
+            response = False
+
+        return bool(response)

tubearchivist/home/src/download/yt_dlp_handler.py

@@ -14,6 +14,7 @@ from time import sleep
 import yt_dlp
 from home.src.download.queue import PendingList
 from home.src.download.subscriptions import PlaylistSubscription
+from home.src.download.yt_cookie import CookieHandler
 from home.src.es.connect import ElasticWrap, IndexPaginate
 from home.src.index.channel import YoutubeChannel
 from home.src.index.playlist import YoutubePlaylist
@@ -40,6 +41,7 @@ class DownloadPostProcess:
         self.auto_delete_all()
         self.auto_delete_overwrites()
         self.validate_playlists()
+        self.pending.close_config()
 
     def auto_delete_all(self):
         """handle auto delete"""
@@ -280,6 +282,7 @@ class VideoDownloader:
             "writethumbnail": False,
             "noplaylist": True,
             "check_formats": "selected",
+            "socket_timeout": 3,
         }
 
     def _build_obs_user(self):
@@ -290,6 +293,9 @@ class VideoDownloader:
             self.obs["ratelimit"] = (
                 self.config["downloads"]["limit_speed"] * 1024
             )
+        if self.config["downloads"]["cookie_import"]:
+            cookie_path = CookieHandler().use()
+            self.obs["cookiefile"] = cookie_path
 
         throttle = self.config["downloads"]["throttledratelimit"]
         if throttle:

tubearchivist/home/src/es/connect.py

@@ -75,8 +75,10 @@ class ElasticWrap:
 
         return response.json(), response.status_code
 
-    def delete(self, data=False):
+    def delete(self, data=False, refresh=False):
         """delete document from es"""
+        if refresh:
+            self.url = f"{self.url}/?refresh=true"
         if data:
             response = requests.delete(self.url, json=data, auth=self.auth)
         else:

tubearchivist/home/src/frontend/api_calls.py

@@ -19,7 +19,6 @@ from home.src.ta.ta_redis import RedisArchivist, RedisQueue
 from home.tasks import (
     download_pending,
     download_single,
-    extrac_dl,
     index_channel_playlists,
     kill_dl,
     re_sync_thumbs,
@@ -111,7 +110,7 @@ class PostData:
     def _ignore(self):
         """ignore from download queue"""
         video_id = self.exec_val
-        print(f"ignore video {video_id}")
+        print(f"{video_id}: ignore video from download queue")
         PendingInteract(video_id=video_id, status="ignore").update_status()
         # also clear from redis queue
         RedisQueue().clear_item(video_id)
@@ -123,7 +122,7 @@ class PostData:
         print("download pending")
         running = download_pending.delay()
         task_id = running.id
-        print("set task id: " + task_id)
+        print(f"{task_id}: set task id")
         RedisArchivist().set_message("dl_queue_id", task_id, expire=False)
         return {"success": True}
 
@@ -146,7 +145,7 @@ class PostData:
     def _unsubscribe(self):
         """unsubscribe from channels or playlists"""
         id_unsub = self.exec_val
-        print("unsubscribe from " + id_unsub)
+        print(f"{id_unsub}: unsubscribe")
         to_unsub_list = UrlListParser(id_unsub).process_list()
         for to_unsub in to_unsub_list:
             unsub_type = to_unsub["type"]
@@ -167,7 +166,7 @@ class PostData:
     def _subscribe(self):
         """subscribe to channel or playlist, called from js buttons"""
         id_sub = self.exec_val
-        print("subscribe to " + id_sub)
+        print(f"{id_sub}: subscribe")
         subscribe_to.delay(id_sub)
         return {"success": True}
 
@@ -203,7 +202,7 @@ class PostData:
     def _dlnow(self):
         """start downloading single vid now"""
         youtube_id = self.exec_val
-        print("downloading: " + youtube_id)
+        print(f"{youtube_id}: downloading now")
         running = download_single.delay(youtube_id=youtube_id)
         task_id = running.id
         print("set task id: " + task_id)
@@ -222,17 +221,15 @@ class PostData:
     def _forget_ignore(self):
         """delete from ta_download index"""
         video_id = self.exec_val
-        print(f"forgetting from download index: {video_id}")
+        print(f"{video_id}: forget from download")
         PendingInteract(video_id=video_id).delete_item()
         return {"success": True}
 
     def _add_single(self):
         """add single youtube_id to download queue"""
         video_id = self.exec_val
-        print(f"add vid to dl queue: {video_id}")
+        print(f"{video_id}: add single vid to download queue")
-        PendingInteract(video_id=video_id).delete_item()
+        PendingInteract(video_id=video_id, status="pending").update_status()
-        video_ids = UrlListParser(video_id).process_list()
-        extrac_dl.delay(video_ids)
         return {"success": True}
 
     def _delete_queue(self):

tubearchivist/home/src/frontend/forms.py

@@ -86,6 +86,12 @@ class ApplicationSettingsForm(forms.Form):
         ("1", "enable subtitle index"),
     ]
 
+    COOKIE_IMPORT_CHOICES = [
+        ("", "-- change cookie settings"),
+        ("0", "disable cookie"),
+        ("1", "enable cookie"),
+    ]
+
     subscriptions_channel_size = forms.IntegerField(required=False)
     downloads_limit_count = forms.IntegerField(required=False)
     downloads_limit_speed = forms.IntegerField(required=False)
@@ -106,6 +112,9 @@ class ApplicationSettingsForm(forms.Form):
     downloads_subtitle_index = forms.ChoiceField(
         widget=forms.Select, choices=SUBTITLE_INDEX_CHOICES, required=False
     )
+    downloads_cookie_import = forms.ChoiceField(
+        widget=forms.Select, choices=COOKIE_IMPORT_CHOICES, required=False
+    )
     downloads_integrate_ryd = forms.ChoiceField(
         widget=forms.Select, choices=RYD_CHOICES, required=False
     )

tubearchivist/home/src/index/filesystem.py

@@ -12,6 +12,7 @@ import shutil
 import subprocess
 
 from home.src.download.queue import PendingList
+from home.src.download.yt_cookie import CookieHandler
 from home.src.download.yt_dlp_handler import VideoDownloader
 from home.src.es.connect import ElasticWrap
 from home.src.index.reindex import Reindex
@@ -308,6 +309,12 @@ def scan_filesystem():
 def reindex_old_documents():
     """daily refresh of old documents"""
     handler = Reindex()
-    handler.check_outdated()
+    if handler.config["downloads"]["cookie_import"]:
-    handler.reindex()
+        CookieHandler().use()
-    RedisArchivist().set_message("last_reindex", handler.now, expire=False)
+    try:
+        handler.check_outdated()
+        handler.reindex()
+        RedisArchivist().set_message("last_reindex", handler.now, expire=False)
+    finally:
+        if handler.config["downloads"]["cookie_import"]:
+            CookieHandler().hide()

tubearchivist/home/src/index/generic.py

@@ -6,6 +6,7 @@ functionality:
 import math
 
 import yt_dlp
+from home.src.download.yt_cookie import CookieHandler
 from home.src.es.connect import ElasticWrap
 from home.src.ta.config import AppConfig
 from home.src.ta.ta_redis import RedisArchivist
@@ -22,6 +23,7 @@ class YouTubeItem:
         "default_search": "ytsearch",
         "skip_download": True,
         "check_formats": "selected",
+        "socket_timeout": 3,
         "noplaylist": True,
     }
 
@@ -37,6 +39,9 @@ class YouTubeItem:
         """read user conf"""
         self.config = AppConfig().config
         self.app_conf = self.config["application"]
+        if self.config["downloads"]["cookie_import"]:
+            cookie_path = CookieHandler().use()
+            self.yt_obs.update({"cookiefile": cookie_path})
 
     def get_from_youtube(self):
         """use yt-dlp to get meta data from youtube"""
@@ -81,7 +86,7 @@ class YouTubeItem:
     def del_in_es(self):
         """delete item from elastic search"""
         print(f"{self.youtube_id}: delete from es")
-        _, _ = ElasticWrap(self.es_path).delete()
+        _, _ = ElasticWrap(self.es_path).delete(refresh=True)
 
 
 class Pagination:
@@ -147,3 +152,4 @@ class Pagination:
         ]
 
         self.pagination["next_pages"] = next_pages
+        self.pagination["total_hits"] = total_hits

tubearchivist/home/src/index/reindex.py

@@ -4,12 +4,15 @@ functionality:
 - index and update in es
 """
 
+import os
+import shutil
 from datetime import datetime
 from math import ceil
 from time import sleep
 
 from home.src.download.queue import PendingList
 from home.src.download.thumbnails import ThumbManager
+from home.src.download.yt_dlp_handler import VideoDownloader
 from home.src.es.connect import ElasticWrap
 from home.src.index.channel import YoutubeChannel
 from home.src.index.playlist import YoutubePlaylist
@@ -211,7 +214,12 @@ class Reindex:
         # videos
         print(f"reindexing {len(self.all_youtube_ids)} videos")
         for youtube_id in self.all_youtube_ids:
-            self._reindex_single_video(youtube_id)
+            try:
+                self._reindex_single_video(youtube_id)
+            except FileNotFoundError:
+                # handle channel name change here
+                ChannelUrlFixer(youtube_id, self.config).run()
+                self._reindex_single_video(youtube_id)
             if sleep_interval:
                 sleep(sleep_interval)
         # channels
@@ -231,3 +239,62 @@ class Reindex:
                 self._reindex_single_playlist(playlist_id, all_indexed_ids)
                 if sleep_interval:
                     sleep(sleep_interval)
+
+
+class ChannelUrlFixer:
+    """fix not matching channel names in reindex"""
+
+    def __init__(self, youtube_id, config):
+        self.youtube_id = youtube_id
+        self.config = config
+        self.video = False
+
+    def run(self):
+        """check and run if needed"""
+        print(f"{self.youtube_id}: failed to build channel path, try to fix.")
+        video_path_is, video_folder_is = self.get_as_is()
+        if not os.path.exists(video_path_is):
+            print(f"giving up reindex, video in video: {self.video.json_data}")
+            raise ValueError
+
+        _, video_folder_should = self.get_as_should()
+
+        if video_folder_is != video_folder_should:
+            self.process(video_path_is)
+        else:
+            print(f"{self.youtube_id}: skip channel url fixer")
+
+    def get_as_is(self):
+        """get video object as is"""
+        self.video = YoutubeVideo(self.youtube_id)
+        self.video.get_from_es()
+        video_path_is = os.path.join(
+            self.config["application"]["videos"],
+            self.video.json_data["media_url"],
+        )
+        video_folder_is = os.path.split(video_path_is)[0]
+
+        return video_path_is, video_folder_is
+
+    def get_as_should(self):
+        """add fresh metadata from remote"""
+        self.video.get_from_youtube()
+        self.video.add_file_path()
+
+        video_path_should = os.path.join(
+            self.config["application"]["videos"],
+            self.video.json_data["media_url"],
+        )
+        video_folder_should = os.path.split(video_path_should)[0]
+        return video_path_should, video_folder_should
+
+    def process(self, video_path_is):
+        """fix filepath"""
+        print(f"{self.youtube_id}: fixing channel rename.")
+        cache_dir = self.config["application"]["cache_dir"]
+        new_file_path = os.path.join(
+            cache_dir, "download", self.youtube_id + ".mp4"
+        )
+        shutil.move(video_path_is, new_file_path)
+        VideoDownloader().move_to_archive(self.video.json_data)
+        self.video.update_media_url()

tubearchivist/home/src/index/video.py

@@ -10,7 +10,6 @@ from datetime import datetime
 
 import requests
 from django.conf import settings
-from home.src.download.thumbnails import ThumbManager
 from home.src.es.connect import ElasticWrap
 from home.src.index import channel as ta_channel
 from home.src.index.generic import YouTubeItem
@@ -152,6 +151,26 @@ class YoutubeSubtitle:
         """send subtitle to es for indexing"""
         _, _ = ElasticWrap("_bulk").post(data=query_str, ndjson=True)
 
+    def delete(self):
+        """delete subtitles from index and filesystem"""
+        youtube_id = self.video.youtube_id
+        # delete files
+        videos_base = self.video.config["application"]["videos"]
+        if not self.video.json_data.get("subtitles"):
+            return
+
+        files = [i["media_url"] for i in self.video.json_data["subtitles"]]
+        for file_name in files:
+            file_path = os.path.join(videos_base, file_name)
+            try:
+                os.remove(file_path)
+            except FileNotFoundError:
+                print(f"{youtube_id}: {file_path} failed to delete")
+        # delete from index
+        path = "ta_subtitle/_delete_by_query?refresh=true"
+        data = {"query": {"term": {"youtube_id": {"value": youtube_id}}}}
+        _, _ = ElasticWrap(path).post(data=data)
+
 
 class SubtitleParser:
     """parse subtitle str from youtube"""
@@ -196,6 +215,11 @@ class SubtitleParser:
 
             if flatten:
                 # fix overlapping retiming issue
+                if "dDurationMs" not in flatten[-1]:
+                    # some events won't have a duration
+                    print(f"failed to parse event without duration: {event}")
+                    continue
+
                 last_end = flatten[-1]["tStartMs"] + flatten[-1]["dDurationMs"]
                 if event["tStartMs"] < last_end:
                     joined = flatten[-1]["segs"][0]["utf8"] + "\n" + text
@@ -437,7 +461,8 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
         upload_date_time = datetime.strptime(upload_date, "%Y%m%d")
         published = upload_date_time.strftime("%Y-%m-%d")
         last_refresh = int(datetime.now().strftime("%s"))
-        base64_blur = ThumbManager().get_base64_blur(self.youtube_id)
+        # base64_blur = ThumbManager().get_base64_blur(self.youtube_id)
+        base64_blur = False
         # build json_data basics
         self.json_data = {
             "title": self.youtube_meta["title"],
@@ -537,22 +562,21 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
         """delete video file, meta data"""
         self.get_from_es()
         video_base = self.app_conf["videos"]
-        to_del = [self.json_data.get("media_url")]
+        media_url = self.json_data.get("media_url")
-
+        file_path = os.path.join(video_base, media_url)
-        all_subtitles = self.json_data.get("subtitles")
+        try:
-        if all_subtitles:
+            os.remove(file_path)
-            to_del = to_del + [i.get("media_url") for i in all_subtitles]
+        except FileNotFoundError:
-
+            print(f"{self.youtube_id}: failed {media_url}, continue.")
-        for media_url in to_del:
-            file_path = os.path.join(video_base, media_url)
-            try:
-                os.remove(file_path)
-            except FileNotFoundError:
-                print(f"{self.youtube_id}: failed {media_url}, continue.")
 
         self.del_in_es()
         self.delete_subtitles()
 
+    def delete_subtitles(self):
+        """delete indexed subtitles"""
+        print(f"{self.youtube_id}: delete subtitles")
+        YoutubeSubtitle(self).delete()
+
     def _get_ryd_stats(self):
         """get optional stats from returnyoutubedislikeapi.com"""
         try:
@@ -587,10 +611,10 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
             self.json_data["subtitles"] = subtitles
             handler.download_subtitles(relevant_subtitles=subtitles)
 
-    def delete_subtitles(self):
+    def update_media_url(self):
-        """delete indexed subtitles"""
+        """update only media_url in es for reindex channel rename"""
-        path = "ta_subtitle/_delete_by_query?refresh=true"
+        data = {"doc": {"media_url": self.json_data["media_url"]}}
-        data = {"query": {"term": {"youtube_id": {"value": self.youtube_id}}}}
+        path = f"{self.index_name}/_update/{self.youtube_id}"
         _, _ = ElasticWrap(path).post(data=data)
 
 

tubearchivist/home/src/ta/config.py

@@ -83,6 +83,7 @@ class AppConfig:
 
     def update_config(self, form_post):
         """update config values from settings form"""
+        updated = []
         for key, value in form_post.items():
             if not value and not isinstance(value, int):
                 continue
@@ -96,8 +97,10 @@ class AppConfig:
 
             config_dict, config_value = key.split("_", maxsplit=1)
             self.config[config_dict][config_value] = to_write
+            updated.append((config_value, to_write))
 
         RedisArchivist().set_message("config", self.config, expire=False)
+        return updated
 
     @staticmethod
     def set_user_config(form_post, user_id):

tubearchivist/home/src/ta/helper.py

@@ -173,7 +173,7 @@ class UrlListParser:
             id_type = "video"
         elif str_len == 24:
             id_type = "channel"
-        elif str_len in [34, 18]:
+        elif str_len in [34, 18] or id_str in ["LL", "WL"]:
             id_type = "playlist"
         else:
             # unable to parse

tubearchivist/home/templates/home/settings.html

@@ -114,6 +114,19 @@
                 {{ app_form.downloads_subtitle_index }}
             </div>
         </div>
+        <div class="settings-group">
+            <h2 id="format">Cookie</h2>
+            <div class="settings-item">
+                <p>Import YouTube cookie: <span class="settings-current">{{ config.downloads.cookie_import }}</span><br></p>
+                <i>Place your cookie file named <span class="settings-current">cookies.google.txt</span> in <span class="settings-current">cache/import</span> before enabling. Instructions in the <a href="https://github.com/tubearchivist/tubearchivist/wiki/Settings" target="_blank">Wiki.</a></i><br>
+                {{ app_form.downloads_cookie_import }}<br>
+                {% if config.downloads.cookie_import %}
+                    <div id="cookieMessage">
+                        <button onclick="handleCookieValidate()" type="button" id="cookieButton">Validate Cookie File</button>
+                    </div>
+                {% endif %}
+            </div>
+        </div>
         <div class="settings-group">
             <h2 id="integrations">Integrations</h2>
             <div class="settings-item">

tubearchivist/home/views.py

@@ -14,6 +14,7 @@ from django.contrib.auth.forms import AuthenticationForm
 from django.http import JsonResponse
 from django.shortcuts import redirect, render
 from django.views import View
+from home.src.download.yt_cookie import CookieHandler
 from home.src.es.index_setup import get_available_backups
 from home.src.frontend.api_calls import PostData
 from home.src.frontend.forms import (
@@ -791,8 +792,7 @@ class SettingsView(View):
         token = Token.objects.get_or_create(user=request.user)[0]
         return token
 
-    @staticmethod
+    def post(self, request):
-    def post(request):
         """handle form post to update settings"""
         user_form = UserSettingsForm(request.POST)
         if user_form.is_valid():
@@ -805,7 +805,8 @@ class SettingsView(View):
             app_form_post = app_form.cleaned_data
             if app_form_post:
                 print(app_form_post)
-                AppConfig().update_config(app_form_post)
+                updated = AppConfig().update_config(app_form_post)
+                self.post_process_updated(updated)
 
         scheduler_form = SchedulerSettingsForm(request.POST)
         if scheduler_form.is_valid():
@@ -817,6 +818,19 @@ class SettingsView(View):
         sleep(1)
         return redirect("settings", permanent=True)
 
+    @staticmethod
+    def post_process_updated(updated):
+        """apply changes for config"""
+        if not updated:
+            return
+
+        for config_value, updated_value in updated:
+            if config_value == "cookie_import":
+                if updated_value:
+                    CookieHandler().import_cookie()
+                else:
+                    CookieHandler().revoke()
+
 
 def progress(request):
     # pylint: disable=unused-argument

tubearchivist/static/script.js

@@ -350,7 +350,10 @@ function createPlayer(button) {
                 `;
             }
         }
+    } else {
+        sponsorBlock = null;
     }
+
     var videoProgress = getVideoProgress(videoId).position;
     var videoName = videoData.data.title;
 
@@ -696,6 +699,24 @@ function postSponsorSegmentVote(videoId, uuid, vote) {
     apiRequest(apiEndpoint, "POST", data);
 }
 
+function handleCookieValidate() {
+    document.getElementById("cookieButton").remove();
+    var cookieMessageElement = document.getElementById("cookieMessage");
+    cookieMessageElement.innerHTML = `<span>Processing.</span>`;
+    response = postCookieValidate();
+    if (response.cookie_validated == true) {
+        cookieMessageElement.innerHTML = `<span>The cookie file is valid.</span>`;
+    } else {
+        cookieMessageElement.innerHTML = `<span class="danger-zone">Warning, the cookie file is invalid.</span>`;
+    }
+}
+
+// Check youtube cookie settings
+function postCookieValidate() {
+    var apiEndpoint = "/api/cookie/";
+    return apiRequest(apiEndpoint, "POST");
+}
+
 // Makes api requests when passed an endpoint and method ("GET", "POST", "DELETE")
 function apiRequest(apiEndpoint, method, data) {
     const xhttp = new XMLHttpRequest();