bump TA_VERSION

Changed:
- Changed more robust channel title building
- Ensure 100 progress message is sent
- Remove mig_path at startup
- Fix comment extraction fail due to redirect
- Fix duplicate notification message threads

.github/ISSUE_TEMPLATE/INSTALLATION-HELP.yml

@@ -38,6 +38,6 @@ body:
     attributes:
       label: Relevant log output
       description: Please copy and paste any relevant Docker logs. This will be automatically formatted into code, so no need for backticks.
-      render: shell
+      render: Shell
     validations:
       required: true

CONTRIBUTING.md

@@ -37,12 +37,12 @@ Please keep in mind:
 - A bug that can't be reproduced, is difficult or sometimes even impossible to fix. Provide very clear steps *how to reproduce*.
 
 ### Feature Request
-This project needs your help to grow further. There is no shortage of ideas, see the open [issues on GH](https://github.com/tubearchivist/tubearchivist/issues?q=is%3Aopen+is%3Aissue+label%3Aenhancement) and the [roadmap](https://github.com/tubearchivist/tubearchivist#roadmap), what this project lacks is contributors to implement these ideas.
+This project needs your help to grow further. There is no shortage of ideas, see the open [issues on GH](https://github.com/tubearchivist/tubearchivist/issues?q=is%3Aopen+is%3Aissue+label%3Aenhancement) and the [roadmap](https://github.com/tubearchivist/tubearchivist#roadmap), what this project lacks is contributors interested in helping with overall improvements of the application. Focus is *not* on adding new features, but improving existing ones.
 
-Existing ideas are easily *multiple years* worth of development effort, at least at current speed. Best and fastest way to implement your feature is to do it yourself, that's why this project is open source after all. This project is *very* selective with accepting new feature requests at this point.  
+Existing ideas are easily *multiple years* worth of development effort, at least at current speed. This project is *very* selective with accepting new feature requests at this point.  
 
 Good feature requests usually fall into one or more of these categories:
-- You want to work on your own idea within the next few days or weeks.
+- You want to work on your own small scoped idea within the next few days or weeks.
 - Your idea is beneficial for a wide range of users, not just for you.
 - Your idea extends the current project by building on and improving existing functionality.
 - Your idea is quick and easy to implement, for an experienced as well as for a first time contributor.
@@ -66,7 +66,9 @@ IMPORTANT: When receiving help, contribute back to the community by improving th
 
 ## How to make a Pull Request
 
-Thank you for contributing and helping improve this project. This is a quick checklist to help streamline the process:
+Thank you for contributing and helping improve this project. Focus for the foreseeable future is on improving and building on existing functionality, *not* on adding and expanding the application.
+
+This is a quick checklist to help streamline the process:
 
 - For **code changes**, make your PR against the [testing branch](https://github.com/tubearchivist/tubearchivist/tree/testing). That's where all active development happens. This simplifies the later merging into *master*, minimizes any conflicts and usually allows for easy and convenient *fast-forward* merging.
 - For **documentation changes**, make your PR directly against the *master* branch.

README.md

@@ -188,9 +188,17 @@ Implemented:
 - [X] Scan your file system to index already downloaded videos [2021-09-14]
 
 ## User Scripts
-This is a list of useful user scripts, generously created from folks like you to extend this project and its functionality. This is your time to shine, [read this](https://github.com/tubearchivist/tubearchivist/blob/master/CONTRIBUTING.md#user-scripts) then open a PR to add your script here.
+This is a list of useful user scripts, generously created from folks like you to extend this project and its functionality. Make sure to check the respective repository links for detailed license information.  
 
-- Example 1
+This is your time to shine, [read this](https://github.com/tubearchivist/tubearchivist/blob/master/CONTRIBUTING.md#user-scripts) then open a PR to add your script here.
+
+- [danieljue/ta_dl_page_script](https://github.com/danieljue/ta_dl_page_script): Helper browser script to prioritize a channels' videos in download queue.
+- [dot-mike/ta-scripts](https://github.com/dot-mike/ta-scripts): A collection of personal scripts for managing TubeArchivist.
+- [DarkFighterLuke/ta_base_url_nginx](https://gist.github.com/DarkFighterLuke/4561b6bfbf83720493dc59171c58ac36): Set base URL with Nginx when you can't use subdomains.
+- [lamusmaser/ta_migration_helper](https://github.com/lamusmaser/ta_migration_helper): Advanced helper script for migration issues to TubeArchivist v0.4.4 or later.
+- [lamusmaser/create_info_json](https://gist.github.com/lamusmaser/837fb58f73ea0cad784a33497932e0dd): Script to generate `.info.json` files using `ffmpeg` collecting information from downloaded videos.
+- [lamusmaser/ta_fix_for_video_redirection](https://github.com/lamusmaser/ta_fix_for_video_redirection): Script to fix videos that were incorrectly indexed by YouTube's "Video is Unavailable" response. 
+- [RoninTech/ta-helper](https://github.com/RoninTech/ta-helper): Helper script to provide a symlink association to reference TubeArchivist videos with their original titles.
 
 ## Donate
 The best donation to **Tube Archivist** is your time, take a look at the [contribution page](CONTRIBUTING.md) to get started.  

docker-compose.yml

@@ -40,7 +40,7 @@ services:
     depends_on:
       - archivist-es
   archivist-es:
-    image: bbilly1/tubearchivist-es         # only for amd64, or use official es 8.11.0
+    image: bbilly1/tubearchivist-es         # only for amd64, or use official es 8.13.2
     container_name: archivist-es
     restart: unless-stopped
     environment:

docker_assets/run.sh

@@ -14,11 +14,10 @@ fi
 python manage.py ta_envcheck
 python manage.py ta_connection
 python manage.py ta_startup
-python manage.py ta_migpath
 
 # start all tasks
 nginx &
-celery -A home.tasks worker --loglevel=INFO &
+celery -A home.tasks worker --loglevel=INFO --max-tasks-per-child 10 &
 celery -A home beat --loglevel=INFO \
     -s "${BEAT_SCHEDULE_PATH:-${cachedir}/celerybeat-schedule}" &
 uwsgi --ini uwsgi.ini

tubearchivist/api/src/aggs.py

@@ -24,62 +24,159 @@ class AggBase:
         raise NotImplementedError
 
 
-class Primary(AggBase):
-    """primary aggregation for total documents indexed"""
+class Video(AggBase):
+    """get video stats"""
 
-    name = "primary"
-    path = "ta_video,ta_channel,ta_playlist,ta_subtitle,ta_download/_search"
+    name = "video_stats"
+    path = "ta_video/_search"
     data = {
         "size": 0,
         "aggs": {
             "video_type": {
-                "filter": {"exists": {"field": "active"}},
-                "aggs": {"filtered": {"terms": {"field": "vid_type"}}},
+                "terms": {"field": "vid_type"},
+                "aggs": {
+                    "media_size": {"sum": {"field": "media_size"}},
+                    "duration": {"sum": {"field": "player.duration"}},
+                },
             },
-            "channel_total": {"value_count": {"field": "channel_active"}},
-            "channel_sub": {"terms": {"field": "channel_subscribed"}},
-            "playlist_total": {"value_count": {"field": "playlist_active"}},
-            "playlist_sub": {"terms": {"field": "playlist_subscribed"}},
-            "download": {"terms": {"field": "status"}},
+            "video_active": {
+                "terms": {"field": "active"},
+                "aggs": {
+                    "media_size": {"sum": {"field": "media_size"}},
+                    "duration": {"sum": {"field": "player.duration"}},
+                },
+            },
+            "video_media_size": {"sum": {"field": "media_size"}},
+            "video_count": {"value_count": {"field": "youtube_id"}},
+            "duration": {"sum": {"field": "player.duration"}},
         },
     }
 
     def process(self):
-        """make the call"""
+        """process aggregation"""
         aggregations = self.get()
 
-        videos = {"total": aggregations["video_type"].get("doc_count")}
-        videos.update(
-            {
-                i.get("key"): i.get("doc_count")
-                for i in aggregations["video_type"]["filtered"]["buckets"]
-            }
-        )
-        channels = {"total": aggregations["channel_total"].get("value")}
-        channels.update(
-            {
-                "sub_" + i.get("key_as_string"): i.get("doc_count")
-                for i in aggregations["channel_sub"]["buckets"]
-            }
-        )
-        playlists = {"total": aggregations["playlist_total"].get("value")}
-        playlists.update(
-            {
-                "sub_" + i.get("key_as_string"): i.get("doc_count")
-                for i in aggregations["playlist_sub"]["buckets"]
-            }
-        )
-        downloads = {
-            i.get("key"): i.get("doc_count")
-            for i in aggregations["download"]["buckets"]
+        duration = int(aggregations["duration"]["value"])
+        response = {
+            "doc_count": aggregations["video_count"]["value"],
+            "media_size": int(aggregations["video_media_size"]["value"]),
+            "duration": duration,
+            "duration_str": get_duration_str(duration),
         }
+        for bucket in aggregations["video_type"]["buckets"]:
+            duration = int(bucket["duration"].get("value"))
+            response.update(
+                {
+                    f"type_{bucket['key']}": {
+                        "doc_count": bucket.get("doc_count"),
+                        "media_size": int(bucket["media_size"].get("value")),
+                        "duration": duration,
+                        "duration_str": get_duration_str(duration),
+                    }
+                }
+            )
+
+        for bucket in aggregations["video_active"]["buckets"]:
+            duration = int(bucket["duration"].get("value"))
+            response.update(
+                {
+                    f"active_{bucket['key_as_string']}": {
+                        "doc_count": bucket.get("doc_count"),
+                        "media_size": int(bucket["media_size"].get("value")),
+                        "duration": duration,
+                        "duration_str": get_duration_str(duration),
+                    }
+                }
+            )
+
+        return response
+
+
+class Channel(AggBase):
+    """get channel stats"""
+
+    name = "channel_stats"
+    path = "ta_channel/_search"
+    data = {
+        "size": 0,
+        "aggs": {
+            "channel_count": {"value_count": {"field": "channel_id"}},
+            "channel_active": {"terms": {"field": "channel_active"}},
+            "channel_subscribed": {"terms": {"field": "channel_subscribed"}},
+        },
+    }
+
+    def process(self):
+        """process aggregation"""
+        aggregations = self.get()
 
         response = {
-            "videos": videos,
-            "channels": channels,
-            "playlists": playlists,
-            "downloads": downloads,
+            "doc_count": aggregations["channel_count"].get("value"),
         }
+        for bucket in aggregations["channel_active"]["buckets"]:
+            key = f"active_{bucket['key_as_string']}"
+            response.update({key: bucket.get("doc_count")})
+        for bucket in aggregations["channel_subscribed"]["buckets"]:
+            key = f"subscribed_{bucket['key_as_string']}"
+            response.update({key: bucket.get("doc_count")})
+
+        return response
+
+
+class Playlist(AggBase):
+    """get playlist stats"""
+
+    name = "playlist_stats"
+    path = "ta_playlist/_search"
+    data = {
+        "size": 0,
+        "aggs": {
+            "playlist_count": {"value_count": {"field": "playlist_id"}},
+            "playlist_active": {"terms": {"field": "playlist_active"}},
+            "playlist_subscribed": {"terms": {"field": "playlist_subscribed"}},
+        },
+    }
+
+    def process(self):
+        """process aggregation"""
+        aggregations = self.get()
+        response = {"doc_count": aggregations["playlist_count"].get("value")}
+        for bucket in aggregations["playlist_active"]["buckets"]:
+            key = f"active_{bucket['key_as_string']}"
+            response.update({key: bucket.get("doc_count")})
+        for bucket in aggregations["playlist_subscribed"]["buckets"]:
+            key = f"subscribed_{bucket['key_as_string']}"
+            response.update({key: bucket.get("doc_count")})
+
+        return response
+
+
+class Download(AggBase):
+    """get downloads queue stats"""
+
+    name = "download_queue_stats"
+    path = "ta_download/_search"
+    data = {
+        "size": 0,
+        "aggs": {
+            "status": {"terms": {"field": "status"}},
+            "video_type": {
+                "filter": {"term": {"status": "pending"}},
+                "aggs": {"type_pending": {"terms": {"field": "vid_type"}}},
+            },
+        },
+    }
+
+    def process(self):
+        """process aggregation"""
+        aggregations = self.get()
+        response = {}
+        for bucket in aggregations["status"]["buckets"]:
+            response.update({bucket["key"]: bucket.get("doc_count")})
+
+        for bucket in aggregations["video_type"]["type_pending"]["buckets"]:
+            key = f"pending_{bucket['key']}"
+            response.update({key: bucket.get("doc_count")})
 
         return response
 

tubearchivist/api/urls.py

@@ -152,9 +152,24 @@ urlpatterns = [
         name="api-notification",
     ),
     path(
-        "stats/primary/",
-        views.StatPrimaryView.as_view(),
-        name="api-stats-primary",
+        "stats/video/",
+        views.StatVideoView.as_view(),
+        name="api-stats-video",
+    ),
+    path(
+        "stats/channel/",
+        views.StatChannelView.as_view(),
+        name="api-stats-channel",
+    ),
+    path(
+        "stats/playlist/",
+        views.StatPlaylistView.as_view(),
+        name="api-stats-playlist",
+    ),
+    path(
+        "stats/download/",
+        views.StatDownloadView.as_view(),
+        name="api-stats-download",
     ),
     path(
         "stats/watch/",

tubearchivist/api/views.py

@@ -1,6 +1,14 @@
 """all API views"""
 
-from api.src.aggs import BiggestChannel, DownloadHist, Primary, WatchProgress
+from api.src.aggs import (
+    BiggestChannel,
+    Channel,
+    Download,
+    DownloadHist,
+    Playlist,
+    Video,
+    WatchProgress,
+)
 from api.src.search_processor import SearchProcess
 from home.src.download.queue import PendingInteract
 from home.src.download.subscriptions import (
@@ -32,7 +40,7 @@ from home.tasks import (
     run_restore_backup,
     subscribe_to,
 )
-from rest_framework import permissions
+from rest_framework import permissions, status
 from rest_framework.authentication import (
     SessionAuthentication,
     TokenAuthentication,
@@ -454,12 +462,26 @@ class PlaylistApiListView(ApiBaseView):
 
     search_base = "ta_playlist/_search/"
     permission_classes = [AdminWriteOnly]
+    valid_playlist_type = ["regular", "custom"]
 
     def get(self, request):
         """handle get request"""
-        self.data.update(
-            {"sort": [{"playlist_name.keyword": {"order": "asc"}}]}
-        )
+        playlist_type = request.GET.get("playlist_type", None)
+        query = {"sort": [{"playlist_name.keyword": {"order": "asc"}}]}
+        if playlist_type is not None:
+            if playlist_type not in self.valid_playlist_type:
+                message = f"invalid playlist_type {playlist_type}"
+                return Response({"message": message}, status=400)
+
+            query.update(
+                {
+                    "query": {
+                        "term": {"playlist_type": {"value": playlist_type}}
+                    },
+                }
+            )
+
+        self.data.update(query)
         self.get_document_list(request)
         return Response(self.response)
 
@@ -503,6 +525,7 @@ class PlaylistApiView(ApiBaseView):
 
     search_base = "ta_playlist/_doc/"
     permission_classes = [AdminWriteOnly]
+    valid_custom_actions = ["create", "remove", "up", "down", "top", "bottom"]
 
     def get(self, request, playlist_id):
         # pylint: disable=unused-argument
@@ -510,6 +533,27 @@ class PlaylistApiView(ApiBaseView):
         self.get_document(playlist_id)
         return Response(self.response, status=self.status_code)
 
+    def post(self, request, playlist_id):
+        """post to custom playlist to add a video to list"""
+        playlist = YoutubePlaylist(playlist_id)
+        if not playlist.is_custom_playlist():
+            message = f"playlist with ID {playlist_id} is not custom"
+            return Response({"message": message}, status=400)
+
+        action = request.data.get("action")
+        if action not in self.valid_custom_actions:
+            message = f"invalid action: {action}"
+            return Response({"message": message}, status=400)
+
+        video_id = request.data.get("video_id")
+        if action == "create":
+            playlist.add_video_to_playlist(video_id)
+        else:
+            hide = UserConfig(request.user.id).get_value("hide_watched")
+            playlist.move_video(video_id, action, hide_watched=hide)
+
+        return Response({"success": True}, status=status.HTTP_201_CREATED)
+
     def delete(self, request, playlist_id):
         """delete playlist"""
         print(f"{playlist_id}: delete playlist")
@@ -1141,16 +1185,52 @@ class NotificationView(ApiBaseView):
         return Response(RedisArchivist().list_items(query))
 
 
-class StatPrimaryView(ApiBaseView):
-    """resolves to /api/stats/primary/
-    GET: return document count
+class StatVideoView(ApiBaseView):
+    """resolves to /api/stats/video/
+    GET: return video stats
     """
 
     def get(self, request):
         """get stats"""
         # pylint: disable=unused-argument
 
-        return Response(Primary().process())
+        return Response(Video().process())
+
+
+class StatChannelView(ApiBaseView):
+    """resolves to /api/stats/channel/
+    GET: return channel stats
+    """
+
+    def get(self, request):
+        """get stats"""
+        # pylint: disable=unused-argument
+
+        return Response(Channel().process())
+
+
+class StatPlaylistView(ApiBaseView):
+    """resolves to /api/stats/playlist/
+    GET: return playlist stats
+    """
+
+    def get(self, request):
+        """get stats"""
+        # pylint: disable=unused-argument
+
+        return Response(Playlist().process())
+
+
+class StatDownloadView(ApiBaseView):
+    """resolves to /api/stats/download/
+    GET: return download stats
+    """
+
+    def get(self, request):
+        """get stats"""
+        # pylint: disable=unused-argument
+
+        return Response(Download().process())
 
 
 class StatWatchProgress(ApiBaseView):

tubearchivist/config/management/commands/ta_migpath.py

@@ -1,4 +1,8 @@
-"""filepath migration from v0.3.6 to v0.3.7"""
+"""
+filepath migration from v0.3.6 to v0.3.7
+not getting called at startup any more, to run manually if needed:
+python manage.py ta_migpath
+"""
 
 import json
 import os

tubearchivist/config/management/commands/ta_startup.py

@@ -8,6 +8,7 @@ import os
 from time import sleep
 
 from django.core.management.base import BaseCommand, CommandError
+from home.src.es.connect import ElasticWrap
 from home.src.es.index_setup import ElasitIndexWrap
 from home.src.es.snapshot import ElasticSnapshot
 from home.src.ta.config import AppConfig, ReleaseVersion
@@ -36,13 +37,15 @@ class Command(BaseCommand):
         self.stdout.write(TOPIC)
         self._sync_redis_state()
         self._make_folders()
-        self._release_locks()
+        self._clear_redis_keys()
         self._clear_tasks()
         self._clear_dl_cache()
+        self._mig_clear_failed_versioncheck()
         self._version_check()
         self._mig_index_setup()
         self._mig_snapshot_check()
         self._mig_move_users_to_es()
+        self._mig_custom_playlist()
 
     def _sync_redis_state(self):
         """make sure redis gets new config.json values"""
@@ -73,10 +76,10 @@ class Command(BaseCommand):
 
         self.stdout.write(self.style.SUCCESS("    ✓ expected folders created"))
 
-    def _release_locks(self):
-        """make sure there are no leftover locks set in redis"""
-        self.stdout.write("[3] clear leftover locks in redis")
-        all_locks = [
+    def _clear_redis_keys(self):
+        """make sure there are no leftover locks or keys set in redis"""
+        self.stdout.write("[3] clear leftover keys in redis")
+        all_keys = [
             "dl_queue_id",
             "dl_queue",
             "downloading",
@@ -85,19 +88,22 @@ class Command(BaseCommand):
             "rescan",
             "run_backup",
             "startup_check",
+            "reindex:ta_video",
+            "reindex:ta_channel",
+            "reindex:ta_playlist",
         ]
 
         redis_con = RedisArchivist()
         has_changed = False
-        for lock in all_locks:
-            if redis_con.del_message(lock):
+        for key in all_keys:
+            if redis_con.del_message(key):
                 self.stdout.write(
-                    self.style.SUCCESS(f"    ✓ cleared lock {lock}")
+                    self.style.SUCCESS(f"    ✓ cleared key {key}")
                 )
                 has_changed = True
 
         if not has_changed:
-            self.stdout.write(self.style.SUCCESS("    no locks found"))
+            self.stdout.write(self.style.SUCCESS("    no keys found"))
 
     def _clear_tasks(self):
         """clear tasks and messages"""
@@ -145,6 +151,10 @@ class Command(BaseCommand):
         self.stdout.write("[MIGRATION] setup snapshots")
         ElasticSnapshot().setup()
 
+    def _mig_clear_failed_versioncheck(self):
+        """hotfix for v0.4.5, clearing faulty versioncheck"""
+        ReleaseVersion().clear_fail()
+
     def _mig_move_users_to_es(self):  # noqa: C901
         """migration: update from 0.4.1 to 0.4.2 move user config to ES"""
         self.stdout.write("[MIGRATION] move user configuration to ES")
@@ -234,3 +244,36 @@ class Command(BaseCommand):
                     "    ✓ Settings for all users migrated to ES"
                 )
             )
+
+    def _mig_custom_playlist(self):
+        """migration for custom playlist"""
+        self.stdout.write("[MIGRATION] custom playlist")
+        data = {
+            "query": {
+                "bool": {"must_not": [{"exists": {"field": "playlist_type"}}]}
+            },
+            "script": {"source": "ctx._source['playlist_type'] = 'regular'"},
+        }
+        path = "ta_playlist/_update_by_query"
+        response, status_code = ElasticWrap(path).post(data=data)
+        if status_code == 200:
+            updated = response.get("updated", 0)
+            if updated:
+                self.stdout.write(
+                    self.style.SUCCESS(
+                        f"    ✓ {updated} playlist_type updated in ta_playlist"
+                    )
+                )
+            else:
+                self.stdout.write(
+                    self.style.SUCCESS(
+                        "    no playlist_type needed updating in ta_playlist"
+                    )
+                )
+            return
+
+        message = "    🗙 ta_playlist playlist_type update failed"
+        self.stdout.write(self.style.ERROR(message))
+        self.stdout.write(response)
+        sleep(60)
+        raise CommandError(message)

tubearchivist/config/settings.py

@@ -269,4 +269,4 @@ CORS_ALLOW_HEADERS = list(default_headers) + [
 
 # TA application settings
 TA_UPSTREAM = "https://github.com/tubearchivist/tubearchivist"
-TA_VERSION = "v0.4.3"
+TA_VERSION = "v0.4.7"

tubearchivist/config/urls.py

@@ -13,6 +13,7 @@ Including another URLconf
     1. Import the include() function: from django.urls import include, path
     2. Add a URL to urlpatterns:  path('blog/', include('blog.urls'))
 """
+
 from django.contrib import admin
 from django.urls import include, path
 

tubearchivist/home/models.py

@@ -1,4 +1,5 @@
 """custom models"""
+
 from django.contrib.auth.models import (
     AbstractBaseUser,
     BaseUserManager,

tubearchivist/home/src/download/thumbnails.py

@@ -75,7 +75,7 @@ class ThumbManagerBase:
                 app_root, "static/img/default-video-thumb.jpg"
             ),
             "playlist": os.path.join(
-                app_root, "static/img/default-video-thumb.jpg"
+                app_root, "static/img/default-playlist-thumb.jpg"
             ),
             "icon": os.path.join(
                 app_root, "static/img/default-channel-icon.jpg"
@@ -202,7 +202,18 @@ class ThumbManager(ThumbManagerBase):
         if skip_existing and os.path.exists(thumb_path):
             return
 
-        img_raw = self.download_raw(url)
+        img_raw = (
+            self.download_raw(url)
+            if not isinstance(url, str) or url.startswith("http")
+            else Image.open(os.path.join(self.CACHE_DIR, url))
+        )
+        width, height = img_raw.size
+
+        if not width / height == 16 / 9:
+            new_height = width / 16 * 9
+            offset = (height - new_height) / 2
+            img_raw = img_raw.crop((0, offset, width, height - offset))
+        img_raw = img_raw.resize((336, 189))
         img_raw.convert("RGB").save(thumb_path)
 
     def delete_video_thumb(self):

tubearchivist/home/src/download/yt_dlp_base.py

@@ -62,8 +62,8 @@ class YtWrap:
         """make extract request"""
         try:
             response = yt_dlp.YoutubeDL(self.obs).extract_info(url)
-        except cookiejar.LoadError:
-            print("cookie file is invalid")
+        except cookiejar.LoadError as err:
+            print(f"cookie file is invalid: {err}")
             return False
         except yt_dlp.utils.ExtractorError as err:
             print(f"{url}: failed to extract with message: {err}, continue...")

tubearchivist/home/src/download/yt_dlp_handler.py

@@ -50,7 +50,7 @@ class DownloadPostProcess:
             return
 
         print(f"auto delete older than {autodelete_days} days")
-        now_lte = self.now - autodelete_days * 24 * 60 * 60
+        now_lte = str(self.now - autodelete_days * 24 * 60 * 60)
         data = {
             "query": {"range": {"player.watched_date": {"lte": now_lte}}},
             "sort": [{"player.watched_date": {"order": "asc"}}],
@@ -63,7 +63,7 @@ class DownloadPostProcess:
             if "autodelete_days" in value:
                 autodelete_days = value.get("autodelete_days")
                 print(f"{channel_id}: delete older than {autodelete_days}d")
-                now_lte = self.now - autodelete_days * 24 * 60 * 60
+                now_lte = str(self.now - autodelete_days * 24 * 60 * 60)
                 must_list = [
                     {"range": {"player.watched_date": {"lte": now_lte}}},
                     {"term": {"channel.channel_id": {"value": channel_id}}},
@@ -177,7 +177,7 @@ class VideoDownloader:
             if not success:
                 continue
 
-            self._notify(video_data, "Add video metadata to index")
+            self._notify(video_data, "Add video metadata to index", progress=1)
 
             vid_dict = index_new_video(
                 youtube_id,
@@ -197,14 +197,16 @@ class VideoDownloader:
 
         return self.videos
 
-    def _notify(self, video_data, message):
+    def _notify(self, video_data, message, progress=False):
         """send progress notification to task"""
         if not self.task:
             return
 
         typ = VideoTypeEnum(video_data["vid_type"]).value.rstrip("s").title()
         title = video_data.get("title")
-        self.task.send_progress([f"Processing {typ}: {title}", message])
+        self.task.send_progress(
+            [f"Processing {typ}: {title}", message], progress=progress
+        )
 
     def _get_next(self, auto_only):
         """get next item in queue"""

tubearchivist/home/src/es/connect.py

@@ -3,6 +3,7 @@ functionality:
 - wrapper around requests to call elastic search
 - reusable search_after to extract total index
 """
+
 # pylint: disable=missing-timeout
 
 import json

tubearchivist/home/src/es/index_mapping.json

@@ -74,7 +74,7 @@
                             "type": "boolean"
                         },
                         "integrate_sponsorblock": {
-                            "type" : "boolean"
+                            "type": "boolean"
                         }
                     }
                 }
@@ -168,7 +168,7 @@
                                     "type": "boolean"
                                 },
                                 "integrate_sponsorblock": {
-                                    "type" : "boolean"
+                                    "type": "boolean"
                                 }
                             }
                         }
@@ -236,19 +236,37 @@
                 "comment_count": {
                     "type": "long"
                 },
-                "stats" : {
-                    "properties" : {
-                        "average_rating" : {
-                            "type" : "float"
+                "stats": {
+                    "properties": {
+                        "average_rating": {
+                            "type": "float"
                         },
-                        "dislike_count" : {
-                            "type" : "long"
+                        "dislike_count": {
+                            "type": "long"
                         },
-                        "like_count" : {
-                            "type" : "long"
+                        "like_count": {
+                            "type": "long"
                         },
-                        "view_count" : {
-                            "type" : "long"
+                        "view_count": {
+                            "type": "long"
+                        }
+                    }
+                },
+                "player": {
+                    "properties": {
+                        "duration": {
+                            "type": "long"
+                        },
+                        "duration_str": {
+                            "type": "keyword",
+                            "index": false
+                        },
+                        "watched": {
+                            "type": "boolean"
+                        },
+                        "watched_date": {
+                            "type": "date",
+                            "format": "epoch_second"
                         }
                     }
                 },
@@ -314,28 +332,28 @@
                         "is_enabled": {
                             "type": "boolean"
                         },
-                        "segments" : {
-                            "properties" : {
-                                "UUID" : {
+                        "segments": {
+                            "properties": {
+                                "UUID": {
                                     "type": "keyword"
                                 },
-                                "actionType" : {
+                                "actionType": {
                                     "type": "keyword"
                                 },
-                                "category" : {
+                                "category": {
                                     "type": "keyword"
                                 },
-                                "locked" : {
-                                    "type" : "short"
+                                "locked": {
+                                    "type": "short"
                                 },
-                                "segment" : {
-                                    "type" : "float"
+                                "segment": {
+                                    "type": "float"
                                 },
-                                "videoDuration" : {
-                                    "type" : "float"
+                                "videoDuration": {
+                                    "type": "float"
                                 },
-                                "votes" : {
-                                    "type" : "long"
+                                "votes": {
+                                    "type": "long"
                                 }
                             }
                         }
@@ -459,6 +477,41 @@
                 "playlist_last_refresh": {
                     "type": "date",
                     "format": "epoch_second"
+                },
+                "playlist_entries": {
+                    "properties": {
+                        "downloaded": {
+                            "type": "boolean"
+                        },
+                        "idx": {
+                            "type": "long"
+                        },
+                        "title": {
+                            "type": "text",
+                            "analyzer": "english",
+                            "fields": {
+                                "keyword": {
+                                    "type": "keyword",
+                                    "ignore_above": 256,
+                                    "normalizer": "to_lower"
+                                }
+                            }
+                        },
+                        "uploader": {
+                            "type": "text",
+                            "analyzer": "english",
+                            "fields": {
+                                "keyword": {
+                                    "type": "keyword",
+                                    "ignore_above": 256,
+                                    "normalizer": "to_lower"
+                                }
+                            }
+                        },
+                        "youtube_id": {
+                            "type": "keyword"
+                        }
+                    }
                 }
             },
             "expected_set": {
@@ -516,7 +569,7 @@
                     "format": "epoch_second"
                 },
                 "subtitle_index": {
-                    "type" : "long"
+                    "type": "long"
                 },
                 "subtitle_lang": {
                     "type": "keyword"
@@ -525,7 +578,7 @@
                     "type": "keyword"
                 },
                 "subtitle_line": {
-                    "type" : "text",
+                    "type": "text",
                     "analyzer": "english"
                 }
             },
@@ -560,14 +613,14 @@
                             "type": "keyword"
                         },
                         "comment_text": {
-                            "type" : "text"
+                            "type": "text"
                         },
                         "comment_timestamp": {
                             "type": "date",
                             "format": "epoch_second"
                         },
                         "comment_time_text": {
-                            "type" : "text"
+                            "type": "text"
                         },
                         "comment_likecount": {
                             "type": "long"
@@ -613,4 +666,4 @@
             }
         }
     ]
-}
+}

tubearchivist/home/src/frontend/forms.py

@@ -105,8 +105,8 @@ class ApplicationSettingsForm(forms.Form):
 
     COOKIE_IMPORT_CHOICES = [
         ("", "-- change cookie settings"),
-        ("0", "disable cookie"),
-        ("1", "enable cookie"),
+        ("0", "remove cookie"),
+        ("1", "import cookie"),
     ]
 
     subscriptions_channel_size = forms.IntegerField(
@@ -265,6 +265,20 @@ class SubscribeToPlaylistForm(forms.Form):
     )
 
 
+class CreatePlaylistForm(forms.Form):
+    """text area form to create a single custom playlist"""
+
+    create = forms.CharField(
+        label="Or create custom playlist",
+        widget=forms.Textarea(
+            attrs={
+                "rows": 1,
+                "placeholder": "Input playlist name",
+            }
+        ),
+    )
+
+
 class ChannelOverwriteForm(forms.Form):
     """custom overwrites for channel settings"""
 

tubearchivist/home/src/frontend/searching.py

@@ -6,7 +6,6 @@ Functionality:
 - calculate pagination values
 """
 
-
 from api.src.search_processor import SearchProcess
 from home.src.es.connect import ElasticWrap
 

tubearchivist/home/src/index/channel.py

@@ -30,6 +30,10 @@ class YoutubeChannel(YouTubeItem):
         self.all_playlists = False
         self.task = task
 
+    def build_yt_url(self):
+        """overwrite base to use channel about page"""
+        return f"{self.yt_base}{self.youtube_id}/about"
+
     def build_json(self, upload=False, fallback=False):
         """get from es or from youtube"""
         self.get_from_es()
@@ -195,11 +199,21 @@ class YoutubeChannel(YouTubeItem):
         }
         _, _ = ElasticWrap("ta_comment/_delete_by_query").post(data)
 
+    def delete_es_subtitles(self):
+        """delete all subtitles from this channel"""
+        data = {
+            "query": {
+                "term": {"subtitle_channel_id": {"value": self.youtube_id}}
+            }
+        }
+        _, _ = ElasticWrap("ta_subtitle/_delete_by_query").post(data)
+
     def delete_playlists(self):
         """delete all indexed playlist from es"""
         all_playlists = self.get_indexed_playlists()
         for playlist in all_playlists:
             playlist_id = playlist["playlist_id"]
+            playlist = YoutubePlaylist(playlist_id)
             YoutubePlaylist(playlist_id).delete_metadata()
 
     def delete_channel(self):
@@ -225,6 +239,7 @@ class YoutubeChannel(YouTubeItem):
         print(f"{self.youtube_id}: delete indexed videos")
         self.delete_es_videos()
         self.delete_es_comments()
+        self.delete_es_subtitles()
         self.del_in_es()
 
     def index_channel_playlists(self):

tubearchivist/home/src/index/comments.py

@@ -68,6 +68,7 @@ class Comments:
                 "youtube": {
                     "max_comments": max_comments_list,
                     "comment_sort": [comment_sort],
+                    "player_client": ["ios", "web"],  # workaround yt-dlp #9554
                 }
             },
         }
@@ -77,7 +78,7 @@ class Comments:
     def get_yt_comments(self):
         """get comments from youtube"""
         yt_obs = self.build_yt_obs()
-        info_json = YtWrap(yt_obs).extract(self.youtube_id)
+        info_json = YtWrap(yt_obs, config=self.config).extract(self.youtube_id)
         if not info_json:
             return False, False
 
@@ -115,6 +116,9 @@ class Comments:
 
         time_text = time_text_datetime.strftime(format_string)
 
+        if not comment.get("author"):
+            comment["author"] = comment.get("author_id", "Unknown")
+
         cleaned_comment = {
             "comment_id": comment["id"],
             "comment_text": comment["text"].replace("\xa0", ""),
@@ -126,7 +130,7 @@ class Comments:
             "comment_author_id": comment["author_id"],
             "comment_author_thumbnail": comment["author_thumbnail"],
             "comment_author_is_uploader": comment.get(
-                "comment_author_is_uploader", False
+                "author_is_uploader", False
             ),
             "comment_parent": comment["parent"],
         }

tubearchivist/home/src/index/playlist.py

@@ -66,6 +66,7 @@ class YoutubePlaylist(YouTubeItem):
             "playlist_thumbnail": playlist_thumbnail,
             "playlist_description": self.youtube_meta["description"] or False,
             "playlist_last_refresh": int(datetime.now().timestamp()),
+            "playlist_type": "regular",
         }
 
     def get_entries(self, playlistend=False):
@@ -178,6 +179,7 @@ class YoutubePlaylist(YouTubeItem):
 
     def delete_metadata(self):
         """delete metadata for playlist"""
+        self.delete_videos_metadata()
         script = (
             "ctx._source.playlist.removeAll("
             + "Collections.singleton(params.playlist)) "
@@ -195,6 +197,30 @@ class YoutubePlaylist(YouTubeItem):
         _, _ = ElasticWrap("ta_video/_update_by_query").post(data)
         self.del_in_es()
 
+    def is_custom_playlist(self):
+        self.get_from_es()
+        return self.json_data["playlist_type"] == "custom"
+
+    def delete_videos_metadata(self, channel_id=None):
+        """delete video metadata for a specific channel"""
+        self.get_from_es()
+        playlist = self.json_data["playlist_entries"]
+        i = 0
+        while i < len(playlist):
+            video_id = playlist[i]["youtube_id"]
+            video = YoutubeVideo(video_id)
+            video.get_from_es()
+            if (
+                channel_id is None
+                or video.json_data["channel"]["channel_id"] == channel_id
+            ):
+                playlist.pop(i)
+                self.remove_playlist_from_video(video_id)
+                i -= 1
+            i += 1
+        self.set_playlist_thumbnail()
+        self.upload_to_es()
+
     def delete_videos_playlist(self):
         """delete playlist with all videos"""
         print(f"{self.youtube_id}: delete playlist")
@@ -208,3 +234,159 @@ class YoutubePlaylist(YouTubeItem):
             YoutubeVideo(youtube_id).delete_media_file()
 
         self.delete_metadata()
+
+    def create(self, name):
+        self.json_data = {
+            "playlist_id": self.youtube_id,
+            "playlist_active": False,
+            "playlist_name": name,
+            "playlist_last_refresh": int(datetime.now().timestamp()),
+            "playlist_entries": [],
+            "playlist_type": "custom",
+            "playlist_channel": None,
+            "playlist_channel_id": None,
+            "playlist_description": False,
+            "playlist_thumbnail": False,
+            "playlist_subscribed": False,
+        }
+        self.upload_to_es()
+        self.get_playlist_art()
+        return True
+
+    def add_video_to_playlist(self, video_id):
+        self.get_from_es()
+        video_metadata = self.get_video_metadata(video_id)
+        video_metadata["idx"] = len(self.json_data["playlist_entries"])
+
+        if not self.playlist_entries_contains(video_id):
+            self.json_data["playlist_entries"].append(video_metadata)
+            self.json_data["playlist_last_refresh"] = int(
+                datetime.now().timestamp()
+            )
+            self.set_playlist_thumbnail()
+            self.upload_to_es()
+            video = YoutubeVideo(video_id)
+            video.get_from_es()
+            if "playlist" not in video.json_data:
+                video.json_data["playlist"] = []
+            video.json_data["playlist"].append(self.youtube_id)
+            video.upload_to_es()
+        return True
+
+    def remove_playlist_from_video(self, video_id):
+        video = YoutubeVideo(video_id)
+        video.get_from_es()
+        if video.json_data is not None and "playlist" in video.json_data:
+            video.json_data["playlist"].remove(self.youtube_id)
+            video.upload_to_es()
+
+    def move_video(self, video_id, action, hide_watched=False):
+        self.get_from_es()
+        video_index = self.get_video_index(video_id)
+        playlist = self.json_data["playlist_entries"]
+        item = playlist[video_index]
+        playlist.pop(video_index)
+        if action == "remove":
+            self.remove_playlist_from_video(item["youtube_id"])
+        else:
+            if action == "up":
+                while True:
+                    video_index = max(0, video_index - 1)
+                    if (
+                        not hide_watched
+                        or video_index == 0
+                        or (
+                            not self.get_video_is_watched(
+                                playlist[video_index]["youtube_id"]
+                            )
+                        )
+                    ):
+                        break
+            elif action == "down":
+                while True:
+                    video_index = min(len(playlist), video_index + 1)
+                    if (
+                        not hide_watched
+                        or video_index == len(playlist)
+                        or (
+                            not self.get_video_is_watched(
+                                playlist[video_index - 1]["youtube_id"]
+                            )
+                        )
+                    ):
+                        break
+            elif action == "top":
+                video_index = 0
+            else:
+                video_index = len(playlist)
+            playlist.insert(video_index, item)
+        self.json_data["playlist_last_refresh"] = int(
+            datetime.now().timestamp()
+        )
+
+        for i, item in enumerate(playlist):
+            item["idx"] = i
+
+        self.set_playlist_thumbnail()
+        self.upload_to_es()
+
+        return True
+
+    def del_video(self, video_id):
+        playlist = self.json_data["playlist_entries"]
+
+        i = 0
+        while i < len(playlist):
+            if video_id == playlist[i]["youtube_id"]:
+                playlist.pop(i)
+                self.set_playlist_thumbnail()
+                i -= 1
+            i += 1
+
+    def get_video_index(self, video_id):
+        for i, child in enumerate(self.json_data["playlist_entries"]):
+            if child["youtube_id"] == video_id:
+                return i
+        return -1
+
+    def playlist_entries_contains(self, video_id):
+        return (
+            len(
+                list(
+                    filter(
+                        lambda x: x["youtube_id"] == video_id,
+                        self.json_data["playlist_entries"],
+                    )
+                )
+            )
+            > 0
+        )
+
+    def get_video_is_watched(self, video_id):
+        video = YoutubeVideo(video_id)
+        video.get_from_es()
+        return video.json_data["player"]["watched"]
+
+    def set_playlist_thumbnail(self):
+        playlist = self.json_data["playlist_entries"]
+        self.json_data["playlist_thumbnail"] = False
+
+        for video in playlist:
+            url = ThumbManager(video["youtube_id"]).vid_thumb_path()
+            if url is not None:
+                self.json_data["playlist_thumbnail"] = url
+                break
+        self.get_playlist_art()
+
+    def get_video_metadata(self, video_id):
+        video = YoutubeVideo(video_id)
+        video.get_from_es()
+        video_json_data = {
+            "youtube_id": video.json_data["youtube_id"],
+            "title": video.json_data["title"],
+            "uploader": video.json_data["channel"]["channel_name"],
+            "idx": 0,
+            "downloaded": "date_downloaded" in video.json_data
+            and video.json_data["date_downloaded"] > 0,
+        }
+        return video_json_data

tubearchivist/home/src/index/reindex.py

@@ -105,11 +105,13 @@ class ReindexPopulate(ReindexBase):
         """get total hits from index"""
         index_name = reindex_config["index_name"]
         active_key = reindex_config["active_key"]
-        path = f"{index_name}/_search?filter_path=hits.total"
-        data = {"query": {"match": {active_key: True}}}
-        response, _ = ElasticWrap(path).post(data=data)
-        total_hits = response["hits"]["total"]["value"]
-        return total_hits
+        data = {
+            "query": {"term": {active_key: {"value": True}}},
+            "_source": False,
+        }
+        total = IndexPaginate(index_name, data, keep_source=True).get_results()
+
+        return len(total)
 
     def _get_daily_should(self, total_hits):
         """calc how many should reindex daily"""
@@ -123,7 +125,7 @@ class ReindexPopulate(ReindexBase):
         """get outdated from index_name"""
         index_name = reindex_config["index_name"]
         refresh_key = reindex_config["refresh_key"]
-        now_lte = self.now - self.interval * 24 * 60 * 60
+        now_lte = str(self.now - self.interval * 24 * 60 * 60)
         must_list = [
             {"match": {reindex_config["active_key"]: True}},
             {"range": {refresh_key: {"lte": now_lte}}},
@@ -241,7 +243,7 @@ class Reindex(ReindexBase):
             return
 
         for name, index_config in self.REINDEX_CONFIG.items():
-            if not RedisQueue(index_config["queue_name"]).has_item():
+            if not RedisQueue(index_config["queue_name"]).length():
                 continue
 
             self.total = RedisQueue(index_config["queue_name"]).length()
@@ -290,6 +292,9 @@ class Reindex(ReindexBase):
 
         # read current state
         video.get_from_es()
+        if not video.json_data:
+            return
+
         es_meta = video.json_data.copy()
 
         # get new
@@ -327,6 +332,9 @@ class Reindex(ReindexBase):
         # read current state
         channel = YoutubeChannel(channel_id)
         channel.get_from_es()
+        if not channel.json_data:
+            return
+
         es_meta = channel.json_data.copy()
 
         # get new
@@ -355,6 +363,12 @@ class Reindex(ReindexBase):
         self._get_all_videos()
         playlist = YoutubePlaylist(playlist_id)
         playlist.get_from_es()
+        if (
+            not playlist.json_data
+            or playlist.json_data["playlist_type"] == "custom"
+        ):
+            return
+
         subscribed = playlist.json_data["playlist_subscribed"]
         playlist.all_youtube_ids = self.all_indexed_ids
         playlist.build_json(scrape=True)

tubearchivist/home/src/index/subtitle.py

@@ -128,6 +128,10 @@ class YoutubeSubtitle:
                 print(response.text)
                 continue
 
+            if not response.text:
+                print(f"{self.video.youtube_id}: skip empty subtitle")
+                continue
+
             parser = SubtitleParser(response.text, lang, source)
             parser.process()
             if not parser.all_cues:

tubearchivist/home/src/index/video.py

@@ -177,6 +177,7 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
 
     def _process_youtube_meta(self):
         """extract relevant fields from youtube"""
+        self._validate_id()
         # extract
         self.channel_id = self.youtube_meta["channel_id"]
         upload_date = self.youtube_meta["upload_date"]
@@ -188,11 +189,11 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
         # build json_data basics
         self.json_data = {
             "title": self.youtube_meta["title"],
-            "description": self.youtube_meta["description"],
-            "category": self.youtube_meta["categories"],
+            "description": self.youtube_meta.get("description", ""),
+            "category": self.youtube_meta.get("categories", []),
             "vid_thumb_url": self.youtube_meta["thumbnail"],
             "vid_thumb_base64": base64_blur,
-            "tags": self.youtube_meta["tags"],
+            "tags": self.youtube_meta.get("tags", []),
             "published": published,
             "vid_last_refresh": last_refresh,
             "date_downloaded": last_refresh,
@@ -202,6 +203,19 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
             "active": True,
         }
 
+    def _validate_id(self):
+        """validate expected video ID, raise value error on mismatch"""
+        remote_id = self.youtube_meta["id"]
+
+        if not self.youtube_id == remote_id:
+            # unexpected redirect
+            message = (
+                f"[reindex][{self.youtube_id}] got an unexpected redirect "
+                + f"to {remote_id}, you are probably getting blocked by YT. "
+                "See FAQ for more details."
+            )
+            raise ValueError(message)
+
     def _add_channel(self):
         """add channel dict to video json_data"""
         channel = ta_channel.YoutubeChannel(self.channel_id)
@@ -210,20 +224,13 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
 
     def _add_stats(self):
         """add stats dicst to json_data"""
-        # likes
-        like_count = self.youtube_meta.get("like_count", 0)
-        dislike_count = self.youtube_meta.get("dislike_count", 0)
-        average_rating = self.youtube_meta.get("average_rating", 0)
-        self.json_data.update(
-            {
-                "stats": {
-                    "view_count": self.youtube_meta["view_count"],
-                    "like_count": like_count,
-                    "dislike_count": dislike_count,
-                    "average_rating": average_rating,
-                }
-            }
-        )
+        stats = {
+            "view_count": self.youtube_meta.get("view_count", 0),
+            "like_count": self.youtube_meta.get("like_count", 0),
+            "dislike_count": self.youtube_meta.get("dislike_count", 0),
+            "average_rating": self.youtube_meta.get("average_rating", 0),
+        }
+        self.json_data.update({"stats": stats})
 
     def build_dl_cache_path(self):
         """find video path in dl cache"""
@@ -312,6 +319,8 @@ class YoutubeVideo(YouTubeItem, YoutubeSubtitle):
                     playlist.json_data["playlist_entries"][idx].update(
                         {"downloaded": False}
                     )
+                    if playlist.json_data["playlist_type"] == "custom":
+                        playlist.del_video(self.youtube_id)
             playlist.upload_to_es()
 
     def delete_subtitles(self, subtitles=False):

tubearchivist/home/src/ta/config.py

@@ -106,7 +106,10 @@ class AppConfig:
 
             # missing nested values
             for sub_key, sub_value in value.items():
-                if sub_key not in redis_config[key].keys():
+                if (
+                    sub_key not in redis_config[key].keys()
+                    or sub_value == "rand-d"
+                ):
                     if sub_value == "rand-d":
                         sub_value = self._build_rand_daily()
 
@@ -266,73 +269,83 @@ class ReleaseVersion:
     REMOTE_URL = "https://www.tubearchivist.com/api/release/latest/"
     NEW_KEY = "versioncheck:new"
 
-    def __init__(self):
-        self.local_version = self._parse_version(settings.TA_VERSION)
-        self.is_unstable = settings.TA_VERSION.endswith("-unstable")
-        self.remote_version = False
-        self.is_breaking = False
-        self.response = False
+    def __init__(self) -> None:
+        self.local_version: str = settings.TA_VERSION
+        self.is_unstable: bool = settings.TA_VERSION.endswith("-unstable")
+        self.remote_version: str = ""
+        self.is_breaking: bool = False
 
-    def check(self):
+    def check(self) -> None:
         """check version"""
         print(f"[{self.local_version}]: look for updates")
         self.get_remote_version()
-        new_version, is_breaking = self._has_update()
+        new_version = self._has_update()
         if new_version:
             message = {
                 "status": True,
                 "version": new_version,
-                "is_breaking": is_breaking,
+                "is_breaking": self.is_breaking,
             }
             RedisArchivist().set_message(self.NEW_KEY, message)
             print(f"[{self.local_version}]: found new version {new_version}")
 
-    def get_local_version(self):
+    def get_local_version(self) -> str:
         """read version from local"""
         return self.local_version
 
-    def get_remote_version(self):
+    def get_remote_version(self) -> None:
         """read version from remote"""
         sleep(randint(0, 60))
-        self.response = requests.get(self.REMOTE_URL, timeout=20).json()
-        remote_version_str = self.response["release_version"]
-        self.remote_version = self._parse_version(remote_version_str)
-        self.is_breaking = self.response["breaking_changes"]
+        response = requests.get(self.REMOTE_URL, timeout=20).json()
+        self.remote_version = response["release_version"]
+        self.is_breaking = response["breaking_changes"]
 
-    def _has_update(self):
+    def _has_update(self) -> str | bool:
         """check if there is an update"""
-        for idx, number in enumerate(self.local_version):
-            is_newer = self.remote_version[idx] > number
-            if is_newer:
-                return self.response["release_version"], self.is_breaking
+        remote_parsed = self._parse_version(self.remote_version)
+        local_parsed = self._parse_version(self.local_version)
+        if remote_parsed > local_parsed:
+            return self.remote_version
 
-        if self.is_unstable and self.local_version == self.remote_version:
-            return self.response["release_version"], self.is_breaking
+        if self.is_unstable and local_parsed == remote_parsed:
+            return self.remote_version
 
-        return False, False
+        return False
 
     @staticmethod
-    def _parse_version(version):
+    def _parse_version(version) -> tuple[int, ...]:
         """return version parts"""
         clean = version.rstrip("-unstable").lstrip("v")
         return tuple((int(i) for i in clean.split(".")))
 
-    def is_updated(self):
+    def is_updated(self) -> str | bool:
         """check if update happened in the mean time"""
         message = self.get_update()
         if not message:
             return False
 
-        if self._parse_version(message.get("version")) == self.local_version:
+        local_parsed = self._parse_version(self.local_version)
+        message_parsed = self._parse_version(message.get("version"))
+
+        if local_parsed >= message_parsed:
             RedisArchivist().del_message(self.NEW_KEY)
             return settings.TA_VERSION
 
         return False
 
-    def get_update(self):
+    def get_update(self) -> dict:
         """return new version dict if available"""
         message = RedisArchivist().get_message(self.NEW_KEY)
         if not message.get("status"):
-            return False
+            return {}
 
         return message
+
+    def clear_fail(self) -> None:
+        """clear key, catch previous error in v0.4.5"""
+        message = self.get_update()
+        if not message:
+            return
+
+        if isinstance(message.get("version"), list):
+            RedisArchivist().del_message(self.NEW_KEY)

tubearchivist/home/src/ta/ta_redis.py

@@ -20,6 +20,7 @@ class RedisBase:
         self.conn = redis.Redis(
             host=EnvironmentSettings.REDIS_HOST,
             port=EnvironmentSettings.REDIS_PORT,
+            decode_responses=True,
         )
 
 
@@ -82,7 +83,7 @@ class RedisArchivist(RedisBase):
         if not reply:
             return []
 
-        return [i.decode().lstrip(self.NAME_SPACE) for i in reply]
+        return [i.lstrip(self.NAME_SPACE) for i in reply]
 
     def list_items(self, query: str) -> list:
         """list all matches"""
@@ -99,65 +100,49 @@ class RedisArchivist(RedisBase):
 
 
 class RedisQueue(RedisBase):
-    """dynamically interact with queues in redis"""
+    """
+    dynamically interact with queues in redis using sorted set
+    - low score number is first in queue
+    - add new items with high score number
+    """
 
     def __init__(self, queue_name: str):
         super().__init__()
         self.key = f"{self.NAME_SPACE}{queue_name}"
 
-    def get_all(self):
+    def get_all(self) -> list[str]:
         """return all elements in list"""
-        result = self.conn.execute_command("LRANGE", self.key, 0, -1)
-        all_elements = [i.decode() for i in result]
-        return all_elements
+        result = self.conn.zrange(self.key, 0, -1)
+        return result
 
     def length(self) -> int:
         """return total elements in list"""
-        return self.conn.execute_command("LLEN", self.key)
+        return self.conn.zcard(self.key)
 
     def in_queue(self, element) -> str | bool:
         """check if element is in list"""
-        result = self.conn.execute_command("LPOS", self.key, element)
+        result = self.conn.zrank(self.key, element)
         if result is not None:
             return "in_queue"
 
         return False
 
-    def add_list(self, to_add):
+    def add_list(self, to_add: list) -> None:
         """add list to queue"""
-        self.conn.execute_command("RPUSH", self.key, *to_add)
-
-    def add_priority(self, to_add: str) -> None:
-        """add single video to front of queue"""
-        item: str = json.dumps(to_add)
-        self.clear_item(item)
-        self.conn.execute_command("LPUSH", self.key, item)
+        mapping = {i: "+inf" for i in to_add}
+        self.conn.zadd(self.key, mapping)
 
     def get_next(self) -> str | bool:
-        """return next element in the queue, False if none"""
-        result = self.conn.execute_command("LPOP", self.key)
+        """return next element in the queue, if available"""
+        result = self.conn.zpopmin(self.key)
         if not result:
             return False
 
-        next_element = result.decode()
-        return next_element
+        return result[0][0]
 
     def clear(self) -> None:
         """delete list from redis"""
-        self.conn.execute_command("DEL", self.key)
-
-    def clear_item(self, to_clear: str) -> None:
-        """remove single item from list if it's there"""
-        self.conn.execute_command("LREM", self.key, 0, to_clear)
-
-    def trim(self, size: int) -> None:
-        """trim the queue based on settings amount"""
-        self.conn.execute_command("LTRIM", self.key, 0, size)
-
-    def has_item(self) -> bool:
-        """check if queue as at least one pending item"""
-        result = self.conn.execute_command("LRANGE", self.key, 0, 0)
-        return bool(result)
+        self.conn.delete(self.key)
 
 
 class TaskRedis(RedisBase):
@@ -170,7 +155,7 @@ class TaskRedis(RedisBase):
     def get_all(self) -> list:
         """return all tasks"""
         all_keys = self.conn.execute_command("KEYS", f"{self.BASE}*")
-        return [i.decode().replace(self.BASE, "") for i in all_keys]
+        return [i.replace(self.BASE, "") for i in all_keys]
 
     def get_single(self, task_id: str) -> dict:
         """return content of single task"""
@@ -178,7 +163,7 @@ class TaskRedis(RedisBase):
         if not result:
             return {}
 
-        return json.loads(result.decode())
+        return json.loads(result)
 
     def set_key(
         self, task_id: str, message: dict, expire: bool | int = False

tubearchivist/home/src/ta/urlparser.py

@@ -92,7 +92,7 @@ class Parser:
             item_type = "video"
         elif len_id_str == 24:
             item_type = "channel"
-        elif len_id_str in (34, 26, 18):
+        elif len_id_str in (34, 26, 18) or id_str.startswith("TA_playlist_"):
             item_type = "playlist"
         else:
             raise ValueError(f"not a valid id_str: {id_str}")

tubearchivist/home/src/ta/users.py

@@ -50,7 +50,14 @@ class UserConfig:
     VALID_STYLESHEETS = get_stylesheets()
     VALID_VIEW_STYLE = ["grid", "list"]
     VALID_SORT_ORDER = ["asc", "desc"]
-    VALID_SORT_BY = ["published", "downloaded", "views", "likes"]
+    VALID_SORT_BY = [
+        "published",
+        "downloaded",
+        "views",
+        "likes",
+        "duration",
+        "filesize",
+    ]
     VALID_GRID_ITEMS = range(3, 8)
 
     def __init__(self, user_id: str):

tubearchivist/home/templates/home/base.html

@@ -33,11 +33,9 @@
 <body>
     <div class="main-content">
         <div class="boxed-content">
-            <div class="top-banner">
-                <a href="{% url 'home' %}">
-                    <img alt="tube-archivist-banner">
-                </a>
-            </div>
+            <a href="{% url 'home' %}">
+                <div class="top-banner"></div>
+            </a>
             <div class="top-nav">
                 <div class="nav-items">
                     <a href="{% url 'home' %}">

tubearchivist/home/templates/home/channel_id.html

@@ -82,6 +82,8 @@
                     <option value="downloaded" {% if sort_by == "downloaded" %}selected{% endif %}>date downloaded</option>
                     <option value="views" {% if sort_by == "views" %}selected{% endif %}>views</option>
                     <option value="likes" {% if sort_by == "likes" %}selected{% endif %}>likes</option>
+                    <option value="duration" {% if sort_by == "duration" %}selected{% endif %}>duration</option>
+                    <option value="filesize" {% if sort_by == "filesize" %}selected{% endif %}>file size</option>
                 </select>
                 <select name="sort_order" id="sort-order" onchange="sortChange(this)">
                     <option value="asc" {% if sort_order == "asc" %}selected{% endif %}>asc</option>

tubearchivist/home/templates/home/home.html

@@ -65,6 +65,8 @@
                     <option value="downloaded" {% if sort_by == "downloaded" %}selected{% endif %}>date downloaded</option>
                     <option value="views" {% if sort_by == "views" %}selected{% endif %}>views</option>
                     <option value="likes" {% if sort_by == "likes" %}selected{% endif %}>likes</option>
+                    <option value="duration" {% if sort_by == "duration" %}selected{% endif %}>duration</option>
+                    <option value="filesize" {% if sort_by == "filesize" %}selected{% endif %}>file size</option>
                 </select>
                 <select name="sort_order" id="sort-order" onchange="sortChange(this)">
                     <option value="asc" {% if sort_order == "asc" %}selected{% endif %}>asc</option>

tubearchivist/home/templates/home/playlist.html

@@ -11,13 +11,18 @@
         {% if request.user|has_group:"admin" or request.user.is_staff %}
 
         <div class="title-split-form">
-            <img id="animate-icon" onclick="showForm()" src="{% static 'img/icon-add.svg' %}" alt="add-icon" title="Subscribe to Playlists">
+            <img id="animate-icon" onclick="showForm();showForm('hidden-form2')" src="{% static 'img/icon-add.svg' %}" alt="add-icon" title="Subscribe to Playlists">
             <div class="show-form">
                 <form id="hidden-form" action="/playlist/" method="post">
                     {% csrf_token %}
                     {{ subscribe_form }}
                     <button type="submit">Subscribe</button>
                 </form>
+                <form id="hidden-form2" action="/playlist/" method="post">
+                    {% csrf_token %}
+                    {{ create_form }}
+                    <button type="submit">Create</button>
+                </form>
             </div>
         </div>
         {% endif %}
@@ -51,14 +56,18 @@
                     </a>
                 </div>
                 <div class="playlist-desc {{ view_style }}">
-                    <a href="{% url 'channel_id' playlist.playlist_channel_id %}"><h3>{{ playlist.playlist_channel }}</h3></a>
+                    {% if playlist.playlist_type != "custom" %}
+                    	<a href="{% url 'channel_id' playlist.playlist_channel_id %}"><h3>{{ playlist.playlist_channel }}</h3></a>
+                    {% endif %}
                     <a href="{% url 'playlist_id' playlist.playlist_id %}"><h2>{{ playlist.playlist_name }}</h2></a>
                     <p>Last refreshed: {{ playlist.playlist_last_refresh }}</p>
-                    {% if playlist.playlist_subscribed %}
-                        <button class="unsubscribe" type="button" data-type="playlist" data-subscribe="" data-id="{{ playlist.playlist_id }}" onclick="subscribeStatus(this)" title="Unsubscribe from {{ playlist.playlist_name }}">Unsubscribe</button>
-                    {% else %}
-                        <button type="button" data-type="playlist" data-subscribe="true" data-id="{{ playlist.playlist_id }}" onclick="subscribeStatus(this)" title="Subscribe to {{ playlist.playlist_name }}">Subscribe</button>
-                    {% endif %}
+                    {% if playlist.playlist_type != "custom" %}
+	                    {% if playlist.playlist_subscribed %}
+	                        <button class="unsubscribe" type="button" data-type="playlist" data-subscribe="" data-id="{{ playlist.playlist_id }}" onclick="subscribeStatus(this)" title="Unsubscribe from {{ playlist.playlist_name }}">Unsubscribe</button>
+	                    {% else %}
+	                        <button type="button" data-type="playlist" data-subscribe="true" data-id="{{ playlist.playlist_id }}" onclick="subscribeStatus(this)" title="Subscribe to {{ playlist.playlist_name }}">Subscribe</button>
+	                    {% endif %}
+	                {% endif %}
                 </div>
             </div>
             {% endfor %}

tubearchivist/home/templates/home/playlist_id.html

@@ -9,37 +9,42 @@
         <h1>{{ playlist_info.playlist_name }}</h1>
     </div>
     <div class="info-box info-box-3">
-        <div class="info-box-item">
-            <div class="round-img">
-                <a href="{% url 'channel_id' channel_info.channel_id %}">
-                    <img src="/cache/channels/{{ channel_info.channel_id }}_thumb.jpg" alt="channel-thumb">
-                </a>
-            </div>
-            <div>
-                <h3><a href="{% url 'channel_id' channel_info.channel_id %}">{{ channel_info.channel_name }}</a></h3>
-                {% if channel_info.channel_subs >= 1000000 %}
-                    <span>Subscribers: {{ channel_info.channel_subs|intword }}</span>
-                {% else %}
-                    <span>Subscribers: {{ channel_info.channel_subs|intcomma }}</span>
-                {% endif %}
-            </div>
-        </div>
+        {% if playlist_info.playlist_type != "custom" %}
+	        <div class="info-box-item">
+	            <div class="round-img">
+	                <a href="{% url 'channel_id' channel_info.channel_id %}">
+	                    <img src="/cache/channels/{{ channel_info.channel_id }}_thumb.jpg" alt="channel-thumb">
+	                </a>
+	            </div>
+	            <div>
+	                <h3><a href="{% url 'channel_id' channel_info.channel_id %}">{{ channel_info.channel_name }}</a></h3>
+	                {% if channel_info.channel_subs >= 1000000 %}
+	                    <span>Subscribers: {{ channel_info.channel_subs|intword }}</span>
+	                {% else %}
+	                    <span>Subscribers: {{ channel_info.channel_subs|intcomma }}</span>
+	                {% endif %}
+	            </div>
+	        </div>
+	    {% endif %}
+        
         <div class="info-box-item">
             <div>
                 <p>Last refreshed: {{ playlist_info.playlist_last_refresh }}</p>
-                <p>Playlist:
-                    {% if playlist_info.playlist_subscribed %}
-                        {% if request.user|has_group:"admin" or request.user.is_staff %}
-                        <button class="unsubscribe" type="button" data-type="playlist" data-subscribe="" data-id="{{ playlist_info.playlist_id }}" onclick="subscribeStatus(this)" title="Unsubscribe from {{ playlist_info.playlist_name }}">Unsubscribe</button>
-                        {% endif %}
-                    {% else %}
-                        <button type="button" data-type="playlist" data-subscribe="true" data-id="{{ playlist_info.playlist_id }}" onclick="subscribeStatus(this)" title="Subscribe to {{ playlist_info.playlist_name }}">Subscribe</button>
-                    {% endif %}
-                </p>
-                {% if playlist_info.playlist_active %}
-                    <p>Youtube: <a href="https://www.youtube.com/playlist?list={{ playlist_info.playlist_id }}" target="_blank">Active</a></p>
-                {% else %}
-                    <p>Youtube: Deactivated</p>
+                {% if playlist_info.playlist_type != "custom" %}
+	                <p>Playlist:
+	                    {% if playlist_info.playlist_subscribed %}
+	                        {% if request.user|has_group:"admin" or request.user.is_staff %}
+	                        <button class="unsubscribe" type="button" data-type="playlist" data-subscribe="" data-id="{{ playlist_info.playlist_id }}" onclick="subscribeStatus(this)" title="Unsubscribe from {{ playlist_info.playlist_name }}">Unsubscribe</button>
+	                        {% endif %}
+	                    {% else %}
+	                        <button type="button" data-type="playlist" data-subscribe="true" data-id="{{ playlist_info.playlist_id }}" onclick="subscribeStatus(this)" title="Subscribe to {{ playlist_info.playlist_name }}">Subscribe</button>
+	                    {% endif %}
+	                </p>
+	                {% if playlist_info.playlist_active %}
+	                    <p>Youtube: <a href="https://www.youtube.com/playlist?list={{ playlist_info.playlist_id }}" target="_blank">Active</a></p>
+	                {% else %}
+	                    <p>Youtube: Deactivated</p>
+	                {% endif %}
                 {% endif %}
                 <button onclick="deleteConfirm()" id="delete-item">Delete Playlist</button>
                 <div class="delete-confirm" id="delete-button">
@@ -63,7 +68,9 @@
                     <p>Reindex scheduled</p>
                 {% else %}
                     <div id="reindex-button" class="button-box">
-                        <button data-id="{{ playlist_info.playlist_id }}" data-type="playlist" onclick="reindex(this)" title="Reindex Playlist {{ playlist_info.playlist_name }}">Reindex</button>
+                    	{% if playlist_info.playlist_type != "custom" %}
+                        	<button data-id="{{ playlist_info.playlist_id }}" data-type="playlist" onclick="reindex(this)" title="Reindex Playlist {{ playlist_info.playlist_name }}">Reindex</button>
+                        {% endif %}
                         <button data-id="{{ playlist_info.playlist_id }}" data-type="playlist" data-extract-videos="true" onclick="reindex(this)" title="Reindex Videos of {{ playlist_info.playlist_name }}">Reindex Videos</button>
                     </div>
                 {% endif %}
@@ -138,15 +145,35 @@
                             {% endif %}
                             <span>{{ video.published }} | {{ video.player.duration_str }}</span>
                         </div>
-                        <div>
-                            <a class="video-more" href="{% url 'video' video.youtube_id %}"><h2>{{ video.title }}</h2></a>
+                        <div class="video-desc-details">
+                            <div>
+                                {% if playlist_info.playlist_type == "custom" %}
+	                            	<a href="{% url 'channel_id' video.channel.channel_id %}"><h3>{{ video.channel.channel_name }}</h3></a>
+	                            {% endif %}
+	                            <a class="video-more" href="{% url 'video' video.youtube_id %}"><h2>{{ video.title }}</h2></a>
+                        	</div>
+                        	{% if playlist_info.playlist_type == "custom" %}
+                        		{% if pagination %}
+                        			{% if pagination.last_page > 0 %}
+	                        			<img id="{{ video.youtube_id }}-button" src="{% static 'img/icon-dot-menu.svg' %}" alt="dot-menu-icon" data-id="{{ video.youtube_id }}" data-context="video" onclick="showCustomPlaylistMenu(this,'{{playlist_info.playlist_id}}',{{pagination.current_page}},{{pagination.last_page}})" class="dot-button" title="More actions">
+	                        		{% else %}
+	                        			<img id="{{ video.youtube_id }}-button" src="{% static 'img/icon-dot-menu.svg' %}" alt="dot-menu-icon" data-id="{{ video.youtube_id }}" data-context="video" onclick="showCustomPlaylistMenu(this,'{{playlist_info.playlist_id}}',{{pagination.current_page}},{{pagination.current_page}})" class="dot-button" title="More actions">
+                        			{% endif %}
+                        		{% else %}
+                        			<img id="{{ video.youtube_id }}-button" src="{% static 'img/icon-dot-menu.svg' %}" alt="dot-menu-icon" data-id="{{ video.youtube_id }}" data-context="video" onclick="showCustomPlaylistMenu(this,'{{playlist_info.playlist_id}}',0,0)" class="dot-button" title="More actions">
+                        		{% endif %}
+                        	{% endif %}
                         </div>
                     </div>
                 </div>
             {% endfor %}
         {% else %}
             <h2>No videos found...</h2>
-            <p>Try going to the <a href="{% url 'downloads' %}">downloads page</a> to start the scan and download tasks.</p>
+            {% if playlist_info.playlist_type == "custom" %}
+               <p>Try going to the <a href="{% url 'home' %}">home page</a> to add videos to this playlist.</p>
+            {% else %}
+            	<p>Try going to the <a href="{% url 'downloads' %}">downloads page</a> to start the scan and download tasks.</p>
+            {% endif %} 
         {% endif %}
     </div>
 </div>

tubearchivist/home/templates/home/settings.html

@@ -5,14 +5,26 @@
     <h1>Your Archive</h1>
 </div>
 <div class="settings-item">
-    <h2>Main overview</h2>
-    <div id="primaryBox" class="info-box info-box-4">
+    <h2>Overview</h2>
+    <div id="activeBox" class="info-box info-box-3">
+        <p id="loading">Loading...</p>
+    </div>
+</div>
+<div class="settings-item">
+    <h2>Video Type</h2>
+    <div id="videoTypeBox" class="info-box info-box-3">
+        <p id="loading">Loading...</p>
+    </div>
+</div>
+<div class="settings-item">
+    <h2>Application</h2>
+    <div id="secondaryBox" class="info-box info-box-3">
         <p id="loading">Loading...</p>
     </div>
 </div>
 <div class="settings-item">
     <h2>Watch Progress</h2>
-    <div id="watchBox" class="info-box info-box-3">
+    <div id="watchBox" class="info-box info-box-2">
         <p id="loading">Loading...</p>
     </div>
 </div>

tubearchivist/home/templates/home/settings_scheduling.html

@@ -51,7 +51,7 @@
         </div>
     </div>
     <div class="settings-group">
-        <h2>Start download</h2>
+        <h2>Start Download</h2>
         <div class="settings-item">
             <p>Current Download schedule: <span class="settings-current">
                 {% if config.scheduler.download_pending %}
@@ -112,7 +112,7 @@
         </div>
     </div>
     <div class="settings-group">
-        <h2>Thumbnail check</h2>
+        <h2>Thumbnail Check</h2>
         <div class="settings-item">
             <p>Current thumbnail check schedule: <span class="settings-current">
                 {% if config.scheduler.thumbnail_check %}

tubearchivist/home/templates/home/video.html

@@ -95,6 +95,7 @@
                     <span>Are you sure? </span><button class="danger-button" onclick="deleteVideo(this)" data-id="{{ video.youtube_id }}" data-redirect = "{{ video.channel.channel_id }}">Delete</button> <button onclick="cancelDelete()">Cancel</button>
                 </div>
                 {% endif %}
+                <button id="{{ video.youtube_id }}-button" data-id="{{ video.youtube_id }}" data-context="video" onclick="showAddToPlaylistMenu(this)">Add To Playlist</button>
             </div>
         </div>
         <div class="info-box-item">

tubearchivist/home/views.py

@@ -3,8 +3,10 @@ Functionality:
 - all views for home app
 - holds base classes to inherit from
 """
+
 import enum
 import urllib.parse
+import uuid
 from time import sleep
 
 from api.src.search_processor import SearchProcess, process_aggs
@@ -26,6 +28,7 @@ from home.src.frontend.forms import (
     AddToQueueForm,
     ApplicationSettingsForm,
     ChannelOverwriteForm,
+    CreatePlaylistForm,
     CustomAuthForm,
     MultiSearchForm,
     SchedulerSettingsForm,
@@ -111,6 +114,8 @@ class ArchivistResultsView(ArchivistViewConfig):
             "likes": "stats.like_count",
             "downloaded": "date_downloaded",
             "published": "published",
+            "duration": "player.duration",
+            "filesize": "media_size",
         }
         sort_by = sort_by_map[self.context["sort_by"]]
 
@@ -519,7 +524,7 @@ class ChannelIdView(ChannelIdBaseView):
 
         self.context.update(
             {
-                "title": "Channel: " + channel_name,
+                "title": f"Channel: {channel_name}",
                 "channel_info": channel_info,
             }
         )
@@ -737,12 +742,12 @@ class PlaylistIdView(ArchivistResultsView):
         # playlist details
         es_path = f"ta_playlist/_doc/{playlist_id}"
         playlist_info = self.single_lookup(es_path)
-
-        # channel details
-        channel_id = playlist_info["playlist_channel_id"]
-        es_path = f"ta_channel/_doc/{channel_id}"
-        channel_info = self.single_lookup(es_path)
-
+        channel_info = None
+        if playlist_info["playlist_type"] != "custom":
+            # channel details
+            channel_id = playlist_info["playlist_channel_id"]
+            es_path = f"ta_channel/_doc/{channel_id}"
+            channel_info = self.single_lookup(es_path)
         return playlist_info, channel_info
 
     def _update_view_data(self, playlist_id, playlist_info):
@@ -800,6 +805,7 @@ class PlaylistView(ArchivistResultsView):
             {
                 "title": "Playlists",
                 "subscribe_form": SubscribeToPlaylistForm(),
+                "create_form": CreatePlaylistForm(),
             }
         )
 
@@ -834,12 +840,19 @@ class PlaylistView(ArchivistResultsView):
     @method_decorator(user_passes_test(check_admin), name="dispatch")
     @staticmethod
     def post(request):
-        """handle post from search form"""
-        subscribe_form = SubscribeToPlaylistForm(data=request.POST)
-        if subscribe_form.is_valid():
-            url_str = request.POST.get("subscribe")
-            print(url_str)
-            subscribe_to.delay(url_str, expected_type="playlist")
+        """handle post from subscribe or create form"""
+        if request.POST.get("create") is not None:
+            create_form = CreatePlaylistForm(data=request.POST)
+            if create_form.is_valid():
+                name = request.POST.get("create")
+                playlist_id = f"TA_playlist_{uuid.uuid4()}"
+                YoutubePlaylist(playlist_id).create(name)
+        else:
+            subscribe_form = SubscribeToPlaylistForm(data=request.POST)
+            if subscribe_form.is_valid():
+                url_str = request.POST.get("subscribe")
+                print(url_str)
+                subscribe_to.delay(url_str, expected_type="playlist")
 
         sleep(1)
         return redirect("playlist")

tubearchivist/requirements.txt

@@ -1,13 +1,13 @@
-apprise==1.6.0
-celery==5.3.4
-Django==4.2.7
-django-auth-ldap==4.6.0
-django-cors-headers==4.3.0
-djangorestframework==3.14.0
-Pillow==10.1.0
-redis==5.0.1
+apprise==1.7.5
+celery==5.3.6
+Django==5.0.4
+django-auth-ldap==4.8.0
+django-cors-headers==4.3.1
+djangorestframework==3.15.1
+Pillow==10.3.0
+redis==5.0.0
 requests==2.31.0
 ryd-client==0.0.6
-uWSGI==2.0.23
+uWSGI==2.0.24
 whitenoise==6.6.0
-yt-dlp==2023.10.13
+yt-dlp==2024.4.9

tubearchivist/static/css/style.css

@@ -159,13 +159,13 @@ button:hover {
 }
 
 .top-banner {
-    text-align: center;
-}
-
-.top-banner img {
-    width: 100%;
-    max-width: 700px;
-    content: var(--banner);
+    background-image: var(--banner);
+    background-repeat: no-repeat;
+    background-size: contain;
+    height: 10vh;
+    min-height: 80px;
+    max-height: 120px;
+    background-position: center center;
 }
 
 .footer {
@@ -180,6 +180,10 @@ button:hover {
     text-decoration: underline;
 }
 
+.footer .boxed-content {
+    text-align: center;
+}
+
 /* toggle on-off */
 .toggle {
     display: flex;
@@ -367,11 +371,23 @@ button:hover {
     filter: var(--img-filter);
 }
 
+.video-popup-menu {
+    border-top: 2px solid;
+    border-color: var(--accent-font-dark);
+    margin: 5px 0;
+    padding-top: 10px;
+}
+
 #hidden-form {
     display: none;
 }
 
-#hidden-form button {
+#hidden-form2 {
+    display: none;
+    margin-top: 10px;
+}
+
+#hidden-form button, #hidden-form2 button  {
     margin-right: 1rem;
 }
 
@@ -560,6 +576,12 @@ video:-webkit-full-screen {
     margin-right: 10px;
 }
 
+.video-popup-menu img.move-video-button {
+	width: 24px;
+	cursor: pointer;
+    filter: var(--img-filter);
+}
+
 .video-desc a {
     text-decoration: none;
     text-align: left;
@@ -588,7 +610,13 @@ video:-webkit-full-screen {
     align-items: center;
 }
 
+.video-desc-details {
+	display: flex;
+	justify-content: space-between;
+}
+
 .watch-button,
+.dot-button,
 .close-button {
     cursor: pointer;
     filter: var(--img-filter);
@@ -678,6 +706,19 @@ video:-webkit-full-screen {
     width: 100%;
 }
 
+.video-popup-menu img {
+	width: 12px;
+	cursor: pointer;
+    filter: var(--img-filter);
+}
+
+
+.video-popup-menu-close-button {
+	cursor: pointer;
+    filter: var(--img-filter);
+    float:right;
+}
+
 .description-text {
     width: 100%;
 }

tubearchivist/static/img/icon-arrow-bottom.svg

@@ -0,0 +1,14 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!-- Generator: Adobe Illustrator 27.5.0, SVG Export Plug-In . SVG Version: 6.00 Build 0)  -->
+<svg version="1.1"
+	 id="svg1303" xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" xmlns:cc="http://creativecommons.org/ns#" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:svg="http://www.w3.org/2000/svg" xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+	 xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px" viewBox="0 0 500 500"
+	 style="enable-background:new 0 0 500 500;" xml:space="preserve">
+<g>
+	<path d="M231.3,382c4.8,5.3,11.6,8.3,18.7,8.3c7.1,0,13.9-3,18.7-8.3l112.1-124.2c6.7-7.4,8.4-18,4.3-27.1
+		c-4-9.1-13.1-14.9-23-14.9h-52.9V64.9c0-13.9-11.3-25.2-25.2-25.2h-68c-13.9,0-25.2,11.3-25.2,25.2v150.9h-52.9
+		c-10,0-19,5.9-23,14.9c-4,9.1-2.3,19.7,4.3,27.1L231.3,382z"/>
+	<path d="M436.1,408.8H63.9c-13.6,0-24.7,11.1-24.7,24.7c0,13.6,11.1,24.7,24.7,24.7h372.2c13.6,0,24.7-11.1,24.7-24.7
+		C460.8,419.9,449.7,408.8,436.1,408.8z"/>
+</g>
+</svg>

tubearchivist/static/img/icon-arrow-down.svg

@@ -0,0 +1,6 @@
+<?xml version="1.0" encoding="utf-8"?>
+<svg viewBox="0 0 500 500" style="enable-background:new 0 0 500 500;" xmlns="http://www.w3.org/2000/svg">
+  <g style="" transform="matrix(0.9999999999999999, 0, 0, -0.9999999999999999, 0, 497.9000392526395)">
+    <path d="M 231.3 48 C 236.1 42.7 242.9 39.7 250 39.7 C 257.1 39.7 263.9 42.7 268.7 48 L 380.8 172.2 C 387.5 179.6 389.2 190.2 385.1 199.3 C 381.1 208.4 372 214.2 362.1 214.2 L 309.2 214.2 L 309.2 365.1 C 309.2 379 297.9 390.3 284 390.3 L 216 390.3 C 202.1 390.3 190.8 379 190.8 365.1 L 190.8 214.2 L 137.9 214.2 C 127.9 214.2 118.9 208.3 114.9 199.3 C 110.9 190.2 112.6 179.6 119.2 172.2 L 231.3 48 Z" style=""/>
+  </g>
+</svg>

tubearchivist/static/img/icon-arrow-top.svg

@@ -0,0 +1,7 @@
+<?xml version="1.0" encoding="utf-8"?>
+<svg viewBox="0 0 500 500" style="enable-background:new 0 0 500 500;" xmlns="http://www.w3.org/2000/svg">
+  <g style="" transform="matrix(0.9999999999999999, 0, 0, -0.9999999999999999, 0, 497.9000392526395)">
+    <path d="M231.3,382c4.8,5.3,11.6,8.3,18.7,8.3c7.1,0,13.9-3,18.7-8.3l112.1-124.2c6.7-7.4,8.4-18,4.3-27.1 c-4-9.1-13.1-14.9-23-14.9h-52.9V64.9c0-13.9-11.3-25.2-25.2-25.2h-68c-13.9,0-25.2,11.3-25.2,25.2v150.9h-52.9 c-10,0-19,5.9-23,14.9c-4,9.1-2.3,19.7,4.3,27.1L231.3,382z"/>
+    <path d="M436.1,408.8H63.9c-13.6,0-24.7,11.1-24.7,24.7c0,13.6,11.1,24.7,24.7,24.7h372.2c13.6,0,24.7-11.1,24.7-24.7 C460.8,419.9,449.7,408.8,436.1,408.8z"/>
+  </g>
+</svg>

tubearchivist/static/img/icon-arrow-up.svg

@@ -0,0 +1,6 @@
+<?xml version="1.0" encoding="utf-8"?>
+<svg viewBox="0 0 500 500" style="enable-background:new 0 0 500 500;" xmlns="http://www.w3.org/2000/svg">
+  <g style="" transform="matrix(0.9999999999999999, 0, 0, -0.9999999999999999, 0, 497.9000392526395)">
+    <path d="M231.3,382c4.8,5.3,11.6,8.3,18.7,8.3c7.1,0,13.9-3,18.7-8.3l112.1-124.2c6.7-7.4,8.4-18,4.3-27.1 c-4-9.1-13.1-14.9-23-14.9h-52.9V64.9c0-13.9-11.3-25.2-25.2-25.2h-68c-13.9,0-25.2,11.3-25.2,25.2v150.9h-52.9 c-10,0-19,5.9-23,14.9c-4,9.1-2.3,19.7,4.3,27.1L231.3,382z"/>
+  </g>
+</svg>

tubearchivist/static/img/icon-dot-menu.svg

@@ -0,0 +1,4 @@
+<?xml version="1.0" encoding="utf-8"?><!-- Uploaded to: SVG Repo, www.svgrepo.com, Generator: SVG Repo Mixer Tools -->
+<svg width="800px" height="800px" viewBox="0 0 16 16" xmlns="http://www.w3.org/2000/svg" fill="#000000" class="bi bi-three-dots-vertical">
+  <path d="M9.5 13a1.5 1.5 0 1 1-3 0 1.5 1.5 0 0 1 3 0zm0-5a1.5 1.5 0 1 1-3 0 1.5 1.5 0 0 1 3 0zm0-5a1.5 1.5 0 1 1-3 0 1.5 1.5 0 0 1 3 0z"/>
+</svg>

tubearchivist/static/img/icon-remove.svg

@@ -0,0 +1,4 @@
+<?xml version="1.0" encoding="utf-8"?>
+<svg viewBox="0 0 500 500" style="enable-background:new 0 0 500 500;" xmlns="http://www.w3.org/2000/svg">
+  <path d="M 408.514 358.563 L 303.835 255.003 L 408.441 149.115 C 424.1 133.406 424.1 107.662 408.441 91.953 C 392.783 76.245 367.12 76.245 351.463 91.953 L 246.857 197.841 L 142.323 91.881 C 126.665 76.172 101.002 76.172 85.344 91.881 C 69.613 107.662 69.613 133.334 85.272 149.115 L 189.805 255.003 L 84.401 359.291 C 68.743 375 68.743 400.744 84.401 416.453 C 100.06 432.161 125.722 432.161 141.381 416.453 L 246.784 312.165 L 351.39 415.726 C 367.048 431.434 392.711 431.434 408.369 415.726 C 424.172 400.017 424.172 374.345 408.514 358.563 Z" style="" transform="matrix(0.9999999999999999, 0, 0, 0.9999999999999999, 0, 0)"/>
+</svg>

tubearchivist/static/progress.js

@@ -12,7 +12,7 @@ checkMessages();
 // start to look for messages
 function checkMessages() {
   let notifications = document.getElementById('notifications');
-  if (notifications) {
+  if (notifications && notifications.childNodes.length === 0 ) {
     let dataOrigin = notifications.getAttribute('data');
     getMessages(dataOrigin);
   }

tubearchivist/static/script.js

@@ -197,6 +197,137 @@ function addToQueue(autostart = false) {
   showForm();
 }
 
+//shows the video sub menu popup
+function showAddToPlaylistMenu(input1) {
+  let dataId, playlists, form_code, buttonId;
+  dataId = input1.getAttribute('data-id');
+  buttonId = input1.getAttribute('id');
+  playlists = getCustomPlaylists();
+  
+  //hide the invoking button
+  input1.style.visibility = "hidden";
+  
+  //show the form
+  form_code = '<div class="video-popup-menu"><img src="/static/img/icon-close.svg" class="video-popup-menu-close-button" title="Close menu" onclick="removeDotMenu(this, \''+buttonId+'\')"/><h3>Add video to...</h3>';
+  
+  for(let i = 0; i < playlists.length; i++) {
+    	let obj = playlists[i];
+	    form_code += '<p onclick="addToCustomPlaylist(this, \''+dataId+'\',\''+obj.playlist_id+'\')"><img class="p-button" src="/static/img/icon-unseen.svg"/>'+obj.playlist_name+'</p>';
+  }
+  
+  form_code += '<p><a href="/playlist">Create playlist</a></p></div>';
+  input1.parentNode.parentNode.innerHTML += form_code;
+}
+
+//handles user action of adding a video to a custom playlist
+function addToCustomPlaylist(input, video_id, playlist_id) {
+  let apiEndpoint = '/api/playlist/' + playlist_id + '/';
+  let data = { "action": "create", "video_id": video_id };
+  apiRequest(apiEndpoint, 'POST', data);
+  
+  //mark the item added in the ui
+  input.firstChild.src='/static/img/icon-seen.svg';
+}
+
+function removeDotMenu(input1, button_id) {
+
+  //show the menu button
+  document.getElementById(button_id).style.visibility = "visible";
+  
+  //remove the form
+  input1.parentNode.remove();
+}
+
+//shows the video sub menu popup on custom playlist page
+function showCustomPlaylistMenu(input1, playlist_id, current_page, last_page) {
+  let dataId, form_code, buttonId;
+  dataId = input1.getAttribute('data-id');
+  buttonId = input1.getAttribute('id');
+  
+  //hide the invoking button
+  input1.style.visibility = "hidden";
+  
+  //show the form
+  form_code = '<div class="video-popup-menu"><img src="/static/img/icon-close.svg" class="video-popup-menu-close-button" title="Close menu" onclick="removeDotMenu(this, \''+buttonId+'\')"/><h3>Move Video</h3>';
+  
+  form_code += '<img class="move-video-button" data-id="'+dataId+'" data-context="top" onclick="moveCustomPlaylistVideo(this,\''+playlist_id+'\','+current_page+','+last_page+')" src="/static/img/icon-arrow-top.svg" title="Move to top"/>';
+  form_code += '<img class="move-video-button" data-id="'+dataId+'" data-context="up" onclick="moveCustomPlaylistVideo(this,\''+playlist_id+'\','+current_page+','+last_page+')" src="/static/img/icon-arrow-up.svg" title="Move up"/>';
+  form_code += '<img class="move-video-button" data-id="'+dataId+'" data-context="down" onclick="moveCustomPlaylistVideo(this,\''+playlist_id+'\','+current_page+','+last_page+')" src="/static/img/icon-arrow-down.svg" title="Move down"/>';
+  form_code += '<img class="move-video-button" data-id="'+dataId+'" data-context="bottom" onclick="moveCustomPlaylistVideo(this,\''+playlist_id+'\','+current_page+','+last_page+')" src="/static/img/icon-arrow-bottom.svg" title="Move to bottom"/>';
+  form_code += '<img class="move-video-button" data-id="'+dataId+'" data-context="remove" onclick="moveCustomPlaylistVideo(this,\''+playlist_id+'\','+current_page+','+last_page+')" src="/static/img/icon-remove.svg" title="Remove from playlist"/>';
+ 
+  
+  form_code += '</div>';
+  input1.parentNode.parentNode.innerHTML += form_code;
+}
+
+//process custom playlist form actions
+function moveCustomPlaylistVideo(input1, playlist_id, current_page, last_page) {
+  let dataId, dataContext;
+  dataId = input1.getAttribute('data-id');
+  dataContext = input1.getAttribute('data-context');
+  
+  let apiEndpoint = '/api/playlist/' + playlist_id + '/';
+  let data = { "action": dataContext, "video_id": dataId };
+  apiRequest(apiEndpoint, 'POST', data);
+  
+  let itemDom = input1.parentElement.parentElement.parentElement;
+  let listDom = itemDom.parentElement;
+  
+  if (dataContext === "up")
+  {
+     let sibling = itemDom.previousElementSibling;
+     if (sibling !== null)
+     {
+        sibling.before(itemDom);
+     }
+     else if (current_page > 1)
+     {
+       itemDom.remove();
+     }
+  }
+  else if (dataContext === "down")
+  {
+     let sibling = itemDom.nextElementSibling;
+     if (sibling !== null)
+     {
+        sibling.after(itemDom);
+     }
+     else if (current_page !== last_page)
+     {
+       itemDom.remove();
+     }
+  }
+  else if (dataContext === "top")
+  {
+     let sibling = listDom.firstElementChild;
+     if (sibling !== null)
+     {
+        sibling.before(itemDom);
+     }
+     if (current_page > 1)
+     {
+       itemDom.remove();
+     }
+  }
+  else if (dataContext === "bottom")
+  {
+     let sibling = listDom.lastElementChild;
+     if (sibling !== null)
+     {
+        sibling.after(itemDom);
+     }
+     if (current_page !== last_page)
+     {
+       itemDom.remove();
+     }
+  }
+  else if (dataContext === "remove")
+  {
+     itemDom.remove();
+  }
+}
+
 function toIgnore(button) {
   let youtube_id = button.getAttribute('data-id');
   let apiEndpoint = '/api/download/' + youtube_id + '/';
@@ -451,7 +582,7 @@ function createPlayer(button) {
   }
   let videoName = videoData.data.title;
 
-  let videoTag = createVideoTag(videoData, videoProgress);
+  let videoTag = createVideoTag(videoData, videoProgress, true);
 
   let playlist = '';
   let videoPlaylists = videoData.data.playlist; // Array of playlists the video is in
@@ -529,7 +660,7 @@ function insertVideoTag(videoData, videoProgress) {
 }
 
 // Generates a video tag with subtitles when passed videoData and videoProgress.
-function createVideoTag(videoData, videoProgress) {
+function createVideoTag(videoData, videoProgress, autoplay = false) {
   let videoId = videoData.data.youtube_id;
   let videoUrl = videoData.data.media_url;
   let videoThumbUrl = videoData.data.vid_thumb_url;
@@ -546,7 +677,9 @@ function createVideoTag(videoData, videoProgress) {
   }
 
   let videoTag = `
-    <video poster="${videoThumbUrl}" onvolumechange="onVolumeChange(this)" onloadstart="this.volume=getPlayerVolume()" ontimeupdate="onVideoProgress()" onpause="onVideoPause()" onended="onVideoEnded()" controls autoplay width="100%" playsinline id="video-item">
+    <video poster="${videoThumbUrl}" onvolumechange="onVolumeChange(this)" onloadstart="this.volume=getPlayerVolume()" ontimeupdate="onVideoProgress()" onpause="onVideoPause()" onended="onVideoEnded()" ${
+    autoplay ? 'autoplay' : ''
+  } controls width="100%" playsinline id="video-item">
         <source src="${videoUrl}#t=${videoProgress}" type="video/mp4" id="video-source" videoid="${videoId}">
         ${subtitles}
     </video>
@@ -650,6 +783,7 @@ function onVideoProgress() {
       }
     }
   }
+  if (currentTime < 10) return;
   if ((currentTime % 10).toFixed(1) <= 0.2) {
     // Check progress every 10 seconds or else progress is checked a few times a second
     postVideoProgress(videoId, currentTime);
@@ -701,6 +835,7 @@ function watchedThreshold(currentTime, duration) {
 function onVideoPause() {
   let videoId = getVideoPlayerVideoId();
   let currentTime = getVideoPlayerCurrentTime();
+  if (currentTime < 10) return;
   postVideoProgress(videoId, currentTime);
 }
 
@@ -769,6 +904,13 @@ function getPlaylistData(playlistId) {
   return playlistData.data;
 }
 
+// Gets custom playlists
+function getCustomPlaylists() {
+  let apiEndpoint = '/api/playlist/?playlist_type=custom';
+  let playlistData = apiRequest(apiEndpoint, 'GET');
+  return playlistData.data;
+}
+
 // Get video progress data when passed video ID
 function getVideoProgress(videoId) {
   let apiEndpoint = '/api/video/' + videoId + '/progress/';
@@ -1379,8 +1521,10 @@ function textExpandButtonVisibilityUpdate() {
 document.addEventListener('readystatechange', textExpandButtonVisibilityUpdate);
 window.addEventListener('resize', textExpandButtonVisibilityUpdate);
 
-function showForm() {
-  let formElement = document.getElementById('hidden-form');
+function showForm(id) {
+  
+  let id2 = id === undefined ? 'hidden-form' : id;
+  let formElement = document.getElementById(id2);
   let displayStyle = formElement.style.display;
   if (displayStyle === '') {
     formElement.style.display = 'block';

tubearchivist/static/stats.js

@@ -5,29 +5,333 @@
 /* globals apiRequest */
 
 function primaryStats() {
-  let apiEndpoint = '/api/stats/primary/';
+  let apiVideoEndpoint = '/api/stats/video/';
+  let responseData = apiRequest(apiVideoEndpoint, 'GET');
+
+  let activeBox = document.getElementById('activeBox');
+  clearLoading(activeBox);
+
+  let totalTile = buildTotalVideoTile(responseData);
+  activeBox.appendChild(totalTile);
+  let activeTile = buildActiveVideoTile(responseData);
+  activeBox.appendChild(activeTile);
+  let inActiveTile = buildInActiveVideoTile(responseData);
+  activeBox.appendChild(inActiveTile);
+
+  let videoTypeBox = document.getElementById('videoTypeBox');
+  clearLoading(videoTypeBox);
+
+  let videosTypeTile = buildVideosTypeTile(responseData);
+  videoTypeBox.appendChild(videosTypeTile);
+  let shortsTypeTile = buildShortsTypeTile(responseData);
+  videoTypeBox.appendChild(shortsTypeTile);
+  let streamsTypeTile = buildStreamsTypeTile(responseData);
+  videoTypeBox.appendChild(streamsTypeTile);
+}
+
+function secondaryStats() {
+  let apiChannelEndpoint = '/api/stats/channel/';
+  let channelResponseData = apiRequest(apiChannelEndpoint, 'GET');
+  let secondaryBox = document.getElementById('secondaryBox');
+  clearLoading(secondaryBox);
+  let channelTile = buildChannelTile(channelResponseData);
+  secondaryBox.appendChild(channelTile);
+
+  let apiPlaylistEndpoint = '/api/stats/playlist/';
+  let playlistResponseData = apiRequest(apiPlaylistEndpoint, 'GET');
+  let playlistTile = buildPlaylistTile(playlistResponseData);
+  secondaryBox.appendChild(playlistTile);
+
+  let apiDownloadEndpoint = '/api/stats/download/';
+  let downloadResponseData = apiRequest(apiDownloadEndpoint, 'GET');
+  let downloadTile = buildDownloadTile(downloadResponseData);
+  secondaryBox.appendChild(downloadTile);
+}
+
+function buildTotalVideoTile(responseData) {
+  const totalCount = responseData.doc_count || 0;
+  const totalSize = humanFileSize(responseData.media_size || 0);
+  const content = {
+    Videos: `${totalCount}`,
+    'Media Size': `${totalSize}`,
+    Duration: responseData.duration_str,
+  };
+  const tile = buildTile('All: ');
+  const table = buildTileContenTable(content, 2);
+  tile.appendChild(table);
+  return tile;
+}
+
+function buildActiveVideoTile(responseData) {
+  const activeCount = responseData?.active_true?.doc_count || 0;
+  const activeSize = humanFileSize(responseData?.active_true?.media_size || 0);
+  const duration = responseData?.active_true?.duration_str || 'NA';
+  const content = {
+    Videos: `${activeCount}`,
+    'Media Size': `${activeSize}`,
+    Duration: duration,
+  };
+  const tile = buildTile('Active: ');
+  const table = buildTileContenTable(content, 2);
+  tile.appendChild(table);
+  return tile;
+}
+
+function buildInActiveVideoTile(responseData) {
+  const inActiveCount = responseData?.active_false?.doc_count || 0;
+  const inActiveSize = humanFileSize(responseData?.active_false?.media_size || 0);
+  const duration = responseData?.active_false?.duration_str || 'NA';
+  const content = {
+    Videos: `${inActiveCount}`,
+    'Media Size': `${inActiveSize}`,
+    Duration: duration,
+  };
+  const tile = buildTile('Inactive: ');
+  const table = buildTileContenTable(content, 2);
+  tile.appendChild(table);
+  return tile;
+}
+
+function buildVideosTypeTile(responseData) {
+  const videosCount = responseData?.type_videos?.doc_count || 0;
+  const videosSize = humanFileSize(responseData?.type_videos?.media_size || 0);
+  const duration = responseData?.type_videos?.duration_str || 'NA';
+  const content = {
+    Videos: `${videosCount}`,
+    'Media Size': `${videosSize}`,
+    Duration: duration,
+  };
+  const tile = buildTile('Regular Videos: ');
+  const table = buildTileContenTable(content, 2);
+  tile.appendChild(table);
+  return tile;
+}
+
+function buildShortsTypeTile(responseData) {
+  const shortsCount = responseData?.type_shorts?.doc_count || 0;
+  const shortsSize = humanFileSize(responseData?.type_shorts?.media_size || 0);
+  const duration = responseData?.type_shorts?.duration_str || 'NA';
+  const content = {
+    Videos: `${shortsCount}`,
+    'Media Size': `${shortsSize}`,
+    Duration: duration,
+  };
+  const tile = buildTile('Shorts: ');
+  const table = buildTileContenTable(content, 2);
+  tile.appendChild(table);
+  return tile;
+}
+
+function buildStreamsTypeTile(responseData) {
+  const streamsCount = responseData?.type_streams?.doc_count || 0;
+  const streamsSize = humanFileSize(responseData?.type_streams?.media_size || 0);
+  const duration = responseData?.type_streams?.duration_str || 'NA';
+  const content = {
+    Videos: `${streamsCount}`,
+    'Media Size': `${streamsSize}`,
+    Duration: duration,
+  };
+  const tile = buildTile('Streams: ');
+  const table = buildTileContenTable(content, 2);
+  tile.appendChild(table);
+  return tile;
+}
+
+function buildChannelTile(responseData) {
+  let tile = buildTile('Channels: ');
+  const total = responseData.doc_count || 0;
+  const subscribed = responseData.subscribed_true || 0;
+  const active = responseData.active_true || 0;
+  const content = {
+    Subscribed: subscribed,
+    Active: active,
+    Total: total,
+  };
+  const table = buildTileContenTable(content, 3);
+  tile.appendChild(table);
+
+  return tile;
+}
+
+function buildPlaylistTile(responseData) {
+  let tile = buildTile('Playlists: ');
+  const total = responseData.doc_count || 0;
+  const subscribed = responseData.subscribed_true || 0;
+  const active = responseData.active_true || 0;
+  const content = {
+    Subscribed: subscribed,
+    Active: active,
+    Total: total,
+  };
+  const table = buildTileContenTable(content, 2);
+  tile.appendChild(table);
+
+  return tile;
+}
+
+function buildDownloadTile(responseData) {
+  const pendingTotal = responseData.pending || 0;
+  let tile = buildTile(`Downloads Pending: ${pendingTotal}`);
+  const pendingVideos = responseData.pending_videos || 0;
+  const pendingShorts = responseData.pending_shorts || 0;
+  const pendingStreams = responseData.pending_streams || 0;
+  const content = {
+    Videos: pendingVideos,
+    Shorts: pendingShorts,
+    Streams: pendingStreams,
+  };
+  const table = buildTileContenTable(content, 3);
+  tile.appendChild(table);
+
+  return tile;
+}
+
+function watchStats() {
+  let apiEndpoint = '/api/stats/watch/';
   let responseData = apiRequest(apiEndpoint, 'GET');
-  let primaryBox = document.getElementById('primaryBox');
+  let watchBox = document.getElementById('watchBox');
+  clearLoading(watchBox);
 
-  clearLoading(primaryBox);
+  let watchedTile = buildWatchTile('watched', responseData.watched);
+  watchBox.appendChild(watchedTile);
 
-  let videoTile = buildVideoTile(responseData);
-  primaryBox.appendChild(videoTile);
+  let unwatchedTile = buildWatchTile('unwatched', responseData.unwatched);
+  watchBox.appendChild(unwatchedTile);
+}
 
-  let channelTile = buildChannelTile(responseData);
-  primaryBox.appendChild(channelTile);
+function buildWatchTile(title, watchDetail) {
+  const items = watchDetail?.items ?? 0;
+  const duration = watchDetail?.duration ?? 0;
+  const duration_str = watchDetail?.duration_str ?? '0s';
+  const hasProgess = !!watchDetail?.progress;
+  const progress = (Number(watchDetail?.progress) * 100).toFixed(2) ?? '0';
 
-  let playlistTile = buildPlaylistTile(responseData);
-  primaryBox.appendChild(playlistTile);
+  let titleCapizalized = capitalizeFirstLetter(title);
 
-  let downloadTile = buildDownloadTile(responseData);
-  primaryBox.appendChild(downloadTile);
+  if (hasProgess) {
+    titleCapizalized = `${progress}% ` + titleCapizalized;
+  }
+
+  let tile = buildTile(titleCapizalized);
+
+  const content = {
+    Videos: items,
+    Seconds: duration,
+    Duration: duration_str,
+  };
+
+  const table = buildTileContenTable(content, 3);
+
+  tile.appendChild(table);
+
+  return tile;
+}
+
+function downloadHist() {
+  let apiEndpoint = '/api/stats/downloadhist/';
+  let responseData = apiRequest(apiEndpoint, 'GET');
+  let histBox = document.getElementById('downHistBox');
+  clearLoading(histBox);
+  if (responseData.length === 0) {
+    let tile = buildTile('No recent downloads');
+    histBox.appendChild(tile);
+    return;
+  }
+
+  for (let i = 0; i < responseData.length; i++) {
+    const dailyStat = responseData[i];
+    let tile = buildDailyStat(dailyStat);
+    histBox.appendChild(tile);
+  }
+}
+
+function buildDailyStat(dailyStat) {
+  let tile = buildTile(dailyStat.date);
+  let message = document.createElement('p');
+  const isExactlyOne = dailyStat.count === 1;
+
+  let text = 'Videos';
+  if (isExactlyOne) {
+    text = 'Video';
+  }
+
+  message.innerText = `+${dailyStat.count} ${text}
+  ${humanFileSize(dailyStat.media_size)}`;
+
+  tile.appendChild(message);
+  return tile;
+}
+
+function buildChannelRow(id, name, value) {
+  let tableRow = document.createElement('tr');
+
+  tableRow.innerHTML = `
+    <td class="agg-channel-name"><a href="/channel/${id}/">${name}</a></td>
+    <td class="agg-channel-right-align">${value}</td>
+  `;
+
+  return tableRow;
+}
+
+function addBiggestChannelByDocCount() {
+  let tBody = document.getElementById('biggestChannelTableVideos');
+
+  let apiEndpoint = '/api/stats/biggestchannels/?order=doc_count';
+  const responseData = apiRequest(apiEndpoint, 'GET');
+
+  for (let i = 0; i < responseData.length; i++) {
+    const { id, name, doc_count } = responseData[i];
+
+    let tableRow = buildChannelRow(id, name, doc_count);
+
+    tBody.appendChild(tableRow);
+  }
+}
+
+function addBiggestChannelByDuration() {
+  const tBody = document.getElementById('biggestChannelTableDuration');
+
+  let apiEndpoint = '/api/stats/biggestchannels/?order=duration';
+  const responseData = apiRequest(apiEndpoint, 'GET');
+
+  for (let i = 0; i < responseData.length; i++) {
+    const { id, name, duration_str } = responseData[i];
+
+    let tableRow = buildChannelRow(id, name, duration_str);
+
+    tBody.appendChild(tableRow);
+  }
+}
+
+function addBiggestChannelByMediaSize() {
+  let tBody = document.getElementById('biggestChannelTableMediaSize');
+
+  let apiEndpoint = '/api/stats/biggestchannels/?order=media_size';
+  const responseData = apiRequest(apiEndpoint, 'GET');
+
+  for (let i = 0; i < responseData.length; i++) {
+    const { id, name, media_size } = responseData[i];
+
+    let tableRow = buildChannelRow(id, name, humanFileSize(media_size));
+
+    tBody.appendChild(tableRow);
+  }
 }
 
 function clearLoading(dashBox) {
   dashBox.querySelector('#loading').remove();
 }
 
+function capitalizeFirstLetter(string) {
+  // source: https://stackoverflow.com/a/1026087
+  return string.charAt(0).toUpperCase() + string.slice(1);
+}
+
+function humanFileSize(size) {
+  let i = size === 0 ? 0 : Math.floor(Math.log(size) / Math.log(1024));
+  return (size / Math.pow(1024, i)).toFixed(1) * 1 + ' ' + ['B', 'kB', 'MB', 'GB', 'TB'][i];
+}
+
 function buildTile(titleText) {
   let tile = document.createElement('div');
   tile.classList.add('info-box-item');
@@ -89,226 +393,6 @@ function buildTileContenTable(content, rowsWanted) {
   return table;
 }
 
-function buildVideoTile(responseData) {
-  let tile = buildTile(`Video types: `);
-
-  const total = responseData.videos.total || 0;
-  const videos = responseData.videos.videos || 0;
-  const shorts = responseData.videos.shorts || 0;
-  const streams = responseData.videos.streams || 0;
-
-  const content = {
-    Videos: `${videos}/${total}`,
-    Shorts: `${shorts}/${total}`,
-    Streams: `${streams}/${total}`,
-  };
-
-  const table = buildTileContenTable(content, 3);
-
-  tile.appendChild(table);
-
-  return tile;
-}
-
-function buildChannelTile(responseData) {
-  let tile = buildTile(`Channels: `);
-
-  const total = responseData.channels.total || 0;
-  const subscribed = responseData.channels.sub_true || 0;
-
-  const content = {
-    Subscribed: `${subscribed}/${total}`,
-  };
-
-  const table = buildTileContenTable(content, 3);
-
-  tile.appendChild(table);
-
-  return tile;
-}
-
-function buildPlaylistTile(responseData) {
-  let tile = buildTile(`Playlists:`);
-
-  const total = responseData.playlists.total || 0;
-  const subscribed = responseData.playlists.sub_true || 0;
-
-  const content = {
-    Subscribed: `${subscribed}/${total}`,
-  };
-
-  const table = buildTileContenTable(content, 3);
-
-  tile.appendChild(table);
-
-  return tile;
-}
-
-function buildDownloadTile(responseData) {
-  let tile = buildTile('Downloads');
-
-  const pending = responseData.downloads.pending || 0;
-  const ignored = responseData.downloads.ignore || 0;
-
-  const content = {
-    Pending: pending,
-    Ignored: ignored,
-  };
-
-  const table = buildTileContenTable(content, 3);
-
-  tile.appendChild(table);
-
-  return tile;
-}
-
-function watchStats() {
-  let apiEndpoint = '/api/stats/watch/';
-  let responseData = apiRequest(apiEndpoint, 'GET');
-  let watchBox = document.getElementById('watchBox');
-  clearLoading(watchBox);
-
-  const { total, watched, unwatched } = responseData;
-
-  let firstCard = buildWatchTile('total', total);
-  watchBox.appendChild(firstCard);
-
-  let secondCard = buildWatchTile('watched', watched);
-  watchBox.appendChild(secondCard);
-
-  let thirdCard = buildWatchTile('unwatched', unwatched);
-  watchBox.appendChild(thirdCard);
-}
-
-function capitalizeFirstLetter(string) {
-  // source: https://stackoverflow.com/a/1026087
-  return string.charAt(0).toUpperCase() + string.slice(1);
-}
-
-function buildWatchTile(title, watchDetail) {
-  const items = watchDetail?.items ?? 0;
-  const duration = watchDetail?.duration ?? 0;
-  const duration_str = watchDetail?.duration_str ?? '0s';
-  const hasProgess = !!watchDetail?.progress;
-  const progress = (Number(watchDetail?.progress) * 100).toFixed(2) ?? '0';
-
-  let titleCapizalized = capitalizeFirstLetter(title);
-
-  if (hasProgess) {
-    titleCapizalized = `${progress}% ` + titleCapizalized;
-  }
-
-  let tile = buildTile(titleCapizalized);
-
-  const content = {
-    Videos: items,
-    Seconds: duration,
-    Playback: duration_str,
-  };
-
-  const table = buildTileContenTable(content, 3);
-
-  tile.appendChild(table);
-
-  return tile;
-}
-
-function downloadHist() {
-  let apiEndpoint = '/api/stats/downloadhist/';
-  let responseData = apiRequest(apiEndpoint, 'GET');
-  let histBox = document.getElementById('downHistBox');
-  clearLoading(histBox);
-  if (responseData.length === 0) {
-    let tile = buildTile('No recent downloads');
-    histBox.appendChild(tile);
-    return;
-  }
-
-  for (let i = 0; i < responseData.length; i++) {
-    const dailyStat = responseData[i];
-    let tile = buildDailyStat(dailyStat);
-    histBox.appendChild(tile);
-  }
-}
-
-function buildDailyStat(dailyStat) {
-  let tile = buildTile(dailyStat.date);
-  let message = document.createElement('p');
-  const isExactlyOne = dailyStat.count === 1;
-
-  let text = 'Videos';
-  if (isExactlyOne) {
-    text = 'Video';
-  }
-
-  message.innerText = `+${dailyStat.count} ${text}
-  ${humanFileSize(dailyStat.media_size)}`;
-
-  tile.appendChild(message);
-  return tile;
-}
-
-function humanFileSize(size) {
-  let i = size === 0 ? 0 : Math.floor(Math.log(size) / Math.log(1024));
-  return (size / Math.pow(1024, i)).toFixed(1) * 1 + ' ' + ['B', 'kB', 'MB', 'GB', 'TB'][i];
-}
-
-function buildChannelRow(id, name, value) {
-  let tableRow = document.createElement('tr');
-
-  tableRow.innerHTML = `
-    <td class="agg-channel-name"><a href="/channel/${id}/">${name}</a></td>
-    <td class="agg-channel-right-align">${value}</td>
-  `;
-
-  return tableRow;
-}
-
-function addBiggestChannelByDocCount() {
-  let tBody = document.getElementById('biggestChannelTableVideos');
-
-  let apiEndpoint = '/api/stats/biggestchannels/?order=doc_count';
-  const responseData = apiRequest(apiEndpoint, 'GET');
-
-  for (let i = 0; i < responseData.length; i++) {
-    const { id, name, doc_count } = responseData[i];
-
-    let tableRow = buildChannelRow(id, name, doc_count);
-
-    tBody.appendChild(tableRow);
-  }
-}
-
-function addBiggestChannelByDuration() {
-  const tBody = document.getElementById('biggestChannelTableDuration');
-
-  let apiEndpoint = '/api/stats/biggestchannels/?order=duration';
-  const responseData = apiRequest(apiEndpoint, 'GET');
-
-  for (let i = 0; i < responseData.length; i++) {
-    const { id, name, duration_str } = responseData[i];
-
-    let tableRow = buildChannelRow(id, name, duration_str);
-
-    tBody.appendChild(tableRow);
-  }
-}
-
-function addBiggestChannelByMediaSize() {
-  let tBody = document.getElementById('biggestChannelTableMediaSize');
-
-  let apiEndpoint = '/api/stats/biggestchannels/?order=media_size';
-  const responseData = apiRequest(apiEndpoint, 'GET');
-
-  for (let i = 0; i < responseData.length; i++) {
-    const { id, name, media_size } = responseData[i];
-
-    let tableRow = buildChannelRow(id, name, humanFileSize(media_size));
-
-    tBody.appendChild(tableRow);
-  }
-}
-
 function biggestChannel() {
   addBiggestChannelByDocCount();
   addBiggestChannelByDuration();
@@ -317,6 +401,7 @@ function biggestChannel() {
 
 async function buildStats() {
   primaryStats();
+  secondaryStats();
   watchStats();
   downloadHist();
   biggestChannel();