bumping versions for new release

CONTRIBUTING.md

@@ -8,6 +8,14 @@ If you haven't already, the best place to start is the README. This will give yo
 If you notice something is not working as expected, check to see if it has been previously reported in the [open issues](https://github.com/bbilly1/tubearchivist/issues).
 If it has not yet been disclosed, go ahead and create an issue.
 
+## Wiki
+
+WIP: The wiki is where all user functions are explained in detail. These pages are mirrored into the **docs** folder of the repo. This allows for pull requests and all other features like regular code. Make any changes there, and I'll sync them  with the wiki tab.
+
+## Implementing a new feature
+
+Do you see anything on the roadmap that you would like to take a closer look at but you are not sure, what's the best way to tackle that? Or anything not on there yet you'd like to implement but are not sure how? Open up an issue and we try to find a solution together.
+
 ## Making changes
 
 To fix a bug or implement a feature, fork the repository and make all changes to the testing branch. When ready, create a pull request.

README.md

@@ -2,6 +2,20 @@
 
 <center><h1>Your self hosted YouTube media server</h1></center>
 
+## Table of contents:
+* [Core functionality](#core-functionality)
+* [Screenshots](#screenshots)
+* [Problem Tube Archivist tries to solve](#problem-tube-archivist-tries-to-solve)
+* [Installing and updating](#installing-and-updating)
+* [Getting Started](#getting-started)
+* [Import your existing library](#import-your-existing-library)
+* [Backup and restore](#backup-and-restore)
+* [Potential pitfalls](#potential-pitfalls)
+* [Roadmap](#roadmap)
+* [Known limitations](#known-limitations)
+* [Donate](#donate)
+
+------------------------
 
 ## Core functionality
 * Subscribe to your favorite YouTube channels
@@ -29,7 +43,7 @@
 ## Problem Tube Archivist tries to solve
 Once your YouTube video collection grows, it becomes hard to search and find a specific video. That's where Tube Archivist comes in: By indexing your video collection with metadata from YouTube, you can organize, search and enjoy your archived YouTube videos without hassle offline through a convenient web interface.
 
-## Installation
+## Installing and updating
 Take a look at the example `docker-compose.yml` file provided. Tube Archivist depends on three main components split up into separate docker containers:  
 
 ### Tube Archivist
@@ -52,6 +66,33 @@ Functions as a cache and temporary link between the application and the file sys
   - Needs to be accessible over the default port `6379`
   - Takes an optional volume at **/data** to make your configuration changes permanent.
 
+### Updating Tube Archivist
+You will see the current version number of **Tube Archivist** in the footer of the interface so you can compare it with the latest release to make sure you are running the *latest and greatest*.  
+* There can be breaking changes between updates, particularly as the application grows, new environment variables or settings might be required for you to set in the your docker-compose file. Any breaking changes will be marked in the **release notes**.  
+* All testing and development is done with the Elasticsearch version number as mentioned in the provided *docker-compose.yml* file. This will be updated when a new release of Elasticsearch is available. Running an older version of Elasticsearch is most likely not going to result in any issues, but it's still recommended to run the same version as mentioned.
+
+## Potential pitfalls
+### vm.max_map_count
+**Elastic Search** in Docker requires the kernel setting of the host machine `vm.max_map_count` to be set to at least 262144.
+
+To temporary set the value run:  
+```
+sudo sysctl -w vm.max_map_count=262144
+```  
+
+To apply the change permanently depends on your host operating system:  
+- For example on Ubuntu Server add `vm.max_map_count = 262144` to the file */etc/sysctl.conf*.
+- On Arch based systems create a file */etc/sysctl.d/max_map_count.conf* with the content `vm.max_map_count = 262144`. 
+- On any other platform look up in the documentation on how to pass kernel parameters.
+
+### Permissions for elasticsearch
+If you see a message similar to `AccessDeniedException[/usr/share/elasticsearch/data/nodes]` when initially starting elasticsearch, that means the container is not allowed to write files to the volume.  
+That's most likely the case when you run `docker-compose` as an unprivileged user. To fix that issue, shutdown the container and on your host machine run:
+```
+chown 1000:0 /path/to/mount/point
+```
+This will match the permissions with the **UID** and **GID** of elasticsearch within the container and should fix the issue.
+
 ## Getting Started
 1. Go through the **settings** page and look at the available options. Particularly set *Download Format* to your desired video quality before downloading. **Tube Archivist** downloads the best available quality by default.
 2. Subscribe to some of your favorite YouTube channels on the **channels** page. 
@@ -87,28 +128,6 @@ The restore functionality will expect the same zip file in *cache/backup* and wi
 
 BE AWARE: This will **replace** your current index with the one from the backup file.
 
-## Potential pitfalls
-### vm.max_map_count
-**Elastic Search** in Docker requires the kernel setting of the host machine `vm.max_map_count` to be set to at least 262144.
-
-To temporary set the value run:  
-```
-sudo sysctl -w vm.max_map_count=262144
-```  
-
-To apply the change permanently depends on your host operating system:  
-- For example on Ubuntu Server add `vm.max_map_count = 262144` to the file */etc/sysctl.conf*.
-- On Arch based systems create a file */etc/sysctl.d/max_map_count.conf* with the content `vm.max_map_count = 262144`. 
-- On any other platform look up in the documentation on how to pass kernel parameters.
-
-### Permissions for elasticsearch
-If you see a message similar to `AccessDeniedException[/usr/share/elasticsearch/data/nodes]` when initially starting elasticsearch, that means the container is not allowed to write files to the volume.  
-That's most likely the case when you run `docker-compose` as an unprivileged user. To fix that issue, shutdown the container and on your host machine run:
-```
-chown 1000:0 /path/to/mount/point
-```
-This will match the permissions with the **UID** and **GID** of elasticsearch within the container and should fix the issue.
-
 ## Roadmap
 This should be considered as a **minimal viable product**, there is an extensive list of future functions and improvements planned.
 
@@ -136,3 +155,11 @@ This should be considered as a **minimal viable product**, there is an extensive
 - Video files created by Tube Archivist need to be **mp4** video files for best browser compatibility.
 - Every limitation of **yt-dlp** will also be present in Tube Archivist. If **yt-dlp** can't download or extract a video for any reason, Tube Archivist won't be able to either.
 - For now this is meant to be run in a trusted network environment.
+
+
+## Donate
+The best donation to **Tube Archivist** is your time, take a look at the [contribution page](CONTRIBUTING) to get started.  
+Second best way to support the development is to provide for caffeinated beverages:
+* [Paypal.me](https://paypal.me/bbilly1) for a one time coffee
+* [Paypal Subscription](https://www.paypal.com/webapps/billing/plans/subscribe?plan_id=P-03770005GR991451KMFGVPMQ) for a monthly coffee
+* [co-fi.com](https://ko-fi.com/bbilly1) for an alternative platform

deploy.sh

@@ -108,7 +108,8 @@ function sync_docker {
     printf "\nlatest images:\n"
     sudo docker image ls bbilly1/tubearchivist
 
-    read -s "Push?"
+    echo "continue?"
+    read -rn 1
 
     # push to docker
     echo "pushing latest:"

docker-compose.yml

@@ -29,7 +29,7 @@ services:
     depends_on:
       - archivist-es
   archivist-es:
-    image: docker.elastic.co/elasticsearch/elasticsearch:7.14.1
+    image: docker.elastic.co/elasticsearch/elasticsearch:7.15.0
     container_name: archivist-es
     restart: always
     environment:

docs/Channels.md

@@ -0,0 +1 @@
+# Channels Overview and Channel Detail Page

docs/Downloads.md

@@ -0,0 +1 @@
+# Downloads Page Functionality

docs/Home.md

@@ -0,0 +1,10 @@
+# Tube Archivist Wiki
+*Documentation of user functionality*  
+
+**WIP**: This is work in progress!
+
+Table of contents:
+* [Main](Main): Tube Archivist landing page
+* [Channels](Channels): Browse your channels, handle subscriptions
+* [Downloads](Downloads): Scanning subscriptions, handle download queue
+* [Settings](Settings): All the configuration options

docs/Main.md

@@ -0,0 +1 @@
+# Tube Archivist Home Page Functionality

docs/Settings.md

@@ -0,0 +1 @@
+# Settings Page Functionality

tubearchivist/home/config.json

@@ -11,7 +11,7 @@
         "channel_size": 50
     },
     "downloads": {
-        "limit_count": 5,
+        "limit_count": false,
         "limit_speed": false,
         "sleep_interval": 3,
         "format": false,

tubearchivist/home/src/download.py

@@ -14,7 +14,13 @@ from time import sleep
 import requests
 import yt_dlp as youtube_dl
 from home.src.config import AppConfig
-from home.src.helper import DurationConverter, clean_string, set_message
+from home.src.helper import (
+    DurationConverter,
+    RedisQueue,
+    clean_string,
+    ignore_filelist,
+    set_message,
+)
 from home.src.index import YoutubeChannel, index_new_video
 
 
@@ -147,7 +153,7 @@ class PendingList:
             "size": 50,
             "query": {"match_all": {}},
             "pit": {"id": pit_id, "keep_alive": "1m"},
-            "sort": [{"timestamp": {"order": "desc"}}],
+            "sort": [{"timestamp": {"order": "asc"}}],
         }
         query_str = json.dumps(data)
         url = self.ES_URL + "/_search"
@@ -214,11 +220,13 @@ class PendingList:
 
     def get_all_downloaded(self):
         """get a list of all videos in archive"""
-        all_channel_folders = os.listdir(self.VIDEOS)
+        channel_folders = os.listdir(self.VIDEOS)
+        all_channel_folders = ignore_filelist(channel_folders)
         all_downloaded = []
         for channel_folder in all_channel_folders:
             channel_path = os.path.join(self.VIDEOS, channel_folder)
-            all_videos = os.listdir(channel_path)
+            videos = os.listdir(channel_path)
+            all_videos = ignore_filelist(videos)
             youtube_vids = [i[9:20] for i in all_videos]
             for youtube_id in youtube_vids:
                 all_downloaded.append(youtube_id)
@@ -400,19 +408,28 @@ def playlist_extractor(playlist_id):
 
 
 class VideoDownloader:
-    """handle the video download functionality"""
+    """
+    handle the video download functionality
+    if not initiated with list, take from queue
+    """
 
-    def __init__(self, youtube_id_list):
+    def __init__(self, youtube_id_list=False):
         self.youtube_id_list = youtube_id_list
         self.config = AppConfig().config
 
-    def download_list(self):
-        """download the list of youtube_ids"""
-        limit_count = self.config["downloads"]["limit_count"]
-        if limit_count:
-            self.youtube_id_list = self.youtube_id_list[:limit_count]
+    def run_queue(self):
+        """setup download queue in redis loop until no more items"""
+        queue = RedisQueue("dl_queue")
+
+        limit_queue = self.config["downloads"]["limit_count"]
+        if limit_queue:
+            queue.trim(limit_queue - 1)
+
+        while True:
+            youtube_id = queue.get_next()
+            if not youtube_id:
+                break
 
-        for youtube_id in self.youtube_id_list:
             try:
                 self.dl_single_vid(youtube_id)
             except youtube_dl.utils.DownloadError:
@@ -421,8 +438,14 @@ class VideoDownloader:
             vid_dict = index_new_video(youtube_id)
             self.move_to_archive(vid_dict)
             self.delete_from_pending(youtube_id)
-            if self.config["downloads"]["sleep_interval"]:
-                sleep(self.config["downloads"]["sleep_interval"])
+
+    @staticmethod
+    def add_pending():
+        """add pending videos to download queue"""
+        all_pending, _ = PendingList().get_all_pending()
+        to_add = [i["youtube_id"] for i in all_pending]
+        queue = RedisQueue("dl_queue")
+        queue.add_list(to_add)
 
     @staticmethod
     def progress_hook(response):
@@ -486,7 +509,8 @@ class VideoDownloader:
 
         # check if already in cache to continue from there
         cache_dir = self.config["application"]["cache_dir"]
-        all_cached = os.listdir(cache_dir + "/download/")
+        cached = os.listdir(cache_dir + "/download/")
+        all_cached = ignore_filelist(cached)
         for file_name in all_cached:
             if youtube_id in file_name:
                 obs["outtmpl"] = cache_dir + "/download/" + file_name
@@ -511,7 +535,9 @@ class VideoDownloader:
         os.makedirs(new_folder, exist_ok=True)
         # find real filename
         cache_dir = self.config["application"]["cache_dir"]
-        for file_str in os.listdir(cache_dir + "/download"):
+        cached = os.listdir(cache_dir + "/download/")
+        all_cached = ignore_filelist(cached)
+        for file_str in all_cached:
             if youtube_id in file_str:
                 old_file = file_str
         old_file_path = os.path.join(cache_dir, "download", old_file)

tubearchivist/home/src/helper.py

@@ -40,12 +40,28 @@ def clean_string(file_name):
     return cleaned
 
 
+def ignore_filelist(filelist):
+    """ignore temp files for os.listdir sanitizer"""
+    to_ignore = ["Icon\r\r", "Temporary Items", "Network Trash Folder"]
+    cleaned = []
+    for file_name in filelist:
+        if file_name.startswith(".") or file_name in to_ignore:
+            continue
+
+        cleaned.append(file_name)
+
+    return cleaned
+
+
 def process_url_list(url_str):
     """parse url_list to find valid youtube video or channel ids"""
     to_replace = ["watch?v=", "playlist?list="]
     url_list = re.split("\n+", url_str[0])
     youtube_ids = []
     for url in url_list:
+        if "/c/" in url or "/user/" in url:
+            raise ValueError("user name is not unique, use channel ID")
+
         url_clean = url.strip().strip("/").split("/")[-1]
         for i in to_replace:
             url_clean = url_clean.replace(i, "")
@@ -85,6 +101,12 @@ def get_message(key):
     return json_str
 
 
+def del_message(key):
+    """delete key from redis"""
+    redis_connection = redis.Redis(host=REDIS_HOST)
+    redis_connection.execute_command("DEL", key)
+
+
 def get_dl_message(cache_dir):
     """get latest message if available"""
     redis_connection = redis.Redis(host=REDIS_HOST)
@@ -109,7 +131,8 @@ def monitor_cache_dir(cache_dir):
     look at download cache dir directly as alternative progress info
     """
     dl_cache = os.path.join(cache_dir, "download")
-    cache_file = os.listdir(dl_cache)
+    all_cache_file = os.listdir(dl_cache)
+    cache_file = ignore_filelist(all_cache_file)
     if cache_file:
         filename = cache_file[0][12:].replace("_", " ").split(".")[0]
         mess_dict = {
@@ -124,6 +147,50 @@ def monitor_cache_dir(cache_dir):
     return mess_dict
 
 
+class RedisQueue:
+    """dynamically interact with the download queue in redis"""
+
+    def __init__(self, key):
+        self.key = key
+        self.conn = redis.Redis(host=REDIS_HOST)
+
+    def get_all(self):
+        """return all elements in list"""
+        result = self.conn.execute_command("LRANGE", self.key, 0, -1)
+        all_elements = [i.decode() for i in result]
+        return all_elements
+
+    def add_list(self, to_add):
+        """add list to queue"""
+        self.conn.execute_command("RPUSH", self.key, *to_add)
+
+    def add_priority(self, to_add):
+        """add single video to front of queue"""
+        self.clear_item(to_add)
+        self.conn.execute_command("LPUSH", self.key, to_add)
+
+    def get_next(self):
+        """return next element in the queue, False if none"""
+        result = self.conn.execute_command("LPOP", self.key)
+        if not result:
+            return False
+
+        next_element = result.decode()
+        return next_element
+
+    def clear(self):
+        """delete list from redis"""
+        self.conn.execute_command("DEL", self.key)
+
+    def clear_item(self, to_clear):
+        """remove single item from list if it's there"""
+        self.conn.execute_command("LREM", self.key, 0, to_clear)
+
+    def trim(self, size):
+        """trim the queue based on settings amount"""
+        self.conn.execute_command("LTRIM", self.key, 0, size)
+
+
 class DurationConverter:
     """
     using ffmpeg to get and parse duration from filepath

tubearchivist/home/src/index_management.py

@@ -13,6 +13,7 @@ from datetime import datetime
 
 import requests
 from home.src.config import AppConfig
+from home.src.helper import ignore_filelist
 
 # expected mapping and settings
 INDEX_CONFIG = [
@@ -433,9 +434,11 @@ class ElasticBackup:
         """extract backup zip and return filelist"""
         cache_dir = self.config["application"]["cache_dir"]
         backup_dir = os.path.join(cache_dir, "backup")
+        backup_files = os.listdir(backup_dir)
+        all_backup_files = ignore_filelist(backup_files)
         all_available_backups = [
             i
-            for i in os.listdir(backup_dir)
+            for i in all_backup_files
             if i.startswith("ta_") and i.endswith(".zip")
         ]
         all_available_backups.sort()

tubearchivist/home/src/reindex.py

@@ -21,6 +21,7 @@ from home.src.helper import (
     clean_string,
     get_message,
     get_total_hits,
+    ignore_filelist,
     set_message,
 )
 from home.src.index import YoutubeChannel, YoutubeVideo, index_new_video
@@ -209,12 +210,15 @@ class FilesystemScanner:
 
     def get_all_downloaded(self):
         """get a list of all video files downloaded"""
-        all_channels = os.listdir(self.VIDEOS)
+        channels = os.listdir(self.VIDEOS)
+        all_channels = ignore_filelist(channels)
         all_channels.sort()
         all_downloaded = []
         for channel_name in all_channels:
             channel_path = os.path.join(self.VIDEOS, channel_name)
-            for video in os.listdir(channel_path):
+            videos = os.listdir(channel_path)
+            all_videos = ignore_filelist(videos)
+            for video in all_videos:
                 youtube_id = video[9:20]
                 all_downloaded.append((channel_name, video, youtube_id))
 
@@ -339,8 +343,8 @@ class ManualImport:
 
     def import_folder_parser(self):
         """detect files in import folder"""
-
-        to_import = os.listdir(self.IMPORT_DIR)
+        import_files = os.listdir(self.IMPORT_DIR)
+        to_import = ignore_filelist(import_files)
         to_import.sort()
         video_files = [i for i in to_import if not i.endswith(".json")]
 

tubearchivist/home/src/searching.py

@@ -13,6 +13,7 @@ from datetime import datetime
 
 import requests
 from home.src.config import AppConfig
+from home.src.helper import ignore_filelist
 from PIL import Image
 
 
@@ -105,7 +106,8 @@ class SearchHandler:
     def cache_dl_vids(self, all_videos):
         """video thumbs links for cache"""
         vid_cache = os.path.join(self.CACHE_DIR, "videos")
-        all_vid_cached = os.listdir(vid_cache)
+        vid_cached = os.listdir(vid_cache)
+        all_vid_cached = ignore_filelist(vid_cached)
         # videos
         for video_dict in all_videos:
             youtube_id = video_dict["youtube_id"]
@@ -124,7 +126,8 @@ class SearchHandler:
     def cache_dl_chan(self, all_channels):
         """download channel thumbs"""
         chan_cache = os.path.join(self.CACHE_DIR, "channels")
-        all_chan_cached = os.listdir(chan_cache)
+        chan_cached = os.listdir(chan_cache)
+        all_chan_cached = ignore_filelist(chan_cached)
         for channel_dict in all_channels:
             channel_id_cache = channel_dict["channel_id"]
             channel_banner_url = channel_dict["chan_banner"]

tubearchivist/home/tasks.py

@@ -9,7 +9,7 @@ import os
 from celery import Celery, shared_task
 from home.src.config import AppConfig
 from home.src.download import ChannelSubscription, PendingList, VideoDownloader
-from home.src.helper import get_lock
+from home.src.helper import RedisQueue, del_message, get_lock, set_message
 from home.src.index_management import backup_all_indexes, restore_from_backup
 from home.src.reindex import ManualImport, reindex_old_documents
 
@@ -37,20 +37,47 @@ def update_subscribed():
 @shared_task
 def download_pending():
     """download latest pending videos"""
-    pending_handler = PendingList()
-    pending_vids = pending_handler.get_all_pending()[0]
-    to_download = [i["youtube_id"] for i in pending_vids]
-    to_download.reverse()
-    if to_download:
-        download_handler = VideoDownloader(to_download)
-        download_handler.download_list()
+
+    have_lock = False
+    my_lock = get_lock("downloading")
+
+    try:
+        have_lock = my_lock.acquire(blocking=False)
+        if have_lock:
+            downloader = VideoDownloader()
+            downloader.add_pending()
+            downloader.run_queue()
+        else:
+            print("Did not acquire download lock.")
+
+    finally:
+        if have_lock:
+            my_lock.release()
 
 
 @shared_task
 def download_single(youtube_id):
     """start download single video now"""
-    download_handler = VideoDownloader([youtube_id])
-    download_handler.download_list()
+
+    queue = RedisQueue("dl_queue")
+    queue.add_priority(youtube_id)
+    print("Added to queue with priority: " + youtube_id)
+
+    # start queue if needed
+    have_lock = False
+    my_lock = get_lock("downloading")
+
+    try:
+        have_lock = my_lock.acquire(blocking=False)
+        if have_lock:
+            VideoDownloader().run_queue()
+        else:
+            print("Download queue already running.")
+
+    finally:
+        # release if only single run
+        if have_lock and not queue.get_next():
+            my_lock.release()
 
 
 @shared_task
@@ -101,3 +128,25 @@ def run_restore_backup():
     """called from settings page, dump backup to zip file"""
     restore_from_backup()
     print("index restore finished")
+
+
+def kill_dl(task_id):
+    """kill download worker task by ID"""
+    app.control.revoke(task_id, terminate=True)
+    del_message("dl_queue_id")
+    RedisQueue("dl_queue").clear()
+
+    # clear cache
+    cache_dir = os.path.join(CONFIG["application"]["cache_dir"], "download")
+    for cached in os.listdir(cache_dir):
+        to_delete = os.path.join(cache_dir, cached)
+        os.remove(to_delete)
+
+    # notify
+    mess_dict = {
+        "status": "downloading",
+        "level": "error",
+        "title": "Brutally killing download queue",
+        "message": "",
+    }
+    set_message("progress:download", mess_dict)

tubearchivist/home/templates/home/base.html

@@ -96,7 +96,7 @@
     </div>
     <div class="footer">
         <div class="boxed-content">
-            <span>© 2021 The Tube Archivist v0.0.3 | <a href="https://github.com/bbilly1/tubearchivist" target="_blank">Github</a> | <a href="https://hub.docker.com/r/bbilly1/tubearchivist" target="_blank">Docker Hub</a></span>
+            <span>© 2021 The Tube Archivist v0.0.4 | <a href="https://github.com/bbilly1/tubearchivist" target="_blank">Github</a> | <a href="https://hub.docker.com/r/bbilly1/tubearchivist" target="_blank">Docker Hub</a></span>
         </div>
     </div>
 </body>

tubearchivist/home/templates/home/downloads.html

@@ -5,6 +5,7 @@
     <h1>Downloads</h1>
 </div>
 <div id="downloadMessage"></div>
+<div id="downloadControl"></div>
 <div class="info-box info-box-3 padding-box">
     <div class="icon-text">
         <img id="rescan-icon" onclick="rescanPending()" src="{% static 'img/icon-rescan.svg' %}" alt="rescan-icon">

tubearchivist/home/views.py

@@ -15,6 +15,7 @@ from django.views import View
 from home.src.config import AppConfig
 from home.src.download import ChannelSubscription, PendingList
 from home.src.helper import (
+    RedisQueue,
     get_dl_message,
     get_message,
     process_url_list,
@@ -26,6 +27,7 @@ from home.tasks import (
     download_pending,
     download_single,
     extrac_dl,
+    kill_dl,
     run_backup,
     run_manual_import,
     run_restore_backup,
@@ -185,7 +187,7 @@ class DownloadView(View):
             "size": page_size,
             "from": page_from,
             "query": {"term": {"status": {"value": "pending"}}},
-            "sort": [{"timestamp": {"order": "desc"}}],
+            "sort": [{"timestamp": {"order": "asc"}}],
         }
         return data
 
@@ -195,16 +197,16 @@ class DownloadView(View):
         download_post = dict(request.POST)
         if "vid-url" in download_post.keys():
             url_str = download_post["vid-url"]
-            print("adding to queue")
-            youtube_ids = process_url_list(url_str)
-            if not youtube_ids:
+            try:
+                youtube_ids = process_url_list(url_str)
+            except ValueError:
                 # failed to process
-                print(url_str)
+                print(f"failed to parse: {url_str}")
                 mess_dict = {
                     "status": "downloading",
                     "level": "error",
                     "title": "Failed to extract links.",
-                    "message": "",
+                    "message": "Not a video, channel or playlist ID or URL",
                 }
                 set_message("progress:download", mess_dict)
                 return redirect("downloads")
@@ -479,6 +481,7 @@ class PostData:
             "rescan_pending": self.rescan_pending,
             "ignore": self.ignore,
             "dl_pending": self.dl_pending,
+            "queue": self.queue_handler,
             "unsubscribe": self.unsubscribe,
             "sort_order": self.sort_order,
             "hide_watched": self.hide_watched,
@@ -506,17 +509,35 @@ class PostData:
 
     def ignore(self):
         """ignore from download queue"""
-        print("ignore video")
+        id_to_ignore = self.exec_val
+        print("ignore video " + id_to_ignore)
         handler = PendingList()
-        ignore_list = self.exec_val
-        handler.ignore_from_pending([ignore_list])
+        handler.ignore_from_pending([id_to_ignore])
+        # also clear from redis queue
+        RedisQueue("dl_queue").clear_item(id_to_ignore)
         return {"success": True}
 
     @staticmethod
     def dl_pending():
         """start the download queue"""
         print("download pending")
-        download_pending.delay()
+        running = download_pending.delay()
+        task_id = running.id
+        print("set task id: " + task_id)
+        set_message("dl_queue_id", task_id, expire=False)
+        return {"success": True}
+
+    def queue_handler(self):
+        """queue controls from frontend"""
+        to_execute = self.exec_val
+        if to_execute == "stop":
+            print("stopping download queue")
+            RedisQueue("dl_queue").clear()
+        elif to_execute == "kill":
+            task_id = get_message("dl_queue_id")
+            print("brutally killing " + task_id)
+            kill_dl(task_id)
+
         return {"success": True}
 
     def unsubscribe(self):
@@ -552,7 +573,10 @@ class PostData:
         """start downloading single vid now"""
         youtube_id = self.exec_val
         print("downloading: " + youtube_id)
-        download_single.delay(youtube_id=youtube_id)
+        running = download_single.delay(youtube_id=youtube_id)
+        task_id = running.id
+        print("set task id: " + task_id)
+        set_message("dl_queue_id", task_id, expire=False)
         return {"success": True}
 
     @staticmethod

tubearchivist/requirements.txt

@@ -6,4 +6,4 @@ redis==3.5.3
 requests==2.26.0
 uWSGI==2.0.19.1
 whitenoise==5.3.0
-yt_dlp==2021.9.2
+yt_dlp==2021.9.25

tubearchivist/static/css/dark.css

@@ -7,4 +7,5 @@
     --accent-font-dark: #259485;
     --accent-font-light: #97d4c8;
     --img-filter: invert(50%) sepia(9%) saturate(2940%) hue-rotate(122deg) brightness(94%) contrast(90%);
+    --img-filter-error: invert(16%) sepia(60%) saturate(3717%) hue-rotate(349deg) brightness(86%) contrast(120%);
 }

tubearchivist/static/css/light.css

@@ -7,4 +7,5 @@
     --accent-font-dark: #259485;
     --accent-font-light: #35b399;
     --img-filter: invert(50%) sepia(9%) saturate(2940%) hue-rotate(122deg) brightness(94%) contrast(90%);
+    --img-filter-error: invert(83%) sepia(35%) saturate(1238%) hue-rotate(297deg) brightness(103%) contrast(97%);
 }

tubearchivist/static/css/style.css

@@ -458,10 +458,29 @@ button:hover {
 }
 
 .dl-desc {
-    padding-left: 15px;
+    padding: 0 15px;
     width: 75%;
 }
 
+.dl-control-icons {
+    display: flex;
+    justify-content: center;
+    padding: 10px 0;
+}
+
+.dl-control-icons img {
+    width: 30px;
+    cursor: pointer;
+    margin: 5px;
+}
+
+#stop-icon {
+    filter: var(--img-filter);
+}
+
+#kill-icon {
+    filter: var(--img-filter-error);
+}
 
 /* status message */
 .download-progress {

tubearchivist/static/img/icon-stop.svg

@@ -0,0 +1,67 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   xmlns:dc="http://purl.org/dc/elements/1.1/"
+   xmlns:cc="http://creativecommons.org/ns#"
+   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+   xmlns:svg="http://www.w3.org/2000/svg"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   width="500"
+   height="500"
+   viewBox="0 0 132.29197 132.29167"
+   version="1.1"
+   id="svg1303"
+   inkscape:version="0.92.4 (5da689c313, 2019-01-14)"
+   sodipodi:docname="Icons_stop.svg">
+  <defs
+     id="defs1297" />
+  <sodipodi:namedview
+     id="base"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageopacity="0.0"
+     inkscape:pageshadow="2"
+     inkscape:zoom="1.0105705"
+     inkscape:cx="43.182711"
+     inkscape:cy="168.09972"
+     inkscape:document-units="mm"
+     inkscape:current-layer="layer1"
+     showgrid="false"
+     units="px"
+     inkscape:window-width="1920"
+     inkscape:window-height="1017"
+     inkscape:window-x="-8"
+     inkscape:window-y="-8"
+     inkscape:window-maximized="1" />
+  <metadata
+     id="metadata1300">
+    <rdf:RDF>
+      <cc:Work
+         rdf:about="">
+        <dc:format>image/svg+xml</dc:format>
+        <dc:type
+           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+        <dc:title></dc:title>
+      </cc:Work>
+    </rdf:RDF>
+  </metadata>
+  <g
+     inkscape:label="Ebene 1"
+     inkscape:groupmode="layer"
+     id="layer1"
+     transform="translate(0,-164.70764)">
+    <rect
+       style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;paint-order:markers fill stroke"
+       id="rect836"
+       width="118.86465"
+       height="118.86465"
+       x="6.7136617"
+       y="171.42116"
+       rx="10.00003"
+       ry="10.00003" />
+  </g>
+</svg>

tubearchivist/static/progress.js

@@ -15,9 +15,13 @@ function checkMessage() {
     req.open('GET', '/downloads/progress', true);
     req.onload = function() {
         var dlProgress = req.response;
+        // var dlStatus = dlProgress['status'];
         if (dlProgress['status']) {
             buildDownloadMessage(dlProgress);
             handleInterval();
+            // if (dlStatus == 'downloading') {
+            //     buildDownloadIcons();
+            // };
         };
     };
     req.send();
@@ -70,4 +74,33 @@ function buildDownloadMessage(dlProgress) {
     message.appendChild(title);
     message.appendChild(messageText);
     box.appendChild(message);
+    if (dlStatus == 'downloading' && dlLevel != 'error') {
+        box.appendChild(buildDownloadIcons());
+    };
 };
+
+
+// add dl control icons
+function buildDownloadIcons() {
+    var iconBox = document.createElement('div');
+    iconBox.classList = 'dl-control-icons';
+    // stop icon
+    var stopIcon = document.createElement('img');
+    stopIcon.setAttribute('id', "stop-icon");
+    stopIcon.setAttribute('title', "Stop Download Queue");
+    stopIcon.setAttribute('src', "/static/img/icon-stop.svg");
+    stopIcon.setAttribute('alt', "stop icon");
+    stopIcon.setAttribute('onclick', 'stopQueue()');
+    // kill icon
+    var killIcon = document.createElement('img');
+    killIcon.setAttribute('id', "kill-icon");
+    killIcon.setAttribute('title', "Kill Download Queue");
+    killIcon.setAttribute('src', "/static/img/icon-close.svg");
+    killIcon.setAttribute('alt', "kill icon");
+    killIcon.setAttribute('onclick', 'killQueue()');
+    // stich together
+    iconBox.appendChild(stopIcon);
+    iconBox.appendChild(killIcon);
+    
+    return iconBox
+}

tubearchivist/static/script.js

@@ -79,6 +79,18 @@ function downloadNow(button) {
     }, 500);
 }
 
+function stopQueue() {
+    var payload = JSON.stringify({'queue': 'stop'});
+    sendPost(payload);
+    document.getElementById('stop-icon').remove();
+}
+
+function killQueue() {
+    var payload = JSON.stringify({'queue': 'kill'});
+    sendPost(payload);
+    document.getElementById('kill-icon').remove();
+}
+
 // settings page buttons
 function manualImport() {
     var payload = JSON.stringify({'manual-import': true});