add timeout for sponsorblock api requests, handle 503

Changed:
- added download filter dropdown
- fix for UnidentifiedImageError thumbnail error
- fix for _update_by_query channel missing watched_date
- fix for chrome compatibility text reveal

.github/ISSUE_TEMPLATE/FEATURE-REQUEST.yml

@@ -15,7 +15,7 @@ body:
       options:
         - label: I have read through the [wiki](https://github.com/tubearchivist/tubearchivist/wiki).
           required: true
-        - label: I understand the [scope](https://github.com/tubearchivist/tubearchivist/wiki/FAQ) of this project and am aware of the [known limitations](https://github.com/tubearchivist/tubearchivist#known-limitations).
+        - label: I understand the [scope](https://github.com/tubearchivist/tubearchivist/wiki/FAQ) of this project and am aware of the [known limitations](https://github.com/tubearchivist/tubearchivist#known-limitations) and my idea is not already on the [roadmap](https://github.com/tubearchivist/tubearchivist#roadmap).
           required: true
 
   - type: textarea

Dockerfile

@@ -3,7 +3,7 @@
 
 
 # First stage to build python wheel
-FROM python:3.10.5-slim-bullseye AS builder
+FROM python:3.10.8-slim-bullseye AS builder
 ARG TARGETPLATFORM
 
 RUN apt-get update
@@ -14,7 +14,7 @@ COPY ./tubearchivist/requirements.txt /requirements.txt
 RUN pip install --user -r requirements.txt
 
 # build final image
-FROM python:3.10.5-slim-bullseye as tubearchivist
+FROM python:3.10.8-slim-bullseye as tubearchivist
 
 ARG TARGETPLATFORM
 ARG INSTALL_DEBUG

README.md

@@ -63,6 +63,8 @@ Once your YouTube video collection grows, it becomes hard to search and find a s
 ## Installing and updating
 Take a look at the example `docker-compose.yml` file provided. Use the *latest* or the named semantic version tag. The *unstable* tag is for intermediate testing and as the name implies, is **unstable** and not be used on your main installation but in a [testing environment](CONTRIBUTING.md).  
 
+For minimal system requirements, the Tube Archivist stack needs around 2GB of available memory for a small testing setup and around 4GB of available memory for a mid to large sized installation.  
+
 Tube Archivist depends on three main components split up into separate docker containers:  
 
 ### Tube Archivist
@@ -72,7 +74,7 @@ The main Python application that displays and serves your video collection, buil
   - And another volume to save application data at **/cache**.
   - The environment variables `ES_URL` and `REDIS_HOST` are needed to tell Tube Archivist where Elasticsearch and Redis respectively are located.
   - The environment variables `HOST_UID` and `HOST_GID` allows Tube Archivist to `chown` the video files to the main host system user instead of the container user. Those two variables are optional, not setting them will disable that functionality. That might be needed if the underlying filesystem doesn't support `chown` like *NFS*. 
-  - Set the environment variable `TA_HOST` to match with the system running Tube Archivist. This can be a domain like *example.com*, a subdomain like *ta.example.com* or an IP address like *192.168.1.20*, add without the protocol and without the port. You can add multiple hostnames separated with a space.
+  - Set the environment variable `TA_HOST` to match with the system running Tube Archivist. This can be a domain like *example.com*, a subdomain like *ta.example.com* or an IP address like *192.168.1.20*, add without the protocol and without the port. You can add multiple hostnames separated with a space. Any wrong configurations here will result in a `Bad Request (400)` response.
   - Change the environment variables `TA_USERNAME` and `TA_PASSWORD` to create the initial credentials. 
   - `ELASTIC_PASSWORD` is for the password for Elasticsearch. The environment variable `ELASTIC_USER` is optional, should you want to change the username from the default *elastic*.
   - For the scheduler to know what time it is, set your timezone with the `TZ` environment variable, defaults to *UTC*.
@@ -91,6 +93,7 @@ You can configure LDAP with the following environment variables:
 
  - `TA_LDAP` (ex: `true`) Set to anything besides empty string to use LDAP authentication **instead** of local user authentication.
  - `TA_LDAP_SERVER_URI` (ex: `ldap://ldap-server:389`) Set to the uri of your LDAP server.
+ - `TA_LDAP_DISABLE_CERT_CHECK` (ex: `true`) Set to anything besides empty string to disable certificate checking when connecting over LDAPS.
  - `TA_LDAP_BIND_DN` (ex: `uid=search-user,ou=users,dc=your-server`) DN of the user that is able to perform searches on your LDAP account.
  - `TA_LDAP_BIND_PASSWORD` (ex: `yoursecretpassword`) Password for the search user.
  - `TA_LDAP_USER_BASE` (ex: `ou=users,dc=your-server`) Search base for user filter.
@@ -182,12 +185,12 @@ We have come far, nonetheless we are not short of ideas on how to improve and ex
 - [ ] Show total video downloaded vs total videos available in channel
 - [ ] Add statistics of index
 - [ ] Download speed schedule ([#198](https://github.com/tubearchivist/tubearchivist/issues/198))
-- [ ] Implement complete offline media file import from json file ([#138](https://github.com/tubearchivist/tubearchivist/issues/138))
 - [ ] Auto ignore videos by keyword ([#163](https://github.com/tubearchivist/tubearchivist/issues/163))
 - [ ] Custom searchable notes to videos, channels, playlists ([#144](https://github.com/tubearchivist/tubearchivist/issues/144))
 - [ ] Download video comments
 
 Implemented:
+- [X] Implement complete offline media file import from json file [2022-08-20]
 - [X] Filter and query in search form, search by url query [2022-07-23]
 - [X] Make items in grid row configurable to use more of the screen [2022-06-04]
 - [X] Add passing browser cookies to yt-dlp [2022-05-08]

deploy.sh

@@ -18,11 +18,7 @@ set -e
 
 function sync_blackhole {
 
-    # docker commands need sudo, only build amd64
     host="blackhole.local"
-
-    read -sp 'Password: ' remote_pw
-    export PASS=$remote_pw
     
     rsync -a --progress --delete-after \
         --exclude ".git" \
@@ -32,8 +28,8 @@ function sync_blackhole {
         --exclude "db.sqlite3" \
         . -e ssh "$host":tubearchivist
 
-    echo "$PASS" | ssh "$host" 'sudo -S docker buildx build --platform linux/amd64 -t bbilly1/tubearchivist:latest tubearchivist --load 2>/dev/null'
-    echo "$PASS" | ssh "$host" 'sudo -S docker compose up -d 2>/dev/null'
+    ssh "$host" 'docker build -t bbilly1/tubearchivist --build-arg TARGETPLATFORM="linux/amd64" tubearchivist'
+    ssh "$host" 'docker compose up -d'
 
 }
 
@@ -197,7 +193,8 @@ function sync_docker_old {
     # start build
     sudo docker buildx build \
         --platform linux/amd64,linux/arm64 \
-        -t bbilly1/tubearchivist:latest \
+        -t bbilly1/tubearchivist \
+        -t bbilly1/tubearchivist:unstable \
         -t bbilly1/tubearchivist:"$VERSION" --push .
 
     # create release tag

docker-compose.yml

@@ -34,7 +34,7 @@ services:
     depends_on:
       - archivist-es
   archivist-es:
-    image: bbilly1/tubearchivist-es         # only for amd64, or use official es 8.3.3
+    image: bbilly1/tubearchivist-es         # only for amd64, or use official es 8.4.3
     container_name: archivist-es
     restart: unless-stopped
     environment:

docs/Channels.md

@@ -24,7 +24,7 @@ Each channel will get a dedicated channel detail page accessible at `/channel/<c
 
 Additionally there is a *Channel Playlist* page, accessible at `/channel/<channel-id>/playlist/` to show all indexed playlists from this channel. 
 
-On the *Channel About* page you can see additional metadata.
+On the *Channel About* page, accessible at `/channel/<channel-id>/about/`, you can see additional metadata.
 - The button **Delete Channel** will delete the channel plus all videos of this channel, both media files and metadata additionally this will also delete playlists metadata belonging to that channel.
 
 The channel customize form gives options to change settings on a per channel basis. Any configurations here will overwrite your configurations from the [settings](Settings) page.
@@ -32,3 +32,5 @@ The channel customize form gives options to change settings on a per channel bas
 - **Auto Delete**: Automatically delete watched videos from this channel after selected days.
 - **Index Playlists**: Automatically add all Playlists with at least a video downloaded to your index. Only do this for channels where you care about playlists as this will slow down indexing new videos for having to check which playlist this belongs to.
 - **SponsorBlock**: Using [SponsorBlock](https://sponsor.ajay.app/) to get and skip sponsored content. Customize per channel: You can *disable* or *enable* SponsorBlock for certain channels only to overwrite the behavior set on the [Settings](settings) page. Selecting *unset* will remove the overwrite and your setting will fall back to the default on the settings page.
+
+If you have any videos pending in the download queue, a *Downloads* link will show, bringing you directly to the [downloads](Downloads) page, filtering the list by the selected channel.

docs/Downloads.md

@@ -29,6 +29,8 @@ The **Add to Download Queue** icon <img src="assets/icon-add.png?raw=true" alt="
 ## The Download Queue
 Below the three buttons you find the download queue. New items will get added at the bottom of the queue, the next video to download once you click on **Start Download** will be the first in the list.
 
+You can filter the download queue with the **filter** dropdown box, the filter will show once you have more than one channel in the download queue. Select the channel to filter by name, the number in parentheses indicates how many videos you have pending from this channel. Reset the filter by selecting *all* from the dropdown. This will generate links for the top 30 channels with pending videos.
+
 Every video in the download queue has two buttons:
 - **Ignore**: This will remove that video from the download queue and this video will not get added again, even when you **Rescan Subscriptions**.
 - **Download now**: This will give priority to this video. If the download process is already running, the prioritized video will get downloaded as soon as the current video is finished. If there is no download process running, this will start downloading this single video and stop after that.  

docs/Settings.md

@@ -133,6 +133,8 @@ Detect the YouTube ID from filename, this accepts the default yt-dlp naming conv
 - The YouTube ID in square brackets at the end of the filename is the crucial part.
 
 ### Offline import:
+**NOTE**: This is untested. Please provide feedback.  
+
 If the video you are trying to import is not available on YouTube any more, **Tube Archivist** can import the required metadata:
 - The file `<base-name>.info.json` is required to extract the required information.
 - Add the thumbnail as `<base-name>.<ext>`, where valid file extensions are *.jpg*, *.png* or *.webp*. If there is no thumbnail file, **Tube Archivist** will try to extract it from the media file or will fallback to a default thumbnail.

tubearchivist/api/README.md

@@ -61,6 +61,7 @@ The list views return a paginate object with the following keys:
 - prev_pages: *array of ints* of previous pages, if available
 - current_page: *int* current page from query
 - max_hits: *bool* if max of 10k results is reached
+- params: *str* additional url encoded query parameters
 - last_page: *int* of last page link
 - next_pages: *array of ints* of next pages
 - total_hits: *int* total results
@@ -169,6 +170,7 @@ GET /api/download/
 
 Parameter:
 - filter: pending, ignore
+- channel: channel-id
 
 ### Add list of videos to download queue
 POST /api/download/

tubearchivist/api/views.py

@@ -53,9 +53,7 @@ class ApiBaseView(APIView):
 
     def initiate_pagination(self, request):
         """set initial pagination values"""
-        user_id = request.user.id
-        page_get = int(request.GET.get("page", 0))
-        self.pagination_handler = Pagination(page_get, user_id)
+        self.pagination_handler = Pagination(request)
         self.data.update(
             {
                 "size": self.pagination_handler.pagination["page_size"],
@@ -368,13 +366,23 @@ class DownloadApiListView(ApiBaseView):
         """get request"""
         query_filter = request.GET.get("filter", False)
         self.data.update({"sort": [{"timestamp": {"order": "asc"}}]})
+
+        must_list = []
         if query_filter:
             if query_filter not in self.valid_filter:
                 message = f"invalid url query filder: {query_filter}"
                 print(message)
                 return Response({"message": message}, status=400)
 
-            self.data["query"] = {"term": {"status": {"value": query_filter}}}
+            must_list.append({"term": {"status": {"value": query_filter}}})
+
+        filter_channel = request.GET.get("channel", False)
+        if filter_channel:
+            must_list.append(
+                {"term": {"channel_id": {"value": filter_channel}}}
+            )
+
+        self.data["query"] = {"bool": {"must": must_list}}
 
         self.get_document_list(request)
         return Response(self.response)

tubearchivist/config/settings.py

@@ -119,6 +119,12 @@ if bool(environ.get("TA_LDAP")):
         "email": "mail",
     }
 
+    if bool(environ.get("TA_LDAP_DISABLE_CERT_CHECK")):
+        global AUTH_LDAP_GLOBAL_OPTIONS
+        AUTH_LDAP_GLOBAL_OPTIONS = {
+            ldap.OPT_X_TLS_REQUIRE_CERT: ldap.OPT_X_TLS_NEVER,
+        }
+
     global AUTHENTICATION_BACKENDS
     AUTHENTICATION_BACKENDS = ("django_auth_ldap.backend.LDAPBackend",)
 
@@ -201,4 +207,4 @@ CORS_ALLOW_HEADERS = list(default_headers) + [
 
 # TA application settings
 TA_UPSTREAM = "https://github.com/tubearchivist/tubearchivist"
-TA_VERSION = "v0.2.1"
+TA_VERSION = "v0.2.3"

tubearchivist/home/src/download/queue.py

@@ -196,6 +196,7 @@ class PendingList(PendingIndex):
         bulk_list = []
 
         for idx, youtube_id in enumerate(self.missing_videos):
+            print(f"{youtube_id}: add to download queue")
             video_details = self.get_youtube_details(youtube_id)
             if not video_details:
                 continue
@@ -245,7 +246,7 @@ class PendingList(PendingIndex):
             print(f"{youtube_id}: skipping premium video, id not matching")
             return False
         # stop if video is streaming live now
-        if vid["is_live"]:
+        if vid["live_status"] in ["is_upcoming", "is_live"]:
             return False
 
         return self._parse_youtube_details(vid)

tubearchivist/home/src/download/subscriptions.py

@@ -63,6 +63,7 @@ class ChannelSubscription:
 
         for idx, channel in enumerate(all_channels):
             channel_id = channel["channel_id"]
+            print(f"{channel_id}: find missing videos.")
             last_videos = self.get_last_youtube_videos(channel_id)
 
             if last_videos:

tubearchivist/home/src/download/thumbnails.py

@@ -14,7 +14,7 @@ from home.src.download import queue  # partial import
 from home.src.es.connect import IndexPaginate
 from home.src.ta.config import AppConfig
 from mutagen.mp4 import MP4, MP4Cover
-from PIL import Image, ImageFile, ImageFilter
+from PIL import Image, ImageFile, ImageFilter, UnidentifiedImageError
 
 ImageFile.LOAD_TRUNCATED_IMAGES = True
 
@@ -42,7 +42,12 @@ class ThumbManagerBase:
             try:
                 response = requests.get(url, stream=True, timeout=5)
                 if response.ok:
-                    return Image.open(response.raw)
+                    try:
+                        return Image.open(response.raw)
+                    except UnidentifiedImageError:
+                        print(f"failed to open thumbnail: {url}")
+                        return self.get_fallback()
+
                 if response.status_code == 404:
                     return self.get_fallback()
 
@@ -124,7 +129,7 @@ class ThumbManager(ThumbManagerBase):
 
         img_raw.convert("RGB").save(thumb_path)
 
-    def vid_thumb_path(self, absolute=False):
+    def vid_thumb_path(self, absolute=False, create_folder=False):
         """build expected path for video thumbnail from youtube_id"""
         folder_name = self.item_id[0].lower()
         folder_path = os.path.join("videos", folder_name)
@@ -132,6 +137,10 @@ class ThumbManager(ThumbManagerBase):
         if absolute:
             thumb_path = os.path.join(self.CACHE_DIR, thumb_path)
 
+        if create_folder:
+            folder_path = os.path.join(self.CACHE_DIR, folder_path)
+            os.makedirs(folder_path, exist_ok=True)
+
         return thumb_path
 
     def download_channel_art(self, urls, skip_existing=False):

tubearchivist/home/src/download/yt_dlp_base.py

@@ -20,7 +20,8 @@ class YtWrap:
         "default_search": "ytsearch",
         "quiet": True,
         "check_formats": "selected",
-        "socket_timeout": 2,
+        "socket_timeout": 3,
+        "extractor_retries": 3,
     }
 
     def __init__(self, obs_request, config=False):

tubearchivist/home/src/download/yt_dlp_handler.py

@@ -175,6 +175,14 @@ class VideoDownloader:
             if not success:
                 continue
 
+            mess_dict = {
+                "status": self.MSG,
+                "level": "info",
+                "title": "Indexing....",
+                "message": "Add video metadata to index.",
+            }
+            RedisArchivist().set_message(self.MSG, mess_dict, expire=60)
+
             vid_dict = index_new_video(
                 youtube_id, video_overwrites=self.video_overwrites
             )
@@ -187,12 +195,17 @@ class VideoDownloader:
             }
             RedisArchivist().set_message(self.MSG, mess_dict)
 
+            if queue.has_item():
+                message = "Continue with next video."
+            else:
+                message = "Download queue is finished."
+
             self.move_to_archive(vid_dict)
             mess_dict = {
                 "status": self.MSG,
                 "level": "info",
                 "title": "Completed",
-                "message": "",
+                "message": message,
             }
             RedisArchivist().set_message(self.MSG, mess_dict, expire=10)
             self._delete_from_pending(youtube_id)

tubearchivist/home/src/frontend/watched.py

@@ -57,6 +57,14 @@ class WatchState:
             print(response)
             raise ValueError("failed to mark video as watched")
 
+    def _get_source(self):
+        """build source line for update_by_query script"""
+        source = [
+            "ctx._source.player['watched'] = true",
+            f"ctx._source.player['watched_date'] = {self.stamp}",
+        ]
+        return "; ".join(source)
+
     def mark_channel_watched(self):
         """change watched status of every video in channel"""
         path = "ta_video/_update_by_query"
@@ -67,7 +75,7 @@ class WatchState:
         data = {
             "query": {"bool": {"must": must_list}},
             "script": {
-                "source": "ctx._source.player['watched'] = true",
+                "source": self._get_source(),
                 "lang": "painless",
             },
         }
@@ -87,7 +95,7 @@ class WatchState:
         data = {
             "query": {"bool": {"must": must_list}},
             "script": {
-                "source": "ctx._source.player['watched'] = true",
+                "source": self._get_source(),
                 "lang": "painless",
             },
         }

tubearchivist/home/src/index/channel.py

@@ -73,13 +73,14 @@ class ChannelScraper:
 
     def _is_deactivated(self):
         """check if channel is deactivated"""
-        alert_text = "This channel does not exist."
         alerts = self.yt_json.get("alerts")
-        if alerts and alert_text in str(alerts):
-            print(f"{self.channel_id}: {alert_text}")
-            return True
+        if not alerts:
+            return False
 
-        return False
+        for alert in alerts:
+            alert_text = alert["alertRenderer"]["text"]["simpleText"]
+            print(f"{self.channel_id}: failed to extract, {alert_text}")
+            return True
 
     def _parse_channel_main(self):
         """extract maintab values from scraped channel json data"""
@@ -226,10 +227,11 @@ class YoutubeChannel(YouTubeItem):
 
             self.json_data.update(
                 {
-                    "channel_subs": content["channel_follower_count"],
-                    "channel_description": content["description"],
+                    "channel_subs": content.get("channel_follower_count", 0),
+                    "channel_description": content.get("description", False),
                 }
             )
+            os.remove(info_json)
 
     def get_channel_art(self):
         """download channel art for new channels"""

tubearchivist/home/src/index/filesystem.py

@@ -217,15 +217,14 @@ class ImportFolderScanner:
         last_base = False
 
         for file_path in all_files:
-            base_name_raw, ext = os.path.splitext(file_path)
-            base_name, _ = os.path.splitext(base_name_raw)
-
+            base_name, ext = self._detect_base_name(file_path)
             key, file_path = self._detect_type(file_path, ext)
             if not key or not file_path:
                 continue
 
             if base_name != last_base:
                 if last_base:
+                    print(f"manual import: {current_video}")
                     self.to_import.append(current_video)
 
                 current_video = self._get_template()
@@ -237,8 +236,21 @@ class ImportFolderScanner:
                 current_video[key] = file_path
 
         if current_video.get("media"):
+            print(f"manual import: {current_video}")
             self.to_import.append(current_video)
 
+    def _detect_base_name(self, file_path):
+        """extract base_name and ext for matching"""
+        base_name_raw, ext = os.path.splitext(file_path)
+        base_name, ext2 = os.path.splitext(base_name_raw)
+
+        if ext2:
+            if ISO639Utils.short2long(ext2.strip(".")) or ext2 == ".info":
+                # valid secondary extension
+                return base_name, ext
+
+        return base_name_raw, ext
+
     def _detect_type(self, file_path, ext):
         """detect metadata type for file"""
 
@@ -260,12 +272,12 @@ class ImportFolderScanner:
             self._convert_thumb(current_video)
             self._get_subtitles(current_video)
             self._convert_video(current_video)
+            print(f"manual import: {current_video}")
 
             ManualImport(current_video, self.CONFIG).run()
 
     def _detect_youtube_id(self, current_video):
         """find video id from filename or json"""
-        print(current_video)
         youtube_id = self._extract_id_from_filename(current_video["media"])
         if youtube_id:
             current_video["video_id"] = youtube_id
@@ -276,7 +288,6 @@ class ImportFolderScanner:
             current_video["video_id"] = youtube_id
             return
 
-        print(current_video["media"])
         raise ValueError("failed to find video id")
 
     @staticmethod
@@ -516,7 +527,8 @@ class ManualImport:
 
         if video.offline_import and self.current_video["thumb"]:
             old_path = self.current_video["thumb"]
-            new_path = ThumbManager(video_id).vid_thumb_path(absolute=True)
+            thumbs = ThumbManager(video_id)
+            new_path = thumbs.vid_thumb_path(absolute=True, create_folder=True)
             shutil.move(old_path, new_path, copy_function=shutil.copyfile)
         else:
             url = video.json_data["vid_thumb_url"]
@@ -555,11 +567,13 @@ class ManualImport:
 
     def _cleanup(self, json_data):
         """cleanup leftover files"""
-        if os.path.exists(self.current_video["metadata"]):
-            os.remove(self.current_video["metadata"])
+        meta_data = self.current_video["metadata"]
+        if meta_data and os.path.exists(meta_data):
+            os.remove(meta_data)
 
-        if os.path.exists(self.current_video["thumb"]):
-            os.remove(self.current_video["thumb"])
+        thumb = self.current_video["thumb"]
+        if thumb and os.path.exists(thumb):
+            os.remove(thumb)
 
         for subtitle_file in self.current_video["subtitle"]:
             if os.path.exists(subtitle_file):

tubearchivist/home/src/index/generic.py

@@ -73,16 +73,25 @@ class Pagination:
     figure out the pagination based on page size and total_hits
     """
 
-    def __init__(self, page_get, user_id, search_get=False):
-        self.user_id = user_id
+    def __init__(self, request):
+        self.request = request
+        self.page_get = False
+        self.params = False
+        self.get_params()
         self.page_size = self.get_page_size()
-        self.page_get = page_get
-        self.search_get = search_get
         self.pagination = self.first_guess()
 
+    def get_params(self):
+        """process url query parameters"""
+        query_dict = self.request.GET.copy()
+        self.page_get = int(query_dict.get("page", 0))
+
+        _ = query_dict.pop("page", False)
+        self.params = query_dict.urlencode()
+
     def get_page_size(self):
         """get default or user modified page_size"""
-        key = f"{self.user_id}:page_size"
+        key = f"{self.request.user.id}:page_size"
         page_size = RedisArchivist().get_message(key)["status"]
         if not page_size:
             config = AppConfig().config
@@ -108,9 +117,9 @@ class Pagination:
             "prev_pages": prev_pages,
             "current_page": page_get,
             "max_hits": False,
+            "params": self.params,
         }
-        if self.search_get:
-            pagination.update({"search_get": self.search_get})
+
         return pagination
 
     def validate(self, total_hits):

tubearchivist/home/src/index/playlist.py

@@ -45,13 +45,19 @@ class YoutubePlaylist(YouTubeItem):
 
     def process_youtube_meta(self):
         """extract relevant fields from youtube"""
+        try:
+            playlist_thumbnail = self.youtube_meta["thumbnails"][-1]["url"]
+        except IndexError:
+            print(f"{self.youtube_id}: thumbnail extraction failed")
+            playlist_thumbnail = False
+
         self.json_data = {
             "playlist_id": self.youtube_id,
             "playlist_active": True,
             "playlist_name": self.youtube_meta["title"],
             "playlist_channel": self.youtube_meta["channel"],
             "playlist_channel_id": self.youtube_meta["channel_id"],
-            "playlist_thumbnail": self.youtube_meta["thumbnails"][-1]["url"],
+            "playlist_thumbnail": playlist_thumbnail,
             "playlist_description": self.youtube_meta["description"] or False,
             "playlist_last_refresh": int(datetime.now().strftime("%s")),
         }

tubearchivist/home/src/index/reindex.py

@@ -72,14 +72,11 @@ class Reindex:
 
     def _get_unrated_vids(self):
         """get max 200 videos without rating if ryd integration is enabled"""
-        data = {
-            "size": 200,
-            "query": {
-                "bool": {
-                    "must_not": [{"exists": {"field": "stats.average_rating"}}]
-                }
-            },
-        }
+        must_not_list = [
+            {"exists": {"field": "stats.average_rating"}},
+            {"term": {"active": {"value": False}}},
+        ]
+        data = {"size": 200, "query": {"bool": {"must_not": must_not_list}}}
         response, _ = ElasticWrap("ta_video/_search").get(data=data)
 
         missing_rating = [i["_id"] for i in response["hits"]["hits"]]

tubearchivist/home/src/index/video.py

@@ -48,9 +48,17 @@ class SponsorBlock:
         url = f"{self.API}/skipSegments?videoID={youtube_id}"
         headers = {"User-Agent": self.user_agent}
         print(f"{youtube_id}: get sponsorblock timestamps")
-        response = requests.get(url, headers=headers)
+        try:
+            response = requests.get(url, headers=headers, timeout=10)
+        except requests.ReadTimeout:
+            print(f"{youtube_id}: sponsorblock API timeout")
+            return False
+
         if not response.ok:
-            print(f"{youtube_id}: sponsorblock failed: {response.text}")
+            print(f"{youtube_id}: sponsorblock failed: {response.status_code}")
+            if response.status_code == 503:
+                return False
+
             sponsor_dict = {
                 "last_refresh": self.last_refresh,
                 "is_enabled": True,

tubearchivist/home/src/ta/ta_redis.py

@@ -143,3 +143,8 @@ class RedisQueue(RedisBase):
     def trim(self, size):
         """trim the queue based on settings amount"""
         self.conn.execute_command("LTRIM", self.key, 0, size)
+
+    def has_item(self):
+        """check if queue as at least one pending item"""
+        result = self.conn.execute_command("LRANGE", self.key, 0, 0)
+        return bool(result)

tubearchivist/home/templates/home/base.html

@@ -79,16 +79,16 @@
             <div class="pagination">
                 {% if pagination %}
                     {% if pagination.current_page > 1 %}
-                        {% if pagination.search_get %}
-                        <a class="pagination-item" href="{{ request.path }}?search={{ pagination.search_get }}">First</a>
+                        {% if pagination.params %}
+                        <a class="pagination-item" href="{{ request.path }}?{{ pagination.params }}">First</a>
                         {% else %}
                         <a class="pagination-item" href="{{ request.path }}">First</a>
                         {% endif %}
                     {% endif %}
                     {% if pagination.prev_pages %}
                         {% for page in pagination.prev_pages %}
-                            {% if pagination.search_get %}
-                            <a class="pagination-item" href="?page={{ page }}&search={{ pagination.search_get }}">{{ page }}</a>
+                            {% if pagination.params %}
+                            <a class="pagination-item" href="?page={{ page }}&{{ pagination.params }}">{{ page }}</a>
                             {% else %}
                             <a class="pagination-item" href="?page={{ page }}">{{ page }}</a>
                             {% endif %}
@@ -100,16 +100,16 @@
                     {% if pagination.next_pages %}
                         <span> ></span>
                         {% for page in pagination.next_pages %}
-                            {% if pagination.search_get %}
-                            <a class="pagination-item" href="?page={{ page }}&search={{ pagination.search_get }}">{{ page }}</a>
+                            {% if pagination.params %}
+                            <a class="pagination-item" href="?page={{ page }}&{{ pagination.params }}">{{ page }}</a>
                             {% else %}
                             <a class="pagination-item" href="?page={{ page }}">{{ page }}</a>
                             {% endif %}
                         {% endfor %}
                     {% endif %}
                     {% if pagination.last_page > 0 %}
-                        {% if pagination.search_get %}
-                        <a class="pagination-item" href="?page={{ pagination.last_page }}&search={{ pagination.search_get }}">
+                        {% if pagination.params %}
+                        <a class="pagination-item" href="?page={{ pagination.last_page }}&{{ pagination.params }}">
                             {% if pagination.max_hits %}
                                 Max ({{ pagination.last_page }})
                             {% else %}

tubearchivist/home/templates/home/channel_id.html

@@ -10,6 +10,9 @@
         <a href="{% url 'channel_id' channel_info.channel_id %}"><h3>Videos</h3></a>
         <a href="{% url 'channel_id_playlist' channel_info.channel_id %}"><h3>Playlists</h3></a>
         <a href="{% url 'channel_id_about' channel_info.channel_id %}"><h3>About</h3></a>
+        {% if has_pending %}
+            <a href="{% url 'downloads' %}?channel={{ channel_info.channel_id }}"><h3>Downloads</h3></a>
+        {% endif %}
     </div>
     <div id="notifications" data="channel_id"></div>
     <div class="info-box info-box-2">

tubearchivist/home/templates/home/channel_id_about.html

@@ -10,6 +10,9 @@
         <a href="{% url 'channel_id' channel_info.channel_id %}"><h3>Videos</h3></a>
         <a href="{% url 'channel_id_playlist' channel_info.channel_id %}"><h3>Playlists</h3></a>
         <a href="{% url 'channel_id_about' channel_info.channel_id %}"><h3>About</h3></a>
+        {% if has_pending %}
+            <a href="{% url 'downloads' %}?channel={{ channel_info.channel_id }}"><h3>Downloads</h3></a>
+        {% endif %}
     </div>
     <div class="info-box info-box-3">
         <div class="info-box-item">

tubearchivist/home/templates/home/channel_id_playlist.html

@@ -10,6 +10,9 @@
         <a href="{% url 'channel_id' channel_info.channel_id %}"><h3>Videos</h3></a>
         <a href="{% url 'channel_id_playlist' channel_info.channel_id %}"><h3>Playlists</h3></a>
         <a href="{% url 'channel_id_about' channel_info.channel_id %}"><h3>About</h3></a>
+        {% if has_pending %}
+            <a href="{% url 'downloads' %}?channel={{ channel_info.channel_id }}"><h3>Downloads</h3></a>
+        {% endif %}
     </div>
     <div class="view-controls">
         <div class="toggle">

tubearchivist/home/templates/home/downloads.html

@@ -3,7 +3,7 @@
 {% block content %}
 <div class="boxed-content">
     <div class="title-bar">
-        <h1>Downloads</h1>
+        <h1>Downloads {% if channel_filter_id %} for {{ channel_filter_name }}{% endif %}</h1>
     </div>
     <div id="notifications" data="download"></div>
     <div id="downloadControl"></div>
@@ -41,38 +41,65 @@
             </div>
         </div>
         <div class="view-icons">
+            {% if channel_agg_list|length > 1 %}
+                <span>Filter:</span>
+                <select name="channel_filter" id="channel_filter" onchange="channelFilterDownload(this.value)">
+                    <option value="all" {% if not channel_filter_id %}selected{% endif %}>all</option>
+                    {% for channel in channel_agg_list %}
+                        <option {% if channel_filter_id == channel.id %}selected{% endif %} value="{{ channel.id }}">{{ channel.name }} ({{channel.count}})</option>
+                    {% endfor %}
+                </select>
+            {% endif %}
+            {% if view_style == "grid" %}
+                <div class="grid-count">
+                    {% if grid_items < 7 %}
+                        <img src="{% static 'img/icon-add.svg' %}" onclick="changeGridItems(this)" data-value="{{ grid_items|add:"1"}}" alt="grid plus row">
+                    {% endif %}
+                    {% if grid_items > 3 %}
+                        <img src="{% static 'img/icon-substract.svg' %}" onclick="changeGridItems(this)" data-value="{{ grid_items|add:"-1"}}" alt="grid minus row">
+                    {% endif %}
+                </div>
+            {% endif %}
             <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="downloads" data-value="grid" alt="grid view">
             <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="downloads" data-value="list" alt="list view">
         </div>
     </div>
-    <h3>Total videos: {{ max_hits }}{% if max_hits == 10000 %}+{% endif %}</h3>
-    <div class="dl-list {{ view_style }}">
+    <h3>Total videos: {{ max_hits }}{% if max_hits == 10000 %}+{% endif %} {% if channel_filter_id %} - from channel <i>{{ channel_filter_name }}</i>{% endif %}</h3>
+</div>
+<div class="boxed-content {% if view_style == "grid" %}boxed-{{ grid_items }}{% endif %}">
+    <div class="video-list {{ view_style }} {% if view_style == "grid" %}grid-{{ grid_items }}{% endif %}">
         {% if results %}
             {% for video in results %}
-                <div class="dl-item {{ view_style }}" id="dl-{{ video.source.youtube_id }}">
-                    <div class="dl-thumb {{ view_style }}">
-                        <img src="/cache/{{ video.source.vid_thumb_url }}" alt="video_thumb">
-                        {% if show_ignored_only %}
-                            <span>ignored</span>
-                        {% else %}
-                            <span>queued</span>
-                        {% endif %}
+                <div class="video-item {{ view_style }}" id="dl-{{ video.source.youtube_id }}">
+                    <div class="video-thumb-wrap {{ view_style }}">
+                        <div class="video-thumb">
+                            <img src="/cache/{{ video.source.vid_thumb_url }}" alt="video_thumb">
+                            {% if show_ignored_only %}
+                                <span>ignored</span>
+                            {% else %}
+                                <span>queued</span>
+                            {% endif %}
+                        </div>
                     </div>
-                    <div class="dl-desc {{ view_style }}">
-                        <h3>{{ video.source.title }}</h3>
-                        {% if video.source.channel_indexed %}
+                    <div class="video-desc {{ view_style }}">
+                        <div>
+                            {% if video.source.channel_indexed %}
                             <a href="{% url 'channel_id' video.source.channel_id %}">{{ video.source.channel_name }}</a>
-                        {% else %}
-                            <span>{{ video.source.channel_name }}</span>
-                        {% endif %}
+                            {% else %}
+                                <span>{{ video.source.channel_name }}</span>
+                            {% endif %}
+                            <h3>{{ video.source.title }}</h3>
+                        </div>
                         <p>Published: {{ video.source.published }} | Duration: {{ video.source.duration }} | {{ video.source.youtube_id }}</p>
-                        {% if show_ignored_only %}
-                            <button data-id="{{ video.source.youtube_id }}" onclick="forgetIgnore(this)">Forget</button>
-                            <button data-id="{{ video.source.youtube_id }}" onclick="addSingle(this)">Add to queue</button>
-                        {% else %}
-                            <button data-id="{{ video.source.youtube_id }}" onclick="toIgnore(this)">Ignore</button>
-                            <button id="{{ video.source.youtube_id }}" data-id="{{ video.source.youtube_id }}" onclick="downloadNow(this)">Download now</button>
-                        {% endif %}
+                        <div>
+                            {% if show_ignored_only %}
+                                <button data-id="{{ video.source.youtube_id }}" onclick="forgetIgnore(this)">Forget</button>
+                                <button data-id="{{ video.source.youtube_id }}" onclick="addSingle(this)">Add to queue</button>
+                            {% else %}
+                                <button data-id="{{ video.source.youtube_id }}" onclick="toIgnore(this)">Ignore</button>
+                                <button id="{{ video.source.youtube_id }}" data-id="{{ video.source.youtube_id }}" onclick="downloadNow(this)">Download now</button>
+                            {% endif %}
+                        </div>
                     </div>
                 </div>
             {% endfor %}

tubearchivist/home/views.py

@@ -31,7 +31,7 @@ from home.src.frontend.forms import (
     UserSettingsForm,
 )
 from home.src.frontend.searching import SearchHandler
-from home.src.index.channel import channel_overwrites
+from home.src.index.channel import YoutubeChannel, channel_overwrites
 from home.src.index.generic import Pagination
 from home.src.index.playlist import YoutubePlaylist
 from home.src.ta.config import AppConfig, ScheduleBuilder
@@ -237,14 +237,10 @@ class ArchivistResultsView(ArchivistViewConfig):
 
     def initiate_vars(self, request):
         """search in es for vidoe hits"""
-        page_get = int(request.GET.get("page", 0))
         self.user_id = request.user.id
         self.config_builder(self.user_id)
         self.search_get = request.GET.get("search", False)
-        search_encoded = self._url_encode(self.search_get)
-        self.pagination_handler = Pagination(
-            page_get=page_get, user_id=self.user_id, search_get=search_encoded
-        )
+        self.pagination_handler = Pagination(request)
         self.sort_by = self._sort_by_overwrite()
         self._initial_data()
 
@@ -362,29 +358,80 @@ class DownloadView(ArchivistResultsView):
     def get(self, request):
         """handle get request"""
         self.initiate_vars(request)
-        self._update_view_data()
+        self._update_view_data(request)
         self.find_results()
         self.context.update(
             {
                 "title": "Downloads",
                 "add_form": AddToQueueForm(),
+                "channel_agg_list": self._get_channel_agg(),
             }
         )
         return render(request, "home/downloads.html", self.context)
 
-    def _update_view_data(self):
+    def _update_view_data(self, request):
         """update downloads view specific data dict"""
         if self.context["show_ignored_only"]:
             filter_view = "ignore"
         else:
             filter_view = "pending"
+
+        must_list = [{"term": {"status": {"value": filter_view}}}]
+
+        channel_filter = request.GET.get("channel", False)
+        if channel_filter:
+            must_list.append(
+                {"term": {"channel_id": {"value": channel_filter}}}
+            )
+
+            channel = YoutubeChannel(channel_filter)
+            channel.get_from_es()
+            self.context.update(
+                {
+                    "channel_filter_id": channel_filter,
+                    "channel_filter_name": channel.json_data["channel_name"],
+                }
+            )
+
         self.data.update(
             {
-                "query": {"term": {"status": {"value": filter_view}}},
+                "query": {"bool": {"must": must_list}},
                 "sort": [{"timestamp": {"order": "asc"}}],
             }
         )
 
+    def _get_channel_agg(self):
+        """get pending channel with count"""
+        data = {
+            "size": 0,
+            "query": {"term": {"status": {"value": "pending"}}},
+            "aggs": {
+                "channel_downloads": {
+                    "multi_terms": {
+                        "size": 30,
+                        "terms": [
+                            {"field": "channel_name.keyword"},
+                            {"field": "channel_id"},
+                        ],
+                        "order": {"_count": "desc"},
+                    }
+                }
+            },
+        }
+        response, _ = ElasticWrap(self.es_search).get(data=data)
+        buckets = response["aggregations"]["channel_downloads"]["buckets"]
+
+        buckets_sorted = []
+        for i in buckets:
+            bucket = {
+                "name": i["key"][0],
+                "id": i["key"][1],
+                "count": i["doc_count"],
+            }
+            buckets_sorted.append(bucket)
+
+        return buckets_sorted
+
     @staticmethod
     def post(request):
         """handle post requests"""
@@ -414,7 +461,37 @@ class DownloadView(ArchivistResultsView):
         return redirect("downloads", permanent=True)
 
 
-class ChannelIdView(ArchivistResultsView):
+class ChannelIdBaseView(ArchivistResultsView):
+    """base class for all channel-id views"""
+
+    def get_channel_meta(self, channel_id):
+        """get metadata for channel"""
+        path = f"ta_channel/_doc/{channel_id}"
+        response, _ = ElasticWrap(path).get()
+        channel_info = SearchProcess(response).process()
+
+        return channel_info
+
+    def channel_has_pending(self, channel_id):
+        """check if channel has pending videos in queue"""
+        path = "ta_download/_search"
+        data = {
+            "size": 1,
+            "query": {
+                "bool": {
+                    "must": [
+                        {"term": {"status": {"value": "pending"}}},
+                        {"term": {"channel_id": {"value": channel_id}}},
+                    ]
+                }
+            },
+        }
+        response, _ = ElasticWrap(path).get(data=data)
+
+        self.context.update({"has_pending": bool(response["hits"]["hits"])})
+
+
+class ChannelIdView(ChannelIdBaseView):
     """resolves to /channel/<channel-id>/
     display single channel page from channel_id
     """
@@ -428,6 +505,7 @@ class ChannelIdView(ArchivistResultsView):
         self._update_view_data(channel_id)
         self.find_results()
         self.match_progress()
+        self.channel_has_pending(channel_id)
 
         if self.context["results"]:
             channel_info = self.context["results"][0]["source"]["channel"]
@@ -478,7 +556,7 @@ class ChannelIdView(ArchivistResultsView):
         return redirect("channel_id", channel_id, permanent=True)
 
 
-class ChannelIdAboutView(ArchivistResultsView):
+class ChannelIdAboutView(ChannelIdBaseView):
     """resolves to /channel/<channel-id>/about/
     show metadata, handle per channel conf
     """
@@ -488,6 +566,7 @@ class ChannelIdAboutView(ArchivistResultsView):
     def get(self, request, channel_id):
         """handle get request"""
         self.initiate_vars(request)
+        self.channel_has_pending(channel_id)
 
         path = f"ta_channel/_doc/{channel_id}"
         response, _ = ElasticWrap(path).get()
@@ -521,7 +600,7 @@ class ChannelIdAboutView(ArchivistResultsView):
         return redirect("channel_id_about", channel_id, permanent=True)
 
 
-class ChannelIdPlaylistView(ArchivistResultsView):
+class ChannelIdPlaylistView(ChannelIdBaseView):
     """resolves to /channel/<channel-id>/playlist/
     show all playlists of channel
     """
@@ -534,8 +613,9 @@ class ChannelIdPlaylistView(ArchivistResultsView):
         self.initiate_vars(request)
         self._update_view_data(channel_id)
         self.find_results()
+        self.channel_has_pending(channel_id)
 
-        channel_info = self._get_channel_meta(channel_id)
+        channel_info = self.get_channel_meta(channel_id)
         channel_name = channel_info["channel_name"]
         self.context.update(
             {
@@ -556,14 +636,6 @@ class ChannelIdPlaylistView(ArchivistResultsView):
 
         self.data["query"] = {"bool": {"must": must_list}}
 
-    def _get_channel_meta(self, channel_id):
-        """get metadata for channel"""
-        path = f"ta_channel/_doc/{channel_id}"
-        response, _ = ElasticWrap(path).get()
-        channel_info = SearchProcess(response).process()
-
-        return channel_info
-
 
 class ChannelView(ArchivistResultsView):
     """resolves to /channel/

tubearchivist/requirements.txt

@@ -1,13 +1,13 @@
 beautifulsoup4==4.11.1
 celery==5.2.7
-Django==4.1
+Django==4.1.2
 django-auth-ldap==4.1.0
 django-cors-headers==3.13.0
-djangorestframework==3.13.1
+djangorestframework==3.14.0
 Pillow==9.2.0
 redis==4.3.4
 requests==2.28.1
 ryd-client==0.0.6
 uWSGI==2.0.20
 whitenoise==6.2.0
-yt_dlp==2022.8.19
+yt_dlp==2022.10.4

tubearchivist/static/css/style.css

@@ -344,6 +344,7 @@ button:hover {
 .grid-count {
     display: flex;
     justify-content: end;
+    align-items: center;
 }
 
 .view-icons img {
@@ -456,6 +457,10 @@ video:-webkit-full-screen {
     overflow: hidden;
 }
 
+.video-item:hover .video-thumb span {
+    opacity: 1;
+}
+
 .video-item.list {
     display: grid;
     grid-template-columns: 26% auto;
@@ -476,6 +481,16 @@ video:-webkit-full-screen {
     position: relative;
 }
 
+.video-thumb span {
+    position: absolute;
+    top: 5px;
+    left: 5px;
+    background-color: var(--accent-font-light);
+    padding: 5px;
+    opacity: 0;
+    transition: 300ms ease-in-out;
+}
+
 .video-play img {
     width: 40px;
     filter: var(--img-filter);
@@ -512,7 +527,7 @@ video:-webkit-full-screen {
     padding: 10px;
     height: 100%;
     display: flex;
-    flex-wrap: wrap-reverse;
+    flex-wrap: wrap;
     align-content: center;
 }
 
@@ -907,73 +922,6 @@ video:-webkit-full-screen {
     cursor: pointer;
 }
 
-.dl-list.list {
-    display: block;
-}
-
-.dl-list.grid {
-    display: grid;
-    grid-template-columns: 1fr 1fr 1fr;
-    grid-gap: 1rem;
-}
-
-.dl-item.list {
-    display: flex;
-    margin: 15px 0;
-    align-items: center;
-    background-color: var(--highlight-bg);
-}
-
-.dl-item.grid {
-    display: flex;
-    flex-wrap: wrap;
-    background-color: var(--highlight-bg);
-}
-
-.dl-check {
-    width: 30px;
-}
-
-.dl-item:hover > .dl-thumb span {
-    opacity: 1;
-}
-
-.dl-thumb {
-    position: relative;
-}
-
-.dl-thumb span {
-    position: absolute;
-    top: 5px;
-    left: 5px;
-    background-color: var(--accent-font-light);
-    padding: 5px;
-    opacity: 0;
-    transition: 300ms ease-in-out;
-}
-
-.dl-thumb.list {
-    width: 25%;
-}
-
-.dl-thumb.grid {
-    width: 100%;
-}
-
-.dl-item img {
-    width: 100%;
-}
-
-.dl-desc.list {
-    padding: 0 15px;
-    width: 75%;
-}
-
-.dl-desc.grid {
-    padding: 15px;
-    width: 100%;
-}
-
 .dl-control-icons {
     display: flex;
     justify-content: center;
@@ -1131,14 +1079,10 @@ video:-webkit-full-screen {
     .video-list.grid.grid-5,
     .video-list.grid.grid-6,
     .video-list.grid.grid-7,
-    .dl-list.grid,
     .channel-list.grid,
     .playlist-list.grid {
         grid-template-columns: 1fr 1fr;
     }
-    .dl-thumb.list {
-        width: 35%;
-    }
     .video-item.list,
     .playlist-item.list {
         display: grid;
@@ -1182,7 +1126,6 @@ video:-webkit-full-screen {
     .video-list.grid.grid-5,
     .video-list.grid.grid-6,
     .video-list.grid.grid-7,
-    .dl-list.grid,
     .channel-list.grid,
     .video-item.list,
     .playlist-list.list,
@@ -1231,15 +1174,6 @@ video:-webkit-full-screen {
     .description-box {
         display: block;
     }
-    .dl-item {
-        display: block;
-    }
-    .dl-thumb {
-        width: 100%;
-    }
-    .dl-desc {
-        padding: 15px;
-    }
     .backup-grid-row {
         display: flex;
         flex-wrap: wrap;

tubearchivist/static/script.js

@@ -140,7 +140,7 @@ function toggleCheckbox(checkbox) {
     var payload = JSON.stringify(payloadDict);
     sendPost(payload);
     setTimeout(function(){
-        var currPage = window.location.pathname;
+        var currPage = window.location.pathname + window.location.search;
         window.location.replace(currPage);
         return false;
     }, 500);
@@ -1102,12 +1102,12 @@ function textReveal() {
 function textExpand() {
     var textBox = document.getElementById("text-expand");
     var button = document.getElementById("text-expand-button");
-    var textBoxLineClamp = textBox.style["-webkit-line-clamp"];
-    if (textBoxLineClamp === "none") {
+    var style = window.getComputedStyle(textBox)
+    if (style.webkitLineClamp === "none") {
         textBox.style["-webkit-line-clamp"] = "4";
         button.innerText = "Show more";
     } else {
-        textBox.style["-webkit-line-clamp"] = "none";
+        textBox.style["-webkit-line-clamp"] = "unset";
         button.innerText = "Show less";
     }
 }
@@ -1119,8 +1119,9 @@ function textExpandButtonVisibilityUpdate() {
     if (!textBox || !button)
         return;
 
-    var textBoxLineClamp = textBox.style["-webkit-line-clamp"];
-    if (textBoxLineClamp === "none")
+    var styles = window.getComputedStyle(textBox);
+    var textBoxLineClamp = styles.webkitLineClamp;
+    if (textBoxLineClamp === "unset")
         return; // text box is in revealed state
 
     if (textBox.offsetHeight < textBox.scrollHeight
@@ -1147,6 +1148,14 @@ function showForm() {
     animate('animate-icon', 'pulse-img');
 }
 
+function channelFilterDownload(value) {
+    if (value === "all") {
+        window.location = "/downloads/";
+    } else {
+        window.location.search = "?channel=" + value;
+    }
+}
+
 function showOverwrite() {
     var overwriteDiv = document.getElementById("overwrite-form");
     if (overwriteDiv.classList.contains("hidden-overwrite")) {