update for v0.0.7

Unraid installation instructions

CONTRIBUTING.md

@@ -11,7 +11,7 @@ If the issue doesn't move forward due to a lack of response, I assume it's solve
 
 ## Wiki
 
-WIP: The wiki is where all user functions are explained in detail. These pages are mirrored into the **docs** folder of the repo. This allows for pull requests and all other features like regular code. Make any changes there, and I'll sync them  with the wiki tab.
+The wiki is where all user functions are explained in detail. These pages are mirrored into the **docs** folder of the repo. This allows for pull requests and all other features like regular code. Make any changes there, and I'll sync them with the wiki tab.
 
 ## Development Environment
 
@@ -20,7 +20,6 @@ I have learned the hard way, that working on a dockerized application outside of
 This is my setup I have landed on, YMMV:
 - Clone the repo, work on it with your favorite code editor in your local filesystem. *testing* branch is the where all the changes are happening, might be unstable and is WIP.
 - Then I have a VM on KVM hypervisor running standard Ubuntu Server LTS with docker installed. The VM keeps my projects separate and offers convenient snapshot functionality. The VM also offers ways to simulate lowend environments by limiting CPU cores and memory. But you could also just run docker on your host system.
-- Additionally to the required services as listed in the example docker-compose file, the **Dev Tools** of [Kibana](https://www.elastic.co/guide/en/kibana/current/docker.html) are invaluable for running and testing Elasticsearch queries.
 - The `Dockerfile` is structured in a way that the actual application code is in the last layer so rebuilding the image with only code changes utilizes the build cache for everything else and will take just 2-3 secs.
 - Take a look at the `deploy.sh` file. I have my local DNS resolve `tubearchivist.local` to the IP of the VM for convenience. To deploy the latest changes and rebuild the application to the testing VM run:
 ```bash
@@ -30,6 +29,11 @@ This is my setup I have landed on, YMMV:
 - The `test` argument takes another optional argument to build for a specific architecture valid options are: `amd64`, `arm64` and `multi`, default is `amd64`.
 - This `deploy.sh` file is not meant to be universally usable for every possible environment but could serve as an idea on how to automatically rebuild containers to test changes - customize to your liking. 
 
+## Working with Elasticsearch
+Additionally to the required services as listed in the example docker-compose file, the **Dev Tools** of [Kibana](https://www.elastic.co/guide/en/kibana/current/docker.html) are invaluable for running and testing Elasticsearch queries.  
+
+If you want to run queries in on the Elasticsearch container directly from your host with for example `curl` or something like *postman*, you might want to **publish** the port 9200 instead of just **exposing** it.
+
 ## Implementing a new feature
 
 Do you see anything on the roadmap that you would like to take a closer look at but you are not sure, what's the best way to tackle that? Or anything not on there yet you'd like to implement but are not sure how? Open up an issue and we try to find a solution together.

README.md

@@ -48,10 +48,12 @@ Take a look at the example `docker-compose.yml` file provided. Tube Archivist de
 ### Tube Archivist
 The main Python application that displays and serves your video collection, built with Django.
   - Serves the interface on port `8000`
-  - Needs a mandatory volume for the video archive at **/youtube**
+  - Needs a volume for the video archive at **/youtube**
-  - And another recommended volume to save the cache for thumbnails and artwork at **/cache**.
+  - And another volume to save application data at **/cache**.
   - The environment variables `ES_URL` and `REDIS_HOST` are needed to tell Tube Archivist where Elasticsearch and Redis respectively are located.
-  - The environment variables `HOST_UID` and `HOST_GID` allows Tube Archivist to `chown` the video files to the main host system user instead of the container user.
+  - The environment variables `HOST_UID` and `HOST_GID` allows Tube Archivist to `chown` the video files to the main host system user instead of the container user. Those two variables are optional, not setting them will disable that functionality. That might be needed if the underlying filesystem doesn't support `chown` like *NFS*. 
+  - Change the environment variables `TA_USERNAME` and `TA_PASSWORD` to create the initial credentials. 
+  - `ELASTIC_PASSWORD` is for the password for Elasticsearch. The environment variable `ELASTIC_USER` is optional, should you want to change the username from the default *elastic*.
 
 ### Elasticsearch
 Stores video meta data and makes everything searchable. Also keeps track of the download queue.
@@ -63,7 +65,7 @@ Follow the [documentation](https://www.elastic.co/guide/en/elasticsearch/referen
 ### Redis JSON
 Functions as a cache and temporary link between the application and the file system. Used to store and display messages and configuration variables.
   - Needs to be accessible over the default port `6379`
-  - Takes an optional volume at **/data** to make your configuration changes permanent.
+  - Needs a volume at **/data** to make your configuration changes permanent.
 
 ### Redis on a custom port
 For some architectures it might be required to run Redis JSON on a nonstandard port. To for example change the Redis port to **6380**, set the following values:
@@ -80,7 +82,8 @@ You will see the current version number of **Tube Archivist** in the footer of t
 - **arm64**: Newest Tube Archivist container is multi arch, so is Elasticsearch. RedisJSON doesn't offer arm builds, you can use `bbilly1/rejson`, an unofficial rebuild for arm64.
   - NOTE: This is untested, looking for feedback.
 - **Synology**: There is a [discussion thread](https://github.com/bbilly1/tubearchivist/discussions/48) with Synology installation instructions.
-- **Unraid**: There is a [template](https://github.com/pairofcrocs/unraid-templates/) and forum [thread](https://forums.unraid.net/topic/114073-support-crocs-tube-archivist/) with additional information.
+- **Unraid**: The three containers needed are all in the Community Applications. First install `TubeArchivist RedisJSON` followed by `TubeArchivist ES`, and finally you can install `TubeArchivist`. If you have unraid specific issues, report those to the [support thread](https://forums.unraid.net/topic/114073-support-crocs-tube-archivist/ "support thread").
+
 
 ## Potential pitfalls
 ### vm.max_map_count
@@ -105,7 +108,10 @@ chown 1000:0 /path/to/mount/point
 This will match the permissions with the **UID** and **GID** of elasticsearch within the container and should fix the issue.
 
 ## Getting Started
-1. Go through the **settings** page and look at the available options. Particularly set *Download Format* to your desired video quality before downloading. **Tube Archivist** downloads the best available quality by default.
+1. Go through the **settings** page and look at the available options. Particularly set *Download Format* to your desired video quality before downloading. **Tube Archivist** downloads the best available quality by default. To support iOS or MacOS a compatible format must be specified. For example:
+```
+bestvideo[VCODEC=avc1]+bestaudio[ACODEC=mp4a]/mp4
+```
 2. Subscribe to some of your favorite YouTube channels on the **channels** page. 
 3. On the **downloads** page, click on *Rescan subscriptions* to add videos from the subscribed channels to your Download queue or click on *Add to download queue* to manually add Video IDs, links, channels or playlists.
 4. Click on *Start download* and let **Tube Archivist** to it's thing. 
@@ -115,11 +121,13 @@ This will match the permissions with the **UID** and **GID** of elasticsearch wi
 This should be considered as a **minimal viable product**, there is an extensive list of future functions and improvements planned.
 
 ### Functionality
-- [ ] Access control
 - [ ] User roles
 - [ ] Create playlists
 - [ ] Podcast mode to serve channel as mp3
 - [ ] Implement [PyFilesystem](https://github.com/PyFilesystem/pyfilesystem2) for flexible video storage
+- [ ] Optional automatic deletion of watched items after a specified time
+- [ ] Subtitle download & indexing
+- [X] Access control [2021-11-01]
 - [X] Delete videos and channel [2021-10-16]
 - [X] Add thumbnail embed option [2021-10-16]
 - [X] Un-ignore videos [2021-10-03]
@@ -138,7 +146,8 @@ This should be considered as a **minimal viable product**, there is an extensive
 ## Known limitations
 - Video files created by Tube Archivist need to be **mp4** video files for best browser compatibility.
 - Every limitation of **yt-dlp** will also be present in Tube Archivist. If **yt-dlp** can't download or extract a video for any reason, Tube Archivist won't be able to either.
-- For now this is meant to be run in a trusted network environment. There is *no* security.
+- For now this is meant to be run in a trusted network environment. Not everything is properly authenticated.
+- There is currently no flexibility in naming of the media files.
 
 
 ## Donate

deploy.sh

@@ -10,6 +10,9 @@
 # docker buildx use tubearchivist
 # docker buildx inspect --bootstrap
 
+# more details:
+# https://github.com/bbilly1/tubearchivist/issues/6
+
 set -e
 
 function sync_blackhole {
@@ -119,20 +122,14 @@ function sync_docker {
     printf "\ncreate new version:\n"
     read -r VERSION
 
-    # start build
+    echo "build and push $VERSION?"
-    sudo docker buildx build --platform linux/amd64,linux/arm64 -t bbilly1/tubearchivist:latest -t bbilly1/tubearchivist:"$VERSION" .
-
-    printf "\nlatest images:\n"
-    sudo docker image ls bbilly1/tubearchivist
-
-    echo "continue?"
     read -rn 1
 
-    # push to docker
+    # start build
-    echo "pushing multiarch latest and $VERSION:"
+    sudo docker buildx build \
-    sudo docker buildx build --platform linux/amd64,linux/arm64 -t bbilly1/tubearchivist:latest -t bbilly1/tubearchivist:"$VERSION" --push .
+        --platform linux/amd64,linux/arm64 \
-    #sudo docker push bbilly1/tubearchivist:latest
+        -t bbilly1/tubearchivist:latest \
-    #sudo docker push bbilly1/tubearchivist:"$VERSION"
+        -t bbilly1/tubearchivist:"$VERSION" --push .
 
     # create release tag
     echo "commits since last version:"

docker-compose.yml

@@ -15,6 +15,9 @@ services:
       - REDIS_HOST=archivist-redis
       - HOST_UID=1000
       - HOST_GID=1000
+      - TA_USERNAME=tubearchivist
+      - TA_PASSWORD=verysecret
+      - ELASTIC_PASSWORD=verysecret
     depends_on:
       - archivist-es
       - archivist-redis
@@ -22,8 +25,8 @@ services:
     image: redislabs/rejson:latest
     container_name: archivist-redis
     restart: always
-    ports:
+    expose:
-      - 6379:6379
+      - "6379"
     volumes:
       - ./volumes/tubearchivist/redis:/data
     depends_on:
@@ -33,6 +36,8 @@ services:
     container_name: archivist-es
     restart: always
     environment:
+      - "xpack.security.enabled=true"
+      - "ELASTIC_PASSWORD=verysecret"
       - "discovery.type=single-node"
       - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
     ulimits:
@@ -41,5 +46,5 @@ services:
         hard: -1
     volumes:
       - ./volumes/tubearchivist/es:/usr/share/elasticsearch/data
-    ports:
+    expose:
-      - 9200:9200
+      - "9200"

docs/Channels.md

@@ -10,9 +10,9 @@ Accessible at `/channel/` of your Tube Archivist, the **Overview Page** shows a
 The **Subscribe to Channels** button <img src="assets/icon-add.png?raw=true" alt="add icon" width="20px" style="margin:0 5px;"> opens a text field to subscribe to a channel. You have a few options:
 - Enter the YouTube channel ID, a 25 character alphanumeric string. For example *UCBa659QWEk1AI4Tg--mrJ2A*
 - Enter the URL to the channel page on YouTube. For example *https://www.youtube.com/channel/UCBa659QWEk1AI4Tg--mrJ2A*
+- Enter the channel name for example: *https://www.youtube.com/c/TomScottGo*.
 - Enter the video URL for any video and let Tube Archivist extract the channel ID for you. For example *https://www.youtube.com/watch?v=2tdiKTSdE9Y*
 - Add one per line.
-- **Note**: Adding a link to a YouTube channel name is not yet supported, for example: *https://www.youtube.com/c/TomScottGo* will fail.
 
 The search icon <img src="assets/icon-search.png?raw=true" alt="search icon" width="20px" style="margin:0 5px;"> opens a text box to search for indexed channel names. Possible matches will show as you type. 
 

docs/Downloads.md

@@ -18,9 +18,9 @@ The **Add to Download Queue** icon <img src="assets/icon-add.png?raw=true" alt="
 - Add a YouTube video ID. For example *2tdiKTSdE9Y*.
 - Add a link to a YouTube video by providing the shortened URL, for example *https://youtu.be/2tdiKTSdE9Y*.
 - Add a Channel ID or Channel URL to add every available video to the download queue. This will ignore the channel page size as described before and is meant for an initial download of the whole channel. You can still ignore selected videos before starting the download.
+- Add a channel name like for example *https://www.youtube.com/c/TomScottGo*.
 - Add a playlist ID or URL to add every available video in the list to the download queue, for example *https://www.youtube.com/playlist?list=PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha* or *PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha*. Note that when you add a link to a video in a playlist, Tube Archivist assumes you want to download only the specific video and not the whole playlist, for example *https://www.youtube.com/watch?v=CINVwWHlzTY&list=PL96C35uN7xGLLeET0dOWaKHkAlPsrkcha* will only add one video *CINVwWHlzTY* to the queue.
 - Add one link per line.
-- **Note**: Adding a link to a YouTube channel name is not yet supported, for example: *https://www.youtube.com/c/TomScottGo* will fail.
 
 ## The Download Queue
 Below the three buttons you find the download queue. New items will get added at the bottom of the queue, the next video to download once you click on **Start Download** will be the first in the list.

docs/Home.md

@@ -6,6 +6,7 @@ Table of contents:
 * [Channels](Channels): Browse your channels, handle subscriptions
 * [Downloads](Downloads): Scanning subscriptions, handle download queue
 * [Settings](Settings): All the configuration options
+* [Users](Users): User management admin interface
 
 ## Getting Started
 1. [Subscribe](Channels#channels-overview) to some of your favourite YouTube channels.

docs/Settings.md

@@ -17,6 +17,7 @@ Settings related to the channel management.
 Settings related to the download process.
 - **Download Limit**: Stop the download process after downloading the set quantity of videos.
 - **Download Speed Limit**: Set your download speed limit in KB/s. This will pass the option `--limit-rate` to yt-dlp.
+- **Throttled Rate Limit**: Restart download if the download speed drops below this value in KB/s. This will pass the option `--throttled-rate` to yt-dlp. Using this option might have a negative effect if you have an unstable or slow internet connection.
 - **Sleep Interval**: Time in seconds to sleep between requests to YouTube. It's a good idea to set this to **3** seconds. Might be necessary to avoid throttling.
 
 ## Download Format
@@ -50,6 +51,9 @@ Detect the YouTube ID from filename, this accepts the default yt-dlp naming conv
 - Maybe start with a subset of your files to import to make sure everything goes well...
 - Follow the logs to monitor progress and errors: `docker-compose logs -f tubearchivist`.
 
+## Embed thumbnails into media file
+This will write or overwrite all thumbnails in the media file using the downloaded thumbnail. This is only necessary if you didn't download the files with the option *Embed Thumbnail* enabled or want to make sure all media files get the newest thumbnail. Follow the docker-compose logs to monitor progress.
+
 ## Backup Database
 This will backup your metadata into a zip file. The file will get stored at *cache/backup* and will contain the necessary files to restore the Elasticsearch index formatted **nd-json** files plus a complete export of the index in a set of conventional **json** files.  
 

docs/Users.md

@@ -0,0 +1,20 @@
+# User Management
+
+For now, **Tube Archivist** is a single user application. You can create multiple users with different names and passwords, they will share the same videos and permissions but some interface configurations are on a per user basis. *More is on the roadmap*.
+
+## Superuser
+The first user gets created with the environment variables **TA_USERNAME** and **TA_PASSWORD** from your docker-compose file. That first user will automatically have *superuser* privileges. 
+
+## Admin Interface
+When logged in from your *superuser* account, you are able to access the admin interface from the settings page or at `/admin/`. This interface holds all functionality for user management.
+
+## Create additional users
+From the admin interface when you click on *Accounts* you will get a list of all users. From there you can create additional users by clicking on *Add Account*, provide a name and confirm password and click on *Save* to create the user.
+
+## Changing users
+You can delete or change permissions and password of a user by clicking on the username from the *Accounts* list page and follow the interface from there. Changing the password of the *superuser* here will overwrite the password originally set with the environment variables.
+
+## Reset
+Delete all user configurations by deleting the file `cache/db.sqlite3` and restart the container. This will create the superuser again from the environment variables.
+
+NOTE: Future improvements here will most likely require such a reset.

run.sh

@@ -1,19 +1,38 @@
 #!/bin/bash
 # startup script inside the container for tubearchivist
 
+# check environment
+if [[ -z "$ELASTIC_USER" ]]; then
+    export ELASTIC_USER=elastic
+fi
+
+ENV_VARS=("TA_USERNAME" "TA_PASSWORD" "ELASTIC_PASSWORD" "ELASTIC_USER")
+for each in "${ENV_VARS[@]}"; do
+    if ! [[ -v $each ]]; then
+        echo "missing environment variable $each"
+        exit 1
+    fi
+done
+
+# wait for elasticsearch
 counter=0
-until curl "$ES_URL" -fs; do
+until curl -u "$ELASTIC_USER":"$ELASTIC_PASSWORD" "$ES_URL" -fs; do
     echo "waiting for elastic search to start"
     counter=$((counter+1))
     if [[ $counter -eq 12 ]]; then
-        # fail after 1 min
+        # fail after 2 min
         echo "failed to connect to elastic search, exiting..."
         exit 1
     fi
-    sleep 5
+    sleep 10
 done
 
+# start python application
+python manage.py makemigrations
 python manage.py migrate
+export DJANGO_SUPERUSER_PASSWORD=$TA_PASSWORD && \
+    python manage.py createsuperuser --noinput --name "$TA_USERNAME"
+
 python manage.py collectstatic --noinput -c
 nginx &
 celery -A home.tasks worker --loglevel=INFO &

tubearchivist/config/settings.py

@@ -10,9 +10,12 @@ For the full list of settings and their values, see
 https://docs.djangoproject.com/en/3.2/ref/settings/
 """
 
-from os import environ
+import hashlib
+from os import environ, path
 from pathlib import Path
 
+from home.src.config import AppConfig
+
 # Build paths inside the project like this: BASE_DIR / 'subdir'.
 BASE_DIR = Path(__file__).resolve().parent.parent
 
@@ -20,8 +23,8 @@ BASE_DIR = Path(__file__).resolve().parent.parent
 # Quick-start development settings - unsuitable for production
 # See https://docs.djangoproject.com/en/3.2/howto/deployment/checklist/
 
-# SECURITY WARNING: keep the secret key used in production secret!
+PW_HASH = hashlib.sha256(environ.get("TA_PASSWORD").encode())
-SECRET_KEY = "Fvid^aUL6LohRZz*kZFvq85B&JW&kB9o*#jdzWsdWE8*XkCLR8"
+SECRET_KEY = PW_HASH.hexdigest()
 
 # SECURITY WARNING: don't run with debug turned on in production!
 DEBUG = bool(environ.get("DJANGO_DEBUG"))
@@ -78,10 +81,12 @@ WSGI_APPLICATION = "config.wsgi.application"
 # Database
 # https://docs.djangoproject.com/en/3.2/ref/settings/#databases
 
+CACHE_DIR = AppConfig().config["application"]["cache_dir"]
+DB_PATH = path.join(CACHE_DIR, "db.sqlite3")
 DATABASES = {
     "default": {
         "ENGINE": "django.db.backends.sqlite3",
-        "NAME": BASE_DIR / "db.sqlite3",
+        "NAME": DB_PATH,
     }
 }
 
@@ -104,18 +109,16 @@ AUTH_PASSWORD_VALIDATORS = [
     },
 ]
 
+AUTH_USER_MODEL = "home.Account"
+
 
 # Internationalization
 # https://docs.djangoproject.com/en/3.2/topics/i18n/
 
 LANGUAGE_CODE = "en-us"
-
 TIME_ZONE = "UTC"
-
 USE_I18N = True
-
 USE_L10N = True
-
 USE_TZ = True
 
 
@@ -123,22 +126,14 @@ USE_TZ = True
 # https://docs.djangoproject.com/en/3.2/howto/static-files/
 
 STATIC_URL = "/static/"
-
-# STATICFILES_DIRS = [
-#     str(BASE_DIR.joinpath('static')),
-#     '/cache/'
-# ]
-
-# STATIC_URL = '/static/'
 STATICFILES_DIRS = (str(BASE_DIR.joinpath("static")),)
-# MEDIA_ROOT = str(BASE_DIR.joinpath('media'))
-# MEDIA_URL = '/media/'
-
 STATIC_ROOT = str(BASE_DIR.joinpath("staticfiles"))
-
 STATICFILES_STORAGE = "whitenoise.storage.CompressedManifestStaticFilesStorage"
 
 # Default primary key field type
 # https://docs.djangoproject.com/en/3.2/ref/settings/#default-auto-field
 
 DEFAULT_AUTO_FIELD = "django.db.models.BigAutoField"
+
+LOGIN_URL = "/login/"
+LOGOUT_REDIRECT_URL = "/login/"

tubearchivist/home/admin.py

@@ -1,3 +1,36 @@
-from django.contrib import admin  # noqa: F401 - Unused import
+"""custom admin classes"""
 
-# Register your models here.
+from django.contrib import admin
+from django.contrib.auth.admin import UserAdmin as BaseUserAdmin
+
+from .models import Account
+
+
+class HomeAdmin(BaseUserAdmin):
+    """register in admin page"""
+
+    list_display = ("name", "is_staff", "is_superuser")
+    list_filter = ("is_superuser",)
+
+    fieldsets = (
+        (None, {"fields": ("is_staff", "is_superuser", "password")}),
+        ("Personal info", {"fields": ("name",)}),
+        ("Groups", {"fields": ("groups",)}),
+        ("Permissions", {"fields": ("user_permissions",)}),
+    )
+    add_fieldsets = (
+        (
+            None,
+            {"fields": ("is_staff", "is_superuser", "password1", "password2")},
+        ),
+        ("Personal info", {"fields": ("name",)}),
+        ("Groups", {"fields": ("groups",)}),
+        ("Permissions", {"fields": ("user_permissions",)}),
+    )
+
+    search_fields = ("name",)
+    ordering = ("name",)
+    filter_horizontal = ()
+
+
+admin.site.register(Account, HomeAdmin)

tubearchivist/home/config.json

@@ -1,6 +1,7 @@
 {
     "archive": {
-        "sort": "published",
+        "sort_by": "published",
+        "sort_order": "desc",
         "page_size": 12
     },
     "default_view": {
@@ -19,7 +20,8 @@
         "sleep_interval": 3,
         "format": false,
         "add_metadata": false,
-        "add_thumbnail": false
+        "add_thumbnail": false,
+        "throttledratelimit": false
     },
     "application": {
         "app_root": "/app",

tubearchivist/home/forms.py

@@ -0,0 +1,114 @@
+"""functionality:
+- hold all form classes used in the views
+"""
+
+from django import forms
+from django.contrib.auth.forms import AuthenticationForm
+from django.forms.widgets import PasswordInput, TextInput
+
+
+class CustomAuthForm(AuthenticationForm):
+    """better styled login form"""
+
+    username = forms.CharField(
+        widget=TextInput(attrs={"placeholder": "Username"}), label=False
+    )
+    password = forms.CharField(
+        widget=PasswordInput(attrs={"placeholder": "Password"}), label=False
+    )
+
+
+class UserSettingsForm(forms.Form):
+    """user configurations values"""
+
+    CHOICES = [
+        ("", "-- change color scheme --"),
+        ("dark", "Dark"),
+        ("light", "Light"),
+    ]
+
+    colors = forms.ChoiceField(
+        widget=forms.Select, choices=CHOICES, required=False
+    )
+    page_size = forms.IntegerField(required=False)
+
+
+class ApplicationSettingsForm(forms.Form):
+    """handle all application settings"""
+
+    METADATA_CHOICES = [
+        ("", "-- change metadata embed --"),
+        ("0", "don't embed metadata"),
+        ("1", "embed metadata"),
+    ]
+
+    THUMBNAIL_CHOICES = [
+        ("", "-- change thumbnail embed --"),
+        ("0", "don't embed thumbnail"),
+        ("1", "embed thumbnail"),
+    ]
+
+    subscriptions_channel_size = forms.IntegerField(required=False)
+    downloads_limit_count = forms.IntegerField(required=False)
+    downloads_limit_speed = forms.IntegerField(required=False)
+    downloads_throttledratelimit = forms.IntegerField(required=False)
+    downloads_sleep_interval = forms.IntegerField(required=False)
+    downloads_format = forms.CharField(required=False)
+    downloads_add_metadata = forms.ChoiceField(
+        widget=forms.Select, choices=METADATA_CHOICES, required=False
+    )
+    downloads_add_thumbnail = forms.ChoiceField(
+        widget=forms.Select, choices=THUMBNAIL_CHOICES, required=False
+    )
+
+
+class VideoSearchForm(forms.Form):
+    """search videos form"""
+
+    searchInput = forms.CharField(
+        label="Search your videos",
+        widget=forms.TextInput(attrs={"autocomplete": "off"}),
+    )
+
+
+class ChannelSearchForm(forms.Form):
+    """search for channels"""
+
+    searchInput = forms.CharField(
+        label="",
+        widget=forms.TextInput(
+            attrs={
+                "oninput": "searchChannels(this.value)",
+                "autocomplete": "off",
+                "list": "resultBox",
+            }
+        ),
+    )
+
+
+class AddToQueueForm(forms.Form):
+    """text area form to add to downloads"""
+
+    vid_url = forms.CharField(
+        label=False,
+        widget=forms.Textarea(
+            attrs={
+                "rows": 4,
+                "placeholder": "Enter Video Urls or IDs here...",
+            }
+        ),
+    )
+
+
+class SubscribeToChannelForm(forms.Form):
+    """text area form to subscribe to multiple channels"""
+
+    subscribe = forms.CharField(
+        label=False,
+        widget=forms.Textarea(
+            attrs={
+                "rows": 3,
+                "placeholder": "Input channel ID, URL or Video of a channel",
+            }
+        ),
+    )

tubearchivist/home/models.py

@@ -1,3 +1,53 @@
-from django.db import models  # noqa: F401 - Unused import
+"""custom models"""
+from django.contrib.auth.models import (
+    AbstractBaseUser,
+    BaseUserManager,
+    PermissionsMixin,
+)
+from django.db import models
 
-# Create your models here.
+
+class AccountManager(BaseUserManager):
+    """manage user creation methods"""
+
+    use_in_migrations = True
+
+    def _create_user(self, name, password, **extra_fields):
+        """create regular user private"""
+        values = [name, password]
+        field_value_map = dict(zip(self.model.REQUIRED_FIELDS, values))
+        for field_name, value in field_value_map.items():
+            if not value:
+                raise ValueError(f"The {field_name} value must be set")
+
+        user = self.model(name=name, **extra_fields)
+        user.set_password(password)
+        user.save(using=self._db)
+        return user
+
+    def create_user(self, name, password):
+        """create regular user public"""
+        return self._create_user(name, password)
+
+    def create_superuser(self, name, password, **extra_fields):
+        """create super user"""
+        extra_fields.setdefault("is_staff", True)
+        extra_fields.setdefault("is_superuser", True)
+
+        if extra_fields.get("is_staff") is not True:
+            raise ValueError("Superuser must have is_staff=True.")
+        if extra_fields.get("is_superuser") is not True:
+            raise ValueError("Superuser must have is_superuser=True.")
+
+        return self._create_user(name, password, **extra_fields)
+
+
+class Account(AbstractBaseUser, PermissionsMixin):
+    """handle account creation"""
+
+    name = models.CharField(max_length=150, unique=True)
+    is_staff = models.BooleanField(default=False)
+    objects = AccountManager()
+
+    USERNAME_FIELD = "name"
+    REQUIRED_FIELDS = ["password"]

tubearchivist/home/src/config.py

@@ -14,8 +14,10 @@ from home.src.helper import RedisArchivist
 class AppConfig:
     """handle user settings and application variables"""
 
-    def __init__(self):
+    def __init__(self, user_id=False):
+        self.user_id = user_id
         self.config = self.get_config()
+        self.colors = self.get_colors()
 
     def get_config(self):
         """get config from default file or redis if changed"""
@@ -23,6 +25,12 @@ class AppConfig:
         if not config:
             config = self.get_config_file()
 
+        if self.user_id:
+            key = f"{self.user_id}:page_size"
+            page_size = RedisArchivist().get_message(key)["status"]
+            if page_size:
+                config["archive"]["page_size"] = page_size
+
         config["application"].update(self.get_config_env())
         return config
 
@@ -39,11 +47,27 @@ class AppConfig:
     @staticmethod
     def get_config_env():
         """read environment application variables"""
+        host_uid_env = os.environ.get("HOST_UID")
+        if host_uid_env:
+            host_uid = int(host_uid_env)
+        else:
+            host_uid = False
+
+        host_gid_env = os.environ.get("HOST_GID")
+        if host_gid_env:
+            host_gid = int(host_gid_env)
+        else:
+            host_gid = False
+
+        es_pass = os.environ.get("ELASTIC_PASSWORD")
+        es_user = os.environ.get("ELASTIC_USER", default="elastic")
+
         application = {
             "REDIS_HOST": os.environ.get("REDIS_HOST"),
             "es_url": os.environ.get("ES_URL"),
-            "HOST_UID": int(os.environ.get("HOST_UID")),
+            "es_auth": (es_user, es_pass),
-            "HOST_GID": int(os.environ.get("HOST_GID")),
+            "HOST_UID": host_uid,
+            "HOST_GID": host_gid,
         }
 
         return application
@@ -70,11 +94,36 @@ class AppConfig:
                 elif to_write.isdigit():
                     to_write = int(to_write)
 
-                config_dict, config_value = key.split(".")
+                config_dict, config_value = key.split("_", maxsplit=1)
                 config[config_dict][config_value] = to_write
 
         RedisArchivist().set_message("config", config, expire=False)
 
+    @staticmethod
+    def set_user_config(form_post, user_id):
+        """set values in redis for user settings"""
+        for key, value in form_post.items():
+            to_write = value[0]
+            if len(to_write):
+                if to_write.isdigit():
+                    to_write = int(to_write)
+                message = {"status": to_write}
+                redis_key = f"{user_id}:{key}"
+                RedisArchivist().set_message(redis_key, message, expire=False)
+
+    def get_colors(self):
+        """overwrite config if user has set custom values"""
+        colors = False
+        if self.user_id:
+            col_dict = RedisArchivist().get_message(f"{self.user_id}:colors")
+            colors = col_dict["status"]
+
+        if not colors:
+            colors = self.config["application"]["colors"]
+
+        self.config["application"]["colors"] = colors
+        return colors
+
     def load_new_defaults(self):
         """check config.json for missing defaults"""
         default_config = self.get_config_file()

tubearchivist/home/src/download.py

@@ -29,8 +29,13 @@ class PendingList:
 
     CONFIG = AppConfig().config
     ES_URL = CONFIG["application"]["es_url"]
+    ES_AUTH = CONFIG["application"]["es_auth"]
     VIDEOS = CONFIG["application"]["videos"]
 
+    def __init__(self):
+        self.all_channel_ids = False
+        self.all_downloaded = False
+
     @staticmethod
     def parse_url_list(youtube_ids):
         """extract youtube ids from list"""
@@ -50,12 +55,14 @@ class PendingList:
             if url_type == "video":
                 missing_videos.append(url)
             elif url_type == "channel":
-                youtube_ids = ChannelSubscription().get_last_youtube_videos(
+                video_results = ChannelSubscription().get_last_youtube_videos(
                     url, limit=False
                 )
+                youtube_ids = [i[0] for i in video_results]
                 missing_videos = missing_videos + youtube_ids
             elif url_type == "playlist":
-                youtube_ids = playlist_extractor(url)
+                video_results = playlist_extractor(url)
+                youtube_ids = [i[0] for i in video_results]
                 missing_videos = missing_videos + youtube_ids
 
         return missing_videos
@@ -65,29 +72,41 @@ class PendingList:
         # check if channel is indexed
         channel_handler = ChannelSubscription()
         all_indexed = channel_handler.get_channels(subscribed_only=False)
-        all_channel_ids = [i["channel_id"] for i in all_indexed]
+        self.all_channel_ids = [i["channel_id"] for i in all_indexed]
         # check if already there
-        all_downloaded = self.get_all_downloaded()
+        self.all_downloaded = self.get_all_downloaded()
-        # loop
+
+        bulk_list, all_videos_added = self.build_bulk(missing_videos)
+        # add last newline
+        bulk_list.append("\n")
+        query_str = "\n".join(bulk_list)
+        headers = {"Content-type": "application/x-ndjson"}
+        url = self.ES_URL + "/_bulk"
+        request = requests.post(
+            url, data=query_str, headers=headers, auth=self.ES_AUTH
+        )
+        if not request.ok:
+            print(request)
+
+        return all_videos_added
+
+    def build_bulk(self, missing_videos):
+        """build the bulk lists"""
         bulk_list = []
         all_videos_added = []
-        for video in missing_videos:
+        counter = 1
-            if isinstance(video, str):
+        for youtube_id in missing_videos:
-                youtube_id = video
+            # check if already downloaded
-            elif isinstance(video, tuple):
+            if youtube_id in self.all_downloaded:
-                youtube_id = video[0]
-            if youtube_id in all_downloaded:
-                # skip already downloaded
                 continue
+
             video = self.get_youtube_details(youtube_id)
             # skip on download error
             if not video:
                 continue
 
-            if video["channel_id"] in all_channel_ids:
+            channel_indexed = video["channel_id"] in self.all_channel_ids
-                video["channel_indexed"] = True
+            video["channel_indexed"] = channel_indexed
-            else:
-                video["channel_indexed"] = False
             thumb_url = video["vid_thumb_url"]
             video["status"] = "pending"
             action = {"create": {"_id": youtube_id, "_index": "ta_download"}}
@@ -95,23 +114,19 @@ class PendingList:
             bulk_list.append(json.dumps(video))
             all_videos_added.append((youtube_id, thumb_url))
             # notify
+            progress = f"{counter}/{len(missing_videos)}"
             mess_dict = {
                 "status": "pending",
                 "level": "info",
-                "title": "Adding to download queue.",
+                "title": "Adding new videos to download queue.",
-                "message": "Processing IDs...",
+                "message": "Progress: " + progress,
             }
             RedisArchivist().set_message("progress:download", mess_dict)
-        # add last newline
+            if counter % 25 == 0:
-        bulk_list.append("\n")
+                print("adding to queue progress: " + progress)
-        query_str = "\n".join(bulk_list)
+            counter = counter + 1
-        headers = {"Content-type": "application/x-ndjson"}
-        url = self.ES_URL + "/_bulk"
-        request = requests.post(url, data=query_str, headers=headers)
-        if not request.ok:
-            print(request)
 
-        return all_videos_added
+        return bulk_list, all_videos_added
 
     @staticmethod
     def get_youtube_details(youtube_id):
@@ -155,7 +170,7 @@ class PendingList:
         headers = {"Content-type": "application/json"}
         # get PIT ID
         url = self.ES_URL + "/ta_download/_pit?keep_alive=1m"
-        response = requests.post(url)
+        response = requests.post(url, auth=self.ES_AUTH)
         json_data = json.loads(response.text)
         pit_id = json_data["id"]
         # query
@@ -170,7 +185,9 @@ class PendingList:
         all_pending = []
         all_ignore = []
         while True:
-            response = requests.get(url, data=query_str, headers=headers)
+            response = requests.get(
+                url, data=query_str, headers=headers, auth=self.ES_AUTH
+            )
             json_data = json.loads(response.text)
             all_hits = json_data["hits"]["hits"]
             if all_hits:
@@ -188,7 +205,12 @@ class PendingList:
                 break
         # clean up PIT
         query_str = json.dumps({"id": pit_id})
-        requests.delete(self.ES_URL + "/_pit", data=query_str, headers=headers)
+        requests.delete(
+            self.ES_URL + "/_pit",
+            data=query_str,
+            headers=headers,
+            auth=self.ES_AUTH,
+        )
         return all_pending, all_ignore
 
     def get_all_indexed(self):
@@ -196,7 +218,7 @@ class PendingList:
         headers = {"Content-type": "application/json"}
         # get PIT ID
         url = self.ES_URL + "/ta_video/_pit?keep_alive=1m"
-        response = requests.post(url)
+        response = requests.post(url, auth=self.ES_AUTH)
         json_data = json.loads(response.text)
         pit_id = json_data["id"]
         # query
@@ -210,7 +232,9 @@ class PendingList:
         url = self.ES_URL + "/_search"
         all_indexed = []
         while True:
-            response = requests.get(url, data=query_str, headers=headers)
+            response = requests.get(
+                url, data=query_str, headers=headers, auth=self.ES_AUTH
+            )
             json_data = json.loads(response.text)
             all_hits = json_data["hits"]["hits"]
             if all_hits:
@@ -224,7 +248,12 @@ class PendingList:
                 break
         # clean up PIT
         query_str = json.dumps({"id": pit_id})
-        requests.delete(self.ES_URL + "/_pit", data=query_str, headers=headers)
+        requests.delete(
+            self.ES_URL + "/_pit",
+            data=query_str,
+            headers=headers,
+            auth=self.ES_AUTH,
+        )
         return all_indexed
 
     def get_all_downloaded(self):
@@ -244,7 +273,7 @@ class PendingList:
     def delete_from_pending(self, youtube_id):
         """delete the youtube_id from ta_download"""
         url = f"{self.ES_URL}/ta_download/_doc/{youtube_id}"
-        response = requests.delete(url)
+        response = requests.delete(url, auth=self.ES_AUTH)
         if not response.ok:
             print(response.text)
 
@@ -266,7 +295,9 @@ class PendingList:
 
         headers = {"Content-type": "application/x-ndjson"}
         url = self.ES_URL + "/_bulk"
-        request = requests.post(url, data=query_str, headers=headers)
+        request = requests.post(
+            url, data=query_str, headers=headers, auth=self.ES_AUTH
+        )
         mess_dict = {
             "status": "ignore",
             "level": "info",
@@ -284,6 +315,7 @@ class ChannelSubscription:
     def __init__(self):
         config = AppConfig().config
         self.es_url = config["application"]["es_url"]
+        self.es_auth = config["application"]["es_auth"]
         self.channel_size = config["subscriptions"]["channel_size"]
 
     def get_channels(self, subscribed_only=True):
@@ -291,7 +323,7 @@ class ChannelSubscription:
         headers = {"Content-type": "application/json"}
         # get PIT ID
         url = self.es_url + "/ta_channel/_pit?keep_alive=1m"
-        response = requests.post(url)
+        response = requests.post(url, auth=self.es_auth)
         json_data = json.loads(response.text)
         pit_id = json_data["id"]
         # query
@@ -313,7 +345,9 @@ class ChannelSubscription:
         url = self.es_url + "/_search"
         all_channels = []
         while True:
-            response = requests.get(url, data=query_str, headers=headers)
+            response = requests.get(
+                url, data=query_str, headers=headers, auth=self.es_auth
+            )
             json_data = json.loads(response.text)
             all_hits = json_data["hits"]["hits"]
             if all_hits:
@@ -328,7 +362,12 @@ class ChannelSubscription:
                 break
         # clean up PIT
         query_str = json.dumps({"id": pit_id})
-        requests.delete(self.es_url + "/_pit", data=query_str, headers=headers)
+        requests.delete(
+            self.es_url + "/_pit",
+            data=query_str,
+            headers=headers,
+            auth=self.es_auth,
+        )
         return all_channels
 
     def get_last_youtube_videos(self, channel_id, limit=True):
@@ -364,7 +403,7 @@ class ChannelSubscription:
                 {
                     "status": "rescan",
                     "level": "info",
-                    "title": "Rescanning: Looking for new videos.",
+                    "title": "Scanning channels: Looking for new videos.",
                     "message": f"Progress: {counter}/{len(all_channels)}",
                 },
             )
@@ -394,7 +433,9 @@ class ChannelSubscription:
             url = self.es_url + "/ta_channel/_update/" + channel_id
             payload = json.dumps({"doc": channel_dict})
         # update channel
-        request = requests.post(url, data=payload, headers=headers)
+        request = requests.post(
+            url, data=payload, headers=headers, auth=self.es_auth
+        )
         if not request.ok:
             print(request.text)
         # sync to videos
@@ -515,6 +556,11 @@ class VideoDownloader:
             obs["format"] = self.config["downloads"]["format"]
         if self.config["downloads"]["limit_speed"]:
             obs["ratelimit"] = self.config["downloads"]["limit_speed"] * 1024
+
+        throttle = self.config["downloads"]["throttledratelimit"]
+        if throttle:
+            obs["throttledratelimit"] = throttle * 1024
+
         external = False
         if external:
             obs["external_downloader"] = "aria2c"
@@ -579,7 +625,8 @@ class VideoDownloader:
         new_folder = os.path.join(videos, channel_name)
         if not os.path.exists(new_folder):
             os.makedirs(new_folder)
-            os.chown(new_folder, host_uid, host_gid)
+            if host_uid and host_gid:
+                os.chown(new_folder, host_uid, host_gid)
         # find real filename
         cache_dir = self.config["application"]["cache_dir"]
         all_cached = ignore_filelist(os.listdir(cache_dir + "/download/"))
@@ -590,12 +637,14 @@ class VideoDownloader:
         new_file_path = os.path.join(videos, vid_dict["media_url"])
         # move media file and fix permission
         shutil.move(old_file_path, new_file_path)
-        os.chown(new_file_path, host_uid, host_gid)
+        if host_uid and host_gid:
+            os.chown(new_file_path, host_uid, host_gid)
 
     def delete_from_pending(self, youtube_id):
         """delete downloaded video from pending index if its there"""
         es_url = self.config["application"]["es_url"]
+        es_auth = self.config["application"]["es_auth"]
         url = f"{es_url}/ta_download/_doc/{youtube_id}"
-        response = requests.delete(url)
+        response = requests.delete(url, auth=es_auth)
         if not response.ok and not response.status_code == 404:
             print(response.text)

tubearchivist/home/src/helper.py

@@ -9,18 +9,20 @@ import re
 import string
 import subprocess
 import unicodedata
+from urllib.parse import parse_qs, urlparse
 
 import redis
 import requests
+import yt_dlp as youtube_dl
 
 
-def get_total_hits(index, es_url, match_field):
+def get_total_hits(index, es_url, es_auth, match_field):
     """get total hits from index"""
     headers = {"Content-type": "application/json"}
     data = {"query": {"match": {match_field: True}}}
     payload = json.dumps(data)
     url = f"{es_url}/{index}/_search?filter_path=hits.total"
-    request = requests.post(url, data=payload, headers=headers)
+    request = requests.post(url, data=payload, headers=headers, auth=es_auth)
     if not request.ok:
         print(request.text)
     total_json = json.loads(request.text)
@@ -51,33 +53,100 @@ def ignore_filelist(filelist):
     return cleaned
 
 
-def process_url_list(url_str):
+class UrlListParser:
-    """parse url_list to find valid youtube video or channel ids"""
+    """take a multi line string and detect valid youtube ids"""
-    to_replace = ["watch?v=", "playlist?list="]
-    url_list = re.split("\n+", url_str[0])
-    youtube_ids = []
-    for url in url_list:
-        if "/c/" in url or "/user/" in url:
-            raise ValueError("user name is not unique, use channel ID")
 
-        url_clean = url.strip().strip("/").split("/")[-1]
+    def __init__(self, url_str):
-        for i in to_replace:
+        self.url_list = [i.strip() for i in url_str.split()]
-            url_clean = url_clean.replace(i, "")
+
-        url_no_param = url_clean.split("&")[0]
+    def process_list(self):
-        str_len = len(url_no_param)
+        """loop through the list"""
+        youtube_ids = []
+        for url in self.url_list:
+            parsed = urlparse(url)
+            print(f"processing: {url}")
+            print(parsed)
+            if not parsed.netloc:
+                # is not a url
+                id_type = self.find_valid_id(url)
+                youtube_id = url
+            elif "youtube.com" not in url:
+                raise ValueError(f"{url} is not a youtube link")
+            elif parsed.path:
+                # is a url
+                youtube_id, id_type = self.detect_from_url(parsed)
+            else:
+                # not detected
+                raise ValueError(f"failed to detect {url}")
+
+            youtube_ids.append({"url": youtube_id, "type": id_type})
+
+        return youtube_ids
+
+    def detect_from_url(self, parsed):
+        """detect from parsed url"""
+        if parsed.netloc == "youtu.be":
+            # shortened
+            youtube_id = parsed.path.strip("/")
+            _ = self.find_valid_id(youtube_id)
+            return youtube_id, "video"
+
+        if parsed.query:
+            # detect from query string
+            query_parsed = parse_qs(parsed.query)
+            if "v" in query_parsed.keys():
+                youtube_id = query_parsed["v"][0]
+                _ = self.find_valid_id(youtube_id)
+                return youtube_id, "video"
+
+            if "list" in query_parsed.keys():
+                youtube_id = query_parsed["list"][0]
+                return youtube_id, "playlist"
+
+        if parsed.path.startswith("/channel/"):
+            # channel id in url
+            youtube_id = parsed.path.split("/")[2]
+            _ = self.find_valid_id(youtube_id)
+            return youtube_id, "channel"
+
+        # dedect channel with yt_dlp
+        youtube_id = self.extract_channel_name(parsed.geturl())
+        return youtube_id, "channel"
+
+    @staticmethod
+    def find_valid_id(id_str):
+        """dedect valid id from length of string"""
+        str_len = len(id_str)
         if str_len == 11:
-            link_type = "video"
+            id_type = "video"
         elif str_len == 24:
-            link_type = "channel"
+            id_type = "channel"
         elif str_len == 34:
-            link_type = "playlist"
+            id_type = "playlist"
         else:
             # unable to parse
-            raise ValueError("not a valid url: " + url)
+            raise ValueError("not a valid id_str: " + id_str)
 
-        youtube_ids.append({"url": url_no_param, "type": link_type})
+        return id_type
 
-    return youtube_ids
+    @staticmethod
+    def extract_channel_name(url):
+        """find channel id from channel name with yt-dlp help"""
+        obs = {
+            "default_search": "ytsearch",
+            "quiet": True,
+            "skip_download": True,
+            "extract_flat": True,
+            "playlistend": 0,
+        }
+        url_info = youtube_dl.YoutubeDL(obs).extract_info(url, download=False)
+        try:
+            channel_id = url_info["channel_id"]
+        except KeyError as error:
+            print(f"failed to extract channel id from {url}")
+            raise ValueError from error
+
+        return channel_id
 
 
 class RedisArchivist:
@@ -85,6 +154,7 @@ class RedisArchivist:
 
     REDIS_HOST = os.environ.get("REDIS_HOST")
     REDIS_PORT = os.environ.get("REDIS_PORT")
+    NAME_SPACE = "ta:"
 
     if not REDIS_PORT:
         REDIS_PORT = 6379
@@ -97,15 +167,19 @@ class RedisArchivist:
     def set_message(self, key, message, expire=True):
         """write new message to redis"""
         self.redis_connection.execute_command(
-            "JSON.SET", key, ".", json.dumps(message)
+            "JSON.SET", self.NAME_SPACE + key, ".", json.dumps(message)
         )
 
         if expire:
-            self.redis_connection.execute_command("EXPIRE", key, 20)
+            self.redis_connection.execute_command(
+                "EXPIRE", self.NAME_SPACE + key, 20
+            )
 
     def get_message(self, key):
         """get message dict from redis"""
-        reply = self.redis_connection.execute_command("JSON.GET", key)
+        reply = self.redis_connection.execute_command(
+            "JSON.GET", self.NAME_SPACE + key
+        )
         if reply:
             json_str = json.loads(reply)
         else:
@@ -115,18 +189,20 @@ class RedisArchivist:
 
     def del_message(self, key):
         """delete key from redis"""
-        response = self.redis_connection.execute_command("DEL", key)
+        response = self.redis_connection.execute_command(
+            "DEL", self.NAME_SPACE + key
+        )
         return response
 
     def get_lock(self, lock_key):
         """handle lock for task management"""
-        redis_lock = self.redis_connection.lock(lock_key)
+        redis_lock = self.redis_connection.lock(self.NAME_SPACE + lock_key)
         return redis_lock
 
     def get_dl_message(self, cache_dir):
         """get latest download progress message if available"""
         reply = self.redis_connection.execute_command(
-            "JSON.GET", "progress:download"
+            "JSON.GET", self.NAME_SPACE + "progress:download"
         )
         if reply:
             json_str = json.loads(reply)
@@ -164,12 +240,13 @@ class RedisQueue:
 
     REDIS_HOST = os.environ.get("REDIS_HOST")
     REDIS_PORT = os.environ.get("REDIS_PORT")
+    NAME_SPACE = "ta:"
 
     if not REDIS_PORT:
         REDIS_PORT = 6379
 
     def __init__(self, key):
-        self.key = key
+        self.key = self.NAME_SPACE + key
         self.conn = redis.Redis(host=self.REDIS_HOST, port=self.REDIS_PORT)
 
     def get_all(self):

tubearchivist/home/src/index.py

@@ -15,7 +15,7 @@ import requests
 import yt_dlp as youtube_dl
 from bs4 import BeautifulSoup
 from home.src.config import AppConfig
-from home.src.helper import DurationConverter, clean_string, process_url_list
+from home.src.helper import DurationConverter, UrlListParser, clean_string
 from home.src.thumbnails import ThumbManager
 
 
@@ -24,6 +24,7 @@ class YoutubeChannel:
 
     CONFIG = AppConfig().config
     ES_URL = CONFIG["application"]["es_url"]
+    ES_AUTH = CONFIG["application"]["es_auth"]
     CACHE_DIR = CONFIG["application"]["cache_dir"]
     VIDEOS = CONFIG["application"]["videos"]
 
@@ -51,7 +52,7 @@ class YoutubeChannel:
         """get from elastic search first if possible"""
         channel_id = self.channel_id
         url = f"{self.ES_URL}/ta_channel/_doc/{channel_id}"
-        response = requests.get(url)
+        response = requests.get(url, auth=self.ES_AUTH)
         if response.ok:
             channel_source = response.json()["_source"]
             self.source = "elastic"
@@ -63,7 +64,7 @@ class YoutubeChannel:
         channel_id = self.channel_id
         url = f"https://www.youtube.com/channel/{channel_id}/about?hl=en"
         cookies = {"CONSENT": "YES+xxxxxxxxxxxxxxxxxxxxxxxxxxx"}
-        response = requests.get(url, cookies=cookies)
+        response = requests.get(url, cookies=cookies, auth=self.ES_AUTH)
         if response.ok:
             channel_page = response.text
         else:
@@ -166,7 +167,7 @@ class YoutubeChannel:
     def upload_to_es(self):
         """upload channel data to elastic search"""
         url = f"{self.ES_URL}/ta_channel/_doc/{self.channel_id}"
-        response = requests.put(url, json=self.channel_dict)
+        response = requests.put(url, json=self.channel_dict, auth=self.ES_AUTH)
         print(f"added {self.channel_id} to es")
         if not response.ok:
             print(response.text)
@@ -183,14 +184,18 @@ class YoutubeChannel:
         data = {"description": channel_id, "processors": processors}
         payload = json.dumps(data)
         url = self.ES_URL + "/_ingest/pipeline/" + channel_id
-        request = requests.put(url, data=payload, headers=headers)
+        request = requests.put(
+            url, data=payload, headers=headers, auth=self.ES_AUTH
+        )
         if not request.ok:
             print(request.text)
         # apply pipeline
         data = {"query": {"match": {"channel.channel_id": channel_id}}}
         payload = json.dumps(data)
         url = self.ES_URL + "/ta_video/_update_by_query?pipeline=" + channel_id
-        request = requests.post(url, data=payload, headers=headers)
+        request = requests.post(
+            url, data=payload, headers=headers, auth=self.ES_AUTH
+        )
         if not request.ok:
             print(request.text)
 
@@ -211,7 +216,9 @@ class YoutubeChannel:
         }
         payload = json.dumps(data)
         url = self.ES_URL + "/ta_video/_delete_by_query"
-        response = requests.post(url, data=payload, headers=headers)
+        response = requests.post(
+            url, data=payload, headers=headers, auth=self.ES_AUTH
+        )
         if not response.ok:
             print(response.text)
 
@@ -220,17 +227,21 @@ class YoutubeChannel:
         print(f"deleting {self.channel_id} and all matching media files")
         folder_path = self.get_folder_path()
         print("delete all media files")
-        all_videos = os.listdir(folder_path)
+        try:
-        for video in all_videos:
+            all_videos = os.listdir(folder_path)
-            video_path = os.path.join(folder_path, video)
+            for video in all_videos:
-            os.remove(video_path)
+                video_path = os.path.join(folder_path, video)
-        os.rmdir(folder_path)
+                os.remove(video_path)
+            os.rmdir(folder_path)
+        except FileNotFoundError:
+            print(f"no videos found for {folder_path}")
+
         ThumbManager().delete_chan_thumb(self.channel_id)
 
         print("delete indexed videos")
         self.delete_es_videos()
         url = self.ES_URL + "/ta_channel/_doc/" + self.channel_id
-        response = requests.delete(url)
+        response = requests.delete(url, auth=self.ES_AUTH)
         if not response.ok:
             print(response.text)
 
@@ -240,6 +251,7 @@ class YoutubeVideo:
 
     CONFIG = AppConfig().config
     ES_URL = CONFIG["application"]["es_url"]
+    ES_AUTH = CONFIG["application"]["es_auth"]
     CACHE_DIR = CONFIG["application"]["cache_dir"]
     VIDEOS = CONFIG["application"]["videos"]
 
@@ -360,7 +372,7 @@ class YoutubeVideo:
     def get_es_data(self):
         """get current data from elastic search"""
         url = self.ES_URL + "/ta_video/_doc/" + self.youtube_id
-        response = requests.get(url)
+        response = requests.get(url, auth=self.ES_AUTH)
         if not response.ok:
             print(response.text)
         es_vid_dict = json.loads(response.text)
@@ -369,7 +381,7 @@ class YoutubeVideo:
     def upload_to_es(self):
         """upload channel data to elastic search"""
         url = f"{self.ES_URL}/ta_video/_doc/{self.youtube_id}"
-        response = requests.put(url, json=self.vid_dict)
+        response = requests.put(url, json=self.vid_dict, auth=self.ES_AUTH)
         if not response.ok:
             print(response.text)
 
@@ -380,7 +392,9 @@ class YoutubeVideo:
         url = f"{self.ES_URL}/ta_video/_update/{youtube_id}"
         data = {"script": "ctx._source.active = false"}
         json_str = json.dumps(data)
-        response = requests.post(url, data=json_str, headers=headers)
+        response = requests.post(
+            url, data=json_str, headers=headers, auth=self.ES_AUTH
+        )
         print(f"deactivated {youtube_id}")
         if not response.ok:
             print(response.text)
@@ -395,7 +409,7 @@ class YoutubeVideo:
         os.remove(to_delete)
         # delete from index
         url = f"{self.ES_URL}/ta_video/_doc/{self.youtube_id}"
-        response = requests.delete(url)
+        response = requests.delete(url, auth=self.ES_AUTH)
         if not response.ok:
             print(response.text)
         # delete thumbs from cache
@@ -407,6 +421,7 @@ class WatchState:
 
     CONFIG = AppConfig().config
     ES_URL = CONFIG["application"]["es_url"]
+    ES_AUTH = CONFIG["application"]["es_auth"]
     HEADERS = {"Content-type": "application/json"}
 
     def __init__(self, youtube_id):
@@ -435,9 +450,8 @@ class WatchState:
 
     def dedect_type(self):
         """find youtube id type"""
-        url_process = process_url_list([self.youtube_id])
+        url_process = UrlListParser(self.youtube_id).process_list()
         url_type = url_process[0]["type"]
-
         return url_type
 
     def mark_vid_watched(self, revert=False):
@@ -450,7 +464,9 @@ class WatchState:
             data["doc"]["player"]["watched"] = False
 
         payload = json.dumps(data)
-        request = requests.post(url, data=payload, headers=self.HEADERS)
+        request = requests.post(
+            url, data=payload, headers=self.HEADERS, auth=self.ES_AUTH
+        )
         if not request.ok:
             print(request.text)
 
@@ -472,7 +488,9 @@ class WatchState:
 
         payload = json.dumps(data)
         url = f"{es_url}/_ingest/pipeline/{youtube_id}"
-        request = requests.put(url, data=payload, headers=headers)
+        request = requests.put(
+            url, data=payload, headers=headers, auth=self.ES_AUTH
+        )
         if not request.ok:
             print(request.text)
             raise ValueError("failed to post ingest pipeline")
@@ -485,7 +503,9 @@ class WatchState:
         data = {"query": {"bool": {"must": must_list}}}
         payload = json.dumps(data)
         url = f"{es_url}/ta_video/_update_by_query?pipeline={youtube_id}"
-        request = requests.post(url, data=payload, headers=headers)
+        request = requests.post(
+            url, data=payload, headers=headers, auth=self.ES_AUTH
+        )
         if not request.ok:
             print(request.text)
 

tubearchivist/home/src/index_management.py

@@ -164,6 +164,7 @@ class ElasticIndex:
 
     CONFIG = AppConfig().config
     ES_URL = CONFIG["application"]["es_url"]
+    ES_AUTH = CONFIG["application"]["es_auth"]
     HEADERS = {"Content-type": "application/json"}
 
     def __init__(self, index_name, expected_map, expected_set):
@@ -176,7 +177,7 @@ class ElasticIndex:
         """check if index already exists and return mapping if it does"""
         index_name = self.index_name
         url = f"{self.ES_URL}/ta_{index_name}"
-        response = requests.get(url)
+        response = requests.get(url, auth=self.ES_AUTH)
         exists = response.ok
 
         if exists:
@@ -274,7 +275,9 @@ class ElasticIndex:
         query = {"source": {"index": source}, "dest": {"index": destination}}
         data = json.dumps(query)
         url = self.ES_URL + "/_reindex?refresh=true"
-        response = requests.post(url=url, data=data, headers=self.HEADERS)
+        response = requests.post(
+            url=url, data=data, headers=self.HEADERS, auth=self.ES_AUTH
+        )
         if not response.ok:
             print(response.text)
 
@@ -284,7 +287,7 @@ class ElasticIndex:
             url = f"{self.ES_URL}/ta_{self.index_name}_backup"
         else:
             url = f"{self.ES_URL}/ta_{self.index_name}"
-        response = requests.delete(url)
+        response = requests.delete(url, auth=self.ES_AUTH)
         if not response.ok:
             print(response.text)
 
@@ -301,7 +304,9 @@ class ElasticIndex:
         # create
         url = f"{self.ES_URL}/ta_{self.index_name}"
         data = json.dumps(payload)
-        response = requests.put(url=url, data=data, headers=self.HEADERS)
+        response = requests.put(
+            url=url, data=data, headers=self.HEADERS, auth=self.ES_AUTH
+        )
         if not response.ok:
             print(response.text)
 
@@ -319,9 +324,10 @@ class ElasticBackup:
         """export all documents of a single index"""
         headers = {"Content-type": "application/json"}
         es_url = self.config["application"]["es_url"]
+        es_auth = self.config["application"]["es_auth"]
         # get PIT ID
         url = f"{es_url}/ta_{index_name}/_pit?keep_alive=1m"
-        response = requests.post(url)
+        response = requests.post(url, auth=es_auth)
         json_data = json.loads(response.text)
         pit_id = json_data["id"]
         # build query
@@ -336,7 +342,9 @@ class ElasticBackup:
         # loop until nothing left
         all_results = []
         while True:
-            response = requests.get(url, data=query_str, headers=headers)
+            response = requests.get(
+                url, data=query_str, headers=headers, auth=es_auth
+            )
             json_data = json.loads(response.text)
             all_hits = json_data["hits"]["hits"]
             if all_hits:
@@ -350,7 +358,9 @@ class ElasticBackup:
                 break
         # clean up PIT
         query_str = json.dumps({"id": pit_id})
-        requests.delete(es_url + "/_pit", data=query_str, headers=headers)
+        requests.delete(
+            es_url + "/_pit", data=query_str, headers=headers, auth=es_auth
+        )
 
         return all_results
 
@@ -416,6 +426,7 @@ class ElasticBackup:
         """send bulk to es"""
         cache_dir = self.config["application"]["cache_dir"]
         es_url = self.config["application"]["es_url"]
+        es_auth = self.config["application"]["es_auth"]
         headers = {"Content-type": "application/x-ndjson"}
         file_path = os.path.join(cache_dir, file_name)
 
@@ -426,7 +437,9 @@ class ElasticBackup:
             return
 
         url = es_url + "/_bulk"
-        request = requests.post(url, data=query_str, headers=headers)
+        request = requests.post(
+            url, data=query_str, headers=headers, auth=es_auth
+        )
         if not request.ok:
             print(request.text)
 

tubearchivist/home/src/reindex.py

@@ -35,6 +35,7 @@ class Reindex:
         config = AppConfig().config
         self.sleep_interval = config["downloads"]["sleep_interval"]
         self.es_url = config["application"]["es_url"]
+        self.es_auth = config["application"]["es_auth"]
         self.refresh_interval = 90
         # scan
         self.video_daily, self.channel_daily = self.get_daily()
@@ -43,10 +44,12 @@ class Reindex:
 
     def get_daily(self):
         """get daily refresh values"""
-        total_videos = get_total_hits("ta_video", self.es_url, "active")
+        total_videos = get_total_hits(
+            "ta_video", self.es_url, self.es_auth, "active"
+        )
         video_daily = ceil(total_videos / self.refresh_interval * 1.2)
         total_channels = get_total_hits(
-            "ta_channel", self.es_url, "channel_active"
+            "ta_channel", self.es_url, self.es_auth, "channel_active"
         )
         channel_daily = ceil(total_channels / self.refresh_interval * 1.2)
         return (video_daily, channel_daily)
@@ -72,7 +75,9 @@ class Reindex:
         }
         query_str = json.dumps(data)
         url = self.es_url + "/ta_video/_search"
-        response = requests.get(url, data=query_str, headers=headers)
+        response = requests.get(
+            url, data=query_str, headers=headers, auth=self.es_auth
+        )
         if not response.ok:
             print(response.text)
         response_dict = json.loads(response.text)
@@ -100,7 +105,9 @@ class Reindex:
         }
         query_str = json.dumps(data)
         url = self.es_url + "/ta_channel/_search"
-        response = requests.get(url, data=query_str, headers=headers)
+        response = requests.get(
+            url, data=query_str, headers=headers, auth=self.es_auth
+        )
         if not response.ok:
             print(response.text)
         response_dict = json.loads(response.text)
@@ -208,6 +215,7 @@ class FilesystemScanner:
 
     CONFIG = AppConfig().config
     ES_URL = CONFIG["application"]["es_url"]
+    ES_AUTH = CONFIG["application"]["es_auth"]
     VIDEOS = CONFIG["application"]["videos"]
 
     def __init__(self):
@@ -329,7 +337,9 @@ class FilesystemScanner:
         # make the call
         headers = {"Content-type": "application/x-ndjson"}
         url = self.ES_URL + "/_bulk"
-        request = requests.post(url, data=query_str, headers=headers)
+        request = requests.post(
+            url, data=query_str, headers=headers, auth=self.ES_AUTH
+        )
         if not request.ok:
             print(request.text)
 
@@ -339,7 +349,7 @@ class FilesystemScanner:
             youtube_id = indexed[0]
             print(f"deleting {youtube_id} from index")
             url = self.ES_URL + "/ta_video/_doc/" + youtube_id
-            request = requests.delete(url)
+            request = requests.delete(url, auth=self.ES_AUTH)
             if not request.ok:
                 print(request.text)
 

tubearchivist/home/src/searching.py

@@ -12,6 +12,7 @@ from datetime import datetime
 
 import requests
 from home.src.config import AppConfig
+from home.src.helper import RedisArchivist
 from home.src.thumbnails import ThumbManager
 
 
@@ -20,6 +21,7 @@ class SearchHandler:
 
     CONFIG = AppConfig().config
     CACHE_DIR = CONFIG["application"]["cache_dir"]
+    ES_AUTH = CONFIG["application"]["es_auth"]
 
     def __init__(self, url, data):
         self.max_hits = None
@@ -29,9 +31,11 @@ class SearchHandler:
     def get_data(self):
         """get the data"""
         if self.data:
-            response = requests.get(self.url, json=self.data).json()
+            response = requests.get(
+                self.url, json=self.data, auth=self.ES_AUTH
+            ).json()
         else:
-            response = requests.get(self.url).json()
+            response = requests.get(self.url, auth=self.ES_AUTH).json()
 
         if "hits" in response.keys():
             self.max_hits = response["hits"]["total"]["value"]
@@ -178,13 +182,23 @@ class Pagination:
     figure out the pagination based on page size and total_hits
     """
 
-    def __init__(self, page_get, search_get=False):
+    def __init__(self, page_get, user_id, search_get=False):
-        config = AppConfig().config
+        self.user_id = user_id
-        self.page_size = config["archive"]["page_size"]
+        self.page_size = self.get_page_size()
         self.page_get = page_get
         self.search_get = search_get
         self.pagination = self.first_guess()
 
+    def get_page_size(self):
+        """get default or user modified page_size"""
+        key = f"{self.user_id}:page_size"
+        page_size = RedisArchivist().get_message(key)["status"]
+        if not page_size:
+            config = AppConfig().config
+            page_size = config["archive"]["page_size"]
+
+        return page_size
+
     def first_guess(self):
         """build first guess before api call"""
         page_get = self.page_get

tubearchivist/home/src/thumbnails.py

@@ -11,6 +11,7 @@ import home.src.download as download
 import requests
 from home.src.config import AppConfig
 from home.src.helper import RedisArchivist, ignore_filelist
+from mutagen.mp4 import MP4, MP4Cover
 from PIL import Image
 
 
@@ -18,6 +19,7 @@ class ThumbManager:
     """handle thumbnails related functions"""
 
     CONFIG = AppConfig().config
+    MEDIA_DIR = CONFIG["application"]["videos"]
     CACHE_DIR = CONFIG["application"]["cache_dir"]
     VIDEO_DIR = os.path.join(CACHE_DIR, "videos")
     CHANNEL_DIR = os.path.join(CACHE_DIR, "channels")
@@ -117,7 +119,11 @@ class ThumbManager:
             ),
         }
         if img_url:
-            response = requests.get(img_url, stream=True)
+            try:
+                response = requests.get(img_url, stream=True)
+            except ConnectionError:
+                sleep(5)
+                response = requests.get(img_url, stream=True)
             if not response.ok and not response.status_code == 404:
                 print("retry thumbnail download for " + img_url)
                 sleep(5)
@@ -136,16 +142,13 @@ class ThumbManager:
 
     def download_vid(self, missing_thumbs, notify=True):
         """download all missing thumbnails from list"""
-        total = len(missing_thumbs)
+        print(f"downloading {len(missing_thumbs)} thumbnails")
-        print(f"downloading {total} thumbnails")
+        counter = 1
-        # videos
-        # counter as update path from v0.0.5, remove in future version
-        counter = 0
         for youtube_id, thumb_url in missing_thumbs:
-            folder_name = youtube_id[0].lower()
+            folder_path = os.path.join(self.VIDEO_DIR, youtube_id[0].lower())
-            folder_path = os.path.join(self.VIDEO_DIR, folder_name)
+            thumb_path = os.path.join(
-            thumb_path_part = self.vid_thumb_path(youtube_id)
+                self.CACHE_DIR, self.vid_thumb_path(youtube_id)
-            thumb_path = os.path.join(self.CACHE_DIR, thumb_path_part)
+            )
 
             os.makedirs(folder_path, exist_ok=True)
             img_raw = self.get_raw_img(thumb_url, "video")
@@ -158,17 +161,18 @@ class ThumbManager:
 
             img_raw.convert("RGB").save(thumb_path)
 
+            progress = f"{counter}/{len(missing_thumbs)}"
             if notify:
                 mess_dict = {
                     "status": "pending",
                     "level": "info",
-                    "title": "Adding to download queue.",
+                    "title": "Downloading Thumbnails",
-                    "message": "Downloading Thumbnails...",
+                    "message": "Progress: " + progress,
                 }
                 RedisArchivist().set_message("progress:download", mess_dict)
 
-            if counter % 50 == 0 and counter != 0:
+            if counter % 25 == 0:
-                print(f"thumbnail progress: {counter}/{total}")
+                print("thumbnail progress: " + progress)
             counter = counter + 1
 
     def download_chan(self, missing_channels):
@@ -232,6 +236,47 @@ class ThumbManager:
                 youtube_id = thumb.rstrip(".jpg")
                 self.delete_vid_thumb(youtube_id)
 
+    def get_thumb_list(self):
+        """get list of mediafiles and matching thumbnails"""
+        all_indexed = download.PendingList().get_all_indexed()
+        video_list = []
+        for video in all_indexed:
+            youtube_id = video["_source"]["youtube_id"]
+            media_url = os.path.join(
+                self.MEDIA_DIR, video["_source"]["media_url"]
+            )
+            thumb_path = os.path.join(
+                self.CACHE_DIR, self.vid_thumb_path(youtube_id)
+            )
+            video_list.append(
+                {
+                    "media_url": media_url,
+                    "thumb_path": thumb_path,
+                }
+            )
+
+        return video_list
+
+    @staticmethod
+    def write_all_thumbs(video_list):
+        """rewrite the thumbnail into media file"""
+
+        counter = 1
+        for video in video_list:
+            # loop through all videos
+            media_url = video["media_url"]
+            thumb_path = video["thumb_path"]
+
+            mutagen_vid = MP4(media_url)
+            with open(thumb_path, "rb") as f:
+                mutagen_vid["covr"] = [
+                    MP4Cover(f.read(), imageformat=MP4Cover.FORMAT_JPEG)
+                ]
+            mutagen_vid.save()
+            if counter % 50 == 0:
+                print(f"thumbnail write progress {counter}/{len(video_list)}")
+            counter = counter + 1
+
 
 def validate_thumbnails():
     """check if all thumbnails are there and organized correctly"""

tubearchivist/home/tasks.py

@@ -9,7 +9,7 @@ import os
 from celery import Celery, shared_task
 from home.src.config import AppConfig
 from home.src.download import ChannelSubscription, PendingList, VideoDownloader
-from home.src.helper import RedisArchivist, RedisQueue
+from home.src.helper import RedisArchivist, RedisQueue, UrlListParser
 from home.src.index_management import backup_all_indexes, restore_from_backup
 from home.src.reindex import (
     ManualImport,
@@ -27,7 +27,7 @@ if not REDIS_PORT:
 
 os.environ.setdefault("DJANGO_SETTINGS_MODULE", "home.settings")
 app = Celery("tasks", broker=f"redis://{REDIS_HOST}:{REDIS_PORT}")
-app.config_from_object("django.conf:settings", namespace="CELERY")
+app.config_from_object("django.conf:settings", namespace="ta:")
 app.autodiscover_tasks()
 
 
@@ -166,3 +166,35 @@ def kill_dl(task_id):
 def rescan_filesystem():
     """check the media folder for mismatches"""
     scan_filesystem()
+
+
+@shared_task
+def re_sync_thumbs():
+    """sync thumbnails to mediafiles"""
+    handler = ThumbManager()
+    video_list = handler.get_thumb_list()
+    handler.write_all_thumbs(video_list)
+
+
+@shared_task
+def subscribe_to(url_str):
+    """take a list of urls to subscribe to"""
+    youtube_ids = UrlListParser(url_str).process_list()
+    for youtube_id in youtube_ids:
+        if youtube_id["type"] == "video":
+            to_sub = youtube_id["url"]
+            vid_details = PendingList().get_youtube_details(to_sub)
+            channel_id_sub = vid_details["channel_id"]
+        elif youtube_id["type"] == "channel":
+            channel_id_sub = youtube_id["url"]
+        else:
+            raise ValueError("failed to subscribe to: " + youtube_id)
+
+        ChannelSubscription().change_subscribe(
+            channel_id_sub, channel_subscribed=True
+        )
+        print("subscribed to: " + channel_id_sub)
+        # notify
+        RedisArchivist().set_message(
+            "progress:subscribe", {"status": "subscribing"}
+        )

tubearchivist/home/templates/home/base.html

@@ -45,10 +45,13 @@
             </div>
             <div class="nav-icons">
                 <a href="{% url 'about' %}">
-                    <img src="{% static 'img/icon-help.svg' %}" alt="help-icon">
+                    <img src="{% static 'img/icon-help.svg' %}" alt="help-icon" title="About">
                 </a>
                 <a href="{% url 'settings' %}">
-                    <img src="{% static 'img/icon-gear.svg' %}" alt="gear-icon">
+                    <img src="{% static 'img/icon-gear.svg' %}" alt="gear-icon" title="Settings">
+                </a>
+                <a href="{% url 'logout' %}">
+                    <img src="{% static 'img/icon-exit.svg' %}" alt="exit-icon" title="Logout">
                 </a>
             </div>
         </div>
@@ -96,7 +99,7 @@
     </div>
     <div class="footer">
         <div class="boxed-content">
-            <span>© 2021 The Tube Archivist v0.0.6 | <a href="https://github.com/bbilly1/tubearchivist" target="_blank">Github</a> | <a href="https://hub.docker.com/r/bbilly1/tubearchivist" target="_blank">Docker Hub</a></span>
+            <span>© 2021 The Tube Archivist v0.0.7 | <a href="https://github.com/bbilly1/tubearchivist" target="_blank">Github</a> | <a href="https://hub.docker.com/r/bbilly1/tubearchivist" target="_blank">Docker Hub</a></span>
         </div>
     </div>
 </body>

tubearchivist/home/templates/home/channel.html

@@ -7,24 +7,28 @@
 </div>
 <div class="info-box info-box-2">
     <div class="icon-text">
-        <img id="add-icon" onclick="showForm()" src="{% static 'img/icon-add.svg' %}" alt="add-icon">
+        {% if running == "subscribing" %}
-        <p>Subscribe to Channels</p>
+            <p>Subscribing in progress, refresh.</p>
-        <div class="show-form">
+        {% else %}
-            <form id="hidden-form" action="/channel/" method="post">
+            <img id="add-icon" onclick="showForm()" src="{% static 'img/icon-add.svg' %}" alt="add-icon">
-                {% csrf_token %}
+            <p>Subscribe to Channels</p>
-                <textarea rows="3" placeholder="Input channel ID, channel URL or Video of a channel" id="subscribe" name="subscribe"></textarea>
+            <div class="show-form">
-                <button type="submit">Subscribe</button>
+                <form id="hidden-form" action="/channel/" method="post">
-            </form>
+                    {% csrf_token %}
-        </div>
+                    {{ subscribe_form }}
+                    <button type="submit">Subscribe</button>
+                </form>
+            </div>
+        {% endif %}
     </div>
     <div class="search-form icon-text">
         <div class="search-icon">
             <img src="{% static 'img/icon-search.svg' %}" alt="search-icon" onclick="showSearch()">
-            <p>Search Channels</p>
+            <p>Search your Channels</p>
         </div>
         <form onSubmit="return channelRedirect();" id="search-box">
             {% csrf_token %}
-            <input name="videoSearch" list="resultBox" type="text" id="searchInput" autocomplete="off" oninput="searchChannels(this.value)">
+            {{ search_form }}
             <datalist id="resultBox">
             </datalist>
         </form>

tubearchivist/home/templates/home/channel_id.html

@@ -34,6 +34,11 @@
                 false
             {% endif %}
             </p>
+            <p>Channel id: {{ channel_info.channel_id }}</p>
+            <button onclick="deleteConfirm()" id="delete-item">Delete Channel</button>
+            <div class="delete-confirm" id="delete-button">
+                <span>Delete {{ channel_info.channel_name }} including all videos? </span><button class="danger-button" onclick="deleteChannel(this)" data-id="{{ channel_info.channel_id }}">Delete</button> <button onclick="cancelDelete()">Cancel</button>
+            </div>
         </div>
     </div>
     <div class="info-box-item">
@@ -47,11 +52,6 @@
                 <p>Total Videos archived: {{ max_hits }}</p>
                 <p>Watched: <button title="Mark all videos from {{ channel_info.channel_name }} as watched" type="button" id="{{ channel_info.channel_id }}" onclick="isWatched(this.id)">Mark as watched</button></p>
             {% endif %}
-            <p>Channel id: {{ channel_info.channel_id }}</p>
-            <button onclick="deleteConfirm()" id="delete-item">Delete Channel</button>
-            <div class="delete-confirm" id="delete-button">
-                <span>Delete {{ channel_info.channel_name }} including all videos? </span><button class="danger-button" onclick="deleteChannel(this)" data-id="{{ channel_info.channel_id }}">Delete</button> <button onclick="cancelDelete()">Cancel</button>
-            </div>
         </div>
     </div>
 </div>
@@ -63,12 +63,49 @@
         </div>
     </div>
 {% endif %}
-<div class="view-icons">
-    <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="grid" alt="grid view">
-    <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="list" alt="list view">
-</div>
 <div id="player" class="video-player"></div>
-<h2>Videos</h2>
+<div class="sort">
+    <p>Sort by <span class="settings-current">{{ sort_by }}</span>
+        <select name="sort" id="sort" onchange="sortChange(this.value)">
+            <option value="" disabled selected> -- change sort by -- </option>
+            <option value="published">date published</option>
+            <option value="downloaded">date downloaded</option>
+            <option value="views">views</option>
+            <option value="likes">likes</option>
+        </select>
+        <select name="sord-order" id="sort-order" onchange="sortChange(this.value)">
+            {% if sort_order == "asc" %}
+                <option value="asc" selected>asc</option>
+            {% else %}
+                <option value="asc">asc</option>
+            {% endif %}
+            {% if sort_order == "desc" %}
+                <option value="desc" selected>desc</option>
+            {% else %}
+                <option value="desc">desc</option>
+            {% endif %}
+        </select>
+    </p>
+</div>
+<div class="view-controls">
+    <div class="toggle">
+        <span>Hide watched videos:</span>
+        <div class="toggleBox">
+            <input 
+                id="hide_watched" onclick="toggleCheckbox(this)" type="checkbox"
+                {% if hide_watched %}
+                    checked
+                {% endif %}
+                >
+            <label for="" class="onbtn">On</label>
+            <label for="" class="ofbtn">Off</label>
+        </div>
+    </div>
+    <div class="view-icons">
+        <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="grid" alt="grid view">
+        <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="home" data-value="list" alt="list view">
+    </div>
+</div>
 <div class="video-list {{ view_style }}">
     {% if videos %}
         {% for video in videos %}

tubearchivist/home/templates/home/downloads.html

@@ -21,7 +21,8 @@
         <div class="show-form">
             <form id='hidden-form' action="/downloads/" method="post">
                 {% csrf_token %}
-                <textarea rows="4" placeholder="Enter Video Urls or IDs here..." id="vid-url" name="vid-url"></textarea>
+                <!-- <textarea rows="4" placeholder="Enter Video Urls or IDs here..." id="vid-url" name="vid-url"></textarea> -->
+                {{ add_form }}
                 <button type="submit">Add to download queue</button>
             </form>
         </div>
@@ -58,13 +59,14 @@
             <div class="dl-item {{ view_style }}" id="dl-{{ video.youtube_id }}">
                 <div class="dl-thumb {{ view_style }}">
                     <img src="/cache/{{ video.vid_thumb_url }}" alt="video_thumb">
+                    {% if show_ignored_only %}
+                        <span>ignored</span>
+                    {% else %}
+                        <span>queued</span>
+                    {% endif %}
                 </div>
                 <div class="dl-desc {{ view_style }}">
-                    {% if show_ignored_only %}
+                    <h3>{{ video.title }}</h3>
-                        <h3>Ignore: {{ video.title }}</h3>
-                    {% else %}
-                        <h3>Download: {{ video.title }}</h3>
-                    {% endif %}
                     {% if video.channel_indexed %}
                         <a href="{% url 'channel_id' video.channel_id %}">{{ video.channel_name }}</a>
                     {% else %}

tubearchivist/home/templates/home/home.html

@@ -6,11 +6,25 @@
 </div>
 <div class="info-box info-box-2">
     <div class="sort">
-        <p>Sort order from <span class="settings-current">{{ sortorder }}</span>
+        <p>Sort by <span class="settings-current">{{ sort_by }}</span>
             <select name="sort" id="sort" onchange="sortChange(this.value)">
-                <option value="" disabled selected> -- change sort order -- </option>
+                <option value="" disabled selected> -- change sort by -- </option>
                 <option value="published">date published</option>
                 <option value="downloaded">date downloaded</option>
+                <option value="views">views</option>
+                <option value="likes">likes</option>
+            </select>
+            <select name="sord-order" id="sort-order" onchange="sortChange(this.value)">
+                {% if sort_order == "asc" %}
+                    <option value="asc" selected>asc</option>
+                {% else %}
+                    <option value="asc">asc</option>
+                {% endif %}
+                {% if sort_order == "desc" %}
+                    <option value="desc" selected>desc</option>
+                {% else %}
+                    <option value="desc">desc</option>
+                {% endif %}
             </select>
         </p>
     </div>
@@ -20,7 +34,7 @@
         </div>
         <form action="/" method="POST" id="search-box">
             {% csrf_token %}
-            <input name="videoSearch" list="resultBox" type="text" id="searchInput" autocomplete="off">
+            {{ search_form }}
         </form>
     </div>
 </div>

tubearchivist/home/templates/home/login.html

@@ -0,0 +1,46 @@
+{% load static %}
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta http-equiv="X-UA-Compatible" content="IE=edge">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>TA | Welcome</title>
+    <link rel="stylesheet" type="text/css" href="{% static 'css/style.css' %}">
+    <link rel="shortcut icon" type="image/png" href="{% static 'favicon.ico' %}"/>
+    {% if colors == "dark" %}
+        <link rel="stylesheet" href="{% static 'css/dark.css' %}">
+    {% else %}
+        <link rel="stylesheet" href="{% static 'css/light.css' %}">
+    {% endif %}
+</head>
+<body>
+    <div class="boxed-content login-page">
+        {% if colors == 'dark' %}
+            <img src="{% static 'img/logo-tube-archivist-dark.png' %}" alt="tube-archivist-logo">
+        {% endif %}
+        {% if colors == 'light' %}
+            <img src="{% static 'img/logo-tube-archivist-light.png' %}" alt="tube-archivist-banner">
+        {% endif %}
+        <h1>Tube Archivist</h1>
+        <h2>Your Self Hosted YouTube Media Server</h2>
+        {% if form_error %}
+            <p class="danger-zone">Failed to login.</p>
+        {% endif %}
+        <form action="/login/" method="POST" name="login">
+            {% csrf_token %}
+            {% for field in form %}
+                {{ field }}<br>
+            {% endfor %}
+            <input type="hidden" name="next" value="{{ request.GET.next }}" />
+            <button type="submit">Login</button>
+        </form>
+        <p class="login-links"><span><a href="https://github.com/bbilly1/tubearchivist" target="_blank">Github</a></span> <span><a href="https://github.com/bbilly1/tubearchivist#donate" target="_blank">Donate</a></span></p>
+    </div>
+    <div class="footer-colors">
+        <div class="col-1"></div>
+        <div class="col-2"></div>
+        <div class="col-3"></div>
+    </div>
+</body>
+</html>

tubearchivist/home/templates/home/settings.html

@@ -1,20 +1,16 @@
 {% extends "home/base.html" %}
 {% block content %}
 <div class="title-bar">
-    <h1>Settings</h1>
+    <h1>User Configurations</h1>
 </div>
-<form action="/settings/" method="POST" name="settings-update">
+<form action="/settings/" method="POST" name="user-update">
     {% csrf_token %}
     <div class="settings-group">
         <h2>Color scheme</h2>
         <div class="settings-item">
             <p>Current color scheme: <span class="settings-current">{{ config.application.colors }}</span></p>
             <i>Select your preferred color scheme between dark and light mode.</i><br>
-            <select name="application.colors" id="application.colors">
+            {{ user_form.colors }}
-                <option value="" disabled selected> -- change color scheme -- </option>
-                <option value="dark">dark mode</option>
-                <option value="light">light mode</option>
-            </select>
         </div>
     </div>
     <div class="settings-group">
@@ -22,15 +18,22 @@
         <div class="settings-item">
             <p>Current page size: <span class="settings-current">{{ config.archive.page_size }}</span></p>
             <i>Result of videos showing in archive page</i><br>
-            <input type="number" name="archive.page_size" id="archive.page_size">
+            {{ user_form.page_size }}
         </div>
     </div>
+    <button type="submit" name="user-settings">Update User Configurations</button>
+</form>
+<div class="title-bar">
+    <h1>Application Configurations</h1>
+</div>
+<form action="/settings/" method="POST" name="application-update">
+    {% csrf_token %}
     <div class="settings-group">
         <h2 id="subscriptions">Subscriptions</h2>
         <div class="settings-item">
             <p>Current channel page size: <span class="settings-current">{{ config.subscriptions.channel_size }}</span></p>
             <i>Recent videos to check on check pending, max recommended 50.</i><br>
-            <input type="number" name="subscriptions.channel_size" id="subscriptions.channel_size">
+            {{ app_form.subscriptions_channel_size }}
         </div>
         <div class="settings-item">
             <p>Auto scan subscribed channels:</p>
@@ -46,17 +49,22 @@
         <div class="settings-item">
             <p>Current download limit: <span class="settings-current">{{ config.downloads.limit_count }}</span></p>
             <i>Limit the number of videos getting downloaded on every run. 0 (zero) to deactivate.</i><br>
-            <input type="number" name="downloads.limit_count" id="downloads.limit_count">
+            {{ app_form.downloads_limit_count }}
         </div>
         <div class="settings-item">
             <p>Current download speed limit: <span class="settings-current">{{ config.downloads.limit_speed }}</span></p>
             <i>Limit download speed. 0 (zero) to deactivate.</i><br>
-            <input type="number" name="downloads.limit_speed" id="downloads.limit_speed"><span>KB/sec</span>
+            {{ app_form.downloads_limit_speed }}
+        </div>
+        <div class="settings-item">
+            <p>Current throttled rate limit: <span class="settings-current">{{ config.downloads.throttledratelimit }}</span></p>
+            <i>Assume the download is being throttled below this speed and restart. 0 (zero) to deactivate, e.g. 100KB/sec</i><br>
+            {{ app_form.downloads_throttledratelimit }}
         </div>
         <div class="settings-item">
             <p>Current scraping sleep interval: <span class="settings-current">{{ config.downloads.sleep_interval }}</p>
             <i>Seconds to sleep between calls to YouTube. Might be necessary to avoid throttling. Recommended 3.</i><br>
-            <input type="number" name="downloads.sleep_interval" id="downloads.sleep_interval">
+            {{ app_form.downloads_sleep_interval }}
         </div>
         <div class="settings-item">
             <p>External downloader:</p>
@@ -73,32 +81,25 @@
             <ul>
                 <li><span class="settings-current">bestvideo[height<=720]+bestaudio/best[height<=720]</span>: best audio and max video height of 720p.</li>
                 <li><span class="settings-current">bestvideo[height<=1080]+bestaudio/best[height<=1080]</span>: best audio and max video height of 1080p.</li>
+                <li><span class="settings-current">bestvideo[height<=1080][VCODEC=avc1]+bestaudio[ACODEC=mp4a]/mp4</span>: Max 1080p video height with iOS compatible video and audio codecs.</li>
                 <li><span class="settings-current">0</span>: deactivate and download the best quality possible as decided by yt-dlp.</li>
             </ul>
             <i>Make sure your custom format gets merged into a single file. Check out the <a href="https://github.com/yt-dlp/yt-dlp#format-selection" target="_blank">documentation</a> for valid configurations.</i><br>
-            <input type="text" name="downloads.format" id="downloads.format">
+            {{ app_form.downloads_format }}
             <br>
         </div>
         <div class="settings-item">
             <p>Current metadata embed setting: <span class="settings-current">{{ config.downloads.add_metadata }}</span></p>
             <i>Metadata is not embedded into the downloaded files by default.</i><br>
-            <select name="downloads.add_metadata" id="downloads.add_metadata"">
+            {{ app_form.downloads_add_metadata }}
-                <option value="" disabled selected> -- change metadata embed -- </option>
-                <option value="0">don't embed metadata</option>
-                <option value="1">embed metadata</option>
-            </select>
         </div>
         <div class="settings-item">
             <p>Current thumbnail embed setting: <span class="settings-current">{{ config.downloads.add_thumbnail }}</span></p>
             <i>Embed thumbnail into the mediafile.</i><br>
-            <select name="downloads.add_thumbnail" id="downloads.add_thumbnail"">
+            {{ app_form.downloads_add_thumbnail }}
-                <option value="" disabled selected> -- change thumbnail embed -- </option>
-                <option value="0">don't embed thumbnail</option>
-                <option value="1">embed thumbnail</option>
-            </select>
         </div>
     </div>
-    <button type="submit">Update Settings</button>
+    <button type="submit" name="application-settings">Update Application Configurations</button>
 </form>
 <div class="title-bar">
     <h1>Actions</h1>
@@ -110,6 +111,13 @@
         <button onclick="manualImport()">Start import</button>
     </div>
 </div>
+<div class="settings-group">
+    <h2>Embed thumbnails into media file.</h2>
+    <p>Set extracted youtube thumbnail as cover art of the media file.</p>
+    <div id="re-embed">
+        <button onclick="reEmbed()">Start process</button>
+    </div>
+</div>
 <div class="settings-group">
     <h2>Backup database</h2>
     <p>Export your database to a zip file stored at <span class="settings-current">cache/backup</span>.</p>
@@ -133,4 +141,14 @@
         <button onclick="fsRescan()">Rescan filesystem</button>
     </div>
 </div>
+{% if request.user.is_superuser %}
+    <div class="title-bar">
+        <h1>Users</h1>
+    </div>
+    <div class="settings-group">
+        <h2>User Management</h2>
+        <p>Access the admin interface for basic user management functionality like adding and deleting users, changing passwords and more.</p>
+        <a href="/admin/"><button>Admin Interface</button></a>
+    </div>
+{% endif %}
 {% endblock content %}

tubearchivist/home/templates/home/video.html

@@ -9,7 +9,7 @@
     <video 
         src="/media/{{ video.media_url }}" 
         poster="/cache/{{ video.vid_thumb_url }}" controls preload="false" 
-        type='video/mp4' width="100%">
+        type='video/mp4' width="100%" playsinline>
     </video>
 </div>
 <div class="info-box info-box-3 padding-box">

tubearchivist/home/urls.py

@@ -1,5 +1,8 @@
 """ all home app urls """
 
+from django.conf import settings
+from django.contrib.auth.decorators import login_required
+from django.contrib.auth.views import LogoutView
 from django.urls import path
 from home.views import (
     AboutView,
@@ -7,24 +10,38 @@ from home.views import (
     ChannelView,
     DownloadView,
     HomeView,
+    LoginView,
     SettingsView,
     VideoView,
+    process,
+    progress,
 )
 
-from . import views
-
 urlpatterns = [
-    path("", HomeView.as_view(), name="home"),
+    path("", login_required(HomeView.as_view()), name="home"),
+    path("login/", LoginView.as_view(), name="login"),
+    path(
+        "logout/",
+        LogoutView.as_view(),
+        {"next_page": settings.LOGOUT_REDIRECT_URL},
+        name="logout",
+    ),
     path("about/", AboutView.as_view(), name="about"),
-    path("downloads/", DownloadView.as_view(), name="downloads"),
+    path(
-    path("settings/", SettingsView.as_view(), name="settings"),
+        "downloads/", login_required(DownloadView.as_view()), name="downloads"
-    path("process/", views.process, name="process"),
+    ),
-    path("downloads/progress/", views.progress, name="progress"),
+    path("settings/", login_required(SettingsView.as_view()), name="settings"),
-    path("channel/", ChannelView.as_view(), name="channel"),
+    path("process/", login_required(process), name="process"),
+    path("downloads/progress/", login_required(progress), name="progress"),
+    path("channel/", login_required(ChannelView.as_view()), name="channel"),
     path(
         "channel/<slug:channel_id_detail>/",
-        ChannelIdView.as_view(),
+        login_required(ChannelIdView.as_view()),
         name="channel_id",
     ),
-    path("video/<slug:video_id>/", VideoView.as_view(), name="video"),
+    path(
+        "video/<slug:video_id>/",
+        login_required(VideoView.as_view()),
+        name="video",
+    ),
 ]

tubearchivist/home/views.py

@@ -8,13 +8,25 @@ import json
 import urllib.parse
 from time import sleep
 
+from django import forms
+from django.contrib.auth import login
+from django.contrib.auth.forms import AuthenticationForm
 from django.http import JsonResponse
 from django.shortcuts import redirect, render
 from django.utils.http import urlencode
 from django.views import View
+from home.forms import (
+    AddToQueueForm,
+    ApplicationSettingsForm,
+    ChannelSearchForm,
+    CustomAuthForm,
+    SubscribeToChannelForm,
+    UserSettingsForm,
+    VideoSearchForm,
+)
 from home.src.config import AppConfig
 from home.src.download import ChannelSubscription, PendingList
-from home.src.helper import RedisArchivist, RedisQueue, process_url_list
+from home.src.helper import RedisArchivist, RedisQueue, UrlListParser
 from home.src.index import WatchState, YoutubeChannel, YoutubeVideo
 from home.src.searching import Pagination, SearchForm, SearchHandler
 from home.tasks import (
@@ -22,10 +34,12 @@ from home.tasks import (
     download_single,
     extrac_dl,
     kill_dl,
+    re_sync_thumbs,
     rescan_filesystem,
     run_backup,
     run_manual_import,
     run_restore_backup,
+    subscribe_to,
     update_subscribed,
 )
 
@@ -40,7 +54,8 @@ class HomeView(View):
 
     def get(self, request):
         """return home search results"""
-        colors, view_style, sort_order, hide_watched = self.read_config()
+        user_id = request.user.id
+        view_config = self.read_config(user_id)
         # handle search
         search_get = request.GET.get("search", False)
         if search_get:
@@ -49,50 +64,63 @@ class HomeView(View):
             search_encoded = False
         # define page size
         page_get = int(request.GET.get("page", 0))
-        pagination_handler = Pagination(page_get, search_encoded)
+        pagination_handler = Pagination(
+            page_get, user_id, search_get=search_encoded
+        )
 
         url = self.ES_URL + "/ta_video/_search"
-
         data = self.build_data(
-            pagination_handler, sort_order, search_get, hide_watched
+            pagination_handler,
+            view_config["sort_by"],
+            view_config["sort_order"],
+            search_get,
+            view_config["hide_watched"],
         )
 
         search = SearchHandler(url, data)
         videos_hits = search.get_data()
         max_hits = search.max_hits
         pagination_handler.validate(max_hits)
+
+        search_form = VideoSearchForm()
         context = {
+            "search_form": search_form,
             "videos": videos_hits,
             "pagination": pagination_handler.pagination,
-            "sortorder": sort_order,
+            "sort_by": view_config["sort_by"],
-            "hide_watched": hide_watched,
+            "sort_order": view_config["sort_order"],
-            "colors": colors,
+            "hide_watched": view_config["hide_watched"],
-            "view_style": view_style,
+            "colors": view_config["colors"],
+            "view_style": view_config["view_style"],
         }
         return render(request, "home/home.html", context)
 
     @staticmethod
-    def build_data(pagination_handler, sort_order, search_get, hide_watched):
+    def build_data(
+        pagination_handler, sort_by, sort_order, search_get, hide_watched
+    ):
         """build the data dict for the search query"""
         page_size = pagination_handler.pagination["page_size"]
         page_from = pagination_handler.pagination["page_from"]
+
+        # overwrite sort_by to match key
+        if sort_by == "views":
+            sort_by = "stats.view_count"
+        elif sort_by == "likes":
+            sort_by = "stats.like_count"
+        elif sort_by == "downloaded":
+            sort_by = "date_downloaded"
+
         data = {
             "size": page_size,
             "from": page_from,
             "query": {"match_all": {}},
-            "sort": [
+            "sort": [{sort_by: {"order": sort_order}}],
-                {"published": {"order": "desc"}},
-                {"date_downloaded": {"order": "desc"}},
-            ],
         }
-        # define sort
-        if sort_order == "downloaded":
-            del data["sort"][0]
-        if search_get:
-            del data["sort"]
         if hide_watched:
             data["query"] = {"term": {"player.watched": {"value": False}}}
         if search_get:
+            del data["sort"]
             query = {
                 "multi_match": {
                     "query": search_get,
@@ -106,24 +134,73 @@ class HomeView(View):
         return data
 
     @staticmethod
-    def read_config():
+    def read_config(user_id):
         """read needed values from redis"""
-        config_handler = AppConfig().config
+        config_handler = AppConfig(user_id)
-        colors = config_handler["application"]["colors"]
+
-        view_style = config_handler["default_view"]["home"]
+        view_key = f"{user_id}:view:home"
-        sort_order = RedisArchivist().get_message("sort_order")
+        view_style = RedisArchivist().get_message(view_key)["status"]
+        if not view_style:
+            view_style = config_handler.config["default_view"]["home"]
+
+        sort_by = RedisArchivist().get_message(f"{user_id}:sort_by")["status"]
+        if not sort_by:
+            sort_by = config_handler.config["archive"]["sort_by"]
+
+        sort_order_key = f"{user_id}:sort_order"
+        sort_order = RedisArchivist().get_message(sort_order_key)["status"]
         if not sort_order:
-            sort_order = "published"
+            sort_order = config_handler.config["archive"]["sort_order"]
-        hide_watched = RedisArchivist().get_message("hide_watched")
+
-        return colors, view_style, sort_order, hide_watched
+        hide_watched_key = f"{user_id}:hide_watched"
+        hide_watched = RedisArchivist().get_message(hide_watched_key)["status"]
+        view_config = {
+            "colors": config_handler.colors,
+            "view_style": view_style,
+            "sort_by": sort_by,
+            "sort_order": sort_order,
+            "hide_watched": hide_watched,
+        }
+        return view_config
 
     @staticmethod
     def post(request):
         """handle post from search form"""
-        post_data = dict(request.POST)
+        search_form = VideoSearchForm(data=request.POST)
-        search_query = post_data["videoSearch"][0]
+        if search_form.is_valid():
-        search_url = "/?" + urlencode({"search": search_query})
+            search_query = request.POST.get("searchInput")
-        return redirect(search_url, permanent=True)
+            print(search_query)
+            search_url = "/?" + urlencode({"search": search_query})
+            return redirect(search_url, permanent=True)
+
+        return redirect("home")
+
+
+class LoginView(View):
+    """resolves to /login/
+    Greeting and login page
+    """
+
+    @staticmethod
+    def get(request):
+        """handle get requests"""
+        failed = bool(request.GET.get("failed"))
+        colors = AppConfig(request.user.id).colors
+        form = CustomAuthForm()
+        context = {"colors": colors, "form": form, "form_error": failed}
+        return render(request, "home/login.html", context)
+
+    @staticmethod
+    def post(request):
+        """handle login post request"""
+        form = AuthenticationForm(data=request.POST)
+        if form.is_valid():
+            next_url = request.POST.get("next") or "home"
+            user = form.get_user()
+            login(request, user)
+            return redirect(next_url)
+
+        return redirect("/login?failed=true")
 
 
 class AboutView(View):
@@ -134,8 +211,7 @@ class AboutView(View):
     @staticmethod
     def get(request):
         """handle http get"""
-        config = AppConfig().config
+        colors = AppConfig(request.user.id).colors
-        colors = config["application"]["colors"]
         context = {"title": "About", "colors": colors}
         return render(request, "home/about.html", context)
 
@@ -147,17 +223,16 @@ class DownloadView(View):
 
     def get(self, request):
         """handle get requests"""
-        config = AppConfig().config
+        user_id = request.user.id
-        colors = config["application"]["colors"]
+        view_config = self.read_config(user_id)
-        view_style = config["default_view"]["downloads"]
-        ignored = RedisArchivist().get_message("show_ignored_only")
-        show_ignored_only = ignored["status"]
 
         page_get = int(request.GET.get("page", 0))
-        pagination_handler = Pagination(page_get)
+        pagination_handler = Pagination(page_get, user_id)
 
-        url = config["application"]["es_url"] + "/ta_download/_search"
+        url = view_config["es_url"] + "/ta_download/_search"
-        data = self.build_data(pagination_handler, show_ignored_only)
+        data = self.build_data(
+            pagination_handler, view_config["show_ignored_only"]
+        )
         search = SearchHandler(url, data)
 
         videos_hits = search.get_data()
@@ -171,17 +246,41 @@ class DownloadView(View):
             all_video_hits = False
             pagination = False
 
+        add_form = AddToQueueForm()
         context = {
+            "add_form": add_form,
             "all_video_hits": all_video_hits,
             "max_hits": max_hits,
             "pagination": pagination,
             "title": "Downloads",
-            "colors": colors,
+            "colors": view_config["colors"],
-            "show_ignored_only": show_ignored_only,
+            "show_ignored_only": view_config["show_ignored_only"],
-            "view_style": view_style,
+            "view_style": view_config["view_style"],
         }
         return render(request, "home/downloads.html", context)
 
+    @staticmethod
+    def read_config(user_id):
+        """read config vars"""
+        config_handler = AppConfig(user_id)
+        view_key = f"{user_id}:view:downloads"
+        view_style = RedisArchivist().get_message(view_key)["status"]
+        if not view_style:
+            view_style = config_handler.config["default_view"]["downloads"]
+
+        ignored = RedisArchivist().get_message(f"{user_id}:show_ignored_only")
+        show_ignored_only = ignored["status"]
+
+        es_url = config_handler.config["application"]["es_url"]
+
+        view_config = {
+            "es_url": es_url,
+            "colors": config_handler.colors,
+            "view_style": view_style,
+            "show_ignored_only": show_ignored_only,
+        }
+        return view_config
+
     @staticmethod
     def build_data(pagination_handler, show_ignored_only):
         """build data dict for search"""
@@ -203,11 +302,12 @@ class DownloadView(View):
     @staticmethod
     def post(request):
         """handle post requests"""
-        download_post = dict(request.POST)
+        to_queue = AddToQueueForm(data=request.POST)
-        if "vid-url" in download_post.keys():
+        if to_queue.is_valid():
-            url_str = download_post["vid-url"]
+            url_str = request.POST.get("vid_url")
+            print(url_str)
             try:
-                youtube_ids = process_url_list(url_str)
+                youtube_ids = UrlListParser(url_str).process_list()
             except ValueError:
                 # failed to process
                 print(f"failed to parse: {url_str}")
@@ -234,27 +334,56 @@ class ChannelIdView(View):
 
     def get(self, request, channel_id_detail):
         """get method"""
-        es_url, colors, view_style = self.read_config()
+        # es_url, colors, view_style = self.read_config()
-        context = self.get_channel_videos(request, channel_id_detail, es_url)
+        view_config = self.read_config(user_id=request.user.id)
-        context.update({"colors": colors, "view_style": view_style})
+        context = self.get_channel_videos(
+            request, channel_id_detail, view_config
+        )
+        context.update(view_config)
         return render(request, "home/channel_id.html", context)
 
     @staticmethod
-    def read_config():
+    def read_config(user_id):
         """read config file"""
-        config = AppConfig().config
+        config_handler = AppConfig(user_id)
-        es_url = config["application"]["es_url"]
+        config = config_handler.config
-        colors = config["application"]["colors"]
-        view_style = config["default_view"]["home"]
-        return es_url, colors, view_style
 
-    def get_channel_videos(self, request, channel_id_detail, es_url):
+        view_key = f"{user_id}:view:home"
+        view_style = RedisArchivist().get_message(view_key)["status"]
+        if not view_style:
+            view_style = config_handler.config["default_view"]["home"]
+
+        sort_by = RedisArchivist().get_message(f"{user_id}:sort_by")["status"]
+        if not sort_by:
+            sort_by = config["archive"]["sort_by"]
+
+        sort_order_key = f"{user_id}:sort_order"
+        sort_order = RedisArchivist().get_message(sort_order_key)["status"]
+        if not sort_order:
+            sort_order = config["archive"]["sort_order"]
+
+        hide_watched_key = f"{user_id}:hide_watched"
+        hide_watched = RedisArchivist().get_message(hide_watched_key)["status"]
+
+        view_config = {
+            "colors": config_handler.colors,
+            "es_url": config["application"]["es_url"],
+            "view_style": view_style,
+            "sort_by": sort_by,
+            "sort_order": sort_order,
+            "hide_watched": hide_watched,
+        }
+        return view_config
+
+    def get_channel_videos(self, request, channel_id_detail, view_config):
         """get channel from video index"""
         page_get = int(request.GET.get("page", 0))
-        pagination_handler = Pagination(page_get)
+        pagination_handler = Pagination(page_get, request.user.id)
         # get data
-        url = es_url + "/ta_video/_search"
+        url = view_config["es_url"] + "/ta_video/_search"
-        data = self.build_data(pagination_handler, channel_id_detail)
+        data = self.build_data(
+            pagination_handler, channel_id_detail, view_config
+        )
         search = SearchHandler(url, data)
         videos_hits = search.get_data()
         max_hits = search.max_hits
@@ -266,7 +395,7 @@ class ChannelIdView(View):
         else:
             # get details from channel index when when no hits
             channel_info, channel_name = self.get_channel_info(
-                channel_id_detail, es_url
+                channel_id_detail, view_config["es_url"]
             )
             videos_hits = False
             pagination = False
@@ -282,21 +411,41 @@ class ChannelIdView(View):
         return context
 
     @staticmethod
-    def build_data(pagination_handler, channel_id_detail):
+    def build_data(pagination_handler, channel_id_detail, view_config):
         """build data dict for search"""
-        page_size = pagination_handler.pagination["page_size"]
+        sort_by = view_config["sort_by"]
-        page_from = pagination_handler.pagination["page_from"]
+        sort_order = view_config["sort_order"]
+
+        # overwrite sort_by to match key
+        if sort_by == "views":
+            sort_by = "stats.view_count"
+        elif sort_by == "likes":
+            sort_by = "stats.like_count"
+        elif sort_by == "downloaded":
+            sort_by = "date_downloaded"
+
         data = {
-            "size": page_size,
+            "size": pagination_handler.pagination["page_size"],
-            "from": page_from,
+            "from": pagination_handler.pagination["page_from"],
             "query": {
-                "term": {"channel.channel_id": {"value": channel_id_detail}}
+                "bool": {
+                    "must": [
+                        {
+                            "term": {
+                                "channel.channel_id": {
+                                    "value": channel_id_detail
+                                }
+                            }
+                        }
+                    ]
+                }
             },
-            "sort": [
+            "sort": [{sort_by: {"order": sort_order}}],
-                {"published": {"order": "desc"}},
-                {"date_downloaded": {"order": "desc"}},
-            ],
         }
+        if view_config["hide_watched"]:
+            to_append = {"term": {"player.watched": {"value": False}}}
+            data["query"]["bool"]["must"].append(to_append)
+
         return data
 
     @staticmethod
@@ -319,81 +468,78 @@ class ChannelView(View):
 
     def get(self, request):
         """handle http get requests"""
-        es_url, colors, view_style = self.read_config()
+        user_id = request.user.id
+        view_config = self.read_config(user_id=user_id)
         page_get = int(request.GET.get("page", 0))
-        pagination_handler = Pagination(page_get)
+        pagination_handler = Pagination(page_get, user_id)
-        page_size = pagination_handler.pagination["page_size"]
+
-        page_from = pagination_handler.pagination["page_from"]
         # get
-        url = es_url + "/ta_channel/_search"
+        url = view_config["es_url"] + "/ta_channel/_search"
         data = {
-            "size": page_size,
+            "size": pagination_handler.pagination["page_size"],
-            "from": page_from,
+            "from": pagination_handler.pagination["page_from"],
             "query": {"match_all": {}},
             "sort": [{"channel_name.keyword": {"order": "asc"}}],
         }
-        show_subed_only = RedisArchivist().get_message("show_subed_only")
+        if view_config["show_subed_only"]:
-        if show_subed_only:
             data["query"] = {"term": {"channel_subscribed": {"value": True}}}
         search = SearchHandler(url, data)
         channel_hits = search.get_data()
-        max_hits = search.max_hits
         pagination_handler.validate(search.max_hits)
+        search_form = ChannelSearchForm()
+        subscribe_form = SubscribeToChannelForm()
         context = {
+            "search_form": search_form,
+            "subscribe_form": subscribe_form,
             "channels": channel_hits,
-            "max_hits": max_hits,
+            "max_hits": search.max_hits,
             "pagination": pagination_handler.pagination,
-            "show_subed_only": show_subed_only,
+            "show_subed_only": view_config["show_subed_only"],
             "title": "Channels",
-            "colors": colors,
+            "colors": view_config["colors"],
-            "view_style": view_style,
+            "view_style": view_config["view_style"],
+            "running": view_config["running"],
         }
         return render(request, "home/channel.html", context)
 
     @staticmethod
-    def read_config():
+    def read_config(user_id):
         """read config file"""
-        config = AppConfig().config
+        config_handler = AppConfig(user_id)
-        es_url = config["application"]["es_url"]
+        view_key = f"{user_id}:view:channel"
-        colors = config["application"]["colors"]
+        view_style = RedisArchivist().get_message(view_key)["status"]
-        view_style = config["default_view"]["channel"]
+        running = RedisArchivist().get_message("progress:subscribe")["status"]
-        return es_url, colors, view_style
+        if not view_style:
+            view_style = config_handler.config["default_view"]["channel"]
 
-    def post(self, request):
+        sub_only_key = f"{user_id}:show_subed_only"
+        show_subed_only = RedisArchivist().get_message(sub_only_key)["status"]
+
+        view_config = {
+            "es_url": config_handler.config["application"]["es_url"],
+            "view_style": view_style,
+            "show_subed_only": show_subed_only,
+            "colors": config_handler.colors,
+            "running": running,
+        }
+
+        return view_config
+
+    @staticmethod
+    def post(request):
         """handle http post requests"""
-        subscriptions_post = dict(request.POST)
+        subscribe_form = SubscribeToChannelForm(data=request.POST)
-        print(subscriptions_post)
+        if subscribe_form.is_valid():
-        subscriptions_post = dict(request.POST)
+            RedisArchivist().set_message(
-        if "subscribe" in subscriptions_post.keys():
+                "progress:subscribe", {"status": "subscribing"}
-            sub_str = subscriptions_post["subscribe"]
+            )
-            try:
+            url_str = request.POST.get("subscribe")
-                youtube_ids = process_url_list(sub_str)
+            print(url_str)
-                self.subscribe_to(youtube_ids)
+            subscribe_to.delay(url_str)
-            except ValueError:
-                print("parsing subscribe ids failed!")
-                print(sub_str)
 
         sleep(1)
         return redirect("channel", permanent=True)
 
-    @staticmethod
-    def subscribe_to(youtube_ids):
-        """process the subscribe ids"""
-        for youtube_id in youtube_ids:
-            if youtube_id["type"] == "video":
-                to_sub = youtube_id["url"]
-                vid_details = PendingList().get_youtube_details(to_sub)
-                channel_id_sub = vid_details["channel_id"]
-            elif youtube_id["type"] == "channel":
-                channel_id_sub = youtube_id["url"]
-            else:
-                raise ValueError("failed to subscribe to: " + youtube_id)
-
-            ChannelSubscription().change_subscribe(
-                channel_id_sub, channel_subscribed=True
-            )
-            print("subscribed to: " + channel_id_sub)
-
 
 class VideoView(View):
     """resolves to /video/<video-id>/
@@ -402,7 +548,7 @@ class VideoView(View):
 
     def get(self, request, video_id):
         """get single video"""
-        es_url, colors = self.read_config()
+        es_url, colors = self.read_config(user_id=request.user.id)
         url = f"{es_url}/ta_video/_doc/{video_id}"
         data = None
         look_up = SearchHandler(url, data)
@@ -419,11 +565,11 @@ class VideoView(View):
         return render(request, "home/video.html", context)
 
     @staticmethod
-    def read_config():
+    def read_config(user_id):
         """read config file"""
-        config = AppConfig().config
+        config_handler = AppConfig(user_id)
-        es_url = config["application"]["es_url"]
+        es_url = config_handler.config["application"]["es_url"]
-        colors = config["application"]["colors"]
+        colors = config_handler.colors
         return es_url, colors
 
     @staticmethod
@@ -450,21 +596,38 @@ class SettingsView(View):
     @staticmethod
     def get(request):
         """read and display current settings"""
-        config = AppConfig().config
+        config_handler = AppConfig(request.user.id)
-        colors = config["application"]["colors"]
+        colors = config_handler.colors
 
-        context = {"title": "Settings", "config": config, "colors": colors}
+        user_form = UserSettingsForm()
+        app_form = ApplicationSettingsForm()
+
+        context = {
+            "title": "Settings",
+            "config": config_handler.config,
+            "colors": colors,
+            "user_form": user_form,
+            "app_form": app_form,
+        }
 
         return render(request, "home/settings.html", context)
 
     @staticmethod
     def post(request):
         """handle form post to update settings"""
-        form_post = dict(request.POST)
+
-        del form_post["csrfmiddlewaretoken"]
+        form_response = forms.Form(request.POST)
-        print(form_post)
+        if form_response.is_valid():
-        config_handler = AppConfig()
+            form_post = dict(request.POST)
-        config_handler.update_config(form_post)
+            print(form_post)
+            del form_post["csrfmiddlewaretoken"]
+            config_handler = AppConfig()
+            if "application-settings" in form_post:
+                del form_post["application-settings"]
+                config_handler.update_config(form_post)
+            elif "user-settings" in form_post:
+                del form_post["user-settings"]
+                config_handler.set_user_config(form_post, request.user.id)
 
         return redirect("settings", permanent=True)
 
@@ -481,8 +644,9 @@ def progress(request):
 def process(request):
     """handle all the buttons calls via POST ajax"""
     if request.method == "POST":
+        current_user = request.user.id
         post_dict = json.loads(request.body.decode())
-        post_handler = PostData(post_dict)
+        post_handler = PostData(post_dict, current_user)
         if post_handler.to_exec:
             task_result = post_handler.run_task()
             return JsonResponse(task_result)
@@ -496,9 +660,10 @@ class PostData:
     handover long running tasks to celery
     """
 
-    def __init__(self, post_dict):
+    def __init__(self, post_dict, current_user):
         self.post_dict = post_dict
         self.to_exec, self.exec_val = list(post_dict.items())[0]
+        self.current_user = current_user
 
     def run_task(self):
         """execute and return task result"""
@@ -525,6 +690,7 @@ class PostData:
             "forgetIgnore": self.forget_ignore,
             "addSingle": self.add_single,
             "manual-import": self.manual_import,
+            "re-embed": self.re_embed,
             "db-backup": self.db_backup,
             "db-restore": self.db_restore,
             "fs-rescan": self.fs_rescan,
@@ -548,9 +714,9 @@ class PostData:
     def change_view(self):
         """process view changes in home, channel, and downloads"""
         origin, new_view = self.exec_val.split(":")
-        print(f"change view on page {origin} to {new_view}")
+        key = f"{self.current_user}:view:{origin}"
-        update_dict = {f"default_view.{origin}": [new_view]}
+        print(f"change view: {key} to {new_view}")
-        AppConfig().update_config(update_dict)
+        RedisArchivist().set_message(key, {"status": new_view}, expire=False)
         return {"success": True}
 
     @staticmethod
@@ -607,26 +773,31 @@ class PostData:
 
     def sort_order(self):
         """change the sort between published to downloaded"""
-        sort_order = self.exec_val
+        sort_order = {"status": self.exec_val}
-        RedisArchivist().set_message("sort_order", sort_order, expire=False)
+        if self.exec_val in ["asc", "desc"]:
+            RedisArchivist().set_message(
+                f"{self.current_user}:sort_order", sort_order, expire=False
+            )
+        else:
+            RedisArchivist().set_message(
+                f"{self.current_user}:sort_by", sort_order, expire=False
+            )
         return {"success": True}
 
     def hide_watched(self):
         """toggle if to show watched vids or not"""
-        hide_watched = bool(int(self.exec_val))
+        key = f"{self.current_user}:hide_watched"
-        print(f"hide watched: {hide_watched}")
+        message = {"status": bool(int(self.exec_val))}
-        RedisArchivist().set_message(
+        print(f"toggle {key}: {message}")
-            "hide_watched", hide_watched, expire=False
+        RedisArchivist().set_message(key, message, expire=False)
-        )
         return {"success": True}
 
     def show_subed_only(self):
         """show or hide subscribed channels only on channels page"""
-        show_subed_only = bool(int(self.exec_val))
+        key = f"{self.current_user}:show_subed_only"
-        print(f"show subed only: {show_subed_only}")
+        message = {"status": bool(int(self.exec_val))}
-        RedisArchivist().set_message(
+        print(f"toggle {key}: {message}")
-            "show_subed_only", show_subed_only, expire=False
+        RedisArchivist().set_message(key, message, expire=False)
-        )
         return {"success": True}
 
     def dlnow(self):
@@ -642,10 +813,10 @@ class PostData:
     def show_ignored_only(self):
         """switch view on /downloads/ to show ignored only"""
         show_value = self.exec_val
+        key = f"{self.current_user}:show_ignored_only"
+        value = {"status": show_value}
         print(f"Filter download view ignored only: {show_value}")
-        RedisArchivist().set_message(
+        RedisArchivist().set_message(key, value, expire=False)
-            "show_ignored_only", {"status": show_value}, expire=False
-        )
         return {"success": True}
 
     def forget_ignore(self):
@@ -660,7 +831,7 @@ class PostData:
         youtube_id = self.exec_val
         print("add vid to dl queue: " + youtube_id)
         PendingList().delete_from_pending(youtube_id)
-        youtube_ids = process_url_list([youtube_id])
+        youtube_ids = UrlListParser(youtube_id).process_list()
         extrac_dl.delay(youtube_ids)
         return {"success": True}
 
@@ -671,6 +842,13 @@ class PostData:
         run_manual_import.delay()
         return {"success": True}
 
+    @staticmethod
+    def re_embed():
+        """rewrite thumbnails into media files"""
+        print("start video thumbnail embed process")
+        re_sync_thumbs.delay()
+        return {"success": True}
+
     @staticmethod
     def db_backup():
         """backup es to zip from settings page"""

tubearchivist/requirements.txt

@@ -6,4 +6,4 @@ redis==3.5.3
 requests==2.26.0
 uWSGI==2.0.20
 whitenoise==5.3.0
-yt_dlp==2021.10.10
+yt_dlp==2021.10.22

tubearchivist/static/css/style.css

@@ -62,7 +62,7 @@ ul {
     margin-left: 20px;
 }
 
-td, span {
+td, span, label {
     font-family: Sen-Regular, sans-serif;
     color: var(--main-font);
 }
@@ -250,7 +250,7 @@ button:hover {
 
 .nav-icons img {
     width: 40px;
-    padding: 0 10px;
+    padding: 0 5px;
     filter: var(--img-filter);
 }
 
@@ -514,6 +514,69 @@ button:hover {
     margin-bottom: 10px;
 }
 
+/* login */
+.login-page {
+    display: flex;
+    flex-wrap: wrap;
+    justify-content: center;
+    text-align: center;
+    align-content: center;
+}
+
+.login-page > * {
+    width: 100%;
+}
+
+.login-page img {
+    width: 100%;
+    max-width: 200px;
+    max-height: 200px;
+    margin-bottom: 40px;
+}
+
+.login-page form {
+    margin: 30px 0;
+}
+
+.login-page input {
+    min-width: 200px;
+}
+
+.login-page button,
+.login-page .danger-zone {
+    width: 210px;
+    margin-top: 5px;
+}
+
+.login-links a {
+    text-decoration: underline;
+    margin: 30px 0;
+    padding: 20px;
+}
+
+.footer-colors {
+    grid-row-start: 2;
+    grid-row-end: 3;
+    display: flex;
+}
+
+.footer-colors div {
+    padding: 20px 0;
+    width: 33.33%;
+}
+
+.col-1 {
+    background-color: var(--highlight-bg);
+}
+
+.col-2 {
+    background-color: var(--accent-font-dark);
+}
+
+.col-3 {
+    background-color: var(--accent-font-light);
+}
+
 /* video page */
 .video-main video {
     max-height: 70vh;
@@ -615,6 +678,24 @@ button:hover {
     width: 30px;
 }
 
+.dl-item:hover > .dl-thumb span {
+    opacity: 1;
+}
+
+.dl-thumb {
+    position: relative;
+}
+
+.dl-thumb span {
+    position: absolute;
+    top: 5px;
+    left: 5px;
+    background-color: var(--accent-font-light);
+    padding: 5px;
+    opacity: 0;
+    transition: 300ms ease-in-out;
+}
+
 .dl-thumb.list {
     width: 25%;
 }

tubearchivist/static/img/icon-exit.svg

@@ -0,0 +1,67 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   xmlns:dc="http://purl.org/dc/elements/1.1/"
+   xmlns:cc="http://creativecommons.org/ns#"
+   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+   xmlns:svg="http://www.w3.org/2000/svg"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   width="500"
+   height="500"
+   viewBox="0 0 132.29197 132.29167"
+   version="1.1"
+   id="svg1303"
+   inkscape:version="0.92.4 (5da689c313, 2019-01-14)"
+   sodipodi:docname="Icons_exit.svg">
+  <defs
+     id="defs1297" />
+  <sodipodi:namedview
+     id="base"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageopacity="0.0"
+     inkscape:pageshadow="2"
+     inkscape:zoom="0.66442107"
+     inkscape:cx="161.45413"
+     inkscape:cy="207.61753"
+     inkscape:document-units="mm"
+     inkscape:current-layer="layer1"
+     showgrid="false"
+     units="px"
+     inkscape:window-width="1169"
+     inkscape:window-height="893"
+     inkscape:window-x="729"
+     inkscape:window-y="13"
+     inkscape:window-maximized="0" />
+  <metadata
+     id="metadata1300">
+    <rdf:RDF>
+      <cc:Work
+         rdf:about="">
+        <dc:format>image/svg+xml</dc:format>
+        <dc:type
+           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+        <dc:title></dc:title>
+      </cc:Work>
+    </rdf:RDF>
+  </metadata>
+  <g
+     inkscape:label="Ebene 1"
+     inkscape:groupmode="layer"
+     id="layer1"
+     transform="translate(0,-164.70764)">
+    <g
+       id="g855"
+       transform="matrix(1.9016362,0,0,1.9016362,-197.93838,-58.9418)">
+      <path
+         inkscape:connector-curvature="0"
+         id="rect1208"
+         style="opacity:1;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0;stroke-linecap:round;stroke-linejoin:bevel;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;paint-order:markers fill stroke"
+         d="m 124.57603,151.92962 c -0.0433,2.30016 2.0751,4.19245 4.75007,4.24278 l 30.26401,0.43007 -6.00195,5.78023 3.43246,3.56154 10.2778,-9.9006 0.002,0.002 3.5183,-3.3908 -3.42991,-3.564 -9.8737,-10.24989 -3.51834,3.39083 5.84388,6.06803 -30.35875,-0.43185 c -2.67494,-0.0503 -4.86301,1.76094 -4.90629,4.06112 z m -17.65039,-32.01644 v 64.95883 h 7.44347 v -58.27707 h 26.3896 v 18.5229 h 7.44296 v -25.20466 z m 33.83307,39.75416 v 25.20467 h 7.44296 v -25.20467 z" />
+    </g>
+  </g>
+</svg>

tubearchivist/static/script.js

@@ -143,6 +143,17 @@ function manualImport() {
     toReplace.appendChild(message);
 }
 
+function reEmbed() {
+    var payload = JSON.stringify({'re-embed': true});
+    sendPost(payload);
+    // clear button
+    var message = document.createElement('p');
+    message.innerText = 'processing thumbnails';
+    var toReplace = document.getElementById('re-embed');
+    toReplace.innerHTML = '';
+    toReplace.appendChild(message);
+}
+
 function dbBackup() {
     var payload = JSON.stringify({'db-backup': true});
     sendPost(payload)
@@ -230,6 +241,7 @@ function createPlayer(button) {
     videoPlayer.setAttribute('controls', true);
     videoPlayer.setAttribute('autoplay', true);
     videoPlayer.setAttribute('width', '100%');
+    videoPlayer.setAttribute('playsinline', true);
     videoPlayer.setAttribute('poster', mediaThumb);
     playerElement.appendChild(videoPlayer);
     // title bar
@@ -371,7 +383,7 @@ function showSearch() {
     } else {
         searchBox.style.display = "";
     }
-    var inputBox = document.getElementById('searchInput');
+    var inputBox = document.getElementById('id_searchInput');
     inputBox.focus();
 }
 

uwsgi.ini

@@ -1,8 +1,8 @@
 [uwsgi]
-; chdir = /app
+module = config.wsgi:application
-module=config.wsgi:application
+master = True
-master=True
+pidfile = /tmp/project-master.pid
-pidfile=/tmp/project-master.pid
+vacuum = True
-vacuum=True
+max-requests = 5000
-max-requests=5000
+socket = :8080
-socket = :8080
+buffer-size = 8192