clarify scope

bump TA_VERSION to v0.2.0
remove synology link, remove security notification
2026-08-09 12:39:29 +00:00 · 2022-07-23 17:32:59 +07:00 · 2022-07-23 16:35:18 +07:00 · 2022-07-23 15:03:41 +07:00 · 2022-07-23 14:38:50 +07:00 · 2022-07-23 14:10:16 +07:00
48 changed files with 1449 additions and 384 deletions
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -30,7 +30,25 @@ This is my setup I have landed on, YMMV:
 - This `deploy.sh` script is not meant to be universally usable for every possible environment but could serve as an idea on how to automatically rebuild containers to test changes - customize to your liking. 

 ## Working with Elasticsearch
-Additionally to the required services as listed in the example docker-compose file, the **Dev Tools** of [Kibana](https://www.elastic.co/guide/en/kibana/current/docker.html) are invaluable for running and testing Elasticsearch queries.  
+Additionally to the required services as listed in the example docker-compose file, the **Dev Tools** of [Kibana](https://www.elastic.co/guide/en/kibana/current/docker.html) are invaluable for running and testing Elasticsearch queries. 
+
+**Quick start**  
+Generate your access token in Elasitcsearch:
+```bash
+bin/elasticsearch-service-tokens create elastic/kibana kibana
+```
+
+Example docker compose, use same version as for Elasticsearch:
+```yml
+kibana:
+  image: docker.elastic.co/kibana/kibana:0.0.0
+  container_name: kibana
+  environment:
+    - "ELASTICSEARCH_HOSTS=http://archivist-es:9200"
+    - "ELASTICSEARCH_SERVICEACCOUNTTOKEN=<your-token-here>"
+  ports:
+    - "5601:5601"
+```

 If you want to run queries on the Elasticsearch container directly from your host with for example `curl` or something like *postman*, you might want to **publish** the port 9200 instead of just **exposing** it.

--- a/9
+++ b/9
@@ -3,7 +3,7 @@


 # First stage to build python wheel
-FROM python:3.10.4-slim-bullseye AS builder
+FROM python:3.10.5-slim-bullseye AS builder
 ARG TARGETPLATFORM

 RUN apt-get update
@@ -14,7 +14,7 @@ COPY ./tubearchivist/requirements.txt /requirements.txt
 RUN pip install --user -r requirements.txt

 # build final image
-FROM python:3.10.4-slim-bullseye as tubearchivist
+FROM python:3.10.5-slim-bullseye as tubearchivist

 ARG TARGETPLATFORM
 ARG INSTALL_DEBUG
@@ -54,12 +54,11 @@ RUN if [ "$INSTALL_DEBUG" ] ; then \
    ; fi

 # make folders
-RUN mkdir /cache
-RUN mkdir /youtube
-RUN mkdir /app
+RUN mkdir /cache /youtube /app

 # copy config files
 COPY docker_assets/nginx.conf /etc/nginx/sites-available/default
+RUN sed -i 's/^user www\-data\;$/user root\;/' /etc/nginx/nginx.conf

 # copy application into container
 COPY ./tubearchivist /app
--- a/README.md
+++ b/README.md
@@ -1,8 +1,11 @@
 ![Tube Archivist](assets/tube-archivist-banner.jpg?raw=true "Tube Archivist Banner")  

-<center><h1>Your self hosted YouTube media server</h1></center>
-
-Tube Archivist has a new home: https://github.com/tubearchivist/tubearchivist
+<h1 align="center">Your self hosted YouTube media server</h1>
+<div align="center">
+<a href="https://github.com/bbilly1/tilefy" target="_blank"><img src="https://tiles.tilefy.me/t/tubearchivist-docker.png" alt="tubearchivist-docker" title="Tube Archivist Docker Pulls" height="50" width="200"/></a>
+<a href="https://github.com/bbilly1/tilefy" target="_blank"><img src="https://tiles.tilefy.me/t/tubearchivist-github-star.png" alt="tubearchivist-github-star" title="Tube Archivist GitHub Stars" height="50" width="200"/></a>
+<a href="https://github.com/bbilly1/tilefy" target="_blank"><img src="https://tiles.tilefy.me/t/tubearchivist-github-forks.png" alt="tubearchivist-github-forks" title="Tube Archivist GitHub Forks" height="50" width="200"/></a>
+</div>

 ## Table of contents:
 * [Wiki](https://github.com/tubearchivist/tubearchivist/wiki) with [FAQ](https://github.com/tubearchivist/tubearchivist/wiki/FAQ)
@@ -69,6 +72,7 @@ The main Python application that displays and serves your video collection, buil
  - And another volume to save application data at **/cache**.
  - The environment variables `ES_URL` and `REDIS_HOST` are needed to tell Tube Archivist where Elasticsearch and Redis respectively are located.
  - The environment variables `HOST_UID` and `HOST_GID` allows Tube Archivist to `chown` the video files to the main host system user instead of the container user. Those two variables are optional, not setting them will disable that functionality. That might be needed if the underlying filesystem doesn't support `chown` like *NFS*. 
+  - Set the environment variable `TA_HOST` to configure from where the interface can be accessed. This can be a domain like *example.com*, a subdomain like *ta.example.com* or an IP address like *192.168.1.20*, add without the protocol and without the port. You can add multiple hostnames separated with a space.
  - Change the environment variables `TA_USERNAME` and `TA_PASSWORD` to create the initial credentials. 
  - `ELASTIC_PASSWORD` is for the password for Elasticsearch. The environment variable `ELASTIC_USER` is optional, should you want to change the username from the default *elastic*.
  - For the scheduler to know what time it is, set your timezone with the `TZ` environment variable, defaults to *UTC*.
@@ -83,7 +87,7 @@ Should that not be an option, the Tube Archivist container takes these two addit
 Changing any of these two environment variables will change the files *nginx.conf* and *uwsgi.ini* at startup using `sed` in your container.

 ### Elasticsearch
-**Note**: Newest Tube Archivist depends on Elasticsearch version 7.17 to provide an automatic updatepath in the future. 
+**Note**: Tube Archivist depends on Elasticsearch 8. 

 Use `bbilly1/tubearchivist-es` to automatically get the recommended version, or use the official image with the version tag in the docker-compose file.

@@ -111,7 +115,6 @@ You will see the current version number of **Tube Archivist** in the footer of t

 ### Alternative installation instructions:
 - **arm64**: The Tube Archivist container is multi arch, so is Elasticsearch. RedisJSON doesn't offer arm builds, you can use `bbilly1/rejson`, an unofficial rebuild for arm64.
- **Synology**: There is a [discussion thread](https://github.com/tubearchivist/tubearchivist/discussions/48) with Synology installation instructions.
 - **Unraid**: The three containers needed are all in the Community Applications. First install `TubeArchivist RedisJSON` followed by `TubeArchivist ES`, and finally you can install `TubeArchivist`. If you have unraid specific issues, report those to the [support thread](https://forums.unraid.net/topic/114073-support-crocs-tube-archivist/ "support thread").
 - **Helm Chart**: There is a Helm Chart available at https://github.com/insuusvenerati/helm-charts. Mostly self-explanatory but feel free to ask questions in the discord / subreddit.

@@ -165,7 +168,6 @@ We have come far, nonetheless we are not short of ideas on how to improve and ex
 - [ ] Show similar videos on video page
 - [ ] Multi language support
 - [ ] Show total video downloaded vs total videos available in channel
- [ ] Make items in grid row configurable to use more of the screen
 - [ ] Add statistics of index
 - [ ] Download speed schedule ([#198](https://github.com/tubearchivist/tubearchivist/issues/198))
 - [ ] Implement complete offline media file import from json file ([#138](https://github.com/tubearchivist/tubearchivist/issues/138))
@@ -175,6 +177,7 @@ We have come far, nonetheless we are not short of ideas on how to improve and ex
 - [ ] Download video comments

 Implemented:
+- [X] Make items in grid row configurable to use more of the screen [2022-06-04]
 - [X] Add passing browser cookies to yt-dlp [2022-05-08]
 - [X] Add [SponsorBlock](https://sponsor.ajay.app/) integration [2022-04-16]
 - [X] Implement per channel settings [2022-03-26]
@@ -196,7 +199,6 @@ Implemented:
 ## Known limitations
 - Video files created by Tube Archivist need to be playable in your browser of choice. Not every codec is compatible with every browser and might require some testing with format selection. 
 - Every limitation of **yt-dlp** will also be present in Tube Archivist. If **yt-dlp** can't download or extract a video for any reason, Tube Archivist won't be able to either.
- For now this is meant to be run in a trusted network environment. Not everything is properly authenticated.
 - There is currently no flexibility in naming of the media files.


--- a/deploy.sh
+++ b/deploy.sh
@@ -144,6 +144,7 @@ function sync_unstable {
 }


+# new function, sync only tag, build with build server
 function sync_docker {

    # check things
@@ -152,13 +153,40 @@ function sync_docker {
        return
    fi

+    echo "latest tags:"
+    git tag | tail -n 5 | sort -r
+
+    printf "\ncreate new version:\n"
+    read -r VERSION
+
+    echo "push new tag: $VERSION?"
+    read -rn 1
+
+    # create release tag
+    echo "commits since last version:"
+    git log "$(git describe --tags --abbrev=0)"..HEAD --oneline
+    git tag -a "$VERSION" -m "new release version $VERSION"
+    git push origin "$VERSION"
+
+}
+
+
+# old builder, sync tag, build and push locally
+function sync_docker_old {
+
+    # check things
+    if [[ $(git branch --show-current) != 'master' ]]; then
+        echo 'you are not on master, dummy!'
+        return
+    fi
+
    if [[ $(systemctl is-active docker) != 'active' ]]; then
        echo "starting docker"
        sudo systemctl start docker
    fi

    echo "latest tags:"
-    git tag | tail -n 10
+    git tag | tail -n 5 | sort -r

    printf "\ncreate new version:\n"
    read -r VERSION
@@ -176,7 +204,7 @@ function sync_docker {
    echo "commits since last version:"
    git log "$(git describe --tags --abbrev=0)"..HEAD --oneline
    git tag -a "$VERSION" -m "new release version $VERSION"
-    git push all "$VERSION"
+    git push origin "$VERSION"

 }

@@ -191,7 +219,6 @@ elif [[ $1 == "validate" ]]; then
    validate "$2"
 elif [[ $1 == "docker" ]]; then
    sync_docker
-    sync_unstable
 elif [[ $1 == "unstable" ]]; then
    sync_unstable
 elif [[ $1 == "es" ]]; then
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -3,8 +3,8 @@ version: '3.3'
 services:
  tubearchivist:
    container_name: tubearchivist
-    restart: always
-    image: bbilly1/tubearchivist:latest
+    restart: unless-stopped
+    image: bbilly1/tubearchivist
    ports:
      - 8000:8000
    volumes:
@@ -15,6 +15,7 @@ services:
      - REDIS_HOST=archivist-redis          # don't add protocol
      - HOST_UID=1000
      - HOST_GID=1000
+      - TA_HOST=tubearchivist.local         # set your host name
      - TA_USERNAME=tubearchivist           # your initial TA credentials
      - TA_PASSWORD=verysecret              # your initial TA credentials
      - ELASTIC_PASSWORD=verysecret         # set password for Elasticsearch
@@ -23,9 +24,9 @@ services:
      - archivist-es
      - archivist-redis
  archivist-redis:
-    image: redislabs/rejson:latest          # for arm64 use bbilly1/rejson
+    image: redislabs/rejson                 # for arm64 use bbilly1/rejson
    container_name: archivist-redis
-    restart: always
+    restart: unless-stopped
    expose:
      - "6379"
    volumes:
@@ -33,9 +34,9 @@ services:
    depends_on:
      - archivist-es
  archivist-es:
-    image: bbilly1/tubearchivist-es         # only for amd64, or use official es 7.17.3
+    image: bbilly1/tubearchivist-es         # only for amd64, or use official es 8.3.2
    container_name: archivist-es
-    restart: always
+    restart: unless-stopped
    environment:
      - "xpack.security.enabled=true"
      - "ELASTIC_PASSWORD=verysecret"       # matching Elasticsearch password
--- a/docker_assets/nginx.conf
+++ b/docker_assets/nginx.conf
@@ -3,18 +3,22 @@ server {
    listen 8000;
    
    location /cache/videos/ {
+        auth_request /api/ping/;
        alias /cache/videos/;
    }
    
    location /cache/channels/ {
+        auth_request /api/ping/;
        alias /cache/channels/;
    }
    
    location /cache/playlists/ {
+        auth_request /api/ping/;
        alias /cache/playlists/;
    }
    
    location /media/ {
+        auth_request /api/ping/;
        alias /youtube/;
        types {
            text/vtt vtt;
--- a/docker_assets/run.sh
+++ b/docker_assets/run.sh
@@ -5,13 +5,15 @@ if [[ -z "$ELASTIC_USER" ]]; then
    export ELASTIC_USER=elastic
 fi

-ENV_VARS=("TA_USERNAME" "TA_PASSWORD" "ELASTIC_PASSWORD" "ELASTIC_USER")
-for each in "${ENV_VARS[@]}"; do
-    if ! [[ -v $each ]]; then
-        echo "missing environment variable $each"
-        exit 1
-    fi
-done
+cachedir=/cache
+[[ -d $cachedir ]] || cachedir=.
+lockfile=${cachedir}/initsu.lock
+
+required="Missing required environment variable"
+[[ -f $lockfile ]] || : "${TA_USERNAME:?$required}"
+: "${TA_PASSWORD:?$required}"
+: "${ELASTIC_PASSWORD:?$required}"
+: "${TA_HOST:?$required}"

 # ugly nginx and uwsgi port overwrite with env vars
 if [[ -n "$TA_PORT" ]]; then
@@ -39,12 +41,27 @@ done
 # start python application
 python manage.py makemigrations
 python manage.py migrate
-export DJANGO_SUPERUSER_PASSWORD=$TA_PASSWORD && \
-    python manage.py createsuperuser --noinput --name "$TA_USERNAME"
+
+if [[ -f $lockfile ]]; then
+    echo -e "\e[33;1m[WARNING]\e[0m This is not the first run! Skipping" \
+        "superuser creation.\nTo force it, remove $lockfile"
+else
+    export DJANGO_SUPERUSER_PASSWORD=$TA_PASSWORD
+    output="$(python manage.py createsuperuser --noinput --name "$TA_USERNAME" 2>&1)"
+
+    case "$output" in
+        *"Superuser created successfully"*)
+            echo "$output" && touch $lockfile ;;
+        *"That name is already taken."*)
+            echo "Superuser already exists. Creation will be skipped on next start."
+            touch $lockfile ;;
+        *) echo "$output" && exit 1
+    esac
+fi

 python manage.py collectstatic --noinput -c
 nginx &
 celery -A home.tasks worker --loglevel=INFO &
 celery -A home beat --loglevel=INFO \
-    -s "${BEAT_SCHEDULE_PATH:-/cache/celerybeat-schedule}" &
+    -s "${BEAT_SCHEDULE_PATH:-${cachedir}/celerybeat-schedule}" &
 uwsgi --ini uwsgi.ini
--- a/docker_assets/uwsgi.ini
+++ b/docker_assets/uwsgi.ini
@@ -5,4 +5,7 @@ pidfile = /tmp/project-master.pid
 vacuum = True
 max-requests = 5000
 socket = :8080
-buffer-size = 8192
+buffer-size = 8192
+log-5xx = true
+log-4xx = true
+disable-logging = true
--- a/docs/Channels.md
+++ b/docs/Channels.md
@@ -17,16 +17,18 @@ The **Subscribe to Channels** button <img src="assets/icon-add.png?raw=true" alt
 You can search your indexed channels by clicking on the search icon <img src="assets/icon-search.png?raw=true" alt="search icon" width="20px" style="margin:0 5px;">. This will open a dedicated page.

 ## Channel Detail
-Each channel will get a dedicated channel detail page accessible at `/channel/<channel-id>/` of your Tube Archivist. This page shows all the videos you have downloaded from this channel plus additional metadata. 
- If you are subscribed to the channel, an *Unsubscribe* button will show, else the *Subscribe* button will show.
- You can *Show* the channel description, that matches with the *About* tab on YouTube.
- The **Mark as Watched** button will mark all videos of this channel as watched.
- The button **Delete Channel** will delete the channel plus all videos of this channel, both media files and metadata additionally this will also delete playlists metadata belonging to that channel.
- The button **Show Playlists** will go to the [playlists](Playlists) page and filter the list to only show playlists from this channel.
+Each channel will get a dedicated channel detail page accessible at `/channel/<channel-id>/` of your Tube Archivist. This page shows all the videos you have downloaded from this channel.

-### Channel Customize
-Clicking on the *Configure* button will open a form with options to configure settings on a per channel basis. Any configurations here will overwrite your settings from the [settings](Settings) page.
- **Download Format**: Overwrite the download qualities for videos from this channel.
+- If you are subscribed to the channel, an *Unsubscribe* button will show, else the *Subscribe* button will show.
+- The **Mark as Watched** button will mark all videos of this channel as watched.
+
+Additionally there is a *Channel Playlist* page, accessible at `/channel/<channel-id>/playlist/` to show all indexed playlists from this channel. 
+
+On the *Channel About* page you can see additional metadata.
+- The button **Delete Channel** will delete the channel plus all videos of this channel, both media files and metadata additionally this will also delete playlists metadata belonging to that channel.
+
+The channel customize form gives options to change settings on a per channel basis. Any configurations here will overwrite your configurations from the [settings](Settings) page.
+- **Download Format**: Overwrite the download quality for videos from this channel.
 - **Auto Delete**: Automatically delete watched videos from this channel after selected days.
 - **Index Playlists**: Automatically add all Playlists with at least a video downloaded to your index. Only do this for channels where you care about playlists as this will slow down indexing new videos for having to check which playlist this belongs to.
 - **SponsorBlock**: Using [SponsorBlock](https://sponsor.ajay.app/) to get and skip sponsored content. Customize per channel: You can *disable* or *enable* SponsorBlock for certain channels only to overwrite the behavior set on the [Settings](settings) page. Selecting *unset* will remove the overwrite and your setting will fall back to the default on the settings page.
--- a/docs/FAQ.md
+++ b/docs/FAQ.md
@@ -15,7 +15,7 @@ Defining the scope is important for the success of any project:
 - A too narrow scope will make this project uninteresting and will exclude audiences that could also benefit from this project.
 - Not defining a scope will easily lead to misunderstandings and false hopes of where this project tries to go.

-Of course this is subject to change, as this project continues to grow and more people contribute.
+Of course this is subject to change: The scope can be expanded as this project continues to grow and more people contribute.

 ## 2. Emby/Plex/Jellyfin/Kodi integrations
 Although there are similarities between these excellent projects and Tube Archivist, they have a very different use case. Trying to fit the metadata relations and database structure of a YouTube archival project into these media servers that specialize in Movies and TV shows is always going to be limiting.
--- a/docs/Home.md
+++ b/docs/Home.md
@@ -9,6 +9,7 @@ Table of contents:
 * [Settings](Settings): All the configuration options
 * [Video](Video): All details of a single video and playlist navigation.
 * [Users](Users): User management admin interface
+* [Search](Search): Search your archive
 * [Installation](Installation): WIP - detailed installation instructions for various platforms.

 ## Getting Started
--- a/docs/Search.md
+++ b/docs/Search.md
@@ -0,0 +1,54 @@
+# Search Page
+Accessible at `/search/` of your **Tube Archivist**, search your archive for Videos, Channels and Playlists - or even full text search throughout your indexed subtitles.
+
+- All your queries are case insensitive and are normalized to lowercase.
+- All your queries are analyzed for the english language, this means *singular*, *plural* and word variations like *-ing*, *-ed*, *-able* etc are treated as synonyms.
+- Fuzzy search is activated for all your searches. This can catch typos in your queries or in the matching documents with one to two letters difference, depending on the query length.
+- All text searches are ranked, meaning the better a match the higher ranked the result. Unless otherwise stated, queries with multiple words are processed with the `and` operator, meaning all words need to match so each word will narrow down the result.
+- This will return 30 results per query, pagination is not implemented yet.
+
+Just start typing to start a *simple* search or start your query with a primary keyword to search for a specific type and narrow down the result with secondary keywords. Secondary keywords can be in any order. Use *yes* or *no* for boolean values.
+
+## Simple
+Start your query without a keyword to make a simple query. This will search in *video titles*, *channel names* and *playlist titles* and will return matching videos, channels and playlists. Keyword searches will return more results in a particular category due to the fact that more fields are searched for matches.
+
+## Video
+Start your query with the primary keyword `video:` to search for videos only. This will search through the *video titles*, *tags* and *category* fields. Narrow your search down with secondary keywords:
+- `channel:` search for videos matching the channel name.
+- `active:` is a boolean value, to search for videos that are still active on youtube or that are not active any more.
+
+**Example**:
+- `video:learn python channel:corey shafer active:yes`: This will return all videos with the term *Learn Python* from the channel *Corey Shafer* that are still *Active* on YouTube.
+- `video: channel:tom scott active:no`: Note the omitted term after the primary key, this will show all videos from the channel *Tom Scott* that are no longer active on YouTube.
+
+## Channel
+Start with the `channel:` primary keyword to search for channels matching your query. This will search through the *channel name* and *channel description* fields. Narrow your search down with secondary keywords:
+- `subscribed:` is a boolean value, search for channels that you are subscribed to or not.
+- `active:` is a boolean value, to search for channels that are still active on YouTube or that are no longer active.
+
+**Example**:
+- `channel:linux subscribed:yes`: Search for channels with the term *Linux* that you are subscribed to.
+- `channel: active:no`: Note the omitted term after the primary key, this will return all channels that are no longer active on YouTube.
+
+## Playlist
+Start your query with the primary keyword `playlist:` to search for playlists only. This will search through the *playlist title* and *playlist description* fields. Narrow down your search with these secondary keywords:
+- `subscribed`: is a boolean value, search for playlists that you are subscribed to or not.
+- `active:` is a boolean value, to search for playlists that are still active on YouTube or that are no longer active.
+
+**Example**:
+- `playlist:backend engineering subscribed:yes`: Search for playlists about *Backend Engineering* that you are subscribed to.
+- `playlist: active:yes subscribed:yes`: Note the omitted primary search term, this will return all playlists active on YouTube that you are subscribed to.
+- `playlist:html css active:yes`: Search for playlists containing *HTML CSS* that are still active on YouTube.
+
+## Full
+Start a full text search by beginning your query with the primary keyword `full:`. This will search through your indexed Subtitles showing segments with possible matches. This will only show any results if you have activated *subtitle download and index* on the settings page. The operator for full text searches is `or` meaning when searching for multiple words not all words need to match, but additional words will change the ranking of the result, the more words match and the better they match, the higher ranked the result. The matching words will get highlighted in the text preview.
+
+Clicking the play button on the thumbnail will open the inplace player at the timestamp from where the segment starts. Same when clicking the video title, this will open the video page and put the player at the segment timestamp. This will overwrite any previous playback position.
+
+Narrow down your search with these secondary keywords:
+- `lang`: Search for matches only within a language. Use the same two letter ISO country code as you have set on the settings page.
+- `source`: Can either be *auto* to search through auto generated subtitles only or *user* to search through user uploaded subtitles only.
+
+**Example**:
+- `full:contribute to open source lang:en` search for subtitle segments matching with the words *Contribute to Open Source* in the language *en*.
+- `full:flight simulator cockpit source:user` to search for the words *Flight Simulator Cockpit* from *user* uploaded subtitle segments.
--- a/docs/Settings.md
+++ b/docs/Settings.md
@@ -36,15 +36,18 @@ Additional settings passed to yt-dlp.
 Importing your YouTube Cookie into Tube Archivist allows yt-dlp to bypass age restrictions, gives access to private videos and your *watch later* or *liked videos*.

 ### Security concerns
-Cookies are used to store your session and contain your access token to your google account, this file can be used to take over your account. Treat that file with utmost care as you would any other password or credential. *Tube Archivist* stores your cookie in Redis and will automatically append it to yt-dlp for every request.
+Cookies are used to store your session and contain your access token to your google account, this information can be used to take over your account. Treat that data with utmost care as you would any other password or credential. *Tube Archivist* stores your cookie in Redis and will automatically append it to yt-dlp for every request.

-### Export your cookie
+### Auto import
+Easiest way to import your cookie is to use the **Tube Archivist Companion** [browser extension](https://github.com/tubearchivist/browser-extension) for Firefox and Chrome.
+
+### Alternative Manual Export your cookie
 - Install **Cookies.txt** addon for [chrome](https://chrome.google.com/webstore/detail/get-cookiestxt/bgaddhkoddajcdgocldbbfleckgcbcid) or [firefox](https://addons.mozilla.org/firefox/addon/cookies-txt).
 - Visit YouTube and login with whichever YouTube account you wish to use to generate the cookies.
 - Click on the extension icon in the toolbar - it will drop down showing the active cookies for YT.
 - Click Export to export the cookies, filename is by default *cookies.google.txt*.

-### Import your cookie
+### Alternative Manual Import your cookie
 Place the file *cookies.google.txt* into the *cache/import* folder of Tube Archivist and enable the cookie import. Once you click on *Update Application Configurations* to save your changes, your cookie will get imported and stored internally.

 Once imported, a **Validate Cookie File** button will show, where you can confirm if your cookie is working or not.
@@ -95,7 +98,7 @@ That's the equivalent task as run from the downloads page looking through your c
 Start downloading all videos currently in the download queue.

 ## Refresh Metadata
-Rescan videos, channels and playlists on youtube and update metadata periodically. This will also deactivate an item and exclude it from future refreshes if the link on YouTube is no longer available. This task is meant to be run once per day, set your schedule accordingly.
+Rescan videos, channels and playlists on youtube and update metadata periodically. This will also refresh your subtitles based on your current settings. If an item is no longer available on YouTube, this will deactivate it and exclude it from future refreshes. This task is meant to be run once per day, set your schedule accordingly.

 The field **Refresh older than x days** takes a number where TubeArchivist will consider an item as *outdated*. This value is used to calculate how many items need to be refreshed today based on the total indexed. This will spread out the requests to YouTube. Sensible value here is **90** days.

@@ -139,7 +142,7 @@ Detect the YouTube ID from filename, this accepts the default yt-dlp naming conv
 This will write or overwrite all thumbnails in the media file using the downloaded thumbnail. This is only necessary if you didn't download the files with the option *Embed Thumbnail* enabled or want to make sure all media files get the newest thumbnail. Follow the docker-compose logs to monitor progress.

 ## Backup Database
-This will backup your metadata into a zip file. The file will get stored at *cache/backup* and will contain the necessary files to restore the Elasticsearch index formatted **nd-json** files plus a complete export of the index in a set of conventional **json** files.  
+This will backup your metadata into a zip file. The file will get stored at *cache/backup* and will contain the necessary files to restore the Elasticsearch index formatted **nd-json** files.  

 BE AWARE: This will **not** backup any media files, just the metadata from the Elasticsearch.

--- a/tubearchivist/api/README.md
+++ b/tubearchivist/api/README.md
@@ -1,6 +1,9 @@
 # TubeArchivist API
 Documentation of available API endpoints.  
-**Note: This is very early alpha and will change!**
+
+Note:  
+- This is very early stages and will change!
+- Check the commit history to see if a documented feature is already in your release

 ## Authentication
 API token will get automatically created, accessible on the settings page. Token needs to be passed as an authorization header with every request. Additionally session based authentication is enabled too: When you are logged into your TubeArchivist instance, you'll have access to the api in the browser for testing.
@@ -202,7 +205,21 @@ When valid returns message with user id:
 ```

 ## Task View
-Start a background task
+GET /api/task/
+POST /api/task/
+
+Check if there is an ongoing task:
+GET /api/task/
+
+Returns:
+```json
+{
+    "rescan": false,
+    "downloading": false
+}
+```
+
+Start a background task  
 POST /api/task/
 ```json
 {
@@ -216,11 +233,14 @@ List of valid task names:


 ## Cookie View
-Check your youtube cookie settings  
+Check your youtube cookie settings, *status* turns to `true` if cookie has been validated.  
 GET /api/cookie/
 ```json
 {
-    "cookie_enabled": true
+    "cookie_enabled": true,
+    "status": true,
+    "validated": <timestamp>,
+    "validated_str": "timestamp"
 }
 ```

@@ -231,3 +251,25 @@ Send empty post request to validate cookie.
    "cookie_validated": true
 }
 ```
+
+PUT /api/cookie/  
+Send put request containing the cookie as a string:
+```json
+{
+    "cookie": "your-cookie-as-string"
+}
+```
+Imports and validates cookie, returns on success:
+```json
+{
+    "cookie_import": "done",
+    "cookie_validated": true
+}
+```
+Or returns status code 400 on failure:
+```json
+{
+    "cookie_import": "fail",
+    "cookie_validated": false
+}
+```
--- a/tubearchivist/api/src/task_processor.py
+++ b/tubearchivist/api/src/task_processor.py
@@ -50,5 +50,5 @@ class TaskHandler:
        print("download pending")
        running = download_pending.delay()
        print("set task id: " + running.id)
-        RedisArchivist().set_message("dl_queue_id", running.id, expire=False)
+        RedisArchivist().set_message("dl_queue_id", running.id)
        return {"success": True}
--- a/tubearchivist/api/views.py
+++ b/tubearchivist/api/views.py
@@ -131,7 +131,7 @@ class VideoProgressView(ApiBaseView):
        position = request.data.get("position", 0)
        key = f"{request.user.id}:progress:{video_id}"
        message = {"position": position, "youtube_id": video_id}
-        RedisArchivist().set_message(key, message, expire=False)
+        RedisArchivist().set_message(key, message)
        self.response = request.data

        return Response(self.response)
@@ -452,9 +452,20 @@ class LoginApiView(ObtainAuthToken):

 class TaskApiView(ApiBaseView):
    """resolves to /api/task/
+    GET: check if ongoing background task
    POST: start a new background task
    """

+    @staticmethod
+    def get(request):
+        """handle get request"""
+        # pylint: disable=unused-argument
+        response = {"rescan": False, "downloading": False}
+        for key in response.keys():
+            response[key] = RedisArchivist().is_locked(key)
+
+        return Response(response)
+
    def post(self, request):
        """handle post request"""

@@ -469,6 +480,7 @@ class CookieView(ApiBaseView):
    """resolves to /api/cookie/
    GET: check if cookie is enabled
    POST: verify validity of cookie
+    PUT: import cookie
    """

    @staticmethod
@@ -476,9 +488,11 @@ class CookieView(ApiBaseView):
        """handle get request"""
        # pylint: disable=unused-argument
        config = AppConfig().config
-        cookie_enabled = config["downloads"]["cookie_import"]
+        valid = RedisArchivist().get_message("cookie:valid")
+        response = {"cookie_enabled": config["downloads"]["cookie_import"]}
+        response.update(valid)

-        return Response({"cookie_enabled": cookie_enabled})
+        return Response(response)

    @staticmethod
    def post(request):
@@ -488,3 +502,27 @@ class CookieView(ApiBaseView):
        validated = CookieHandler(config).validate()

        return Response({"cookie_validated": validated})
+
+    @staticmethod
+    def put(request):
+        """handle put request"""
+        # pylint: disable=unused-argument
+        config = AppConfig().config
+        cookie = request.data.get("cookie")
+        if not cookie:
+            message = "missing cookie key in request data"
+            print(message)
+            return Response({"message": message}, status=400)
+
+        print(f"cookie preview:\n\n{cookie[:300]}")
+        handler = CookieHandler(config)
+        handler.set_cookie(cookie)
+        validated = handler.validate()
+        if not validated:
+            handler.revoke()
+            message = {"cookie_import": "fail", "cookie_validated": validated}
+            print(f"cookie: {message}")
+            return Response({"message": message}, status=400)
+
+        message = {"cookie_import": "done", "cookie_validated": validated}
+        return Response(message)
--- a/tubearchivist/config/settings.py
+++ b/tubearchivist/config/settings.py
@@ -30,7 +30,7 @@ SECRET_KEY = PW_HASH.hexdigest()
 # SECURITY WARNING: don't run with debug turned on in production!
 DEBUG = bool(environ.get("DJANGO_DEBUG"))

-ALLOWED_HOSTS = ["*"]
+ALLOWED_HOSTS = [i.strip() for i in environ.get("TA_HOST").split()]


 # Application definition
@@ -163,4 +163,4 @@ CORS_ALLOW_HEADERS = list(default_headers) + [

 # TA application settings
 TA_UPSTREAM = "https://github.com/tubearchivist/tubearchivist"
-TA_VERSION = "v0.1.6"
+TA_VERSION = "v0.2.0"
--- a/tubearchivist/home/apps.py
+++ b/tubearchivist/home/apps.py
@@ -13,8 +13,8 @@ from home.src.ta.ta_redis import RedisArchivist
 class StartupCheck:
    """checks to run at application startup"""

-    MIN_MAJOR, MAX_MAJOR = 7, 7
-    MIN_MINOR = 17
+    MIN_MAJOR, MAX_MAJOR = 8, 8
+    MIN_MINOR = 0

    def __init__(self):
        self.config_handler = ArchivistConfig()
@@ -28,6 +28,7 @@ class StartupCheck:
        self.release_lock()
        index_check()
        self.sync_redis_state()
+        self.set_redis_conf()
        self.make_folders()
        self.set_has_run()

@@ -45,6 +46,10 @@ class StartupCheck:
        print("sync redis")
        self.config_handler.load_new_defaults()

+    def set_redis_conf(self):
+        """set conf values for redis"""
+        self.redis_con.conn.config_set("timeout", 3600)
+
    def make_folders(self):
        """make needed cache folders here so docker doesn't mess it up"""
        folders = [
--- a/tubearchivist/home/config.json
+++ b/tubearchivist/home/config.json
@@ -36,7 +36,6 @@
        "app_root": "/app",
        "cache_dir": "/cache",
        "videos": "/youtube",
-        "file_template": "%(id)s_%(title)s.mp4",
        "colors": "dark",
        "enable_cast": false
    },
--- a/tubearchivist/home/src/download/queue.py
+++ b/tubearchivist/home/src/download/queue.py
@@ -150,7 +150,7 @@ class PendingList(PendingIndex):
                "title": "Adding to download queue.",
                "message": "Extracting lists",
            }
-            RedisArchivist().set_message("message:add", mess_dict)
+            RedisArchivist().set_message("message:add", mess_dict, expire=True)
            self._process_entry(entry)

    def _process_entry(self, entry):
@@ -229,10 +229,11 @@ class PendingList(PendingIndex):
            "message": "Progress: " + progress,
        }
        if idx + 1 == len(self.missing_videos):
-            RedisArchivist().set_message("message:add", mess_dict, expire=4)
+            expire = 4
        else:
-            RedisArchivist().set_message("message:add", mess_dict)
+            expire = True

+        RedisArchivist().set_message("message:add", mess_dict, expire=expire)
        if idx + 1 % 25 == 0:
            print("adding to queue progress: " + progress)

--- a/tubearchivist/home/src/download/subscriptions.py
+++ b/tubearchivist/home/src/download/subscriptions.py
@@ -76,11 +76,13 @@ class ChannelSubscription:
                "message": f"Progress: {idx + 1}/{len(all_channels)}",
            }
            if idx + 1 == len(all_channels):
-                RedisArchivist().set_message(
-                    "message:rescan", message=message, expire=4
-                )
+                expire = 4
            else:
-                RedisArchivist().set_message("message:rescan", message=message)
+                expire = True
+
+            RedisArchivist().set_message(
+                "message:rescan", message=message, expire=expire
+            )

        return missing_videos

@@ -152,7 +154,7 @@ class PlaylistSubscription:
                "message": f"Processing {idx + 1} of {len(new_playlists)}",
            }
            RedisArchivist().set_message(
-                "message:subplaylist", message=message
+                "message:subplaylist", message=message, expire=True
            )

        return new_thumbs
@@ -206,7 +208,9 @@ class PlaylistSubscription:
                "title": "Scanning playlists: Looking for new videos.",
                "message": f"Progress: {idx + 1}/{len(all_playlists)}",
            }
-            RedisArchivist().set_message("message:rescan", message=message)
+            RedisArchivist().set_message(
+                "message:rescan", message=message, expire=True
+            )

            for video in all_missing:
                youtube_id = video["youtube_id"]
--- a/tubearchivist/home/src/download/thumbnails.py
+++ b/tubearchivist/home/src/download/thumbnails.py
@@ -193,11 +193,13 @@ class ThumbManager:
                    "message": "Downloading Thumbnails, Progress: " + progress,
                }
                if idx + 1 == len(missing_thumbs):
-                    RedisArchivist().set_message(
-                        "message:add", mess_dict, expire=4
-                    )
+                    expire = 4
                else:
-                    RedisArchivist().set_message("message:add", mess_dict)
+                    expire = True
+
+                RedisArchivist().set_message(
+                    "message:add", mess_dict, expire=expire
+                )

            if idx + 1 % 25 == 0:
                print("thumbnail progress: " + progress)
@@ -226,7 +228,8 @@ class ThumbManager:
                "title": "Processing Channels",
                "message": "Downloading Channel Art.",
            }
-            RedisArchivist().set_message("message:download", mess_dict)
+            key = "message:download"
+            RedisArchivist().set_message(key, mess_dict, expire=True)

    def download_playlist(self, missing_playlists):
        """download needed artwork for playlists"""
@@ -243,7 +246,8 @@ class ThumbManager:
                "title": "Processing Playlists",
                "message": "Downloading Playlist Art.",
            }
-            RedisArchivist().set_message("message:download", mess_dict)
+            key = "message:download"
+            RedisArchivist().set_message(key, mess_dict, expire=True)

    def get_base64_blur(self, youtube_id):
        """return base64 encoded placeholder"""
--- a/tubearchivist/home/src/download/yt_dlp_base.py
+++ b/tubearchivist/home/src/download/yt_dlp_base.py
@@ -5,6 +5,8 @@ functionality:
 """

 import os
+from datetime import datetime
+from http import cookiejar
 from io import StringIO

 import yt_dlp
@@ -54,9 +56,12 @@ class YtWrap:
        """make extract request"""
        try:
            response = yt_dlp.YoutubeDL(self.obs).extract_info(url)
+        except cookiejar.LoadError:
+            print("cookie file is invalid")
+            return False
        except (yt_dlp.utils.ExtractorError, yt_dlp.utils.DownloadError):
            print(f"{url}: failed to get info from youtube")
-            response = False
+            return False

        return response

@@ -78,25 +83,74 @@ class CookieHandler:
        """import cookie from file"""
        cache_path = self.config["application"]["cache_dir"]
        import_path = os.path.join(cache_path, "import", "cookies.google.txt")
-        with open(import_path, encoding="utf-8") as cookie_file:
-            cookie = cookie_file.read()

-        RedisArchivist().set_message("cookie", cookie, expire=False)
+        try:
+            with open(import_path, encoding="utf-8") as cookie_file:
+                cookie = cookie_file.read()
+        except FileNotFoundError as err:
+            print(f"cookie: {import_path} file not found")
+            raise err
+
+        self.set_cookie(cookie)

        os.remove(import_path)
        print("cookie: import successful")

+    def set_cookie(self, cookie):
+        """set cookie str and activate in cofig"""
+        RedisArchivist().set_message("cookie", cookie)
+        path = ".downloads.cookie_import"
+        RedisArchivist().set_message("config", True, path=path)
+        self.config["downloads"]["cookie_import"] = True
+        print("cookie: activated and stored in Redis")
+
    @staticmethod
    def revoke():
        """revoke cookie"""
        RedisArchivist().del_message("cookie")
+        RedisArchivist().del_message("cookie:valid")
+        RedisArchivist().set_message(
+            "config", False, path=".downloads.cookie_import"
+        )
        print("cookie: revoked")

    def validate(self):
        """validate cookie using the liked videos playlist"""
+        print("validating cookie")
        obs_request = {
            "skip_download": True,
            "extract_flat": True,
        }
-        response = YtWrap(obs_request, self.config).extract("LL")
-        return bool(response)
+        validator = YtWrap(obs_request, self.config)
+        response = bool(validator.extract("LL"))
+        self.store_validation(response)
+
+        # update in redis to avoid expiring
+        modified = validator.obs["cookiefile"].getvalue()
+        if modified:
+            RedisArchivist().set_message("cookie", modified)
+
+        if not response:
+            mess_dict = {
+                "status": "message:download",
+                "level": "error",
+                "title": "Cookie validation failed, exiting...",
+                "message": "",
+            }
+            RedisArchivist().set_message(
+                "message:download", mess_dict, expire=4
+            )
+            print("cookie validation failed, exiting...")
+
+        return response
+
+    @staticmethod
+    def store_validation(response):
+        """remember last validation"""
+        now = datetime.now()
+        message = {
+            "status": response,
+            "validated": int(now.strftime("%s")),
+            "validated_str": now.strftime("%Y-%m-%d %H:%M"),
+        }
+        RedisArchivist().set_message("cookie:valid", message)
--- a/tubearchivist/home/src/download/yt_dlp_handler.py
+++ b/tubearchivist/home/src/download/yt_dlp_handler.py
@@ -12,7 +12,7 @@ from datetime import datetime

 from home.src.download.queue import PendingList
 from home.src.download.subscriptions import PlaylistSubscription
-from home.src.download.yt_dlp_base import YtWrap
+from home.src.download.yt_dlp_base import CookieHandler, YtWrap
 from home.src.es.connect import ElasticWrap, IndexPaginate
 from home.src.index.channel import YoutubeChannel
 from home.src.index.playlist import YoutubePlaylist
@@ -125,18 +125,19 @@ class DownloadPostProcess:
            + f"{id_c + 1}/{len(self.download.channels)}"
        )
        message = f"Progress: {id_p + 1}/{len(all_channel_playlist)}"
+        key = "message:download"
        mess_dict = {
-            "status": "message:download",
+            "status": key,
            "level": "info",
            "title": title,
            "message": message,
        }
        if id_p + 1 == len(all_channel_playlist):
-            RedisArchivist().set_message(
-                "message:download", mess_dict, expire=4
-            )
+            expire = 4
        else:
-            RedisArchivist().set_message("message:download", mess_dict)
+            expire = True
+
+        RedisArchivist().set_message(key, mess_dict, expire=expire)


 class VideoDownloader:
@@ -145,6 +146,8 @@ class VideoDownloader:
    if not initiated with list, take from queue
    """

+    MSG = "message:download"
+
    def __init__(self, youtube_id_list=False):
        self.obs = False
        self.video_overwrites = False
@@ -155,10 +158,7 @@ class VideoDownloader:

    def run_queue(self):
        """setup download queue in redis loop until no more items"""
-        pending = PendingList()
-        pending.get_download()
-        pending.get_channels()
-        self.video_overwrites = pending.video_overwrites
+        self._setup_queue()

        queue = RedisQueue()

@@ -180,37 +180,48 @@ class VideoDownloader:
            )
            self.channels.add(vid_dict["channel"]["channel_id"])
            mess_dict = {
-                "status": "message:download",
+                "status": self.MSG,
                "level": "info",
                "title": "Moving....",
                "message": "Moving downloaded file to storage folder",
            }
-            RedisArchivist().set_message("message:download", mess_dict, False)
+            RedisArchivist().set_message(self.MSG, mess_dict)

            self.move_to_archive(vid_dict)
            mess_dict = {
-                "status": "message:download",
+                "status": self.MSG,
                "level": "info",
                "title": "Completed",
                "message": "",
            }
-            RedisArchivist().set_message("message:download", mess_dict, 10)
+            RedisArchivist().set_message(self.MSG, mess_dict, expire=10)
            self._delete_from_pending(youtube_id)

        # post processing
        self._add_subscribed_channels()
        DownloadPostProcess(self).run()

-    @staticmethod
-    def add_pending():
+    def _setup_queue(self):
+        """setup required and validate"""
+        if self.config["downloads"]["cookie_import"]:
+            valid = CookieHandler(self.config).validate()
+            if not valid:
+                return
+
+        pending = PendingList()
+        pending.get_download()
+        pending.get_channels()
+        self.video_overwrites = pending.video_overwrites
+
+    def add_pending(self):
        """add pending videos to download queue"""
        mess_dict = {
-            "status": "message:download",
+            "status": self.MSG,
            "level": "info",
            "title": "Looking for videos to download",
            "message": "Scanning your download queue.",
        }
-        RedisArchivist().set_message("message:download", mess_dict)
+        RedisArchivist().set_message(self.MSG, mess_dict, expire=True)
        pending = PendingList()
        pending.get_download()
        to_add = [i["youtube_id"] for i in pending.all_pending]
@@ -218,25 +229,20 @@ class VideoDownloader:
            # there is nothing pending
            print("download queue is empty")
            mess_dict = {
-                "status": "message:download",
+                "status": self.MSG,
                "level": "error",
                "title": "Download queue is empty",
                "message": "Add some videos to the queue first.",
            }
-            RedisArchivist().set_message("message:download", mess_dict)
+            RedisArchivist().set_message(self.MSG, mess_dict, expire=True)
            return

        RedisQueue().add_list(to_add)

-    @staticmethod
-    def _progress_hook(response):
+    def _progress_hook(self, response):
        """process the progress_hooks from yt_dlp"""
-        # title
-        path = os.path.split(response["filename"])[-1][12:]
-        filename = os.path.splitext(os.path.splitext(path)[0])[0]
-        filename_clean = filename.replace("_", " ")
-        title = "Downloading: " + filename_clean
-        # message
+        title = "Downloading: " + response["info_dict"]["title"]
+
        try:
            percent = response["_percent_str"]
            size = response["_total_bytes_str"]
@@ -245,13 +251,14 @@ class VideoDownloader:
            message = f"{percent} of {size} at {speed} - time left: {eta}"
        except KeyError:
            message = "processing"
+
        mess_dict = {
-            "status": "message:download",
+            "status": self.MSG,
            "level": "info",
            "title": title,
            "message": message,
        }
-        RedisArchivist().set_message("message:download", mess_dict)
+        RedisArchivist().set_message(self.MSG, mess_dict, expire=True)

    def _build_obs(self):
        """collection to build all obs passed to yt-dlp"""
@@ -264,11 +271,9 @@ class VideoDownloader:
        self.obs = {
            "default_search": "ytsearch",
            "merge_output_format": "mp4",
-            "restrictfilenames": True,
            "outtmpl": (
                self.config["application"]["cache_dir"]
-                + "/download/"
-                + self.config["application"]["file_template"]
+                + "/download/%(id)s.mp4"
            ),
            "progress_hooks": [self._progress_hook],
            "noprogress": True,
@@ -385,12 +390,12 @@ class VideoDownloader:
        for file_str in all_cached:
            if vid_dict["youtube_id"] in file_str:
                old_file = file_str
-        old_file_path = os.path.join(cache_dir, "download", old_file)
-        new_file_path = os.path.join(videos, vid_dict["media_url"])
+        old_path = os.path.join(cache_dir, "download", old_file)
+        new_path = os.path.join(videos, vid_dict["media_url"])
        # move media file and fix permission
-        shutil.move(old_file_path, new_file_path)
+        shutil.move(old_path, new_path, copy_function=shutil.copyfile)
        if host_uid and host_gid:
-            os.chown(new_file_path, host_uid, host_gid)
+            os.chown(new_path, host_uid, host_gid)

    @staticmethod
    def _delete_from_pending(youtube_id):
--- a/tubearchivist/home/src/es/connect.py
+++ b/tubearchivist/home/src/es/connect.py
@@ -91,16 +91,22 @@ class ElasticWrap:


 class IndexPaginate:
-    """use search_after to go through whole index"""
+    """use search_after to go through whole index
+    kwargs:
+    - size: int, overwrite DEFAULT_SIZE
+    - keep_source: bool, keep _source key from es resutls
+    - callback: obj, Class with run method collback for every loop
+    """

    DEFAULT_SIZE = 500

-    def __init__(self, index_name, data, size=False, keep_source=False):
+    def __init__(self, index_name, data, **kwargs):
        self.index_name = index_name
        self.data = data
        self.pit_id = False
-        self.size = size
-        self.keep_source = keep_source
+        self.size = kwargs.get("size")
+        self.keep_source = kwargs.get("keep_source")
+        self.callback = kwargs.get("callback")

    def get_results(self):
        """get all results"""
@@ -122,14 +128,13 @@ class IndexPaginate:
            print(self.data)
            raise ValueError("missing sort key in data")

-        size = self.size or self.DEFAULT_SIZE
-
-        self.data["size"] = size
+        self.data["size"] = self.size or self.DEFAULT_SIZE
        self.data["pit"] = {"id": self.pit_id, "keep_alive": "10m"}

    def run_loop(self):
        """loop through results until last hit"""
        all_results = []
+        counter = 0
        while True:
            response, _ = ElasticWrap("_search").get(data=self.data)
            all_hits = response["hits"]["hits"]
@@ -139,10 +144,18 @@ class IndexPaginate:
                        source = hit
                    else:
                        source = hit["_source"]
-                    search_after = hit["sort"]
-                    all_results.append(source)
+
+                    if not self.callback:
+                        all_results.append(source)
+
+                if self.callback:
+                    self.callback(all_hits, self.index_name).run()
+                    if counter % 10 == 0:
+                        print(f"{self.index_name}: processing page {counter}")
+                    counter = counter + 1
+
                # update search_after with last hit data
-                self.data["search_after"] = search_after
+                self.data["search_after"] = all_hits[-1]["sort"]
            else:
                break

--- a/tubearchivist/home/src/es/index_setup.py
+++ b/tubearchivist/home/src/es/index_setup.py
@@ -137,35 +137,24 @@ class ElasticIndex:
        _, _ = ElasticWrap(f"ta_{self.index_name}").put(data)


-class ElasticBackup:
-    """dump index to nd-json files for later bulk import"""
+class BackupCallback:
+    """handle backup ndjson writer as callback for IndexPaginate"""

-    def __init__(self, index_config, reason):
-        self.config = AppConfig().config
-        self.cache_dir = self.config["application"]["cache_dir"]
-        self.index_config = index_config
-        self.reason = reason
+    def __init__(self, source, index_name):
+        self.source = source
+        self.index_name = index_name
        self.timestamp = datetime.now().strftime("%Y%m%d")
-        self.backup_files = []

-    @staticmethod
-    def get_all_documents(index_name):
-        """export all documents of a single index"""
-        data = {
-            "query": {"match_all": {}},
-            "sort": [{"_doc": {"order": "desc"}}],
-        }
-        paginate = IndexPaginate(f"ta_{index_name}", data, keep_source=True)
-        all_results = paginate.get_results()
+    def run(self):
+        """run the junk task"""
+        file_content = self._build_bulk()
+        self._write_es_json(file_content)

-        return all_results
-
-    @staticmethod
-    def build_bulk(all_results):
+    def _build_bulk(self):
        """build bulk query data from all_results"""
        bulk_list = []

-        for document in all_results:
+        for document in self.source:
            document_id = document["_id"]
            es_index = document["_index"]
            action = {"index": {"_index": es_index, "_id": document_id}}
@@ -179,40 +168,56 @@ class ElasticBackup:

        return file_content

-    def write_es_json(self, file_content, index_name):
+    def _write_es_json(self, file_content):
        """write nd-json file for es _bulk API to disk"""
-        file_name = f"es_{index_name}-{self.timestamp}.json"
-        file_path = os.path.join(self.cache_dir, "backup", file_name)
-        with open(file_path, "w", encoding="utf-8") as f:
+        cache_dir = AppConfig().config["application"]["cache_dir"]
+        file_name = f"es_{self.index_name.lstrip('ta_')}-{self.timestamp}.json"
+        file_path = os.path.join(cache_dir, "backup", file_name)
+        with open(file_path, "a+", encoding="utf-8") as f:
            f.write(file_content)

-        self.backup_files.append(file_path)

-    def write_ta_json(self, all_results, index_name):
-        """write generic json file to disk"""
-        file_name = f"ta_{index_name}-{self.timestamp}.json"
-        file_path = os.path.join(self.cache_dir, "backup", file_name)
-        to_write = [i["_source"] for i in all_results]
-        file_content = json.dumps(to_write)
-        with open(file_path, "w", encoding="utf-8") as f:
-            f.write(file_content)
+class ElasticBackup:
+    """dump index to nd-json files for later bulk import"""

-        self.backup_files.append(file_path)
+    def __init__(self, index_config, reason):
+        self.config = AppConfig().config
+        self.cache_dir = self.config["application"]["cache_dir"]
+        self.timestamp = datetime.now().strftime("%Y%m%d")
+        self.index_config = index_config
+        self.reason = reason
+
+    @staticmethod
+    def backup_index(index_name):
+        """export all documents of a single index"""
+        data = {
+            "query": {"match_all": {}},
+            "sort": [{"_doc": {"order": "desc"}}],
+        }
+        paginate = IndexPaginate(
+            f"ta_{index_name}", data, keep_source=True, callback=BackupCallback
+        )
+        _ = paginate.get_results()

    def zip_it(self):
        """pack it up into single zip file"""
        file_name = f"ta_backup-{self.timestamp}-{self.reason}.zip"
-        backup_folder = os.path.join(self.cache_dir, "backup")
-        backup_file = os.path.join(backup_folder, file_name)
+        folder = os.path.join(self.cache_dir, "backup")

-        with zipfile.ZipFile(
-            backup_file, "w", compression=zipfile.ZIP_DEFLATED
-        ) as zip_f:
-            for backup_file in self.backup_files:
+        to_backup = []
+        for file in os.listdir(folder):
+            if file.endswith(".json"):
+                to_backup.append(os.path.join(folder, file))
+
+        backup_file = os.path.join(folder, file_name)
+
+        comp = zipfile.ZIP_DEFLATED
+        with zipfile.ZipFile(backup_file, "w", compression=comp) as zip_f:
+            for backup_file in to_backup:
                zip_f.write(backup_file, os.path.basename(backup_file))

        # cleanup
-        for backup_file in self.backup_files:
+        for backup_file in to_backup:
            os.remove(backup_file)

    def post_bulk_restore(self, file_name):
@@ -380,10 +385,7 @@ def backup_all_indexes(reason):
        print(f"backup: export in progress for {index_name}")
        if not backup_handler.index_exists(index_name):
            continue
-        all_results = backup_handler.get_all_documents(index_name)
-        file_content = backup_handler.build_bulk(all_results)
-        backup_handler.write_es_json(file_content, index_name)
-        backup_handler.write_ta_json(all_results, index_name)
+        backup_handler.backup_index(index_name)

    backup_handler.zip_it()

--- a/tubearchivist/home/src/frontend/api_calls.py
+++ b/tubearchivist/home/src/frontend/api_calls.py
@@ -98,7 +98,7 @@ class PostData:
        origin, new_view = self.exec_val.split(":")
        key = f"{self.current_user}:view:{origin}"
        print(f"change view: {key} to {new_view}")
-        RedisArchivist().set_message(key, {"status": new_view}, expire=False)
+        RedisArchivist().set_message(key, {"status": new_view})
        return {"success": True}

    def _change_grid(self):
@@ -109,7 +109,7 @@ class PostData:

        key = f"{self.current_user}:grid_items"
        print(f"change grid items: {grid_items}")
-        RedisArchivist().set_message(key, {"status": grid_items}, expire=False)
+        RedisArchivist().set_message(key, {"status": grid_items})
        return {"success": True}

    @staticmethod
@@ -135,7 +135,7 @@ class PostData:
        running = download_pending.delay()
        task_id = running.id
        print(f"{task_id}: set task id")
-        RedisArchivist().set_message("dl_queue_id", task_id, expire=False)
+        RedisArchivist().set_message("dl_queue_id", task_id)
        return {"success": True}

    def _queue_handler(self):
@@ -187,11 +187,11 @@ class PostData:
        sort_order = {"status": self.exec_val}
        if self.exec_val in ["asc", "desc"]:
            RedisArchivist().set_message(
-                f"{self.current_user}:sort_order", sort_order, expire=False
+                f"{self.current_user}:sort_order", sort_order
            )
        else:
            RedisArchivist().set_message(
-                f"{self.current_user}:sort_by", sort_order, expire=False
+                f"{self.current_user}:sort_by", sort_order
            )
        return {"success": True}

@@ -200,7 +200,7 @@ class PostData:
        key = f"{self.current_user}:hide_watched"
        message = {"status": bool(int(self.exec_val))}
        print(f"toggle {key}: {message}")
-        RedisArchivist().set_message(key, message, expire=False)
+        RedisArchivist().set_message(key, message)
        return {"success": True}

    def _show_subed_only(self):
@@ -208,7 +208,7 @@ class PostData:
        key = f"{self.current_user}:show_subed_only"
        message = {"status": bool(int(self.exec_val))}
        print(f"toggle {key}: {message}")
-        RedisArchivist().set_message(key, message, expire=False)
+        RedisArchivist().set_message(key, message)
        return {"success": True}

    def _dlnow(self):
@@ -218,7 +218,7 @@ class PostData:
        running = download_single.delay(youtube_id=youtube_id)
        task_id = running.id
        print("set task id: " + task_id)
-        RedisArchivist().set_message("dl_queue_id", task_id, expire=False)
+        RedisArchivist().set_message("dl_queue_id", task_id)
        return {"success": True}

    def _show_ignored_only(self):
@@ -227,7 +227,7 @@ class PostData:
        key = f"{self.current_user}:show_ignored_only"
        value = {"status": show_value}
        print(f"Filter download view ignored only: {show_value}")
-        RedisArchivist().set_message(key, value, expire=False)
+        RedisArchivist().set_message(key, value)
        return {"success": True}

    def _forget_ignore(self):
--- a/tubearchivist/home/src/frontend/searching.py
+++ b/tubearchivist/home/src/frontend/searching.py
@@ -136,6 +136,11 @@ class SearchHandler:
                date_str = datetime.strftime(date_refresh, "%d %b, %Y")
                hit["source"]["channel"]["channel_last_refresh"] = date_str

+        if "subtitle_fragment_id" in hit_keys:
+            youtube_id = hit["source"]["youtube_id"]
+            thumb_path = ThumbManager().vid_thumb_path(youtube_id)
+            hit["source"]["vid_thumb_url"] = f"/cache/{thumb_path}"
+
        return hit


@@ -146,38 +151,12 @@ class SearchForm:

    def multi_search(self, search_query):
        """searching through index"""
-        path = "ta_video,ta_channel,ta_playlist/_search"
-        data = {
-            "size": 30,
-            "query": {
-                "multi_match": {
-                    "query": search_query,
-                    "type": "bool_prefix",
-                    "operator": "and",
-                    "fuzziness": "auto",
-                    "fields": [
-                        "category",
-                        "channel_description",
-                        "channel_name._2gram",
-                        "channel_name._3gram",
-                        "channel_name.search_as_you_type",
-                        "playlist_description",
-                        "playlist_name._2gram",
-                        "playlist_name._3gram",
-                        "playlist_name.search_as_you_type",
-                        "tags",
-                        "title._2gram",
-                        "title._3gram",
-                        "title.search_as_you_type",
-                    ],
-                }
-            },
-        }
-        look_up = SearchHandler(path, config=self.CONFIG, data=data)
+        path, query, query_type = SearchParser(search_query).run()
+        look_up = SearchHandler(path, config=self.CONFIG, data=query)
        search_results = look_up.get_data()
        all_results = self.build_results(search_results)

-        return {"results": all_results}
+        return {"results": all_results, "queryType": query_type}

    @staticmethod
    def build_results(search_results):
@@ -185,6 +164,7 @@ class SearchForm:
        video_results = []
        channel_results = []
        playlist_results = []
+        fulltext_results = []
        if search_results:
            for result in search_results:
                if result["_index"] == "ta_video":
@@ -193,11 +173,337 @@ class SearchForm:
                    channel_results.append(result)
                elif result["_index"] == "ta_playlist":
                    playlist_results.append(result)
+                elif result["_index"] == "ta_subtitle":
+                    fulltext_results.append(result)

        all_results = {
            "video_results": video_results,
            "channel_results": channel_results,
            "playlist_results": playlist_results,
+            "fulltext_results": fulltext_results,
        }

        return all_results
+
+
+class SearchParser:
+    """handle structured searches"""
+
+    def __init__(self, search_query):
+        self.query_words = search_query.lower().split()
+        self.query_map = False
+        self.append_to = "term"
+
+    def run(self):
+        """collection, return path and query dict for es"""
+        print(f"query words: {self.query_words}")
+        query_type = self._find_map()
+        self._run_words()
+        self._delete_unset()
+        self._match_data_types()
+
+        path, query = QueryBuilder(self.query_map, query_type).run()
+
+        return path, query, query_type
+
+    def _find_map(self):
+        """find query in keyword map"""
+        first_word = self.query_words[0]
+        key_word_map = self._get_map()
+
+        if ":" in first_word:
+            index_match, query_string = first_word.split(":")
+            if index_match in key_word_map:
+                self.query_map = key_word_map.get(index_match)
+                self.query_words[0] = query_string
+                return index_match
+
+        self.query_map = key_word_map.get("simple")
+        print(f"query_map: {self.query_map}")
+
+        return "simple"
+
+    @staticmethod
+    def _get_map():
+        """return map to build on"""
+        return {
+            "simple": {
+                "index": "ta_video,ta_channel,ta_playlist",
+                "term": [],
+            },
+            "video": {
+                "index": "ta_video",
+                "term": [],
+                "channel": [],
+                "active": [],
+            },
+            "channel": {
+                "index": "ta_channel",
+                "term": [],
+                "active": [],
+                "subscribed": [],
+            },
+            "playlist": {
+                "index": "ta_playlist",
+                "term": [],
+                "active": [],
+                "subscribed": [],
+            },
+            "full": {
+                "index": "ta_subtitle",
+                "term": [],
+                "lang": [],
+                "source": [],
+            },
+        }
+
+    def _run_words(self):
+        """append word by word"""
+        for word in self.query_words:
+            if ":" in word:
+                keyword, search_string = word.split(":")
+                if keyword in self.query_map:
+                    self.append_to = keyword
+                    word = search_string
+
+            if word:
+                self.query_map[self.append_to].append(word)
+
+    def _delete_unset(self):
+        """delete unset keys"""
+        new_query_map = {}
+        for key, value in self.query_map.items():
+            if value:
+                new_query_map.update({key: value})
+        self.query_map = new_query_map
+
+    def _match_data_types(self):
+        """match values with data types"""
+        for key, value in self.query_map.items():
+            if key in ["term", "channel"]:
+                self.query_map[key] = " ".join(self.query_map[key])
+            if key in ["active", "subscribed"]:
+                self.query_map[key] = "yes" in value
+
+
+class QueryBuilder:
+    """build query for ES from form data"""
+
+    def __init__(self, query_map, query_type):
+        self.query_map = query_map
+        self.query_type = query_type
+
+    def run(self):
+        """build query"""
+        path = self._build_path()
+        query = self.build_query()
+        print(f"es path: {path}")
+        print(f"query: {query}")
+
+        return path, query
+
+    def _build_path(self):
+        """build es index search path"""
+        return f"{self.query_map.get('index')}/_search"
+
+    def build_query(self):
+        """build query based on query_type"""
+
+        exec_map = {
+            "simple": self._build_simple,
+            "video": self._build_video,
+            "channel": self._build_channel,
+            "playlist": self._build_playlist,
+            "full": self._build_fulltext,
+        }
+
+        build_must_list = exec_map[self.query_type]
+
+        if self.query_type == "full":
+            query = build_must_list()
+        else:
+            query = {
+                "size": 30,
+                "query": {"bool": {"must": build_must_list()}},
+            }
+
+        return query
+
+    def _build_simple(self):
+        """build simple cross index query"""
+        must_list = []
+
+        if (term := self.query_map.get("term")) is not None:
+            must_list.append(
+                {
+                    "multi_match": {
+                        "query": term,
+                        "type": "bool_prefix",
+                        "fuzziness": "auto",
+                        "operator": "and",
+                        "fields": [
+                            "channel_name._2gram",
+                            "channel_name._3gram",
+                            "channel_name.search_as_you_type",
+                            "playlist_name._2gram",
+                            "playlist_name._3gram",
+                            "playlist_name.search_as_you_type",
+                            "title._2gram",
+                            "title._3gram",
+                            "title.search_as_you_type",
+                        ],
+                    }
+                }
+            )
+
+        return must_list
+
+    def _build_video(self):
+        """build video query"""
+        must_list = []
+
+        if (term := self.query_map.get("term")) is not None:
+            must_list.append(
+                {
+                    "multi_match": {
+                        "query": term,
+                        "type": "bool_prefix",
+                        "fuzziness": "auto",
+                        "operator": "and",
+                        "fields": [
+                            "title._2gram^2",
+                            "title._3gram^2",
+                            "title.search_as_you_type^2",
+                            "tags",
+                            "category",
+                        ],
+                    }
+                }
+            )
+
+        if (active := self.query_map.get("active")) is not None:
+            must_list.append({"term": {"active": {"value": active}}})
+
+        if (channel := self.query_map.get("channel")) is not None:
+            must_list.append(
+                {
+                    "multi_match": {
+                        "query": channel,
+                        "type": "bool_prefix",
+                        "fuzziness": "auto",
+                        "operator": "and",
+                        "fields": [
+                            "channel.channel_name._2gram",
+                            "channel.channel_name._3gram",
+                            "channel.channel_name.search_as_you_type",
+                        ],
+                    }
+                }
+            )
+
+        return must_list
+
+    def _build_channel(self):
+        """build query for channel"""
+        must_list = []
+
+        if (term := self.query_map.get("term")) is not None:
+            must_list.append(
+                {
+                    "multi_match": {
+                        "query": term,
+                        "type": "bool_prefix",
+                        "fuzziness": "auto",
+                        "operator": "and",
+                        "fields": [
+                            "channel_description",
+                            "channel_name._2gram^2",
+                            "channel_name._3gram^2",
+                            "channel_name.search_as_you_type^2",
+                        ],
+                    }
+                }
+            )
+
+        if (active := self.query_map.get("active")) is not None:
+            must_list.append({"term": {"channel_active": {"value": active}}})
+
+        if (subscribed := self.query_map.get("subscribed")) is not None:
+            must_list.append(
+                {"term": {"channel_subscribed": {"value": subscribed}}}
+            )
+
+        return must_list
+
+    def _build_playlist(self):
+        """build query for playlist"""
+        must_list = []
+
+        if (term := self.query_map.get("term")) is not None:
+            must_list.append(
+                {
+                    "multi_match": {
+                        "query": term,
+                        "type": "bool_prefix",
+                        "fuzziness": "auto",
+                        "operator": "and",
+                        "fields": [
+                            "playlist_description",
+                            "playlist_name._2gram^2",
+                            "playlist_name._3gram^2",
+                            "playlist_name.search_as_you_type^2",
+                        ],
+                    }
+                }
+            )
+
+        if (active := self.query_map.get("active")) is not None:
+            must_list.append({"term": {"playlist_active": {"value": active}}})
+
+        if (subscribed := self.query_map.get("subscribed")) is not None:
+            must_list.append(
+                {"term": {"playlist_subscribed": {"value": subscribed}}}
+            )
+
+        return must_list
+
+    def _build_fulltext(self):
+        """build query for fulltext search"""
+        must_list = []
+
+        if (term := self.query_map.get("term")) is not None:
+            must_list.append(
+                {
+                    "match": {
+                        "subtitle_line": {
+                            "query": term,
+                            "fuzziness": "auto",
+                        }
+                    }
+                }
+            )
+
+        if (lang := self.query_map.get("lang")) is not None:
+            must_list.append({"term": {"subtitle_lang": {"value": lang[0]}}})
+
+        if (source := self.query_map.get("source")) is not None:
+            must_list.append(
+                {"term": {"subtitle_source": {"value": source[0]}}}
+            )
+
+        query = {
+            "size": 30,
+            "_source": {"excludes": "subtitle_line"},
+            "query": {"bool": {"must": must_list}},
+            "highlight": {
+                "fields": {
+                    "subtitle_line": {
+                        "number_of_fragments": 0,
+                        "pre_tags": ['<span class="settings-current">'],
+                        "post_tags": ["</span>"],
+                    }
+                }
+            },
+        }
+
+        return query
--- a/tubearchivist/home/src/index/channel.py
+++ b/tubearchivist/home/src/index/channel.py
@@ -37,6 +37,9 @@ class ChannelScraper:
        """main method to return channel dict"""
        self.get_soup()
        self._extract_yt_json()
+        if self._is_deactivated():
+            return False
+
        self._parse_channel_main()
        self._parse_channel_meta()
        return self.json_data
@@ -68,6 +71,16 @@ class ChannelScraper:
        json_raw = script_content.rstrip(";</script>")
        self.yt_json = json.loads(json_raw)

+    def _is_deactivated(self):
+        """check if channel is deactivated"""
+        alert_text = "This channel does not exist."
+        alerts = self.yt_json.get("alerts")
+        if alerts and alert_text in str(alerts):
+            print(f"{self.channel_id}: {alert_text}")
+            return True
+
+        return False
+
    def _parse_channel_main(self):
        """extract maintab values from scraped channel json data"""
        main_tab = self.yt_json["header"]["c4TabbedHeaderRenderer"]
@@ -153,6 +166,7 @@ class YoutubeChannel(YouTubeItem):
    es_path = False
    index_name = "ta_channel"
    yt_base = "https://www.youtube.com/channel/"
+    msg = "message:playlistscan"

    def __init__(self, youtube_id):
        super().__init__(youtube_id)
@@ -252,12 +266,12 @@ class YoutubeChannel(YouTubeItem):
        self.get_from_es()
        channel_name = self.json_data["channel_name"]
        mess_dict = {
-            "status": "message:playlistscan",
+            "status": self.msg,
            "level": "info",
            "title": "Looking for playlists",
            "message": f"{channel_name}: Scanning channel in progress",
        }
-        RedisArchivist().set_message("message:playlistscan", mess_dict)
+        RedisArchivist().set_message(self.msg, mess_dict, expire=True)
        self.get_all_playlists()
        if not self.all_playlists:
            print(f"{self.youtube_id}: no playlists found.")
@@ -272,12 +286,12 @@ class YoutubeChannel(YouTubeItem):
        """send notification"""
        channel_name = self.json_data["channel_name"]
        mess_dict = {
-            "status": "message:playlistscan",
+            "status": self.msg,
            "level": "info",
            "title": f"{channel_name}: Scanning channel for playlists",
            "message": f"Progress: {idx + 1}/{len(self.all_playlists)}",
        }
-        RedisArchivist().set_message("message:playlistscan", mess_dict)
+        RedisArchivist().set_message(self.msg, mess_dict, expire=True)
        print("add playlist: " + playlist[1])

    @staticmethod
--- a/tubearchivist/home/src/index/filesystem.py
+++ b/tubearchivist/home/src/index/filesystem.py
@@ -266,7 +266,7 @@ class ManualImport:
        if ext == ".mp4":
            new_file = video_file + ext
            dest_path = os.path.join(self.CACHE_DIR, "download", new_file)
-            shutil.move(video_path, dest_path)
+            shutil.move(video_path, dest_path, copy_function=shutil.copyfile)
        else:
            print(f"processing with ffmpeg: {video_file}")
            new_file = video_file + ".mp4"
@@ -310,4 +310,4 @@ def reindex_old_documents():
    handler = Reindex()
    handler.check_outdated()
    handler.reindex()
-    RedisArchivist().set_message("last_reindex", handler.now, expire=False)
+    RedisArchivist().set_message("last_reindex", handler.now)
--- a/tubearchivist/home/src/index/reindex.py
+++ b/tubearchivist/home/src/index/reindex.py
@@ -12,6 +12,7 @@ from time import sleep

 from home.src.download.queue import PendingList
 from home.src.download.thumbnails import ThumbManager
+from home.src.download.yt_dlp_base import CookieHandler
 from home.src.download.yt_dlp_handler import VideoDownloader
 from home.src.es.connect import ElasticWrap
 from home.src.index.channel import YoutubeChannel
@@ -40,6 +41,13 @@ class Reindex:
        self.all_channel_ids = False
        self.all_playlist_ids = False

+    def check_cookie(self):
+        """validate cookie if enabled"""
+        if self.config["downloads"]["cookie_import"]:
+            valid = CookieHandler(self.config).validate()
+            if not valid:
+                return
+
    def _get_daily(self):
        """get daily refresh values"""
        total_videos = self._get_total_hits("ta_video")
@@ -187,6 +195,10 @@ class Reindex:
        subscribed = channel.json_data["channel_subscribed"]
        overwrites = channel.json_data.get("channel_overwrites", False)
        channel.get_from_youtube()
+        if not channel.json_data:
+            channel.deactivate()
+            return
+
        channel.json_data["channel_subscribed"] = subscribed
        if overwrites:
            channel.json_data["channel_overwrites"] = overwrites
@@ -293,9 +305,9 @@ class ChannelUrlFixer:
        """fix filepath"""
        print(f"{self.youtube_id}: fixing channel rename.")
        cache_dir = self.config["application"]["cache_dir"]
-        new_file_path = os.path.join(
+        new_path = os.path.join(
            cache_dir, "download", self.youtube_id + ".mp4"
        )
-        shutil.move(video_path_is, new_file_path)
+        shutil.move(video_path_is, new_path, copy_function=shutil.copyfile)
        VideoDownloader().move_to_archive(self.video.json_data)
        self.video.update_media_url()
--- a/tubearchivist/home/src/index/video.py
+++ b/tubearchivist/home/src/index/video.py
@@ -339,7 +339,7 @@ class SponsorBlock:
        sb_id = RedisArchivist().get_message(key)
        if not sb_id["status"]:
            sb_id = {"status": randomizor(32)}
-            RedisArchivist().set_message(key, sb_id, expire=False)
+            RedisArchivist().set_message(key, sb_id)

        return sb_id

--- a/tubearchivist/home/src/ta/config.py
+++ b/tubearchivist/home/src/ta/config.py
@@ -99,7 +99,7 @@ class AppConfig:
            self.config[config_dict][config_value] = to_write
            updated.append((config_value, to_write))

-        RedisArchivist().set_message("config", self.config, expire=False)
+        RedisArchivist().set_message("config", self.config)
        return updated

    @staticmethod
@@ -111,7 +111,7 @@ class AppConfig:

            message = {"status": value}
            redis_key = f"{user_id}:{key}"
-            RedisArchivist().set_message(redis_key, message, expire=False)
+            RedisArchivist().set_message(redis_key, message)

    def get_colors(self):
        """overwrite config if user has set custom values"""
@@ -133,6 +133,8 @@ class AppConfig:

        # check for customizations
        if not redis_config:
+            config = self.get_config()
+            RedisArchivist().set_message("config", config)
            return

        needs_update = False
@@ -151,7 +153,7 @@ class AppConfig:
                    needs_update = True

        if needs_update:
-            RedisArchivist().set_message("config", redis_config, expire=False)
+            RedisArchivist().set_message("config", redis_config)


 class ScheduleBuilder:
@@ -165,6 +167,7 @@ class ScheduleBuilder:
        "run_backup": "0 18 0",
    }
    CONFIG = ["check_reindex_days", "run_backup_rotate"]
+    MSG = "message:setting"

    def __init__(self):
        self.config = AppConfig().config
@@ -180,25 +183,27 @@ class ScheduleBuilder:
                except ValueError:
                    print(f"failed: {key} {value}")
                    mess_dict = {
-                        "status": "message:setting",
+                        "status": self.MSG,
                        "level": "error",
                        "title": "Scheduler update failed.",
                        "message": "Invalid schedule input",
                    }
-                    RedisArchivist().set_message("message:setting", mess_dict)
+                    RedisArchivist().set_message(
+                        self.MSG, mess_dict, expire=True
+                    )
                    return

                redis_config["scheduler"][key] = to_write
            if key in self.CONFIG and value:
                redis_config["scheduler"][key] = int(value)
-        RedisArchivist().set_message("config", redis_config, expire=False)
+        RedisArchivist().set_message("config", redis_config)
        mess_dict = {
-            "status": "message:setting",
+            "status": self.MSG,
            "level": "info",
            "title": "Scheduler changed.",
            "message": "Please restart container for changes to take effect",
        }
-        RedisArchivist().set_message("message:setting", mess_dict)
+        RedisArchivist().set_message(self.MSG, mess_dict, expire=True)

    def value_builder(self, key, value):
        """validate single cron form entry and return cron dict"""
--- a/tubearchivist/home/src/ta/helper.py
+++ b/tubearchivist/home/src/ta/helper.py
@@ -105,6 +105,18 @@ def date_praser(timestamp):
    return datetime.strftime(date_obj, "%d %b, %Y")


+def time_parser(timestamp):
+    """return seconds from timestamp, false on empty"""
+    if not timestamp:
+        return False
+
+    if timestamp.isnumeric():
+        return int(timestamp)
+
+    hours, minutes, seconds = timestamp.split(":", maxsplit=3)
+    return int(hours) * 60 * 60 + int(minutes) * 60 + float(seconds)
+
+
 class UrlListParser:
    """take a multi line string and detect valid youtube ids"""

--- a/tubearchivist/home/src/ta/ta_redis.py
+++ b/tubearchivist/home/src/ta/ta_redis.py
@@ -34,10 +34,10 @@ class RedisArchivist(RedisBase):
        "setting",
    ]

-    def set_message(self, key, message, expire=True):
+    def set_message(self, key, message, path=".", expire=False):
        """write new message to redis"""
        self.conn.execute_command(
-            "JSON.SET", self.NAME_SPACE + key, ".", json.dumps(message)
+            "JSON.SET", self.NAME_SPACE + key, path, json.dumps(message)
        )

        if expire:
@@ -80,6 +80,12 @@ class RedisArchivist(RedisBase):
        redis_lock = self.conn.lock(self.NAME_SPACE + lock_key)
        return redis_lock

+    def is_locked(self, lock_key):
+        """check if lock is set"""
+        lock_name = self.NAME_SPACE + lock_key
+        lock_status = bool(self.conn.execute_command("GET", lock_name))
+        return lock_status
+
    def get_progress(self):
        """get a list of all progress messages"""
        all_messages = []
--- a/tubearchivist/home/tasks.py
+++ b/tubearchivist/home/tasks.py
@@ -48,7 +48,7 @@ def update_subscribed():
        "title": "Rescanning channels and playlists.",
        "message": "Looking for new videos.",
    }
-    RedisArchivist().set_message("message:rescan", message)
+    RedisArchivist().set_message("message:rescan", message, expire=True)

    have_lock = False
    my_lock = RedisArchivist().get_lock("rescan")
@@ -108,13 +108,14 @@ def download_single(youtube_id):
    try:
        have_lock = my_lock.acquire(blocking=False)
        if have_lock:
+            key = "message:download"
            mess_dict = {
-                "status": "message:download",
+                "status": key,
                "level": "info",
                "title": "Download single video",
                "message": "processing",
            }
-            RedisArchivist().set_message("message:download", mess_dict)
+            RedisArchivist().set_message(key, mess_dict, expire=True)
            VideoDownloader().run_queue()
        else:
            print("Download queue already running.")
@@ -196,7 +197,7 @@ def kill_dl(task_id):
        "title": "Canceling download process",
        "message": "Canceling download queue now.",
    }
-    RedisArchivist().set_message("message:download", mess_dict)
+    RedisArchivist().set_message("message:download", mess_dict, expire=True)


@shared_task
@@ -245,13 +246,14 @@ def subscribe_to(url_str):
            channel_id_sub, channel_subscribed=True
        )
        # notify for channels
+        key = "message:subchannel"
        message = {
-            "status": "message:subchannel",
+            "status": key,
            "level": "info",
            "title": "Subscribing to Channels",
            "message": f"Processing {counter} of {len(to_subscribe_list)}",
        }
-        RedisArchivist().set_message("message:subchannel", message=message)
+        RedisArchivist().set_message(key, message=message, expire=True)
        counter = counter + 1


@@ -260,13 +262,14 @@ def index_channel_playlists(channel_id):
    """add all playlists of channel to index"""
    channel = YoutubeChannel(channel_id)
    # notify
+    key = "message:playlistscan"
    mess_dict = {
-        "status": "message:playlistscan",
+        "status": key,
        "level": "info",
        "title": "Looking for playlists",
        "message": f'Scanning channel "{channel.youtube_id}" in progress',
    }
-    RedisArchivist().set_message("message:playlistscan", mess_dict)
+    RedisArchivist().set_message(key, mess_dict, expire=True)
    channel.index_channel_playlists()


--- a/tubearchivist/home/templates/home/channel_id.html
+++ b/tubearchivist/home/templates/home/channel_id.html
@@ -6,8 +6,13 @@
    <div class="channel-banner">
        <a href="/channel/{{ channel_info.channel_id }}/"><img src="/cache/channels/{{ channel_info.channel_id }}_banner.jpg" alt="channel_banner"></a>
    </div>
+    <div class="info-box-item channel-nav">
+        <a href="{% url 'channel_id' channel_info.channel_id %}"><h3>Videos</h3></a>
+        <a href="{% url 'channel_id_playlist' channel_info.channel_id %}"><h3>Playlists</h3></a>
+        <a href="{% url 'channel_id_about' channel_info.channel_id %}"><h3>About</h3></a>
+    </div>
    <div id="notifications" data="channel_id"></div>
-    <div class="info-box info-box-3">
+    <div class="info-box info-box-2">
        <div class="info-box-item">
            <div class="round-img">
                <a href="{% url 'channel_id' channel_info.channel_id %}">
@@ -30,88 +35,13 @@
        </div>
        <div class="info-box-item">
            <div>
-                <p>Last refreshed: {{ channel_info.channel_last_refresh }}</p>
-                {% if channel_info.channel_active %}
-                    <p>Youtube: <a href="https://www.youtube.com/channel/{{ channel_info.channel_id }}" target="_blank">Active</a></p>
-                {% else %}
-                    <p>Youtube: Deactivated</p>
-                {% endif %}
-                <button onclick="deleteConfirm()" id="delete-item">Delete Channel</button>
-                <div class="delete-confirm" id="delete-button">
-                    <span>Delete {{ channel_info.channel_name }} including all videos? </span><button class="danger-button" onclick="deleteChannel(this)" data-id="{{ channel_info.channel_id }}">Delete</button> <button onclick="cancelDelete()">Cancel</button>
-                </div>
-            </div>
-        </div>
-        <div class="info-box-item">
-            <div>
-                {% if channel_info.channel_views >= 1000000 %}
-                    <p>Channel views: {{ channel_info.channel_views|intword }}</p>
-                {% elif channel_info.channel_views > 0 %}
-                    <p>Channel views: {{ channel_info.channel_views|intcomma }}</p>
-                {% endif %}
                {% if max_hits %}
-                    <p>Total Videos archived: {{ max_hits }}</p>
-                    <p>Watched: <button title="Mark all videos from {{ channel_info.channel_name }} as watched" type="button" id="watched-button" data-id="{{ channel_info.channel_id }}" onclick="isWatchedButton(this)">Mark as watched</button></p>
+                    <p>Total Videos: {{ max_hits }}</p>
+                    <button title="Mark all videos from {{ channel_info.channel_name }} as watched" type="button" id="watched-button" data-id="{{ channel_info.channel_id }}" onclick="isWatchedButton(this)">Mark as watched</button>
                {% endif %}
-                <button {% if channel_info.channel_overwrites %} class="danger-button"{% endif %} onclick="showOverwrite()" title="Overwrite settings for channel {{ channel_info.channel_name }}">Configure</button>
-                <a href="/playlist/?search={{ channel_info.channel_id }}" title="Show all playlists belonging to {{ channel_info.channel_name }}"><button>Show Playlists</button></a>
            </div>
        </div>
    </div>
-    <div id="overwrite-form" class="info-box{% if not channel_info.channel_overwrites %} hidden-overwrite{% endif %}">
-        <div class="info-box-item">
-            <form class="overwrite-form" action="/channel/{{ channel_info.channel_id }}/" method="POST">
-                {% csrf_token %}
-                <div class="overwrite-form-item">
-                    <p>Download format: <span class="settings-current">
-                        {% if channel_info.channel_overwrites.download_format %}
-                            {{ channel_info.channel_overwrites.download_format }}
-                        {% else %}
-                            False
-                        {% endif %}</span></p>
-                    {{ channel_overwrite_form.download_format }}<br>
-                </div>
-                <div class="overwrite-form-item">
-                    <p>Auto delete watched videos after x days: <span class="settings-current">
-                        {% if channel_info.channel_overwrites.autodelete_days %}
-                            {{ channel_info.channel_overwrites.autodelete_days }}
-                        {% else %}
-                            False
-                        {% endif %}</span></p>
-                    {{ channel_overwrite_form.autodelete_days }}<br>
-                </div>
-                <div class="overwrite-form-item">
-                    <p>Index playlists: <span class="settings-current">
-                        {% if channel_info.channel_overwrites.index_playlists %}
-                            {{ channel_info.channel_overwrites.index_playlists }}
-                        {% else %}
-                            False
-                        {% endif %}</span></p>
-                    {{ channel_overwrite_form.index_playlists }}<br>
-                </div>
-                <div class="overwrite-form-item">
-                    <p>Enable <a href="https://sponsor.ajay.app/" target="_blank">SponsorBlock</a>: <span class="settings-current">
-                        {% if channel_info.channel_overwrites.integrate_sponsorblock %}
-                            {{ channel_info.channel_overwrites.integrate_sponsorblock }}
-                        {% elif channel_info.channel_overwrites.integrate_sponsorblock == False %}
-                            Disabled
-                        {% else %}
-                            False
-                        {% endif %}</span></p>
-                    {{ channel_overwrite_form.integrate_sponsorblock }}<br>
-                </div>
-                <button type="submit">Save Channel Overwrites</button>
-            </form>
-        </div>
-    </div>
-    {% if channel_info.channel_description %}
-        <div class="info-box-item description-box">
-            <p>Description: <button onclick="textReveal()" id="text-reveal-button">Show</button></p>
-            <div id="text-reveal" class="description-text">
-                {{ channel_info.channel_description|linebreaks }}
-            </div>
-        </div>
-    {% endif %}
 </div>
 <div class="boxed-content {% if view_style == "grid" %}boxed-{{ grid_items }}{% endif %}">
    <div class="view-controls">
--- a/tubearchivist/home/templates/home/channel_id_about.html
+++ b/tubearchivist/home/templates/home/channel_id_about.html
@@ -0,0 +1,112 @@
+{% extends "home/base.html" %}
+{% block content %}
+{% load static %}
+{% load humanize %}
+<div class="boxed-content">
+    <div class="channel-banner">
+        <a href="{% url 'channel_id' channel_info.channel_id %}"><img src="{{ channel_info.channel_banner_url }}" alt="channel_banner"></a>
+    </div>
+    <div class="info-box-item channel-nav">
+        <a href="{% url 'channel_id' channel_info.channel_id %}"><h3>Videos</h3></a>
+        <a href="{% url 'channel_id_playlist' channel_info.channel_id %}"><h3>Playlists</h3></a>
+        <a href="{% url 'channel_id_about' channel_info.channel_id %}"><h3>About</h3></a>
+    </div>
+    <div class="info-box info-box-3">
+        <div class="info-box-item">
+            <div class="round-img">
+                <a href="{% url 'channel_id' channel_info.channel_id %}">
+                    <img src="{{ channel_info.channel_thumb_url }}" alt="channel-thumb">
+                </a>
+            </div>
+            <div>
+                <h3><a href="{% url 'channel_id' channel_info.channel_id %}">{{ channel_info.channel_name }}</a></h3>
+                {% if channel_info.channel_subs >= 1000000 %}
+                    <p>Subscribers: {{ channel_info.channel_subs|intword }}</p>
+                {% else %}
+                    <p>Subscribers: {{ channel_info.channel_subs|intcomma }}</p>
+                {% endif %}
+            </div>
+        </div>
+        <div class="info-box-item">
+            <div>
+                <p>Last refreshed: {{ channel_info.channel_last_refresh }}</p>
+                {% if channel_info.channel_active %}
+                    <p>Youtube: <a href="https://www.youtube.com/channel/{{ channel_info.channel_id }}" target="_blank">Active</a></p>
+                {% else %}
+                    <p>Youtube: Deactivated</p>
+                {% endif %}
+            </div>
+        </div>
+        <div class="info-box-item">
+            <div>
+                {% if channel_info.channel_views >= 1000000 %}
+                    <p>Channel views: {{ channel_info.channel_views|intword }}</p>
+                {% elif channel_info.channel_views > 0 %}
+                    <p>Channel views: {{ channel_info.channel_views|intcomma }}</p>
+                {% endif %}
+                <button onclick="deleteConfirm()" id="delete-item">Delete Channel</button>
+                <div class="delete-confirm" id="delete-button">
+                    <span>Delete {{ channel_info.channel_name }} including all videos? </span><button class="danger-button" onclick="deleteChannel(this)" data-id="{{ channel_info.channel_id }}">Delete</button> <button onclick="cancelDelete()">Cancel</button>
+                </div>
+            </div>
+        </div>
+    </div>
+    {% if channel_info.channel_description %}
+        <div class="description-box">
+            <p id="text-expand" class="description-text">
+                {{ channel_info.channel_description|linebreaksbr|urlizetrunc:50 }}
+            </p>
+            <button onclick="textExpand()" id="text-expand-button">Show more</button>
+        </div>
+    {% endif %}
+    <div class="description-box">
+        <h2>Customize {{ channel_info.channel_name }}</h2>
+    </div>
+    <div id="overwrite-form" class="info-box">
+        <div class="info-box-item">
+            <form class="overwrite-form" action="/channel/{{ channel_info.channel_id }}/about/" method="POST">
+                {% csrf_token %}
+                <div class="overwrite-form-item">
+                    <p>Download format: <span class="settings-current">
+                        {% if channel_info.channel_overwrites.download_format %}
+                            {{ channel_info.channel_overwrites.download_format }}
+                        {% else %}
+                            False
+                        {% endif %}</span></p>
+                    {{ channel_overwrite_form.download_format }}<br>
+                </div>
+                <div class="overwrite-form-item">
+                    <p>Auto delete watched videos after x days: <span class="settings-current">
+                        {% if channel_info.channel_overwrites.autodelete_days %}
+                            {{ channel_info.channel_overwrites.autodelete_days }}
+                        {% else %}
+                            False
+                        {% endif %}</span></p>
+                    {{ channel_overwrite_form.autodelete_days }}<br>
+                </div>
+                <div class="overwrite-form-item">
+                    <p>Index playlists: <span class="settings-current">
+                        {% if channel_info.channel_overwrites.index_playlists %}
+                            {{ channel_info.channel_overwrites.index_playlists }}
+                        {% else %}
+                            False
+                        {% endif %}</span></p>
+                    {{ channel_overwrite_form.index_playlists }}<br>
+                </div>
+                <div class="overwrite-form-item">
+                    <p>Enable <a href="https://sponsor.ajay.app/" target="_blank">SponsorBlock</a>: <span class="settings-current">
+                        {% if channel_info.channel_overwrites.integrate_sponsorblock %}
+                            {{ channel_info.channel_overwrites.integrate_sponsorblock }}
+                        {% elif channel_info.channel_overwrites.integrate_sponsorblock == False %}
+                            Disabled
+                        {% else %}
+                            False
+                        {% endif %}</span></p>
+                    {{ channel_overwrite_form.integrate_sponsorblock }}<br>
+                </div>
+                <button type="submit">Save Channel Overwrites</button>
+            </form>
+        </div>
+    </div>
+</div>
+{% endblock content %}
--- a/tubearchivist/home/templates/home/channel_id_playlist.html
+++ b/tubearchivist/home/templates/home/channel_id_playlist.html
@@ -0,0 +1,57 @@
+{% extends "home/base.html" %}
+{% block content %}
+{% load static %}
+{% load humanize %}
+<div class="boxed-content">
+    <div class="channel-banner">
+        <a href="{% url 'channel_id' channel_info.channel_id %}"><img src="{{ channel_info.channel_banner_url }}" alt="channel_banner"></a>
+    </div>
+    <div class="info-box-item channel-nav">
+        <a href="{% url 'channel_id' channel_info.channel_id %}"><h3>Videos</h3></a>
+        <a href="{% url 'channel_id_playlist' channel_info.channel_id %}"><h3>Playlists</h3></a>
+        <a href="{% url 'channel_id_about' channel_info.channel_id %}"><h3>About</h3></a>
+    </div>
+    <div class="view-controls">
+        <div class="toggle">
+            <span>Show subscribed only:</span>
+            <div class="toggleBox">
+                <input id="show_subed_only" onclick="toggleCheckbox(this)" type="checkbox" {% if show_subed_only %}checked{% endif %}>
+                {% if not show_subed_only %}
+                    <label for="" class="ofbtn">Off</label>
+                {% else %}
+                    <label for="" class="onbtn">On</label>
+                {% endif %}
+            </div>
+        </div>
+        <div class="view-icons">
+            <img src="{% static 'img/icon-gridview.svg' %}" onclick="changeView(this)" data-origin="playlist" data-value="grid" alt="grid view">
+            <img src="{% static 'img/icon-listview.svg' %}" onclick="changeView(this)" data-origin="playlist" data-value="list" alt="list view">
+        </div>
+    </div>
+    <div class="playlist-list {{ view_style }}">
+        {% if results %}
+            {% for playlist in results %}
+            <div class="playlist-item {{ view_style }}">
+                <div class="playlist-thumbnail">
+                    <a href="{% url 'playlist_id' playlist.source.playlist_id %}">
+                        <img src="/cache/playlists/{{ playlist.source.playlist_id }}.jpg" alt="{{ playlist.source.playlist_id }}-thumbnail">
+                    </a>
+                </div>
+                <div class="playlist-desc {{ view_style }}">
+                    <a href="{% url 'channel_id' playlist.source.playlist_channel_id %}"><h3>{{ playlist.source.playlist_channel }}</h3></a>
+                    <a href="{% url 'playlist_id' playlist.source.playlist_id %}"><h2>{{ playlist.source.playlist_name }}</h2></a>
+                    <p>Last refreshed: {{ playlist.source.playlist_last_refresh }}</p>
+                    {% if playlist.source.playlist_subscribed %}
+                        <button class="unsubscribe" type="button" id="{{ playlist.source.playlist_id }}" onclick="unsubscribe(this.id)" title="Unsubscribe from {{ playlist.source.playlist_name }}">Unsubscribe</button>
+                    {% else %}
+                        <button type="button" id="{{ playlist.source.playlist_id }}" onclick="subscribe(this.id)" title="Subscribe to {{ playlist.source.playlist_name }}">Subscribe</button>
+                    {% endif %}
+                </div>
+            </div>
+            {% endfor %}
+        {% else %}
+            <h2>No playlists found...</h2>
+        {% endif %}
+    </div>
+</div>
+{% endblock content %}
--- a/tubearchivist/home/templates/home/playlist_id.html
+++ b/tubearchivist/home/templates/home/playlist_id.html
@@ -56,11 +56,11 @@
        </div>
    </div>
    {% if playlist_info.playlist_description %}
-        <div class="info-box-item description-box">
-            <p>Description: <button onclick="textReveal()" id="text-reveal-button">Show</button></p>
-            <div id="text-reveal" class="description-text">
-                {{ playlist_info.playlist_description|linebreaks }}
-            </div>
+        <div class="description-box">
+            <p id="text-expand" class="description-text">
+                {{ playlist_info.playlist_description|linebreaksbr|urlizetrunc:50 }}
+            </p>
+            <button onclick="textExpand()" id="text-expand-button">Show more</button>
        </div>
    {% endif %}
 </div>
--- a/tubearchivist/home/templates/home/search.html
+++ b/tubearchivist/home/templates/home/search.html
@@ -7,6 +7,7 @@
    </div>
    <div class="multi-search-box">
        {{ search_form }}
+        <p>Start typing or use <span class="settings-current">video:</span>, <span class="settings-current">channel:</span>, <span class="settings-current">playlist:</span> or <span class="settings-current">full:</span> keywords for advanced queries. <a href="https://github.com/tubearchivist/tubearchivist/wiki/Search" target="_blank">Learn more</a>.</p>
    </div>
    <div class="multi-search-result">
        <h2>Video Results</h2>
@@ -26,5 +27,11 @@
            <p>No playlists found.</p>
        </div>
    </div>
+    <div class="multi-search-result">
+        <h2>Fulltext Results</h2>
+        <div id="fulltext-results" class="video-list list">
+            <p>No fulltext results found.</p>
+        </div>
+    </div>
 </div>
 {% endblock content %}
--- a/tubearchivist/home/templates/home/settings.html
+++ b/tubearchivist/home/templates/home/settings.html
@@ -118,7 +118,8 @@
            <h2 id="format">Cookie</h2>
            <div class="settings-item">
                <p>Import YouTube cookie: <span class="settings-current">{{ config.downloads.cookie_import }}</span><br></p>
-                <i>Place your cookie file named <span class="settings-current">cookies.google.txt</span> in <span class="settings-current">cache/import</span> before enabling. Instructions in the <a href="https://github.com/tubearchivist/tubearchivist/wiki/Settings" target="_blank">Wiki.</a></i><br>
+                <p>For automatic cookie import use <b>Tube Archivist Companion</b> <a href="https://github.com/tubearchivist/browser-extension" target="_blank">browser extension</a>.</p> 
+                <i>For manual cookie import, place your cookie file named <span class="settings-current">cookies.google.txt</span> in <span class="settings-current">cache/import</span> before enabling. Instructions in the <a href="https://github.com/tubearchivist/tubearchivist/wiki/Settings" target="_blank">Wiki.</a></i><br>
                {{ app_form.downloads_cookie_import }}<br>
                {% if config.downloads.cookie_import %}
                    <div id="cookieMessage">
--- a/tubearchivist/home/templates/home/video.html
+++ b/tubearchivist/home/templates/home/video.html
@@ -77,11 +77,11 @@
        </div>
    </div>
    {% if video.description %}
-        <div class="info-box-item description-box">
-            <p>Description: <button onclick="textReveal()" id="text-reveal-button">Show</button></p>
-            <div id="text-reveal" class="description-text">
-                {{ video.description|linebreaks }}
-            </div>
+        <div class="description-box">
+            <p id="text-expand" class="description-text">
+                {{ video.description|linebreaksbr|urlizetrunc:50 }}
+            </p>
+            <button onclick="textExpand()" id="text-expand-button">Show more</button>
        </div>
    {% endif %}
    {% if playlist_nav %}
@@ -125,7 +125,11 @@
 <script>
    var videoData = getVideoData('{{ video.youtube_id }}');
    sponsorBlock = videoData.data.sponsorblock;
-    var videoProgress = getVideoProgress('{{ video.youtube_id }}').position;
+    {% if position %}
+        var videoProgress = {{ position }}
+    {% else %}
+        var videoProgress = getVideoProgress('{{ video.youtube_id }}').position;
+    {% endif %}
    window.onload = insertVideoTag(videoData, videoProgress);
 </script>
 {% endblock content %}
--- a/tubearchivist/home/urls.py
+++ b/tubearchivist/home/urls.py
@@ -6,6 +6,8 @@ from django.contrib.auth.views import LogoutView
 from django.urls import path
 from home.views import (
    AboutView,
+    ChannelIdAboutView,
+    ChannelIdPlaylistView,
    ChannelIdView,
    ChannelView,
    DownloadView,
@@ -42,6 +44,16 @@ urlpatterns = [
        login_required(ChannelIdView.as_view()),
        name="channel_id",
    ),
+    path(
+        "channel/<slug:channel_id>/about/",
+        login_required(ChannelIdAboutView.as_view()),
+        name="channel_id_about",
+    ),
+    path(
+        "channel/<slug:channel_id>/playlist/",
+        login_required(ChannelIdPlaylistView.as_view()),
+        name="channel_id_playlist",
+    ),
    path(
        "video/<slug:video_id>/",
        login_required(VideoView.as_view()),
--- a/tubearchivist/home/views.py
+++ b/tubearchivist/home/views.py
@@ -8,6 +8,7 @@ import json
 import urllib.parse
 from time import sleep

+from api.src.search_processor import SearchProcess
 from django.conf import settings
 from django.contrib.auth import login
 from django.contrib.auth.forms import AuthenticationForm
@@ -15,6 +16,7 @@ from django.http import JsonResponse
 from django.shortcuts import redirect, render
 from django.views import View
 from home.src.download.yt_dlp_base import CookieHandler
+from home.src.es.connect import ElasticWrap
 from home.src.es.index_setup import get_available_backups
 from home.src.frontend.api_calls import PostData
 from home.src.frontend.forms import (
@@ -33,7 +35,7 @@ from home.src.index.channel import channel_overwrites
 from home.src.index.generic import Pagination
 from home.src.index.playlist import YoutubePlaylist
 from home.src.ta.config import AppConfig, ScheduleBuilder
-from home.src.ta.helper import UrlListParser
+from home.src.ta.helper import UrlListParser, time_parser
 from home.src.ta.ta_redis import RedisArchivist
 from home.tasks import extrac_dl, index_channel_playlists, subscribe_to
 from rest_framework.authtoken.models import Token
@@ -394,14 +396,15 @@ class DownloadView(ArchivistResultsView):
                youtube_ids = UrlListParser(url_str).process_list()
            except ValueError:
                # failed to process
+                key = "message:add"
                print(f"failed to parse: {url_str}")
                mess_dict = {
-                    "status": "message:add",
+                    "status": key,
                    "level": "error",
                    "title": "Failed to extract links.",
                    "message": "Not a video, channel or playlist ID or URL",
                }
-                RedisArchivist().set_message("message:add", mess_dict)
+                RedisArchivist().set_message(key, mess_dict, expire=True)
                return redirect("downloads")

            print(youtube_ids)
@@ -439,7 +442,6 @@ class ChannelIdView(ArchivistResultsView):
            {
                "title": "Channel: " + channel_name,
                "channel_info": channel_info,
-                "channel_overwrite_form": ChannelOverwriteForm,
            }
        )

@@ -476,6 +478,93 @@ class ChannelIdView(ArchivistResultsView):
        return redirect("channel_id", channel_id, permanent=True)


+class ChannelIdAboutView(ArchivistResultsView):
+    """resolves to /channel/<channel-id>/about/
+    show metadata, handle per channel conf
+    """
+
+    view_origin = "channel"
+
+    def get(self, request, channel_id):
+        """handle get request"""
+        self.initiate_vars(request)
+
+        path = f"ta_channel/_doc/{channel_id}"
+        response, _ = ElasticWrap(path).get()
+
+        channel_info = SearchProcess(response).process()
+        channel_name = channel_info["channel_name"]
+
+        self.context.update(
+            {
+                "title": "Channel: About " + channel_name,
+                "channel_info": channel_info,
+                "channel_overwrite_form": ChannelOverwriteForm,
+            }
+        )
+
+        return render(request, "home/channel_id_about.html", self.context)
+
+    @staticmethod
+    def post(request, channel_id):
+        """handle post request"""
+        print(f"handle post from {channel_id}")
+        channel_overwrite_form = ChannelOverwriteForm(request.POST)
+        if channel_overwrite_form.is_valid():
+            overwrites = channel_overwrite_form.cleaned_data
+            print(f"{channel_id}: set overwrites {overwrites}")
+            channel_overwrites(channel_id, overwrites=overwrites)
+            if overwrites.get("index_playlists") == "1":
+                index_channel_playlists.delay(channel_id)
+
+        sleep(1)
+        return redirect("channel_id_about", channel_id, permanent=True)
+
+
+class ChannelIdPlaylistView(ArchivistResultsView):
+    """resolves to /channel/<channel-id>/playlist/
+    show all playlists of channel
+    """
+
+    view_origin = "playlist"
+    es_search = "ta_playlist/_search"
+
+    def get(self, request, channel_id):
+        """handle get request"""
+        self.initiate_vars(request)
+        self._update_view_data(channel_id)
+        self.find_results()
+
+        channel_info = self._get_channel_meta(channel_id)
+        channel_name = channel_info["channel_name"]
+        self.context.update(
+            {
+                "title": "Channel: Playlists " + channel_name,
+                "channel_info": channel_info,
+            }
+        )
+
+        return render(request, "home/channel_id_playlist.html", self.context)
+
+    def _update_view_data(self, channel_id):
+        """update view specific data dict"""
+        self.data["sort"] = [{"playlist_name.keyword": {"order": "asc"}}]
+        must_list = [{"match": {"playlist_channel_id": channel_id}}]
+
+        if self.context["show_subed_only"]:
+            must_list.append({"match": {"playlist_subscribed": True}})
+
+        self.data["query"] = {"bool": {"must": must_list}}
+
+    def _get_channel_meta(self, channel_id):
+        """get metadata for channel"""
+        path = f"ta_channel/_doc/{channel_id}"
+        response, _ = ElasticWrap(path).get()
+        channel_info = SearchProcess(response).process()
+
+        return channel_info
+
+
 class ChannelView(ArchivistResultsView):
    """resolves to /channel/
    handle functionality for channel overview page, subscribe to channel,
@@ -512,13 +601,14 @@ class ChannelView(ArchivistResultsView):
        """handle http post requests"""
        subscribe_form = SubscribeToChannelForm(data=request.POST)
        if subscribe_form.is_valid():
+            key = "message:subchannel"
            message = {
-                "status": "message:subchannel",
+                "status": key,
                "level": "info",
                "title": "Subscribing to Channels",
                "message": "Parsing form data",
            }
-            RedisArchivist().set_message("message:subchannel", message=message)
+            RedisArchivist().set_message(key, message=message, expire=True)
            url_str = request.POST.get("subscribe")
            print(url_str)
            subscribe_to.delay(url_str)
@@ -659,15 +749,14 @@ class PlaylistView(ArchivistResultsView):
        if subscribe_form.is_valid():
            url_str = request.POST.get("subscribe")
            print(url_str)
+            key = "message:subplaylist"
            message = {
-                "status": "message:subplaylist",
+                "status": key,
                "level": "info",
                "title": "Subscribing to Playlists",
                "message": "Parsing form data",
            }
-            RedisArchivist().set_message(
-                "message:subplaylist", message=message
-            )
+            RedisArchivist().set_message(key, message=message, expire=True)
            subscribe_to.delay(url_str)

        sleep(1)
@@ -682,6 +771,7 @@ class VideoView(View):
    def get(self, request, video_id):
        """get single video"""
        config_handler = AppConfig(request.user.id)
+        position = time_parser(request.GET.get("t"))
        path = f"ta_video/_doc/{video_id}"
        look_up = SearchHandler(path, config=False)
        video_hit = look_up.get_data()
@@ -707,6 +797,7 @@ class VideoView(View):
            "cast": config_handler.config["application"]["enable_cast"],
            "version": settings.TA_VERSION,
            "config": config_handler.config,
+            "position": position,
        }
        return render(request, "home/video.html", context)

@@ -829,18 +920,45 @@ class SettingsView(View):
        sleep(1)
        return redirect("settings", permanent=True)

-    @staticmethod
-    def post_process_updated(updated, config):
+    def post_process_updated(self, updated, config):
        """apply changes for config"""
        if not updated:
            return

        for config_value, updated_value in updated:
            if config_value == "cookie_import":
-                if updated_value:
-                    CookieHandler(config).import_cookie()
-                else:
-                    CookieHandler(config).revoke()
+                self.process_cookie(config, updated_value)
+
+    def process_cookie(self, config, updated_value):
+        """import and validate cookie"""
+        handler = CookieHandler(config)
+        if updated_value:
+            try:
+                handler.import_cookie()
+            except FileNotFoundError:
+                print("cookie: import failed, file not found")
+                handler.revoke()
+                self._fail_message("Cookie file not found.")
+                return
+
+            valid = handler.validate()
+            if not valid:
+                handler.revoke()
+                self._fail_message("Failed to validate cookie file.")
+        else:
+            handler.revoke()
+
+    @staticmethod
+    def _fail_message(message_line):
+        """notify our failure"""
+        key = "message:setting"
+        message = {
+            "status": key,
+            "level": "error",
+            "title": "Cookie import failed",
+            "message": message_line,
+        }
+        RedisArchivist().set_message(key, message=message, expire=True)


 def progress(request):
--- a/tubearchivist/requirements.txt
+++ b/tubearchivist/requirements.txt
@@ -1,12 +1,12 @@
 beautifulsoup4==4.11.1
 celery==5.2.7
-Django==4.0.4
-django-cors-headers==3.12.0
+Django==4.0.6
+django-cors-headers==3.13.0
 djangorestframework==3.13.1
-Pillow==9.1.1
-redis==4.3.1
-requests==2.27.1
+Pillow==9.2.0
+redis==4.3.4
+requests==2.28.1
 ryd-client==0.0.3
 uWSGI==2.0.20
-whitenoise==6.1.0
-yt_dlp==2022.5.18
+whitenoise==6.2.0
+yt_dlp==2022.7.18
--- a/tubearchivist/static/css/style.css
+++ b/tubearchivist/static/css/style.css
@@ -365,10 +365,17 @@ button:hover {
 }

 #text-reveal {
-    height: 0px;
+    height: 0;
    overflow: hidden;
 }

+#text-expand {
+    overflow: hidden;
+    display: -webkit-inline-box;
+    -webkit-box-orient: vertical;
+    -webkit-line-clamp: 4;
+}
+

 /* video player */
 .player-wrapper {
@@ -408,6 +415,12 @@ button:hover {
    margin: 10px 10px 10px 0;
 }

+/* fix for safari full screen not scaling full */
+video:-webkit-full-screen {
+    max-height: unset !important;
+    max-width: unset !important;
+}
+
 /* video list */
 .video-list {
    display: grid;
@@ -445,7 +458,7 @@ button:hover {

 .video-item.list {
    display: grid;
-    grid-template-columns: 25% auto;
+    grid-template-columns: 26% auto;
 }

 .video-progress-bar {
@@ -591,6 +604,8 @@ button:hover {

 .description-box {
    margin-top: 1rem;
+    padding: 15px;
+    background-color: var(--highlight-bg);
 }

 .info-box-3 {
@@ -772,6 +787,10 @@ button:hover {
 }

 /* multi search page */
+.multi-search-box {
+    padding-right: 20px;
+}
+
 .multi-search-box input {
    width: 100%;
 }
@@ -817,6 +836,18 @@ button:hover {
    transform: translateX(-30%);
 }

+.info-box-item.channel-nav {
+    justify-content: center;
+}
+
+.info-box-item.channel-nav a {
+    padding: 0 1rem;
+}
+
+.info-box-item.channel-nav a:hover {
+    text-decoration: underline;
+}
+
 /* playlist overview page */
 .playlist-list.list {
    display: grid;
@@ -1086,8 +1117,12 @@ button:hover {
 }

 /* tablet */
-@media screen and (max-width: 1000px) {
-    .boxed-content {
+@media screen and (max-width: 1000px), screen and (max-height: 850px) {
+    .boxed-content,
+    .boxed-content.boxed-4,
+    .boxed-content.boxed-5,
+    .boxed-content.boxed-6,
+    .boxed-content.boxed-7 {
        width: 90%;
    }
    .video-list.grid.grid-3,
--- a/tubearchivist/static/script.js
+++ b/tubearchivist/static/script.js
@@ -336,10 +336,18 @@ function cancelDelete() {
    document.getElementById("delete-item").style.display = 'block';
 }

+// get seconds from hh:mm:ss.ms timestamp
+function getSeconds(timestamp) {
+    var elements = timestamp.split(":", 3);
+    var secs = parseInt(elements[0]) * 60 * 60 + parseInt(elements[1]) * 60 + parseFloat(elements[2])
+    return secs
+}
+
 // player
 var sponsorBlock = [];
 function createPlayer(button) {
    var videoId = button.getAttribute('data-id');
+    var videoPosition = button.getAttribute('data-position');
    var videoData = getVideoData(videoId);

    var sponsorBlockElements = '';
@@ -363,8 +371,11 @@ function createPlayer(button) {
    } else {
        sponsorBlock = null;
    }
-
-    var videoProgress = getVideoProgress(videoId).position;
+    if (videoPosition) {
+        var videoProgress = getSeconds(videoPosition)
+    } else {
+        var videoProgress = getVideoProgress(videoId).position;
+    }
    var videoName = videoData.data.title;

    var videoTag = createVideoTag(videoData, videoProgress);
@@ -761,7 +772,9 @@ function removePlayer() {
        playerElement.innerHTML = '';
        // append played status
        var videoInfo = document.getElementById('video-info-' + youtubeId);
-        videoInfo.insertBefore(playedStatus, videoInfo.firstChild);
+        if (videoInfo) {
+            videoInfo.insertBefore(playedStatus, videoInfo.firstChild);
+        }
    }
 }

@@ -794,8 +807,8 @@ function searchMulti(query) {
            var http = new XMLHttpRequest();
            http.onreadystatechange = function() {
                if (http.readyState === 4) {
-                    allResults = JSON.parse(http.response).results;
-                    populateMultiSearchResults(allResults);
+                    response = JSON.parse(http.response);
+                    populateMultiSearchResults(response.results, response.queryType);
                }
            };
            http.open("POST", "/process/", true);
@@ -811,36 +824,83 @@ function getViewDefaults(view) {
    return defaultView;
 }

-function populateMultiSearchResults(allResults) {
+function populateMultiSearchResults(allResults, queryType) {
    // videos
    var defaultVideo = getViewDefaults("home");
    var allVideos = allResults.video_results;
    var videoBox = document.getElementById("video-results");
    videoBox.innerHTML = "";
-    for (let index = 0; index < allVideos.length; index++) {
-        const video = allVideos[index].source;
-        const videoDiv = createVideo(video, defaultVideo);
-        videoBox.appendChild(videoDiv);
+    videoBox.parentElement.style.display = "block";
+    if (allVideos.length > 0) {
+        for (let index = 0; index < allVideos.length; index++) {
+            const video = allVideos[index].source;
+            const videoDiv = createVideo(video, defaultVideo);
+            videoBox.appendChild(videoDiv);
+        }
+    } else {
+        if (queryType === "simple" || queryType == "video") {
+            videoBox.innerHTML = "<p>No videos found.</p>";
+        } else {
+            videoBox.parentElement.style.display = "none";
+        }
    }
    // channels
    var defaultChannel = getViewDefaults("channel");
    var allChannels = allResults.channel_results;
    var channelBox = document.getElementById("channel-results");
    channelBox.innerHTML = "";
-    for (let index = 0; index < allChannels.length; index++) {
-        const channel = allChannels[index].source;
-        const channelDiv = createChannel(channel, defaultChannel);
-        channelBox.appendChild(channelDiv);
+    channelBox.parentElement.style.display = "block";
+    if (allChannels.length > 0) {
+        for (let index = 0; index < allChannels.length; index++) {
+            const channel = allChannels[index].source;
+            const channelDiv = createChannel(channel, defaultChannel);
+            channelBox.appendChild(channelDiv);
+        }
+    } else {
+        if (queryType === "simple" || queryType == "channel") {
+            channelBox.innerHTML = "<p>No channels found.</p>";
+        } else {
+            channelBox.parentElement.style.display = "none";
+        }
    }
    // playlists
    var defaultPlaylist = getViewDefaults("playlist");
    var allPlaylists = allResults.playlist_results;
    var playlistBox = document.getElementById("playlist-results");
    playlistBox.innerHTML = "";
-    for (let index = 0; index < allPlaylists.length; index++) {
-        const playlist = allPlaylists[index].source;
-        const playlistDiv = createPlaylist(playlist, defaultPlaylist);
-        playlistBox.appendChild(playlistDiv);
+    playlistBox.parentElement.style.display = "block";
+    if (allPlaylists.length > 0) {
+        for (let index = 0; index < allPlaylists.length; index++) {
+            const playlist = allPlaylists[index].source;
+            const playlistDiv = createPlaylist(playlist, defaultPlaylist);
+            playlistBox.appendChild(playlistDiv);
+        }
+    } else {
+        if (queryType === "simple" || queryType == "playlist") {
+            playlistBox.innerHTML = "<p>No playlists found.</p>";
+        } else {
+            playlistBox.parentElement.style.display = "none";
+        }
+    }
+    // fulltext
+    var allFullText = allResults.fulltext_results;
+    var fullTextBox = document.getElementById("fulltext-results");
+    fullTextBox.innerHTML = "";
+    fullTextBox.parentElement.style.display = "block";
+    if (allFullText.length > 0) {
+        for (let i = 0; i < allFullText.length; i++) {
+            const fullText = allFullText[i];
+            if ("highlight" in fullText) {
+                const fullTextDiv = createFulltext(fullText);
+                fullTextBox.appendChild(fullTextDiv);
+            }
+        }
+    } else {
+        if (queryType === "simple" || queryType == "full") {
+            fullTextBox.innerHTML = "<p>No fulltext items found.</p>";
+        } else {
+            fullTextBox.parentElement.style.display = "none";
+        }
    }
 }

@@ -862,7 +922,7 @@ function createVideo(video, viewStyle) {
    const channelName = video.channel.channel_name;
    // build markup
    const markup = `
-    <a href="#player" data-src="/media/${mediaUrl}" data-thumb="${thumbUrl}" data-title="${videoTitle}" data-channel="${channelName}" data-channel-id="${channelId}" data-id="${videoId}" onclick="createPlayer(this)">
+    <a href="#player" data-id="${videoId}" onclick="createPlayer(this)">
        <div class="video-thumb-wrap ${viewStyle}">
            <div class="video-thumb">
                <img src="${thumbUrl}" alt="video-thumb">
@@ -965,8 +1025,43 @@ function createPlaylist(playlist, viewStyle) {
    return playlistDiv;
 }

+function createFulltext(fullText) {
+    const videoId = fullText.source.youtube_id;
+    const videoTitle = fullText.source.title;
+    const thumbUrl = fullText.source.vid_thumb_url;
+    const channelId = fullText.source.subtitle_channel_id;
+    const channelName = fullText.source.subtitle_channel;
+    const subtitleLine = fullText.highlight.subtitle_line[0];
+    const subtitle_start = fullText.source.subtitle_start.split(".")[0];
+    const subtitle_end = fullText.source.subtitle_end.split(".")[0];
+    const markup = `
+    <a href="#player" data-id="${videoId}" data-position="${subtitle_start}" onclick="createPlayer(this)">
+        <div class="video-thumb-wrap list">
+            <div class="video-thumb">
+                <img src="${thumbUrl}" alt="video-thumb">
+            </div>
+            <div class="video-play">
+                <img src="/static/img/icon-play.svg" alt="play-icon">
+            </div>
+        </div>
+    </a>
+    <div class="video-desc list">
+        <p>${subtitle_start} - ${subtitle_end}</p>
+        <p>${subtitleLine}</p>    
+        <div>
+            <a href="/channel/${channelId}/"><h3>${channelName}</h3></a>
+            <a class="video-more" href="/video/${videoId}/?t=${subtitle_start}"><h2>${videoTitle}</h2></a>
+        </div>
+    </div>
+    `
+    const fullTextDiv = document.createElement("div");
+    fullTextDiv.setAttribute("class", "video-item list");
+    fullTextDiv.innerHTML = markup;
+    return fullTextDiv
+}

 // generic
+
 function sendPost(payload) {
    var http = new XMLHttpRequest();
    http.open("POST", "/process/", true);
@@ -991,19 +1086,57 @@ function getCookie(c_name) {


 // animations
+
 function textReveal() {
-    var textBox = document.getElementById('text-reveal');
-    var button = document.getElementById('text-reveal-button');
+    var textBox = document.getElementById("text-reveal");
+    var button = document.getElementById("text-reveal-button");
    var textBoxHeight = textBox.style.height;
-    if (textBoxHeight === 'unset') {
-        textBox.style.height = '0px';
-        button.innerText = 'Show';
+    if (textBoxHeight === "unset") {
+        textBox.style.height = "0px";
+        button.innerText = "Show";
    } else {
-        textBox.style.height = 'unset';
-        button.innerText = 'Hide';
+        textBox.style.height = "unset";
+        button.innerText = "Hide";
    }
 }

+function textExpand() {
+    var textBox = document.getElementById("text-expand");
+    var button = document.getElementById("text-expand-button");
+    var textBoxLineClamp = textBox.style["-webkit-line-clamp"];
+    if (textBoxLineClamp === "none") {
+        textBox.style["-webkit-line-clamp"] = "4";
+        button.innerText = "Show more";
+    } else {
+        textBox.style["-webkit-line-clamp"] = "none";
+        button.innerText = "Show less";
+    }
+}
+
+// hide "show more" button if all text is already visible
+function textExpandButtonVisibilityUpdate() {
+    var textBox = document.getElementById("text-expand");
+    var button = document.getElementById("text-expand-button");
+    if (!textBox || !button)
+        return;
+
+    var textBoxLineClamp = textBox.style["-webkit-line-clamp"];
+    if (textBoxLineClamp === "none")
+        return; // text box is in revealed state
+
+    if (textBox.offsetHeight < textBox.scrollHeight
+        || textBox.offsetWidth < textBox.scrollWidth) {
+        // the element has an overflow, show read more button
+        button.style.display = "inline-block";
+    } else {
+        // the element doesn't have overflow
+        button.style.display = "none";
+    }
+}
+
+document.addEventListener("readystatechange", textExpandButtonVisibilityUpdate);
+window.addEventListener("resize", textExpandButtonVisibilityUpdate);
+
 function showForm() {
    var formElement = document.getElementById('hidden-form');
    var displayStyle = formElement.style.display;
Author	SHA1	Message	Date
simon	83866a0d79	clarify scope	2022-07-23 17:32:59 +07:00
simon	aa0739bfbe	bump TA_VERSION to v0.2.0	2022-07-23 16:35:18 +07:00
simon	d797f47bb8	remove synology link, remove security notification	2022-07-23 15:03:41 +07:00
simon	09f3ce0109	add TA_HOST and ES8 documentation	2022-07-23 14:38:50 +07:00
simon	2ba31a3937	add search help text	2022-07-23 14:10:16 +07:00
simon	31e2fcb322	update backup section for v0.2	2022-07-23 13:41:24 +07:00
simon	e69cd1129c	add result size info	2022-07-23 13:41:07 +07:00
simon	8a59b66a0f	finetune full text thumb width	2022-07-23 13:28:49 +07:00
simon	dcf7658972	add kibana quick start guide	2022-07-22 18:26:12 +07:00
simon	e79d64d707	update channel wiki page for v0.2	2022-07-22 18:15:27 +07:00
simon	503a4ed822	force es8 at startup	2022-07-22 17:22:06 +07:00
simon	f0a44678ed	unset video max-height max-width for fullscreen webkit, #264	2022-07-22 14:50:53 +07:00
simon	4b1ebf44c9	handle filenotfounderror for manual cookie import	2022-07-21 23:01:32 +07:00
simon	eb229440d4	load config into redis on fresh install at startup	2022-07-21 23:01:01 +07:00
simon	88b412a2e9	keyword search, #build Changed: - added keyword search queries - added fulltext search - authenticate all user created media - reduce docker log output	2022-07-21 20:15:15 +07:00
simon	14d3a7586d	validate es8 at startup	2022-07-21 20:09:06 +07:00
simon	6ec65f9fe8	add search wiki page	2022-07-21 20:00:12 +07:00
simon	0ec6558282	jump to timestamp from full search	2022-07-21 17:15:36 +07:00
simon	52d31f9cca	simplify createVideo player link building	2022-07-21 15:51:33 +07:00
simon	07016f2cc5	ignore full: result without highlight	2022-07-21 15:47:29 +07:00
simon	55a73995c8	implement source and lang keywords for full text search	2022-07-21 15:39:27 +07:00
simon	a40ebc7bbc	tweak with for video-item.list	2022-07-21 15:36:36 +07:00
simon	e4dd47aed0	implement fulltext search keyword	2022-07-20 21:49:32 +07:00
simon	eb5a7338ab	search with and operator, increase result size	2022-07-20 18:25:39 +07:00
simon	b77d50d4ed	use queryType in populateMultiSearchResults to hide irrelevant	2022-07-20 18:01:57 +07:00
simon	b520a9bfb0	boost title result scores in multi_match	2022-07-20 16:20:23 +07:00
simon	242ca19586	return query_type from SearchParser	2022-07-20 16:10:57 +07:00
simon	6a6665bdf9	implement simple, aka none keyword query	2022-07-20 15:04:10 +07:00
simon	8e8c00af40	implement playlist keyword search, fix channel subscribed keyword	2022-07-20 14:58:04 +07:00
simon	ae3cf7eb80	add channel keyword search	2022-07-20 14:48:41 +07:00
simon	0a2b6ee90a	add keyword search framework, implement video search	2022-07-19 19:23:59 +07:00
simon	d065b5b90a	reduce uwsgi logging to errors only	2022-07-18 17:17:20 +07:00
simon	0528220b5e	authenticate media and thumb requests	2022-07-18 14:10:34 +07:00
simon	f0850522a7	Merge branch 'master' into testing	2022-07-18 11:49:42 +07:00
simon	cc2fff8bef	use info_dict for message, simplify cache template, #271	2022-07-18 11:37:43 +07:00
simon	2d13011866	bump yt-dlp	2022-07-18 11:29:28 +07:00
simon	146f17bf74	shutil.move fix take 2, #build Changed: - use shutil.copyfile for copy_function - fixed typo	2022-07-17 07:48:54 +07:00
simon	b406250ae2	use shutil.copyfile for shutil.move calls	2022-07-17 07:47:27 +07:00
simon	ed9237b69d	fix typo	2022-07-17 07:46:52 +07:00
simon	393e00866b	Desc preview, nginx fix, #build Changed: - Requires new env var: TA_HOST, hostname of your application - Showing preview for description text, by @p0358 - Fix nginx user permission, by @p0358 - Refactor IndexPaginate to take loop callback - Fix backup for large index using loop callback - Fix deactivating missing channels on reindex	2022-07-16 22:11:29 +07:00
p0358	411b09629a	Better description view for videos and playlists (#272 ) * draft * proper implementation of description line clamping * use a JS event that's potentially faster for text expand button visibility update * remove desc title text, move show button after desc Co-authored-by: simon <simobilleter@gmail.com>	2022-07-16 22:06:26 +07:00
p0358	79996f6838	Fix permission/mount issues from #268 (#270 ) * make nginx run as root within the container (#268) * move file to videos path with fallback to regular copy with no metadata copying (#268) * use `copy_function=shutil.copy` everywhere with `shutil.move`	2022-07-16 21:54:39 +07:00
simon	6bb7f80ea2	Refactor IndexPaginate - add callback function - implement callback for ElasticBackup	2022-07-16 17:55:18 +07:00
simon	3b17c01c6d	add TA_HOST env for ALLOWED_HOSTS	2022-07-14 17:26:10 +07:00
simon	91bccfd057	deactivate non existing channels on reindex	2022-07-12 11:41:03 +07:00
simon	8cca09e0ac	remove ta_json backup file, keep only es_ndjson	2022-07-07 12:34:57 +07:00
simon	1479516689	Reimagining the channel pages, #build Changed: - Split the channel-id page into subpages - Channel videos, channel playlist, channel about - Disable es startup check to test ES8	2022-07-04 21:23:50 +07:00
simon	831e59a626	bump django	2022-07-04 21:23:32 +07:00
simon	6e012383ab	toggle playlist subscribed filter	2022-07-04 21:12:54 +07:00
simon	e988192343	add channel_id_about subheading	2022-07-04 20:44:12 +07:00
simon	ef3abd01b1	implement channel_id_playlist, handle channel_id_about post	2022-07-04 20:25:52 +07:00
simon	e664338845	implement basic channel_id_about view	2022-07-04 16:44:37 +07:00
simon	63a1237683	temporarily disable es version check for testing	2022-07-04 16:43:13 +07:00
simon	6d59da4c86	remove redundant latest tags	2022-07-03 18:20:30 +07:00
simon	5bc67104ea	set restart policy to unless-stopped	2022-07-03 17:23:02 +07:00
simon	72ec78e011	browser extension as prefered cookie sync method	2022-07-03 16:46:45 +07:00
simon	de8b7d6b43	bump ES version	2022-07-03 16:09:09 +07:00
simon	048409bb1e	bump TA_VERSION to v0.1.7	2022-07-03 16:08:30 +07:00
simon	efff4cd96d	change cookie documentation for v0.1.7	2022-07-03 16:06:28 +07:00
simon	78d7947a5c	a note about subtitle refresh, #260	2022-07-03 15:36:21 +07:00
simon	c132a37f03	bump dependencies	2022-07-03 15:30:46 +07:00
simon	4678a7507b	bump dependencies	2022-06-30 18:44:33 +07:00
simon	e8e240c89b	Startup fix, cookie improvements, #build Changed - lock creating superuser at startup by @dshoreman - auto validate cookie - API: return cookie validation state for browser extension - Finetuning CSS mobile layout	2022-06-27 10:14:59 +07:00
simon	1857ac44ca	fix tablet css breakpoints and grid spacing	2022-06-27 09:48:36 +07:00
simon	8c07932071	delete cookie:valid key when revoking	2022-06-23 19:21:13 +07:00
simon	74d74d95a1	store validation result, return in API	2022-06-23 06:48:55 +07:00
simon	b4add20d10	bump yt-dlp	2022-06-22 19:30:38 +07:00
Dave Shoreman	e9619c1242	Fix repeated superuser creations after rename/removal (#263 ) * Avoid creating superuser if variables are removed Enables the removal of `TA_USERNAME` and/or `TA_PASSWORD` as a way of preventing an additional superuser being created when another exists. In the old code, `ELASTIC_USER` would never error due to the default, so this replaces the loop with one requirement on `ELASTIC_PASSWORD`. * Use .superuser_created to prevent re-creation Creates an empty `.superuser_created` file after successful creation, and uses this to skip the command on future runs. If the file doesn't exist, then `TA_USERNAME` and `TA_PASSWORD` will also be required. * The superuser password can't be empty ...well—it can—but the password input of the login form requires a value so even though the account is created, you can't actually login. * Parse createsuperuser output for 'exists' error If a user already exists with the same name as the superuser being created, the command fails with "Error: That name is already taken". It's probably safe to assume that user is the one we expect. This parses the output, treating it as successful so it can be skipped next time. If it hits the catch-all, something probably failed so we exit. * Use the cache for persistence Updates the .superuser_created file to be placed in /cache so that it gets persisted in a volume between config tweaks and container rebuilds. Also makes `TA_PASSWORD` required always, because it seems to be needed by tubearchivist/config/settings.py to generate a secret key. * Let it work outside Docker If /cache doesn't exist (i.e. when it's not running in Docker), this falls back to a `cache` dir in the current working directory instead. Also renames the file to 'initsu.lock' rather than having it hidden. * Use flexible fallback path for `celery` command	2022-06-22 18:29:46 +07:00
simon	49d33a2765	Improved Cookie handeling, #build Changed: - Cookie: Use IO Stream from Redis directly for cookies - Cookie: Don't load invalid cookie - API: Import cookie with PUT request - Redis: Don't expire Redis messages by default - Redis: Connections expire automatically	2022-06-21 08:14:01 +07:00
simon	ecc58f6c11	API: note about release lagging behind	2022-06-21 08:10:00 +07:00
simon	e5f43e5fe9	API: implement cookie PUT request to import cookie	2022-06-21 08:06:38 +07:00
simon	499c47c7a1	move set_cookie to separate method for reusabliity	2022-06-21 08:01:33 +07:00
simon	2cf30e1127	refactor: default set_message in RedisArchivist to True	2022-06-16 10:37:46 +07:00
simon	ebfc4a349f	center title and tiles	2022-06-15 18:30:06 +07:00
simon	150c1dc273	add tiles to readme	2022-06-15 18:18:52 +07:00
simon	30abbe9be7	rewrite cookie into redis from io stream, auto validate	2022-06-15 17:54:05 +07:00
simon	40e4ef0e05	auto expire redis connections after 3600secs	2022-06-15 16:36:41 +07:00
simon	2ceb1b701a	consolidate config of cookie revoke	2022-06-15 15:39:24 +07:00
simon	2ad093a9a8	handle cookiejar.LoadError dont import invalid cookie	2022-06-15 10:39:48 +07:00
simon	09a94d0df5	set relative values directly with path arg	2022-06-15 10:27:29 +07:00
simon	f7f4ca7728	API update, #build Changed: - API: Added Task get view - Added is_locked method for redis lock check - Updaded base and libs	2022-06-15 09:24:01 +07:00
simon	83a90000c6	bump base image	2022-06-15 09:23:41 +07:00
lamusmaser	bbb16bb3c2	Update TaskAPIView with GET - #257 PR (#258 ) * Initial commit for GET response for TaskAPIView. * Update for missing space - linting issue. * Additional linting fixes. * add is_locked method to check if lock is set * Update to use `.is_locked` method and add decorator. * Fix linting issue. * Fix doubled "rescan" call for locked file. Removed call to ".owned()" under "is_locked". * Commenting out GET call in TaskAPI view to revert change for testing. * Commenting is_locked function to see if Server Response 500 is stopped. * Reassert is_locked function within RedisArchivist class. * Create test GET responder. * Reverting simple GET response change. * Reapplying simple GET response for `/api/task/`. * Reapplying change with modification to lock key. * Documentation update for new GET calls. * README: fix returned value as `bool` instead of `str`. * Updating the `is_locked` key reference to "rescan" and "downloading" to fix which key is being retrieved. * Use dictionary keys, rather than line-by-line statements. * Fix typo in README for Task View. Co-authored-by: simon <simobilleter@gmail.com>	2022-06-15 09:15:59 +07:00
simon	4d9ee4494f	bump requests	2022-06-14 14:26:49 +07:00
simon	eeeabca8c2	bump dependencies	2022-06-07 17:09:49 +07:00
simon	bb0f3dad1a	update roadmap	2022-06-04 11:19:47 +07:00
simon	9c0c263fce	new docker build for build server task	2022-06-04 11:16:46 +07:00