diff --git a/README.md b/README.md index 182d00fe..e2a751ce 100644 --- a/README.md +++ b/README.md @@ -66,10 +66,10 @@ There's dedicated user-contributed install steps under [docs/Installation.md](./ For minimal system requirements, the Tube Archivist stack needs around 2GB of available memory for a small testing setup and around 4GB of available memory for a mid to large sized installation. Minimal with dual core with 4 threads, better quad core plus. -Note for arm64 hosts: The Tube Archivist container is multi arch, so is Elasticsearch. RedisJSON doesn't offer arm builds, but you can use the image `bbilly1/rejson`, an unofficial rebuild for arm64. - This project requires docker. Ensure it is installed and running on your system. +Note for **arm64**: Tube Archivist is a multi arch container, same for redis. For Elasitc Search use the official image for arm64 support. Other architectures are not supported. + Save the [docker-compose.yml](./docker-compose.yml) file from this reposity somewhere permanent on your system, keeping it named `docker-compose.yml`. You'll need to refer to it whenever starting this application. Edit the following values from that file: @@ -153,6 +153,8 @@ Wildcards "*" can not be used for the Access-Control-Allow-Origin header. If the Use `bbilly1/tubearchivist-es` to automatically get the recommended version, or use the official image with the version tag in the docker-compose file. +Use official Elastic Search for **arm64**. + Stores video meta data and makes everything searchable. Also keeps track of the download queue. - Needs to be accessible over the default port `9200` - Needs a volume at **/usr/share/elasticsearch/data** to store data diff --git a/docker-compose.yml b/docker-compose.yml index 0004d15d..72b65b90 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -24,7 +24,7 @@ services: - archivist-es - archivist-redis archivist-redis: - image: redislabs/rejson # for arm64 use bbilly1/rejson + image: redis/redis-stack-server container_name: archivist-redis restart: unless-stopped expose: diff --git a/docs/Settings.md b/docs/Settings.md index 2ec38c4d..3b9007ba 100644 --- a/docs/Settings.md +++ b/docs/Settings.md @@ -121,8 +121,8 @@ The field **Refresh older than x days** takes a number where TubeArchivist will ## Thumbnail check This will check if all expected thumbnails are there and will delete any artwork without matching video. -## Index backup -Create a zip file of the metadata and select **Max auto backups to keep** to automatically delete old backups created from this task. +## ZIP file index backup +Create a zip file of the metadata and select **Max auto backups to keep** to automatically delete old backups created from this task. For data consistency, make sure there aren't any other tasks running that will change the index during the backup process. This is very slow, particularly for large archives. Use snapshots instead. # Actions @@ -166,8 +166,8 @@ If the video you are trying to import is not available on YouTube any more, **Tu ## Embed thumbnails into media file This will write or overwrite all thumbnails in the media file using the downloaded thumbnail. This is only necessary if you didn't download the files with the option *Embed Thumbnail* enabled or want to make sure all media files get the newest thumbnail. Follow the docker-compose logs to monitor progress. -## Backup Database -This will backup your metadata into a zip file. The file will get stored at *cache/backup* and will contain the necessary files to restore the Elasticsearch index formatted **nd-json** files. +## ZIP file index backup +This will backup your metadata into a zip file. The file will get stored at *cache/backup* and will contain the necessary files to restore the Elasticsearch index formatted **nd-json** files. For data consistency, make sure there aren't any other tasks running that will change the index during the backup process. This is very slow, particularly for large archives. BE AWARE: This will **not** backup any media files, just the metadata from the Elasticsearch. diff --git a/tubearchivist/config/management/commands/ta_startup.py b/tubearchivist/config/management/commands/ta_startup.py index b4c09409..852447a6 100644 --- a/tubearchivist/config/management/commands/ta_startup.py +++ b/tubearchivist/config/management/commands/ta_startup.py @@ -168,4 +168,4 @@ class Command(BaseCommand): message = f" 🗙 {index_name} vid_type update failed" self.stdout.write(self.style.ERROR(message)) self.stdout.write(response) - CommandError(message) + raise CommandError(message) diff --git a/tubearchivist/config/settings.py b/tubearchivist/config/settings.py index 8c222cab..87e283e2 100644 --- a/tubearchivist/config/settings.py +++ b/tubearchivist/config/settings.py @@ -32,7 +32,9 @@ SECRET_KEY = PW_HASH.hexdigest() # SECURITY WARNING: don't run with debug turned on in production! DEBUG = bool(environ.get("DJANGO_DEBUG")) -ALLOWED_HOSTS = [i.strip() for i in environ.get("TA_HOST").split()] +ALLOWED_HOSTS = [] +if environ.get("TA_HOST"): + ALLOWED_HOSTS = [i.strip() for i in environ.get("TA_HOST").split()] CSRF_TRUSTED_ORIGINS = [] for host in ALLOWED_HOSTS: diff --git a/tubearchivist/home/src/ta/config.py b/tubearchivist/home/src/ta/config.py index 9b3d7c51..f191eae9 100644 --- a/tubearchivist/home/src/ta/config.py +++ b/tubearchivist/home/src/ta/config.py @@ -266,6 +266,7 @@ class ScheduleBuilder: def build_schedule(self): """build schedule dict as expected by app.conf.beat_schedule""" AppConfig().load_new_defaults() + self.config = AppConfig().config schedule_dict = {} for schedule_item in self.SCHEDULES: diff --git a/tubearchivist/home/templates/home/settings.html b/tubearchivist/home/templates/home/settings.html index ec3d4c97..af0eb665 100644 --- a/tubearchivist/home/templates/home/settings.html +++ b/tubearchivist/home/templates/home/settings.html @@ -286,8 +286,9 @@
-

Index backup

+

ZIP file index backup

+

Zip file backups are very slow for large archives and consistency is not guaranteed, use snapshots instead. Make sure no other tasks are running when creating a Zip file backup.

Current index backup schedule: {% if config.scheduler.run_backup %} {% for key, value in config.scheduler.run_backup.items %} @@ -332,8 +333,9 @@

-

Backup database

+

ZIP file index backup

Export your database to a zip file stored at cache/backup.

+

Zip file backups are very slow for large archives and consistency is not guaranteed, use snapshots instead. Make sure no other tasks are running when creating a Zip file backup.