ytdl-server

ytdl-server

ytdl-server is a program that allows you to download videos onto a remote server using youtube-dl.

Using youtube-dl to download videos onto a network file share is inefficient since the video file has to pass through your workstation. Additionally, many of the post-processing options that youtube-dl provides will create a temporary copy of the file, which means that the amount of data uploaded over the network can be 3-4x the size of the actual file.

ytdl-server solves this by allowing you to download videos directly to the remote server via a REST API.

ytdl-server also supports high-availability, which means that you can perform maintenance without downtime.

[TOC]

How it works

ytdl-server has two main components: the Flask REST API and the Celery worker.

The REST API communicates with the user in order to create jobs. The jobs are then run by the worker in order to download the desired videos.

The complete structure of ytdl-server is shown in the below diagram:

Included in ytdl-server means that the component is part of ytdl-server.

Included in the Docker images means that the component is built into the Docker images. You'll need to install it separately if you're not using them.

Not included means that the component is not part of ytdl-server and needs to be installed separately.

• Reverse proxy / Load balancer (not included)

  Used to forward user-requests to the WSGI server(s).

  Some available options are NGINX and Caddy.

  Load balancing is only needed if you're running multiple WSGI server instances.

• WSGI server (included in the Docker images)

  Used to run the Flask REST API(s).

  Some available options are Gunicorn and gevent.

  If you're using the Docker images, Gunicorn is included.

  If you're running multiple REST API instances, each one needs its own WSGI server instance.

• Flask REST API (included in ytdl-server)

  Used to create and manage jobs.

  You can run multiple instances if you want high-availability. Otherwise, a single instance is fine.

• Celery broker (not included)

  Used to dispatch jobs created by the REST API to a Celery worker.

  You can use any broker supported by Celery, including RabbitMQ and Redis.

• Celery worker (included in ytdl-server)

  Used to run jobs (download videos).

  You can run multiple instances if you want high-availability. Otherwise, a single instance is fine.

• PostgreSQL database (not included)

  Used to store information about jobs, and also functions as the Celery result backend.

Installation

Docker (recommended)

Prebuilt Docker images are provided. See Registry for a list of supported tags.

An example Docker Compose file is provided here. To use it, copy the Compose file to a directory, and run the following command to start it:

docker-compose up

The REST API will be accessible at port 8000. See Usage for how to use it.

If you want to use a custom config file, overwrite the existing config file at /config/ytdl-server.yml. See Configuration for available options.

Manual

ytdl-server requires Python 3.9+.

Install from PyPI:

pip3 install 'ytdl-server[dbapi]'

Install from source:

git clone 'https://gitlab.com/adralioh/ytdl-server.git'
pip3 install './ytdl-server[dbapi]'

The [dbapi] extra installs Psycopg2, which is used to connect to the PostgreSQL database. You must install a DBAPI for ytdl-server to work. You can alternatively install psycopg2-binary if you don't want to compile it from source. See Installation - Psycopg 2 for details.

If you're using Redis or Amazon SQS as the broker, you must also install the following extra dependencies:

# Redis
pip3 install 'celery[redis]'
# Amazon SQS
pip3 install 'celery[sqs]'

Run the REST API using a WSGI server such as Gunicorn:

gunicorn -w 4 'ytdl_server.flask:with_logging()'

Run the Celery worker using Celery:

celery -A 'ytdl_server.run_celery' worker

You also need to set up the reverse proxy, Celery broker, and PostgreSQL database. See How it works for information, and see Configuration for how to configure ytdl-server to use the database and broker.

Updating

When updating ytdl-server, refer to the versioning scheme and the changelog for information about what has changed.

Configuration

See Configuration.

Usage

Videos are downloaded via jobs. You create a new job whenever you want to download one or more videos.

The following examples access the ytdl-server directly using curl. See Frontends for more user-friendly ways of accessing it.

Create a job:

curl ytdl.example.com/jobs/ \
     -H 'Content-Type: application/json' \
     -d '{
           "urls": [ "https://www.youtube.com/watch?v=AAAAAAAAAAA",
                     "https://www.youtube.com/watch?v=BBBBBBBBBBB" ],
           "ytdl_opts": { "format": "best",
                          "outtmpl": "%(id)s.%(ext)s" }
         }'

Jobs are created asynchronously, which means that the videos will be downloaded in the background.

You can check the status of a job as follows:

curl ytdl.example.com/jobs/ffedda2d-7fa4-4839-a705-88c893e3f942

ffedda2d-7fa4-4839-a705-88c893e3f942 is the job ID. This is obtained from the output of the previous command where you created the job.

You can also cancel a running job:

curl ytdl.example.com/jobs/ffedda2d-7fa4-4839-a705-88c893e3f942 \
     -X PATCH \
     -H 'Content-Type: application/json' \
     -d '{
           "status": "cancelled"
         }'

For complete documentation about the REST API, see REST API.

Registry

Prebuit Docker images are provided at registry.gitlab.com/adralioh/ytdl-server for the REST API and the Celery worker.

The supported tags are rebuilt monthly in order to provide the latest security updates.

REST API

Images for the REST API are available at registry.gitlab.com/adralioh/ytdl-server/api.

The images include Gunicorn, which is used as the WSGI server.

Supported tags:

• latest, 1, 1.1, 1.1.8

Celery worker

Images for the Celery worker are available at registry.gitlab.com/adralioh/ytdl-server/worker.

The yt_dlp tags use yt-dlp instead of youtube-dl. You also need to set the YTDL_MODULE env-var to yt_dlp on the REST API when using these.

The ffmpeg tags include FFmpeg, which is required for many of youtube-dl's post-processing options.

Supported tags:

• latest, 1, 1.1, 1.1.8
• yt_dlp, 1-yt_dlp, 1.1-yt_dlp, 1.1.8-yt_dlp
• ffmpeg, 1-ffmpeg, 1.1-ffmpeg, 1.1.8-ffmpeg
• yt_dlp-ffmpeg, 1-yt_dlp-ffmpeg, 1.1-yt_dlp-ffmpeg, 1.1.8-yt_dlp-ffmpeg

Testing and development

See tests

Frontends

• ytcl - Command-line interface with syntax similar to youtube-dl

Frequently asked questions

See Frequently asked questions.