A dynamic and extensible music library organizer
Demlo is a music library organizer. It can encode, fix case, change folder
hierarchy according to tags or file properties, tag from an online database,
copy covers while ignoring duplicates or those below a quality threshold, and
much more. It makes it possible to manage your libraries uniformly and
dynamically. You can write your own rules to fit your needs best.
Demlo aims at being as lightweight and portable as possible. Its major
runtime dependency is the transcoder FFmpeg. The scripts are written in Lua for
portability and speed while allowing virtually unlimited extensibility.
Usage:
For usage options, see:
First Demlo creates a list of all input files. When a folder is specified, all
files matching the extensions from the 'extensions' variable will be appended to
the list. Identical files are appended only once.
Next all files get analyzed:
- The audio file details (tags, stream properties, format properties, etc.) are
stored into the 'input' variable. The 'output' variable gets its default values
from 'input', or from an index file if specified from command-line. If no index
has been specified and if an attached cuesheet is found, all cuesheet details
are appended accordingly. Cuesheet tags override stream tags, which override
format tags. Finally, still without index, tags can be retrieved from Internet
if the command-line option is set.
- If a prescript has been specified, it gets executed. It makes it possible to
adjust the input values and global variables before running the other scripts.
- The scripts, if any, get executed in the lexicographic order of their
basename. The 'output' variable is transformed accordingly. Scripts may contain
rules such as defining a new file name, new tags, new encoding properties, etc.
You can use conditions on input values to set the output properties, which makes
it virtually possible to process a full music library in one single run.
- If a postscript has been specified, it gets executed. It makes it possible to
adjust the output of the script for the current run only.
- Demlo makes some last-minute tweaking if need be: it adjusts the bitrate, the
path, the encoding parameters, and so on.
- A preview of changes is displayed.
- When applying changes, the covers get copied if required and the audio file
gets processed: tags are modified as specified, the file is re-encoded if
required, and the output is written to the appropriate folder. When destination
already exists, the 'exist' action is executed.
The program's default behaviour can be changed from the user configuration file.
(See the 'Files' section for a template.) Most command-line flags default value
can be changed. The configuration file is loaded on startup, before parsing the
command-line options. Review the default value of the CLI flags with 'demlo -h'.
If you wish to use no configuration file, set the environment variable DEMLORC
to ".".
Scripts can contain any safe Lua code. Some functions like 'os.execute' are not
available for security reasons. It is not possible to print to the standard
output/error unless running in debug mode and using the 'debug' function.
See the 'sandbox.go' file for a list of allowed functions and variables.
Lua patterns are replaced by Go regexps. See
https://github.com/google/re2/wiki/Syntax.
Scripts have no requirements at all. However, to be useful, they should set
values of the 'output' table detailed in the 'Variables' section. You can use
the full power of the Lua to set the variables dynamically. For instance:
'input' and 'output' are both accessible from any script.
All default functions and variables (excluding 'output') are reset on every
script call to enforce consistency. Local variables are lost from one script
call to another. Global variables are preserved. Use this feature to pass data
like options or new functions.
'output' structure consistency is guaranteed at the start of every script. Demlo
will only extract the fields with the right type as described in the 'Variables'
section.
Warning: Do not abuse of global variables, especially when processing non-fixed
size data (e.g. tables). Data could grow big and slow down the program.
By default, when the destination exists, Demlo will append a suffix to the
output destination. This behaviour can be changed from the 'exist' action
specified by the user. Demlo comes with a few default actions.
The 'exist' action works just like scripts with the following differences:
- Any change to 'output.path' will be skipped.
- An additional variable is accessible from the action: 'existinfo' holds the file
details of the existing files in the same fashion as 'input'. This allows for
comparing the input file and the existing destination.
The writing rules can be tweaked the following way:
Word of caution: overwriting breaks Demlo's rule of not altering existing files.
It can lead to undesired results if the overwritten file is also part of the
(yet to be processed) input. The overwrite capability can be useful when syncing
music libraries however.
The user scripts should be generic. Therefore they may not properly handle some
uncommon input values. Tweak the input with temporary overrides from
command-line.
The prescript and postscript defined on command-line will let you run arbitrary
code that is run before and after all other scripts, respectively.
Use global variables to transfer data and parameters along.
If the prescript and postscript end up being too long, consider writing a demlo
script. You can also define shell aliases or use wrapper scripts as convenience.
The 'input' table describes the file:
Bitrate is in bits per seconds (bps). That is, for 320 kbps you would specify
The 'time' is the modification time of the file. It holds the sec seconds and
nsec nanoseconds since January 1, 1970 UTC.
The entry 'streams' and 'format' are as returned by
It gives access to most metadata that FFmpeg can return. For instance, to get
the duration of the track in seconds, query the variable
'input.format.duration'.
Since there may be more than one stream (covers, other data), the first audio
stream is assumed to be the music stream. For convenience, the index of the
music stream is stored in 'audioindex'.
The tags returned by FFmpeg are found in streams, format and in the cuesheet.
To make tag queries easier, all tags are stored in the 'tags' table, with the
following precedence:
You can remove a tag by setting it to 'nil' or the empty string. This is
equivalent, except that 'nil' saves some memory during the process.
The 'output' table describes the transformation to apply to the file:
The 'parameters' array holds the CLI parameters passed to FFmpeg. It can be
anything supported by FFmpeg, although this variable is supposed to hold
encoding information. See the 'Examples' section.
The 'embeddedcovers', 'externalcovers' and 'onlinecover' variables are detailed
in the 'Covers' section.
The 'write' variable is covered in the 'Existing destination' section.
The 'rmsrc' variable is a boolean: when true, Demlo removes the source file
after processing. This can speed up the process when not re-encoding. This
option is ignored for multi-track files.
For convenience, the following shortcuts are provided:
Demlo provides some non-standard Lua functions to ease scripting.
Display a message on stderr if debug mode is on.
Return lowercase string without non-alphanumeric characters nor leading zeros.
Return the relation coefficient of the two input strings. The result is a float
in 0.0...1.0, 0.0 means no relation at all, 1.0 means identical strings.
A format is a container in FFmpeg's terminology.
'output.parameters' contains CLI flags passed to FFmpeg. They are meant to set
the stream codec, the bitrate, etc.
If 'output.parameters' is {'-c:a', 'copy'} and the format is identical, then
taglib will be used instead of FFmpeg. Use this rule from a (post)script to
disable encoding by setting the same format and the copy parameters. This speeds
up the process.
The official scripts are usually very smart at guessing the right values. They
might make mistakes however. If you are unsure, you can (and you are advised to)
preview the results before proceeding. The 'diff' preview is printed to stderr.
A JSON preview of the changes is printed to stdout if stdout is redirected.
The initial values of the 'output' table can be completed with tags fetched from
the MusicBrainz database. Audio files are fingerprinted for the queries, so even
with initially wrong file names and tags, the right values should still be
retrieved. The front album cover can also be retrieved.
Proxy parameters will be fetched automatically from the 'http_proxy'
and 'https_proxy' environment variables.
As this process requires network access it can be quite slow. Nevertheless,
Demlo is specifically optimized for albums, so that network queries are
used for only one track per album, when possible.
Some tracks can be released on different albums: Demlo tries to guess it from
the tags, but if the tags are wrong there is no way to know which one it is.
There is a case where the selection can be controlled: let's assume we have
tracks A, B and C from the same album Z. A and B were also released in album Y,
whereas C was release in Z only.
Tags for A will be checked online; let's assume it gets tagged to album Y. B
will use A details, so album Y too. Then C does not match neither A's nor B's
album, so another online query will be made and it will be tagged to album Z.
This is slow and does not yield the expected result.
Now let's call
Tags for C will be queried online, and C will be tagged to Z. Then both A and B
will match album Z so they will be tagged using C details, which is the desired
result.
Conclusion: when using online tagging, the first argument should be the lesser
known track of the album.
Demlo can set the output variables according to the values set in a text file
before calling the script. The input values are ignored as well as online
tagging, but it is still possible to access the input table from scripts. This
'index' file is formatted in JSON. It corresponds to what Demlo outputs when
printing the JSON preview. This is valid JSON except for the missing beginning
and the missing end. It makes it possible to concatenate and to append to
existing index files. Demlo will automatically complete the missing parts so
that it becomes valid JSON.
The index file is useful when you want to edit tags manually: You can redirect
the output to a file, edit the content manually with your favorite text editor,
then run Demlo again with the index as argument. See the 'Examples' section.
This feature can also be used to interface Demlo with other programs.
Demlo can manage embedded covers as well as external covers.
External covers are queried from files matching known extensions in the file's
folder.
Embedded covers are queried from static video streams in the file.
Covers are accessed from
The embedded covers are indexed numerically by order of appearance in the
streams. The first cover will be at index 1 and so on. This is not necessarily
the index of the stream.
'inputcover' is the following structure:
'format' is the picture format. FFmpeg makes a distinction between format and
codec, but it is not useful for covers. The name of the format is specified by
Demlo, not by FFmpeg. Hence the 'jpeg' name, instead of 'mjpeg' as FFmpeg puts
it.
'width' and 'height' hold the size in pixels.
'checksum' can be used to identify files uniquely. For performance reasons, only
a partial checksum is performed. This variable is typically used for skipping
duplicates.
Cover transformations are specified in
'outputcover' has the following structure:
The format is specified by FFmpeg this time. See the comments on 'format' for
'inputcover'.
'parameters' is used in the same fashion as 'output.parameters'.
User configuration:
This must be a Lua file. See the 'demlorc' file provided with this package for
an exhaustive list of options.
Folder containing the official scripts:
User script folder:
Create this folder and add your own scripts inside. This folder takes precedence
over the system folder, so scripts with the same name will be found in the user
folder first.
The following examples will not proceed unless the '-p' command-line option is true.
Important: you _must_ use single quotes for the runtime Lua command to prevent
expansion. Inside the Lua code, use double quotes for strings and escape single
quotes.
Show default options:
Preview changes made by the default scripts:
Use 'alternate' script if found in user or system script folder (user folder first):
Add the Lua file to the list of scripts. This feature is convenient if you want
to write scripts that are too complex to fit on the command-line, but not
generic enough to fit the user or system script folders.
Remove all script from the list, then add '30-case' and '60-path' scripts. Note
that '30-case' will be run before '60-path'.
Do not use any script but '60-path'. The file content is unchanged and the file
is renamed to a dynamically computed destination. Demlo performs an instant
rename if destination is on the same device. Otherwise it copies the file and
removes the source.
Use the default scripts (if set in configuration file), but do not re-encode:
Set 'artist' to the value of 'composer', and 'title' to be preceded by the new
value of 'artist', then apply the default script. Do not re-encode. Order in
runtime script matters. Mind the double quotes.
Set track number to first number in input file name:
Use the default scripts but keep original value for the 'artist' tag:
1) Preview default scripts transformation and save it to an index. 2) Edit file
to fix any potential mistake. 3) Run Demlo over the same files using the index
information only.
Same as above but generate output filename according to the custom '61-rename'
script. The numeric prefix is important: it ensures that '61-rename' will be run
after all the default tag related scripts and after '60-path'. Otherwise, if a
change in tags would occur later on, it would not affect the renaming script.
Retrieve tags from Internet:
Same as above but for a whole album, and saving the result to an index:
Only download the cover for the album corresponding to the track. Use 'rmsrc' to
avoid duplicating the audio file.
Change tags inplace with entries from MusicBrainz:
Set tags to titlecase while casing AC-DC correctly:
To easily switch between formats from command-line, create one script per format
(see 50-encoding.lua), e.g. ogg.lua and flac.lua. Then
Add support for non-default formats from CLI:
Overwrite existing destination if input is newer:
ffmpeg(1), ffprobe(1),
http://www.lua.org/pil/contents.html