@knennigtri/merge-markdown

merge-markdown

Overview

Takes in a list of markdown files and merges them into a single output.md file with optional Word, HTML, or PDF output. Other advantages:

• Auto-create a manifest.yml (based on the directory), sample theme folder, and package.json
• Run in docker to reduce dependencies and discrepancies
• auto-resolution of all relative links in files for assets, other markdown files no matter their location locally
• built in link checker of final file
• Use the Manifest file (json/yaml):
  • Specify input files, folders, or patterns (relative or absolute)
  • Specify the output file (relative or absolute)
    • Add pandoc arguments for Word/HTML output
    • Add wkhtmltopdf arguments for PDF output
    • Create a TOC with doctoc
  • QA feature for optional file exclusions for reviewing
  • Per input file or globally:
    • Find/replace with regex (ex: names, titles, chapter #s, timestamps, etc)
    • Create TOC with doctoc
    • Remove yaml from top of md file

Contents

• Installation
• Basic Use
• Usage
• Manifest file format
  • Supported Options
  • Supported Output Options
  • Special Modes
• Manifest Examples
  • YAML used as input
  • JSON used as input
  • Replace keys within a single file
  • Options applied to all files
  • Apply output options
• Using Docker
  • Full CLI
  • Configurable Build

Installation

To install the command line tool globally, run:

  npm install -g @knennigtri/merge-markdown
  #See full list of commands
  merge-markdown -h

Basic Use

Create a new merge-markdown project with existing markdown files:

  # Create an initial manifest with markdown files in a directory
  > merge-markdown -c my/path/src

  # Create a full npm project with a theme folder
  > merge-markdown -c my/path/src --fullProject

Run merge-markdown

  # Merge based on existing manifest file, outputs markdown file
  > merge-markdown -m manifest.yml

  # Output to Word
  > merge-markdown -m myManifest.yml --word

  # Output to html
  > merge-markdown -m myManifest.yml --html
  
  # Output to PDF
  > merge-markdown -m myManifest.yml --pdf
  
  # Requires docker to be installed
  > merge-markdown -m myManifest.yml --pdf --docker

Using Docker is preferred. Using --docker simplifies external installation requirements like pandoc and wkhtmltopdf.

Usage

Usage: merge-markdown [ARGS]
Arguments:
  -m, --manifest <manifestFile>            Path to input folder, yaml, or json manifest
  -v, --version                            Displays version of this package
  -c, --create <path>                      auto-creates ./manifest.yml with input files from <path>
  --fullProject                            Add to -c to include a basic theme, frontmatter, and npm scripts
  --docker                                 Run merge-markdown commands in docker
  --getDockerFiles                         Downloads the Docker files to your local project
  --qa                                     QA mode.
  --skipLinkcheck                          Skips linkchecking
  --maintainAssetPaths                     Retains original asset paths
  --lang <language>                        Language code for multilingual support (e.g., --lang=es)
  --clean-assets [directory]               Clean unused assets in specified directory (or current directory)
  --assets-dir <name>                      Name of assets folder (default: 'assets')
  --fix                                    Actually delete unused assets (default: preview mode)
  --no-asset-preview                       Disable asset preview during normal merge workflow
  --pdf                                    Output to PDF. Must have Pandoc and wkhtmltopdf installed!
  --html --word                            Output to HTML or Word. Must have Pandoc installed!
  -h, --help                               Displays this screen
  -h manifest | options |
    outputOptions | qa | docker | assets   See examples
  -d, --debug                              See debug Options
Default is manifest[.yml|.yaml|.json] unless specified in -m.

Download Docker: https://docs.docker.com/get-docker/

Manifest file format

manifest[.yml|.yaml|.json]: This file can be in YAML or JSON format.

• input:
  • myFile1.md: {options} File input with local options
  • myFolder/: {options} Folder input (all immediate .md files)
  • "pattern:regex": {options} Pattern input (regex matching, recursive)
• output:
  • name: path/name.md: of the resultant file
  • {outputOptions} See Supported Output Options
• {options}: global options

See Supported Options Relative or absolute are accepted

Input Types:

• File: Standard file path (e.g., docs/readme.md)
• Folder: Path ending with / includes all immediate .md files (e.g., chapters/)
• Pattern: Starts with pattern: followed by regex to match files recursively (e.g., "pattern:.*chapter.*\\.md")

Supported Options

Options can be applied to an individual input or at a globally to apply to all inputs

noYAML

Optionlly removes YAML from top of input file. Default=false

---
  noYAML: true|false
---

replace

Optionally find/replace in an input file.

• string: Specify a string to find and replace
• regex: Specify a regex to find and replace

---
  replace:
      <!--{timestamp}-->: 09/01/2022
      ({#(.*?)}): ""                  
---

doctoc

Optionally add a table of contents to files using doctoc. This will allow for a local navigation within a module/chapter of your merged document.

# Use Default doctoc values:
---
 doctoc: true
---
# Add a unique title:
---
 doctoc: "Module Contents"
---
# Fully configure doctoc by overriding the default values from below:
---
  doctoc:
   mode: github
   maxlevel: 3
   title: ""
   notitle: true
   entryprefix: ""
   all: false
   stdout: true
   update-only: false                
---

Based on doctoc, Set where you would like for the TOC to exist in the markdown file

  <!-- START auto-update -->
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
# Contents

  - [Special Modes](#special-modes)
- [Manifest Examples](#manifest-examples)
  - [YAML used as input](#yaml-used-as-input)
  - [JSON used as input](#json-used-as-input)
  - [Replace keys within a single file](#replace-keys-within-a-single-file)
  - [Options applied to all files](#options-applied-to-all-files)
  - [Apply output options](#apply-output-options)
  - [Use folder inputs](#use-folder-inputs)
  - [Use pattern matching](#use-pattern-matching)
  - [Mix files, folders, and patterns](#mix-files-folders-and-patterns)
- [Using Docker](#using-docker)
  - [Full CLI](#full-cli)
  - [Configurable Build](#configurable-build)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->
  <!-- END auto-update -->

Supported Output Options

You can output to HTML or PDF. Pandoc is used to generate HTML and wkhtmltopdf is used to generate a PDF.

Merged file TOC

Similar to adding a TOC to the input files, you can add a TOC for the entire merged document. See doctoc options to configure.

 output:
   doctoc:
     key: value

HTML and Word Output

You can optionally add pandoc parameters to the manifest. The key doesn't matter, only the value is evalutated based on pandoc args.

If you aren't using Docker, pandoc must be installed!

  # HTML parameters
  output:
    pandoc:
      latexTemplate: --template path/to/my/latex/template.latex
      css: -c path/to/my/css/main.css

  # Word parameters
  output:
    pandoc:
      referenceDoc: '--reference-doc theme/reference.docx'

Generate HTML/Word only:

  #HTML
  > merge-markdown -m manifest.yml --html
  # Word
  > merge-markdown -m manifest.yml --word

The following arguments cannot be changes for pandoc:

• -o < fileName > - can only be modified using manifest.output.name

PDF Output

You can optionally add wkhtmltopdf options to the manifest.

If you aren't using Docker, pandoc and wkhtmltopdf must be installed!

  output:
    # html parameters are still needed since conversion is markdown > html > PDF
    pandoc:
      latexTemplate: --template path/to/my/latex/template.latex
      css: -c path/to/my/css/main.css
    wkhtmltopdf:
      marginBottom: 1in
      marginTop: 1in
      pageSize: Letter

The following options cannot be changes for wkhtmltopdf:

• enableLocalFileAccess - always true
• disableSmartShrinking - always true
• output - can only be modified using manifest.output.name

Generate a PDF:

 > merge-markdown -m manifest.yml --pdf

Example files can be found in test/pdf/src. You can also checkout a working project for css development using webpack.

Special Modes

QA Mode

  > merge-markdown -m manifest.yml --qa

Output will omit all filenames with frontmatter by default Add a regex to the manifest.json to customize exclusion:

---
  qa: {exclude: "(frontmatter|preamble)"}
---

nolinkcheck Mode

Sometimes the markdown-link-check tool might produce an error. To skip linkcheck:

> merge-markdown -m mymanifest.yml --nolinkcheck

Debug Mode

Debug is used in this tool:

Mac or Linux:

 > DEBUG:options merge-markdown -m file

Windows:

> set DEBUG=options & merge-markdown -m file

Options: {
  "*": "Output all debugging messages",
  "args": "See CLI argument messages",
  "cli": "Validate CLI logic",
  "manifest": "",
  "manifest:deprecation": "",
  "manifest:json": "",
  "merge": "messages for merge process",
  "rellinks": "relative links",
  "o:yaml": "yaml removal",
  "o:doctoc": "doctoc messages",
  "o:replace": "regex replace messages",
  "linkcheck": "linkcheck validation",
  "linkcheck:deep": "deep linkcheck validation",
  "presentation": "",
  "html": "pandoc messages for html",
  "html:options": "pandoc options messages",
  "pdf": "wkhtmltopdf messages for pdf",
  "pdf:options": "wkhtmltopdf options messages",
  "docker": ""
}

Manifest Examples

YAML used as input

---
input:
  frontmatter.md: ""
  file1.md: {noYAML: true, doctoc: "#### Section Contents"}
  file2.md: {noYAML: true, doctoc: "#### Section Contents"}
output: 
  name: myOutput.md
---

JSON used as input

{
  "input": {
    "frontmatter.md": {"replace": {"timestamp":true}},
    "file1.md": {"noYAML":true,"doctoc":"#### Section Contents"},
    "file2.md": {"noYAML":true,"doctoc":"#### Section Contents"}
  },
  "output": {
    "name": "myOutput.md"
  }
}

Replace keys within a single file

{
  "input": {
    "folder1/folder1/file1.md": {"replace": {
      "<!--{timestamp}-->": "06/01/2021",
      "<!--{endOfSection}-->": "> To learn more on this subject, visit: www.example.com",
      "({#(.*?)})": ""
      }},
    "folder2/folder2/file2.md": {"noYAML":true}
  },
  "output": {
    "name": "path/to/myOutput.md"
  }
}

Options applied to all files

---
input:
  frontmatter.md: ""
  folder1/file1.md: ""
  file2.md: ""
output: 
  name: myOutput.md
replace:
  ${timestamp}: 06/01/2021
  ({#(.*?)}): ""
doctoc: "#### Chapter contents"
noYAML: true
---

Apply output options

---
input:
 frontmatter.md: ''
 m1/m1-example.md: {noYAML: true, doctoc: true, replace: {<!--#-->: "Module 1:"}}
 m2/m2-example.md: {noYAML: true, doctoc: true, replace: {<!--#-->: "Module 2:"}}
output: 
  name: merged/myOutput.md
  doctoc:
   mode: bitbucket
   title: "Course Contents"
   maxlevel: 2
  pandoc:
    css: -c path/to/main.css
    title: -M title:Example
  wkhtmltopdf:
    pageSize: Letter
    footerLine: true
    footerCenter: Page [page]
---

Use folder inputs

---
input:
  frontmatter.md: ""
  chapters/: {noYAML: true, doctoc: true}
  appendix/: {noYAML: true}
output:
  name: merged/book.md
---

Use pattern matching

---
input:
  "pattern:lessons/lesson-[0-9]+\\.md": {doctoc: true}
  "pattern:.*exercise.*\\.md": {noYAML: true}
output:
  name: merged/course.md
---

Mix files, folders, and patterns

---
input:
  docs/intro.md: ""
  tutorials/: {doctoc: true}
  "pattern:m[0-9]+/.*\\.md": {noYAML: true}
output:
  name: merged/documentation.md
---

Using Docker

To use docker, make sure you have docker downloaded and started. Using docker sidesteps the requirements of installing pandoc and wkhtmltopdf locally and makes this tool more agnostic.

• Full CLI
• Configurable Build

Full CLI

• Run the docker application
• Run your merge-markdown command with the --docker parameter

 merge-markdown -m path/to/manifest.yml --pdf --docker

Exclude certain files from copying into Docker

For speed purposes, you can optionally exclude certain files and folders from copying into docker. By default, these folders are not copied:

const EXCLUDE_PATHS = [
  /.*\/node_modules\/.*/,
  /.*\/merged\/.*/,
  /.*\/target\/.*/,
];

To add more paths, update the manifest file:

docker:
  excludePaths: ["my/excluded/path", "temp.txt"]

Configurable Build

Download the Dockerfile and docker-compose.yml files:

 merge-markdown --getDockerFiles

Dockerfile and docker-compose.yml files need to be in the same directory as your project and set up Docker Compose with the following command:

 docker compose up -d --build

The docker image will copy all local structure of files and directories of the project into the current image's working directory. Once there, the command merge-markdown needs to be executed on the node service of docker compose to generate the desired output, e.g:

docker compose exec node merge-markdown -m manifest.yml --pdf

The command above assumes the manifest.yml file is in the root directory. An example of the project file structure could be:

project
└── assets
    ├── image1.svg
    └── ...
├── docker-compose.yml
├── Dockerfile
├── manifest.yml
├── README.md
├── README_2.md
├── README_3.md
└── README_4.md

Getting the outputs from the container's image could be done with the following command:

 docker compose cp node:/home/runner/workspace/output.pdf .