Incremental, cacheable builds for large Javascript monorepos. Uses [Bazel](https://bazel.build)
Incremental, cacheable builds for large Javascript monorepos. Uses Bazel
Getting started
Reference
Misc
Jazelle is designed for large organizations where different teams own different projects within a monorepo, and where projects depend on compiled assets from other projects in the monorepo. In terms of developer experience, it's meant to be a low-impact drop-in replacement for common day-to-day web stack commands such as yarn add, yarn build and yarn test.
Jazelle leverages Bazel for incremental/cacheable builds and should be able to integrate with Bazel rules from non-JS stacks. This is helpful if the rest of your organization is also adopting Bazel, especially if others in your organization are already investing into advanced Bazel features such as distributed caching. Jazelle can also be suitable if you develop libraries and want to test for regressions in downstream projects as part of your regular development workflow.
Due to its integration w/ Bazel, Jazelle can be a suitable solution if long CI times are a problem caused by running too many tests.
Jazelle can also be a suitable solution if the frequency of commits affecting a global lockfile impacts developer velocity.
If you just have a library of decoupled components, Jazelle might be overkill. In those cases, you could probably get away with using a simpler solution, such as Yarn workspaces, Lerna or Rush.
mkdir my-monorepo
cd my-monorepo
jazelle init
The jazelle init command generates Bazel WORKSPACE, BUILD.bazel and .bazelversion files, along with the Jazelle configuration file manifest.json. If you are setting up Jazelle on an existing Bazel workspace, see Bazel rules.
Check that the .bazelversion file at the root of your repo contains your desired Bazel version. For example:
5.1.0
Check that the WORKSPACE file at the root of your repo is using the desired versions of Jazelle, Node and Yarn:
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "jazelle",
url = "https://registry.yarnpkg.com/jazelle/-/jazelle-[version].tgz",
sha256 = "SHA 256 goes here",
strip_prefix = "package",
)
load("@jazelle//:workspace-rules.bzl", "jazelle_dependencies")
jazelle_dependencies(
node_version = "16.15.0",
node_sha256 = {
"darwin-x64": "a6bb12bbf979d32137598e49d56d61bcddf8a8596c3442b44a9b3ace58dd4de8",
"linux-x64": "ebdf4dc9d992d19631f0931cca2fc33c6d0d382543639bc6560d31d5060a8372",
"win-x64": "dbe04e92b264468f2e4911bc901ed5bfbec35e0b27b24f0d29eff4c25e428604",
"darwin-arm64": "ad8d8fc5330ef47788f509c2af398c8060bb59acbe914070d0df684cd2d8d39b",
"linux-arm64": "b4080b86562c5397f32da7a0723b95b1df523cab4c757688a184e3f733a7df56",
},
yarn_version = "1.19.1",
yarn_sha256 = "fdbc534294caef9cc0d7384fb579ec758da7fc033392ce54e0e8268e4db24baf",
)
Jazelle SHA256 checksum can be computed through the following command:
curl -fLs https://registry.yarnpkg.com/jazelle/-/jazelle-[version].tgz | openssl sha256
Node SHA256 checksums can be found at https://nodejs.org/dist/v[version]/SHASUMS256.txt. Use the checksums for these files:
node-v[version]-darwin-x64.tar.gznode-v[version]-linux-x64.tar.xznode-v[version]-win-x64.zipnode-v[version]-darwin-arm64.tar.gznode-v[version]-linux-arm64.tar.xzYarn SHA256 checksum can be computed through the following command:
curl -fLs https://github.com/yarnpkg/yarn/releases/download/v[version]/yarn-[version].js | openssl sha256
Double check that the BUILD.bazel at the root of your repo contains this code:
load("@jazelle//:build-rules.bzl", "jazelle")
jazelle(name = "jazelle", manifest = "manifest.json")
There should be a package.json at the root of the monorepo, with a workspaces field:
{
"workspaces": [
"path/to/project-1",
"path/to/project-2"
]
}
The workspaces field in this file should list every project that you want Jazelle to manage.
Add the following entries to .gitignore
third_party/jazelle/temp
bazel-*
DO commit
package.json filemanifest.json fileWORKSPACE fileBUILD.bazel files.bazelversion file.bazelignore filethird_party/jazelle/BUILD.bazel filethird_party/jazelle/scripts folderyarn.lock fileDO NOT commit
/third_party/jazelle/temp foldernode_modules foldersbazel-[*] foldersInstall the CLI globally:
# install
yarn global add jazelle
# verify it's installed
jazelle version
If the repo is already scaffolded, you can use the script in it:
third_party/jazelle/scripts/install-run-jazelle.sh version
yarn global upgrade jazelle
If upgrading fails, it's probably because you didn't follow the installation instructions. In that case, try reinstalling:
npm uninstall jazelle --global
yarn global remove jazelle
yarn global add jazelle
It's ok for users to have different versions of Jazelle installed. Jazelle runs all commands via Bazelisk, which enforces that the Bazel version specified in .bazelversion is used. Bazel, in turn, enforces that the Node and Yarn versions specified in WORKSPACE are used.
package.json file and add the path to your project in workspaces.scripts fields called build, test, lint and flow.jazelle check. To upgrade a dependency, run jazelle upgrade [the-dependency] to get the latest or jazelle upgrade [the-dependency]@[version] from your project folder.jazelle install from your project folder to generate Bazel BUILD files, and install dependencies. This may take a few minutes the first time around since Bazel needs to install itself and its dependencies. Subsequent calls to jazelle install will be faster.jazelle test to verify that your project builds and tests pass. Optionally run jazelle run lint and jazelle run flow.If building your project fails, open the BUILD.bazel files and double check that the dist argument in the web_library call points to the folder where you expect compiled assets to be placed. This folder is often called dist or bin. Note that BUILD.bazel files may also be created in dependencies' folders, if they did not already have them. Use version control to identify where newly generated BUILD.bazel files were created and review their dist arguments.
If you get permission errors (EPERM), it's likely because the Bazel sandbox disables write permissions on input files and there are compiled assets in your source code tree that are being picked up by the glob call in the BUILD.bazel file.
Delete the compiled assets that are generated by your NPM build script. You could also use the exclude argument of the glob in your BUILD.bazel file to help team members avoid the pitfall.
web_library(
name = "library",
deps = [
"//third_party/jazelle:node_modules",
],
srcs = glob(["**/*"], exclude = ["dist/**"]),
)
This error happens if running an app and Node is unable to find the dependency when requireing it. It can also happen if static analysis tooling depends on build output of dependencies and you use a command that bypasses Bazel.
jazelle purge && jazelle install from your project folder.build script does not run other tools (e.g. lint).bazel --help (e.g. jazelle lint), try running the Bazel-enabled equivalent (jazelle run lint) instead.If you get an error saying a script must exist, make sure your project has the relevant NPM script. For example, if you ran jazelle build, make sure your package.json has a scripts.build field. If it doesn't need to have one, simply create one with an empty value. If you do have that field, one of your project's local dependencies may be missing it.
Navigate to a project in the monorepo, then use CLI commands, similar to how you would with yarn
# navigate to your project folder
cd path/to/project-1
# generates Bazel build files for relevant projects, if needed
jazelle install
# start project in dev mode
jazelle run
# run tests
jazelle test
# lint
jazelle run lint
# type check
jazelle run flow
# add dependency
jazelle add react@16.8.2
Jazelle provides six build rules: jazelle, web_library, web_binary, web_executable, web_test and flow_test.
jazelle allows you to run Jazelle as a Bazel target (e.g. if you have Bazel installed globally, but not Jazelle)web_library defines what source files and dependencies comprise a project. The jazelle install command automatically generates a web_library() declaration with the correct list of dependencies (by looking into the project's package.json)web_binary builds a project and runs a projectweb_executable runs a project (without building)web_test runs a test script for a projectflow_test type checks the projectIf you add or remove an entry in your package.json that points to a local project, Jazelle updates the yarn.lock file and adds the dependency to the deps field of the web_library declation in your project's BUILD.bazel file. In Bazel, dependencies are declared using label syntax. A label consists of a // followed by the path to the project, followed by a : followed by the name field of the web_library declaration of the project.
For example, if you have a project in folder path/to/my-project whose web_library has name = "hello", then its label is //path/to/my-project:hello.
# an example BUILD.bazel file for a project with a dependency
web_library(
name = "my-project",
deps = [
# depend on a project that lives under ./my-other-project
"//my-other-project:my-other-project",
],
srcs = glob(["**/*"]),
dist = "dist",
)
While Jazelle attempts to always keep the workspace in a good state, it may be possible to get into a corrupt state, for example, if you manually edit system files (such as generated files in the /third_party/jazelle/temp folder).
Another way to get into a bad state is to change the name of a project. Currently, Jazelle does not support re-syncing depenency graphs after project name changes, since this use case is rare and the required checks would slow down CLI commands.
If you get into a bad state, here are some things you can try:
jazelle purge and run jazelle install from your project folder again/third_party/jazelle/temp folder and run jazelle install[your-project]/BUILD.bazel and run jazelle installmanifest.json is valid JSONShorthandsjazelle --helpjazelle versionjazelle initjazelle scaffoldjazelle installjazelle cijazelle focusjazelle addjazelle removejazelle upgradejazelle purgejazelle checkjazelle outdatedjazelle resolutionsjazelle alignjazelle localizejazelle chunkjazelle changesjazelle planjazelle batchjazelle buildjazelle devjazelle testjazelle lintjazelle flowjazelle startjazelle scriptjazelle bazeljazelle nodejazelle yarnjazelle execjazelle eachjazelle bumpjazelle doctorjazelle setup--name argument can omit the word --name. For example, jazelle add foo is equivalent to jazelle add --name foo.--cwd argument can be run without it from the project folder. For example, jazelle add foo --cwd my-project is equivalent to cd my-project && jazelle add foo.jazelle --helpDisplays help information
jazelle versionDisplays installed version. You may see two values: actual is the version of Jazelle being run. system is the version of Jazelle that is globally installed in the machine. Note that they may be different because the actual version is set in a repository's WORKSPACE file.
jazelle initScaffolds required workspace files
jazelle scaffoldprescaffold and postscaffold hooksjazelle scaffold --from from --to to --name name
--from - Folder to copy from. Can be an absolute path or relative to process.cwd()--to - Folder to copy to. Can be an absolute path or relative to process.cwd()--name - The name field in package.jsonjazelle installjazelle install --cwd [cwd]
--cwd - Project folder (absolute or relative to shell cwd). Defaults to process.cwd()--skipPreinstall - Skip the preinstall hook--skipPostinstall - Skip the postinstall hookverbose - Prints more Yarn warningsjazelle ciDownloads external dependencies and links local dependencies for all monorepo projects. Does not create or modify source files. Useful for CI checks.
jazelle ci --cwd [cwd]
--cwd - Project folder (absolute or relative to shell cwd). Defaults to process.cwd()jazelle focusIt's typically faster to run jazelle focus than to run jazelle install, because focus does not install packages that are not part of the build graph for the specified package(s). Runs yarn workspaces focus under the hood.
You can specify packages that always get installed even when using the jazelle focus command by specifying a focusRequirements field in manifest.json, containing an array of package names. This is useful if you have utility tools that normally run outside of the scope of a focused project. See focus requirements for more information.
jazelle focus [packages...] --cwd [cwd]
[packages...] - A list of packages to install--cwd - Project folder (absolute or relative to shell cwd). Defaults to process.cwd()--skipPreinstall - Skip the preinstall hook--skipPostinstall - Skip the postinstall hookverbose - Prints more Yarn warningsjazelle addAdds a dependency to the project's package.json, syncing the yarn.lock file, and the matching web_library rule in the relevant BUILD.bazel file if needed
jazelle add [deps...] --dev --cwd [cwd]
[deps...] - Name(s) of dependency and it's version to add. ie., foo@1.2.3. If version is not specified, defaults to npm info [name] version for 3rd party packages, or the local version for local packages.--dev - Whether to install as a devDependency. Default to false--cwd - Project folder (absolute or relative to shell cwd). Defaults to process.cwd()jazelle removeRemoves a dependency from the project's package.json, syncing the yarn.lock file, and the matching web_library rule in the relevant BUILD.bazel file if needed
jazelle remove [deps...] --cwd [cwd]
[deps...] - Name(s) of dependency to remove--cwd - Project folder (absolute or relative to shell cwd). Defaults to process.cwd()jazelle upgradeUpgrades a dependency across all local projects that use it
jazelle upgrade [args...]
args - Space-separated list of dependency names and optionally their desired version ranges. e.g., foo@^1.2.3. If version is not specified, defaults to npm info [name] version for 3rd party packages, or the local version for local packages. Note that local packages must be pinned to an exact version.jazelle purgeRemoves generated files (i.e. node_modules folders and bazel output files)
jazelle purge --force
--force - Whether to also purge Bazel cache. Defaults to falsejazelle checkShows a report of out-of-sync top level dependencies across projects
jazelle check --json
--json - Whether to output as JSON. This is useful if you want to pipe the report to jq (e.g jazelle changes --json | jq .jest to see report for only jest)// sample report
{
"valid": false,
"policy": {
"lockstep": false,
"exceptions": [
"my-dependency"
],
},
"reported": {
"my-dependency": {
"1.0.0": [
"my-project-1",
"my-project-2",
]
}
}
}
jazelle outdatedList packages that are outdated and their versions
jazelle outdated --json --dedup --limit [number]
--json - Whether to output as JSON. Useful for piping to jq. Defaults to false.--dedup - De-duplicates results by combining used versions into a single result. Defaults to false.--limit - Limits the number of external packages to simultaneously resolve with requests to npm. Defaults to 100.jazelle resolutionsList yarn resolutions per package in JSON format
jazelle resolutions
jazelle alignAlign a project's dependency versions to respect the version policy, if there is one
jazelle align --cwd [cwd]
--cwd - Project folder (absolute or relative to shell cwd). Defaults to process.cwd()jazelle localizeAlign dependency versions to local versions, if a local version exists
jazelle localize
jazelle chunkPrints a glob pattern representing a chunk of files matching a given list of glob patterns. Useful for splitting tests across multiple CI jobs.
Glob patterns are matched via minimatch
jazelle chunk --projects [projects] --jobs [jobs] --index [index]
--patterns - A pipe separated list of glob patterns. Patterns can be negated by prepending a ! (e.g. !tests/fixtures/*)--jobs - Divide the files among this number of chunks--index - Which chunk. For example, if patterns find 10 files, jobs is 5 and index is 1, then the third and fourth files will be returned as a .*/file-3.js|.*/file-4.js pattern.For example, it's possible to parallelize Jest tests across multiple CI jobs via a script like this:
jest --testPathPattern=$(jazelle chunk --projects "tests/**/*|!tests/fixtures/**/*" --jobs $CI_JOB_COUNT --index $CI_JOB_INDEX)
jazelle changesList projects that have changed since the last git commit.
jazelle changes
--files - A file containing a list of changed files, one per line. Defaults to stdin--format - 'targets' or 'dirs'. Defaults to 'targets'. Determine whether to return directory paths or bazel targetsThe files file can be generated via git:
git diff-tree --no-commit-id --name-only -r HEAD origin/master > files.txt
jazelle changes files.txt
Bazel targets can be tested via the bazel test [target] command.
jazelle planList an efficient grouping of test jobs to run on a server cluster.
For example, if jazelle changes reports 8 projects have changed, and there are 4 cloud nodes, this command will create 4 groupings, each containing the test, lint and flow jobs for 2 projects. If they are distributed over 24 nodes, the command will create 24 groupings, one for each job.
jazelle plan [targets] --nodes [nodes]
[targets] - A file containing a list of targets (typically from jazelle changes). Defaults to stdin--nodes - The number of nodes (i.e. cloud machines). Defaults to 1jazelle batchRuns a plan from jazelle plan, parallelizing tests across CPUs
jazelle batch [plan] --index [index]
[plan] - A file containing a plan (typically from jazelle plan). Defaults to stdin--index - Which group of tests to execute. Defaults to 0--cores - Number of cpus to use. Defaults to os.cpus().lengthjazelle buildBuilds a project and its dependencies in topological order. Calls scripts.build in package.json. See also direct Bazel usage
jazelle build --cwd [cwd]
--cwd - Project folder (absolute or relative to shell cwd). Defaults to process.cwd()jazelle devRuns a project in development mode. Calls scripts.dev in package.json
jazelle dev --cwd [cwd] [args...]
--cwd - Project folder (absolute or relative to shell cwd). Defaults to process.cwd()args - A space separated list of arguments to pass to the dev scriptjazelle testTests a project. Calls scripts.test in package.json
jazelle test --cwd [cwd] [args...]
--cwd - Project folder (absolute or relative to shell cwd). Defaults to process.cwd()args - A space separated list of arguments to pass to the test scriptjazelle lintLints a project. Calls scripts.lint in package.json
jazelle lint --cwd [cwd] [args...]
--cwd - Project folder (absolute or relative to shell cwd). Defaults to process.cwd()args - A space separated list of arguments to pass to the lint scriptjazelle flowType-checks a project. Calls scripts.flow in package.json
jazelle flow --cwd [cwd] [args...]
--cwd - Project folder (absolute or relative to shell cwd). Defaults to process.cwd()args - A space separated list of arguments to pass to the flow scriptjazelle startRuns a project. Calls scripts.start in package.json
jazelle start --cwd [cwd] [args...]
--cwd - Project folder (absolute or relative to shell cwd). Defaults to process.cwd()args - A space separated list of arguments to pass to the start scriptjazelle scriptRuns a npm script. Calls scripts[command] in package.json
jazelle script [command] [args...] --cwd [cwd]
command - A npm script to runargs - A space separated list of arguments to pass to the script--cwd - Project folder (absolute or relative to shell cwd). Defaults to process.cwd()Npm scripts can also be called by omitting the script command, for example jazelle [command].
jazelle bin-pathPrint the local path of a binary
jazelle bin-path [name]
name - 'bazel', 'node', or 'yarn'jazelle bazelRuns a Bazel command
jazelle bazel --cwd [cwd] [args...]
--cwd - Project folder (absolute or relative to shell cwd). Defaults to process.cwd()args - A space separated list of Bazel argumentsjazelle nodeRuns a Node script
jazelle node --cwd [cwd] [args...]
--cwd - Project folder (absolute or relative to shell cwd). Defaults to process.cwd()args - A space separated list of argumentsjazelle yarnRuns a Yarn command
jazelle yarn --cwd [cwd] [args...]
--cwd - Project folder (absolute or relative to shell cwd). Defaults to process.cwd()args - A space separated list of Yarn argumentsjazelle execRuns a bash script
jazelle exec --cwd [cwd] [args...]
--cwd - Project folder (absolute or relative to shell cwd). Defaults to process.cwd()args - List of shell argsjazelle eachRuns a bash script in all projects, parallelizing across CPUs
jazelle each --cores [cores] [...args]
cores - Number of cores to use. Defaults to os.cpus().length - 1args - List of shell argsjazelle bumpBumps a package and its dependencies to the next version. It also updates all matching local packages to match. Note that if a local package depends on the bumped package via workspace:*, it won't be touched.
jazelle bump [type] [--frozePackageJson] --cwd [cwd]
type - Must be one of major, premajor, minor, preminor, patch, prepatch, prerelease or nonefrozenPackageJson - If this flag is present, throws if changes to package.json are required. Useful for warning users to commit version bumps before publishing--cwd - Project folder (absolute or relative to shell cwd). Defaults to process.cwd()The bump command is idempotent, i.e. running it twice without publishing results in the same versions.
jazelle doctorSuggests fixes for some types of issues
jazelle doctor --cwd [cwd]
--cwd - Project folder (absolute or relative to shell cwd). Defaults to process.cwd()jazelle setupInstalls Jazelle hermetically. Useful for priming CI.
jazelle setup
You can run NPM scripts via jazelle yarn. For example, if you have a script called upload-files, you can call it by running jazelle yarn upload-files.
If you want commands to display colorized output, run their respective NPM scripts directly without going through Bazel (e.g. jazelle yarn lint instead of jazelle lint). This will preserve stdout/stderr colors.
const {runCLI, install, add, remove, upgrade, dedupe, check, build, test, run} = require('jazelle')
runCLIRuns a CLI command given a list of arguments
let runCLI: (args: Array<string>) => Promise<void>
args - An array of arguments, e.g. ['build', '--cwd', cwd]versionThe currently installed version. Note: this is a property, not a function.
let version: string
initGenerates Bazel files required to make Jazelle run in a workspace
let version: ({cwd: string}) => Promise<void>
cwd - Project folder (absolute path)scaffoldprescaffold and postscaffold hookslet scaffold: ({root: string, cwd: string, from: string, to: string, name?: string})
root - Monorepo root folder (absolute path)cwd - Project folder (absolute path)from - Folder to copy from. Can be an absolute path or relative to process.cwd()to - Folder to copy to. Can be an absolute path or relative to process.cwd()name - The name field in package.jsoninstalllet install: ({root: string, cwd: string}) => Promise<void>
root - Monorepo root folder (absolute path)cwd - Project folder (absolute path)ciDownloads external dependencies and links local dependencies for all projects. Does not create or modify source files. Useful for CI checks.
let ci: ({root: string, cwd: string}) => Promise<void>
root - Monorepo root folder (absolute path)cwd - Project folder (absolute path)focusDownloads external dependencies and links local dependencies for individual projects. Does not create or modify source files. Typically faster than install, but does not install packages that are not part of the requested build graph.
let focus: ({packages: Array<string>, root: string, cwd: string}) => Promise<void>
packages - List of package names to focus. Note that these should not be project paths.root - Monorepo root folder (absolute path)cwd - Project folder (absolute path)addAdds a dependency to the project's package.json, syncing the yarn.lock file, and the matching web_library rule in the relevant BUILD.bazel file if needed
let add: ({root: string, cwd: string, args: string[], version: string, dev: boolean}) => Promise<void>
root - Monorepo root folder (absolute path)args - Array of dependency names to add with their version (e.g. foo@^1.2.3). If version is not specified, defaults to npm info [name] version for 3rd party packages, or the local version for local packages.dev - Whether to install as a devDependencycwd - Project folder (absolute path)removeRemoves a dependency from the project's package.json, syncing the yarn.lock file, and the matching web_library rule in the relevant BUILD.bazel file if needed
let remove: ({root: string, cwd: string, args: string[]}) => Promise<void>
root - Monorepo root folder (absolute path)args - Array of dependency names to removecwd - Project folder (absolute path)upgradeUpgrades a dependency across all local projects that use it
let upgrade: ({root: string, args: Array<string>}) => Promise<void>
args - Space-separated list of dependency names and optionally their desired version ranges. e.g., foo@^1.2.3. If version is not specified, defaults to npm info [name] version for 3rd party packages, or the local version for local packages. Note that local packages must be pinned to an exact version.purgeRemoves generated files (i.e. node_modules folders and bazel output files)
let purge: ({root: string, force?: boolean}) => Promise<void>
root - Monorepo root folder (absolute path)force - Whether to also clear bazel cache. Defaults to falsecheckReturns a report of out-of-sync top level dependencies across projects
// sample report
{
"valid": false,
"policy": {
"lockstep": false,
"exceptions": [
"my-dependency"
],
},
"reported": {
"my-dependency": {
"1.0.0": [
"my-project-1",
"my-project-2",
]
}
}
}
type ExceptionMetadata = {
name: string,
versions: Array<string>
};
type VersionPolicy = {
lockstep: boolean,
exceptions: Array<string | ExceptionMetadata>,
}
type Report = {
valid: string,
policy: {
lockstep: boolean,
exceptions: Array<string>
},
reported: {[string]: {[string]: Array<string>}},
}
let check: ({root: string, projects: Array<string>, versionPolicy: VersionPolicy}) => Promise<Report>
root - Monorepo root folder (absolute path)outdatedList packages that are outdated and their versions
let outdated = ({root: string}) => Promise<void>
resolutionsList yarn resolutions per package in JSON format
let resolutions = ({root: string}) => Promise<{[string]: {[string]: string}}>
alignAlign a project's dependency versions to respect the version policy, if there is one
let align: ({root: string, cwd: string}) => Promise<void>
root - Monorepo root folder (absolute path)cwd - Project folder (absolute path)localizeAlign dependency versions to local versions, if a local version exists
let localize: ({root: string}) => Promise<void>
root - Monorepo root folder (absolute path)chunkReturns a glob pattern representing a chunk of files matching a given list of glob patterns. Useful for splitting tests across multiple CI jobs.
Glob patterns are matched via minimatch
let chunk: ({root: string, patterns: Array<string>, jobs: number, index: number}) => Promise<string>
root - Monorepo root folder (absolute path)patterns - A list of glob patterns. Patterns can be negated by prepending a ! (e.g. !tests/fixtures/*)jobs - Divide the files among this number of chunksindex - Which chunk. For example, if patterns find 10 files, jobs is 5 and index is 1, then the third and fourth files will be returned as a .*/file-3.js|.*/file-4.js pattern.For example, it's possible to parallelize Jest tests across multiple CI jobs via a script like this:
jest --testPathPattern=$(node -e "console.log(require('jazelle').chunk({projects: ['tests/**/*', '!tests/fixtures/**/*'], jobs: $CI_JOB_COUNT, index: $CI_JOB_INDEX}))")
changesList projects that have changed since the last git commit.
let changed: ({root: string, files: string, type: string}) => Promise<Array<string>>
root - Monorepo root folder (absolute path)files - The path to a file containing a list of changed files, one per lineformat - 'targets' or 'dirs'. Defaults to 'targets'. Determine whether to return directory paths or bazel targetsThe files file can be generated via git:
git diff-tree --no-commit-id --name-only -r HEAD origin/master > files.txt
Bazel targets can be tested via the bazel test [target] command.
planList an efficient grouping of test jobs to run on a server cluster.
For example, if changes reports 8 projects have changed, and there are 4 servers, this function will create 4 groupings, each containing the test, lint and flow jobs for 2 projects.
type PayloadMetadata = {type: string, dir: string, action: string}
let plan: ({root: string, data: Array<string>, nodes: number}) => Promise<Array<Array<PayloadMetadata>>>
root - Monorepo root folder (absolute path)data - A report of targets (typically from the changes method)nodes - The number of nodes (i.e. cloud machines)batchRuns a plan from jazelle plan, parallelizing tests across CPUs
type DirTestMetadata = {dir: string, script: string}
type BazelTestMetadata = {target: string}
type TestMetadata = DirTestMetadata | BazelTestMetadata
type TestGroup = Array<TestMetadata>
let batch: ({root: string, data: Array<TestGroup>, index: number, cores: number}) => Promise<void>
root - Monorepo root folder (absolute path)plan - A file containing a plan (typically from jazelle plan). Defaults to stdinindex - Which group of tests to executecores - Number of cpus to use. Defaults to os.cpus().lengthbuildBuilds a projects and its dependencies in topological order. Calls scripts.build in package.json
let build: ({root: string, cwd: string}) => Promise<void>
root - Monorepo root folder (absolute path)cwd - Project folder (absolute path)testTests a project. Calls scripts.test in package.json
let test: ({root: string, cwd: string, args: Array<string>}) => Promise<void>
root - Monorepo root folder (absolute path)cwd - Project folder (absolute path)args - A list of arguments to pass to the test scriptdevRuns a project in development mode. Calls scripts.dev in package.json
let dev: ({root: string, cwd: string, args: Array<string>}) => Promise<void>
root - Monorepo root folder (absolute path)cwd - Project folder (absolute path)args - A list of arguments to pass to the dev scriptlintLints a project. Calls scripts.lint in package.json
let lint: ({root: string, cwd: string, args: Array<string>}) => Promise<void>
root - Monorepo root folder (absolute path)cwd - Project folder (absolute path)args - A list of arguments to pass to the lint scriptflowType-checks a project. Calls scripts.flow in package.json
let flow: ({root: string, cwd: string, args: Array<string>}) => Promise<void>
root - Monorepo root folder (absolute path)cwd - Project folder (absolute path)args - A list of arguments to pass to the flow scriptstartRuns a project. Calls scripts.start in package.json
let start: ({root: string, cwd: string, args: Array<string>}) => Promise<void>
root - Monorepo root folder (absolute path)cwd - Project folder (absolute path)args - A list of arguments to pass to the start scriptscriptRuns a npm script. Calls scripts[command] in package.json
let script: ({root: string, cwd: string, args: Array<string>}) => Promise<void>
root - Monorepo root folder (absolute path)cwd - Project folder (absolute path)args - A list of arguments to pass to the script. The first argument should be a command namebinPathPrint the local path of a binary
let binPath: = (name: 'bazel' | 'node' | 'yarn') => string
name - 'bazel', 'node', or 'yarn'bazelRuns a Bazel command
let bazel: ({root: string, cwd: string, args: Array<string>}) => Promise<void>
root - Monorepo root folder (absolute path)cwd - Project folder (absolute path)args - List of Bazel argsnodeRuns a Node script
let node: ({root: string, cwd: string, args: Array<string>}) => Promise<void>
cwd - Project folder (absolute path)args - List of argsyarnRuns a Yarn command
let yarn: ({root: string, cwd: string, args: Array<string>}) => Promise<void>
cwd - Project folder (absolute path)args - List of Yarn argsexecRuns a bash script
let exec: ({root: string, cwd: string, args: Array<string>}) => Promise<void>
root - Monorepo root folder (absolute path)cwd - Project folder (absolute path)args - List of shell argseachRuns a bash script in all projects, parallelizing across CPUs
let each: ({root: string, args: Array<string>, cores: string}) => Promise<void>
cwd - Project folder (absolute path)args - List of shell argscores - Number of cores to use. Defaults to os.cpus().length - 1bumpBumps a package and its dependencies to the next version. It also updates all matching local packages to match
let bump = ({root: string, cwd: string, type: string, frozenPackageJson?: boolean})
root - Monorepo root folder (absolute path)cwd - Project folder (absolute path)type - Must be one of major, premajor, minor, preminor, patch, prepatch, prerelease or nonefrozenPackageJson - If true, throws if changes to package.json are required. Useful for warning users to commit version bumps before publishing. Defaults to false.The bump command is idempotent, i.e. running it twice without publishing results in the same versions.
doctorSuggests fixes for some types of issues
let doctor: ({root: string, cwd: string}) => Promise<void>
root - Monorepo root folder (absolute path)cwd - Project folder (absolute path)getRootDirFinds the absolute path of the monorepo root folder
let getRootDir: ({dir: string}) => Promise<string>
dir - Any absolute path inside the monorepoNote: The manifest.json file does not allow comments; they are present here for informational purposes only.
{
// Optional workspace mode
"workspace": "sandbox",
// Optional installation hooks
"hooks": {
"prescaffold": "echo before",
"postscaffold": "echo after",
"preinstall": "echo before",
"postinstall": "echo after",
"postcommand": "echo after command",
"bool_shouldinstall": "echo true",
},
// Optional version policy
"versionPolicy": {
"lockstep": true,
"exceptions": [
"foo",
{ "name": "bar", "versions": ["1.0.0", "2.3.7"] },
]
},
// Optional focus requirements
"focusRequirements": ["my-project-a", "my-project-b"],
// Optional rule name to use when auto-updating target `deps` in BUILD.bazel
"dependencySyncRule": "my_repo_target",
}
Projects paths must be listed in the workspaces field of the root level package.json file. Paths must be relative to the root of the monorepo.
{
"workspaces": [
"path/to/project-1",
"path/to/project-2",
],
}
The workspace field enables Bazel if set to sandbox, or uses a pure JS implementation if set to host. Defaults to host.
The difference between the two modes is that host mode lets you build projects even if you don't specify all of your non-NPM dependencies. An example of a non-NPM dependency is a configuration file outside of the project's folder (e.g. if you are implementing consistent configuration across projects). In sandbox mode, you must add this file to the deps field of the web_library rule in the project's BUILD.bazel file. In host mode, Jazelle will let you build your project without looking at BUILD.bazel files and without nagging about the fact that the configuration file was not explicitly specified as a dependency.
Note that host mode is only meant to be used to help importing projects. In host mode, dependency graph resolution is only guaranteed for dependencies tracked via package.json. Changes in dependencies that tracked exclusively in BUILD files and/or untracked dependencies are not accounted for when determining when a build cache needs to be invalidated, and may result in overly aggressive caching despite changes that affect a project.
Also note that currently, jazelle changes will only report changes that Bazel can detect. This means that if you never used sandbox mode, it will not report any changes.
It's strongly recommended that you use sandbox mode.
Scaffold hooks run shell scripts before/after a project is scaffolded.
{
"hooks": {
"prescaffold": "echo before",
"postscaffold": "echo after"
}
}
Installation hooks run shell scripts before/after dependency installation.
{
"hooks": {
"preinstall": "echo before",
"postinstall": "echo after",
"postcommand": "echo after command",
}
}
Boolean hooks are a special type of hook that can conditionally enable/disable jazelle behavior. They work by emitting true or false to stdout (must be the last thing emitted in the script).
The bool_shouldinstall hook should echo either true or false. If false, calling jazelle install will bypass yarn install. This is useful if you have custom logic that caches installations or if you are using zero-installs.
{
"hooks": {
"bool_shouldinstall": "echo false",
}
}
Example bool_shouldinstall hook:
#!/usr/bin/env bash
if ./pull_yarn_cache.sh; then
# dependency cache was successfully pulled; don't install
echo 'false'
exit 0
fi
echo 'true'
The version policy structure specifies which direct dependencies must be kept in the same version in all projects that use them within the monorepo.
The version policy is enforced when running jazelle install, jazelle add, jazelle remove and jazelle upgrade.
If you change the version policy, it's your responsibility to run jazelle check to ensure that projects conform to the new policy, and to run jazelle upgrade to fix version policy violations.
{
"versionPolicy": {
"lockstep": true,
"exceptions": [
"foo"
]
}
}
The lockstep field indicates whether ALL dependencies should be kept in the same version.
The exceptions field is a list of package names that should ignore the lockstep policy. For example, if lockstep is true and exceptions includes a package named foo, all dependency versions must be in lockstep, except foo. Conversely, if lockstep is false, and exceptions include a package foo, then all projects that use foo must use the same version of foo, but are free to use any version of any other package.
It's recommended that you set the policy to the following:
{
"versionPolicy": {
"lockstep": true,
}
}
You should avoid adding exceptions if using this policy.
Here's an alternative policy that may be more pragmatic for large existing codebases where only some packages are kept up-to-date by a platform team:
{
"versionPolicy": {
"lockstep": false,
"exceptions": [
"foo",
"bar"
]
}
}
The exceptions field may also identify specific versions for dependencies. These must match the declared versions in package.json exactly (i.e. 1.4.0 and ^1.4.0 are considered two separate versions).
{
"versionPolicy": {
"lockstep": true,
"exceptions": [
"foo",
{ "name": "bar", "versions": [ "1.4.0", "2.3.7" ] }
]
}
}
Jazelle supports partial installation via jazelle focus. It may be desirable to always forcefully install a set of packages if they are frequently used tools that cannot be invoked from individual projects (for example, codemodding tools)
In the example below, it becomes possible to run commands from the my-tool package, even if the user only runs jz focus my-package-a (i.e. they didn't explicitly ask to focus my-tool)
// manifest.json
{
"focusRequirements": ["my-tool"]
}
The dependencySyncRule field takes a string containing the name of a bazel rule. By default this is web_library. This is configurable for cases where the web_library rule is generated via a Bazel macro, so that Jazelle is still able to codemod deps fields in BUILD.bazel files that consume these macros instead of consuming web_library directly.
# rules/my-macro.bzl
load(
"@jazelle//:build-rules.bzl",
# underscore prefix to prevent re-exporting
_web_library = "web_library",
_web_binary = "web_binary",
_web_executable = "web_executable",
_web_test = "web_test",
)
def my_macro(srcs = [], deps = []):
_web_library(
name = "library",
deps = deps + ["//:some-centrally-managed-target"],
srcs = srcs,
)
#...
# my-project/BUILD.bazel
load("//rules:my_macro.bzl", "my_macro")
package(default_visibility = ["//visibility:public"])
my_macro(
srcs = glob(["**"]),
deps = [
# jazelle can codemod this field on add/remove commands if `dependencySyncRule` is set to `my_macro`
]
)
The third_party/jazelle/build-file-template.js file should be a js file that exports a named export called template.
module.exports.template = async ({name, path, label, dependencies}) => `# a BUILD.bazel template`
let template: ({name: string, path: string, label: string, dependencies: Array<string>}) => Promise<string>
name - The shorthand name of the target. For example, if the path to the project is path/to/foo, name is foopath - The project's path, relative to the monorepo rootlabel - The fully qualified Bazel label. For example //path/to/foo:foodependencies - A list of labels for Bazel targets that are dependencies of the current ruleHere's an example of a custom build that changes the dist folder of library compilations and sets up a flow command for type checking:
// @flow
/*::
type TemplateArgs = {
name: string,
path: string,
label: string,
dependencies: Array<string>,
}
type Template = (TemplateArgs) => Promise<string>;
*/
const template /*: Template */ = async ({name, path, dependencies}) => `
package(default_visibility = ["//visibility:public"])
load("@jazelle//:build-rules.bzl", "web_library", "web_binary", "web_executable", "web_test", "flow_test")
web_library(
name = "library",
deps = [
"//third_party/jazelle:node_modules",
${dependencies.map(d => `"${d}",`).join('\n ')}
],
srcs = glob(["**/*"], exclude = ["dist/**"]),
)
web_binary(
name = "${name}",
build = "build",
command = "start",
deps = [
"//${path}:library",
],
dist = ["dist"],
)
web_executable(
name = "dev",
command = "dev",
deps = [
"//${path}:library",
],
)
web_test(
name = "test",
command = "test",
deps = [
"//${path}:library",
],
)
web_test(
name = "lint",
command = "lint",
deps = [
"//${path}:library",
],
)
flow_test(
name = "flow",
deps = [
"//${path}:library",
],
)`;
module.exports = {template};
Note that BUILD.bazel files are not regenerated once they have been created. You can edit them after they've been created if you need to name certain targets differently in specific projects, or if you need to add custom Bazel rules or non-JS Bazel dependencies.
Jazelle edits web_library rules when jazelle add and jazelle remove commands are issued, in order to update the deps list. If you use different rules to build web projects, you must keep the BUILD.bazel file in sync with your package.json file yourself.
The easiest to setup Bazel in an empty repository is to run jazelle init. If you are setting up Jazelle on an existing Bazel workspace, you need to manually add Jazelle rules to the root WORKSPACE and BUILD.bazel files.
Before Jazelle rules can be used in Bazel, you must first import them:
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "jazelle",
url = "https://registry.yarnpkg.com/jazelle/-/jazelle-[version].tgz",
sha256 = "SHA 256 goes here",
)
Jazelle SHA256 checksum can be computed through the following command:
curl -fLs https://registry.yarnpkg.com/jazelle/-/jazelle-[version].tgz | openssl sha256
Rules that should be used from a monorepo's WORKSPACE file.
load("@jazelle//:workspace-rules.bzl", "jazelle_dependencies")
jazelle_dependencies ruleDownload and install binary dependencies (i.e. Node, Yarn) hermetically.
jazelle_dependencies(
node_version = string,
node_sha256 = {
"mac": string,
"linux": string,
"windows": string,
},
yarn_version = string,
yarn_sha256 = string,
)
node_version - The version of Node that should be installednode_sha256 - The checksum of the architecture-specific distribution files of the chosen Node versionyarn_version - The version of Yarn that should be installedyarn_sha256 - The checksum of the chosen Yarn versionJazelle SHA256 checksum can be computed through the following command:
curl -fLs https://github.com/lhorie/jazelle/releases/download/v[version]/jazelle-[version].tar.gz | openssl sha256
Node SHA256 checksums can be found at https://nodejs.org/dist/v[version]/SHASUMS256.txt. Use the checksums for these files:
node-v[version]-darwin-x64.tar.gznode-v[version]-linux-x64.tar.xznode-v[version]-win-x64.zipYarn SHA256 checksum can be computed through the following command:
curl -fLs https://github.com/yarnpkg/yarn/releases/download/v[version]/yarn-[version].js | openssl sha256
Rules that should be used from a project's BUILD.bazel file.
load("@jazelle//:build-rules.bzl", "web_library", "web_binary", "web_test")
jazelle ruleRun the Jazelle CLI from Bazel. This rule should be added to the root of the monorepo. Note that all arguments must be passed after --
bazel run //:jazelle -- version
jazelle(name = "jazelle", manifest = "manifest.json")
name - Should be jazellemanifest - Should be manifest.jsonweb_library ruleDescribes a set of files as a library
web_library(
deps = [string],
srcs = [string],
)
deps - A list of target labels that are dependencies of this rulesrcs - A list of source code files for the projectThe deps argument in this rule is dynamically updated by Jazelle when you run jazelle add or jazelle remove. You should always declare project dependencies in this rule, rather than on the rules below.
This rule collects transitive files from the DefaultInfo(files) provider of targets specified by deps and outputs them as transitive files via the DefaultInfo(files) provider.
web_binary ruleBuilds a project via an npm script and optionally runs it. If this rule is run via bazel build, it generates a output.tgz file representing the project's compiled assets. If this rule is run via bazel run, it additionally extracts the output file and runs the project using the NPM script specified by command. The build step is cacheable, while the run step is not.
web_binary(
build = string,
command = string,
deps = [string],
dist = string,
preserve_symlinks = string,
skip_pnp = boolean,
)
build - The npm script to build the project. Defaults to buildcommand - The npm script to run the project. Defaults to start. If the command is run, the rule acts like yarn run [command]deps - A list of target labels that are dependencies of this ruledist - The name of the output folder where compiled assets are saved topreserve_symlinks - Whether to use the Node PRESERVE_SYMLINKS flag. Set to "1" for true or "" for false.skip_pnp - Boolean flag to bypass the auto require of the Yarn 2 .pnp.cjs file when executing Node commands.This rule consumes transitive files from the DefaultInfo(files) provider of targets specified by deps. If the transitive files include output.tgz files, they are extracted into the root folder of their respective project (in the Bazel sandbox).
web_executable ruleRuns a npm script (e.g. yarn start). Meant to be used with bazel run
web_executable(
command = string,
deps = [string],
)
command - The npm script to executedeps - A list of target labels that are dependencies of this rulegen_srcs - A list of regexp strings for files or folders that should be copied back to the source folder after sandbox execution. Useful for code generation (e.g. jest snapshots)preserve_symlinks - Whether to use the Node PRESERVE_SYMLINKS flag. Set to "1" for true or "" for false.This rule consumes transitive files from the DefaultInfo(files) provider of targets specified by deps. If the transitive files include output.tgz files, they are extracted into the root folder of their respective project (in the Bazel sandbox).
web_test ruleRuns a npm script as a cacheable test (e.g. yarn test). Meant to be used with bazel test
web_test(
command = string,
deps = [string],
)
command - The npm script to executedeps - A list of target labels that are dependencies of this rulegen_srcs - A list of regexp strings for files or folders that should be copied back to the source folder after sandbox execution. Useful for code generation (e.g. jest snapshots)preserve_symlinks - Whether to use the Node PRESERVE_SYMLINKS flag. Set to "1" for true or "" for false.This rule consumes transitive files from the DefaultInfo(files) provider of targets specified by deps. If the transitive files include output.tgz files, they are extracted into the root folder of their respective project (in the Bazel sandbox).
flow_test ruleRuns yarn flow
flow_test(
command = string,
deps = [string],
)
deps - A list of target labels that are dependencies of this ruleThis rule consumes transitive files from the DefaultInfo(files) provider of targets specified by deps. If the transitive files include output.tgz files, they are extracted into the root folder of their respective project (in the Bazel sandbox).
Jazelle commands are similar to yarn commands, but not exactly equivalent. Here's a table showing similar commands and their differences.
| Jazelle | Yarn | Key differences |
|---|---|---|
jazelle install | yarn install | The Jazelle command also sets up local dependencies |
jazelle build | yarn run build | The Jazelle command also builds (and caches) local dependencies |
jazelle test | yarn run test | The Jazelle command caches tests for projects whose code didn't change |
jazelle add x | yarn add x | The Jazelle command also manages deps declared in BUILD.bazel files |
jazelle foo | yarn run foo | The Jazelle command also manages deps declared in BUILD.bazel files |
You should always use Jazelle commands instead of Yarn commands.
Jazelle allows using Bazel directly for building targets. Here's a table showing equivalent commands:
| Jazelle | Bazel |
|---|---|
cd a && jazelle install | N/A |
cd a && jazelle add x | N/A |
cd a && jazelle build | bazel build //a:a |
cd a && jazelle start | bazel run //a:a |
cd a && jazelle dev | bazel run //a:dev |
cd a && jazelle test | bazel test //a:test |
cd a && jazelle lint | bazel run //a:lint |
cd a && jazelle flow | bazel run //a:flow |
cd a && jazelle foo | bazel run //a:script -- foo |
You can use either Jazelle commands or Bazel commands interchangeably. This is helpful if your team is already invested into a Bazel-centric workflow.
It's recommended that you use Jazelle commands instead of Bazel, because Jazelle uses Bazelisk to enforce a Bazel version. You could also use Bazelisk itself.
Jazelle supports using a preinstalled Bazel binary (for example, you may want to preinstall it in a docker layer for CI)
If Jazelle detects that Bazel is installed via which bazel and the installed version matches the version specified in the .bazelversion file, Jazelle will use the installed version instead of attempting to download it.
If you want to use a Bazel binary that is not in your PATH, you can also specify it via the BAZEL_PATH environment variable:
BAZEL_PATH=/path/to/my/bazel jazelle ...
In addition to package.json scripts, Jazelle supports the ability to execute node scripts.
web_binary(
build = "${NODE} ${ROOT_DIR}/foo.js"
# ...
)
${NODE} is a special variable that refers to the node binary, and automatically requires .pnp.cjs${ROOT_DIR} refers to the root of the repoNote that these are not bash variables (i.e. $ROOT_DIR doesn't work)
If you are a monorepo maintainer and you need to implement static analysis logic that runs against files of every project in a monorepo, it's not feasible to depend on all projects at build time, since the build graph could conceivably require rebuilding every project in the monorepo. Instead, you can depend only on specific files.
The simplest way to do that is to add exports_files() or filegroup() declarations in buildFileTemplate to expose the desired files. This way you can put your logic in a package that depends on files from several projects:
# BUILD.bazel file in analyzable projects
exports_files([
"//my-project:package.json", # expose the file we need for static analysis
])
# BUILD.bazel in static analysis project
js_binary(
name = "check"
command = "check",
deps = [
"//my-project:package.json",
"//my-other-project:package.json",
# ...
]
)
You can dynamically update the deps argument of the static analysis project BUILD.bazel file by writing a preinstall script that parses and edits the BUILD.bazel file. The list of monorepo projects is conveniently available in the root level package.json file.
Note that updating buildFileTemplate does not change existing BUILD.bazel files (since they could contain custom rules and modifications). If you want the same changes in existing files, you will have to edit those files yourself.
By default you must declare //:yarn.lock and //:.pnp.cjs as explicit deps in any target that needs to consume Javascript packages if you want fully hermetic lockfiles.
However, Jazelle does work even if they are not specified. Jazelle supports the ability of inferring the location of yarn.lock (and .pnp.cjs) without those files being present in the Bazel graph. This allows a repo to implement a custom yarn plugin that outputs files describing the version locking requirements per project (as opposed to having all of that information tied to a single top-level file). See yarn-plugin-workspace-deps for an example.
The benefit of inferred top-level lockfile is that in large enough repos, yarn.lock is changed frequently by multiple different teams, and it leads to poor cacheability of targets if any team's changes can blow away the cache of unrelated projects.
jazelle changesFAQs
Incremental, cacheable builds for large Javascript monorepos. Uses [Bazel](https://bazel.build)
We found that jazelle demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
PyPI confirms no security flaws were exploited in the Ultralytics supply chain attack and highlights improvements for safer package publishing.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.