@yarnpkg/shell

@yarnpkg/shell

A JavaScript implementation of a bash-like shell (we use it in Yarn 2 to provide cross-platform scripting). This package exposes an API that abstracts both the parser and the interpreter; should you only need the parser you can check out @yarnpkg/parsers, but you probably won't need it.

Usage

import {execute} from '@yarnpkg/shell';

process.exitCode = await execute(`ls "$0" | wc -l`, [process.cwd()]);

Features

• Typechecked
• Portable across systems
• Supports custom JS builtins
• Supports pipes
• Supports glob patterns (only for files that exist on the filesystem: ls *.txt)
• Supports logical operators
• Supports subshells
• Supports variables
• Supports string manipulators
• Supports argc/argv
• Supports background jobs with color-coded output
• Supports the most classic builtins
• Doesn't necessarily need to access the fs

Help Wanted

• Full glob support (mv build/{index.js,index.build.js}, echo {foo,bar}, FOO=a,b echo {$FOO,x})
• More string manipulators

Non-Goals

• Perfect POSIX compliance (basic scripting is enough for now)
• Multiline scripts (we mostly target one-liners)
• Control structures (same reason)

Changelog

4.0.0

Major Changes

• With Node.js 16's now being End of Life'd, we dropped support for Node.js versions lower than 18.12.

• Some important defaults have changed:

  • yarn init and yarn set version will prefer using packageManager rather than yarnPath when possible (when they detect COREPACK_ROOT in your environment variables).

  • yarn init will no longer use zero-install by default. You still can enable it, but it should make it easier to start one-of projects without having to rewrite the configuration afterwards.

    • As a result, enableGlobalCache now defaults to true. If your project uses Zero-Installs, the first yarn install you run after migrating to 4.0 will automatically set enableGlobalCache: false in your local .yarnrc.yml.

  • yarn workspaces foreach now requires one of --all, --recursive, --since, or --worktree to be explicitly specified; the previous default was --worktree, but it was rarely what users expected.

  • compressionLevel now defaults to 0 rather than mixed. It's been proved significantly faster on installs, and the size impact was reasonable enough to change the default. Note that it benefits you even if you use Zero-Installs: as per our tests, a zero-compression is actually easier to handle for Git (you can see by yourself with those examples using compressionLevel: 0 vs compressionLevel: mixed).

    • To avoid making the upgrade too disruptive, Yarn will check whether Zero-Installs are enabled the first time you run yarn install after migrating from 3.6 to 4.0. If you do, it will automatically set the old default (compressionLevel: mixed) in your .yarnrc.yml file. You can then remove it whenever you feel ready to actually change the compression settings.

• All official Yarn plugins are now included by default in the bundle we provide. You no longer need to run yarn plugin import for official plugins (you still need to do it for third-party plugins, of course).

  • This doesn't change anything to the plugin API we provide, which will keep being maintained.
  • Yarn still has a modular architecture and uses the exact same APIs as contrib plugins; all that changes is how we distribute our own features.

• Yarn's UI during installs has been greatly improved:

  • Packages added and removed from the lockfile are now explicitly reported.
  • Fluctuations in the project cache size are now reported as a single line.
  • Unactionable warnings (node-gyp and transitive peer dependency errors) have been removed.
  • Skipped builds are now only reported during initial installs and manual yarn rebuild calls.
  • The Yarn version is now displayed on installs to help us investigate issues when reported as screenshots.
  • Deprecation checks have been moved to yarn npm audit.

• Some settings were renamed or removed:

  • caFilePath is now httpsCaFilePath
  • preferAggregateCacheInfo has been removed (it's now always on)
  • pnpDataPath has been removed to adhere to our new PnP specification. For consistency, all PnP files will now be hardcoded to a single value so that third-party tools can implement the PnP specification without relying on the Yarn configuration.

• The yarn npm audit command has been reimplemented:

  • The audit registry must now implement the /-/npm/v1/security/advisories/bulk endpoint.
  • The npmAuditRegistry can be used to temporarily route audit queries to the npm registry.
  • Deprecations are now returned by default. To silence them, use yarn npm audit ! --no-deprecations.

• Some legacy layers have been sunset:

  • Plugins cannot access the Clipanion 2 APIs anymore (upgrade to Clipanion 3)
  • Plugins cannot access the internal copy of Yup anymore (use Typanion instead)
  • Yarn will no longer remove the old Yarn 2.x .pnp.js file when migrating.
  • The --assume-fresh-project flag of yarn init has been removed.

API Changes

The following changes only affect people writing Yarn plugins:

• The ZipFS and ZipOpenFS classes have been moved from @yarnpkg/fslib to @yarnpkg/libzip. They no longer need or accept the libzip parameter.

  • Reading the zip archives is now done on the Node.js side for performance; as a result, the open, ZIP_CREATE, and ZIP_TRUNCATE bindings are no longer needed for ZipFS and have also been removed.

• The dependencies field sent returned by Resolver#resolve must now be the result of a Configuration#normalizeDependencyMap call. This change is prompted by a refactoring of how default protocols (ie npm:) are injected into descriptors. The previous implementation caused various descriptors to never be normalized, which made it difficult to know what were the descriptors each function should expect.

  • Similarly, the descriptors returned by Resolve#getResolutionDependencies are now expected to be the result of Configuration#normalizeDependency calls.

  • Note that this only applies to the dependencies field; the peerDependencies field is unchanged, as it must only contains semver ranges without any protocol (with an exception for workspace:, but that's not relevant here).

• The Resolve#getResolutionDependencies function must now return an object of arbitrary string keys and descriptor values (instead of a map with DescriptorHash keys). Those descriptors will be resolved and assigned to the same keys as the initial object. This change allows resolvers to wrap resolution dependencies from other resolvers, which wasn't possible before since it'd have caused the key to change.

• The generateLoader function in @yarnpkg/pnp no longer generates the $$SETUP_STATE function, it now needs to be present in the loader passed to the function.

• The getCustomDataKey function in Installer from @yarnpkg/core has been moved to Linker.

• renderForm's options argument is now required to enforce that custom streams are always specified.

• npmConfigUtils.getAuditRegistry no longer takes a Manifest as its first argument.

• The FetchOptions.skipIntegrityCheck option has been removed. Use FetchOptions.cacheOptions.skipIntegrityCheck instead.

• MapConfigurationValue has been removed. Use miscUtils.ToMapValue instead.

• Manifest.isManifestFieldCompatible and Manifest.prototype.isCompatibleWith{OS,CPU} have been removed. Use Manifest.prototype.getConditions and structUtils.isPackageCompatible instead.

• versionUtils.{fetchBase,fetchRoot,fetchChangedFiles} have been moved from @yarnpkg/plugin-version to @yarnpkg/plugin-git. Use gitUtils.{fetchBase,fetchRoot,fetchChangedFiles} instead.

• For consistency reasons:

  • Link{Resolver,Fetcher} have been renamed to Portal{Resolver,Fetcher}
  • RawLink{Resolver,Fetcher} have been renamed to Link{Resolver,Fetcher}

• FakeFS classes are now required to implement lutimes{Sync,Promise}.

• workspace.dependencies has been removed. Use workspace.anchoredPackage.dependencies instead.

• The Installer class must now return BuildRequest structures instead of BuildDirective[]. This lets you mark that the build must be skipped, and the reason why.

• startCacheReport has been removed, and is now part of the output generated by fetchEverything.

• forgettableNames & forgettableBufferSize have been removed (the only messages using them have been removed, making the forgettable logs implementation obsolete).

• workspace.locator has been removed. You can instead use:

  • workspace.anchoredLocator to get the locator that's used throughout the dependency tree.
  • workspace.manifest.version to get the workspace version.

• configuration.{packageExtensions,refreshPackageExtensions} have been removed. Use configuration.getPackageExtensions instead.

• configuration.normalizePackage now requires a packageExtensions option.

• ProjectLookup has been removed. Both Configuration.find and Configuration.findProjectCwd now always do a lockfile lookup.

Installs

• Yarn now caches npm version metadata, leading to faster resolution steps and decreased network data usage.
• The pnpm linker avoids creating symlinks that lead to loops on the file system, by moving them higher up in the directory structure.
• The pnpm linker no longer reports duplicate "incompatible virtual" warnings.

Features

• enableOfflineMode is a new setting that, when set, will instruct Yarn to only use the metadata and archives already stored on the local machine rather than download them from the registry. This can be useful when performing local development under network-constrained environments (trains, planes, ...).
• yarn run bin now injects the environment variables defined in .env.yarn when spawning a process. This can be configured using the injectEnvironmentFiles variable.
• yarn workspaces foreach now automatically enables the yarn workspaces foreach ! --verbose flag in interactive terminals.
• Constraints can now be written in JavaScript. See the revamped documentation for more information.

Bugfixes

• yarn dlx will no longer report false-positive UNUSED_PACKAGE_EXTENSION warnings
• yarn workspace will now set $INIT_CWD to the CLI working directory rather than the workspace root.

Shell

• The builtin shell now supports whitespace-only commands.

Compatibility

• The patched filesystem now supports FileHandle.readLines.
• PnP now reports missing files when in watch mode.