oriservice-darwin-arm64

ori

Tools and plugins to run innerloop builds of typescript monorepos using esbuild.

Configuration & Usage

ori -h

  -blockFollowUp
        Wait for an initial build before running non-build tasks (implied by -traceInitialBuild)
  -config string
        Path to ori.json (default "./ori.json")
  -cpuprofile string
        Generate a cpu profile at the given path
  -entry string
        Use a given entry or entry group (from the values specified in ori.json)
  -gitRef string
        initial set of changed files to use when starting the typescript process (default "HEAD")
  -initialOnly
        Validate the initial build can complete, and exit with an error if it had issues
  -logLevel string
        log level (error|warning|info|verbose|debug)
  -logTs
        log typescript output to stdio
  -noTui
        Disable the tui and print everything to stdio
  -nosplit
        Disable codesplitting. Allows for bundling without esm.
  -port int
        Port to run the http server on (default 3000)
  -snoop
        log imports to a snoop.json file, which can be analyzed to determine why a module is included in a build.
  -sourcemap string
        Generate sourcemaps (one of 'none', 'inline', 'external', 'linked', 'inline-and-external') (default "none")
  -strategy string
        Build Strategy (default "vendor")
  -trace string
        Generate an event trace at the given path
  -traceIncrementalBuilds
        Collect a pprof trace of each incremental build
  -traceInitialBuild
        Collect a pprof trace of the initial build
  -version
        Print the version and exit
  -write
        Write to disk

ori.json fields

ori.json fields

{
    // The path that esbuild should output to
    "outPath": "dist/esbuild",
    // Where to find resource.json files
    //
    // TODO: document resource.json files
    //
    // Should be deprecated by #4
    "resourceRoots": ["packages", "shared"],
    // Where to find source files to watch
    //
    // Should be deprecated by #4
    "watchSourceRoots": ["packages", "shared"],
    // Directories the webserver should serve in addition to serving
    // resources from resources.json and the built scripts + chunks
    "directServeDirectories": ["resources"],
    // constants to define, passed to esbuild's define property
    // see https://esbuild.github.io/api/#define
    "defineConstants": {
        "global": "self",
        "process.env.IS_WEBPACK": "false"
    },
    // A map of entry script names to the packages they are built from
    // (packages are read from tsconfig.paths.json, should be replaced
    // with packageNames in the workspaces we crawl)
    "entry": {
        "mailindex": "mail-index-package-name",
    },
    // Entrypoints to workers
    //
    // Workers are built separately, see WorkerLoader for details
    //
    // Worker entrypoints not in this map will be built inline in the main,
    // build, at significant performance cost
    "workerRawEntries": {
        "pdfjsworker": "node_modules/pdfjs-dist/build/pdf.worker.js",
        "pdfjsworkermin": "node_modules/pdfjs-dist/build/pdf.worker.min.js",
        "owadataworker": "packages/libraries/worker/owa-data-worker-bootstrap/src/index.worker.ts"
    },
    // Human readable groups of entries from the above entries map
    // as well as custom extensions to defineConstants
    //
    // For use on the cli for common entry goups.
    "entryGroups": {
        "OWA Mail": {
            "entries": ["mailindex"],
            "defineConstants": {
                "OWA_BUILD_CONSTANTS.ENTRIES.mail": "true",
                "OWA_BUILD_CONSTANTS.BUILD_ALL": "false"
            }
        }
    }
}

Build Strategies

Rather than alwasy running a single esbuild build, it is sometimes more performant or practical to run multiple separate builds. The different ways of coordinating and connecting these separate builds are called "build strategies".

Strategy single

This build strategy will run everything as a single giant esbuild run. This is the simplest and least error-prone approach, but will also tend to be the least performant

Strategy vendor

This build strategy scans your repository for information about external dependencies on startup, and uses that information to build all of your external dependenices as a separate build.

This tends to speed up interactive iterative builds, because it cuts out the dependencies from the code being rebuilt. However, if new imports are requested from the import set, the whole vendor build will have to re-run. Depending on the size of your main build, & your dependencies, rebuilding both may be slower than rebuilding both with single.

Strategy partitioned

This build strategy splits the entrypoints to a build between multiple other strategies, called partitions. No attempt to resolve imports or deduplicate between the separate builds will be made, so this is for scenarios where part of an app needs to be built separately from the rest. It also supports settings overrides, so it can be useful if part of your app needs to build a CJS bundle for legacy reasons.

As it is a parameterized strategy, It must be configured via the strategies section of ori.json, and later referenced by name.

Working on ori

Getting Dependencies

• install go 1.18 https://go.dev/doc/install

• If on windows, install mingw-gcc. This is to support building libsass on windows https://github.com/wellington/go-libsass/issues/37

  There are two ways to do this right now:

  1. Via Chocolatey:

     • Install Chocolatey
     • Install mingw from chocolatey choco install mingw. At time of writing this installs mingw 11.2.0.07112021

  2. Manually: do this only if you encounter issues with the version distributed by chocolatey:

     • Get a build from https://www.mingw-w64.org/downloads/#mingw-builds (the sourceforge link)
     • If the live installer fails, you can download the prebuilt binaries directly and add those to your path https://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/
     • Prebuilt binaries require 7zip to extract https://www.7-zip.org/
     • I installed x86_64-posix-seh

• Add the path of mingw-gcc's bin to your path (in my case /c/Program Files/mingw-w64/x86_64-8.1.0-posix-seh-rt_v6-rev0/mingw64/bin)

• (Optional, but recommended) Install the go vscode plugin, and click "Install All" when it prompts you to install missing golang components (godef, gopkgs, gopls)

Before Running

In order to get git working against private repos (which ori is in, for now) you have to configure git to go through authentication for github.

You can do this by putting a token in your .netrc, or you can route requests through https with:

git config --global [e-mail]:.insteadOf https://github.com/

Running ori from this repo

1. Set up a ori.json and patches directory in your target project.

   See above for the ori.json fields

   TODO: document the patches directory

   TODO: make a an example of an oribuild project + config (#10)

2. Building and Running

   cd oribuild
   go run . -c ../path/to/ori.json`
   

   The first time you run this, go will fetch and build all the dependencies in oribuild/go.mod

Catches

Add more here as you hit unexpected situations

• in client-web: yarn gulp gqlgen:generate needs to be run manually after any graphql change.

• node_modules are not monitored and assumed to be always stable. If you edit node_modules, you will need to save another file to refresh. Once separate builds are implemented (#8), you will have to restart the whole build agent, unless you specifically omit that node_module from the build cache

• ori exits with error 0xc0000139 on windows

  $ go run . -h
  exit status 0xc0000139
  

  This translates to STATUS_ENTRYPOINT_NOT_FOUND https://pkg.go.dev/golang.org/x/sys/windows

  This might mean you have the wrong mingw install version and windows can't find the entrypoint symbols for the libsass binary at runtime? not 100% sure but changing the mingw version to the one specified above fixes the issue.

Useful Snippets

# with mingw on your path

# Build entries
go run . -config=../ori.json

# Build an entry named "OWA Mail" from the entrypoints map, with codesplitting
# Note that this has to be loaded with a script type="module" entrypoint,
# since esbuild codesplitting forces esm modules
go run . -config=../ori.json -entry="OWA Mail" -split

# Generate a cpu profile for initial and incrmental builds (the traces directory must already exist)
go run . -config=../ori.json -entry="OWA Mail" -traceInitialBuild -traceIncrementalBuilds -cpuprofile=traces/cpu.pprof

# Analyse cpu profiles (constains overview of CPU time)
go tool pprof -http=localhost:8080 traces/cpu.pprof.initial*
go tool pprof -http=localhost:8080 traces/cpu.pprof.incremental*

# Analyse traces
go tool trace traces/trace.out.*

# cutting a new release, from root dir
# first, update the version numbers in js/packages/oribuild/package.json,
# and update the dependency versions to the same version number.
git commit -m "bump to 0.0.0-pre-alpha.4"
git tag v0.0.0-pre-alpha.4
git push
git push --tags
# this reads the version numbers from js/packages/oribuild/package.json
# and generates new packages.
./scripts/build-nonmac.sh
# this publishes to npm (you'll have to npm login separately)
./scripts/publish-nonmac.sh

FAQs

• Why not use the esbuild node API?

  In short, we tried it and it was slow. Initial build times were several minutes, compared to the 40-odd seconds we see with the go api because of all the time plugins spent waiting to run on the node main thread.

• Can I customize ori for my monorepo?

  For now, ori will remain extremly opinionared on what the monorepo shape must look like. As much as possible, we want to prefer convention over configuration.

  In the same vein, rather than implementing plugins or encouraging people to fork and make their own custom builds of ori, new functionality will be added to the same ori binaries as needed.

• Why is it called ori?

  ori was started by the Outlook Web team, and is short for OWA Rapid Innerloop.

  It can also be easily typed on a single row of a QWERTY keyboard without using your fifth fingers, which I value because I have ulnar neuropathy.

TODO: Populate this section as people ask more questions

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [e-mail] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.