oribuild

ori

Tools and plugins to run innerloop builds of typescript monorepos using esbuild.

Configuration & Usage

ori -h

  -blockFollowUp
        Wait for an initial build before running non-build tasks (implied by -traceInitialBuild)
  -config string
        Path to ori.json (default "./ori.json")
  -cpuprofile string
        Generate a cpu profile at the given path
  -entry string
        Use a given entry or entry group (from the values specified in ori.json)
  -gitRef string
        initial set of changed files to use when starting the typescript process (default "HEAD")
  -initialOnly
        Validate the initial build can complete, and exit with an error if it had issues
  -logLevel string
        log level (error|warning|info|debug)
  -noTui
        Disable the tui and print everything to stdio
  -nosplit
        Disable codesplitting. Allows for bundling without esm. (default true)
  -port int
        Port to run the http server on (default 3000)
  -snoop
        log every import as it happens. To help debug why a given module is imported.
  -trace string
        Generate an event trace at the given path
  -traceIncrementalBuilds
        Collect a pprof trace of each incremental build
  -traceInitialBuild
        Collect a pprof trace of the initial build
  -version
        Print the version and exit

ori.json fields

{
    // The path that esbuild should output to
    "outPath": "dist/esbuild",
    // Where to find resource.json files
    // 
    // TODO: document resource.json files
    //
    // Should be deprecated by #4
    "resourceRoots": ["packages", "shared"],
    // Where to find source files to watch
    // 
    // Should be deprecated by #4
    "watchSourceRoots": ["packages", "shared"],
    // Directories the webserver should serve in addition to serving
    // resources from resources.json and the built scripts + chunks
    "directServeDirectories": ["resources"],
    // constants to define, passed to esbuild's define property
    // see https://esbuild.github.io/api/#define
    "defineConstants": {
        "global": "self",
        "process.env.IS_WEBPACK": "false"
    },
    // A map of entry script names to the packages they are built from
    // (packages are read from tsconfig.paths.json, should be replaced
    // with packageNames in the workspaces we crawl)
    "entry": {
        "mailindex": "mail-index-package-name",
    },
    // Entrypoints to workers
    // 
    // Workers are built separately, see WorkerLoader for details
    //
    // Worker entrypoints not in this map will be built inline in the main,
    // build, at significant performance cost
    "workerRawEntries": {
        "pdfjsworker": "node_modules/pdfjs-dist/build/pdf.worker.js",
        "pdfjsworkermin": "node_modules/pdfjs-dist/build/pdf.worker.min.js",
        "owadataworker": "packages/libraries/worker/owa-data-worker-bootstrap/src/index.worker.ts"
    },
    // Human readable groups of entries from the above entries map
    // as well as custom extensions to defineConstants
    //
    // For use on the cli for common entry goups.
    "entryGroups": {
        "OWA Mail": {
            "entries": ["mailindex"],
            "defineConstants": {
                "OWA_BUILD_CONSTANTS.ENTRIES.mail": "true",
                "OWA_BUILD_CONSTANTS.BUILD_ALL": "false"
            }
        }
    }
}

Working on ori

Getting Dependencies

• install go 1.18 https://go.dev/doc/install

• If on windows, install mingw-gcc. This is to support building libsass on windows https://github.com/wellington/go-libsass/issues/37

  There are two ways to do this right now:

  1. Via Chocolatey:

     • Install Chocolatey
     • Install mingw from chocolatey choco install mingw. At time of writing this installs mingw 11.2.0.07112021

  2. Manually: do this only if you encounter issues with the version distributed by chocolatey:

     • Get a build from https://www.mingw-w64.org/downloads/#mingw-builds (the sourceforge link)
     • If the live installer fails, you can download the prebuilt binaries directly and add those to your path https://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/
     • Prebuilt binaries require 7zip to extract https://www.7-zip.org/
     • I installed x86_64-posix-seh

• Add the path of mingw-gcc's bin to your path (in my case /c/Program Files/mingw-w64/x86_64-8.1.0-posix-seh-rt_v6-rev0/mingw64/bin)

• (Optional, but recommended) Install the go vscode plugin, and click "Install All" when it prompts you to install missing golang components (godef, gopkgs, gopls)

Running ori from this repo

1. Set up a ori.json and patches directory in your target project.

   See above for the ori.json fields

   TODO: document the patches directory

   TODO: make a an example of an oribuild project + config (#10)

2. Building and Running

   cd oribuild
   go run . -c ../path/to/ori.json`
   

   The first time you run this, go will fetch and build all the dependencies in oribuild/go.mod

Catches

Add more here as you hit unexpected situations

• in client-web: yarn gulp gqlgen:generate needs to be run manually after any graphql change.

• node_modules are not monitored and assumed to be always stable. If you edit node_modules, you will need to save another file to refresh. Once separate builds are implemented (#8), you will have to restart the whole build agent, unless you specifically omit that node_module from the build cache

• ori exits with error 0xc0000139 on windows

  $ go run . -h
  exit status 0xc0000139
  

  This translates to STATUS_ENTRYPOINT_NOT_FOUND https://pkg.go.dev/golang.org/x/sys/windows

  This might mean you have the wrong mingw install version and windows can't find the entrypoint symbols for the libsass binary at runtime? not 100% sure but changing the mingw version to the one specified above fixes the issue.

Useful Snippets

# with mingw on your path

# Build entries
go run . -config=../ori.json

# Build an entry named "OWA Mail" from the entrypoints map, with codesplitting
# Note that this has to be loaded with a script type="module" entrypoint,
# since esbuild codesplitting forces esm modules
go run . -config=../ori.json -entry="OWA Mail" -split

# Generate a cpu profile for initial and incrmental builds (the traces directory must already exist)
go run . -config=../ori.json -entry="OWA Mail" -traceInitialBuild -traceIncrementalBuilds -cpuprofile=traces/cpu.pprof

# Analyse cpu profiles (constains overview of CPU time)
go tool pprof -http=localhost:8080 traces/cpu.pprof.initial*
go tool pprof -http=localhost:8080 traces/cpu.pprof.incremental*

# Analyse traces
go tool trace traces/trace.out.*

# cutting a new release, from root dir
# first, update the version numbers in dist/oribuild/package.json,
# and update the dependency versions to the same version number.
git commit -m "bump to 0.0.0-pre-alpha.4"
git tag v0.0.0-pre-alpha.4
git push
git push --tags
# this reads the version numbers from dist/oribuild/package.json
# and generates new packages.
./scripts/build-nonmac.sh
# this publishes to npm (you'll have to npm login separately)
./scripts/publish-nonmac.sh

FAQs

• Why not use the esbuild node API?

  In short, we tried it and it was slow. Initial build times were several minutes, compared to the 40-odd seconds we see with the go api because of all the time plugins spent waiting to run on the node main thread.

• Can I customize ori for my monorepo?

  For now, ori will remain extremly opinionared on what the monorepo shape must look like. As much as possible, we want to prefer convention over configuration.

  In the same vein, rather than implementing plugins or encouraging people to fork and make their own custom builds of ori, new functionality will be added to the same ori binaries as needed.

• Why is it called ori?

  ori was started by the Outlook Web team, and is short for OWA Rapid Innerloop.

  It can also be easily typed on a single row of a QWERTY keyboard without using your fifth fingers, which I value because I have ulnar neuropathy.

TODO: Populate this section as people ask more questions

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [e-mail] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.