nx-remotecache-azure

The future of nx-remotecache-*

Nrwl Nx just introduced us to Powerpack. It's the clear successor of the remote cache setup and officially supports custom caching solutions. This will mark the end of the nx-remotecache-* packages as custom caching solutions based on the filesystem won't work anymore starting with Nx 21.

Powerpack fills exactly the void that nx-remotecache-custom filled. Therefore I'm happy to give this topic back to the creators behind Nx. Thanks for the great ride – we reached over 114,000 weekly downloads on npm as I'm writing this 🥳

Feel free to read the Introduction to Powerpack by Jeff yourself. If you have any further questions checkout the pinned GitHub issue.

Cheers, Niklas 👋

nx-remotecache-azure

A task runner for @nrwl/nx that uses an Azure Blob Storage as a remote cache. This enables all team members and CI servers to share a single cache. The concept and benefits of computation caching are explained in the NX documentation.

This package was built with nx-remotecache-custom 🙌

Compatability

Nx	Remote Cache
>= 21	Deprecated
>= 20.0.0 < 21	>= 20.0.0
>= 19.0.0 < 20	>= 19.0.0 < 20
>= 18.0.0 < 19	>= 18.0.0 < 19
>= 17.0.0 < 18	>= 17.0.0 < 18
>= 16.9.0 < 17	>= 5.0.0 < 17
< 16.9.0	< 5.0.0

Setup

npm install --save-dev nx-remotecache-azure

Parameter	Description	Environment Variable / .env	nx.json
Connection String	Connect to an Azure Storage blob via a single URL.	NXCACHE_AZURE_CONNECTION_STRING	connectionString
Container SAS URL	Connect to an Azure Storage blob via a single container SAS URL.	NXCACHE_AZURE_SAS_URL	sasUrl
Account Name	Use together with Account Key for Azure Credentials Authentication	NXCACHE_AZURE_ACCOUNT_NAME	accountName
Account Key	Use together with Account Name for Azure Credentials Authentication	NXCACHE_AZURE_ACCOUNT_KEY	accountKey
Container	Required. Specify which container should be used for storing the cache.	NXCACHE_AZURE_CONTAINER	container
Azure URL	Optional. Can be used to overwrite Azure URL for local debugging.	NXCACHE_AZURE_URL	azureUrl
Azure AD Auth	Use together with Account Name for Azure Active Directory Authentication.	NXCACHE_AZURE_AD_AUTH	adAuth

{
  "tasksRunnerOptions": {
    "default": {
      "runner": "nx-remotecache-azure",
      "options": {
        // All of the azure specific options can also be inserted via environment variables! ⬆️
        "accountName": "MyAzureAccountName",
        "accountKey": "my-azure-account-key-11223-22..",
        "container": "test",
        "cacheableOperations": ["build", "test", "lint", "e2e"]
      }
    }
  }
}

Run it 🚀

Running tasks should now show the storage or retrieval from the remote cache:

------------------------------------------------------------------------
Built Angular Package
 - from: /Users/name/example-workspace/libs/example-lib
 - to:   /Users/name/example-workspace/dist/libs/example-lib
------------------------------------------------------------------------
------------------------------------------------------------------------
Stored output to remote cache: Azure Blob Storage
Hash: d3d2bea71ea0f3004304c5cc88cf91be50b02bb636ebbdfcc927626fd8edf1ae
------------------------------------------------------------------------

Advanced Configuration

Option	Environment Variable / .env	Description
name	NXCACHE_NAME	Set to provide task runner name for logging. Overrides name provided in implementation.
verbose		Set to receive full stack traces whenever errors occur. Best used for debugging. Default: false
silent		Set to mute success and info logs. Default: false
read	NXCACHE_READ	Set to enable / disable reading from the remote cache. Default: true
write	NXCACHE_WRITE	Set to enable / disable writing to the remote cache. Default: true
dotenv		Set to false to disable reading .env into process.env. Default: true
dotenvPath		Set to read .env files from a different folder.

"tasksRunnerOptions": {
  "default": {
    "options": {
      "name": "My Storage",
      "verbose": true,
      "silent": true
    }
  }
}

Cleanup

Over time, cached items accumulate on the remote storage, potentially leading to uncontrolled growth of storage usage. Failure to clean up outdated items can result in excessive costs. However, you can effectively manage storage usage and costs by configuring Azure storage with lifecycle rules.

To begin, ensure that the last access time of stored items is monitored. Access the Lifecycle management section within your storage account and enable monitoring as illustrated below:

Next, tailor the rules according to your requirements. For example, consider the following rules that will manage a container named nx-cache:

• Blobs not accessed for 30 days will be moved to cold storage. If accessed again, they will be returned to hot storage.
• Blobs not accessed at all for 90 days will be automatically deleted.

Configure these rules to strike a balance between cost optimization and access efficiency:

{
  "rules": [
    {
      "enabled": true,
      "name": "Cleanup",
      "type": "Lifecycle",
      "definition": {
        "actions": {
          "baseBlob": {
            "enableAutoTierToHotFromCool": true,
            "tierToCool": {
              "daysAfterLastAccessTimeGreaterThan": 30
            },
            "delete": {
              "daysAfterLastAccessTimeGreaterThan": 90
            }
          }
        },
        "filters": {
          "blobTypes": ["blockBlob"],
          "prefixMatch": ["nx-cache"]
        }
      }
    }
  ]
}

More info on Configure a lifecycle management policy

All Custom Runners

Runner	Storage
nx-remotecache-azure	Azure Blob Storage
nx-remotecache-minio	MinIO Storage