@parity/parachains-integration-tests

Parachains Integration Tests ✅

Since the arrival of XCMP-Lite, communication between different consensus systems became a reality in the Polkadot ecosystem. Parachains Integration Tests is a tool that was created with the ambition of easing testing interactions between Substrate based blockchains.

This tool allows you to develop tests rapidly describing them in a YAML file. Behind the scenes, the YAML files are converted to Mocha tests with Chai assertions.

It can work alongside with Zombienet or you can run your tests against the testnet of your choice.

Under the ./examples folder, this repository contains integration tests for the System Parachains. You can take them as examples of how to write tests with this tool.

Set Up

Requirements

• node v18 or higher.

Versioning

• v2.0.0 contains BREAKING CHANGES. Tests based on ^1.0.0 will stop working properly from v2.0.0 onwards. Check the GitHub release for more info and how to migrate the tests.

• v2.3.0 contains BREAKING CHANGES. Polkadot Launch is not supported anymore.

Installation

It can be installed to be run in two different ways:

• Installing the npm package globally

  yarn global add ts-node
  
  yarn global add @parity/parachains-integration-tests
  

• From the repository

  yarn
  

How to use

The tool implements a simple CLI.

parachains-integration-tests -m <mode> -c <path> -t <path> -to <millisecons> -el <milliseconds> -qd <milliseconds>

• -e, --env:
  • prod: for compiled TypeScript to Javascript (default)
  • dev: for development environment in TypeScript
• -m, --mode:
  • checker: checks the format integtity of the yaml test files
  • test: for running your tests (the checker will be autmatically run prior to the tests)
  • zombienet: only deploy a Zombienet network
  • zombienet-test: deploy a Zombienet testnet and run your tests against it
• -c, --config: path to the Zombienet or Polkadot Launch config file.
• -t, --test: path to the tests folder or to a single test yaml file. All files under the path with a yml extension will be run. To choose the order, is necessary to add an index in front of the file name. E.g: 0_my_test.yml, 1_my_other_test.yml
• -to, --timeout: overrides the default Mocha tests timeout set to 300000
• -el, --event-listener-timeout: overrides the default event listener timeout set to 40000
• -ad, --action-delay: delay before state queries, rpc calls and extrinsics. Overrides the default delay set to 40000. Some delay is necessary to make sure the state is already updated. In the case of extrinsics, it is also necessary until ID hashes are available in XCM v3. Without an identifier, it is not possible to distinguish what XCM message event was triggered as a result of a specific extrinsic from another chain/context. For this reason, it is necessary to add a big delay between XCM messages, to avoid interferences from other unrelated events.
• -cl, --chain-logs: path to the log file to redirect stdout and stderr from the testnets deployment tool, either Zombienet or Polkadot Launch.
• -tl, --test-logs: path to the log file to redirect stdout and stderr from this testing tool.

Examples:

• NPM package

  • Check the integrity of the tests format

    parachains-integration-tests -m checker -t <tests_path>
    

  • Run tests using other testnet

    parachains-integration-tests -m test -t <tests_path>
    

  • Only deploy a testnet with Zombienet

    parachains-integration-tests -m zombienet -c <zombienet_config_path>
    

  • Run tests using Zombienet as testnet

    parachains-integration-tests -m zombienet-test -t <tests_path> -c <zombienet_config_path>
    

• From the repository

  • Check the integrity of the tests format

    yarn checker -t <tests_path>
    

  • Run tests using other testnet

    yarn test -t <tests_path>
    

  • Only deploy a testnet with Zombienet

    yarn zombienet -c <zombienet_config_path>
    

  • Run tests using as testnet

    yarn zombienet-test -t <tests_path> -c <zombienet_config_path>
    

Table of Contents

• YAML Schema
  • Settings
  • Tests
  • Hook & It
  • Action
  • Extrinsic
  • Event
  • Query
  • Rpc
  • Assert
  • Custom

YAML Schema

It is formed by two main sections: settings and tests.

settings:
  # Declaration of the chains the tests should connect to
  chains: # { [key: string]: Chain }
  # Arbitrary declaration of constants to be used across the tests
  variables: # { [key: string]: any }
  # Calls that want to be encoded to be used in the tests
  decodedCalls: # { [key: string]: Call }

tests: # Describe[]

export interface TestsConfig {
  settings: Settings;
  tests: Describe[];
}

Settings

• chains: connection values for all the different chains we want to connect to. Defining wsPort should be enough unless you want to override the default ws URL (ws://localhost).

• variables: section that allows you to define your own variables following the schema that better suits your test's logic.

• decodedCalls: declaration of the different calls you want to calculate their encoded call hex value or use them inside a batch call. Each result is stored in a variable that will become available in the rest of the file ONLY after its declaration. The way to access those variables is appending a $ symbol to the defined decodedCalls key. For instance, in the following example, the encoded call result for my_call_id will be accessible from $my_call_id. If you want to use the call inside a batch call, the attribute encode: false should be added. That attribute indicates if the call should be encoded or if it should be treated as Submittable Polkadot JS object.

Example:

settings: # Settings
  chains:
    my_chain_id: &relay_chain # a Relay Chain, for instance
      wsPort: 9966
      ws: ws://my-custom-url
    my_other_chain_id: &parachain # a Parachain, for instance
      wsPort: 9988
      # It is also possible to add the variables that you consider
      # are useful and related to the chain
      for_example_paraId: &paraId 2000

  variables:
    my_variable: &my_variable 0x0011
    my_arbitrary_schema: &my_schema
      object:
        a: 1
        b: 2

  decodedCalls:
    my_call_id:
        chain: *relay_chain
        pallet: system
        call: remark
        args: [ *my_variable ]
    my_call_id:
        chain: *relay_chain
        encode: false # Indicates the call will not be encoded and used instead as Submittable
        pallet: system
        call: remark
        args: [ *my_variable ]

interface Settings {
  chains: { [key: string]: Chain };
  variables: { [key: string]: any };
  decodedCalls: { [key: string]: Call };
}

interface Chain {
  wsPort: number;
  ws?: string; // if 'undefined', it fallback to the default value -> ws://localhost
  paraId: number; // parachain id
}

interface Call {
  chain: Chain;
  sudo?: boolean; // if 'true', the call will be wrapped with 'sudo.sudo()'
  pallet: string;
  call: string;
  args: any[];
}

Tests

Tests are formed by an array of Describe interfaces. Tests can be nested through the describes attribute.

Example:

tests: # Describe[]
  - name: My Describe
    before: # Before[]
      - name: 'before' description to console log
        actions: [...]
    beforeEach: ... # BeforeEach[]
    after: ... # After[]
    afterEach: ... # AfterEach[]
    its: # It[]
      - name: It should do something
        actions: [...]
    describes: # Describe[]
      - name: My nested Describe

  - name: My other Describe
    its: [...] # It[]

Interfaces:

interface Describe {
  name: string;
  before?: Before[];
  beforeEach?: BeforeEach[];
  after?: After[];
  afterEach?: AfterEach[];
  its: It[];
  describes?: Describe[]; // It is possible to nest Describes
}

Hook & It

Both have a similar interface. They are formed by a name for descriptions and by the actions attribute.

The available hooks are: before, beforeEach, after and afterEach

Example:

tests: # Describe[]
  - name: My Describe
    before: # Before[]
      - name: 'before' description to console log
        actions: [...] # Action[]
      - name: another description for a 'before'
        actions: [...] # Action[]
    its: # It[]
      - name: It should do something
        actions: [...] # Action[]
      - name: It should not do something
        actions: [...] # Action[]
    ...

Interfaces:

type Hook = Before | BeforeEach | After | AfterEach

 // Same for BeforeEach, After, AfterEach
interface Before {
  name?: string; // optional description
  actions: Action[];
}

interface It {
  name: string;
  actions: Action[];
}

Action

There are five available actions types that can be performed inside a Hook or an It: extrinsics, queries, rpcs, asserts and customs. The order they are executed depends on the order they are defined in the Action array. Since actions is an array, multiple actions of the same type can be declared.

Example:

tests: # Describe[]
  - name: My Describe
    before: # Before[]
      - name: 'before' description to console log
        actions: # Action[]
          - extrinsics: [...] # Extrinsic[]
          - queries: [...] # Query[]
          - ...
    its: # It[]
      - name: It should do something
        actions: # Action[]
          - extrinsics: [...] # Extrinsic[]
          - rpcs: [...] # RPC[]
          - asserts: [...] # Assert[]
          - customs: [...] # Custom[]
          - queries: [...] # Query []
          - asserts: [...] # Assert[]
    ...

Interfaces:

export type ExtrinsicAction = {
  extrinsics: Extrinsic[];
}

export type QueryAction = {
  queries: { [key: string]: Query };
}

export type RpcAction = {
  rpcs: { [key: string]: Rpc };
}

export type AsserAction = {
  asserts: { [key: string]: AssertOrCustom };
}

export type CustomAction = {
  customs: Custom[];
}

export type Action = ExtrinsicAction | QueryAction | AsserAction | RpcAction | CustomAction;

Extrinsic

Extends the Call interface adding two new attributes: signer (indispensable) and events (optional). A Extrinsic by itself will not perform any chai assertion. Assertions are build based on the events that the extrinsic is expected to trigger. Each event defined under the events attribute will build and perform its corresponding chai assertion.

Example:

settings:
  chains:
    relay_chain: &relay_chain
      wsPort: 9900
    parachain: &parachain
      wsPort: 9910
      paraId: &id 2000

  variables:
    common:
      require_weight_at_most: &weight_at_most 1000000000
    relay_chain:
      signer: &signer //Alice
      parachain_destination: &dest { v1: { 0, interior: { x1: { parachain: *id }}}}
      my_variable: &my_variable 0x0011

  decodedCalls:
    force_create_asset:
      chain: *parachain
      pallet: assets
      call: forceCreate
      args: [
        1, # assetId
        { # owner
          Id: HNZata7iMYWmk5RvZRTiAsSDhV8366zq2YGb3tLH5Upf74F
        },
        true, # isSufficient
        1000 # minBalance
      ]
    to_be_batched:
      chain: *relay_chain
      encode: false # Indicates the call will not be encoded and used instead as Submittable instead
      pallet: system
      call: remark
      args: [ *my_variable ]

tests: # Describe[]
  - name: My Describe
    its: # It[]
      - name: It should do something
        actions: # Action[]
          - extrinsics: # Extrinsic[]
            - chain: *relay_chain # Chain
              signer: *signer
              sudo: true
              pallet: xcmPallet
              call: send
              args: [
                *ap_dest, # destination
                {
                  v2: [ # message
                    {
                      Transact: {
                        originType: Superuser,
                        requireWeightAtMost: *weight_at_most,
                        call: $force_create_asset # encoded call hex
                      }
                    }
                  ]
                }
              ]
              events: [...]
             - chain: *relay_chain # Chain
                signer: *signer
                pallet: utility
                call: batchAll
                args: [
                  [$to_be_batched]
                ]
    ...

Interfaces:

interface Call {
  encode?: boolean; // Indicates if the Call should be encoded
  chain: Chain;
  sudo?: boolean; // if 'true', the call will be wrapped with 'sudo.sudo()'
  pallet: string;
  call: string;
  args: any[];
}

interface Extrinsic extends Call {
  signer: string;
  delay?: number; // Overrides the default action delay
  events: Event[];
}

Event

If the chain attribute is not defined, it means the event is expected to happen in the same chain context where the extrinsic was dispatched and as a result of it. Otherwise, the chain attribute referring to another context must be defined.

Default event listener timeout can be overridden by the timeout attribute.

There are two different and compatible ways (you can apply both at the same time) of checking if an event returns the expected values: comparing the "whole" result, or comparing by atrributes.

• result: When the event is defined in the runtime as a Tuple, the event result is returned as an ordered array of its elements. In case it is defined as a Struct, the event result is returned as an object.

  E.g:

  • Tuple

    Sent(MultiLocation, MultiLocation, Xcm<()>) // Event from 'pallet_xcm'
    

    result: [..., ..., ...] # order and indexes matters
    

  • Struct

    Transfer { from: T::AccountId, to: T::AccountId, amount: T::Balance } // Event from 'pallet_balances'
    

    result: { from: ..., to: ..., amount: ... }
    

  strict is set to false by default. It allows to check if result is just contained in the event result instead of expecting a perfect match. For a Tuple it means that the provided array is a subset (array items exist & order and index matter) of the event result array. For a Struct it means that the provided object is also a subset (keys/values exist) of the event result object.

• attributes: Event's attributes must be identified either by type, key or both. When the event is defined in the runtime as a Tuple, the only way to identify the attributes is by their type. Be aware that in that case the order you declare the attributes in the test matters. That is because there could be multiple attributes with the same type in the Tuple. However, if the event is defined as a Struct, its attributes can be also identified by their key.

  By setting isRange: true you are letting know to the tool that the expected value should be within the range defined in the value attribute. The expected value's format is: <lower_limit>..<upper_limit>.

  In addition, a threshold attribute can be used to define an upper and lower limit the value attribute should be within. It is expecting a percentage value. E.g: threshold: [10, 20] means that the value can be 10% lower and 20% higher. It can be used either for an attribute value or a event result. For assessing a result treshold should be an object where its keys are the fields to be assessed from result. Example below checking Weight values ref_time and proof_size.

  For obvious reason, isRange and threshold can not be used at the same time. These features are especially useful when checking variables that often change such as Weights.

  There is a special treatment for the attribute type XcmV2TraitsOutcome. Only in that case, xcmOutput and value can be set to replace an event result with the format { outcome: { <xcmOutput>: <value> }}. Valid xcmOutput are Complete, Incomplete and Error.

Example:

settings:
  chains:
    relay_chain: &relay_chain
      wsPort: 9900
    parachain: &parachain
      wsPort: 9910
  variables:
    ...
  encodedCalls:
    my_encoded_call:
      ...

tests: # Describe[]
  - name: My Describe
    its: # It[]
      - name: It should do something
        actions: # Action[]
          - extrinsics: # Extrinsic[]
           - chain: *relay_chain
              signer: *signer
              sudo: true
              pallet: xcmPallet
              call: send
              args: [
                *dest, # destination
                {
                  v2: [ #message
                    {
                      Transact: {
                        originType: Superuser,
                        requireWeightAtMost: *weight_at_most,
                        call: $my_encoded_call
                      }
                    }
                  ]
                }
              ]
              events: # Event[]
                - name: sudo.Sudid
                  attributes:
                    - type: Result<Null, SpRuntimeDispatchError>
                      key: sudoResult
                      value: Ok
                - name: xcmPallet.Sent
                - name: dmpQueue.ExecutedDownward
                  chain: *collectives_parachain
                  threshold: { refTime: [10, 10], proofSize: [10, 10] }
                  result: {
                    outcome: { Complete: { refTime: '3,000,000,000', proofSize: '1,000,000' }}
                  }
                - name: polkadotXcm.Sent
                  chain: *parachain
                - name: ump.ExecutedUpward
                  timeout: 40000
                  attributes: # Attribute[]
                    - type: XcmV2TraitsOutcome
                      xcmOutcome: Complete
                      isRange: true
                      value: 4,000,000..5,000,000 # value should be within 4,000,000..5,000,000
    ...

Interfaces:

interface Event {
  chain: Chain;
  name: string;
  remote: boolean; // indicates the event is considered as a remote (different chain context)
  timeout?: number; // overrides de default event listener timeout
  result?: object; // Either {..} or [..]
  strict: boolean;
  attributes?: Attribute[];
  threshold?: any;
}

interface Attribute {
  type?: string;
  key?: string;
  isRange?: boolean; // indicates the value is a range
  threshold: [number, number]; // defines the percentages a value can vary
  value?: any;
  xcmOutcome?: XcmOutcome; // only for 'XcmV2TraitsOutcome' type
}

export enum XcmOutcome {
  Complete = 'Complete',
  Incomplete = 'Incomplete',
  Error = 'Error'
}

Query

Query the chain state. The result of the query will be stored in a new variable based on the key name of the Query. The variable naming follows the same format of decodedCalls. Therefore, for the following example, the result of the query is stored in: $balance_sender_before. The variable becomes available in the rest of the file ONLY after its declaration.

Example:

settings:
  chains:
    relay_chain: &relay_chain
      wsPort: 9900

  variables:
    ...
  encodedCalls:
    ...
tests: # Describe[]
  - name: My Describe
    before: # Before[]
      - name: Get the balance of an account
        actions: # Action[]
          - queries: # { key: Query }
              balance_sender_before:
                chain: *relay_chain
                pallet: system
                call: account
                args: [
                  HNZata7iMYWmk5RvZRTiAsSDhV8366zq2YGb3tLH5Upf74F
                ]
    its: [...]

Interfaces:

interface Query {
  chain: Chain;
  delay?: number;
  pallet: string;
  call: string;
  args: any[];
}

Rpc

RPC call to the chain's node. Same approach as Query. For the following example, the result of the RPC call will be stored in $block.

Example:

settings:
  chains:
    relay_chain: &relay_chain
      wsPort: 9900

  variables:
    ...
  encodedCalls:
    ...
tests: # Describe[]
  - name: My Describe
    before: # Before[]
      - name: Get the last block
        actions: # Action[]
          - rpcs: # { key: Rpc }
              block:
                chain: *relay_chain
                method: chain
                call: getBlock
                args: []
                events: []
    its: [...]

Interfaces:

interface Rpc {
  chain: Chain;
  delay?: number;
  method: string;
  call: string;
  args: any[];
  events?: Event[];
}

Assert

Unlike Query and Rpc where their keys can be arbitrarily chosen to generate a new variable, AssertOrCustom keys can only be chosen from a list of built-in asserts.

• equal: it has a single attribute args which is expecting an array of two values to be deepEqual() compared.
• isNone: the argument is null.
  • ./src/asserts/isNone.ts
• isSome: the argument is not null.
  • ./src/asserts/isSome.ts
• balanceDecreased: compares balances queried with system.account. If amount and fees are not included as arguments, it will just check that after is lower than before
  • ./src/asserts/balanceDecreased.ts
• balanceIncreased: compares balances queried with system.account. If amount and fees(only for XCM messages) are not included as arguments, it will just check that after is bigger than before
  • ./src/asserts/balanceIncreased.ts
• assetsDecreased: compares balances queried with assets.account. If amount and fees are not included as arguments, it will just check that after is lower than before
  • ./src/asserts/assetsDecreased.ts
• assetsIncreased: compares balances queried with assets.account. If amount and fees(only for XCM messages) are not included as arguments, it will just check that after is bigger than before
  • ./src/asserts/assetsIncreased.ts
• custom: assertion cases can be endless, therefore they are difficult to standardize. custom solves that issue providing the path argument. Its value should point to a file where the desired asserts are performed based on the provided args. It can not be any kind of file though, and it should export a specific function signature. To learn more about this files see Custom. Notice that you will have to include a tsconfig.json file with typeRoots and types attributes pointing to your types in case of adding paths to a typescript file.

These methods are extensible opening a PR to include them:

• Add a new assertion key to REGISTERED_ASSERTIONS in ./src/constants.ts
• Add a new assertion file under ./src/asserts. The filename needs to match with the previously registered assertion key.

Example:

settings:
  chains:
    relay_chain: &relay_chain
      wsPort: 9900

  variables:
    relay_chain:
      sender: &sender HNZata7iMYWmk5RvZRTiAsSDhV8366zq2YGb3tLH5Upf74F
    ...
  encodedCalls:
    ...
tests: # Describe[]
  - name: My Describe
    before: # Before[]
      - name: Get the balance of an account before an event
        actions:
          - queries:
              balance_sender_before:
                chain: *relay_chain
                pallet: system
                call: account
                args: [
                  *sender
                ]
    after: # After[]
      - name: Get the balance of an account after an event
        actions:
          - queries:
              balance_sender_after:
                chain: *relay_chain
                pallet: system
                call: account
                args: [
                  *sender
                ]
    its: # It[]
      - name: Something happens here than modifies the balance
        actions: [...]
      - name: Should reduce the balance of the sender
        actions: # Action[]
          - asserts: # { [key: string]: AssertOrCustom }
              customs:
                path: ./asserts/checkSenderBalances.ts
                args:
                  {
                    balances: {
                      before: $balance_rc_sender_before,
                      after: $balance_rc_sender_after,
                    },
                    amount: *amount,
                  }
              equal:
                args: [true, true]

Interfaces:

interface Assert {
  args: any[];
}

interface Custom {
  path: string;
  args: any;
}

type AssertOrCustom = Assert | Custom;

Custom

This Action type enables the possibility of referring to your own files to perform those actions that a constrained YAML schema can not provide. The file must export a very specific function signature that the tool is expecting to import: async (context, ...args)

• context: corresponds to the test's this object. All user created variables (in encodedCalls, queries and rpcs) are stored and accessible from the this.variables key. In a similar way, context can be used to stored new variables that will become available in the rest of the tests.
• args: the arguments used as input for your custom file function.

The following example shows how to use a custom action to perform an assertion, but there are no limitations about what to achieve.

Example:

settings:
  ...
tests: # Describe[]
  - name: My Describe
    before: # I declare $coin_symbol
    its: # It[]
      ...
      - name: My custom action should do something
        actions: # Action[]
          customs: # Custom[]
            - path: ./queryExternalOracle.ts
              args: [
                {
                  url: https://www.my-oracle.com/price/
                }
              ]
              events: []
          asserts:
            equal: [$dot_price, 30]

// queryExternalOracle.ts

const myCustomFunction = async (context, ...args) => {
  const { url } = args[0]

  let coinSymbol = context.variables.$coin_symbol

  let price = myApi.get(url + coinSymbol)

  // Save the result in context (this) variables
  // to make it available for the rests of the tests
  context.variables['$dot_price'] = price
}

export default myCustomFunction

Interfaces:

interface Custom {
  path: string;
  args: any[];
  events?: Event[];
}

Get Help

Open an issue if you have problems.

Contributions

PRs and contributions are welcome :)