apify-cli

Apify command-line interface (Apify CLI)

Apify command-line interface (Apify CLI) helps you create, develop, build and run Apify actors, and manage the Apify cloud platform from any computer.

Apify actors are cloud programs that can perform arbitrary web scraping, automation or data processing job. They accept input, perform their job and generate output. While you can develop actors in an online IDE directly in the Apify web application, for complex projects it is more convenient to develop actors locally on your computer using Apify SDK and only push the actors to the Apify cloud during deployment. This is where the Apify CLI comes in.

Note that actors running on the Apify platform are executed in Docker containers, so with an appropriate Dockerfile you can build your actors in any programming language. However, we recommend using JavaScript / Node.js, for which we provide most libraries and support.

Installation

First, make sure you have Node.js version 10 or higher with NPM installed on your computer:

node --version
npm --version

Install or upgrade Apify CLI by running:

npm -g install apify-cli

If you receive an EACCES error, you might need to run the commnad as root:

sudo npm -g install apify-cli

Alternativaly, you can use Node Version Manager (nvm) and install Apify CLI only into a selected user-level Node version without requiring root privileges:

nvm install 12
nvm use 12
npm -g install apify-cli

Finally, verify that Apify CLI was installed correctly by running:

apify --version

which should print something like:

apify-cli/0.5.3 darwin-x64 node-v12.16.1

Basic usage

The following examples demonstrate the basic usage of Apify CLI.

Create a new actor from scratch

apify create my-hello-world

First, you will be prompted to select a template with the boilerplate for the actor, to help you get started quickly. The command will create a directory called my-hello-world that contains a Node.js project for the actor and a few configuration files.

Create a new actor from existing project

cd ./my/awesome/project
apify init

This command will only set up local actor development environment in an existing directory, i.e. it will create the apify.json file and apify_storage directory.

Before you can run your project using apify run, you have to set up the right start command in package.json under scripts.start. For example:

{
    ...
    "scripts": {
        "start": "node your_main_file.js",
    },
    ...
}

You can find more information about by running apify help run.

Run the actor locally

cd my-hello-world
apify run

This command runs the actor on your local machine. Now's your chance to develop the logic - or magic :smirk:

Login with your Apify account

apify login

Before you can interact with the Apify cloud, you need to create an Apify account and log in to it using the above command. You will be prompted for your Apify API token. Note that the command will store the API token and other sensitive information to ~/.apify.

Push the actor to the Apify cloud

apify push

This command uploads your project to the Apify cloud and builds an actor from it.

Run an actor on the Apify cloud

apify call

Runs the actor corresponding to the current directory on the Apify platform.

This command can also be used to run other actors, for example:

apify call apify/hello-world

So what's in this apify.json file?

This file associates your local development project with an actor on the Apify platform. It contains information such as actor name, version, build tag and environment variables. Make sure you commit this file to the Git repository.

For example, apify.json file can look as follows:

{
    "name": "dataset-to-mysql",
    "version": "0.1",
    "buildTag": "latest",
    "env": {
      "MYSQL_USER": "my_username",
      "MYSQL_PASSWORD": "@mySecretPassword"
    },
    "template": "basic"
}

Environment variables

There are two options how you can set up environment variables for actors.

1. Set up environment variables in apify.json

All keys from env will be set as environment variables into Apify platform after you push actor to Apify. Current values on Apify will be overridden.

{
    "name": "dataset-to-mysql",
    "version": "0.1",
    "buildTag": "latest",
    "env": {
      "MYSQL_USER": "my_username",
      "MYSQL_PASSWORD": "@mySecretPassword"
    },
    "template": "basic"
}

2. Set up environment variables in Apify app

In Apify app select your actor, you can set up variables into Source tab. After setting up variables in the app, set up env to null apify.json. Otherwise, variables from apify.json will override variables in the app.

{
    "name": "dataset-to-mysql",
    "version": "0.1",
    "buildTag": "latest",
    "env": null,
    "template": "basic"
}

How to set secret environment variables in apify.json

CLI provides commands to manage secrets environment variables. Secrets are stored to the ~/.apify directory. Adds a new secret using command:

apify secrets:add mySecretPassword pwd1234

After adding a new secret you can use the secret in apify.json

{
    "name": "dataset-to-mysql",
    ...
    "env": {
      "MYSQL_PASSWORD": "@mySecretPassword"
    },
    ...
}

Need help?

To see all CLI commands simply run:

apify help

To get information about a specific command run:

apify help COMMAND

Still haven't found what you were looking for? Please go to Apify Help center or contact us.

Command reference

This section contains printouts of apify help for all commands.

• apify call [ACTID]
• apify create [ACTORNAME]
• apify info
• apify init [ACTORNAME]
• apify login
• apify logout
• apify push [ACTORID]
• apify run
• apify secrets
• apify secrets:add NAME VALUE
• apify secrets:rm NAME
• apify vis [PATH]

apify call [ACTID]

Runs a specific actor remotely on the Apify cloud platform.

USAGE
  $ apify call [ACTID]

ARGUMENTS
  ACTID  Name or ID of the actor to run (e.g. "apify/hello-world" or "E2jjCZBezvAZnX8Rb"). If not provided, the command
         runs the remote actor specified in the "apify.json" file.

OPTIONS
  -b, --build=build                      Tag or number of the build to run (e.g. "latest" or "1.2.34").
  -m, --memory=memory                    Amount of memory allocated for the actor run, in megabytes.
  -t, --timeout=timeout                  Timeout for the actor run in seconds. Zero value means there is no timeout.
  -w, --wait-for-finish=wait-for-finish  Seconds for waiting to run to finish, if no value passed, it waits forever.

DESCRIPTION
  The actor is run under your current Apify account, therefore you need to be logged in by calling "apify login". It
  takes input for the actor from the default local key-value store by default.

See code: src/commands/call.js

apify create [ACTORNAME]

Creates a new actor project directory from a selected boilerplate template.

USAGE
  $ apify create [ACTORNAME]

ARGUMENTS
  ACTORNAME  Name of the actor and its directory

OPTIONS
  -t, --template=template  Template for the actor. If not provided, the command will prompt for it.Visit
                           https://github.com/apifytech/actor-templates/raw/master/templates/manifest.json to find
                           available template names.

See code: src/commands/create.js

apify info

Displays information about the currently active Apify account.

USAGE
  $ apify info

DESCRIPTION
  The information is printed to the console.

See code: src/commands/info.js

apify init [ACTORNAME]

Initializes a new actor project in an existing directory.

USAGE
  $ apify init [ACTORNAME]

ARGUMENTS
  ACTORNAME  Name of the actor. If not provided, you will be prompted for it.

DESCRIPTION
  The command only creates the "apify.json" file and the "apify_storage" directory in the current directory, but will
  not touch anything else.

  WARNING: The directory at "apify_storage" will be overwritten if it already exists.

See code: src/commands/init.js

apify login

Logs in to your Apify account using a provided API token.

USAGE
  $ apify login

OPTIONS
  -t, --token=token  [Optional] Apify API token

DESCRIPTION
  The API token and other account information is stored in the ~/.apify directory, from where it is read by all other
  "apify" commands. To log out, call "apify logout".

See code: src/commands/login.js

apify logout

Logs out of your Apify account.

USAGE
  $ apify logout

DESCRIPTION
  The command deletes the API token and all other account information stored in the ~/.apify directory. To log in again,
  call "apify login".

See code: src/commands/logout.js

apify push [ACTORID]

Uploads the actor to the Apify platform and builds it there.

USAGE
  $ apify push [ACTORID]

ARGUMENTS
  ACTORID  ID of an existing actor on the Apify platform where the files will be pushed. If not provided, the command
           will create or modify the actor with the name specified in "apify.json" file.

OPTIONS
  -b, --build-tag=build-tag              Build tag to be applied to the successful actor build. By default, it is taken
                                         from the "apify.json" file

  -v, --version=version                  Actor version number to which the files should be pushed. By default, it is
                                         taken from the "apify.json" file.

  -w, --wait-for-finish=wait-for-finish  Seconds for waiting to build to finish, if no value passed, it waits forever.

  --version-number=version-number        DEPRECATED: Use flag version instead. Actor version number to which the files
                                         should be pushed. By default, it is taken from the "apify.json" file.

DESCRIPTION
  The actor settings are read from the "apify.json" file in the current directory, but they can be overridden using
  command-line options.
  NOTE: If the source files are smaller than 3 MB then they are uploaded as
  "Multiple source files", otherwise they are uploaded as "Zip file".

  WARNING: If the target actor already exists in your Apify account, it will be overwritten!

See code: src/commands/push.js

apify run

Runs the actor locally in the current directory by executing "npm start".

USAGE
  $ apify run

OPTIONS
  -p, --purge              Shortcut that combines the --purge-queue, --purge-dataset and --purge-key-value-store
                           options.

  --purge-dataset          Deletes the local directory containing the default dataset before the run starts.

  --purge-key-value-store  Deletes all records from the default key-value store in the local directory before the run
                           starts, except for the "INPUT" key.

  --purge-queue            Deletes the local directory containing the default request queue before the run starts.

DESCRIPTION
  It sets various APIFY_XYZ environment variables in order to provide a working execution environment for the actor. For
  example, this causes the actor input, as well as all other data in key-value stores, datasets or request queues to be
  stored in the "apify_storage" directory, rather than on the Apify platform.

  NOTE: You can override the default behaviour of command overriding npm start script value in a package.json file. You
  can set up your own main file or environment variables by changing it.

See code: src/commands/run.js

apify secrets

Manages secret values for actor environment variables.

USAGE
  $ apify secrets

DESCRIPTION
  Example:
  $ apify secrets:add mySecret TopSecretValue123

  Now the "mySecret" value can be used in an environment variable defined in "apify.json" file by adding the "@" prefix:

  {
     "name": "my_actor",
     "env": { "SECRET_ENV_VAR": "@mySecret" },
     "version": "0.1
  }

  When the actor is pushed to Apify cloud, the "SECRET_ENV_VAR" and its value is stored as a secret environment variable
  of the actor.

See code: src/commands/secrets/index.js

apify secrets:add NAME VALUE

Adds a new secret value.

USAGE
  $ apify secrets:add NAME VALUE

ARGUMENTS
  NAME   Name of the secret
  VALUE  Value of the secret

DESCRIPTION
  The secrets are stored to a file at ~/.apify

See code: src/commands/secrets/add.js

apify secrets:rm NAME

Removes the secret.

USAGE
  $ apify secrets:rm NAME

ARGUMENTS
  NAME  Name of the secret

See code: src/commands/secrets/rm.js

apify vis [PATH]

Validates INPUT_SCHEMA.json file and prints errors found.

USAGE
  $ apify vis [PATH]

ARGUMENTS
  PATH  Optional path to your INPUT_SCHEMA.json file. If not provided ./INPUT_SCHEMA.json is used.

See code: src/commands/vis.js

Changelog

0.6.1 / 2020-05-18

• BREAKING: Templates are now fully decoupled from this project and the templates repository serves as the single source of truth. Some templates were replaced and others were renamed to better clarify their purpose.

• BREAKING: Providing an invalid template in apify.json no longer throws, but rather silently uses a reasonable default configuration. This is to support regular changes to templates without breaking older versions of the CLI.

  0.5.3 / 2020-03-03

• Moved templates to separate repository

• Fixed: creating apify_storage in root folder after apify create command

  0.5.2 / 2020-01-22

• Added bot(dependabot.com) to check latest Apify SDK version in all templates

• Updated apify package in all templates

• Updated npm packages and fixed all npm audit issues

  0.5.1 / 2019-12-19

• Added warning about outdated node.js version

• Fixed infinite push, when the previous one was interrupted

• Fixed calling public actors with apify call

• apify init create empty INPUT.json file

  0.5.0 / 2019-11-27

• Drop support for node 8 and 9

• Fix: Pass the --max-http-header-size=80000 to the nodeJs process

  0.4.1 / 2019-10-02

• New actor template for Apify projects, you can create it with apify create --template apify_project

• apify vis - Using improved schema validator

  0.4.0 / 2019-09-23

• Breaking Change - apify push: Pushes source code as a "Multiple source files" in case source code is less that 3 MB

  0.3.12 / 2019-09-18

  Bug fixes:

• apify create: Added validation for actor name

• apify init skips creation of apify.json if already exists

• apify run -p runs actor, if apify_storage doesn't exist

• Updated packages

• Additional minor fixes

  0.3.11 / 2019-07-26

• Updated packages

• Updated Cheerio Crawler template

• Updated Apify package version in all templates

  0.3.10 / 2019-06-03

• Updated packages

  0.3.9 / 2019-05-15

• Improved the templates and texts

  0.3.8 / 2019-03-29

• Updated all templates regarding the last version of apify SDK.

  0.3.7 / 2019-03-18

• Fixed templates to use Apify.getInput(), replaced deprecated function and options, added debug fields, added .idea to .gitignore

• Updated packages

• Fixed bug: Users without username can use push/call command

  0.3.6 / 2019-01-29

• Added command apify vis that validates actor input schema.

  0.3.5 / 2019-01-25

• Upgraded to apify@0.11 in templates

  0.3.3 / 2018-12-12

• Omitted CMD command in all templates Dockerfile.

  0.3.2 / 2018-12-05

• Updated apify-client package. It fixed bug, when user can not push actor, whe he changed version in apify.json.

  0.3.1 / 2018-11-29

• :tada: New commands to manage secret environment variables: apify secrets:add, apify secrets:rm.

• New documentation how to set environment variable in apify.json, see documentation.

• BREAKING CHANGES: Simplified apify.json structure. It will be updated automatically before execution apify run and push command.

• Command apify create now shows progress bar of npm install.

• Small bugs fixes

  0.2.7 / 2018-11-27

• Updated all templates to latest apify packages

  0.2.6 / 2018-11-09

• Added warning if apify run reuse old state in storage

• Fixed issues #70 #65 #68

  0.2.5 / 2018-10-31

• Updated NPM dependencies

• Upgraded to apify-shared@0.1.6

• Fixed templates to use apify/actor-node-chrome Docker image instead of outdated apify/actor-node-puppeteer

  0.2.3 / 2018-09-17

• Updated all templates to apify version 0.8.*

• Added template named hello_word

  0.2.1 / 2018-09-17

• BREAKING CHANGES: The local storage directories have been renamed and package.json files needs a new start command. See migration guide for existing projects if you are upgrading from 0.1._ to 0.2._.

• You can specified another file that main.js for apify run command using npm start script.

  0.2.0 / 2018-09-12

• BREAKING CHANGES: Version 0.2.0 of Apify CLI supports only version 0.7.0 of API SDK or newer as management of environment variables has been changed according to Apify SDK version 0.7.0.

• Dropped support for Node 7

  0.1.18 / 2018-09-12

• Updated NPM dependencies, npm-shrinkwrap.json replaced with package-lock.json

• Updated NPM dependencies in code templates

  0.1.15 / 2018-07-23

• Rename act to actor

  0.1.13 / 2018-07-12

• Add environment variables for enable live view for local actors.

  0.1.12 / 2018-06-28

• From now apify call and apify push commands stream live logs from run and build to your terminal

• Add options -p, --purge, --purge-dataset, --purge-key-value-store, --purge-queue in apify run to clean stores before runs actor locally

• Add option -w, --wait-for-finish=wait-for-finish in apify push and apify call - command waits x seconds to finish run or build on Apify

• Fixes #26, #33, #34, #36, #38, #39, #37, #35

  0.1.11 / 2018-05-30

• Use npm-shrinkwrap.json instead of package-lock.json for published module

• Update template, where we using proxy

• Fix #30

  0.1.9 / 2018-04-18

• apify run takes APIFY_USER_ID and APIFY_TOKEN as environments variables, if client is logged locally

• apify call takes input from default local key-value-store

• Fix: duplicates new lines in log

  0.1.8 / 2018-04-17

• Print warning if you have old version of cli

• apify run - kills all sub processes for SIGINT signal (ctrl+c) - It kills all related browsers in apify run command, related issue: https://github.com/apify/apify-js/issues/72

  0.1.7 / 2018-04-12

• Readme and templates updates

  0.1.6 / 2018-04-11

• Add support for request queue

  0.1.5 / 2018-04-09

• Works for windows

• New command apify info

  0.1.x / 2018-04-01

• The first public release

  0.0.x / 2018-03-01

• Initial development, lot of new stuff