titanium-code-processor

Titanium Code Processor

The Titanium Code Processor is a tool for analyzing JavaScript code in Titanium Mobile projects. It provide a wide variety of useful functionality, including runtime error detection, Titanium API deprecation warnings, platform specific API validation, and more. It is built using a robust plugin solution that makes it relatively easy to add new analyses to the code processor. The tool can be used as a stand-alone binary, used as part of the Titanium CLI, or incorporated as a library into other node.js applications (See the API documentation in the 'docs' folder for more information on using the code processor as a library).

Note: In this document, whenever a data structure is defined, it uses the following convention:

[{name}] [<optional>] {type} [{description}]

Fields in [] are optional. If the type definition does not explicitly say <optional>, then it is required. If a type has children, i.e. it is an object or array, then it's children are specified using indented lines immediately below the type definition. Object properties always have a name and array entries never have a name.

Table of Contents

• Quick Start
  • Install Using NPM
  • Run
• Running Using the CLI
  • Command Options
  • Stream Output Format
    • Low Level Packet Format
    • High Level Packet Format
    • Message Types
  • Config File
• Running as Part of a Build
• Runtime Options
• Built-in Plugins
• Internal Concepts
• Running the ECMA-262 Unit Tests

Quick Start

Install Using NPM

[sudo] npm install -g titanium-code-processor

The code processor works as a command in the Titanium CLI, so make sure it is installed before continuing. If you have a 3.0.0. or newer Titanium SDK installed, you should already have the CLI. There are two ways to include the code processor as part of the CLI:

Automatic Method: run the included install script

node /path/to/titanium-code-processor/bin/install

Manual Method: configure the cli by hand

titanium config paths.commands --append /path/to/titanium-code-processor/commands
titanium config paths.hooks --append /path/to/titanium-code-processor/hooks

Note: On *NIX systems, the code processor is typically installed in /usr/local/lib/node_modules/titanium-code-processor

Run

From within your project directory, run:

titanium analyze -p iphone -A

Running Using the CLI

titanium analyze [options]

Analyzes a project.

Command Options

Option	Description
-A, --all-plugins	loads all plugins in the default search path
--exact-mode	enables exact mode evaluation. Exact mode does not use ambiguous modes and throws an exception if an Unknown type is encountered (ignored if --config-file is specified) [default: false]
--no-console-passthrough	Prevents console.* calls in a project from being logged to the console (ignored if --config-file is specified) [default: false]
--no-loop-evaluation	Whether or not to evaluate loops (ignored if --config-file is specified) [default: false]
--no-method-invokation	prevents methods from being invoked (ignored if --config-file is specified) [default: false]
--no-native-exception-recovery	disables recovering from native exceptions when not in try/catch statements (ignored if --config-file is specified) [default: false]
--process-unvisited-code	when set to true, all nodes and files that are not visited/skipped will be processed in ambiguous mode after all other code has been processed. While this will cause more of a project to be analyzed, this will decrease accuracy and can generate a lot of false positives (ignored if --config-file is specified) [default: false]
-F, --config-file [value]	the path to the config file, note: most options and flags are ignored with this option
--cycle-detection-stack-size [size]	the size of the cycle detection stack. Cycles that are larger than this size will not be caught [default: 10000]
--execution-time-limit [time limit]	the maximum time the app is allowed to run before erroring. 0 means no time limit (ignored if --config-file is specified) [default: 300000]
--log-level [level]	minimum logging level (ignored if --config-file is specified) [trace, debug, info, warn, error]
--max-cycles [size]	The maximum number of cycles to allow before throwing an exception [default: 200001]
--max-loop-iterations [iterations]	the maximum number of iterations a loop can iterate before falling back to an unknown evaluation (ignored if --config-file is specified) [default: 200000]
--max-recursion-limit [recursion limit]	the maximum recursion depth to evaluate before throwing a RangeError exception (ignored if --config-file is specified) [default: 500]
-o, --output [format]	output format [report, json, stream]
-p, --platform [platform]	the name of the OS being built-for, reflected in code via Ti.Platform.osname (ignored if --config-file is specified)
--plugins [plugins]	a comma separated list of plugin names to load (ignored if --config-file is specified)
-d, --project-dir [value]	the directory containing the project, otherwise the current working directory (ignored if --config-file is specified)
-R, --results-dir [value]	the path to the directory that will contain the generated results pages (ignored if --config-file is specified)

Stream Output Format

titanium-code-processor subprocess <path/to/config/file>

The stream output format allows for JSON data to be passed in packets as the code processor runs. Data is passed back and forth using a custom two-layer packet format. The lower layer is completely custom, while the upper layer is formatted JSON and is encapsulated by the lower layer

Low Level Packet Format

The low level packet consists of four comma separated fields that forms a message

[Message Type],[Sequence ID],[Message Length],[data]

Note: the packet header at this level is ASCII formatted, although the data can theoretically be in any format

Name	Description
MessageType	A three character sequence that currently is always 'REQ' (request)
Sequence ID	A 32-bit, base 16 number that identifies the message. This value is always 8 characters long, and includes 0 padding if necessary.
Message Length	A 32-bit, base 16 number that identifies the length of the message. This value is always 8 characters long, and includes 0 padding if necessary. Hex letters must be lower case.
Data	The data for the message as specified in the [High Level Packet Format](#high-level-packet-format) section

Example:

REQ,000079AC,0000000C,{foo: 'bar'}

High Level Packet Format

The high level packet is just a JSON string. The contents of the JSON object vary depending on message type and context.

A request always has the following definition:

• messageType string A free form string defining the type of message, and is typically the name of an event
• data any Any valid JSON value. Set to null if there is no data.

Example:

{
	"messageType": "enteredFile",
	"data": {
		"filename": "path/to/file"
	}
}

Message Types

The code processor currently only sends one message, but others are planned for the future

results

The results from the project

• errors array The errors from the project. The array is empty, but exists, if no errors were found
  • type string The type of error
  • description string A description of the error
  • filename string The full path to the file where the error was detected
  • line number The line number where the error was detected
  • column number The column number where the error was detected
• warnings array The warnings from the project. The array is empty, but exists, if no warnings were found
  • type string The type of warning
  • description string A description of the warning
  • filename string The full path to the file where the warning was detected
  • line number The line number where the warning was detected
  • column number The column number where the warning was detected
• plugins array
  • name string The name of the plugin, e.g. "ti-api-deprecation-finder"
  • <other key> Some other key specific to the plugin. See the plugin's README for detailed information
• elapsedTime number The amount of time it took to process the project, in ms
• resultsPath string The value of resultsPath passed to the code processor, for easy reference

Example:

{
	"errors": [{
		"name": "SyntaxError",
		"description": "The description of the error",
		"data": {
			"otherKeys": "other data, including message, type, etc"
		},
		"filename": "path/to/file",
		"line": 0,
		"column": 0,
		"occurances": 0
	}],
	"warnings": [{
		"name": "SyntaxError",
		"description": "The description of the error",
		"data": {
			"otherKeys": "other data, including message, type, etc"
		},
		"filename": "path/to/file",
		"line": 0,
		"column": 0,
		"occurances": 0
	}],
	"plugins": [{
		"name": "plugin-name",
		"otherKeys": "other values"
	}],
	"elapsedTime": 0,
	"resultsPath": "resultsPath/from/config/file"
}

Config File

The config file contains everything necessary for processing a project. Below is it's definition

• sourceInformation object The path to the entry point
  • sourceDir The directory containing the source code to be analyzed
  • entryPoint The entry point for the project
  • projectDir The project directory
  • originalSourceDir <optional> The original directory that contained source code. Source maps map from here to sourceDir
  • sourceMaps <optional> The source maps in key-value form. The key is a relative path to the file with sourceDir as the base, i.e. an absolute path to a file is at sourceDir + '/' + sourceMapKey
• logging object Logging configuration
  • file <optional> object The configuration for logging to a file
    • level string The log level, e.g. "debug"
    • path string The full path to the log file. The file does not have to previously exist
• options object The options for the project. See Runtime Options for details
• plugins array The plugins to load
  • object The configuration for a plugin to load
    • path string The path to the plugin
    • options object The plugin options. See the plugin's README for details

Note: all paths are relative to the CWD. ~ is not supported, and it is recommended to use absolute paths

Example config file for an Alloy application:

{
	"sourceInformation": {
		"projectDir": "/path/to/project",
		"entryPoint": "path/to/project/Resources/app.js",
		"sourceDir": "/path/to/project/Resources",
		"sourceMaps": "/path/to/project/build/map/Resources",
		"originalSourceDir": "/path/to/project/app"
	},
	"logging": {
		"file": {
			"level": "debug",
			"path": "path/to/log"
		}
	},
	"options": {
		"resultsPath": "path/to/results/directory",
		"processUnvisitedCode": true,
		"maxRecursionLimit": 500
	},
	"plugins": [
		{
			"path": "path/to/common-globals",
			"options": {}
		},
		{
			"path": "path/to/require-provider",
			"options": {
				"platform": "iphone",
				"modules": []
			}
		},
		{
			"path": "path/to/ti-api-processor",
			"options": {
				"platform": "iphone",
				"sdkPath": "path/to/sdk",
				"values": {
					"Titanium.Platform.displayCaps.platformWidth": 720
				}
			}
		},
		{
			"path": "path/to/ti-api-usage-finder",
			"options": {}
		},
		{
			"path": "path/to/ti-api-platform-validator",
			"options": {
				"platform": "iphone"
			}
		}
	]
}

Running as Part of a Build

The code processor is integrated as a build step in the CLI. To enable it, add the following to your tiapp.xml:

<code-processor>
	<enabled>true</enabled>
</code-processor>

Options and plugins can also be specified in the tiapp.xml, as the following example shows:

<code-processor>
	<enabled>true</enabled>
	<options>
		<nativeExceptionRecovery>true</nativeExceptionRecovery>
		<invokeMethods>false</invokeMethods>
	</options>
	<plugins>
		<plugin>require-provider</plugin>
	</plugins>
</code-processor>

Running as part of a build will report errors and warnings, and is used in Mobile Web to compress the size of index.html

Runtime Options

These options can be set at the command line by using the '-c' flag from the code processor command, or by setting the option in the tiapp.xml file if using the Titanium CLI.

name	type	default	description
invokeMethods	boolean	true	Indicates whether or not to invoke methods. If set to false, the method is evaluated once in ambiguous context mode.
evaluateLoops	boolean	true	Indicates whether or not to evaluate loops. If set to false, the loop body and any loop conditionals are evaluated once in ambiguous block mode.
maxLoopIterations	integer	1000000000000	The maximum number of iterations of a loop to evaluate. If this threshold is exceeded, the block is evaluated once in ambiguous block mode. Note: this threshold makes it impossible to analyze infinite loops.
maxRecursionLimit	integer	1000	The maximum function call depth to evaluate, similar to a stack size limit. If this threshold is exceeded, function bodies are not evaluated and unknown is returned. Note: this threshold makes it impossible to analyze infinite recursion.
logConsoleCalls	boolean	true	If enabled, all console.* calls in a user's code are logged to the terminal using the appropriate log level.
executionTimeLimit	integer	undefined	The maximum time to execute the code before throwing an exception. If not defined, the code processor will run as long as it takes to complete the analysis.
exactMode	boolean	false	Enables exact mode and causes the code processor act exactly like a standard JavaScript interpreter. Intended primarily for unit testing and is not recommended for project .analysis
nativeExceptionRecovery	boolean	false	When enabled, the code processor will recover from many types of native exceptions and continue analysis. Enabling this has the potential of generating incorrect results, but can be used to parse code that normally wouldn't be parsed because of an error.

Built-in Plugins

Plugins are informally grouped into two types: analyzers and providers. Providers provide some sort of feature in the runtime that is not included in the ECMAScript specification, such as the Titanium Mobile API. Providers do not report any results. Analyzers do not provide any features in the runtime but instead analyze code and do report results. Many analyzers depend on providers to work. All of the current plugins are listed below, along with their type and if they have any other dependencies

name	type	dependencies	description
common-globals	provider	<none>	Provides implementations for common globals that aren't part of the JavaScript spec but are provided on all Titanium Mobile platforms (setTimeout, console, etc).
require-provider	provider	<none>	Provides an implementation of ```require()``` that matches the Titanium Mobile implementation, including its inconsistencies with CommonJS.
require-finder	analyzer	require-provider	Reports all files that are ```require()```'d in a project.
ti-api-provider	provider	require-provider, common-globals	Provides an implementation of the Titanium Mobile API. This implementation reads the API documentation for the SDK used by the project to create the API implementation. As such, the SDK specified in the project's tiapp.xml file *must* be installed.
ti-api-deprecation-finder	analyzer	ti-api-provider	Reports all deprecated APIs used by the project.
ti-api-platform-validator	analyer	ti-api-provider	Reports all instances where a platform specific feature is used on the wrong platform, e.g. calling ```Ti.Android.createIntent``` on iOS.
ti-api-usage-finder	analyzer	ti-api-provider	Reports all Titanium Mobile APIs used by the project.
ti-api-include-finder	analyzer	ti-api-provider	Reports all files that are ```Ti.include()```'d by the project.

Internal Concepts

At the core of the code processor is an ECMAScript 5 interpreter that has been specially designed to work offline. To make this work, two new concepts have been introduced: an 'unknown' data type and 'ambiguous modes.'

The unknown data type is pretty self-explanatory; it's a value that we don't know the value of. For example, if the following code is run:

var x = Date.now();

x will be set to unknown since the date changes from run to run and isn't known at compile time. Operations on unknown values always produce unknown values. For example, y evaluates to unknown in all of the following circumstances:

var x = Date.now(),
	y;
y = x + 20;
y = x > 100;
y = x.foo();
y = x.toString();
y = typeof x;

Ambiguous modes occur when we are evaluating code without knowing exactly how it is invoked. There are two types of ambiguous mode: ambiguous context mode and ambiguous block mode.

An ambiguous context is a function or module that is invoked without knowing exactly how it was invoked. All callbacks passed to the Titanium API are evaluated as ambiguous contexts, and any functions called from an ambiguous block is called as an ambiguous context. In the following example, y is set to unknown:

var y;
setTimeout(function () {
	y = 20;
}, 10)

An ambiguous block is a loop/conditional body that is evaluated without knowing the exact circumstances it is evaluated in. If statements and while/do-while loops are evaluated as an ambiguous block if the conditional is unknown. For and for-in loops are evaluated as an ambiguous block if some part of the iteration conditions are unknown. All assignments in an ambiguous block evaluate to unknown and all functions called from an ambiguous block are evaluated in an ambiguous context. In the following example, y is set to unknown:

var x = Date.now(),
	y;
if (x) {
	y = 10;
} else {
	y = 20;
}

Running the ECMA-262 Unit Tests

The ECMA working group, who maintains the ECMA-262 specification (the JavaScript spec), also maintains a series of unit tests. To run the unit tests:

• Clone the test-262 Mercurial Repository
  • Note: you will need install Mercurial before you can clone the repository. Git will not work.
• Add the following to your titanium config file at ~/.titanium/config.json:

{
	"code-processor": {
		"test": {
			"test-262-directory": "/path/to/test-262/repo"
		}
	}
}

• Run the test script at <titanium code processor dir>/tests/bin/tests
  • You can run tests --help to see options for controlling the test process

Contributing

Titanium is an open source project. Titanium wouldn't be where it is now without contributions by the community. Please consider forking this repo to improve, enhance or fix issues. If you feel like the community will benefit from your fork, please open a pull request.

To protect the interests of the Titanium contributors, Appcelerator, customers and end users we require contributors to sign a Contributors License Agreement (CLA) before we pull the changes into the main repository. Our CLA is simple and straightforward - it requires that the contributions you make to any Appcelerator open source project are properly licensed and that you have the legal authority to make those changes. This helps us significantly reduce future legal risk for everyone involved. It is easy, helps everyone, takes only a few minutes, and only needs to be completed once.

You can digitally sign the CLA online. Please indicate your email address in your first pull request so that we can make sure that will locate your CLA. Once you've submitted it, you no longer need to send one for subsequent submissions.

(C) Copyright 2012-2013, Appcelerator Inc. All Rights Reserved.