karma-accessibility-checker
Overview
karma-accessibility-checker
is a Karma plugin that allows you to perform the following:
- integrate accessibility testing within a continuous integration pipeline, such as Travis CI.
- scan HTML nodes/widgets, URLs, local files, HTML documents, and allows you to scan HTML content in the form of a string
- aside from just performing accessibility scanning, it provides a framework to validate accessibility scan results against baseline files and/or simply failing the test cases based on the levels of violations found during the scan
Table of Contents
Usage
The tools that have been deployed to NPM so it can be easily downloaded and installed:
Quick start
Grab a boilerplate
Getting started
- Setup and initialize - Follow the instructions in the Prerequisites and Install sections.
- Configure Karma and the karma-accessibility-checker plugin - Follow the Configuration instructions.
- Learn how to use the
karma-accessibility-checker
APIs to perform accessibility scans - Refer to the Usage and API documentation.
Prerequisites
Install Node.js and NPM
Note: Use the latest available Node.js version.
Basic understanding of testing with Karma testrunner
Install
Note: For Windows, you may have to install and use cygwin CLI with curl package in order to run curl commands on windows.
Install npm
Install the karma-accessibility-checker
plugin from the npm repository:
$ npm install --save-dev karma-accessibility-checker
Configuration
Karma needs to be configured before you load the karma-accessibility-checker
plugin - and then you need to configure the karma-accessibility-checker
plugin before using it.
Configuring Karma
Configuring plugins
If you are using the plugins array in the Karma configuration file (karma.conf.js
), you will need to add this to the
plugins array to load the karma-accessibility-checker
plugin.
module.exports = function (config) {
config.set({
plugins: [require("karma-accessibility-checker")],
});
};
Configuring framework
and reporters
Add aChecker
to the framework
, reporters
array in the Karma configuration file (karma.conf.js
). This will load the karma-accessibility-checker
plugin.
module.exports = function (config) {
config.set({
frameworks: ["jasmine", "aChecker"],
reporters: ["progress", "aChecker"],
});
};
Configuring preprocessor
Use aChecker
in preprocessor
to process the baseline files and load them into memory. This saved copy of the files, can then be used for accessibility scan results comparison.
module.exports = function (config) {
config.set({
preprocessors: {
"test/baseline/**/*.json": ["aChecker"],
},
files: ["test/baseline/**/*.json", "test/**/*.js"],
});
};
Configuring the plugin
Configuring the karma-accessibility-checker
plugin involves constructing a .achecker.yml
file in the project root. This file, will contain all of the configuration
options for karma-accessibility-checker
. This is the structure of the .achecker.yml
file:
ruleArchive: latest
policies:
- IBM_Accessibility
failLevels:
- violation
- potentialviolation
reportLevels:
- violation
- potentialviolation
- recommendation
- potentialrecommendation
- manual
outputFormat:
- json
outputFilenameTimestamp: true
label:
- master
outputFolder: results
cacheFolder: /tmp/accessibility-checker
API-based usage
karma-accessibility-checker
is solely an API-based Karma plugin, therefore APIs should be used within test cases on a browser launched by Karma.
To perform an accessibility scan within your test cases and verify the scan results:
aChecker.getCompliance(testDataFileContent, testFile, function (results) {
var returnCode = aChecker.assertCompliance(results);
expect(returnCode).toBe(
0,
"Scanning " + testFile + " failed." + JSON.stringify(results)
);
done();
});
Refer to Examples for sample usage scenarios.
API
aChecker.getCompliance(content
, label
, callback
)
Execute accessibility scan on provided content. The content can be in these forms:
- HTML (String)
- Single node/widget (HTMLElement)
- Local file path (String)
- URL (String)
- Document node (HTMLDocument)
Use a callback mechanism (callback
) to extract the results and perform assertion using accessibility-checker APIs.
content
- (String | HTMLElement | HTMLDocument) content to be scanned for accessibility violations.label
- (String) unique label to identify this accessibility scan from others. Using "/" in the label allows for directory hierarchy when results are saved.callback
- (Function) callback that is invoked (indicating success). The callback will be invoked with the results
as a parameter to the callback function provided. This is the outline of the results object which will be passed to the callback function:
{
report: {
scanID: "18504e0c-fcaa-4a78-a07c-4f96e433f3e7",
toolID: "accessibility-checker-v3.0.0",
label: "MyTestLabel",
numExecuted: 137,
nls: {
"WCAG20_Html_HasLang": {
"Pass_0": "Page language detected as {0}"
},
},
summary: {
URL: "https://www.ibm.com",
counts: {
violation: 1,
potentialviolation: 0,
recommendation: 0,
potentialrecommendation: 0,
manual: 0,
pass: 136,
ignored: 0
},
scanTime: 29,
ruleArchive: "September 2019 Deployment (2019SeptDeploy)",
policies: [
"IBM_Accessibility"
],
reportLevels: [
"violation",
"potentialviolation",
"recommendation",
"potentialrecommendation",
"manual"
],
startScan: 1470103006149
},
results: [
{
"ruleId": "WCAG20_Html_HasLang",
"reasonId": "Pass_0",
"value": [
"VIOLATION",
"PASS"
],
"path": {
"dom": "/html[1]",
"aria": "/document[1]"
},
"ruleTime": 0,
"message": "Page language detected as en",
"messageArgs": [
"en"
],
"apiArgs": [],
"bounds": {
"left": 0,
"top": 0,
"height": 143,
"width": 800
},
"snippet": "<html lang=\"en\">",
"category": "Accessibility",
"ignored": false,
"level": "pass"
},
]
}
}
Refer to the actualResults
parameter in aChecker.assertCompliance API for more details about the properties of results object.
aChecker.assertCompliance(actualResults
)
Assertion will be perform one of the following ways based on the condition that is met:
-
If a baseline file of scan results is available in memory, it will be compared to the actualResults
. If the actualResults
matches the baseline, it will return 0. If not, it will return 1. In this case, assertion is only run on the XPath
and ruleId
.
-
If there is no baseline file, assertion will be made based on the provided failLevels
. In this case, 2 is returned if there are failures based on failLevels
.
-
actualResults
- (Object) results for which assertion needs to be run. Properties include:
- scanID, (String) auto generated UUID used to associate a session.
- toolID, (String) tool ID for the tool that generated these results.
- label, (String) label provided to identify unique scan results. (provided through the aChecker.getCompliance API).
- URL, (String) contains the URL that was scanned using aChecker.getCompliance API. (provided through aChecker.getCompliance API).
- issueMessages, (Object) violation messages based on language.
- messages, (Object) violation messages indexed by message ID.
- lang, (String) locale of the violation messages.
- summary, (Object) summary of the scan that these results are for. Properties include:
- counts, (Object) number of violations based on the violation level. Properties include:
- violation, (Int) total number of violations.
- potentialviolation, (Int) total number of potential violations.
- recommendation, (Int) total number of recommendations.
- potentialrecommendation, (Int) total number of potential recommendations.
- manual, (Int) total number of manual checks.
- scanTime, (Int) total number of milliseconds it took to perform the scan.
- ruleArchive, (String) rule archive used for this particular scan result.
- policies, (Array) policies used for the scan result.
- reportLevels, (Array) list of violation levels to include in the report. (save to file).
- startScan, (Int) start time of the scan in milliseconds since epoch, GMT.
- reports, (Array) list of reports in the case of multiple iframes are present on the single page. (iframe scanning not support yet) Each array element is an object with these properties:
- frameIdx, (Int) frame index in the page represented as an integer value.
- frameTitle, (String) title of the frame on the page that was scanned.
- issues, (Array) detailed list of violations. Each array element is an objects with these properties:
- severityCode, (String) severity code of the violation.
- messageCode, (String) message code of the violation. Used to map the localized message string.
- ruleId, (String) rule ID of the violation.
- help, (String) help file name of the violation.
- msgArgs, (Array) list of tokens to help provide a detailed error description of the issue.
- bounds, (Object) provides the pixel position of the element that triggered the violation. Properties include:
- left, (Int) left pixel position of the element.
- top, (Int) top pixel position of the element.
- height, (Int) height pixel position of the element.
- width, (Int) width pixel position of the element.
- level, (String) level of the violation.
- xpath, (String) XPath of the element that triggered the violation.
- snippet, (String) snippet for the element that triggered the violation.
-
Returns 0
if actualResults
matches baseline or if no violations match the failLevels
-
Returns 1
if actualResults
do not match the baseline
-
Returns 2
if there is a failure based on failLevels
-
Returns -1
if an exception has occurred during scanning and the results reflected that
aChecker.getDiffResults(label
)
Retrieve the diff results for a specified scan (denoted by its label) when API aChecker.assertCompliance(...)
returns 1 (when the actualResults
do not match the baseline).
label
- (String) label to use when getting the diff results. The label should match the one provided for aChecker.getCompliance(...).
Returns a diff object, where the left hand side (lhs) is actualResults and the right hand side (rhs) is baseline.
Refer to the deep-diff documentation for the format of the diff object, and how to interpret the object.
Returns undefined
if there are no differences.
aChecker.getBaseline(label
)
Retrieve the baseline result object based on the label provided.
label
- (String) label for which to get the baseline. The label should match the one provided for aChecker.getCompliance(...).
Returns object
that follows the same structure as the results object outlined in the aChecker.getCompliance
and aChecker.assertCompliance API.
Returns undefined
if a baseline is not found for the label provided.
aChecker.diffResultsWithExpected(actual
, expected
, clean
)
Compare provided actual
and expected
objects and get the differences between them.
actual
- (Object) actual results that you want to compare. Refer to aChecker.assertCompliance for details about available properties to include.expected
- (Object) expected results to compare to. Refer to aChecker.assertCompliance for details about available properties to include.clean
- (boolean) clean the actual
and expected
results by converting the objects to match with a basic compliance compare of only xpath
and ruleID
.
Returns a diff object, where the left hand side (lhs) is actualResults and the right hand side (rhs) is baseline.
Refer to the deep-diff documentation for the format of the diff object, and how to interpret the object.
Returns undefined
if there are no differences.
aChecker.stringifyResults(results
)
Retrieve the readable stringified representation of the scan results.
results
- (Object) results which need to be stringified. Refer to aChecker.assertCompliance for details about available properties to include.
Returns a String
representation of the scan results which can be logged to a console.
Errors
Error: labelNotProvided
This is a subtype of Error
defined by the karma-accessibility-checker
plugin. It is considered a programming error.
labelNotProvided
is thrown from the aChecker.getCompliance(...)
method call when a label is not provided to a function call for the scan that is to be performed.
Note: A label must always be provided when calling the aChecker.getCompliance(...)
function.
Error: labelNotUnique
This is a subtype of Error
defined by the karma-accessibility-checker
plugin. It is considered a programming error.
labelNotUnique
is thrown from aChecker.getCompliance(...)
method call when a unique label is not provided to
function call for the scan that is to be performed. Note: Across all accessibility scans, the label provided
must always be unique.
Error: slackNotificationTargetIsNotValid
This is a subtype of Error
defined by the karma-accessibility-checker
plugin. It is considered a programming error.
slackNotificationTargetIsNotValid
is thrown from [framework.aChecker]
during Slack configuration verification.
The error occurs when the provided Slack configuration does not follow one of these formats: <account>:<token>#<channel>
or webhook URL.
Error: SlackAPIError
This is a subtype of Error
defined by the karma-accessibility-checker
plugin. It is considered a programming error.
SlackAPIError
is thrown from [reporter.aChecker]
during Slack notification dispatch using the Slack API.
The error occurs when the provided Slack token and/or channel is incorrect.
A detailed explanation of the error will be provided when this error occurs.
Error: SlackWebHookError
This is a subtype of Error
defined by the karma-accessibility-checker
plugin. It is considered a programming error.
SlackWebHookError
is thrown from [reporter.aChecker]
during Slack notification dispatch using webhook.
The error occurs when the provided webhook URL is incorrect.
Error: LoadingConfigError
This is a subtype of Error
defined by the karma-accessibility-checker
plugin. It is considered a programming error.
LoadingConfigError
is thrown from [framework.aChecker]
during loading/verification of the configuration file.
The error occurs when the provided configuration file contains syntax errors.
Error: RuleArchiveInvalid
This is a subtype of Error
defined by the karma-accessibility-checker
plugin. It is considered a programming error.
RuleArchiveInvalid
is thrown from [framework.aChecker]
during verification of rule archive in the configuration file.
The error occurs when the provided ruleArchive
value in the configuration file is invalid.
Error: ValidPoliciesMissing
This is a subtype of Error
defined by the karma-accessibility-checker
plugin. It is considered a programming error.
ValidPoliciesMissing
is thrown from [aChecker.getCompliance(...)]
method call when no valid policies are in the configuration file.
Note: The valid policies will vary depending on the selected ruleArchive
.
FAQ
- How do I get a list of the available
ruleArchive, policies
and their ID's?
- run
npx achecker_policies
Known Issues
- Unable to scan URLs due to "permission denied to access property "document"" when trying to access document of generated iframe. This is due to cross domain frame access restrictions in browsers. On firefox there is no provided alternative, Chrome provides a way to override this by adding the following to karma.config.js:
module.exports = function (config) {
config.set({
browsers: ['ChromeCustom'],
customLaunchers: {
ChromeCustom: {
base: 'ChromeHeadless',
flags: ['--disable-web-security']
}
}
....
});
}
- Unable to scan local files when provided to
aChecker.getCompliance(...)
API as a local file URL. This is due to a limitation in Karma where it is not able to load local files using file://
protocol. For scanning local file, there is a work around which can be used to scan them, following are the steps to update karma.config.js file with the following:
module.exports = function (config) {
config.set({
browsers: ['Chrome'],
frameworks: ['jasmine', 'AAT'],
files: [
'src/**/*.html',
"test/**/*.js"
],
preprocessors: {
'src/**/*.html': ['html2js']
},
reporters: ['progress', 'AAT'],
....
});
}
Feedback and Reporting bugs
If you think you've found a bug or have feedback, report them in the Git issues, tagged with karma-accessibility-checker
.
To help us fix the bug, please try to provide a log. You can enable Karma debugging in the Karma configuration file by changing the logLevel to config.LOG_DEBUG
. Pipe (karma start > debug.log
) console output to a file and attach the file to the bug report that you open.
If you are an IBM employee, feel free to ask questions in the IBM internal Slack channel #accessibility-at-ibm
.
License