Copy/paste detector for programming source code, supports 150+ formats.
Copy/paste is a common technical debt on a lot of projects. The jscpd gives the ability to find duplicated blocks implemented on more than 150 programming languages and digital formats of documents. The jscpd tool implements Rabin-Karp algorithm for searching duplications.
<script> or <style> sections in html$ npm install -g jscpd
$ npx jscpd /path/to/source
or
$ jscpd /path/to/code
or
$ jscpd --pattern "src/**/*.js"
Glob pattern for find files to detect
--pattern, -pExample:
$ jscpd --pattern "**/*.js"
Minimal block size of code in tokens. The block of code less than min-tokens will be skipped.
--min-tokens, -kThis option is called minTokens in the config file.
Minimal block size of code in lines. The block of code less than min-lines will be skipped.
--min-lines, -lMaximum file size in lines. The file bigger than max-lines will be skipped.
--max-lines, -xMaximum file size in bytes. The file bigger than max-size will be skipped.
--max-size, -zThe threshold for duplication level, check if current level of duplications bigger than threshold jscpd exit with error.
--threshold, -tThe path to configuration file. The config should be in json format. Supported options in config file can be the same with cli options.
--config, -cThe option with glob patterns to ignore from analyze. For multiple globs you can use comma as separator. Example:
$ jscpd --ignore "**/*.min.js,**/*.map" /path/to/files
--ignore, -iThe list of reporters. Reporters use for output information of clones and duplication process.
Available reporters:
jscpd-report.json file with clones report in json format;jscpd-report.xml file with clones report in xml format;jscpd-report.csv file with clones report in csv format;jscpd-report.md file with clones report in markdown format;html/ folder;jscpd-sarif.json file;Note: A reporter can be developed manually, see @jscpd/finder package.
--reporters, -rThe path to directory for reports. JSON and XML reports will be saved there.
--output, -oThe mode of detection quality.
strict - use all types of symbols as token, skip only blocks marked as ignored.mild - skip blocks marked as ignored and new lines and empty symbols.weak - skip blocks marked as ignored and new lines and empty symbols and comments.Note: A mode can be developed manually, see API section.
--mode, -mThe list of formats to detect for duplications. Available over 150 formats.
Example:
$ jscpd --format "php,javascript,markup,css" /path/to/files
--format, -fGet information about authors and dates of duplicated blocks from git.
--blame, -bDon't write a lot of information to a console.
Example:
$ jscpd /path/to/source --silent
Duplications detection: Found 60 exact clones with 3414(46.81%) duplicated lines in 100 (31 formats) files.
Execution Time: 1381.759ms
--silent, -sUse the absolute path in reports.
--absolute, -aIgnore case of symbols in code (experimental).
--ignoreCaseDo not follow symlinks.
--noSymlinks, -nUse for detect duplications in different folders only. For correct usage of --skipLocal option you should provide list of path's with more than one item.
Example:
jscpd --skipLocal /path/to/folder1/ /path/to/folder2/
will detect clones in separate folders only, clones from same folder will be skipped.
--skipLocalDefine the list of formats with file extensions. Available over 150 formats.
In following example jscpd will analyze files *.es and *.es6 as javascript and *.dt files as dart:
$ jscpd --formats-exts javascript:es,es6;dart:dt /path/to/code
Note: formats defined in the option redefine default configuration, you should define all need formats manually or create two configuration for run
jscpd
--formats-extsStores used for collect information about code, by default all information collect in memory.
Available stores:
Note: A store can be developed manually, see @jscpd/finder package and @jscpd/leveldb-store as example.
--storeIgnore code blocks matching the regexp patterns.
--ignore-patternExample:
$ jscpd /path/to/source --ignore-pattern "import.*from\s*'.*'"
Excludes import statements from the calculation.
Put .jscpd.json file in the root of the projects:
{
"threshold": 0,
"reporters": ["html", "console", "badge"],
"ignore": ["**/__snapshots__/**"],
"absolute": true
}
Also you can use section in package.json:
{
...
"jscpd": {
"threshold": 0.1,
"reporters": ["html", "console", "badge"],
"ignore": ["**/__snapshots__/**"],
"absolute": true,
"gitignore": true
}
...
}
By default, the tool exits with code 0 even code duplications were detected. This behaviour can be changed by specifying a custom exit code for error states.
Example:
jscpd --exitCode 1 .
--exitCodeMark blocks in code as ignored:
/* jscpd:ignore-start */
import lodash from 'lodash';
import React from 'react';
import {User} from './models';
import {UserService} from './services';
/* jscpd:ignore-end */
<!--
// jscpd:ignore-start
-->
<meta data-react-helmet="true" name="theme-color" content="#cb3837"/>
<link data-react-helmet="true" rel="stylesheet" href="https://static.npmjs.com/103af5b8a2b3c971cba419755f3a67bc.css"/>
<link data-react-helmet="true" rel="stylesheet" href="https://static.npmjs.com/cms/flatpages.css"/>
<link data-react-helmet="true" rel="apple-touch-icon" sizes="120x120" href="https://static.npmjs.com/58a19602036db1daee0d7863c94673a4.png"/>
<link data-react-helmet="true" rel="apple-touch-icon" sizes="144x144" href="https://static.npmjs.com/7a7ffabbd910fc60161bc04f2cee4160.png"/>
<link data-react-helmet="true" rel="apple-touch-icon" sizes="152x152" href="https://static.npmjs.com/34110fd7686e2c90a487ca98e7336e99.png"/>
<link data-react-helmet="true" rel="apple-touch-icon" sizes="180x180" href="https://static.npmjs.com/3dc95981de4241b35cd55fe126ab6b2c.png"/>
<link data-react-helmet="true" rel="icon" type="image/png" href="https://static.npmjs.com/b0f1a8318363185cc2ea6a40ac23eeb2.png" sizes="32x32"/>
<!--
// jscpd:ignore-end
-->
More info jscpd-badge-reporter
<?xml version="1.0" encoding="utf-8"?>
<pmd-cpd>
<duplication lines="10">
<file path="/path/to/file" line="1">
<codefragment><![CDATA[ ...first code fragment... ]]></codefragment>
</file>
<file path="/path/to/file" line="5">
<codefragment><![CDATA[ ...second code fragment...}]]></codefragment>
</file>
<codefragment><![CDATA[ ...duplicated fragment... ]]></codefragment>
</duplication>
</pmd-cpd>
{
"duplicates": [{
"format": "javascript",
"lines": 27,
"fragment": "...code fragment... ",
"tokens": 0,
"firstFile": {
"name": "tests/fixtures/javascript/file2.js",
"start": 1,
"end": 27,
"startLoc": {
"line": 1,
"column": 1
},
"endLoc": {
"line": 27,
"column": 2
}
},
"secondFile": {
"name": "tests/fixtures/javascript/file1.js",
"start": 1,
"end": 24,
"startLoc": {
"line": 1,
"column": 1
},
"endLoc": {
"line": 24,
"column": 2
}
}
}],
"statistic": {
"detectionDate": "2018-11-09T15:32:02.397Z",
"formats": {
"javascript": {
"sources": {
"/path/to/file": {
"lines": 24,
"sources": 1,
"clones": 1,
"duplicatedLines": 26,
"percentage": 45.33,
"newDuplicatedLines": 0,
"newClones": 0
}
},
"total": {
"lines": 297,
"sources": 1,
"clones": 1,
"duplicatedLines": 26,
"percentage": 45.33,
"newDuplicatedLines": 0,
"newClones": 0
}
}
},
"total": {
"lines": 297,
"sources": 6,
"clones": 5,
"duplicatedLines": 26,
"percentage": 45.33,
"newDuplicatedLines": 0,
"newClones": 0
}
}
}
For integration copy/paste detection to your application you can use programming API:
jscpd Promise API
import {IClone} from '@jscpd/core';
import {jscpd} from 'jscpd';
const clones: Promise<IClone[]> = jscpd(process.argv);
jscpd async/await API
import {IClone} from '@jscpd/core';
import {jscpd} from 'jscpd';
(async () => {
const clones: IClone[] = await jscpd(['', '', __dirname + '/../fixtures', '-m', 'weak', '--silent']);
console.log(clones);
})();
detectClones API
import {detectClones} from "jscpd";
(async () => {
const clones = await detectClones({
path: [
__dirname + '/../fixtures'
],
silent: true
});
console.log(clones);
})()
detectClones with persist store
import {detectClones} from "jscpd";
import {IMapFrame, MemoryStore} from "@jscpd/core";
(async () => {
const store = new MemoryStore<IMapFrame>();
await detectClones({
path: [
__dirname + '/../fixtures'
],
}, store);
await detectClones({
path: [
__dirname + '/../fixtures'
],
silent: true
}, store);
})()
In case of deep customisation of detection process you can build your own tool: If you are going to detect clones in file system you can use @jscpd/finder for make a powerful detector. In case of detect clones in browser or not node.js environment you can build your own solution base on @jscpd/code
This project exists thanks to all the people who contribute.
Thank you to all our backers! 🙏 [Become a backer]
Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [Become a sponsor]
MIT © Andrey Kucherenko
FAQs
detector of copy/paste in files
We found that jscpd demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.
Security News
The Ultralytics' PyPI Package was compromised four times in one weekend through GitHub Actions cache poisoning and failure to rotate previously compromised API tokens.