Security News
How Threat Actors are Abusing GitHub’s File Upload Feature to Host Malware
GitHub is susceptible to a CDN flaw that allows attackers to host malware on any public repository.
iconv-lite
Advanced tools
Package description
The iconv-lite npm package provides utilities for converting character encodings in pure JavaScript. It supports many different encodings and can convert to and from Buffer objects without the need for a native C++ binding. This makes it a lightweight and portable solution for encoding conversion.
Encoding Conversion
Converts text from one character encoding to another. The example shows how to decode a buffer to a string and encode a string to a buffer using Windows-1251 encoding.
const iconv = require('iconv-lite');
// Convert from an encoded buffer to js string.
const str = iconv.decode(Buffer.from([0x63, 0x61, 0x66, 0xe9]), 'win1251');
// Convert from js string to an encoded buffer.
const buf = iconv.encode('Sample input text', 'win1251');
Streaming Conversion
Provides a streaming interface for encoding conversion. This example demonstrates how to create a read stream from a file and pipe it through iconv-lite's decode stream.
const iconv = require('iconv-lite');
const fs = require('fs');
// Decode stream (from a file, for example)
const readStream = fs.createReadStream('file.txt');
const decodeStream = iconv.decodeStream('win1251');
readStream.pipe(decodeStream);
decodeStream.on('data', function(str) {
console.log(str); // converted text
});
Encoding Detection
Checks if a particular encoding is supported by iconv-lite. The example checks if UTF-8 encoding is supported.
const iconv = require('iconv-lite');
// Check if encoding is supported
const encodingSupported = iconv.encodingExists('utf-8');
console.log(encodingSupported); // true or false
The 'buffer' package is a Node.js core module that provides a way to handle binary data. It does not offer direct encoding conversion like iconv-lite, but it can be used in conjunction with other modules to achieve similar results.
The 'iconv' package is similar to iconv-lite but relies on native C++ bindings. It provides more comprehensive encoding support but is not as lightweight or portable as iconv-lite due to its compiled components.
The 'string_decoder' module is a Node.js core module that provides an API for decoding Buffer objects into strings using a specified character encoding. It is less feature-rich compared to iconv-lite and does not support as many encodings.
Readme
npm install
two more modules: buffer
and stream
).var iconv = require('iconv-lite');
// Convert from an encoded buffer to js string.
str = iconv.decode(Buffer.from([0x68, 0x65, 0x6c, 0x6c, 0x6f]), 'win1251');
// Convert from js string to an encoded buffer.
buf = iconv.encode("Sample input string", 'win1251');
// Check if encoding is supported
iconv.encodingExists("us-ascii")
// Decode stream (from binary stream to js strings)
http.createServer(function(req, res) {
var converterStream = iconv.decodeStream('win1251');
req.pipe(converterStream);
converterStream.on('data', function(str) {
console.log(str); // Do something with decoded strings, chunk-by-chunk.
});
});
// Convert encoding streaming example
fs.createReadStream('file-in-win1251.txt')
.pipe(iconv.decodeStream('win1251'))
.pipe(iconv.encodeStream('ucs2'))
.pipe(fs.createWriteStream('file-in-ucs2.txt'));
// Sugar: all encode/decode streams have .collect(cb) method to accumulate data.
http.createServer(function(req, res) {
req.pipe(iconv.decodeStream('win1251')).collect(function(err, body) {
assert(typeof body == 'string');
console.log(body); // full request body string
});
});
NOTE: This doesn't work on latest Node versions. See details.
// After this call all Node basic primitives will understand iconv-lite encodings.
iconv.extendNodeEncodings();
// Examples:
buf = new Buffer(str, 'win1251');
buf.write(str, 'gbk');
str = buf.toString('latin1');
assert(Buffer.isEncoding('iso-8859-15'));
Buffer.byteLength(str, 'us-ascii');
http.createServer(function(req, res) {
req.setEncoding('big5');
req.collect(function(err, body) {
console.log(body);
});
});
fs.createReadStream("file.txt", "shift_jis");
// External modules are also supported (if they use Node primitives, which they probably do).
request = require('request');
request({
url: "http://github.com/",
encoding: "cp932"
});
// To remove extensions
iconv.undoExtendNodeEncodings();
See all supported encodings on wiki.
Most singlebyte encodings are generated automatically from node-iconv. Thank you Ben Noordhuis and libiconv authors!
Multibyte encodings are generated from Unicode.org mappings and WHATWG Encoding Standard mappings. Thank you, respective authors!
Comparison with node-iconv module (1000x256kb, on MacBook Pro, Core i5/2.6 GHz, Node v0.12.0). Note: your results may vary, so please always check on your hardware.
operation iconv@2.1.4 iconv-lite@0.4.7
----------------------------------------------------------
encode('win1251') ~96 Mb/s ~320 Mb/s
decode('win1251') ~95 Mb/s ~246 Mb/s
stripBOM: false
in options
(f.ex. iconv.decode(buf, enc, {stripBOM: false})
).
A callback might also be given as a stripBOM
parameter - it'll be called if BOM character was actually found.addBOM: true
option.This library supports UTF-16LE, UTF-16BE and UTF-16 encodings. First two are straightforward, but UTF-16 is trying to be smart about endianness in the following ways:
defaultEncoding: 'utf-16be'
option. Strips BOM unless stripBOM: false
.addBOM: false
to override.This library supports UTF-32LE, UTF-32BE and UTF-32 encodings. Like the UTF-16 encoding above, UTF-32 defaults to UTF-32LE, but uses BOM and 'spaces heuristics' to determine input endianness.
defaultEncoding: 'utf-32be'
option. Strips BOM unless stripBOM: false
.addBOM: false
to override. (defaultEncoding: 'utf-32be'
can also be used here to change encoding.)When decoding, be sure to supply a Buffer to decode() method, otherwise bad things usually happen.
Untranslatable characters are set to � or ?. No transliteration is currently supported.
Node versions 0.10.31 and 0.11.13 are buggy, don't use them (see #65, #77).
$ git clone git@github.com:ashtuchkin/iconv-lite.git
$ cd iconv-lite
$ npm install
$ npm test
$ # To view performance:
$ node test/performance.js
$ # To view test coverage:
$ npm run coverage
$ open coverage/lcov-report/index.html
FAQs
Convert character encodings in pure javascript.
The npm package iconv-lite receives a total of 57,771,239 weekly downloads. As such, iconv-lite popularity was classified as popular.
We found that iconv-lite demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
GitHub is susceptible to a CDN flaw that allows attackers to host malware on any public repository.
Security News
At Node Congress, Socket CEO Feross Aboukhadijeh uncovers the darker aspects of open source, where applications that rely heavily on third-party dependencies can be exploited in supply chain attacks.
Research
Security News
The Socket Research team found this npm package includes code for collecting sensitive developer information, including your operating system username, Git username, and Git email.