encoding-japanese
Advanced tools
The encoding-japanese npm package provides functionalities for encoding and decoding Japanese text. It supports various character encodings such as UTF-8, Shift_JIS, EUC-JP, and ISO-2022-JP. The package is useful for converting between different encodings, detecting encoding types, and handling Japanese text in web applications.
Encoding Conversion
This feature allows you to convert text from one encoding to another. In this example, the Japanese text 'こんにちは' is converted to a UTF-8 byte array.
const Encoding = require('encoding-japanese');
const utf8Array = Encoding.convert('こんにちは', 'UTF8');
console.log(utf8Array);Encoding Detection
This feature detects the encoding of a given byte array. In this example, the byte array is detected to determine its encoding.
const Encoding = require('encoding-japanese');
const detectedEncoding = Encoding.detect(new Uint8Array([0x82, 0xa0, 0x82, 0xa2, 0x82, 0xa4]));
console.log(detectedEncoding);String to Byte Array
This feature converts a string to a byte array. In this example, the Japanese text 'こんにちは' is converted to a byte array.
const Encoding = require('encoding-japanese');
const byteArray = Encoding.stringToCode('こんにちは');
console.log(byteArray);Byte Array to String
This feature converts a byte array back to a string. In this example, a byte array is converted back to the Japanese text 'こんにちは'.
const Encoding = require('encoding-japanese');
const byteArray = [0x82, 0xa0, 0x82, 0xa2, 0x82, 0xa4];
const string = Encoding.codeToString(byteArray);
console.log(string);iconv-lite is a popular package for converting character encodings. It supports a wide range of encodings and is known for its performance and ease of use. Unlike encoding-japanese, iconv-lite is not specifically focused on Japanese encodings but supports them among many others.
jconv is a package specifically designed for converting between Japanese encodings. It supports Shift_JIS, EUC-JP, and UTF-8. While it has a narrower focus compared to encoding-japanese, it is a good alternative for handling Japanese text encodings.
Convert or detect character encoding in JavaScript.
encoding.js is a JavaScript library for converting and detecting character encodings
that support Japanese character encodings such as Shift_JIS, EUC-JP, JIS, and Unicode such as UTF-8 and UTF-16.
Since JavaScript string values are internally encoded as UTF-16 code units (ref: ECMAScript® 2019 Language Specification - 6.1.4 The String Type), they cannot properly handle other character encodings as they are, but encoding.js enables conversion by handling them as arrays instead of strings.
Each character encoding is handled as an array of numbers with character code values, for example [130, 160] ("あ" in UTF-8).
The array of character codes passed to each method of encoding.js can also be used with TypedArray such as Uint8Array, and Buffer in Node.js.
Numeric arrays of character codes can be converted to strings with methods such as Encoding.codeToString ,
but because of the above JavaScript specifications, some character encodings cannot be handled properly when converted to strings.
So if you want to use strings instead of arrays, convert it to percent-encoded strings like '%82%A0' by using Encoding.urlEncode and Encoding.urlDecode to passed to other resources.
Or, Encoding.base64Encode and Encoding.base64Decode can be passed as strings in the same way.
encoding.js is published under the package name encoding-japanese on npm.
$ npm install --save encoding-japanese
importimport Encoding from 'encoding-japanese';
requireconst Encoding = require('encoding-japanese');
TypeScript type definitions for encoding.js are available at @types/encoding-japanese (thanks @rhysd).
$ npm install --save-dev @types/encoding-japanese
Install from npm or download from the release list and use encoding.js or encoding.min.js in the package.
*Please note that if you git clone, even the master branch may be under development.
<script src="encoding.js"></script>
Or use the minified encoding.min.js
<script src="encoding.min.js"></script>
When the script is loaded, the object Encoding is defined in the global scope (ie window.Encoding).
You can use the encoding.js (package name: encoding-japanese) CDN on cdnjs.com.
| Value in encoding.js | detect() | convert() | MIME Name (Note) |
|---|---|---|---|
| ASCII | ✓ | US-ASCII (Code point range: 0-127) | |
| BINARY | ✓ | (Binary strings. Code point range: 0-255) | |
| EUCJP | ✓ | ✓ | EUC-JP |
| JIS | ✓ | ✓ | ISO-2022-JP |
| SJIS | ✓ | ✓ | Shift_JIS |
| UTF8 | ✓ | ✓ | UTF-8 |
| UTF16 | ✓ | ✓ | UTF-16 |
| UTF16BE | ✓ | ✓ | UTF-16BE (big-endian) |
| UTF16LE | ✓ | ✓ | UTF-16LE (little-endian) |
| UTF32 | ✓ | UTF-32 | |
| UNICODE | ✓ | ✓ | (JavaScript's internal encoding. *See About UNICODE below) |
UNICODEIn encoding.js, the internal character encoding that can be handled in JavaScript is defined as UNICODE.
As mentioned above (Features), JavaScript strings are internally encoded in UTF-16 code units, and other character encodings cannot be handled properly.
Therefore, to convert to a character encoding properly represented in JavaScript, specify UNICODE.
(*Even if the HTML file encoding is UTF-8, specify UNICODE instead of UTF8 when handling it in JavaScript.)
The value of each character code array returned from Encoding.convert is a number of 0-255 if you specify a character code other than UNICODE such as UTF8 or SJIS,
or a number of 0-65535 (range of String.prototype.charCodeAt() values = Code Unit) if you specify UNICODE.
Convert character encoding from JavaScript string (UNICODE) to SJIS.
const unicodeArray = Encoding.stringToCode('こんにちは'); // Convert string to code array
const sjisArray = Encoding.convert(unicodeArray, {
to: 'SJIS',
from: 'UNICODE'
});
console.log(sjisArray);
// [130, 177, 130, 241, 130, 201, 130, 191, 130, 205] ('こんにちは' array in SJIS)
Convert character encoding from SJIS to UNICODE.
var sjisArray = [
130, 177, 130, 241, 130, 201, 130, 191, 130, 205
]; // 'こんにちは' array in SJIS
var unicodeArray = Encoding.convert(sjisArray, {
to: 'UNICODE',
from: 'SJIS'
});
var str = Encoding.codeToString(unicodeArray); // Convert code array to string
console.log(str); // 'こんにちは'
Detect character encoding.
var data = [
227, 129, 147, 227, 130, 147, 227, 129, 171, 227, 129, 161, 227, 129, 175
]; // 'こんにちは' array in UTF-8
var detectedEncoding = Encoding.detect(data);
console.log('Character encoding is ' + detectedEncoding); // 'Character encoding is UTF8'
(Node.js) Example of reading a text file written in SJIS.
const fs = require('fs');
const Encoding = require('encoding-japanese');
const sjisBuffer = fs.readFileSync('./sjis.txt');
const unicodeArray = Encoding.convert(sjisBuffer, {
to: 'UNICODE',
from: 'SJIS'
});
console.log(Encoding.codeToString(unicodeArray));
The return value is one of the above "Supported encodings" or false if it cannot be detected.
var sjisArray = [130, 168, 130, 205, 130, 230]; // 'おはよ' array in SJIS
var detectedEncoding = Encoding.detect(sjisArray);
console.log('Encoding is ' + detectedEncoding); // 'Encoding is SJIS'
Example of specifying the character encoding to be detected.
If the second argument encodings is specified, returns true when it is the specified character encoding, false otherwise.
var sjisArray = [130, 168, 130, 205, 130, 230];
var isSJIS = Encoding.detect(sjisArray, 'SJIS');
if (isSJIS) {
console.log('Encoding is SJIS');
}
Example of converting a character code array to Shift_JIS from UTF-8.
var utf8Array = [227, 129, 130]; // "あ" in UTF-8
var sjisArray = Encoding.convert(utf8Array, 'SJIS', 'UTF8');
console.log(sjisArray); // [130, 160] ("あ" in SJIS)
TypedArray such as Uint8Array, and Buffer of Node.js can be converted in the same usage.
var utf8Array = new Uint8Array([227, 129, 130]);
Encoding.convert(utf8Array, 'SJIS', 'UTF8');
Converts character encoding by auto-detecting the encoding name of the source.
// The character encoding is automatically detected when the from_encoding argument is omitted
var utf8Array = [227, 129, 130];
var sjisArray = Encoding.convert(utf8Array, 'SJIS');
// Or explicitly specify 'AUTO' to auto-detecting
sjisArray = Encoding.convert(utf8Array, 'SJIS', 'AUTO');
to_encoding as an objectYou can specify the second argument to_encoding as an object for improving readability.
var sjisArray = Encoding.convert(utf8Array, {
to: 'SJIS', // to_encoding
from: 'UTF8' // from_encoding
});
type optionconvert returns an array by default, but you can change the return type by specifying the type option.
Also, if the argument data is passed as a string and the type option is not specified, then type ='string' is assumed (returns as a string).
var sjisArray = [130, 168, 130, 205, 130, 230]; // 'おはよ' array in SJIS
var unicodeString = Encoding.convert(sjisArray, {
to: 'UNICODE',
from: 'SJIS',
type: 'string' // Specify 'string' to return as string
});
console.log(unicodeString); // 'おはよ'
The following type options are supported
Uint16Array)Characters that cannot be represented in the target character set are replaced with '?' (U+003F) by default but can be replaced with HTML entities by specifying the fallback option.
The fallback option supports the following values.
Example of specifying { fallback: 'html-entity' } option
var unicodeArray = Encoding.stringToCode('寿司🍣ビール🍺');
// No fallback specified
var sjisArray = Encoding.convert(unicodeArray, {
to: 'SJIS',
from: 'UNICODE'
});
console.log(sjisArray); // Converted to a code array of '寿司?ビール?'
// Specify `fallback: html-entity`
sjisArray = Encoding.convert(unicodeArray, {
to: 'SJIS',
from: 'UNICODE',
fallback: 'html-entity'
});
console.log(sjisArray); // Converted to a code array of '寿司🍣ビール🍺'
Example of specifying { fallback: 'html-entity-hex' } option
var unicodeArray = Encoding.stringToCode('ホッケの漢字は𩸽');
var sjisArray = Encoding.convert(unicodeArray, {
to: 'SJIS',
from: 'UNICODE',
fallback: 'html-entity-hex'
});
console.log(sjisArray); // Converted to a code array of 'ホッケの漢字は𩸽'
You can add a BOM (byte order mark) by specifying the bom option when converting to UTF16.
The default is no BOM.
var utf16Array = Encoding.convert(utf8Array, {
to: 'UTF16', // to_encoding
from: 'UTF8', // from_encoding
bom: true // Add BOM
});
UTF16 byte order is big-endian by default.
If you want to convert as little-endian, specify the { bom: 'LE' } option.
var utf16leArray = Encoding.convert(utf8Array, {
to: 'UTF16', // to_encoding
from: 'UTF8', // from_encoding
bom: 'LE' // With BOM (little-endian)
});
If you do not need BOM, use UTF16BE or UTF16LE.
UTF16BE is big-endian, and UTF16LE is little-endian, and both have no BOM.
var utf16beArray = Encoding.convert(utf8Array, {
to: 'UTF16BE',
from: 'UTF8'
});
{string} Encoding.urlEncode ( data )
URL(percent) encode.
@param {Array|TypedArray} data Target data.
@return {string} Return the encoded string.
{Array} Encoding.urlDecode ( string )
URL(percent) decode.
@param {string} string Target data.
@return {Array} Return the decoded array.
var sjisArray = [130, 177, 130, 241, 130, 201, 130, 191, 130, 205];
var encoded = Encoding.urlEncode(sjisArray);
console.log(encoded); // '%82%B1%82%F1%82%C9%82%BF%82%CD'
var decoded = Encoding.urlDecode(encoded);
console.log(decoded); // [130, 177, 130, 241, 130, 201, 130, 191, 130, 205]
{string} Encoding.base64Encode ( data )
Base64 encode.
@param {Array|TypedArray} data Target data.
@return {string} Return the Base64 encoded string.
{Array} Encoding.base64Decode ( string )
Base64 decode.
@param {string} string Target data.
@return {Array} Return the Base64 decoded array.
var sjisArray = [130, 177, 130, 241, 130, 201, 130, 191, 130, 205];
var encoded = Encoding.base64Encode(sjisArray);
console.log(encoded); // 'grGC8YLJgr+CzQ=='
var decoded = Encoding.base64Decode(encoded);
console.log(decoded); // [130, 177, 130, 241, 130, 201, 130, 191, 130, 205]
{string} Encoding.codeToString ( {Array|TypedArray} data )
Joins a character code array to string.
{Array} Encoding.stringToCode ( {string} string )
Splits string to an array of character codes.
{Array|string} Encoding.toHankakuCase ( {Array|string} data )
Convert the ascii symbols and alphanumeric characters to the zenkaku symbols and alphanumeric characters.
{Array|string} Encoding.toZenkakuCase ( {Array|string} data )
Convert to the zenkaku symbols and alphanumeric characters from the ascii symbols and alphanumeric characters.
{Array|string} Encoding.toHiraganaCase ( {Array|string} data )
Convert to the zenkaku hiragana from the zenkaku katakana.
{Array|string} Encoding.toKatakanaCase ( {Array|string} data )
Convert to the zenkaku katakana from the zenkaku hiragana.
{Array|string} Encoding.toHankanaCase ( {Array|string} data )
Convert to the hankaku katakana from the zenkaku katakana.
{Array|string} Encoding.toZenkanaCase ( {Array|string} data )
Convert to the zenkaku katakana from the hankaku katakana.
{Array|string} Encoding.toHankakuSpace ({Array|string} data )
Convert the em space(U+3000) to the single space(U+0020).
{Array|string} Encoding.toZenkakuSpace ( {Array|string} data )
Convert the single space(U+0020) to the em space(U+3000).
This sample reads the text file written in Shift_JIS as binary data, and displays a string that is converted to Unicode by Encoding.convert.
var req = new XMLHttpRequest();
req.open('GET', '/my-shift_jis.txt', true);
req.responseType = 'arraybuffer';
req.onload = function (event) {
var buffer = req.response;
if (buffer) {
// Shift_JIS Array
var sjisArray = new Uint8Array(buffer);
// Convert encoding to UNICODE (JavaScript Unicode Array).
var unicodeArray = Encoding.convert(sjisArray, {
to: 'UNICODE',
from: 'SJIS'
});
// Join to string.
var unicodeString = Encoding.codeToString(unicodeArray);
console.log(unicodeString);
}
};
req.send(null);
Reads file using the File APIs.
Detect file encoding and convert to Unicode, and display it.
<input type="file" id="file">
<div id="encoding"></div>
<textarea id="result" rows="5" cols="80"></textarea>
<script>
function onFileSelect(event) {
var file = event.target.files[0];
var reader = new FileReader();
reader.onload = function(e) {
var codes = new Uint8Array(e.target.result);
var encoding = Encoding.detect(codes);
document.getElementById('encoding').textContent = encoding;
// Convert encoding to unicode
var unicodeString = Encoding.convert(codes, {
to: 'unicode',
from: encoding,
type: 'string'
});
document.getElementById('result').value = unicodeString;
};
reader.readAsArrayBuffer(file);
}
document.getElementById('file').addEventListener('change', onFileSelect, false);
</script>
We welcome contributions from everyone. For bug reports and feature requests, please create an issue on GitHub.
Please run $ npm run test before the pull request to confirm there are no errors.
We only accept requests without errors.
MIT
FAQs
Convert and detect character encoding in JavaScript
We found that encoding-japanese demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Research
A malicious npm package disguised as a WhatsApp client is exploiting authentication flows with a remote kill switch to exfiltrate data and destroy files.
Security News
PyPI now supports digital attestations, enhancing security and trust by allowing package maintainers to verify the authenticity of Python packages.
Security News
GitHub removed 27 malicious pull requests attempting to inject harmful code across multiple open source repositories, in another round of low-effort attacks.