tldts

tldts - Blazing Fast URL Parsing

tldts is a JavaScript library to extract hostnames, domains, public suffixes, top-level domains and subdomains from URLs.

Features:

• Tuned for performance (order of 0.1 to 1 μs per input)
• Handles both URLs and hostnames
• Full Unicode/IDNA support
• Support parsing email addresses
• Detect IPv4 and IPv6 addresses
• Continuously updated version of the public suffix list
• TypeScript, ships with umd, esm, cjs bundles and type definitions
• Small bundles and small memory footprint
• Battle tested: full test coverage and production use

Install

npm install --save tldts

Usage

Using the command-line interface:

$ npx tldts 'http://www.writethedocs.org/conf/eu/2017/'
{
  "domain": "writethedocs.org",
  "domainWithoutSuffix": "writethedocs",
  "hostname": "www.writethedocs.org",
  "isIcann": true,
  "isIp": false,
  "isPrivate": false,
  "publicSuffix": "org",
  "subdomain": "www"
}

Programmatically:

const { parse } = require('tldts');

// Retrieving hostname related informations of a given URL
parse('http://www.writethedocs.org/conf/eu/2017/');
// { domain: 'writethedocs.org',
//   domainWithoutSuffix: 'writethedocs',
//   hostname: 'www.writethedocs.org',
//   isIcann: true,
//   isIp: false,
//   isPrivate: false,
//   publicSuffix: 'org',
//   subdomain: 'www' }

Modern ES6 modules import is also supported:

import { parse } from 'tldts';

Alternatively, you can try it directly in your browser here: https://npm.runkit.com/tldts

API

• tldts.parse(url | hostname, options)
• tldts.getHostname(url | hostname, options)
• tldts.getDomain(url | hostname, options)
• tldts.getFullDomain(url | hostname, options)
• tldts.getPublicSuffix(url | hostname, options)
• tldts.getSubdomain(url, | hostname, options)
• tldts.getDomainWithoutSuffix(url | hostname, options)

The behavior of tldts can be customized using an options argument for all the functions exposed as part of the public API. This is useful to both change the behavior of the library as well as fine-tune the performance depending on your inputs.

{
  // Use suffixes from ICANN section (default: true)
  allowIcannDomains: boolean;
  // Use suffixes from Private section (default: false)
  allowPrivateDomains: boolean;
  // Extract and validate hostname (default: true)
  // When set to `false`, inputs will be considered valid hostnames.
  extractHostname: boolean;
  // Validate hostnames after parsing (default: true)
  // If a hostname is not valid, not further processing is performed. When set
  // to `false`, inputs to the library will be considered valid and parsing will
  // proceed regardless.
  validateHostname: boolean;
  // Perform IP address detection (default: true).
  detectIp: boolean;
  // Detect IANA special-use domains (RFC 6761 et al.) and expose the result as
  // `isSpecialUse` (default: false). Off by default so the common path does no
  // extra work; the field stays `null` unless this is enabled.
  detectSpecialUse: boolean;
  // Assume that both URLs and hostnames can be given as input (default: true)
  // If set to `false` we assume only URLs will be given as input, which
  // speed-ups processing.
  mixedInputs: boolean;
  // Specifies extra valid suffixes (default: null)
  validHosts: string[] | null;
}

The parse method returns handy properties about a URL or a hostname.

const tldts = require('tldts');

tldts.parse('https://spark-public.s3.amazonaws.com/dataanalysis/loansData.csv');
// { domain: 'amazonaws.com',
//   domainWithoutSuffix: 'amazonaws',
//   hostname: 'spark-public.s3.amazonaws.com',
//   isIcann: true,
//   isIp: false,
//   isPrivate: false,
//   publicSuffix: 'com',
//   subdomain: 'spark-public.s3' }

tldts.parse(
  'https://spark-public.s3.amazonaws.com/dataanalysis/loansData.csv',
  { allowPrivateDomains: true },
);
// { domain: 'spark-public.s3.amazonaws.com',
//   domainWithoutSuffix: 'spark-public',
//   hostname: 'spark-public.s3.amazonaws.com',
//   isIcann: false,
//   isIp: false,
//   isPrivate: true,
//   publicSuffix: 's3.amazonaws.com',
//   subdomain: '' }

tldts.parse('gopher://domain.unknown/');
// { domain: 'domain.unknown',
//   domainWithoutSuffix: 'domain',
//   hostname: 'domain.unknown',
//   isIcann: false,
//   isIp: false,
//   isPrivate: false,
//   publicSuffix: 'unknown',
//   subdomain: '' }

tldts.parse('https://192.168.0.0'); // IPv4
// { domain: null,
//   domainWithoutSuffix: null,
//   hostname: '192.168.0.0',
//   isIcann: null,
//   isIp: true,
//   isPrivate: null,
//   publicSuffix: null,
//   subdomain: null }

tldts.parse('https://[::1]'); // IPv6
// { domain: null,
//   domainWithoutSuffix: null,
//   hostname: '::1',
//   isIcann: null,
//   isIp: true,
//   isPrivate: null,
//   publicSuffix: null,
//   subdomain: null }

tldts.parse('[e-mail]'); // email
// { domain: 'emailprovider.co.uk',
//   domainWithoutSuffix: 'emailprovider',
//   hostname: 'emailprovider.co.uk',
//   isIcann: true,
//   isIp: false,
//   isPrivate: false,
//   publicSuffix: 'co.uk',
//   subdomain: '' }

Property Name	Type	Description
hostname	str	hostname of the input extracted automatically
domain	str	Domain (tld + sld)
domainWithoutSuffix	str	Domain without public suffix
subdomain	str	Sub domain (what comes after domain)
publicSuffix	str	Public Suffix (tld) of hostname
isIcann	bool	Does TLD come from ICANN part of the list
isPrivate	bool	Does TLD come from Private part of the list
isIP	bool	Is hostname an IP address?
isSpecialUse	bool	Is hostname an IANA special-use domain?

Special-use domains (RFC 6761 / IANA)

Set { detectSpecialUse: true } to flag reserved special-use names such as localhost, *.test, *.local, *.onion, and home.arpa via the isSpecialUse result field. isIcann/isPrivate don't identify these: most aren't in the Public Suffix List, and the few that are (e.g. onion, home.arpa) appear there as ordinary ICANN suffixes. The field is null unless the option is enabled, so the default path does no extra work:

parse('http://printer.local/', { detectSpecialUse: true });
// { ...
//   isSpecialUse: true,
//   publicSuffix: 'local',
//   subdomain: '' }

The list tracks the IANA Special-Use Domain Names registry.

Single purpose methods

These methods are shorthands if you want to retrieve only a single value (and will perform better than parse because less work will be needed).

getHostname(url | hostname, options?)

Returns the hostname from a given string.

const { getHostname } = require('tldts');

getHostname('google.com'); // returns `google.com`
getHostname('fr.google.com'); // returns `fr.google.com`
getHostname('fr.google.google'); // returns `fr.google.google`
getHostname('foo.google.co.uk'); // returns `foo.google.co.uk`
getHostname('t.co'); // returns `t.co`
getHostname('fr.t.co'); // returns `fr.t.co`
getHostname(
  'https://user:[e-mail]:8080/some/path?and&query#hash',
); // returns `example.co.uk`

getDomain(url | hostname, options?)

Returns the fully qualified domain from a given string.

const { getDomain } = require('tldts');

getDomain('google.com'); // returns `google.com`
getDomain('fr.google.com'); // returns `google.com`
getDomain('fr.google.google'); // returns `google.google`
getDomain('foo.google.co.uk'); // returns `google.co.uk`
getDomain('t.co'); // returns `t.co`
getDomain('fr.t.co'); // returns `t.co`
getDomain('https://user:[e-mail]:8080/some/path?and&query#hash'); // returns `example.co.uk`

getFullDomain(url | hostname, options?)

Returns the full domain — the subdomain together with the registrable domain (as returned by getDomain(...)), i.e. the whole hostname including any subdomain — or null when the input has no registrable domain (IP address, single label, bare public suffix, …). The result is the normalized hostname (lower-cased, trailing dot stripped); it is not a DNS-absolute name (no trailing root dot) and no IDNA/punycode conversion is performed.

const { getFullDomain } = require('tldts');

getFullDomain('google.com'); // returns `google.com`
getFullDomain('fr.google.com'); // returns `fr.google.com`
getFullDomain('foo.google.co.uk'); // returns `foo.google.co.uk`
getFullDomain('t.co'); // returns `t.co`
getFullDomain('fr.t.co'); // returns `fr.t.co`
getFullDomain('1.2.3.4'); // returns null (no registrable domain)
getFullDomain('localhost'); // returns null

getDomainWithoutSuffix(url | hostname, options?)

Returns the domain (as returned by getDomain(...)) without the public suffix part.

const { getDomainWithoutSuffix } = require('tldts');

getDomainWithoutSuffix('google.com'); // returns `google`
getDomainWithoutSuffix('fr.google.com'); // returns `google`
getDomainWithoutSuffix('fr.google.google'); // returns `google`
getDomainWithoutSuffix('foo.google.co.uk'); // returns `google`
getDomainWithoutSuffix('t.co'); // returns `t`
getDomainWithoutSuffix('fr.t.co'); // returns `t`
getDomainWithoutSuffix(
  'https://user:[e-mail]:8080/some/path?and&query#hash',
); // returns `example`

getSubdomain(url | hostname, options?)

Returns the complete subdomain for a given string.

const { getSubdomain } = require('tldts');

getSubdomain('google.com'); // returns ``
getSubdomain('fr.google.com'); // returns `fr`
getSubdomain('google.co.uk'); // returns ``
getSubdomain('foo.google.co.uk'); // returns `foo`
getSubdomain('moar.foo.google.co.uk'); // returns `moar.foo`
getSubdomain('t.co'); // returns ``
getSubdomain('fr.t.co'); // returns `fr`
getSubdomain(
  'https://user:[e-mail]:443/some/path?and&query#hash',
); // returns `secure`

getPublicSuffix(url | hostname, options?)

Returns the public suffix for a given string.

const { getPublicSuffix } = require('tldts');

getPublicSuffix('google.com'); // returns `com`
getPublicSuffix('fr.google.com'); // returns `com`
getPublicSuffix('google.co.uk'); // returns `co.uk`
getPublicSuffix('s3.amazonaws.com'); // returns `com`
getPublicSuffix('s3.amazonaws.com', { allowPrivateDomains: true }); // returns `s3.amazonaws.com`
getPublicSuffix('tld.is.unknown'); // returns `unknown`

Troubleshooting

Retrieving subdomain of localhost and custom hostnames

tldts methods getDomain and getSubdomain are designed to work only with known and valid TLDs. This way, you can trust what a domain is.

localhost is a valid hostname but not a TLD. You can pass additional options to each method exposed by tldts:

const tldts = require('tldts');

tldts.getDomain('localhost'); // returns null
tldts.getSubdomain('vhost.localhost'); // returns null

tldts.getDomain('localhost', { validHosts: ['localhost'] }); // returns 'localhost'
tldts.getSubdomain('vhost.localhost', { validHosts: ['localhost'] }); // returns 'vhost'

Updating the TLDs List

tldts made the opinionated choice of shipping with a list of suffixes directly in its bundle. There is currently no mechanism to update the lists yourself, but we make sure that the version shipped is always up-to-date.

If you keep tldts updated, the lists should be up-to-date as well!

Performance

tldts is the fastest JavaScript library available for parsing hostnames. It is able to parse millions of inputs per second (typically 2-3M depending on your hardware and inputs). It also offers granular options to fine-tune the behavior and performance of the library depending on the kind of inputs you are dealing with (e.g.: if you know you only manipulate valid hostnames you can disable the hostname extraction step with { extractHostname: false }).

Please see this detailed comparison with other available libraries.

Contributors

tldts is based upon the excellent tld.js library and would not exist without the many contributors who worked on the project:

This project would not be possible without the amazing Mozilla's public suffix list. Thank you for your hard work!

License

MIT License.

Similar packages

Other packages similar to tldts

psl

The 'psl' package is a similar tool that provides functions to parse domain names and extract the public suffix. It is often used for similar purposes as 'tldts', such as URL validation and domain categorization. However, 'tldts' offers more comprehensive parsing capabilities and additional features like subdomain extraction.

url-parse

The 'url-parse' package is a robust URL parser that can decompose URLs into their constituent parts. While it provides general URL parsing capabilities, it does not specialize in domain-specific parsing like 'tldts'. 'tldts' offers more focused functionality for domain and TLD extraction.

parse-domain

The 'parse-domain' package is another tool for parsing domain names and extracting subdomains, domains, and TLDs. It is similar to 'tldts' but may not be as actively maintained or feature-rich. 'tldts' provides a more modern and comprehensive solution for domain parsing.