Uniqhtt
Welcome to Uniqhtt, a powerful and versatile Node.js library designed to handle advanced HTTP requests, response processing, and web automation. This library stands out with its comprehensive support for cookie management, HTML parsing, CSS transformation, and Wget-like recursive website cloning features. Uniqhtt provides a solid solution for web scraping, data extraction, and automation tasks, making it a valuable tool for developers working in diverse environments ranging from Node.js to serverless platforms and edge computing.
Table of Contents
Key Features
Uniqhtt brings a suite of robust features to empower your HTTP requests and web automation workflows:
- Cross-Platform Compatibility: Runs seamlessly in Node.js environments, browsers, and edge computing platforms.
- Advanced Cookie Management: Utilizes
tough-cookies
for sophisticated session handling, making it easy to manage and persist cookies. - HTML Parsing & Manipulation: Powered by
jsdom
, enabling developers to parse and manipulate HTML structures with ease. - CSS Processing: Supports CSS transformations and processing via
postcss
. - Recursive Website Cloning: Emulate Wget-like behavior for cloning entire websites with options for recursion, rate limiting, and user-agent customization.
- Flexible Storage Options: Store data locally, on S3-compatible storage (including Cloudflare R2), or via FTP.
- Wget-like Functionality: Clone websites, respect robots.txt, and support recursive downloads with granular control.
- Customizable User Agents: Define custom user agents to mimic browser requests or other clients.
- Rate Limiting: Prevent overloading servers with configurable rate limits.
- Parallel Downloads: Handle multiple downloads concurrently to increase performance.
- Error Handling: Advanced retry mechanisms, detailed error reporting, and customizable error handling.
- TypeScript Support: Full support for TypeScript, offering type safety and enhanced development experience.
Installation
To install Uniqhtt, use npm or yarn:
npm install uniqhtt
Or, using yarn:
yarn add uniqhtt
This will install Uniqhtt along with its dependencies, including jsdom
, fetch-cookie
, and postcss
.
Basic Usage
Uniqhtt provides a simplified API for making HTTP requests similar to popular libraries like Axios, while offering advanced features for handling cookies, scraping, and cloning websites.
Example: Simple GET Request
const uniqhtt = require('uniqhtt');
uniqhtt.get('https://example.com')
.then(response => {
console.log(response.data);
})
.catch(error => {
console.error('Error occurred:', error);
});
Example: POST Request with Cookies
uniqhtt.post('https://example.com/login', {
data: {
username: 'user',
password: 'pass',
},
cookies: true
})
.then(response => {
console.log('Login successful:', response.data);
})
.catch(error => {
console.error('Login failed:', error);
});
For a comprehensive guide on using Uniqhtt, keep reading the detailed documentation below.
HTTP Methods
Uniqhtt supports all major HTTP methods, enabling it to cover a wide range of use cases for interacting with RESTful APIs or handling custom HTTP interactions.
- GET: Retrieve data from a server.
- POST: Submit data to be processed by a server.
- PUT: Update existing data on a server.
- DELETE: Remove data from a server.
- PATCH: Apply partial modifications to data.
- HEAD: Retrieve metadata about a resource.
- OPTIONS: Get supported HTTP methods for a given URL.
Request Options
Uniqhtt's request options mirror the popular Axios-like structure, making it easy for developers familiar with other libraries to integrate.
Common Options:
- url: The URL to which the request is sent (string or URL object).
- method: The HTTP method (default:
GET
). - headers: An object representing custom HTTP headers to send along with the request.
- body: The body of the request, used in methods like POST, PUT, and PATCH.
- params: URL parameters to append to the request URL (object).
- timeout: Request timeout in milliseconds.
- followRedirect: Automatically follow HTTP 3xx responses (boolean).
- maxRedirects: The maximum number of redirects to follow.
- compress: Request a compressed response if supported by the server (boolean).
- agent: Custom HTTP or HTTPS agent for more advanced network handling.
- auth: Basic authentication credentials (object with
username
and password
). - responseType: Expected type of response (
json
, text
, arraybuffer
, etc.). - validateStatus: Function to determine if the HTTP response status code should be considered successful.
- transformRequest: Function to modify request data before sending it to the server.
- transformResponse: Function to process response data before returning it to the caller.
uniqhtt.get('https://api.example.com/data', {
headers: {
'Authorization': 'Bearer token',
'Content-Type': 'application/json'
}
})
.then(response => console.log(response.data));
HTTPS Options
For secure communication, Uniqhtt offers flexible HTTPS options to handle advanced scenarios, such as mutual TLS.
- rejectUnauthorized: Disables SSL certificate verification (boolean).
- cert: Client certificate for mutual TLS authentication.
- key: Client key for mutual TLS.
- ca: Custom Certificate Authority (CA) bundle.
- pfx: PKCS#12 formatted key and certificate chain.
- passphrase: Passphrase for encrypted private key or PFX file.
Cookie Management
Uniqhtt integrates tough-cookies for advanced cookie handling, making it a great tool for web scraping or interacting with stateful HTTP sessions.
Key Cookie Management Methods
- setCookies(cookies, startNew): Set cookies for subsequent requests.
- getCookies(): Retrieve current cookies in use.
- clearCookies(): Clear all cookies.
- parseCookies(cookieString, url): Parse a raw cookie string into individual cookies based on the provided URL.
Example: Persistent Session with Cookies
uniqhtt.setCookies([{ name: 'session', value: 'abc123', domain: 'example.com' }]);
uniqhtt.get('https://example.com/dashboard')
.then(response => {
console.log('Dashboard data:', response.data);
});
Wget-like Options
Uniqhtt can emulate Wget functionality, enabling recursive website downloading, link rewriting, and local storage of complete websites.
Key Options:
- recursive: Download recursively (boolean).
- level: Set the depth of recursion.
- waitBetweenRequests: Pause between requests (milliseconds).
- limitRate: Throttle the download speed (bytes per second).
- userAgent: Specify a custom user agent string.
- convertLinks: Convert links from relative to absolute.
- retryAttempts: Set the number of retries for failed requests.
- cookies: Enable or disable cookie support during scraping.
Example: Cloning a Website
uniqhtt.clone('https://example.com', {
recursive: true,
level: 3,
convertLinks: true,
outputDirectory: './website_clone'
})
.then(() => console.log('Website cloned successfully!'))
.catch(error => console.error('Error while cloning website:', error));
Storage Options
Uniqhtt supports various storage options to fit different needs for persisting downloaded content.
- Local Storage: Save data directly to the local file system.
- S3-Compatible Storage: Compatible with AWS S3 and services like Cloudflare R2.
- s3Config: Configuration object with
bucket
, region
, accessKeyId
, and secretAccessKey
.
- FTP: Save content via FTP.
- ftpConfig: Configuration object with
host
, port
, user
, and password
.
Advanced Features
Uniqhtt offers several advanced features to enhance its capabilities:
- Parallel Downloads: Support for downloading multiple files simultaneously.
- Rate Limiting: Prevent overwhelming servers with too many requests too quickly.
- OnProgress & OnFinished Callbacks: Track progress and completion of long-running tasks.
- Uniqhtt Instance for Cloudflare Workers: Use the library with Cloudflare R2 by passing an R2 bucket to the
createUniqhtt()
function.
Error Handling
Uniqhtt provides comprehensive error handling, ensuring developers have access to all necessary details when something goes wrong:
- Detailed Error Information: Errors include status codes, request details, and response data.
- Custom Error Classes: Handle specific error scenarios more effectively by extending built-in JavaScript error types to provide better context. For example, a
RequestError
might include details about the failed HTTP request, while a ResponseError
could provide insight into the server response that caused the failure. - Retry Mechanisms: Automatically retry failed requests based on customizable retry logic. You can specify the number of retry attempts and the delay between retries, making Uniqhtt robust for unstable network environments or transient server issues.
Example: Custom Error Handling with Retries
uniqhtt.get('https://example.com/data', {
retryAttempts: 3,
retryDelay: 1000
})
.then(response => {
console.log('Data received:', response.data);
})
.catch(error => {
if (error.isRetryError) {
console.error('Request failed after multiple retries:', error);
} else {
console.error('An unexpected error occurred:', error);
}
});
TypeScript Support
Uniqhtt is fully compatible with TypeScript, offering a strong typing system that enhances developer experience by catching errors at compile time and providing better IntelliSense support. TypeScript definitions are bundled with the package, ensuring you don’t need to install additional typings.
Example: TypeScript Usage
import uniqhtt from 'uniqhtt';
uniqhtt.get('https://example.com')
.then((response: uniqhtt.UniqhttResponse<string>) => {
console.log(response.data);
})
.catch((error: uniqhtt.UniqhttError) => {
console.error('Error occurred:', error.message);
});
TypeScript Types Overview
- UniqhttRequestConfig: Interface defining the options for making HTTP requests.
- UniqhttResponse: Interface for handling the structure of HTTP responses, with generic typing to represent the expected response data format.
- UniqhttError: Custom error type to encapsulate errors encountered during requests.
- UniqhttCookieOptions: Interface defining cookie-related options for requests.
By utilizing TypeScript, Uniqhtt ensures that developers can take full advantage of type safety, making it easier to spot errors early in the development process and ensuring higher quality code.
Contributing
We welcome contributions to Uniqhtt! Whether you're fixing bugs, improving documentation, or adding new features, your input is valuable.
Steps for Contributing
- Fork the repository on GitHub.
- Create a new branch for your feature or bugfix.
- Write clear commit messages and document any changes made.
- Ensure all new features include appropriate unit tests and documentation updates.
- Open a pull request, clearly describing the changes and providing context if necessary.
Before submitting, make sure your code passes all tests and linting checks to ensure consistency and quality.
Conclusion
Uniqhtt is a comprehensive, advanced HTTP client for Node.js, offering a range of features designed to handle everything from basic HTTP requests to complex web scraping and automation tasks. Its support for cookie management, recursive website cloning, parallel downloads, flexible storage, and advanced error handling makes it a powerful tool for developers working on a wide variety of projects.
Whether you're building a scraping engine, an API client, or an automated testing tool, Uniqhtt provides the tools you need in a simple, highly customizable package. With features like TypeScript support, retry mechanisms, and seamless cookie handling, Uniqhtt is designed to handle modern web interactions efficiently and securely.
For more detailed examples, head over to the [official documentation] and start exploring all that Uniqhtt has to offer!
Happy coding! 🎉
License
Uniqhtt is released under the MIT License. See the LICENSE file for more details.