Security News
Research
Data Theft Repackaged: A Case Study in Malicious Wrapper Packages on npm
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
flightplan-tool
Advanced tools
Flightplan is a Javascript library that makes it easy to scrape and parse airline websites for award inventory. It uses Puppeteer for scraping, which is built on top of headless Chrome, meaning it behaves just like a full-fledged Chrome browser, but it can be run from the command line with no visible window (allowing you to use your computer for other things). Furthermore, it can run on any platform supported by headless Chrome, which is just about everything (Windows, Mac, or Linux).
If you're sitting on a pile of airline miles or credit card points, you know that redeeming them can be difficult. Often, planning for my own trips, I would spend hours clicking through an airline's website, searching for available awards, while writing down what I found in a notebook. Eventually, I decided to automate that process, so I could free up my time. Flightplan doesn't scrape much faster than a human would, it simply will do it for hours on end without complaining or making mistakes. This can make planning complex award itineraries much less stressful!
Disclaimer: Scraping is generally against an airline's website's terms of service. As mentioned above, Flightplan typically doesn't place more load on a website than a normal human would, but unlike the human, it can run 24/7. So please use responsibly! Use of any scraping tool (or even excessive non-automated usage) can cause an airline to temporarily (or permanently) ban your IP or member account.
Airline | Website | Search | Parse |
---|---|---|---|
CX (Cathay Pacific) | AsiaMiles | :white_check_mark: | :white_check_mark: |
KE (Korean Air) | SKYPASS | :white_check_mark: | :white_check_mark: |
NH (All Nippon Airways) | ANA Mileage Club | :white_check_mark: | :white_check_mark: |
SQ (Singapore Airlines) | KrisFlyer | :white_check_mark: | :white_check_mark: |
To use Flightplan, there are a few prerequisites that must be installed:
powershell
, right-click on "Windows PowerShell" and select "Run as Administrator"). Then run npm install --add-python-to-path --global --production windows-build-tools
. Once the install is complete, you must restart your computer.If you simply wish to install and run the Flightplan tools, use:
yarn global add flightplan-tool
# Create a directory to store data in
mkdir flightplan
cd flightplan
# Explain all available commands
flightplan
If you're a developer, you can install Flightplan to an existing Javascript project with:
yarn add flightplan-tool
# or "npm i flightplan-tool"
Note: When you install Flightplan, it will be bundled with a recent version of Chromium automatically (so you do not need Chrome installed on your machine to use Flightplan).
For developers who wish to use Flightplan in their own Javascript projects, skip to the Library Usage section below. For everyone else, you're in the right place! If you'd like a video introduction to using Flightplan, check out the tutorial on YouTube:
http://www.youtube.com/watch?feature=player_embedded&v=QMtiucIPOxs
# Create a directory to store data in
mkdir flightplan
cd flightplan
# Explain all available commands
flightplan
# Run a search
flightplan search
The first time you run the search
command, you will be prompted to create a config/accounts.json
file. You must enter valid credentials in this file, for any engine you will be using, or login will fail.
After you've run a search, you should see the raw HTML and screenshots in the data
subdirectory. To extract award fares from the HTML and populate the database, run flightplan parse
.
When you are ready to run the React web UI, run both flightplan server
and flightplan client
. When the React app has finished loading, a browser will automatically be opened, pointing to http://localhost:3000/
.
With Flightplan, you specify an airline and get back an engine, which supports two operations: searching and parsing.
This is useful, because searching is expensive, but parsing is cheap. So it makes sense to search once, but be able to parse many times (perhaps due to bug fixes or new features being added).
const fp = require('flightplan');
const cx = fp.new('cx');
(async () => {
// Must call initialize before searching
await cx.initialize({ username: '1234567890', 'password': 'passw0rd' });
// Do a one-way search (replace credentials with real ones below)
const { responses, error } = await cx.search({
fromCity: 'HKG', toCity: 'LHR',
departDate: '2019-03-06', cabin: 'first'
});
// Check for an error
if (error) {
console.log(error);
return;
}
// Parse out awards from the responses
const { awards } = cx.parse(responses);
console.log(awards);
})();
You can also instruct the search engine to save both the HTML output, and even screenshots! :tada: This makes debugging what might've gone wrong later much easier. Let's try it out:
const fp = require('flightplan');
const sq = fp.new('sq');
(async () => {
await sq.initialize({ username: '1234567890', password: '123456' });
const { htmlFiles, screenshots, fileCount, error } = await sq.search({
fromCity: 'SIN', toCity: 'HKG',
departDate: '2019-03-06', cabin: 'business',
htmlFile: 'output.html', screenshot: 'output.jpg'
});
if (!error) {
console.log('Files Saved:', fileCount);
console.log('HTML:', htmlFiles);
console.log('Screenshots:', screenshots);
}
})();
More API details to come later...
FAQs
Search for award inventory using Node.js and Headless Chrome!
The npm package flightplan-tool receives a total of 3 weekly downloads. As such, flightplan-tool popularity was classified as not popular.
We found that flightplan-tool demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.
Security News
The Ultralytics' PyPI Package was compromised four times in one weekend through GitHub Actions cache poisoning and failure to rotate previously compromised API tokens.