New Case Study:See how Anthropic automated 95% of dependency reviews with Socket.Learn More →
Socket
Sign inDemoInstall
Socket
Sign inDemoInstall

precious-data

Package Overview
Dependencies
Maintainers
1
Versions
4
Alerts
File Explorer

Advanced tools

License
Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

precious-data - npm Package Compare versions

Comparing version 0.2.0 to 0.3.0

.gitignore~
fixtures/HMK_01-001.html

520

docs/index.md

@@ -1,65 +0,519 @@

# Precious Data
## Classes
*a.k.a. Pirate Memories because that's more fun.*
<dl>
<dt><a href="#Ripper">Ripper</a></dt>
<dd></dd>
</dl>
Precious Data is a project to copy and mirror trading card data from the official Precious Memories (P-Memories) website.
## Functions
P-Memories is a Japanese exclusive card game with card titles, text, and flavor text primarily written in Hiragana, Katakana, and Kanji.
<dl>
<dt><a href="#splitTextList">splitTextList(textList)</a> ⇒ <code>Array</code></dt>
<dd><p>splitTextList</p>
<p>converts a {String} list such as &#39;&#39; to an array, using comma as delimiter</p>
</dd>
<dt><a href="#parseCardDataFromHtml">parseCardDataFromHtml(html)</a> ⇒ <code>Promise</code></dt>
<dd><p>parseCardDataFromHtml</p>
</dd>
<dt><a href="#parseCardId">parseCardId(cardId)</a> ⇒ <code>Promise</code></dt>
<dd><p>parseCardId</p>
<p>parses the card ID and returns an object containing</p>
<ul>
<li>setAbbr</li>
<li>release</li>
<li>number</li>
<li>num</li>
<li>id</li>
<li>variation</li>
</ul>
</dd>
</dl>
The goal of this project is to create a dataset of English translated P-Memories cards. With this dataset, myself and third-parties will be empowered to create apps which encourage English speaking players to better enjoy P-Memories card game.
<a name="Ripper"></a>
## What the Precious Data project does
## Ripper
**Kind**: global class
### Data Ripper
* [Ripper](#Ripper)
* [.buildImagePath(imageUrl)](#Ripper+buildImagePath) ⇒ <code>Promise</code>
* [.buildCardDataPath(cardUrl)](#Ripper+buildCardDataPath) ⇒ <code>Promise</code>
* [.downloadImage(targetUrl)](#Ripper+downloadImage) ⇒ <code>Promise</code>
* [.ripSetData(setUrl, dataAcc)](#Ripper+ripSetData) ⇒ <code>Promise</code>
* [.isValidPMemoriesUrl(url)](#Ripper+isValidPMemoriesUrl) ⇒ <code>Boolean</code>
* [.ripCardData(cardRef)](#Ripper+ripCardData) ⇒ <code>Promise</code>
* [.lookupCardUrl(cardId)](#Ripper+lookupCardUrl) ⇒ <code>Promise</code>
* [.getCardUrlsFromSetPage()](#Ripper+getCardUrlsFromSetPage)
* [.getSets()](#Ripper+getSets) ⇒ <code>Promise</code>
* [.getSetNames()](#Ripper+getSetNames) ⇒ <code>Promise</code>
* [.getSetUrls()](#Ripper+getSetUrls) ⇒ <code>Promise</code>
* [.ripAll()](#Ripper+ripAll) ⇒ <code>Promise</code>
* [.ripUrl(url)](#Ripper+ripUrl) ⇒ <code>Promise</code>
* [.saveCardData(cardData)](#Ripper+saveCardData) ⇒ <code>Promise</code>
* [.isLocalData(cardData)](#Ripper+isLocalData) ⇒ <code>Promise</code>
* [.identifyUrl(url)](#Ripper+identifyUrl) ⇒ <code>String</code>
* [.rip()](#Ripper+rip) ⇒ <code>Promise</code>
* [.loadSetAbbrIndex()](#Ripper+loadSetAbbrIndex) ⇒ <code>Promise</code>
* [.getSetSuggestion(setAbbrIndex, setSuggestion)](#Ripper+getSetSuggestion) ⇒ <code>String</code> \| <code>undefined</code>
* [.getSetUrlFromSetAbbr(setAbbr)](#Ripper+getSetUrlFromSetAbbr) ⇒ <code>Promise</code>
* [.getImageUrlFromEachSet()](#Ripper+getImageUrlFromEachSet) ⇒ <code>Promise</code>
* [.createSetAbbreviationIndex()](#Ripper+createSetAbbreviationIndex) ⇒ <code>Promise</code>
* [.getSetAbbrFromImageUrl(imageUrl)](#Ripper+getSetAbbrFromImageUrl) ⇒ <code>String</code>
* [.getFirstCardImageUrl(setUrl)](#Ripper+getFirstCardImageUrl) ⇒ <code>Promise</code>
The data ripper uses Javascript to request, parse, and organize card data from P-Memories.com. Once parsed, card data is saved on disk in set-labelled folders as JSON files. Because of the nature of this script, this functionality is throttled, and not meant to be used often, as doing so too fast or too often could be seen as malicious.
<a name="Ripper+buildImagePath"></a>
Additionally, card images are downloaded and stored on disk in the same set-labelled folders.
### ripper.buildImagePath(imageUrl) ⇒ <code>Promise</code>
buildImagePath
For example, cards belonging to the Hatsune Miku card set will be saved in the abbreviated set-labelled folder, `HMK`. SSSS.GRIDMAN cards are saved in the set-labelled folder, `SSSS`.
Accepts an image URL as it's parameter and returns
a string of the perfect path on disk where the image should be saved.
The perfect path includes set abbreviation (ex: HMK,)
release number (ex: 01) and image name. (ex: HMK_01-001.json.)
As future updates and additions to P-Memories cards are anticipated, the ripper is smart enough to merge official data with unofficial translations. This means that the card JSON files are safe to edit manually, and work will not be lost during future data rips.
**Kind**: instance method of [<code>Ripper</code>](#Ripper)
**Returns**: <code>Promise</code> - - A promise that returns an array if resolved
or an error if rejected.
**Resolve**: <code>String</code> - An absolute path on disk.
**Rejects**: <code>Error</code> - An error which states the cause.
#### Usage:
| Param | Type | Description |
| --- | --- | --- |
| imageUrl | <code>String</code> | the URL to the image. |
npm run rip
**Example**
```js
buildImagePath('http://p-memories.com/images/product/SSSS/SSSS_01-001.jpg')
=> "@/data/SSSS/01/SSSS_01-001.jpg" (where @ is this project root.)
```
<a name="Ripper+buildCardDataPath"></a>
#### Advanced Usage:
### ripper.buildCardDataPath(cardUrl) ⇒ <code>Promise</code>
buildCardDataPath
Advanced usage of the CLI tool `p-data.js` allows the user to download specific sets, specific URLs, or all precious-memories cards in existence. The `-i` (incremental) flag can significantly reduce network usage by only downloading card data which has not already been downloaded.
Accepts a card URL as it's parameter and returns a string of the
perfect path on disk where the card data JSON should be saved.
The perfect path includes set abbreviation (ex: HMK,)
release number (ex: 01) and image name. (ex: HMK_01-001.json.)
##### Examples
**Kind**: instance method of [<code>Ripper</code>](#Ripper)
**Returns**: <code>Promise</code> - - A promise that returns a string if resolved
or an error if rejected.
**Resolve**: <code>String</code> - An absolute path on disk.
**Rejects**: <code>Error</code> - An error which states the cause.
Download all cards in the Hatsune Miku set.
| Param | Type | Description |
| --- | --- | --- |
| cardUrl | <code>String</code> | the URL to the card page on p-memories website. |
./p-data.js rip -s HMK
**Example**
```js
buildCardDataPath({"set": "HMK", "number": "01-001", ... })
=> "@/data/HMK/01/HMK_01-001.json" (where @ is project root)
```
<a name="Ripper+downloadImage"></a>
Download a card from a p-memories.com URL
### ripper.downloadImage(targetUrl) ⇒ <code>Promise</code>
downloadImage
./p-data.js rip -u http://p-memories.com/node/942168
Downloads an image from the internet and resolves with a buffer of the image.
Accepts a card image URL OR card URL as it's parameter.
Download a card given it's unique ID
**Kind**: instance method of [<code>Ripper</code>](#Ripper)
**Returns**: <code>Promise</code> - - A promise that returns an array if resolved
or an error if rejected.
**Resolve**: <code>Buffer</code> - A buffer of the downloaded image.
**Rejects**: <code>Error</code> - An error which states the cause.
./p-data.js rip -n ERMG_01-001
| Param | Type | Description |
| --- | --- | --- |
| targetUrl | <code>String</code> | the URL to the image or card page |
Download all P-Memories cards in existence.
<a name="Ripper+ripSetData"></a>
./p-data.js rip -a
### ripper.ripSetData(setUrl, dataAcc) ⇒ <code>Promise</code>
ripSetData
Download all cards in existence, waiting 3 seconds between each network request. Only download card data which hasn't already been downloaded.
Accepts a set URL as parameter and returns a list of card URLs
which belong in the set.
./p-data.js rip -a -t 3 -i
**Kind**: instance method of [<code>Ripper</code>](#Ripper)
**Returns**: <code>Promise</code> - - A promise that returns an Array if resolved
or an error if rejected
**Resolve**: <code>Array</code> setData - An array of objects which contain cardUrl and cardImageUrl
**Rejects**: <code>Error</code> - An error which states the cause
Show commandline usage and help
| Param | Type | Description |
| --- | --- | --- |
| setUrl | <code>String</code> | the URL to the card set |
| dataAcc | <code>Array</code> \| <code>undefined</code> | object accumulator which contains a list of card URLs and cardImageUrls. Used for recursive calls of this function during ripping of multi-page sets. |
./p-data.js -h
./p-data.js rip -h
<a name="Ripper+isValidPMemoriesUrl"></a>
## Code Documentation
### ripper.isValidPMemoriesUrl(url) ⇒ <code>Boolean</code>
isValidPMemoriesUrl
[util/ripper.js](ripper)
Returns true or false depending on whether or not a valid P-memories.com
URL was passed as parameter.
**Kind**: instance method of [<code>Ripper</code>](#Ripper)
**Returns**: <code>Boolean</code> - isValid - true if the url was p-memories.com url, false otherwise.
## Code Repository
| Param | Type |
| --- | --- |
| url | <code>String</code> |
[https://github.com/insanity54/precious-data/](https://github.com/insanity54/precious-data/}
<a name="Ripper+ripCardData"></a>
### ripper.ripCardData(cardRef) ⇒ <code>Promise</code>
ripCardData
accepts a card URL as it's parameter and returns an object containing card
data and card image URL.
**Kind**: instance method of [<code>Ripper</code>](#Ripper)
**Returns**: <code>Promise</code> - - A promise that returns an array if resolved
or an error if rejected
**Resolve**: <code>Object</code> - An object containing card data such as title,
description, rarity, type, AP, DP, image URL, etc.
**Rejects**: <code>Error</code> - An error which states the cause
| Param | Type | Description |
| --- | --- | --- |
| cardRef | <code>String</code> | A reference to a specific card. Can be a URL or a Card ID |
<a name="Ripper+lookupCardUrl"></a>
### ripper.lookupCardUrl(cardId) ⇒ <code>Promise</code>
lookupCardUrl
Accepts a card ID as parameter, and resolves the appropriate cardUrl and
cardImageUrl belonging to that card.
**Kind**: instance method of [<code>Ripper</code>](#Ripper)
**Returns**: <code>Promise</code> - - A promise that returns an object if resolved
or an error if rejected
**Resolve**: <code>Object</code> card
**Resolve**: <code>String</code> card.cardUrl - the url to the card page.
Example: http://p-memories.com/node/926791
**Resolve**: <code>String</code> card.cardImageUrl - the image url of the card.
Example: http://p-memories.com/images/product/SSSS/SSSS_01-001.jpg
| Param | Type |
| --- | --- |
| cardId | <code>String</code> |
<a name="Ripper+getCardUrlsFromSetPage"></a>
### ripper.getCardUrlsFromSetPage()
getCardUrlsFromSetPage
Accepts a cardNumber and setUrl as parameters, and returns
an object with cardUrl, and cardImageUrl.
**Kind**: instance method of [<code>Ripper</code>](#Ripper)
<a name="Ripper+getSets"></a>
### ripper.getSets() ⇒ <code>Promise</code>
getSets
Gets a list of set names and urls on the p-memories.com
**Kind**: instance method of [<code>Ripper</code>](#Ripper)
**Rejects**: <code>Error</code>
**Resolve**: <code>Array</code> - An array containing objects in the shape
{ setName, setUrl }
<a name="Ripper+getSetNames"></a>
### ripper.getSetNames() ⇒ <code>Promise</code>
getSetNames
accepts no params and returns a list of set names found on p-memories.com
**Kind**: instance method of [<code>Ripper</code>](#Ripper)
**Resolve**: <code>Array</code>
**Rejects**: <code>Error</code>
<a name="Ripper+getSetUrls"></a>
### ripper.getSetUrls() ⇒ <code>Promise</code>
getSetUrls
accepts no parameters and returns a list of all set URLs found on p-memories
website.
**Kind**: instance method of [<code>Ripper</code>](#Ripper)
**Returns**: <code>Promise</code> - - A promise that returns an array if resolved
or an error if rejected
**Resolve**: <code>Array</code> - An array containing set URLs
**Rejects**: <code>Error</code> - An error which states the cause
<a name="Ripper+ripAll"></a>
### ripper.ripAll() ⇒ <code>Promise</code>
ripAll
accepts no parameters and downloads all card data and card images
found p-memories.com.
**Kind**: instance method of [<code>Ripper</code>](#Ripper)
**Returns**: <code>Promise</code> - - A promise that returns a number if resolved
or an error if rejected
**Resolve**: <code>Number</code> - The number of card data ripped from p-memories
**Rejects**: <code>Error</code> - An error which states the cause
<a name="Ripper+ripUrl"></a>
### ripper.ripUrl(url) ⇒ <code>Promise</code>
ripUrl
Rip a resource. Used by the CLI.
url could be one of several resources.
* Card URL (defers to ripCardData)
* Set URL (defers to ripSetData)
* undefined (defers to ripAll)
**Kind**: instance method of [<code>Ripper</code>](#Ripper)
**Returns**: <code>Promise</code> - - A promise that returns a number if resolved
or an error if rejected
**Resolve**: <code>Number</code> - The number of card data ripped
**Rejects**: <code>Error</code> - An error which states the cause
| Param | Type | Description |
| --- | --- | --- |
| url | <code>String</code> | The URL to rip |
<a name="Ripper+saveCardData"></a>
### ripper.saveCardData(cardData) ⇒ <code>Promise</code>
saveCardData
Download the card data and image file
**Kind**: instance method of [<code>Ripper</code>](#Ripper)
**Returns**: <code>Promise</code> - - A promise that returns a number if resolved
or an error if rejected
**Resolve**: <code>Array</code> - An array containing result of this.downloadImage
and this.writeCardData.
**Rejects**: <code>Error</code> - An error which states the cause
| Param | Type | Description |
| --- | --- | --- |
| cardData | <code>Object</code> | The cardData |
<a name="Ripper+isLocalData"></a>
### ripper.isLocalData(cardData) ⇒ <code>Promise</code>
isLocalData
Returns a promise of True or False depending on whether or not the
card data exists on disk.
**Kind**: instance method of [<code>Ripper</code>](#Ripper)
**Resolve**: <code>Boolean</code>
**Rejects**: <code>Error</code>
| Param | Type |
| --- | --- |
| cardData | <code>Object</code> |
<a name="Ripper+identifyUrl"></a>
### ripper.identifyUrl(url) ⇒ <code>String</code>
identifyUrl
Identify the type of URL the user is sending us. Can be either:
* card URL
* Set URL
* undefined
**Kind**: instance method of [<code>Ripper</code>](#Ripper)
**Returns**: <code>String</code> - urlType - either, "card", or "set"
| Param | Type | Description |
| --- | --- | --- |
| url | <code>String</code> | the URL to identify |
<a name="Ripper+rip"></a>
### ripper.rip() ⇒ <code>Promise</code>
rip
Rip card data
Determines the correct method to use to rip card data based on input.
Defers to more specific functions for data rippage.
**Kind**: instance method of [<code>Ripper</code>](#Ripper)
**Returns**: <code>Promise</code> - - A promise that returns a string if resolved
or an error if rejected
**Resolve**: <code>String</code> - A report of ripped card data
**Rejects**: <code>Error</code> - An error which states the cause
<a name="Ripper+loadSetAbbrIndex"></a>
### ripper.loadSetAbbrIndex() ⇒ <code>Promise</code>
loadSetAbbrIndex
load the Set Abbreviation Index from disk
**Kind**: instance method of [<code>Ripper</code>](#Ripper)
**Returns**: <code>Promise</code> - - A promise that returns an Array if resolved
**Resolve**: <code>Array</code> - The set abbreviation list
**Rejects**: <code>Error</code> - An error which states the cause
**Todo**
- [ ] https://github.com/insanity54/precious-data/issues/5
<a name="Ripper+getSetSuggestion"></a>
### ripper.getSetSuggestion(setAbbrIndex, setSuggestion) ⇒ <code>String</code> \| <code>undefined</code>
getSetSuggestion
**Kind**: instance method of [<code>Ripper</code>](#Ripper)
**Returns**: <code>String</code> \| <code>undefined</code> - setSuggestion
| Param | Type |
| --- | --- |
| setAbbrIndex | <code>Array</code> |
| setSuggestion | <code>String</code> |
<a name="Ripper+getSetUrlFromSetAbbr"></a>
### ripper.getSetUrlFromSetAbbr(setAbbr) ⇒ <code>Promise</code>
getSetUrlFromSetAbbr
taking a set abbreviation as it's sole parameter, resolve a setURL
of the matching card set.
**Kind**: instance method of [<code>Ripper</code>](#Ripper)
**Returns**: <code>Promise</code> - - A promise that returns a string if resolved
or an error if rejected
**Resolve**: <code>String</code> - A p-memories.com card set URL
**Rejects**: <code>Error</code> - An error which states the cause
| Param | Type | Description |
| --- | --- | --- |
| setAbbr | <code>String</code> | The set abbreviation |
<a name="Ripper+getImageUrlFromEachSet"></a>
### ripper.getImageUrlFromEachSet() ⇒ <code>Promise</code>
getImageUrlFromEachSet
Returns an array containing the url of the first card of each set
of the card set.
**Kind**: instance method of [<code>Ripper</code>](#Ripper)
**Resolves**: <code>Array</code> - A p-memories.com card set URL
**Example**
```js
[
{
setUrl: 'http://p-memories.com/card_product_list_page?field_title_nid=919863-SSSS.GRIDMAN&s_flg=on',
sampleCardUrl: 'http://p-memories.com/images/product/SSSS/SSSS_01-001.jpg'
}
(...)
]
```
<a name="Ripper+createSetAbbreviationIndex"></a>
### ripper.createSetAbbreviationIndex() ⇒ <code>Promise</code>
createSetAbbreviationIndex
Create a mapping of set abbreviations to set urls.
This map is used to get set URLs from a set Abbreviation.
**Kind**: instance method of [<code>Ripper</code>](#Ripper)
**Resolves**: <code>String</code> - An stringified array of setAbbr/setUrl pairs
**Example**
```js
[
{
"setAbbr": "SSSS",
"setUrl": "http://p-memories.com/card_product_list_page?field_title_nid=919863-SSSS.GRIDMAN&s_flg=on"
},
...
]
```
<a name="Ripper+getSetAbbrFromImageUrl"></a>
### ripper.getSetAbbrFromImageUrl(imageUrl) ⇒ <code>String</code>
getSetAbbrFromImageUrl
Determines the set abbreviation given a card image URL.
**Kind**: instance method of [<code>Ripper</code>](#Ripper)
**Returns**: <code>String</code> - - A p-memories set abbreviation.
| Param | Type | Description |
| --- | --- | --- |
| imageUrl | <code>String</code> | A p-memories card image URL. |
<a name="Ripper+getFirstCardImageUrl"></a>
### ripper.getFirstCardImageUrl(setUrl) ⇒ <code>Promise</code>
getFirstCardImageUrl
Accepts a set URL as parameter and returns the URL of the first card in that set.
**Kind**: instance method of [<code>Ripper</code>](#Ripper)
**Returns**: <code>Promise</code> - - A promise that returns an string if resolved
or an error if rejected
**Resolve**: <code>String</code> - An image URL of the first card in the set
**Rejects**: <code>Error</code> - An error which states the cause
| Param | Type | Description |
| --- | --- | --- |
| setUrl | <code>String</code> | the URL to the card set |
<a name="splitTextList"></a>
## splitTextList(textList) ⇒ <code>Array</code>
splitTextList
converts a {String} list such as '' to an array, using comma as delimiter
**Kind**: global function
**Returns**: <code>Array</code> - list
| Param | Type |
| --- | --- |
| textList | <code>String</code> |
<a name="parseCardDataFromHtml"></a>
## parseCardDataFromHtml(html) ⇒ <code>Promise</code>
parseCardDataFromHtml
**Kind**: global function
**Returns**: <code>Promise</code> - - A promise which resolves with an object if successful
or an error if failed
**Resolve**: <code>Object</code> - An object containing card data such as title,
description, rarity, type, AP, DP, image URL, etc.
**Rejects**: <code>Error</code> - An error which states the cause
| Param | Type | Description |
| --- | --- | --- |
| html | <code>String</code> | the html to parse |
<a name="parseCardId"></a>
## parseCardId(cardId) ⇒ <code>Promise</code>
parseCardId
parses the card ID and returns an object containing
* setAbbr
* release
* number
* num
* id
* variation
**Kind**: global function
**Returns**: <code>Promise</code> - - A promise that returns an object if resolved
or an error if rejected
**Resolve**: <code>Object</code>
**Rejects**: <code>Error</code>
| Param | Type |
| --- | --- |
| cardId | <code>String</code> |

3

index.js

@@ -20,3 +20,4 @@

function collectSets() {
const setAbbrIndex = require(path.join(__dirname, 'data', 'setAbbrIndex.json'));
const json = require(path.join(__dirname, 'data', 'setAbbrIndex.json'));
const setAbbrIndex = JSON.parse(json);
return setAbbrIndex;

@@ -23,0 +24,0 @@ }

27

lib/constants.js

@@ -0,5 +1,30 @@

const path = require('path');
const rootUrl = 'http://p-memories.com';
const urlRegex = /(?:http(?:s?):\/\/)?p-memories\.com/;
const relativeUrlRegex = /\/node\/\d+/;
const cardPageRegex = /p-memories.com\/node\/\d+/;
const setPageRegex = /p-memories.com\/card_product_list_page.+field_title_nid/;
const setAbbrRegex = /product\/(.+)\//;
const imageNameRegex = /\/product\/.+\/(.+_.+-.+.jpg)/;
const releaseNameRegex = /\/product\/.+\/.+_(.+)-.+.jpg/;
const dataDir = path.join(__dirname, '..', 'data');
const setAbbrIndexPath = path.join(__dirname, '..', 'data', 'setAbbrIndex.json');
const cardIdRegex = /([\w\d-]+)(?:_| )(([\w\d]+)-([\d]+)([\w\d]*))/;
module.exports = {
rootUrl
rootUrl,
urlRegex,
relativeUrlRegex,
cardPageRegex,
setPageRegex,
setAbbrRegex,
imageNameRegex,
releaseNameRegex,
dataDir,
setAbbrIndexPath,
cardIdRegex
}

4

lib/httpAgent.js
const axios = require('axios');
// override the adapter so nock can intercept during testing
// https://github.com/nock/nock#axios
axios.defaults.adapter = require('axios/lib/adapters/http')

@@ -7,2 +10,3 @@

} = require('./misc');
const {

@@ -9,0 +13,0 @@ rootUrl

14

lib/parsers.js

@@ -5,7 +5,13 @@

const cheerio = require('cheerio');
const { rootUrl } = require('./constants');
// constants
const cardIdRegex = /([\w\d-]+)(?:_| )(([\w\d]+)-([\d]+)([\w\d]*))/;
const {
setAbbrRegex,
cardIdRegex
} = require('../lib/constants')
function normalizeUrl (url) {

@@ -73,5 +79,7 @@ if (typeof url === 'undefined') {

/** Data that I think is good which isn't explicitly in the page */
data.image = normalizeUrl($('.Images_card > img:nth-child(1)').attr('src'))
data.url = `http://p-memories.com/node/${$('body').attr('id').split('-').pop()}`;
data.setAbbr = setAbbrRegex.exec(data.image)[1];
data.setAbbr = (typeof data.image === 'undefined') ? null : setAbbrRegex.exec(data.image)[1];
let {

@@ -81,3 +89,3 @@ num,

id
} = this.parseCardId(data.image);
} = parseCardId(data.image);
data.num = num;

@@ -84,0 +92,0 @@ data.release = release;

147

lib/ripper.js

@@ -30,15 +30,16 @@ // Rip P-Memories card data and card images from P-Memories official website.

// constants
const urlRegex = /(?:http(?:s?):\/\/)?p-memories\.com/;
const relativeUrlRegex = /\/node\/\d+/;
const cardPageRegex = /p-memories.com\/node\/\d+/;
const setPageRegex = /p-memories.com\/card_product_list_page.+field_title_nid/;
const setAbbrRegex = /product\/(.+)\//;
const imageNameRegex = /\/product\/.+\/(.+_.+-.+.jpg)/;
const releaseNameRegex = /\/product\/.+\/.+_(.+)-.+.jpg/;
const dataDir = path.join(__dirname, '..', 'data');
const setAbbrIndexPath = path.join(__dirname, '..', 'data', 'setAbbrIndex.json');
const {
rootUrl,
urlRegex,
relativeUrlRegex,
cardPageRegex,
setPageRegex,
setAbbrRegex,
imageNameRegex,
releaseNameRegex,
dataDir,
setAbbrIndexPath
} = require('./constants');
class Ripper {

@@ -118,5 +119,4 @@

*
* Downloads an image from the internet and saves it to disk.
* Downloads an image from the internet and resolves with a buffer of the image.
* Accepts a card image URL OR card URL as it's parameter.
* Returns a string of the path on disk where the image was saved.
*

@@ -126,3 +126,3 @@ * @param {String} targetUrl - the URL to the image or card page

* or an error if rejected.
* @resolve {String} - A string which tells where the image was saved.
* @resolve {Buffer} - A buffer of the downloaded image.
* @rejects {Error} - An error which states the cause.

@@ -150,11 +150,4 @@ */

.then((res) => {
// ensure dir exists
let imagePath = this.buildImagePath(targetUrl);
debug(`writing image to ${imagePath}`);
return fsp.mkdir(path.dirname(imagePath), {
recursive: true
}).then(() => {
res.data.pipe(fs.createWriteStream(imagePath));
return imagePath;
})
debug('hello world!')
return res.data;
})

@@ -172,11 +165,11 @@ }

*
* @param {String} setUrl - the URL to the card set
* @param {Array|undefined} dataAcc - object accumulator which contains a list of card
* URLs and cardImageUrls.
* Used for recursive calls of this function
* during ripping of multi-page sets.
* @returns {Promise} - A promise that returns an Array if resolved
* or an error if rejected
* @param {String} setUrl - the URL to the card set
* @param {Array|undefined} dataAcc - object accumulator which contains a list of card
* URLs and cardImageUrls.
* Used for recursive calls of this function
* during ripping of multi-page sets.
* @returns {Promise} - A promise that returns an Array if resolved
* or an error if rejected
* @resolve {Array} setData - An array of objects which contain cardUrl and cardImageUrl
* @rejects {Error} - An error which states the cause
* @rejects {Error} - An error which states the cause
*/

@@ -317,59 +310,4 @@ ripSetData(setUrl, dataAcc) {

/**
* writeCardData
*
* Accepts an object containing card data, and creates a JSON string
* which is written to the appropriate location on disk.
* To prevent the ripper from destroying local english translations,
* writes merge JSON data files rather than blindly overwriting.
*
* @param {Object} cardData - the card data
* @returns {Promise} - A promise that returns an array if resolved
* or an error if rejected
* @resolve {String} - The abs location on disk where the JSON was saved.
* @rejects {Error} - An error which states the cause
*/
writeCardData(newCardData) {
let cardDataPath = this.buildCardDataPath(newCardData);
return fsp.mkdir(path.dirname(cardDataPath), {
recursive: true
}).then(() => {
let existingData = {};
try {
existingData = require(cardDataPath);
} catch (e) {
debug(`${newCardData.number} has no existing data.`);
}
let mergedData = Object.assign({}, existingData, newCardData);
return fsp.writeFile(
cardDataPath, JSON.stringify(mergedData), {
encoding: 'utf-8'
}
).then(() => {
return cardDataPath;
})
});
}
/**
* readCardData
*
* reads the card data on disk
*
* @param {String} cardId - the card ID number.
* @example
* readCardData('HMK_01-001');
* @returns {Promise} - A promise that returns an object if resolved
* or an error if rejected
* @resolve {Object} - the card data read from disk
* @rejects {Error}
*/
readCardData(cardId) {
return fsp.readFile(
path.join(__dirname, '..', 'data', 'setAbbrIndex.json'),
JSON.stringify(index)
)
}
/**
* getSets

@@ -442,16 +380,4 @@ *

/**
* ripCardDataAndSave
*/
ripCardDataAndSave(cardUrl, cardImageUrl) {
return this.ripCardData(cardUrl, cardImageUrl).then((cardData) => {
return this.saveCardData(cardData).then((writeResult) => {
if (writeResult[0]) this.dataCounter++;
if (writeResult[1]) this.imageCounter++;
})
}).catch((e) => {
debug(`${cardImageUrl} already exists locally. skipping.`)
})
}
/**

@@ -575,20 +501,4 @@ * ripAll

/**
* isLocalCard
*
* Returns a promise of True or False depending on whether or not the
* card data exists on disk.
*
* @param {String} cardId
* @returns {Promise}
* @resolve {Boolean}
* @rejects {Error}
*/
isLocalCard(cardId) {
let cardData = parseCardId(cardId);
return this.isLocalData(cardData);
}
/**

@@ -713,2 +623,5 @@ * identifyUrl

.then((setAbbrIndex) => {
debug('the following is the setAbbrIndex')
debug(setAbbrIndex)
// reject if the index doesn't exist

@@ -728,3 +641,3 @@ // @todo https://github.com/insanity54/precious-data/issues/5

if (typeof matchingPair === 'undefined') {
let errorMessage = 'matchingPair not found. '
let errorMessage = `matchingPair not found. getSetUrlFromSetAbbr() was unable to find a URL which matches "${setAbbr}" `;

@@ -794,3 +707,3 @@ // check to see if there is a set with a similar name

createSetAbbreviationIndex() {
let setAbbrIndexPath = path.join(__dirname, '..', 'data', 'setAbbrIndex.json')
let setAbbrIndexPath = path.join(__dirname, '..', 'data', 'setAbbrIndex.json');
return this.getImageUrlFromEachSet().then((imageUrls) => {

@@ -797,0 +710,0 @@ return Promise.map(imageUrls, (imageUrl) => {

15

package.json
{
"name": "precious-data",
"version": "0.2.0",
"version": "0.3.0",
"description": "Precious Memories Card Data",

@@ -9,5 +9,6 @@ "main": "index.js",

"dev": "npx supervisor index",
"jsdoc": "npx jsdoc-to-markdown ./util/ripper.js > ./docs/ripper.md",
"test": "NOCK_BACK_MODE=lockdown jest --verbose",
"fixtures": "NOCK_BACK_MODE=record jest"
"jsdoc": "jsdoc2md ./lib/*.js > ./docs/index.md",
"test": "jest --verbose",
"fixtures": "NOCK_BACK_MODE=record jest",
"coverage": "NOCK_BACK_MODE=lockdown jest --coverage && coveralls < coverage/lconv.info"
},

@@ -27,5 +28,8 @@ "keywords": [

"devDependencies": {
"coveralls": "^3.1.1",
"jest": "^26.6.0",
"jest-extended": "^0.11.5",
"nock": "^13.0.4"
"jsdoc-to-markdown": "^7.0.1",
"nock": "^13.0.4",
"nock-record": "^0.3.9"
},

@@ -39,2 +43,3 @@ "dependencies": {

"globby": "^11.0.3",
"jest-nock": "^0.2.1",
"jest-tobetype": "^1.2.3",

@@ -41,0 +46,0 @@ "yargs": "^16.0.3"

15

README.md

@@ -5,2 +5,4 @@ # Precious Data

[![Coverage Status](https://coveralls.io/repos/github/insanity54/precious-data/badge.svg?branch=master)](https://coveralls.io/github/insanity54/precious-data?branch=master)
*a.k.a. Pirate Memories because that's more fun.*

@@ -24,3 +26,3 @@

#### Installation:
## Installation:

@@ -31,3 +33,3 @@ git clone https://github.com/insanity54/precious-data

#### Usage:
## Usage:

@@ -38,8 +40,13 @@ npm run rip

#### Advanced Usage:
## Advanced Usage:
Advanced usage of the CLI tool `p-data.js` allows the user to download specific sets, specific URLs, or all precious-memories cards in existence. The `-i` (incremental) flag can significantly reduce network usage by only downloading card data which has not already been downloaded.
##### Examples
## Documentation
See ./docs
## Examples
Download all cards in the Hatsune Miku set.

@@ -46,0 +53,0 @@

12

test/export.test.js

@@ -1,4 +0,14 @@

const preciousData = require('../index');
const path = require('path');
const setAbbrIndexPath = path.join(__dirname, '..', 'data', 'setAbbrIndex.json');
const setAbbrIndexFixturePath = path.join(__dirname, '..', 'fixtures', 'setAbbrIndex.json');
const setAbbrIndexFixture = require(setAbbrIndexFixturePath);
jest.doMock(setAbbrIndexPath, () => {
return JSON.stringify(setAbbrIndexFixture)
}, { virtual: true })
const toBeType = require('jest-tobetype');
expect.extend(toBeType)
const preciousData = require('../index');

@@ -5,0 +15,0 @@ describe('precious-data exports', () => {

43

test/parsers.test.js

@@ -5,5 +5,12 @@

parseCardId,
parseCardDataFromHtml
parseCardDataFromHtml,
normalizeUrl,
} = require('../lib/parsers')
const fs = require('fs');
const path = require('path');
const fixturesPath = path.join(__dirname, '..', 'fixtures')
describe('parsers', () => {

@@ -189,2 +196,36 @@

describe('parseCardDataFromHtml', () => {
it('should get card data from a {String} html', () => {
const html = fs.readFileSync(path.join(fixturesPath, 'HMK_01-001.html'), { encoding: 'utf-8' })
return parseCardDataFromHtml(html)
.then((data) => {
expect(data).toHaveProperty('number', '01-001')
expect(data).toHaveProperty('rarity', 'SR（サイン）')
expect(data).toHaveProperty('setName', '初音ミク')
expect(data).toHaveProperty('name', '初音 ミク')
expect(data).toHaveProperty('type', 'キャラクター')
expect(data).toHaveProperty('cost', '4')
expect(data).toHaveProperty('source', '1')
expect(data).toHaveProperty('color', '緑')
expect(data).toHaveProperty('characteristic', ['ヘッドフォン', '音楽'])
expect(data).toHaveProperty('ap', '40')
expect(data).toHaveProperty('dp', '30')
expect(data).toHaveProperty('parallel', '')
expect(data).toHaveProperty('text', 'このカードが登場した場合、手札から『初音 ミク』のキャラ1枚を場に出すことができる。[アプローチ/両方]:《0》自分の「初音 ミク」2枚を休息状態にする。その場合、自分のキャラ1枚は、ターン終了時まで+10/±0または±0/+10を得る。')
expect(data).toHaveProperty('flavor', '-')
expect(data).toHaveProperty('image', 'http://p-memories.com/images/product/HMK/HMK_01-001.jpg')
expect(data).toHaveProperty('url', 'http://p-memories.com/node/383031')
expect(data).toHaveProperty('setAbbr', 'HMK')
expect(data).toHaveProperty('num', '001')
expect(data).toHaveProperty('release', '01')
expect(data).toHaveProperty('id', 'HMK 01-001')
})
})
it('should reject to throw an error if not receiving any param', () => {
return expect(parseCardDataFromHtml()).rejects.toThrow('parameter')
})
})
})

740

test/ripper.test.js

@@ -5,43 +5,40 @@ const Ripper = require('../lib/ripper')

const axios = require('axios')
const nock = require('nock')
const { Readable } = require('stream');
const { setupRecorder } = require('nock-record');
const fs = require('fs');
const fsp = fs.promises;
jest.mock('fs')
const setAbbrIndexPath = path.join(__dirname, '..', 'data', 'setAbbrIndex.json');
const setAbbrIndexFixturePath = path.join(__dirname, '..', 'fixtures', 'setAbbrIndex.json');
const setAbbrIndexFixture = require(setAbbrIndexFixturePath);
const hmk01001DataPath = path.join(__dirname, '..', 'data', 'HMK', '01', 'HMK_01-001.json');
const hmk01001FixturePath = path.join(__dirname, '..', 'fixtures', 'HMK_01-001.json');
const hmk01001Fixture = require(hmk01001FixturePath);
nockBack = nock.back
nockBack.fixtures = path.join(__dirname, '..', 'fixtures')
const setAbbrIndexFixture = require(path.join(__dirname, '..', 'fixtures', 'setAbbrIndex.json'))
const mockFileStructureA = {
[path.join(__dirname, '..', 'data', 'setAbbrIndex.json')]: JSON.stringify(setAbbrIndexFixture)
}
const record = setupRecorder();
let ripper
beforeEach(() => {
ripper = new Ripper();
setAbbrIndexSpy = jest.spyOn(ripper, 'loadSetAbbrIndex');
setAbbrIndexSpy.mockImplementation(() => {
return Promise.resolve(setAbbrIndexFixture)
})
})
describe('Ripper', () => {
beforeEach(() => {
nockBack.setMode('record')
})
describe('getCardUrlsFromSetPage', () => {
it(
'should accept a card number and setUrl and resolve an object with cardUrl and cardImageUrl',
() => {
return nockBack('getCardUrlsFromSetPage.json')
.then(({
nockDone,
context
}) => {
return ripper
.getCardUrlsFromSetPage('01-050', 'http://p-memories.com/card_product_list_page?field_title_nid=919863-SSSS.GRIDMAN&s_flg=on')
.then((card) => {
expect(typeof card).toBe('object')
expect(card).toHaveProperty('cardUrl', 'http://p-memories.com/node/926840')
expect(card).toHaveProperty('cardImageUrl', 'http://p-memories.com/images/product/SSSS/SSSS_01-050.jpg')
context.assertScopesFinished()
})
.then(nockDone)
})
async () => {
const { completeRecording, assertScopesFinished } = await record("getCardUrlsFromSetPage");
const card = await ripper.getCardUrlsFromSetPage('01-050', 'http://p-memories.com/card_product_list_page?field_title_nid=919863-SSSS.GRIDMAN&s_flg=on')
completeRecording();
assertScopesFinished();
expect(typeof card).toBe('object');
expect(card).toHaveProperty('cardUrl', 'http://p-memories.com/node/926840');
expect(card).toHaveProperty('cardImageUrl', 'http://p-memories.com/images/product/SSSS/SSSS_01-050.jpg');
}

@@ -52,5 +49,2 @@ )

describe('lookupCardUrl', () => {
beforeEach(() => {
require('fs').__setMockFiles(mockFileStructureA)
})
it('should throw an error if not receiving a parameter', () => {

@@ -68,13 +62,9 @@ return expect(() => {

'should resolve { cardUrl, cardImageUrl } when given a card ID',
() => {
return nockBack('lookupCardUrl.1.json').then(({
nockDone,
context
}) => {
return ripper.lookupCardUrl('SSSS_01-001').then((card) => {
expect(card).toHaveProperty('cardUrl', 'http://p-memories.com/node/926791')
expect(card).toHaveProperty('cardImageUrl', 'http://p-memories.com/images/product/SSSS/SSSS_01-001.jpg')
context.assertScopesFinished()
}).then(nockDone)
})
async () => {
const { completeRecording, assertScopesFinished } = await record("lookupCardUrl1");
const card = await ripper.lookupCardUrl('SSSS_01-001')
completeRecording();
assertScopesFinished();
expect(card).toHaveProperty('cardUrl', 'http://p-memories.com/node/926791')
expect(card).toHaveProperty('cardImageUrl', 'http://p-memories.com/images/product/SSSS/SSSS_01-001.jpg')
}

@@ -110,9 +100,2 @@ )

describe('isLocalData', () => {
beforeEach(() => {
let data = require('../fixtures/HMK_01-001.json')
let mockFileStructureB = {
[path.join(__dirname, '..', 'data', 'HMK', '01', 'HMK_01-001.json')]: JSON.stringify(data)
}
require('fs').__setMockFiles(mockFileStructureB)
})
it(

@@ -137,17 +120,13 @@ 'should return a promise with true for a card that exists on disk',

describe('getSets', () => {
it('should resolve a list objects which contains {String} setUrl and {String} setName', () => {
return nockBack('getSets.1.json')
.then(({ nockDone, context }) => {
return ripper.getSets()
.then((sets) => {
expect(sets).toEqual(expect.arrayContaining([
{
setUrl: 'http://p-memories.com/card_product_list_page?field_title_nid=919863-SSSS.GRIDMAN&s_flg=on',
setName: 'SSSS.GRIDMAN'
}
]))
context.assertScopesFinished()
})
.then(nockDone)
})
it('should resolve a list objects which contains {String} setUrl and {String} setName', async () => {
const { completeRecording, assertScopesFinished } = await record("getSets");
const sets = await ripper.getSets()
completeRecording();
assertScopesFinished();
expect(sets).toEqual(expect.arrayContaining([
{
setUrl: 'http://p-memories.com/card_product_list_page?field_title_nid=919863-SSSS.GRIDMAN&s_flg=on',
setName: 'SSSS.GRIDMAN'
}
]))
})

@@ -157,16 +136,12 @@ })

describe('getSetNames', () => {
it('should resolve a list of {String} setNames', () => {
return nockBack('getSetNames.1.json')
.then(({ nockDone, context }) => {
return ripper.getSetNames()
.then((names) => {
expect(names).toEqual(expect.arrayContaining([
'SSSS.GRIDMAN',
'初音ミク',
'ハロー！！きんいろモザイク'
]))
context.assertScopesFinished()
})
.then(nockDone)
})
it('should resolve a list of {String} setNames', async () => {
const { completeRecording, assertScopesFinished } = await record("getSetNames1");
const names = await ripper.getSetNames()
completeRecording();
assertScopesFinished();
expect(names).toEqual(expect.arrayContaining([
'SSSS.GRIDMAN',
'初音ミク',
'ハロー！！きんいろモザイク'
]))
})

@@ -178,22 +153,15 @@ })

'should return a list of all set URLs found on p-memories.com',
() => {
return nockBack('getSetUrls.json')
.then(({
nockDone,
context
}) => {
return ripper.getSetUrls()
.then((setList) => {
expect(setList).toBeInstanceOf(Array)
expect(setList.length).toBeGreaterThanOrEqual(94)
expect(setList[0]).toEqual(expect.stringMatching(/http:\/\/p-memories.com\/card_product_list_page\?/))
expect(setList).toEqual(expect.arrayContaining([
'http://p-memories.com/card_product_list_page?field_title_nid=241831-ClariS&s_flg=on',
'http://p-memories.com/card_product_list_page?field_title_nid=919863-SSSS.GRIDMAN&s_flg=on',
'http://p-memories.com/card_product_list_page?field_title_nid=280695-%E5%88%9D%E9%9F%B3%E3%83%9F%E3%82%AF&s_flg=on'
]));
context.assertScopesFinished()
})
.then(nockDone)
})
async () => {
const { completeRecording, assertScopesFinished } = await record("placeholder");
const setList = await ripper.getSetUrls()
completeRecording();
assertScopesFinished();
expect(setList).toBeInstanceOf(Array)
expect(setList.length).toBeGreaterThanOrEqual(94)
expect(setList[0]).toEqual(expect.stringMatching(/http:\/\/p-memories.com\/card_product_list_page\?/))
expect(setList).toEqual(expect.arrayContaining([
'http://p-memories.com/card_product_list_page?field_title_nid=241831-ClariS&s_flg=on',
'http://p-memories.com/card_product_list_page?field_title_nid=919863-SSSS.GRIDMAN&s_flg=on',
'http://p-memories.com/card_product_list_page?field_title_nid=280695-%E5%88%9D%E9%9F%B3%E3%83%9F%E3%82%AF&s_flg=on'
]));
}

@@ -207,8 +175,9 @@ );

describe('ripSetData', () => {
it('should accept a setURL as parameter and return a promise', () => {
return nockBack('ripSetData.5.json')
.then(({ nockDone, context }) => {
let promise = ripper.ripSetData('http://p-memories.com/card_product_list_page?field_title_nid=919863-SSSS.GRIDMAN&s_flg=on')
return expect(promise).resolves.toStrictEqual(expect.anything())
})
jest.setTimeout(300000) // 5 minutes
it('should accept a setURL as parameter and return a promise', async () => {
const { completeRecording, assertScopesFinished } = await record("ripSetData1");
const imagePromise = ripper.ripSetData('http://p-memories.com/card_product_list_page?field_title_nid=241831-ClariS&s_flg=on');
await expect(imagePromise).resolves.toStrictEqual(expect.anything())
completeRecording();
assertScopesFinished();
})

@@ -219,107 +188,37 @@ it('should throw if not receiving a setURL as param', () => {

})
it('should return an array of objects containing cardUrl and cardImageUrl', () => {
return nockBack('ripSetData.1.json')
.then(({ nockDone, context }) => {
return ripper
.ripSetData('http://p-memories.com/card_product_list_page?field_title_nid=241831-ClariS&s_flg=on')
.then((data) => {
expect(data).toBeArray()
expect(data.length).toBe(1)
expect(data[0].cardImageUrl).toBeString()
expect(data[0].cardUrl).toBeString()
context.assertScopesFinished()
})
.then(nockDone)
})
}
)
it('should rip a set which contains more than one page', () => {
return nockBack('ripSetData.2.json')
.then(({
nockDone,
context
}) => {
return ripper
.ripSetData('http://p-memories.com/card_product_list_page?field_title_nid=280695-%E5%88%9D%E9%9F%B3%E3%83%9F%E3%82%AF&s_flg=on')
.then((data) => {
expect(data).toBeArray()
expect(data.length).toBeGreaterThanOrEqual(403);
context.assertScopesFinished()
})
.then(nockDone)
})
it('should resolve with an array of objects containing cardUrl and cardImageUrl', async () => {
const { completeRecording, assertScopesFinished } = await record("ripSetData2");
const data = await ripper.ripSetData('http://p-memories.com/card_product_list_page?field_title_nid=241831-ClariS&s_flg=on')
completeRecording();
assertScopesFinished();
expect(data).toBeArray();
expect(data.length).toBe(1);
expect(data[0].cardImageUrl).toBeString();
expect(data[0].cardUrl).toBeString();
})
it('should cope with a relative p-memories.com URL', () => {
return nockBack('ripSetData.3.json')
.then(({
nockDone,
context
}) => {
return ripper.ripSetData('/card_product_list_page?field_title_nid=241831-ClariS&s_flg=on')
.then((data) => {
expect(data).toBeInstanceOf(Array)
expect(data.length).toBe(1)
context.assertScopesFinished()
})
.then(nockDone)
})
it('should rip a set which contains more than one page', async () => {
const { completeRecording, assertScopesFinished } = await record("ripSetData3");
const data = await ripper.ripSetData('http://p-memories.com/card_product_list_page?field_title_nid=280695-%E5%88%9D%E9%9F%B3%E3%83%9F%E3%82%AF&s_flg=on')
completeRecording();
assertScopesFinished();
expect(data).toBeArray();
expect(data.length).toBeGreaterThanOrEqual(403);
})
it('should download madoka release 03', () => {
return nockBack('ripSetData.4.json')
.then(({
nockDone,
context
}) => {
return ripper.ripSetData('http://p-memories.com/card_product_list_page?field_title_nid=313372-%E5%8A%87%E5%A0%B4%E7%89%88+%E9%AD%94%E6%B3%95%E5%B0%91%E5%A5%B3%E3%81%BE%E3%81%A9%E3%81%8B%E2%98%86%E3%83%9E%E3%82%AE%E3%82%AB&s_flg=on')
.then((data) => {
expect(data).toBeInstanceOf(Array);
expect(data.length).toBeGreaterThan(1);
context.assertScopesFinished()
})
.then(nockDone)
})
it('should cope with a relative p-memories.com URL', async () => {
const { completeRecording, assertScopesFinished } = await record("ripSetData4");
const data = await ripper.ripSetData('/card_product_list_page?field_title_nid=241831-ClariS&s_flg=on')
completeRecording();
assertScopesFinished();
expect(data).toBeInstanceOf(Array)
expect(data.length).toBe(1)
})
})
describe('isLocalCard', () => {
it(
'should return a promise resolving true if the card exists on disk',
() => {
return ripper.isLocalCard('HMK_01-001').then((realCardOnDisk) => {
expect(realCardOnDisk).toBe(true);
});
}
);
it(
'should return a promise resolving false if the card does not exist on disk',
() => {
return ripper.isLocalCard('TTQ_05-003').then((fakeCardNotOnDisk) => {
expect(fakeCardNotOnDisk).toBe(false);
});
}
);
it('should handle relative image urls as parameter', () => {
return ripper.isLocalCard('/images/product/PM_HS/PM_HS_01-002.jpg').then((realCardOnDisk) => {
expect(realCardOnDisk).toBe(true);
});
});
it('should handle absolute image urls as parameter', () => {
return ripper.isLocalCard('http://p-memories.com/images/product/HMK/HMK_01-001.jpg').then((realCardOnDisk) => {
expect(realCardOnDisk).toBe(true);
});
});
})
xdescribe('ripCardById', () => {
it('should accept a card ID as param and rip the card to disk', () => {
return ripper.ripCardById('ERMG 01-001').then((writeResult) => {
expect(typeof writeResult).toBe('object');
expect(writeResult).toHaveProperty('imagePath')
expect(writeResult).toHaveProperty('dataPath')
});
it('should download madoka release 03', async () => {
const { completeRecording, assertScopesFinished } = await record("ripSetData5");
const setData = await ripper.ripSetData('http://p-memories.com/card_product_list_page?field_title_nid=313372-%E5%8A%87%E5%A0%B4%E7%89%88+%E9%AD%94%E6%B3%95%E5%B0%91%E5%A5%B3%E3%81%BE%E3%81%A9%E3%81%8B%E2%98%86%E3%83%9E%E3%82%AE%E3%82%AB&s_flg=on')
completeRecording();
assertScopesFinished();
expect(setData).toBeInstanceOf(Array);
expect(setData.length).toBeGreaterThan(1);
})

@@ -329,36 +228,3 @@ })

describe('parseCardDataFromHtml', () => {
it('should get card data from a {String} html', () => {
const html = require(path.join(nockBack.fixtures, 'HMK_01-001.html.json'))[0].response
return ripper
.parseCardDataFromHtml(html)
.then((data) => {
expect(data).toHaveProperty('number', '01-001')
expect(data).toHaveProperty('rarity', 'SR（サイン）')
expect(data).toHaveProperty('setName', '初音ミク')
expect(data).toHaveProperty('name', '初音 ミク')
expect(data).toHaveProperty('type', 'キャラクター')
expect(data).toHaveProperty('cost', '4')
expect(data).toHaveProperty('source', '1')
expect(data).toHaveProperty('color', '緑')
expect(data).toHaveProperty('characteristic', ['ヘッドフォン', '音楽'])
expect(data).toHaveProperty('ap', '40')
expect(data).toHaveProperty('dp', '30')
expect(data).toHaveProperty('parallel', '')
expect(data).toHaveProperty('text', 'このカードが登場した場合、手札から『初音 ミク』のキャラ1枚を場に出すことができる。[アプローチ/両方]:《0》自分の「初音 ミク」2枚を休息状態にする。その場合、自分のキャラ1枚は、ターン終了時まで+10/±0または±0/+10を得る。')
expect(data).toHaveProperty('flavor', '-')
expect(data).toHaveProperty('image', 'http://p-memories.com/images/product/HMK/HMK_01-001.jpg')
expect(data).toHaveProperty('url', 'http://p-memories.com/node/383031')
expect(data).toHaveProperty('setAbbr', 'HMK')
expect(data).toHaveProperty('num', '001')
expect(data).toHaveProperty('release', '01')
expect(data).toHaveProperty('id', 'HMK 01-001')
})
})
it('should reject to throw an error if not receiving any param', () => {
return expect(ripper.parseCardDataFromHtml()).rejects.toThrow('parameter')
})
})
describe('ripCardData', () => {

@@ -375,58 +241,41 @@ it('Should throw an error if not receiving any param', () => {

})
it('Should accept a card URL and resolve to card data', () => {
return nockBack('ripCardData.1.json')
.then(({
nockDone,
context
}) => {
return ripper
.ripCardData('http://p-memories.com/node/926791')
.then((data) => {
expect(typeof data).toBe('object');
expect(data.number).toEqual('01-001');
expect(data.rarity).toEqual('SR');
expect(data.setName).toEqual('SSSS.GRIDMAN');
expect(data.name).toEqual('響 裕太');
expect(data.type).toEqual('キャラクター');
expect(data.cost).toEqual('6');
expect(data.source).toEqual('3');
expect(data.color).toEqual('赤');
expect(data.characteristic).toEqual(['制服']);
expect(data.ap).toEqual('-');
expect(data.dp).toEqual('-');
expect(data.parallel).toEqual('');
expect(data.text).toEqual(
'このカードが登場した場合、手札から名称に「グリッドマン」を含むキャラ1枚を場に出すことができる。[メイン/自分]:《休》名称に「グリッドマン」を含む自分のキャラ1枚は、ターン終了時まで+20/+20を得る。その場合、カードを1枚引く。'
);
expect(data.flavor).toEqual('グリッドマン…。オレと一緒に戦ってくれ！');
expect(data.url).toEqual('http://p-memories.com/node/926791');
expect(data.image).toEqual('http://p-memories.com/images/product/SSSS/SSSS_01-001.jpg');
expect(data.setAbbr).toEqual('SSSS');
expect(data.id).toEqual('SSSS 01-001');
expect(data.num).toEqual('001');
expect(data.release).toEqual('01');
context.assertScopesFinished()
})
.then(nockDone)
})
it('Should accept a card URL and resolve to card data', async () => {
const { completeRecording, assertScopesFinished } = await record("ripCardData1");
const data = await ripper.ripCardData('http://p-memories.com/node/926791')
completeRecording();
assertScopesFinished();
expect(typeof data).toBe('object');
expect(data.number).toEqual('01-001');
expect(data.rarity).toEqual('SR');
expect(data.setName).toEqual('SSSS.GRIDMAN');
expect(data.name).toEqual('響 裕太');
expect(data.type).toEqual('キャラクター');
expect(data.cost).toEqual('6');
expect(data.source).toEqual('3');
expect(data.color).toEqual('赤');
expect(data.characteristic).toEqual(['制服']);
expect(data.ap).toEqual('-');
expect(data.dp).toEqual('-');
expect(data.parallel).toEqual('');
expect(data.text).toEqual(
'このカードが登場した場合、手札から名称に「グリッドマン」を含むキャラ1枚を場に出すことができる。[メイン/自分]:《休》名称に「グリッドマン」を含む自分のキャラ1枚は、ターン終了時まで+20/+20を得る。その場合、カードを1枚引く。'
);
expect(data.flavor).toEqual('グリッドマン…。オレと一緒に戦ってくれ！');
expect(data.url).toEqual('http://p-memories.com/node/926791');
expect(data.image).toEqual('http://p-memories.com/images/product/SSSS/SSSS_01-001.jpg');
expect(data.setAbbr).toEqual('SSSS');
expect(data.id).toEqual('SSSS 01-001');
expect(data.num).toEqual('001');
expect(data.release).toEqual('01');
});
it('should accept a card ID as first param', () => {
// @TODO https://github.com/insanity54/precious-data/issues/3
require('fs').__setMockFiles(mockFileStructureA)
return nockBack('ripCardData.2.json')
.then(({
nockDone,
context
}) => {
return ripper
.ripCardData('SSSS 01-001')
.then((data) => {
expect(data).toBeObject()
expect(data).toHaveProperty('number', '01-001')
expect(data).toHaveProperty('url', 'http://p-memories.com/node/926791')
expect(data).toHaveProperty('image', 'http://p-memories.com/images/product/SSSS/SSSS_01-001.jpg')
context.assertScopesFinished()
}).then(nockDone)
})
it('should accept a card ID as first param', async () => {
const { completeRecording, assertScopesFinished } = await record("ripCardData2");
const data = await ripper.ripCardData('http://p-memories.com/node/926791');
completeRecording();
assertScopesFinished();
expect(data).toBeObject()
expect(data).toHaveProperty('number', '01-001')
expect(data).toHaveProperty('url', 'http://p-memories.com/node/926791')
expect(data).toHaveProperty('image', 'http://p-memories.com/images/product/SSSS/SSSS_01-001.jpg')
});

@@ -436,136 +285,88 @@

'should accept a second parameter, a cardImageUrl, which will be used to determine whether or not to make a network request to rip card data.',
() => {
async () => {
// https://github.com/insanity54/precious-data/issues/4
return nockBack('ripCardData.3.json')
.then(({
nockDone
}) => {
return ripper
.ripCardData('http://p-memories.com/node/932341', '/images/product/GPFN/GPFN_01-030a.jpg')
.then((data) => {
expect(typeof data).toBe('object');
expect(data.number).toEqual('01-030a');
expect(data.url).toEqual('http://p-memories.com/node/932341');
expect(data.id).toEqual('GPFN 01-030a');
})
.then(nockDone)
})
}
);
const { completeRecording, assertScopesFinished } = await record("ripCardData3");
const imageStream = await ripper.ripCardData('http://p-memories.com/node/932341', '/images/product/GPFN/GPFN_01-030a.jpg')
completeRecording();
assertScopesFinished();
expect(typeof data).toBe('object');
expect(data.number).toEqual('01-030a');
expect(data.url).toEqual('http://p-memories.com/node/932341');
expect(data.id).toEqual('GPFN 01-030a');
});
xit(
'should return a promise which rejects with an error if receiving a URL to a card which has already been downloaded',
() => {
// this should happen elsewhere, before calling ripCardData in order to keep functions neat and tidy
// https://github.com/insanity54/precious-data/issues/4
return nockBack('rpCardData.4.json').then(({
nockDone
}) => {
let targetCardPath = path.join(__dirname, '..', 'data', 'HMK', '01', 'HMK_01-001.json')
let creationTimeBefore = fs.statSync(targetCardPath).mtimeMs;
let cd = ripper.ripCardData('http://p-memories.com/node/383031', 'http://p-memories.com/images/product/HMK/HMK_01-001.jpg');
return expect(cd).rejects.toThrow(/EEXIST/)
.then(nockDone)
})
}
);
it('should cope with a relative P-memories URL', () => {
return nockBack('ripCardData.5.json')
.then(({ nockDone, context }) => {
return ripper.ripCardData('/node/926791')
.then((data) => {
expect(typeof data).toBe('object');
expect(data.number).toEqual('01-001');
expect(data.url).toEqual('http://p-memories.com/node/926791');
context.assertScopesFinished()
})
.then(nockDone)
})
})
it('should cope with a relative P-memories URL', async () => {
const { completeRecording, assertScopesFinished } = await record("ripCardData5");
const data = await ripper.ripCardData('/node/926791')
completeRecording();
assertScopesFinished();
expect(typeof data).toBe('object');
expect(data.number).toEqual('01-001');
expect(data.url).toEqual('http://p-memories.com/node/926791');
})
});
describe('writeCardData', () => {
it('should create a JSON file in the appropriate folder', () => {
let cardData = require('../fixtures/HMK_01-001.json');
return ripper
.writeCardData(cardData)
.then((res) => {
let cardDataResult = require('../data/HMK/01/HMK_01-001.json');
expect(cardDataResult.name).toEqual('初音 ミク');
expect(res).toMatch(/\/data\/HMK\/01\/HMK_01-001.json/);
});
});
describe('downloadImage', () => {
jest.setTimeout(20000)
it(
'Should accept a card image URL, image and resolve with a Readable stream',
async () => {
it('should not overwrite locally modified JSON files.', () => {
let cardData = require('../fixtures/HMK_01-001.json');
return ripper
.writeCardData(cardData)
.then((res) => {
let cardDataResult = require('../data/HMK/01/HMK_01-001.json');
expect(cardDataResult.nameEn).toEqual('Hatsune Miku');
expect(cardDataResult.setNameEn).toEqual('Hatsune Miku');
expect(res).toMatch(/\/data\/HMK\/01\/HMK_01-001.json/);
});
});
});
const { completeRecording, assertScopesFinished } = await record("downloadImage1");
const imageStream = await ripper.downloadImage('http://p-memories.com/images/product/SSSS/SSSS_01-001.jpg');
describe('downloadImage', () => {
beforeEach(() => {})
let correctImagePath = path.join(
__dirname,
'..',
'data',
'SSSS',
'01',
'SSSS_01-001.jpg'
);
it(
'Should accept a card URL and download the card image and write it to the correct folder',
() => {
return nockBack('downloadImage.1.json')
.then(({ nockDone, context }) => {
return ripper
.downloadImage('http://p-memories.com/node/926791')
.then((imagePath) => {
expect(imagePath).toBeString()
expect(imagePath).toEqual(correctImagePath)
context.assertScopesFinished()
})
.then(nockDone)
})
// put the stream into flowing mode and save the file (straight to /dev/null is fine)
// so nock actually records the stream's contents
await new Promise((resolve, reject) => {
imageStream.pipe(fs.createWriteStream('/dev/null'))
imageStream.on('end', resolve);
imageStream.on('error', reject);
})
// Complete the recording, allow for Nock to write fixtures
completeRecording();
// Optional; assert that all recorded fixtures have been called
assertScopesFinished();
// Perform your own assertions
expect(imageStream).toBeInstanceOf(Readable)
}
);
it(
'Should download a card image and place it in the correct folder',
() => {
return nockBack('downloadImage.2.json')
.then(({ nockDone, context }) => {
return ripper
.downloadImage('http://p-memories.com/images/product/SSSS/SSSS_01-001.jpg')
.then((imagePath) => {
expect(imagePath).toBeString()
expect(imagePath).toEqual(correctImagePath)
context.assertScopesFinished()
})
.then(nockDone)
})
'Should accept a card URL, download the card image and resolve with a Readable stream',
async () => {
const { completeRecording, assertScopesFinished } = await record("downloadImage2");
const imageStream = await ripper.downloadImage('http://p-memories.com/node/926791');
// put the stream into flowing mode and save the file (straight to /dev/null is fine)
// so nock actually records the stream's contents
await new Promise((resolve, reject) => {
imageStream.pipe(fs.createWriteStream('/dev/null'))
imageStream.on('end', resolve);
imageStream.on('error', reject);
})
completeRecording();
assertScopesFinished();
expect(imageStream).toBeInstanceOf(Readable);
}
);
it(
'should accept a card data object, download the image specified within, and place it in the correct folder',
() => {
return nockBack('downloadImage.3.json')
.then(({ nockDone, context }) => {
let cardData = require(path.join(__dirname, '..', 'fixtures', 'HMK_01-001.json'));
return ripper
.downloadImage(cardData)
.then((imagePath) => {
expect(imagePath).toBeString()
expect(imagePath).toEqual(path.join(__dirname, '..', 'data', 'HMK', '01', 'HMK_01-001.jpg'))
context.assertScopesFinished()
})
.then(nockDone)
})
'should accept a card data object, download the image specified within, and resolve with a Readable stream',
async () => {
let cardData = require(path.join(__dirname, '..', 'fixtures', 'HMK_01-001.json'));
const { completeRecording, assertScopesFinished } = await record("downloadImage3");
const imageStream = await ripper.downloadImage(cardData);
// put the stream into flowing mode and save the file (straight to /dev/null is fine)
// so nock actually records the stream's contents
await new Promise((resolve, reject) => {
imageStream.pipe(fs.createWriteStream('/dev/null'))
imageStream.on('end', resolve);
imageStream.on('error', reject);
})
completeRecording();
assertScopesFinished();
expect(imageStream).toBeInstanceOf(Readable);
}

@@ -614,6 +415,2 @@ )

describe('loadSetAbbrIndex', () => {
beforeEach(() => {
require('fs').__setMockFiles(mockFileStructureA)
})
it('should return a Promise', () => {

@@ -633,3 +430,5 @@ let index = ripper.loadSetAbbrIndex()

it('should reject with an Error if setAbbrIndex.json does not exist', () => {
require('fs').__setMockFiles(null)
setAbbrIndexSpy.mockImplementation(() => {
return Promise.reject(Error('ENOENT: no such file or directory'))
})
let index = ripper.loadSetAbbrIndex()

@@ -666,4 +465,5 @@ return expect(index).rejects.toThrow(/ENOENT/)

describe('getSetUrlFromSetAbbr', () => {
beforeEach(() => {
require('fs').__setMockFiles(mockFileStructureA)
it('Should call loadSetAbbrIndex()', () => {
let url = ripper.getSetUrlFromSetAbbr('HMK');
expect(setAbbrIndexSpy).toHaveBeenCalled()
})

@@ -674,4 +474,6 @@ it('Should return a promise', () => {

})
it('Should reject with an error if the Set Index does not exist', () => {
require('fs').__setMockFiles(null)
it('Should throw an error if the Set Index does not exist', () => {
setAbbrIndexSpy.mockImplementation(() => {
return Promise.reject(Error('ENOENT: no such file or directory'))
})
let url = ripper.getSetUrlFromSetAbbr('HMK')

@@ -726,28 +528,20 @@ return expect(url).rejects.toThrow('ENOENT')

'Should accept a set URL and return the image URL for the first card in the set',
() => {
return nockBack('getFirstCardImageUrl.1.json')
.then(({ nockDone, context }) => {
return ripper
.getFirstCardImageUrl('http://p-memories.com/card_product_list_page?field_title_nid=919863-SSSS.GRIDMAN&s_flg=on')
.then((imageUrl) => {
expect(imageUrl).toEqual('http://p-memories.com/images/product/SSSS/SSSS_01-001.jpg')
context.assertScopesFinished()
})
.then(nockDone)
async () => {
const { completeRecording, assertScopesFinished } = await record("getFirstCardImageUrl1");
const imageUrl = await ripper.getFirstCardImageUrl('http://p-memories.com/card_product_list_page?field_title_nid=919863-SSSS.GRIDMAN&s_flg=on');
completeRecording();
assertScopesFinished();
expect(imageUrl).toEqual('http://p-memories.com/images/product/SSSS/SSSS_01-001.jpg')
})
})
it('should return normalized URLs', () => {
return nockBack('getFirstCardImageUrl.2.json')
.then(({ nockDone, context }) => {
return ripper
.getFirstCardImageUrl('http://p-memories.com/card_product_list_page?field_title_nid=4539-%E3%82%AA%E3%82%AA%E3%82%AB%E3%83%9F%E3%81%95%E3%82%93%E3%81%A8%E4%B8%83%E4%BA%BA%E3%81%AE%E4%BB%B2%E9%96%93%E3%81%9F%E3%81%A1&s_flg=on')
.then(imageUrl => {
expect(imageUrl).toEqual('http://p-memories.com/images/product/ookami/ookami_01-001.jpg')
context.assertScopesFinished()
})
.then(nockDone)
})
it(
'should return normalized URLs',
async () => {
const { completeRecording, assertScopesFinished } = await record("getFirstCardImageUrl2");
const imageUrl = await ripper.getFirstCardImageUrl('http://p-memories.com/card_product_list_page?field_title_nid=4539-%E3%82%AA%E3%82%AA%E3%82%AB%E3%83%9F%E3%81%95%E3%82%93%E3%81%A8%E4%B8%83%E4%BA%BA%E3%81%AE%E4%BB%B2%E9%96%93%E3%81%9F%E3%81%A1&s_flg=on')
completeRecording();
assertScopesFinished();
expect(imageUrl).toEqual('http://p-memories.com/images/product/ookami/ookami_01-001.jpg')
})
})
})

@@ -771,18 +565,12 @@

'Should accept no parameters and return an array of objects with sampleCardUrl and setUrl k/v',
() => {
return nockBack('getImageUrlFromEachSet.1.json')
.then(({ nockDone, context }) => {
// mock the size of the card_product_list_page to make the test fast
return ripper.getImageUrlFromEachSet()
.then((imageUrls) => {
let indexPath = path.join(__dirname, '..', 'data', 'setAbbrIndex.json')
expect(imageUrls).toBeArray()
expect(imageUrls[0]).toBeObject()
expect(imageUrls[0].setUrl).toBeString()
expect(imageUrls[0].sampleCardUrl).toBeString()
context.assertScopesFinished()
})
.then(nockDone)
})
}, 1000*60*5);
async () => {
const { completeRecording, assertScopesFinished } = await record("getImageUrlFromEachSet");
const imageUrls = await ripper.getImageUrlFromEachSet()
completeRecording();
assertScopesFinished();
expect(imageUrls).toBeArray()
expect(imageUrls[0]).toBeObject()
expect(imageUrls[0].setUrl).toBeString()
expect(imageUrls[0].sampleCardUrl).toBeString()
})
});

@@ -804,33 +592,11 @@

})
it('should create setAbbrIndex.json in the data folder', () => {
return nockBack('createSetAbbreviationIndex.1.json')
.then(({ nockDone, context }) => {
return ripper.createSetAbbreviationIndex()
.then((setAbbrIndexPath) => {
expect(setAbbrIndexPath).toEqual(path.join(__dirname, '..', 'data', 'setAbbrIndex.json'))
context.assertScopesFinished()
})
.then(nockDone)
})
}, 1000*60*3)
it('should create setAbbrIndex.json in the data folder', async () => {
const { completeRecording, assertScopesFinished } = await record("createSetAbbreviationIndex1");
const imageUrls = await ripper.createSetAbbreviationIndex();
completeRecording();
assertScopesFinished();
expect(setAbbrIndexPath).toEqual(path.join(__dirname, '..', 'data', 'setAbbrIndex.json'));
})
})
describe('saveCardData', () => {
it('Should accept an object and resolve to be an {Array} containing paths of image & json on disk', () => {
return nockBack('saveCardData.1.json')
.then(({ nockDone, context }) => {
let cardData = require('../fixtures/HMK_01-001.json');
return ripper
.saveCardData(cardData)
.then((savePaths) => {
expect(savePaths).toBeArray();
const [imagePath, jsonPath] = savePaths
expect(imagePath).toBe(path.join(__dirname, '..', 'data', 'HMK', '01', 'HMK_01-001.jpg'))
expect(jsonPath).toBe(path.join(__dirname, '..', 'data', 'HMK', '01', 'HMK_01-001.json'))
context.assertScopesFinished()
})
.then(nockDone)
})
})
})
})
docs/ripper.md
fixtures/createSetAbbreviationIndex.1.json
fixtures/downloadImage.1.json
fixtures/downloadImage.2.json
fixtures/downloadImage.3.json
fixtures/getCardUrlsFromSetPage.json
fixtures/getFirstCardImageUrl.1.json
fixtures/getFirstCardImageUrl.2.json
fixtures/getImageUrlFromEachSet.1.json
fixtures/getSetNames.1.json
fixtures/getSets.1.json
fixtures/getSetUrls.json
fixtures/GPFN_01-030a.html.json
fixtures/HMK_01-001.html.json
fixtures/HMK_01-001.jpg.json
fixtures/lookupCardUrl.1.json
fixtures/ripCardData.1.json
fixtures/ripCardData.2.json
fixtures/ripCardData.5.json
fixtures/ripSetData.1.json
fixtures/ripSetData.2.json
fixtures/ripSetData.3.json
fixtures/ripSetData.4.json
fixtures/saveCardData.1.json
fixtures/SSSS_01-001.html.json
fixtures/SSSS.html.json
test/httpAgent.test.js
precious-data.sublime-workspace

Sorry, the diff of this file is not supported yet

SocketSocket SOC 2 Logo

Product

  • Package Alerts
  • Integrations
  • Docs
  • Pricing
  • FAQ
  • Roadmap
  • Changelog

About

Packages

npm

Go

Maven

PyPI

Rubygems

Stay in touch

Get open source security insights delivered straight into your inbox.

  • Terms
  • Privacy
  • Security

Made with ⚡️ by Socket Inc