Huge News!Announcing our $40M Series B led by Abstract Ventures.Learn More
Socket
Sign inDemoInstall
Socket

bandcamp-fetch

Package Overview
Dependencies
Maintainers
1
Versions
24
Alerts
File Explorer

Advanced tools

Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

bandcamp-fetch - npm Package Compare versions

Comparing version 0.1.0-a-20210213.3 to 0.1.0-a-20210216

examples/getReleasesByTag_output.txt

5

examples/getReleasesByTag.js

@@ -8,3 +8,3 @@ const bcfetch = require('../');

filters: {
tags: [ 'dark-ambient', 'electronica' ],
tags: [ 'electronica' ],
sort: 'random'

@@ -16,3 +16,4 @@ },

const options = {
imageFormat: 2
imageFormat: 2,
useHardcodedDefaultFilters: true
}

@@ -19,0 +20,0 @@

88

examples/getTags_output.txt

@@ -16,3 +16,2 @@ { tags:

{ name: 'noise', url: 'https://bandcamp.com/tag/noise' },
{ name: 'ambient', url: 'https://bandcamp.com/tag/ambient' },
{ name: 'techno', url: 'https://bandcamp.com/tag/techno' },

@@ -22,11 +21,5 @@ { name: 'pop', url: 'https://bandcamp.com/tag/pop' },

url: 'https://bandcamp.com/tag/indie-rock' },
{ name: 'pop', url: 'https://bandcamp.com/tag/pop' },
{ name: 'instrumental',
url: 'https://bandcamp.com/tag/instrumental' },
{ name: 'hip hop', url: 'https://bandcamp.com/tag/hip-hop' },
{ name: 'experimental',
url: 'https://bandcamp.com/tag/experimental' },
{ name: 'electronic',
url: 'https://bandcamp.com/tag/electronic' },
{ name: 'rock', url: 'https://bandcamp.com/tag/rock' },
{ name: 'acoustic', url: 'https://bandcamp.com/tag/acoustic' },

@@ -36,3 +29,2 @@ { name: 'folk', url: 'https://bandcamp.com/tag/folk' },

{ name: 'drone', url: 'https://bandcamp.com/tag/drone' },
{ name: 'folk', url: 'https://bandcamp.com/tag/folk' },
{ name: 'electronica',

@@ -54,9 +46,7 @@ url: 'https://bandcamp.com/tag/electronica' },

url: 'https://bandcamp.com/tag/alternative-rock' },
{ name: 'jazz', url: 'https://bandcamp.com/tag/jazz' },
{ name: 'punk', url: 'https://bandcamp.com/tag/punk' },
{ name: 'hip-hop', url: 'https://bandcamp.com/tag/hip-hop' },
{ name: 'experimental electronic',
url: 'https://bandcamp.com/tag/experimental-electronic' },
{ name: 'world', url: 'https://bandcamp.com/tag/world' },
{ name: 'indie pop', url: 'https://bandcamp.com/tag/indie-pop' },
{ name: 'world', url: 'https://bandcamp.com/tag/world' },
{ name: 'beats', url: 'https://bandcamp.com/tag/beats' },

@@ -66,25 +56,17 @@ { name: 'electro', url: 'https://bandcamp.com/tag/electro' },

url: 'https://bandcamp.com/tag/black-metal' },
{ name: 'lofi', url: 'https://bandcamp.com/tag/lofi' },
{ name: 'post-rock', url: 'https://bandcamp.com/tag/post-rock' },
{ name: 'soundtrack',
url: 'https://bandcamp.com/tag/soundtrack' },
{ name: 'post-rock', url: 'https://bandcamp.com/tag/post-rock' },
{ name: 'lofi', url: 'https://bandcamp.com/tag/lofi' },
{ name: 'techno', url: 'https://bandcamp.com/tag/techno' },
{ name: 'acoustic', url: 'https://bandcamp.com/tag/acoustic' },
{ name: 'punk rock', url: 'https://bandcamp.com/tag/punk-rock' },
{ name: 'soul', url: 'https://bandcamp.com/tag/soul' },
{ name: 'alternative',
url: 'https://bandcamp.com/tag/alternative' },
{ name: 'post-punk', url: 'https://bandcamp.com/tag/post-punk' },
{ name: 'shoegaze', url: 'https://bandcamp.com/tag/shoegaze' },
{ name: 'house', url: 'https://bandcamp.com/tag/house' },
{ name: 'soundtrack',
url: 'https://bandcamp.com/tag/soundtrack' },
{ name: 'deep house',
url: 'https://bandcamp.com/tag/deep-house' },
{ name: 'downtempo', url: 'https://bandcamp.com/tag/downtempo' },
{ name: 'death metal',
url: 'https://bandcamp.com/tag/death-metal' },
{ name: 'downtempo', url: 'https://bandcamp.com/tag/downtempo' },
{ name: 'funk', url: 'https://bandcamp.com/tag/funk' },
{ name: 'dance', url: 'https://bandcamp.com/tag/dance' },
{ name: 'indie', url: 'https://bandcamp.com/tag/indie' },
{ name: 'avant-garde',

@@ -98,5 +80,5 @@ url: 'https://bandcamp.com/tag/avant-garde' },

{ name: 'vaporwave', url: 'https://bandcamp.com/tag/vaporwave' },
{ name: 'emo', url: 'https://bandcamp.com/tag/emo' },
{ name: 'improvisation',
url: 'https://bandcamp.com/tag/improvisation' },
{ name: 'emo', url: 'https://bandcamp.com/tag/emo' },
{ name: 'dub', url: 'https://bandcamp.com/tag/dub' },

@@ -108,6 +90,5 @@ { name: 'trap', url: 'https://bandcamp.com/tag/trap' },

url: 'https://bandcamp.com/tag/underground' },
{ name: 'garage', url: 'https://bandcamp.com/tag/garage' },
{ name: 'indie folk',
url: 'https://bandcamp.com/tag/indie-folk' },
{ name: 'hip hop', url: 'https://bandcamp.com/tag/hip-hop' },
{ name: 'garage', url: 'https://bandcamp.com/tag/garage' },
{ name: 'underground hip hop',

@@ -120,19 +101,16 @@ url: 'https://bandcamp.com/tag/underground-hip-hop' },

{ name: 'r&b', url: 'https://bandcamp.com/tag/r-b' },
{ name: 'rap', url: 'https://bandcamp.com/tag/rap' },
{ name: 'dubstep', url: 'https://bandcamp.com/tag/dubstep' },
{ name: 'metal', url: 'https://bandcamp.com/tag/metal' },
{ name: 'piano', url: 'https://bandcamp.com/tag/piano' },
{ name: 'pop rock', url: 'https://bandcamp.com/tag/pop-rock' },
{ name: 'instrumental hip-hop',
url: 'https://bandcamp.com/tag/instrumental-hip-hop' },
{ name: 'americana', url: 'https://bandcamp.com/tag/americana' },
{ name: 'ambient electronic',
url: 'https://bandcamp.com/tag/ambient-electronic' },
{ name: 'instrumental hip-hop',
url: 'https://bandcamp.com/tag/instrumental-hip-hop' },
{ name: 'pop rock', url: 'https://bandcamp.com/tag/pop-rock' },
{ name: 'chillout', url: 'https://bandcamp.com/tag/chillout' },
{ name: 'funk', url: 'https://bandcamp.com/tag/funk' },
{ name: 'guitar', url: 'https://bandcamp.com/tag/guitar' },
{ name: 'pop punk', url: 'https://bandcamp.com/tag/pop-punk' },
{ name: 'chill', url: 'https://bandcamp.com/tag/chill' },
{ name: 'hardcore punk',
url: 'https://bandcamp.com/tag/hardcore-punk' },
{ name: 'chill', url: 'https://bandcamp.com/tag/chill' },
{ name: 'progressive rock',

@@ -142,5 +120,29 @@ url: 'https://bandcamp.com/tag/progressive-rock' },

url: 'https://bandcamp.com/tag/electronic-music' },
{ name: 'psychedelic rock',
url: 'https://bandcamp.com/tag/psychedelic-rock' },
{ name: 'classical', url: 'https://bandcamp.com/tag/classical' },
{ name: 'tech house',
url: 'https://bandcamp.com/tag/tech-house' },
{ name: 'garage rock',
url: 'https://bandcamp.com/tag/garage-rock' },
{ name: 'soundscape',
url: 'https://bandcamp.com/tag/soundscape' },
... 298 more items ],
{ name: 'grunge', url: 'https://bandcamp.com/tag/grunge' },
{ name: 'atmospheric',
url: 'https://bandcamp.com/tag/atmospheric' },
{ name: 'harsh noise',
url: 'https://bandcamp.com/tag/harsh-noise' },
{ name: 'hard rock', url: 'https://bandcamp.com/tag/hard-rock' },
{ name: 'edm', url: 'https://bandcamp.com/tag/edm' },
{ name: 'dream pop', url: 'https://bandcamp.com/tag/dream-pop' },
{ name: 'psytrance', url: 'https://bandcamp.com/tag/psytrance' },
{ name: 'reggae', url: 'https://bandcamp.com/tag/reggae' },
{ name: 'devotional',
url: 'https://bandcamp.com/tag/devotional' },
{ name: 'diy', url: 'https://bandcamp.com/tag/diy' },
{ name: 'hiphop', url: 'https://bandcamp.com/tag/hiphop' },
{ name: 'country', url: 'https://bandcamp.com/tag/country' },
{ name: 'doom', url: 'https://bandcamp.com/tag/doom' },
{ name: 'grindcore', url: 'https://bandcamp.com/tag/grindcore' },
... 225 more items ],
locations:

@@ -180,4 +182,4 @@ [ { name: 'United Kingdom',

{ name: 'Ohio', url: 'https://bandcamp.com/tag/ohio' },
{ name: 'Oregon', url: 'https://bandcamp.com/tag/oregon' },
{ name: 'Michigan', url: 'https://bandcamp.com/tag/michigan' },
{ name: 'Oregon', url: 'https://bandcamp.com/tag/oregon' },
{ name: 'Netherlands',

@@ -208,5 +210,5 @@ url: 'https://bandcamp.com/tag/netherlands' },

{ name: 'Virginia', url: 'https://bandcamp.com/tag/virginia' },
{ name: 'Minnesota', url: 'https://bandcamp.com/tag/minnesota' },
{ name: 'San Francisco',
url: 'https://bandcamp.com/tag/san-francisco' },
{ name: 'Minnesota', url: 'https://bandcamp.com/tag/minnesota' },
{ name: 'Boston', url: 'https://bandcamp.com/tag/boston' },

@@ -227,8 +229,8 @@ { name: 'Portugal', url: 'https://bandcamp.com/tag/portugal' },

url: 'https://bandcamp.com/tag/new-zealand' },
{ name: 'Scotland', url: 'https://bandcamp.com/tag/scotland' },
{ name: 'Atlanta', url: 'https://bandcamp.com/tag/atlanta' },
{ name: 'Missouri', url: 'https://bandcamp.com/tag/missouri' },
{ name: 'Scotland', url: 'https://bandcamp.com/tag/scotland' },
{ name: 'Vancouver', url: 'https://bandcamp.com/tag/vancouver' },
{ name: 'Chile', url: 'https://bandcamp.com/tag/chile' },
{ name: 'Wisconsin', url: 'https://bandcamp.com/tag/wisconsin' },
{ name: 'Chile', url: 'https://bandcamp.com/tag/chile' },
{ name: 'Greece', url: 'https://bandcamp.com/tag/greece' },

@@ -238,12 +240,12 @@ { name: 'Indiana', url: 'https://bandcamp.com/tag/indiana' },

{ name: 'Brooklyn', url: 'https://bandcamp.com/tag/brooklyn' },
{ name: 'Austria', url: 'https://bandcamp.com/tag/austria' },
{ name: 'Denmark', url: 'https://bandcamp.com/tag/denmark' },
{ name: 'NRW', url: 'https://bandcamp.com/tag/nrw' },
{ name: 'Denver', url: 'https://bandcamp.com/tag/denver' },
{ name: 'NRW', url: 'https://bandcamp.com/tag/nrw' },
{ name: 'Minneapolis',
url: 'https://bandcamp.com/tag/minneapolis' },
{ name: 'Austria', url: 'https://bandcamp.com/tag/austria' },
{ name: 'Ireland', url: 'https://bandcamp.com/tag/ireland' },
{ name: 'Detroit', url: 'https://bandcamp.com/tag/detroit' },
{ name: 'Norway', url: 'https://bandcamp.com/tag/norway' },
{ name: 'Sydney', url: 'https://bandcamp.com/tag/sydney' },
{ name: 'Norway', url: 'https://bandcamp.com/tag/norway' },
{ name: 'Nashville', url: 'https://bandcamp.com/tag/nashville' },

@@ -257,5 +259,5 @@ { name: 'IDF', url: 'https://bandcamp.com/tag/idf' },

{ name: 'Israel', url: 'https://bandcamp.com/tag/israel' },
{ name: 'Tokyo', url: 'https://bandcamp.com/tag/tokyo' },
{ name: 'Connecticut',
url: 'https://bandcamp.com/tag/connecticut' },
{ name: 'Tokyo', url: 'https://bandcamp.com/tag/tokyo' },
{ name: 'Hungary', url: 'https://bandcamp.com/tag/hungary' },

@@ -268,7 +270,7 @@ { name: 'Alberta', url: 'https://bandcamp.com/tag/alberta' },

url: 'https://bandcamp.com/tag/district-of-columbia' },
{ name: 'Louisiana', url: 'https://bandcamp.com/tag/louisiana' },
{ name: 'Oakland', url: 'https://bandcamp.com/tag/oakland' },
{ name: 'Madrid', url: 'https://bandcamp.com/tag/madrid' },
{ name: 'Louisiana', url: 'https://bandcamp.com/tag/louisiana' },
{ name: 'Pittsburgh',
url: 'https://bandcamp.com/tag/pittsburgh' },
... 240 more items ] }
... 241 more items ] }
const fetch = require('node-fetch');
const Bottleneck = require('bottleneck');
const utils = require('./utils.js');

@@ -290,2 +291,7 @@ const parser = require('./parser.js');

async function getTagInfo(tagUrl) {
return _fetchPage(tagUrl)
.then( html => parser.parseTagInfo(html, {tagUrl}) );
}
async function getReleasesByTagFilterOptions(tagUrl) {

@@ -319,23 +325,54 @@ return getReleasesByTagFilterValueNames(tagUrl)

return getReleasesByTagFilterOptions(tagUrl)
.then( filterOptions => {
const defaultFilters = {};
filterOptions.forEach( filter => {
let selectedOption = filter.options.find( o => o.selected );
let defaultOption = filter.options.find( o => o.default );
if (selectedOption) {
if (filter.name === 'tags') {
defaultFilters[filter.name] = [selectedOption.value];
}
else {
defaultFilters[filter.name] = selectedOption.value;
}
}
else if (defaultOption) {
defaultFilters[filter.name] = defaultOption.value;
}
const _getDefaultFilters = tagUrl => {
if (options.useHardcodedDefaultFilters) {
const tagUrlPath = utils.splitUrl(tagUrl).path;
if (tagUrlPath.endsWith('/')) {
tagUrlPath = tagUrlPath.substr(0, tagUrlPath.length - 1);
}
const tagValue = tagUrlPath.split('/').pop();
return Promise.resolve({
tags: [tagValue],
location: 0,
format: 'all',
sort: 'pop'
});
}
else {
return getReleasesByTagFilterOptions(tagUrl)
.then( filterOptions => {
const defaultFilters = {};
filterOptions.forEach( filter => {
let selectedOption = filter.options.find( o => o.selected );
let defaultOption = filter.options.find( o => o.default );
if (selectedOption) {
if (filter.name === 'tags') {
defaultFilters[filter.name] = [selectedOption.value];
}
else {
defaultFilters[filter.name] = selectedOption.value;
}
}
else if (defaultOption) {
defaultFilters[filter.name] = defaultOption.value;
}
});
const paramFilters = params.filters ? Object.assign(defaultFilters, params.filters) : defaultFilters;
return defaultFilters;
});
}
}
return _getDefaultFilters(tagUrl)
.then( defaultFilters => {
const tagsFilter = defaultFilters.tags ? defaultFilters.tags.slice(0) : [];
if (params.filters && Array.isArray(params.filters.tags)) {
params.filters.tags.forEach( tag => {
if (!tagsFilter.includes(tag)) {
tagsFilter.push(tag);
}
})
}
const paramFilters = params.filters ? Object.assign(defaultFilters, params.filters, {tags: tagsFilter}) : defaultFilters;
return {

@@ -374,3 +411,12 @@ filters: paramFilters,

const doFetch = fetchOptions ? fetch(url, fetchOptions) : fetch(url);
return doFetch.then( res => json ? res.json() : res.text() );
return doFetch.then( res => {
if (res.status === 429) {
const err = new Error('429 Too Many Requests');
err.code = '429';
throw err;
}
else {
return json ? res.json() : res.text();
}
});
});

@@ -396,3 +442,4 @@ }

module.exports = {
// Exported functions
const _exportFn = {
discover,

@@ -411,3 +458,2 @@ getDiscoverOptions,

getTags,
cache,
getAllShows,

@@ -418,2 +464,3 @@ getShow,

getArticle,
getTagInfo,
getReleasesByTagFilterOptions,

@@ -423,2 +470,16 @@ getReleasesByTag,

searchLocation
};
};
// Bottleneck limiter
const _limiter = new Bottleneck({
maxConcurrent: 5,
minTime: 200
});
const limiter = {};
for (const [fnName, fn] of Object.entries(_exportFn)) {
limiter[fnName] = _limiter.wrap(fn);
}
limiter.updateSettings = _limiter.updateSettings.bind(_limiter);
// Module exports
module.exports = Object.assign({}, _exportFn, { cache, limiter });

@@ -617,2 +617,7 @@ const cheerio = require('cheerio');

const $ = cheerio.load(html);
const _findTag = (tagUrl, tagName, tags) => {
return tags.find( t => t.url === tagUrl && t.name === tagName);
}
const _parseCloud = (id) => {

@@ -623,6 +628,10 @@ const cloud = $(`#${id}`);

link = $(link);
tagsInCloud.push({
name: link.text(),
url: utils.getUrl(link.attr('href'))
});
const name = link.text().trim();
const url = utils.getUrl(link.attr('href'));
if (name && link.attr('href') !== '/tag/' && !_findTag(url, name, tagsInCloud)) { // Skip blank or repeating tags
tagsInCloud.push({
name,
url
});
}
});

@@ -1004,2 +1013,34 @@ return tagsInCloud;

function parseTagInfo(html, opts) {
const $ = cheerio.load(html);
const blob = decode($('#pagedata[data-blob]').attr('data-blob'));
const parsed = JSON.parse(blob);
if (typeof parsed === 'object' && parsed.hub) {
const tag = {
type: 'tag',
name: parsed.hub.name,
url: opts.tagUrl,
value: parsed.hub.norm_name,
relatedTags: []
};
if (Array.isArray(parsed.hub.related_tags)) {
parsed.hub.related_tags.forEach( related => {
const relatedTag = {
type: 'tag',
name: related.name,
url: utils.getUrl(related.url),
value: related.norm_name,
isLocation: related.isloc
};
tag.relatedTags.push(relatedTag);
});
}
return tag;
}
else {
console.log('Failed to parse tag info');
return null;
}
}
function parseHubJSPath(html) {

@@ -1136,8 +1177,8 @@ const jsMatch = /src="((?:.+?)hub-(?:.+?).js)"/g.exec(html);

},
featuredTrack: ''
featuredTrack: null
};
if (item.type === 'a') {
if (item.item_type === 'a') {
mediaItem.type = 'album';
}
else if (item.type === 't') {
else if (item.item_type === 't') {
mediaItem.type = 'track';

@@ -1151,2 +1192,3 @@ }

name: item.featured_track_title,
position: item.featured_track_number,
streamUrl: (item.audio_url ? item.audio_url['mp3-128'] : null) || null

@@ -1220,2 +1262,3 @@ };

parseArticle,
parseTagInfo,
parseHubJSPath,

@@ -1222,0 +1265,0 @@ parseHubJSFilterValueNames,

{
"name": "bandcamp-fetch",
"version": "0.1.0a-20210213.3",
"version": "0.1.0a-20210216",
"description": "JS library for scraping Bandcamp content",

@@ -25,2 +25,3 @@ "main": "lib/index.js",

"dependencies": {
"bottleneck": "^2.19.5",
"cheerio": "^1.0.0-rc.5",

@@ -27,0 +28,0 @@ "html-entities": "^2.0.2",

@@ -214,2 +214,80 @@ # bandcamp-fetch

### `getTagInfo(tagUrl)`
[**Example**](examples/getTagInfo.js) ([output](examples/getTagInfo_output.txt))
Fetches information about the tag referred to by `tagUrl`.
### `getReleasesByTag(tagUrl, [params], [options])`
[**Example**](examples/getReleasesByTag.js) ([output](examples/getReleasesByTag_output.txt))
Fetches releases matching the tag referred to by `tagUrl`.
- `tagUrl`
- `params` (optional)
- filters:
- location
- tags: array of tag values to match, in addition to the one referred to by `tagUrl`.
- sort
- format
- page (1 if omitted)
All properties are optional. For omitted properties, default values obtained from `tagUrl` will be used. Possible filter values can be obtained by calling `getReleasesByTagFilterOptions()`. For `filters.location` and `filters.tag`, you may look up additional values not returned by `getReleasesByTagFilterOptions()` through `searchLocation()` and `searchTag()`, respectively.
- `options` (optional)
- imageFormat
- useHardcodedDefaultFilters: if `true`, use hardcoded default values for filters not specified in `params.filters`. If `false` or unspecified, default filter values will be obtained by calling `getReleasesByTagFilterOptions()` (extra query means slower performance).
### `getReleasesByTagFilterOptions(tagUrl)`
[**Example**](examples/getReleasesByTagFilterOptions.js) ([output](examples/getReleasesByTagFilterOptions_output.txt))
Fetches the list of possible filter values for `getReleasesByTag()`. For `location` and `tag` filters, this function does not return a conclusive list of values. You may use `searchLocation()` and `searchTag()` to look up additional values.
- `tagUrl`: the URL of the tag for which filter values should be returned
### `searchLocation(params)`
[**Example**](examples/searchLocation.js) ([output](examples/searchLocation_output.txt))
Fetches the list of locations matching `params.q`. Results include both partial and full matches. Each item in the returned array corresponds to a matching location, and its `value` property can be used for setting the `location` filter in `getReleasesByTag()`.
- `params`:
- q: the string to match
- limit: the maximum number of results to return
### `searchTag(params)`
[**Example**](examples/searchTag.js) ([output](examples/searchTag_output.txt))
Fetches the list of tags matching `params.q`. Results include both partial and full matches. Each item in the returned array corresponds to a matching tag, and its `value` property can be used for setting the `tags` filter in `getReleasesByTag()`.
- `params`:
- q: the string to match
- limit: the maximum number of results to return
## Rate Limiting
The API functions can be called with rate limiting like this:
```
bcfetch.limiter.getAlbumInfo(...);
```
[**Example**](examples/limiter.js) ([output](examples/limiter_output.txt))
Rate limiting is useful when you need to make a large number of queries and don't want to run the risk of getting rejected by the server for making too many requests within a short time interval. If you get a '429 Too Many Requests' error, then you should consider using the rate limiter.
The library uses [Bottleneck](https://www.npmjs.com/package/bottleneck) for rate limiting. You can configure the rate limiter like this:
```
bcfetch.limiter.updateSettings({
maxConcurrent: 10, // default: 5
minTime: 100 // default: 200
});
```
`updateSettings()` is just a passthrough function to Bottleneck. Check the [Bottleneck doc](https://www.npmjs.com/package/bottleneck#docs) for the list of options you can set.
## Caching

@@ -216,0 +294,0 @@

SocketSocket SOC 2 Logo

Product

  • Package Alerts
  • Integrations
  • Docs
  • Pricing
  • FAQ
  • Roadmap
  • Changelog

Packages

npm

Stay in touch

Get open source security insights delivered straight into your inbox.


  • Terms
  • Privacy
  • Security

Made with ⚡️ by Socket Inc