
Security News
Deno 2.2 Improves Dependency Management and Expands Node.js Compatibility
Deno 2.2 enhances Node.js compatibility, improves dependency management, adds OpenTelemetry support, and expands linting and task automation for developers.
scrapy-crawlbase-middleware
Advanced tools
Scrapy Crawlbase Proxy Middleware: Crawlbase interfacing middleware for Scrapy
Processes Scrapy requests using Crawlbase services either with Normal or Javascript tokens
Choose a way of installing:
python setup.py install
pip install scrapy-crawlbase-middleware
Then in your Scrapy settings.py
add the following lines:
# Activate the middleware
CRAWLBASE_ENABLED = True
# The Crawlbase API token you wish to use, either normal of javascript token
CRAWLBASE_TOKEN = 'your token'
# Enable the middleware
DOWNLOADER_MIDDLEWARES = {
'scrapy_crawlbase.CrawlbaseMiddleware': 610
}
Use the scrapy_crawlbase.CrawlbaseRequest instead of the scrapy built-in Request. The scrapy_crawlbase.CrawlbaseRequest accepts additional arguments, used in Proxy Crawl API:
from scrapy_crawlbase import CrawlbaseRequest
class ExampleScraper(Spider):
def start_requests(self):
yield CrawlbaseRequest(
"http://target-url",
callback=self.parse_result
device='desktop',
country='US',
page_wait=1000,
ajax_wait=True,
dont_filter=True
)
The target url will be replaced with proxy crawl url and parameters will be encoded into the url by the middleware automatically.
If you have questions or need help using the library, please open an issue or contact us.
Copyright 2023 Crawlbase
FAQs
Scrapy Crawlbase Proxy Middleware: Crawlbase interfacing middleware for Scrapy
We found that scrapy-crawlbase-middleware demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Deno 2.2 enhances Node.js compatibility, improves dependency management, adds OpenTelemetry support, and expands linting and task automation for developers.
Security News
React's CRA deprecation announcement sparked community criticism over framework recommendations, leading to quick updates acknowledging build tools like Vite as valid alternatives.
Security News
Ransomware payment rates hit an all-time low in 2024 as law enforcement crackdowns, stronger defenses, and shifting policies make attacks riskier and less profitable.