cloudproxy
Advanced tools
The purpose of CloudProxy is to hide your scrapers IP behind the cloud. It allows you to spin up a pool of proxies using popular cloud providers with just an API token. No configuration needed.
CloudProxy exposes an API and modern UI for managing your proxy infrastructure. It includes:
Please always scrape nicely, respectfully.
To get a local copy up and running follow these simple steps.
CloudProxy is designed to run as a Docker container:
CloudProxy is distributed as a Docker image for easy deployment:
# Quick start with DigitalOcean
docker run -d \
-e PROXY_USERNAME='your_username' \
-e PROXY_PASSWORD='your_password' \
-e DIGITALOCEAN_ENABLED=True \
-e DIGITALOCEAN_ACCESS_TOKEN='your_token' \
-p 8000:8000 \
laffin/cloudproxy:latest
# Using environment file (recommended for production)
docker run -d \
--env-file .env \
-p 8000:8000 \
laffin/cloudproxy:0.6.0-beta # Use specific version tag
Docker Compose Example:
version: '3.8'
services:
cloudproxy:
image: laffin/cloudproxy:latest
ports:
- "8000:8000"
env_file:
- .env
restart: unless-stopped
It is recommended to use a Docker image tagged to a specific version (e.g., laffin/cloudproxy:0.6.0-beta). See releases for the latest version.
CloudProxy requires authentication configuration for the proxy servers:
PROXY_USERNAME, PROXY_PASSWORD - Basic authentication credentials (alphanumeric characters only)ONLY_HOST_IP - Set to True to restrict access to the host server IP onlyAGE_LIMIT - Proxy age limit in seconds (0 = disabled, default: disabled)See individual provider documentation for provider-specific environment variables.
For development or integrating CloudProxy into existing Python applications:
# Install from PyPI
pip install cloudproxy
# Or install from source for development
git clone https://github.com/claffin/cloudproxy.git
cd cloudproxy
pip install -e .
See the Python Package Usage Guide for development and integration examples.
CloudProxy includes a comprehensive test suite to ensure reliability:
# Run all tests
pytest -v
# Run specific test file
pytest tests/test_specific.py -v
# Run with coverage
pytest --cov=cloudproxy tests/
Warning: These tests create real cloud instances and will incur costs!
# Run full end-to-end test
./test_cloudproxy.sh
# Run without cleanup (for debugging)
./test_cloudproxy.sh --no-cleanup --skip-connection-test
# Test specific providers
./test_cloudproxy.sh --provider digitalocean
# Clone the repository
git clone https://github.com/claffin/cloudproxy.git
cd cloudproxy
# Install in development mode
pip install -e .
# Install development dependencies
pip install -r requirements.txt
# Run the application locally
python -m cloudproxy
# Navigate to UI directory
cd cloudproxy-ui
# Install dependencies
npm install
# Run development server (hot reload enabled)
npm run serve
# Build for production
npm run build
cloudproxy/providers/main.py with the provider orchestration logicfunctions.py with cloud API interactionsproviders/config.pytests/test_provider_name.pyCloudProxy provides multiple interfaces for managing your proxy infrastructure:
Access the UI at http://localhost:8000/ui to:
Access the interactive API documentation at http://localhost:8000/docs to:
CloudProxy exposes a RESTful API on localhost:8000. Your application can use the API to retrieve and manage proxy servers. All responses include metadata with request ID and timestamp for tracking.
Example of retrieving and using a random proxy:
import random
import requests
def get_random_proxy():
response = requests.get("http://localhost:8000/random").json()
return response["proxy"]["url"]
proxies = {
"http": get_random_proxy(),
"https": get_random_proxy()
}
my_request = requests.get("https://api.ipify.org", proxies=proxies)
For more detailed examples of using CloudProxy as a Python package, see the Python Package Usage Guide.
CloudProxy supports rolling deployments to ensure zero-downtime proxy recycling. This feature maintains a minimum number of healthy proxies during age-based recycling operations.
Enable rolling deployments with these environment variables:
# Enable rolling deployments
ROLLING_DEPLOYMENT=True
# Minimum proxies to keep available during recycling
ROLLING_MIN_AVAILABLE=3
# Maximum proxies to recycle simultaneously
ROLLING_BATCH_SIZE=2
When proxies reach their age limit:
Check rolling deployment status via the API:
# Get overall status
curl http://localhost:8000/rolling
# Get provider-specific status
curl http://localhost:8000/rolling/digitalocean
For detailed documentation, see the Rolling Deployments Guide.
CloudProxy now supports multiple accounts per provider, allowing you to:
Each provider can have multiple "instances", which represent different accounts or configurations. Each instance has its own:
To configure multiple instances, use environment variables with the instance name in the format:
PROVIDERNAME_INSTANCENAME_VARIABLE
For example, to configure two DigitalOcean accounts:
# Default DigitalOcean account
DIGITALOCEAN_ENABLED=True
DIGITALOCEAN_ACCESS_TOKEN=your_first_token
DIGITALOCEAN_DEFAULT_REGION=lon1
DIGITALOCEAN_DEFAULT_MIN_SCALING=2
# Second DigitalOcean account
DIGITALOCEAN_SECONDACCOUNT_ENABLED=True
DIGITALOCEAN_SECONDACCOUNT_ACCESS_TOKEN=your_second_token
DIGITALOCEAN_SECONDACCOUNT_REGION=nyc1
DIGITALOCEAN_SECONDACCOUNT_MIN_SCALING=3
CloudProxy exposes a comprehensive REST API for managing your proxy infrastructure. Here are some common examples:
curl -X 'GET' 'http://localhost:8000/random' -H 'accept: application/json'
curl -X 'GET' 'http://localhost:8000/' -H 'accept: application/json'
# CloudProxy will maintain exactly 5 proxies (DigitalOcean)
curl -X 'PATCH' 'http://localhost:8000/providers/digitalocean' \
-H 'Content-Type: application/json' \
-d '{"min_scaling": 5, "max_scaling": 5}'
# Or for Vultr
curl -X 'PATCH' 'http://localhost:8000/providers/vultr' \
-H 'Content-Type: application/json' \
-d '{"min_scaling": 3, "max_scaling": 3}'
import requests
# Get a random proxy
response = requests.get("http://localhost:8000/random").json()
proxy_url = response["proxy"]["url"]
# Use the proxy
proxies = {"http": proxy_url, "https": proxy_url}
result = requests.get("https://api.ipify.org", proxies=proxies)
For comprehensive API documentation with all endpoints, request/response schemas, and advanced usage examples, see the API Examples Documentation.
CloudProxy runs on a schedule of every 30 seconds to maintain the target number of proxies specified by MIN_SCALING. If the current count differs from the target, it will create or remove proxies as needed. The new proxy info will appear in IPs once they are deployed and ready to be used.
The project is at early alpha with limited features. Future enhancements may include:
See the open issues for a list of proposed features (and known issues).
This method of scraping via cloud providers has limitations, many websites have anti-bot protections and blacklists in place which can limit the effectiveness of CloudProxy. Many websites block datacenter IPs and IPs may be tarnished already due to IP recycling. Rotating the CloudProxy proxies regularly may improve results. The best solution for scraping is via proxy services providing residential IPs, which are less likely to be blocked, however are much more expensive. CloudProxy is a much cheaper alternative for scraping sites that do not block datacenter IPs nor have advanced anti-bot protection. This a point frequently made when people share this project which is why I am including this in the README.
Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated.
git checkout -b feature/AmazingFeature)git commit -m 'Add some AmazingFeature')git push origin feature/AmazingFeature)My target is to review all PRs within a week of being submitted, though sometimes it may be sooner or later.
Distributed under the MIT License. See LICENSE for more information.
Christian Laffin - @christianlaffin - christian.laffin@gmail.com
Project Link: https://github.com/claffin/cloudproxy
DIGITALOCEAN_ENABLED=True)PROXY_USERNAME and PROXY_PASSWORD are setONLY_HOST_IP is restricting access# Check container logs
docker logs <container-id>
# Verify environment variables are passed correctly
docker exec <container-id> env | grep PROXY
# Test connectivity to the API
curl http://localhost:8000/providers
AGE_LIMIT setting - proxies older than this will be automatically replaced/ui not just the root URLnpm run build in cloudproxy-ui/)For more detailed debugging, enable verbose logging by checking the application logs in the logs/ directory.
FAQs
A tool to manage cloud-based proxies for scraping
We found that cloudproxy demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
/Product
Socket Firewall Free is now bundled into Docker Hardened Images, adding build-time and dependency-install supply chain protection on top of hardened base images for Node.js, Python, and Rust.
Security News
Season’s greetings from Socket, and here’s to a calm end of year: clean dependencies, boring pipelines, no surprises.
Research
/Security News
Impostor NuGet package Tracer.Fody.NLog typosquats Tracer.Fody and its author, using homoglyph tricks, and exfiltrates Stratis wallet JSON/passwords to a Russian IP address.