Security News
Introducing the Socket Python SDK
The initial version of the Socket Python SDK is now on PyPI, enabling developers to more easily interact with the Socket REST API in Python projects.
With automated multithreaded image downloading,
caching and optional dataset conversion.
Example reconstructions of dataset elements
using a simple Beta-VAE
Install mtgdata
with pip install mtgdata
Prepare or convert the MTG data using the command line with python -m mtgdata --help
MTG Card Face Dataset
Convert to HDF5
jpg
or png
image accessesPickle HD5F Dataset Class
You can prepare (download) all the normal
quality images
from the default
Scryfall bulk data
by running mtgdata/__main__.py
:
python3 mtgdata prepare --help
Otherwise, you can instead convert the downloaded images into an hdf5 dataset by running:
python3 mtgdata convert --help
Alternatively you can download the images from within python by simply instantiating
the mtgdata.ScryfallDataset
object. Similar arguments can be specified as that of the
command line approach.
from mtgdata import ScryfallDataset
data = ScryfallDataset(
img_type='border_crop',
bulk_type='default_cards',
transform=None,
)
# you can access the dataset elements like usual
The scrape logic used to obtain the proxy list for mtgdata.utils.proxy.ProxyDownloader
will
probably go out of date. You can override the default scrape logic used by the Dataset download
logic by registering a new scrape function.
from mtgdata.util.proxy import register_proxy_scraper
from typing import List, Dict
@register_proxy_scraper(name='my_proxy_source', is_default=True)
def custom_proxy_scraper(proxy_type: str) -> List[Dict[str, str]]:
# you should respect this setting, but we will just ignore it
assert proxy_type in ('all', 'http', 'https')
# proxies is a list of dictionaries, where each dictionary only has one entry:
# - the key is the protocol
# - the value is the matching full url
return [
{'HTTP': 'http://<my-http-proxy>.com'},
{'HTTPS': 'https://<my-https-proxy>.com'},
]
The images can be convert to hdf5 format by running the file mtgdata.scryfall_convert
.
Various arguments can be specified, please see the argparse arguments at the bottom of
the file for more information.
python3 mtgdata/scryfall_convert.py
The resulting data file will have the data
key corresponding to the images data.
Alternatively you can convert and generate the hdf5 dataset from within python by simply calling
the mtgdata.scryfall_convert.generate_converted_dataset
function. Similar arguments can be specified
as that of the command line approach.
from mtgdata import generate_converted_dataset
generate_converted_dataset(
out_img_type='border_crop',
out_bulk_type='default_cards',
save_root='./data/converted/',
out_obs_size=(224, 160),
convert_speed_test=True,
)
We provide a helper dataset class for loading this generated file.
from torch.utils.data import DataLoader
from mtgdata import Hdf5Dataset
# this h5py dataset supports pickling, and can be wrapped with a pytorch dataset.
data = Hdf5Dataset(
h5_path='data/converted/mtg-default_cards-normal-60459x224x160x3.h5', # name will differ
h5_dataset_name='data',
transform=None,
)
# you can wrap the dataset with a pytorch dataloader like usual, and specify more than one worker
dataloader = DataLoader(data, shuffle=True, num_workers=2, batch_size=64)
# to load the data into memory as a numpy array, you can call `data = data.numpy()`
# this will take a long time depending on your disk speed and use a lot of memory.
FAQs
MTG image dataset with automatic image scraping and conversion.
We found that mtgdata demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
The initial version of the Socket Python SDK is now on PyPI, enabling developers to more easily interact with the Socket REST API in Python projects.
Security News
Floating dependency ranges in npm can introduce instability and security risks into your project by allowing unverified or incompatible versions to be installed automatically, leading to unpredictable behavior and potential conflicts.
Security News
A new Rust RFC proposes "Trusted Publishing" for Crates.io, introducing short-lived access tokens via OIDC to improve security and reduce risks associated with long-lived API tokens.