Product
Introducing License Enforcement in Socket
Ensure open-source compliance with Socket’s License Enforcement Beta. Set up your License Policy and secure your software!
A Python package to read and write files in CDM format. Customized for SkyPoint use cases.
Python Spark CDM Connector by SkyPoint.
Apache Spark connector for the Microsoft Azure "Common Data Model". Reading and writing is supported and it is a work in progress. Please file issues for any bugs that you find.
For more information about the Azure Common Data Model, check out this page.
We support Azure Data Lake Service (ADLS) and AWS S3 as storage, historical data preservation using snapshots of the schema & data files and usage within PySpark, Azure Functions etc.
*Upcoming Support for incremental data refresh handling, [CDM 1.1](https://docs.microsoft.com/en-us/common-data-model/cdm-manifest and Google Cloud (Cloud Storage).
# Initialize empty model
m = Model()
# Sample dataframe
df = {"country": ["Brazil", "Russia", "India", "China", "South Africa", "ParaSF"],
"currentTime": [datetime.now(), datetime.now(), datetime.now(), datetime.now(), datetime.now(), datetime.now()],
"area": [8.516, 17.10, 3.286, 9.597, 1.221, 2.222],
"capital": ["Brasilia", "Moscow", "New Dehli", "Beijing", "Pretoria", "ParaSF"],
"population": [200.4, 143.5, 1252, 1357, 52.98, 12.34] }
df = pd.DataFrame(df)
# Generate entity from the dataframe
entity = Model.generate_entity(df, "customEntity")
# Add generated entity to model
m.add_entity(entity)
# Add model level annotation
# Annotation can be added at entity level as well as attribute level
Model.add_annotation("modelJsonAnnotation", "modelJsonAnnotationValue", m)
# Create an ADLSWriter to write into ADLS
writer = ADLSWriter("ACCOUNT_NAME", "ACCOUNT_KEY",
"CONTAINER_NAME", "STORAGE_NAME", "DATAFLOW_NAME")
# Write data as well as model.json in ADLS storage
m.write_to_storage("customEntity", df, writer)
This project welcomes contributions and suggestions.
A clean implementation for Python Objects from/to model.json file
FAQs
A Python package to read and write files in CDM format. Customized for SkyPoint use cases.
We found that cdm-connector demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Product
Ensure open-source compliance with Socket’s License Enforcement Beta. Set up your License Policy and secure your software!
Product
We're launching a new set of license analysis and compliance features for analyzing, managing, and complying with licenses across a range of supported languages and ecosystems.
Product
We're excited to introduce Socket Optimize, a powerful CLI command to secure open source dependencies with tested, optimized package overrides.