azure-schemaregistry
Advanced tools
Azure Schema Registry is a schema repository service hosted by Azure Event Hubs, providing schema storage, versioning, and management. The registry is leveraged by encoders to reduce payload size while describing payload structure with schema identifiers rather than full schemas. This package provides:
A client library to register and retrieve schemas and their respective properties.
An JSON schema-based encoder capable of encoding and decoding payloads containing Schema Registry schema identifiers, corresponding to JSON schemas used for validation, and encoded content.
Source code | Package (PyPi) | Package (Conda) | API reference documentation | Samples | Changelog
Azure SDK Python packages support for Python 2.7 has ended on 01 January 2022. For more information and questions, please refer to https://github.com/Azure/azure-sdk-for-python/issues/20691
Install the Azure Schema Registry client library for Python with pip:
pip install azure-schemaregistry
To use the built-in jsonschema validators with the JSON Schema Encoder, install jsonencoder extras:
pip install azure-schemaregistry[jsonencoder]
To use this package, you must have:
Interaction with Schema Registry starts with an instance of SchemaRegistryClient class. The client constructor takes an Azure Event Hubs fully qualified namespace and an Azure Active Directory credential:
The fully qualified namespace of the Schema Registry instance should follow the format: <yournamespace>.servicebus.windows.net.
An AAD credential that implements the TokenCredential protocol should be passed to the constructor. There are implementations of the TokenCredential protocol available in the
azure-identity package. To use the credential types provided by azure-identity, please install the Azure Identity client library for Python with pip:
pip install azure-identity
pip install aiohttp
Create client using the azure-identity library:
from azure.schemaregistry import SchemaRegistryClient
from azure.identity import DefaultAzureCredential
credential = DefaultAzureCredential()
# Namespace should be similar to: '<your-eventhub-namespace>.servicebus.windows.net/'
fully_qualified_namespace = os.environ['SCHEMAREGISTRY_FULLY_QUALIFIED_NAMESPACE']
schema_registry_client = SchemaRegistryClient(fully_qualified_namespace, credential)
Create JsonSchemaEncoder using the azure-schemaregistry library:
import os
from azure.schemaregistry import SchemaRegistryClient
from azure.schemaregistry.encoder.jsonencoder import JsonSchemaEncoder
from azure.identity import DefaultAzureCredential
credential = DefaultAzureCredential()
# Namespace should be similar to: '<your-eventhub-namespace>.servicebus.windows.net'
fully_qualified_namespace = os.environ['SCHEMAREGISTRY_FULLY_QUALIFIED_NAMESPACE']
group_name = os.environ['SCHEMAREGISTRY_GROUP']
schema_registry_client = SchemaRegistryClient(fully_qualified_namespace, credential)
encoder = JsonSchemaEncoder(client=schema_registry_client, group_name=group_name)
Schema: Schema is the organization or structure for data. More detailed information can be found here.
Schema Group: A logical group of similar schemas based on business criteria, which can hold multiple versions of a schema. More detailed information can be found here.
SchemaRegistryClient: SchemaRegistryClient provides the API for storing and retrieving schemas in schema registry.
JsonSchemaEncoder: Provides API to encode content to and decode content from Binary Encoding, validate content against a JSON Schema, and cache schemas/schema IDs retrived from the registry using the SchemaRegistryClient locally.
OutboundMessageContent: Protocol defined under azure.schemaregistry that allows for JsonSchemaEncoder.encode interoperability with certain Azure Messaging SDK message types. Support has been added to:
azure.eventhub.EventData for azure-eventhub>=5.9.0InboundMessageContent: Protocol defined under azure.schemaregistry that allows for JsonSchemaEncoder.decode interoperability with certain Azure Messaging SDK message types. Support has been added to:
azure.eventhub.EventData for azure-eventhub>=5.9.0If a message type that follows the OutboundMessageContent protocol is provided to the JsonSchemaEncoder, it will set the corresponding content and content type properties. If a message type object that follows the InboundMessageContent protocol is provided to the encoder, it will get the corresponding content and content type properties. These are defined as:
content: Binary-encoded, JSON schema-validated payload (in general, format-specific payload)
content type: a string of the format application/json;serialization=Json+<schema ID>, where:
application/json;serialization=Json is the format indicator<schema ID> is the hexadecimal representation of GUID, same format and byte order as the string from the Schema Registry service.If EventData is passed in as the message type, the following properties will be set on the EventData object:
The body property will be set to the encoded content value.
The content_type property will be set to the content type value.
If message type is not provided, and by default, the encoder will create the following dict:
{"content": <encoded payload>, "content_type": 'application/json;serialization=Json+<schema ID>'}
The following sections provide several code snippets covering some of the most common Schema Registry and Json Schema Encoder tasks, including:
Use SchemaRegistryClient.register_schema method to register a schema.
import os
from azure.identity import DefaultAzureCredential
from azure.schemaregistry import SchemaRegistryClient
token_credential = DefaultAzureCredential()
fully_qualified_namespace = os.environ['SCHEMAREGISTRY_AVRO_FULLY_QUALIFIED_NAMESPACE']
group_name = os.environ['SCHEMA_REGISTRY_GROUP']
name = "your-schema-name"
format = "Avro"
definition = """
{"namespace": "example.avro",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
"""
schema_registry_client = SchemaRegistryClient(fully_qualified_namespace=fully_qualified_namespace, credential=token_credential)
with schema_registry_client:
schema_properties = schema_registry_client.register_schema(group_name, name, definition, format)
id = schema_properties.id
Get the schema definition and its properties by schema id.
import os
from azure.identity import DefaultAzureCredential
from azure.schemaregistry import SchemaRegistryClient
token_credential = DefaultAzureCredential()
fully_qualified_namespace = os.environ['SCHEMAREGISTRY_AVRO_FULLY_QUALIFIED_NAMESPACE']
schema_id = 'your-schema-id'
schema_registry_client = SchemaRegistryClient(fully_qualified_namespace=fully_qualified_namespace, credential=token_credential)
with schema_registry_client:
schema = schema_registry_client.get_schema(schema_id)
definition = schema.definition
properties = schema.properties
Get the schema definition and its properties by schema version.
import os
from azure.identity import DefaultAzureCredential
from azure.schemaregistry import SchemaRegistryClient
token_credential = DefaultAzureCredential()
fully_qualified_namespace = os.environ['SCHEMAREGISTRY_AVRO_FULLY_QUALIFIED_NAMESPACE']
group_name = os.environ["SCHEMAREGISTRY_GROUP"]
name = "your-schema-name"
version = int("<your schema version>")
schema_registry_client = SchemaRegistryClient(fully_qualified_namespace=fully_qualified_namespace, credential=token_credential)
with schema_registry_client:
schema = schema_registry_client.get_schema(group_name=group_name, name=name, version=version)
definition = schema.definition
properties = schema.properties
Get the schema id of a schema by schema definition and its properties.
import os
from azure.identity import DefaultAzureCredential
from azure.schemaregistry import SchemaRegistryClient
token_credential = DefaultAzureCredential()
fully_qualified_namespace = os.environ['SCHEMAREGISTRY_AVRO_FULLY_QUALIFIED_NAMESPACE']
group_name = os.environ['SCHEMA_REGISTRY_GROUP']
name = "your-schema-name"
format = "Avro"
definition = """
{"namespace": "example.avro",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
"""
schema_registry_client = SchemaRegistryClient(fully_qualified_namespace=fully_qualified_namespace, credential=token_credential)
with schema_registry_client:
schema_properties = schema_registry_client.register_schema(group_name, name, definition, format)
id = schema_properties.id
Use the SchemaRegistryClient to pre-register the schema. Encode and validate the content with the JsonSchemaEncoder.
The encode method automatically retrieves the schema from the Schema Registry Service, validates against the content, and caches the schema locally.
import os
import json
from azure.schemaregistry import SchemaRegistryClient, SchemaFormat
from azure.schemaregistry.encoder.jsonencoder import JsonSchemaEncoder
from azure.identity import DefaultAzureCredential
from azure.eventhub import EventData
token_credential = DefaultAzureCredential()
fully_qualified_namespace = os.environ['SCHEMAREGISTRY_JSON_FULLY_QUALIFIED_NAMESPACE']
group_name = os.environ['SCHEMAREGISTRY_GROUP']
format = SchemaFormat.JSON
DRAFT2020_12_SCHEMA_IDENTIFIER = "https://json-schema.org/draft/2020-12/schema"
schema = {
"$id": "https://example.com/person.schema.json",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "Person",
"type": "object",
"properties": {
"name": {
"type": "string",
"description": "Person's name."
},
"favorite_color": {
"type": "string",
"description": "Favorite color."
},
"favorite_number": {
"description": "Favorite number.",
"type": "integer",
}
}
}
name = schema["title"]
definition = json.dumps(schema)
schema_registry_client = SchemaRegistryClient(fully_qualified_namespace, token_credential)
schema_properties = schema_registry_client.register_schema(group_name, name, definition, format)
schema_id = schema_properties.id
# group_name only needed if passing `schema` to encode
encoder = JsonSchemaEncoder(client=schema_registry_client, validate=DRAFT2020_12_SCHEMA_IDENTIFIER, group_name=group_name)
with encoder:
dict_content = {"name": "Ben", "favorite_number": 7, "favorite_color": "red"}
event_data = encoder.encode(dict_content, schema_id=schema_id, message_type=EventData)
# OR
message_content_dict = encoder.encode(dict_content, schema_id=schema_id)
event_data = EventData.from_message_content(message_content_dict["content"], message_content_dict["content_type"])
# OR
dict_content = {"name": "Ben", "favorite_number": 7, "favorite_color": "red"}
message_content = encoder.encode(dict_content, schema=definition) # group_name required in constructor when `schema` is passed
Decode the content with the JsonSchemaEncoder.
The decode method automatically retrieves the schema from the Schema Registry Service, validates against the content, and caches the schema locally.
import os
from azure.schemaregistry import SchemaRegistryClient
from azure.schemaregistry.encoder.jsonencoder import JsonSchemaEncoder
from azure.identity import DefaultAzureCredential
token_credential = DefaultAzureCredential()
fully_qualified_namespace = os.environ['SCHEMAREGISTRY_FULLY_QUALIFIED_NAMESPACE']
group_name = os.environ["SCHEMAREGISTRY_GROUP"]
DRAFT2020_12_SCHEMA_IDENTIFIER = "https://json-schema.org/draft/2020-12/schema"
schema_registry_client = SchemaRegistryClient(fully_qualified_namespace, token_credential)
encoder = JsonSchemaEncoder(client=schema_registry_client, validate=DRAFT2020_12_SCHEMA_IDENTIFIER)
with encoder:
# event_data is an EventData object with encoded body
decoded_content = encoder.decode(event_data)
# OR
# content_dict is a TypedDict with encoded content and JSON content type
decoded_content = encoder.decode(content_dict)
Integration with Event Hubs to send an EventData object with body set to encoded content and corresponding content_type.
import os
from azure.eventhub import EventHubProducerClient, EventData
from azure.schemaregistry import SchemaRegistryClient
from azure.schemaregistry.encoder.jsonencoder import JsonSchemaEncoder
from azure.identity import DefaultAzureCredential
token_credential = DefaultAzureCredential()
fully_qualified_namespace = os.environ['SCHEMAREGISTRY_JSON_FULLY_QUALIFIED_NAMESPACE']
group_name = os.environ['SCHEMAREGISTRY_GROUP']
eventhub_connection_str = os.environ['EVENT_HUB_CONN_STR']
eventhub_name = os.environ['EVENT_HUB_NAME']
schema_id = os.environ['PERSON_JSON_SCHEMA_ID']
DRAFT2020_12_SCHEMA_IDENTIFIER = "https://json-schema.org/draft/2020-12/schema"
schema_registry_client = SchemaRegistryClient(fully_qualified_namespace, token_credential)
json_schema_encoder = JsonSchemaEncoder(client=schema_registry_client, validate=DRAFT2020_12_SCHEMA_IDENTIFIER)
eventhub_producer = EventHubProducerClient.from_connection_string(
conn_str=eventhub_connection_str,
eventhub_name=eventhub_name
)
with eventhub_producer, json_schema_encoder:
event_data_batch = eventhub_producer.create_batch()
dict_content = {"name": "Bob", "favorite_number": 7, "favorite_color": "red"}
event_data = json_schema_encoder.encode(dict_content, schema_id=schema_id, message_type=EventData)
event_data_batch.add(event_data)
eventhub_producer.send_batch(event_data_batch)
Integration with Event Hubs to receive an EventData object and decode the encoded body value.
import os
from azure.eventhub import EventHubConsumerClient
from azure.schemaregistry import SchemaRegistryClient
from azure.schemaregistry.encoder.jsonencoder import JsonSchemaEncoder
from azure.identity import DefaultAzureCredential
token_credential = DefaultAzureCredential()
fully_qualified_namespace = os.environ['SCHEMAREGISTRY_JSON_FULLY_QUALIFIED_NAMESPACE']
group_name = os.environ['SCHEMAREGISTRY_GROUP']
eventhub_connection_str = os.environ['EVENT_HUB_CONN_STR']
eventhub_name = os.environ['EVENT_HUB_NAME']
DRAFT2020_12_SCHEMA_IDENTIFIER = "https://json-schema.org/draft/2020-12/schema"
schema_registry_client = SchemaRegistryClient(fully_qualified_namespace, token_credential)
json_schema_encoder = JsonSchemaEncoder(client=schema_registry_client, validate=DRAFT2020_12_SCHEMA_IDENTIFIER)
eventhub_consumer = EventHubConsumerClient.from_connection_string(
conn_str=eventhub_connection_str,
consumer_group='$Default',
eventhub_name=eventhub_name,
)
def on_event(partition_context, event):
decoded_content = json_schema_encoder.decode(event)
with eventhub_consumer, json_schema_encoder:
eventhub_consumer.receive(on_event=on_event, starting_position="-1")
Schema Registry clients raise exceptions defined in Azure Core if errors are encountered when communicating with the Schema Registry service.
Errors when encoding and decoding related to invalid content/content types will be raised as azure.schemaregistry.encoder.jsonencoder.InvalidContentError, where __cause__ will possibly contain an underlying exception.
This library uses the standard logging library for logging. Basic information about HTTP sessions (URLs, headers, etc.) is logged at INFO level.
Detailed DEBUG level logging, including request/response bodies and unredacted
headers, can be enabled on a client with the logging_enable argument:
import sys
import os
import logging
from azure.schemaregistry import SchemaRegistryClient
from azure.schemaregistry.encoder.jsonencoder import JsonSchemaEncoder
from azure.identity import DefaultAzureCredential
# Create a logger for the SDK
logger = logging.getLogger('azure.schemaregistry')
logger.setLevel(logging.DEBUG)
# Configure a console output
handler = logging.StreamHandler(stream=sys.stdout)
logger.addHandler(handler)
fully_qualified_namespace = os.environ['SCHEMAREGISTRY_FULLY_QUALIFIED_NAMESPACE']
group_name = os.environ['SCHEMAREGISTRY_GROUP']
DRAFT2020_12_SCHEMA_IDENTIFIER = "https://json-schema.org/draft/2020-12/schema"
credential = DefaultAzureCredential()
# This client will log detailed information about its HTTP sessions, at DEBUG level
schema_registry_client = SchemaRegistryClient(fully_qualified_namespace, credential, logging_enable=True)
encoder = JsonSchemaEncoder(client=schema_registry_client, validate=DRAFT2020_12_SCHEMA_IDENTIFIER)
Similarly, logging_enable can enable detailed logging for a single operation,
even when it isn't enabled for the client:
schema_registry_client.get_schema(schema_id, logging_enable=True)
Please take a look at the samples directory for detailed examples of how to use this library to register and retrieve schema to/from Schema Registry.
This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.
When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.
This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.
This version and all future versions will require Python 3.8+. Python 3.7 is no longer supported.
JsonSchemaEncoder have been added under azure.schemaregistry.encoder.jsonencoder.InvalidContentError have been added under azure.schemaregistry.encoder.jsonencoder for use with the JsonSchemaEncoder.MessageContent, OutboundMessageContent,InboundMessageContent, and SchemaContentValidate have been added under azure.schemaregistry as protocols for use with the JsonSchemaEncoder and/or future encoder implementations.Json and Custom have been added to supported formats in SchemaFormat.V2022_10 has been added to ApiVersion and set as the default API version.register_schema and get_schema_properties that did not accept case insensitive strings as an argument to the format parameter.format property.V2023_07_01 in ApiVersion.
PROTOBUF in SchemaFormat.V2023_07_01 has been added to ApiVersion and set as the default API version.
Protobuf has been added to supported formats in SchemaFormat.The following features are experimental and may be removed:
JsonSchemaEncoder have been added under azure.schemaregistry.encoder.jsonencoder.InvalidContentError and JsonSchemaDraftIdentifier have been added under azure.schemaregistry.encoder.jsonencoder for use with the JsonSchemaEncoder.MessageType, MessageContent, SchemaContentValidate, SchemaEncoder have been added under azure.schemaregistry as protocols to define/for use with the JsonSchemaEncoder and future encoder implementations.V2022_10 has been added to ApiVersion and set as the default api version.
Json and Custom have been added to supported formats in SchemaFormat.register_schema and get_schema_properties that did not accept case insensitive strings as an argument to the format parameter.This version and all future versions will require Python 3.7+, Python 3.6 is no longer supported.
group_name, name, and version have been added as optional parameters to the get_schema method on the sync and async SchemaRegistryClient.version has been added to SchemaProperties.SchemaRegistryClient.This version and all future versions will require Python 3.6+. Python 2.7 is no longer supported.
group_name and name have been added as instance variables to SchemaProperties.Note: This is the first stable release of our efforts to create a user-friendly and Pythonic client library for Azure Schema Registry.
SchemaRegistryClient is the top-level client class interacting with the Azure Schema Registry Service. It provides three methods:
register_schema: Store schema in the service by providing schema group name, schema name, schema definition, and schema format.get_schema: Get schema definition and its properties by schema id.get_schema_properties: Get schema properties by providing schema group name, schema name, schema definition, and schema format.SchemaProperties has the following instance variables: id and format:
format has been changed from str to SchemaFormat.Schema has the following properties: properties and definition.SchemaFormat provides the schema format to be stored by the service. Currently, the only supported format is Avro.api_version has been added as a keyword arg to the sync and async SchemaRegistryClient constructors.version instance variable in SchemaProperties has been removed.schema_definition instance variable in Schema has been renamed definition.id parameter in get_schema method on sync and async SchemaRegistryClient has been renamed schema_id.schema_definition parameter in register_schema and get_schema_properties methods on sync and async SchemaRegistryClient has been renamed definition.serializer namespace has been removed from azure.schemaregistry.get_schema_id method on sync and async SchemaRegistryClient has been renamed get_schema_properties.schema_id parameter in get_schema method on sync and async SchemaRegistryClient has been renamed id.register_schema and get_schema_properties methods on sync and async SchemaRegistryClient now take in the following parameters in the given order:
group_name, which has been renamed from schema_groupname, which has been renamed from schema_nameschema_definition, which has been renamed from schema_contentformat, which has been renamed from serialization_typeendpoint parameter in SchemaRegistryClient constructor has been renamed fully_qualified_namespacelocation instance variable in SchemaProperties has been removed.Schema and SchemaProperties no longer have positional parameters, as they will not be constructed by the user.This version and all future versions will require Python 2.7 or Python 3.6+, Python 3.5 is no longer supported.
Version 1.0.0b1 is the first preview of our efforts to create a user-friendly and Pythonic client library for Azure Schema Registry.
New features
SchemaRegistryClient is the top-level client class interacting with the Azure Schema Registry Service. It provides three methods:
register_schema: Store schema into the service.get_schema: Get schema content and its properties by schema id.get_schema_id: Get schema id and its properties by schema group, schema name, serialization type and schema content.FAQs
Microsoft Azure Azure Schema Registry Client Library for Python
We found that azure-schemaregistry demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 2 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
Malicious Go packages are impersonating popular libraries to install hidden loader malware on Linux and macOS, targeting developers with obfuscated payloads.
Security News
Bybit's $1.46B hack by North Korea's Lazarus Group pushes 2025 crypto losses to $1.6B in just two months, already surpassing all of 2024's $1.49B total.
Security News
OpenSSF has published OSPS Baseline, an initiative designed to establish a minimum set of security-related best practices for open source software projects.