Universal ID leverages both MessagePack and Brotli (a combo built for speed and best-in-class data compression). When combined, these libraries are up to 30% faster and within 2-5% compression rates compared to Protobuf. ↗
Universal ID introduces a paradigm shift that enables straightforward simple solutions ↗ for a variety of complex problem domains.
[!TIP] All the code examples below can be tested on your local machine. Just clone the repo (↑or use Gitpod above↑) and run
bin/consoleto begin exploring. Don't forget to executebundlefirst to ensure all dependencies are up to date. Happy coding!
Universal ID introduces a new URI defintion that can recursively serialize any Ruby object into an URL-safe string which can be safely transported via HTTP.
[!NOTE] The payload is optimized to be as small as possible... especially notable with large objects.
The best part: The API is simple.
data = :ANY_OBJECT_YOU_CAN_IMAGINE
uid = URI::UID.build(data)
#<URI::UID payload=Cw6AxxoAQU5ZX09CSkVDVF9ZT1VfQ0FOX0lNQ..., fingerprint=CwWAkccHf6ZTeW1ib2wD>
uid.payload
"Cw6AxxoAQU5ZX09CSkVDVF9ZT1VfQ0FOX0lNQUdJTkUD"
uid.fingerprint
"CwWAkccHf6ZTeW1ib2wD"
uri = uid.to_s
"uid://universalid/Cw6AxxoAQU5ZX09CSkVDVF9ZT1VfQ0FOX0lNQUdJTkUD#CwWAkccHf6ZTeW1ib2wD"
parsed = URI::UID.parse(uri)
#<URI::UID payload=Cw6AxxoAQU5ZX09CSkVDVF9ZT1VfQ0FOX0lNQ..., fingerprint=CwWAkccHf6ZTeW1ib2wD>
parsed.decode
:ANY_OBJECT_YOU_CAN_IMAGINE
# it's also possible to parse the payload by itself
parsed = URI::UID.from_payload(uid.payload)
#<URI::UID payload=Cw6AxxoAQU5ZX09CSkVDVF9ZT1VfQ0FOX0lNQ..., fingerprint=CwWAkccHf6ZTeW1ib2wD>
parsed.decode
:ANY_OBJECT_YOU_CAN_IMAGINE
Universal ID supports most native Ruby primitives:
NilClassBigDecimalComplexDateDateTimeFalseClassFloatIntegerRangeRationalRegexpStringSymbolTimeTrueClassYou can use Universal ID to serialize individual primitives, but this actually serves as the foundation for more advanced use-cases.
uri = URI::UID.build(:demo).to_s
#=> "uid://universalid/iwKA1gBkZW1vAw#CwWAkccHf6ZTeW1ib2wD"
uid = URI::UID.parse(uri)
#=> #<URI::UID payload=iwKA1gBkZW1vAw, fingerprint=CwWAkccHf6ZTeW1ib2wD>
uid.decode
#=> :demo
Composite (or complex, compound, etc.) datatype support is where things start to get interesting. Universal ID supports the following native Ruby composite datatypes:
ArrayHashOpenStructSetStructarray = [1, 2, 3, [:a, :b, :c, [true]]]
uri = URI::UID.build(array).to_s
#=> "uid://universalid/iweAlAECA5TUAGHUAGLUAGORwwM#iwSAkccGf6VBcnJheQM"
uid = URI::UID.parse(uri)
#=> #<URI::UID payload=iweAlAECA5TUAGHUAGLUAGORwwM, fingerprint=iwSAkccGf6VBcnJheQM>
uid.decode
#=> [1, 2, 3, [:a, :b, :c, [true]]]
uid.decode == array
#=> true
hash = {a: 1, b: 2, c: 3, array: [1, 2, 3, [:a, :b, :c, [true]]]}
uri = URI::UID.build(hash).to_s
#=> "uid://universalid/CxKAhNQAYQHUAGIC1ABjA8cFAGFycmF5lAECA5TUAGHUAGLUAGORwwM#CwS..."
uid = URI::UID.parse(uri)
#=> #<URI::UID payload=CxKAhNQAYQHUAGIC1ABjA8cFAGFycmF5lAECA..., fingerprint=CwSAkccFf6RIYXNoAw>
uid.decode
#=> {:a=>1, :b=>2, :c=>3, :array=>[1, 2, 3, [:a, :b, :c, [true]]]}
uid.decode == hash
#=> true
Book = Struct.new(:title, :author, :isbn, :published_year)
book = Book.new("The Great Gatsby", "F. Scott Fitzgerald", "9780743273565", 1925)
uri = URI::UID.build(book).to_s
#=> "uid://universalid/G2YAoGTomv9tT1ilLRgVC9vIpmuBo-k84FZ0G8-siFMBNsbW0dpBE0Tnm96..."
uid = URI::UID.parse(uri)
#=> #<URI::UID payload=G2YAoGTomv9tT1ilLRgVC9vIpmuBo-k84FZ0G..., fingerprint=CwSAkccFf6RCb29rAw>
uid.decode
#=> #<struct Book title="The Great Gatsby", author="F. Scott Fitzgerald", isbn="9780743273565", published_year=1925>
uid.decode == book
#=> true
The following extension datatypes ship with Universal ID:
ActiveRecord::BaseActiveRecord::RelationActiveSupport::Cache::EntryActiveSupport::Cache::StoreActiveSupport::TimeWithZoneGlobalIDSignedGlobalID[!NOTE] Extensions are autoloaded whenever the related datatype is detected.
[!IMPORTANT] Why Universal ID with ActiveRecord? ActiveRecord already has GlobalID, a robust library for serializing individual models. Universal ID covers a much wider range of use cases.
Here are a few reasons you may want to consider Universal ID with ActiveRecord.
New Records: Universal ID can serialize models that haven't been saved to the database yet.
Changesets: Universal ID can serialize ActiveRecord models with unsaved changes, ensuring that even transient states are captured.
Associations: Universal ID goes beyond single models. It can include associated records, even those with unsaved changes, creating a comprehensive snapshot of complex record states.
Copying/Cloning: Universal ID supports making copies of records (including associations), making it ideal for duplicating complex datasets.
More Control: Universal ID gives you control over the serialization process. You can choose which columns to include/exclude, allowing for tailored, optimized payloads to fit your needs.
Queries/Relations: Universal ID also supports ActiveRecord::Relations, enabling the serialization of complex database queries and scopes.
In summary, while GlobalID excels in its specific use case, Universal ID offers more power for use-cases that involve unsaved records, complex associations, data cloning, and database queries.
# setup some records
campaign = Campaign.create(name: "My Campaign")
email = campaign.emails.create(subject: "First Email")
attachment = email.attachments.create(file_name: "data.pdf")
# ensure associations are loaded so they can be included in an UID
campaign.emails.load
campaign.emails.each { |e| e.attachments.load }
# make some unsaved changes
email.subject = "1st Email"
# add an unsaved record
campaign.emails.build(subject: "2nd Email")
# introspection
campaign.emails.size #=> 2
campaign.emails.loaded? #=> true
campaign.emails.last.new_record? #=> true
options = {
include_changes: true,
include_descendants: true,
descendant_depth: 2
}
uri = URI::UID.build(campaign, options).to_s
#=> "uid://universalid/GxYBYGT6_Xn_OrelIDRWhQQgvbS5gQxV7EJKe3paIiEFmEEc1gLKw8Pl2-k..."
uid = URI::UID.parse(uri)
#=> #<URI::UID payload=GxYBYGT6_Xn_OrelIDRWhQQgvbS5gQxV7EJKe..., fingerprint=CwuAkscJf6hDYW1wYWlnbtf_ReuZnGWeG5MD>
decoded = uid.decode
#=> "#<Campaign id: 13, name: \"My Campaign\" ...>
decoded == campaign
#=> true
# introspection
decoded.emails.size #=> 2
decoded.emails.loaded? #=> true
decoded.emails.first.changed? #=> true
decoded.emails.first.changes #=> {"subject"=>["First Email", "1st Email"]}
decoded.emails.last.new_record? #=> true
decoded.save #=> true
decoded.emails.last.persisted? #=> true
Universal ID is extensible, enabling you to register your own datatypes with custom serialization rules. Simply convert the required data to a Ruby primitive or composite value.
# create a custom type
class UserSettings
attr_accessor :user_id, :preferences
def initialize(user_id, preferences = {})
@user_id = user_id
@preferences = preferences
end
end
# register the custom type with Universal ID
UniversalID::MessagePackFactory.register(
type: UserSettings,
packer: ->(user_preferences, packer) do
packer.write user_preferences.user_id
packer.write user_preferences.preferences
end,
unpacker: ->(unpacker) do
user_id = unpacker.read
preferences = unpacker.read
UserSettings.new user_id, preferences
end
)
# create an instance of the custom type
settings = UserSettings.new(1,
theme: "dark",
notifications: "email",
language: "en",
layout: "grid",
privacy: "private"
)
# serialize the custom type
uri = URI::UID.build(settings).to_s
#=> "uid://universalid/G1QAQAT-c_cO7qJcAk-TtsAiadci_IA5xoH7NV3bYttEww7xuUkzasu2HEO..."
# deserialize the custom type
uid = URI::UID.parse(uri)
#=> #<URI::UID payload=G1QAQAT-c_cO7qJcAk-TtsAiadci_IA5xoH7N..., fingerprint=CwiAkccNf6xVc2VyU2V0dGluZ3MD>
uid.decode
=> #<UserSettings:0x000000011d0deb20 @preferences={:theme=>"dark", :notifications=>"email", :language=>"en", :layout=>"grid", :privacy=>"private"}, @user_id=1>
Universal ID supports a small, but powerful, set of options used to "prepack" the object before it's packed with MessagePack. These options instruct Universal ID on how to prepare the object for serialization.
prepack:
# ..........................................................................................................
# A list of attributes to exclude (for objects like Hash, OpenStruct, Struct, etc.)
# Takes prescedence over the`include` list
exclude: []
# ..........................................................................................................
# A list of attributes to include (for objects like Hash, OpenStruct, Struct, etc.)
include: []
# ..........................................................................................................
# Whether or not to include blank values when packing (nil, {}, [], "", etc.)
include_blank: true
# ==========================================================================================================
# Database records
database:
# ......................................................................................................
# Whether or not to include primary/foreign keys
# Setting this to `false` can be used to make a copy of an existing record
include_keys: true
# ......................................................................................................
# Whether or not to include date/time timestamps (created_at, updated_at, etc.)
# Setting this to `false` can be used to make a copy of an existing record
include_timestamps: true
# ......................................................................................................
# Whether or not to include unsaved changes
# Assign to `true` when packing new records
include_changes: false
# ......................................................................................................
# Whether or not to include loaded in-memory descendants (i.e. child associations)
include_descendants: false
# ......................................................................................................
# The max depth (number) of loaded in-memory descendants to include when `include_descendants == true`
# For example, a value of (2) would include the following:
# Parent > Child > Grandchild
descendant_depth: 0
Options can be applied whenever creating a UID.
hash = { a: 1, b: 2, c: 3 }
uri = URI::UID.build(hash, exclude: [:b]).to_s
#=> "uid://universalid/CwSAgtQAYQHUAGMDAw#CwSAkccFf6RIYXNoAw"
uid = URI::UID.parse(uri)
#=> #<URI::UID payload=CwSAgtQAYQHUAGMDAw, fingerprint=CwSAkccFf6RIYXNoAw>
uid.decode
#=> {:a=>1, :c=>3}
[!NOTE] Options can be passed in structured or flat format.
It's also possible to register frequently used options.
# app/config/changed.yml
prepack:
include_blank: false
database:
include_changes: true
include_descendants: true
descendant_depth: 2
UniversalID::Settings.register :changed, File.expand_path("app/config/changed.yml", __dir__)
uid = URI::UID.build(record, UniversalID::Settings[:changed])
Each UID is fingerprinted as part of the serialization process.
Fingerprints are comprised of the following components:
Class (Class) - The encoded object's classTimestamp (Time) - The mtime (UTC) of the file that defined the object's classFingerprints provide a simple mechanism to help manage data format versions... minimizing the need for custom versioning solutions.
Whenever the class definition changes, the mtime updates, resulting in a different fingerprint.
This is especially useful in scenarios where the data format evolves over time, such as in long-lived applications.
uid = URI::UID.build(campaign)
uid.fingerprint
#=> "CwuAkscJf6hDYW1wYWlnbtf_ReuZnGWeG5MD"
uid.fingerprint(decode: true)
#=> [Campaign(id: integer, ...), <Time>]
[!NOTE] The timestamp or
mtimeis determined the moment a UID is created.
[!TIP] Fingerprints can help you maintain consistency and reliability when working with serialized data over time. While fingerpint creation is automatic and implicit, usage is optional... ready whenever you need it.
Make a copy of an ActiveRecord model (with loaded associations).
campaign = Campaign.first
# ensure desired associations are loaded so they can be included in an UID
campaign.emails.load
campaign.emails.each { |e| e.attachments.load }
# introspection
campaign.id #=> 1
campaign.emails.map(&:id) #=> [1, 2]
campaign.emails.map(&:attachments).flatten.map(&:id)
#=> [1, 2, 3, 4]
# setup options for copying
options = {
include_blank: false,
include_keys: false,
include_timestamps: false,
include_descendants: true,
descendant_depth: 2
}
uri = URI::UID.build(campaign, options).to_s
#=> "uid://universalid/G7kAIBylMxZa7MouY3gUqHKkIx3hk4s8NT5xWwQsDc7lKUkGWM4DHsCxQZK..."
uid = URI::UID.parse(uri)
#=> #<URI::UID payload=G7kAIBylMxZa7MouY3gUqHKkIx3hk4s8NT5xW..., fingerprint=CwuAkscJf6hDYW1wYWlnbtf_ReuZnGWeG5MD>
copy = uid.decode
#=> #<Campaign:0x00000001135c7448 id: nil, name: "My Campaign", ...>
copy == campaign
#=> false
# introspection
copy.new_record? #=> true
copy.id #=> nil
copy.emails.map(&:id) #=> [nil, nil]
copy.emails.map(&:attachments).flatten.map(&:id)
#=> [nil, nil, nil, nil]
# create the copy (new records) in the database
copy.save #=> true
[!TIP] If you don't need a URL-Safe UID, you can use
UniversalID::Packerto speed things up a bit.
packed = UniversalID::Packer.pack(campaign, options)
copy = UniversalID::Packer.unpack(packed)
copy.save
Universal ID also supports ActiveRecord relations/scopes. You can easily serialize complex queries into a portable and sharable format.
relation = Campaign.joins(:emails).where("emails.subject LIKE ?", "Flash Sale%")
uri = URI::UID.build(relation).to_s
#=> "uid://universalid/G90EQCwLeEP1oQtHFksrdN5YS4ju5TryFZwBJgh2toqS3SKEVSl1FoNtZjI..."
uid = URI::UID.parse(encoded)
#=> #<URI::UID payload=G90EQCwLeEP1oQtHFksrdN5YS4ju5TryFZwBJ..., fingerprint=CxKAkscXf7ZBY3RpdmVSZWNvcmQ6OlJlbGF0a...>
decoded = uid.decode
# introspection
decoded == relation #=> true
decoded.is_a? ActiveRecord::Relation #=> true
decoded.loaded? #=> false
# run the query
campaigns = decoded.load
[!NOTE] Universal ID clears cached data within the relation before encoding. This minimizes payload size while preserving the integrity of the underlying query.
Features like signing (to prevent tampering), purpose, and expiration are provided by SignedGlobalIDs.
These features (and more) will eventually be added to Universal ID, but until then...
simply convert your UID to a SignedGlobalID to add these features to any Universal ID.
data = OpenStruct.new(name: "Demo", value: "Example")
sgid = URI::UID.build(data).to_sgid_param(for: "purpose", expires_in: 1.hour)
#=> "eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaEpJZ0plQTJkcFpEb3ZMM1Z1YVhabGNuTmhiQzFwWkM5V..."
uid = URI::UID.from_sgid(sgid, for: "purpose")
#=> #<URI::UID payload=Cw-Axxx-gtYAbmFtZaREZW1vxwUAdmFsdWWnR..., fingerprint=ixqAkscof9kmVW5pdmVyc2FsSUQ6OkV4dGVuc...>
decoded = uid.decode
#=> #<OpenStruct name="Demo", value="Example">
# a mismatched purpose returns nil... as expected
URI::UID.from_sgid(sgid, for: "mismatch")
#=> nil
Proudly sponsored by
The gem is available as open source under the terms of the MIT License.
FAQs
Unknown package
We found that universalid demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.
Security News
The Ultralytics' PyPI Package was compromised four times in one weekend through GitHub Actions cache poisoning and failure to rotate previously compromised API tokens.