@dramaorg/animi-quas-maiores
Advanced tools
> A CRDT framework with a powerful abstraction of shared data
A CRDT framework with a powerful abstraction of shared data
Yjs is a CRDT implementation that exposes its internal
data structure as shared types. Shared types are common data types like Map
or Array with superpowers: changes are automatically distributed to other
peers and merged without merge conflicts.
Yjs is network agnostic (p2p!), supports many existing rich text editors, offline editing, version snapshots, undo/redo and shared cursors. It scales well with an unlimited number of users and is well suited for even large documents.
:construction_worker_woman: If you are looking for professional support, please consider supporting this project via a "support contract" on GitHub Sponsors. I will attend your issues quicker and we can discuss questions and problems in regular video conferences. Otherwise you can find help on our community discussion board.
Please contribute to the project financially - especially if your company relies
on Yjs.
This repository contains a collection of shared types that can be observed for changes and manipulated concurrently. Network functionality and two-way-bindings are implemented in separate modules.
| Name | Cursors | Binding | Demo |
|---|---|---|---|
| ProseMirror | ✔ | y-prosemirror | demo |
| Quill | ✔ | y-quill | demo |
| CodeMirror | ✔ | y-codemirror | demo |
| Monaco | ✔ | y-monaco | demo |
| Slate | ✔ | slate-yjs | demo |
| BlockSuite | ✔ | (native) | demo |
| valtio | valtio-yjs | demo | |
| immer | immer-yjs | demo | |
| React / Vue / Svelte / MobX | SyncedStore | demo | |
| mobx-keystone | mobx-keystone-yjs | demo |
Setting up the communication between clients, managing awareness information, and storing shared data for offline usage is quite a hassle. Providers manage all that for you and are the perfect starting point for your collaborative app.
This list of providers is incomplete. Please open PRs to add your providers to this list!
There are several Yjs-compatible ports to other programming languages.
Install Yjs and a provider with your favorite package manager:
npm i yjs y-websocket
Start the y-websocket server:
PORT=1234 node ./node_modules/y-websocket/bin/server.js
import * as Y from 'yjs';
const doc = new Y.Doc();
const yarray = doc.getArray('my-array')
yarray.observe(event => {
console.log('yarray was modified')
})
// every time a local or remote client modifies yarray, the observer is called
yarray.insert(0, ['val']) // => "yarray was modified"
Remember, shared types are just plain old data types. The only limitation is that a shared type must exist only once in the shared document.
const ymap = doc.getMap('map')
const foodArray = new Y.Array()
foodArray.insert(0, ['apple', 'banana'])
ymap.set('food', foodArray)
ymap.get('food') === foodArray // => true
ymap.set('fruit', foodArray) // => Error! foodArray is already defined
Now you understand how types are defined on a shared document. Next you can jump to the demo repository or continue reading the API docs.
Any of the Yjs providers can be combined with each other. So you can sync data over different network technologies.
In most cases you want to use a network provider (like y-websocket or y-webrtc) in combination with a persistence provider (y-indexeddb in the browser). Persistence allows you to load the document faster and to persist data that is created while offline.
For the sake of this demo we combine two different network providers with a persistence provider.
import * as Y from 'yjs'
import { WebrtcProvider } from 'y-webrtc'
import { WebsocketProvider } from 'y-websocket'
import { IndexeddbPersistence } from 'y-indexeddb'
const ydoc = new Y.Doc()
// this allows you to instantly get the (cached) documents data
const indexeddbProvider = new IndexeddbPersistence('count-demo', ydoc)
indexeddbProvider.whenSynced.then(() => {
console.log('loaded data from indexed db')
})
// Sync clients with the y-webrtc provider.
const webrtcProvider = new WebrtcProvider('count-demo', ydoc)
// Sync clients with the y-websocket provider
const websocketProvider = new WebsocketProvider(
'wss://demos.yjs.dev', 'count-demo', ydoc
)
// array of numbers which produce a sum
const yarray = ydoc.getArray('count')
// observe changes of the sum
yarray.observe(event => {
// print updates when the data changes
console.log('new sum: ' + yarray.toArray().reduce((a,b) => a + b))
})
// add 1 to the sum
yarray.push([1]) // => "new sum: 1"
import * as Y from 'yjs'
A shareable Array-like type that supports efficient insert/delete of elements at any position. Internally it uses a linked list of Arrays that is split when necessary.
const yarray = new Y.Array()
parent:Y.AbstractType|null
insert(index:number, content:Array<object|boolean|Array|string|number|null|Uint8Array|Y.Type>)
array.insert(0, [1]) splices the list and inserts 1 at
position 0.
push(Array<Object|boolean|Array|string|number|null|Uint8Array|Y.Type>)
unshift(Array<Object|boolean|Array|string|number|null|Uint8Array|Y.Type>)
delete(index:number, length:number)
get(index:number)
slice(start:number, end:number):Array<Object|boolean|Array|string|number|null|Uint8Array|Y.Type>
length:number
forEach(function(value:object|boolean|Array|string|number|null|Uint8Array|Y.Type,
index:number, array: Y.Array))
map(function(T, number, YArray):M):Array<M>
toArray():Array<object|boolean|Array|string|number|null|Uint8Array|Y.Type>
toJSON():Array<Object|boolean|Array|string|number|null>
toJSON method.
[Symbol.Iterator]
for (let value of yarray) { .. }
observe(function(YArrayEvent, Transaction):void)
unobserve(function(YArrayEvent, Transaction):void)
observe event listener from this type.
observeDeep(function(Array<YEvent>, Transaction):void)
unobserveDeep(function(Array<YEvent>, Transaction):void)
observeDeep event listener from this type.
A shareable Map type.
const ymap = new Y.Map()parent:Y.AbstractType|null
size: number
get(key:string):object|boolean|string|number|null|Uint8Array|Y.Type
set(key:string, value:object|boolean|string|number|null|Uint8Array|Y.Type)
delete(key:string)
has(key:string):boolean
get(index:number)
clear()
clone():Y.Map
toJSON():Object<string, Object|boolean|Array|string|number|null|Uint8Array>
[key,value] pairs of this YMap to a new Object.It
transforms all child types to JSON using their toJSON method.
forEach(function(value:object|boolean|Array|string|number|null|Uint8Array|Y.Type,
key:string, map: Y.Map))
[Symbol.Iterator]
[key, value] pairs.
for (let [key, value] of ymap) { .. }
entries()
[key, value] pairs.
values()
keys()
observe(function(YMapEvent, Transaction):void)
unobserve(function(YMapEvent, Transaction):void)
observe event listener from this type.
observeDeep(function(Array<YEvent>, Transaction):void)
unobserveDeep(function(Array<YEvent>, Transaction):void)
observeDeep event listener from this type.
A shareable type that is optimized for shared editing on text. It allows to assign properties to ranges in the text. This makes it possible to implement rich-text bindings to this type.
This type can also be transformed to the delta format. Similarly the YTextEvents compute changes as deltas.
const ytext = new Y.Text()
parent:Y.AbstractType|null
insert(index:number, content:string, [formattingAttributes:Object<string,string>])
ytext.insert(0, 'bold text', { bold: true })
delete(index:number, length:number)
format(index:number, length:number, formattingAttributes:Object<string,string>)
applyDelta(delta: Delta, opts:Object<string,any>)
ytext.applyDelta(delta, { sanitize: false })
length:number
toString():string
toJSON():string
toStringtoDelta():Delta
observe(function(YTextEvent, Transaction):void)
unobserve(function(YTextEvent, Transaction):void)
observe event listener from this type.
observeDeep(function(Array<YEvent>, Transaction):void)
unobserveDeep(function(Array<YEvent>, Transaction):void)
observeDeep event listener from this type.
A container that holds an Array of Y.XmlElements.
const yxml = new Y.XmlFragment()parent:Y.AbstractType|null
firstChild:Y.XmlElement|Y.XmlText|null
insert(index:number, content:Array<Y.XmlElement|Y.XmlText>)
delete(index:number, length:number)
get(index:number)
slice(start:number, end:number):Array<Y.XmlElement|Y.XmlText>
length:number
clone():Y.XmlFragment
toArray():Array<Y.XmlElement|Y.XmlText>
toDOM():DocumentFragment
toString():string
toJSON():string
toString.createTreeWalker(filter: function(AbstractType<any>):boolean):Iterable
observe(function(YXmlEvent, Transaction):void)
unobserve(function(YXmlEvent, Transaction):void)
observe event listener from this type.
observeDeep(function(Array<YEvent>, Transaction):void)
unobserveDeep(function(Array<YEvent>, Transaction):void)
observeDeep event listener from this type.
A shareable type that represents an XML Element. It has a nodeName,
attributes, and a list of children. But it makes no effort to validate its
content and be actually XML compliant.
const yxml = new Y.XmlElement()parent:Y.AbstractType|null
firstChild:Y.XmlElement|Y.XmlText|null
nextSibling:Y.XmlElement|Y.XmlText|null
prevSibling:Y.XmlElement|Y.XmlText|null
insert(index:number, content:Array<Y.XmlElement|Y.XmlText>)
delete(index:number, length:number)
get(index:number)
length:number
setAttribute(attributeName:string, attributeValue:string)
removeAttribute(attributeName:string)
getAttribute(attributeName:string):string
getAttributes():Object<string,string>
get(i:number):Y.XmlElement|Y.XmlText
slice(start:number, end:number):Array<Y.XmlElement|Y.XmlText>
clone():Y.XmlElement
toArray():Array<Y.XmlElement|Y.XmlText>
toDOM():Element
toString():string
toJSON():string
toString.observe(function(YXmlEvent, Transaction):void)
unobserve(function(YXmlEvent, Transaction):void)
observe event listener from this type.
observeDeep(function(Array<YEvent>, Transaction):void)
unobserveDeep(function(Array<YEvent>, Transaction):void)
observeDeep event listener from this type.
const doc = new Y.Doc()
clientID
gc
transact(function(Transaction):void [, origin:any])
update event are called after each transaction. You should
bundle changes into a single transaction to reduce the amount of event
calls. I.e. doc.transact(() => { yarray.insert(..); ymap.set(..) })
triggers a single change event. origin
parameter that is stored on transaction.origin and
on('update', (update, origin) => ..).
toJSON():any
ydoc.getType(..)).
get(string, Y.[TypeClass]):[Type]
getArray(string):Y.Array
y.get(string, Y.Array).getMap(string):Y.Map
y.get(string, Y.Map).getText(string):Y.Text
y.get(string, Y.Text).getXmlElement(string, string):Y.XmlElement
y.get(string, Y.XmlElement).getXmlFragment(string):Y.XmlFragment
y.get(string, Y.XmlFragment).on(string, function)
off(string, function)
on('update', function(updateMessage:Uint8Array, origin:any, Y.Doc):void)
on('beforeTransaction', function(Y.Transaction, Y.Doc):void)
on('afterTransaction', function(Y.Transaction, Y.Doc):void)
on('beforeAllTransactions', function(Y.Doc):void)
on('afterAllTransactions', function(Y.Doc, Array<Y.Transaction>):void)
Changes on the shared document are encoded into document updates. Document updates are commutative and idempotent. This means that they can be applied in any order and multiple times.
const doc1 = new Y.Doc()
const doc2 = new Y.Doc()
doc1.on('update', update => {
Y.applyUpdate(doc2, update)
})
doc2.on('update', update => {
Y.applyUpdate(doc1, update)
})
// All changes are also applied to the other document
doc1.getArray('myarray').insert(0, ['Hello doc2, you got this?'])
doc2.getArray('myarray').get(0) // => 'Hello doc2, you got this?'
Yjs internally maintains a state vector that denotes the next expected clock from each client. In a different interpretation it holds the number of structs created by each client. When two clients sync, you can either exchange the complete document structure or only the differences by sending the state vector to compute the differences.
const state1 = Y.encodeStateAsUpdate(ydoc1)
const state2 = Y.encodeStateAsUpdate(ydoc2)
Y.applyUpdate(ydoc1, state2)
Y.applyUpdate(ydoc2, state1)
This example shows how to sync two clients with the minimal amount of exchanged data by computing only the differences using the state vector of the remote client. Syncing clients using the state vector requires another roundtrip, but can save a lot of bandwidth.
const stateVector1 = Y.encodeStateVector(ydoc1)
const stateVector2 = Y.encodeStateVector(ydoc2)
const diff1 = Y.encodeStateAsUpdate(ydoc1, stateVector2)
const diff2 = Y.encodeStateAsUpdate(ydoc2, stateVector1)
Y.applyUpdate(ydoc1, diff2)
Y.applyUpdate(ydoc2, diff1)
It is possible to sync clients and compute delta updates without loading the Yjs document to memory. Yjs exposes an API to compute the differences directly on the binary document updates.
// encode the current state as a binary buffer
let currentState1 = Y.encodeStateAsUpdate(ydoc1)
let currentState2 = Y.encodeStateAsUpdate(ydoc2)
// now we can continue syncing clients using state vectors without using the Y.Doc
ydoc1.destroy()
ydoc2.destroy()
const stateVector1 = Y.encodeStateVectorFromUpdate(currentState1)
const stateVector2 = Y.encodeStateVectorFromUpdate(currentState2)
const diff1 = Y.diffUpdate(currentState1, stateVector2)
const diff2 = Y.diffUpdate(currentState2, stateVector1)
// sync clients
currentState1 = Y.mergeUpdates([currentState1, diff2])
currentState2 = Y.mergeUpdates([currentState2, diff1])
If one of your users runs into a weird bug (e.g. the rich-text editor throws error messages), then you don't have to request the full document from your user. Instead, they can obfuscate the document (i.e. replace the content with meaningless generated content) before sending it to you. Note that someone might still deduce the type of content by looking at the general structure of the document. But this is much better than requesting the original document.
Obfuscated updates contain all the CRDT-related data that is required for merging. So it is safe to merge obfuscated updates.
const ydoc = new Y.Doc()
// perform some changes..
ydoc.getText().insert(0, 'hello world')
const update = Y.encodeStateAsUpdate(ydoc)
// the below update contains scrambled data
const obfuscatedUpdate = Y.obfuscateUpdate(update)
const ydoc2 = new Y.Doc()
Y.applyUpdate(ydoc2, obfuscatedUpdate)
ydoc2.getText().toString() // => "00000000000"
Yjs implements two update formats. By default you are using the V1 update format.
You can opt-in into the V2 update format which provides much better compression.
It is not yet used by all providers. However, you can already use it if
you are building your own provider. All below functions are available with the
suffix "V2". E.g. Y.applyUpdate ⇒ Y.applyUpdateV2. Also when listening to updates
you need to specifically need listen for V2 events e.g. yDoc.on('updateV2', …).
We also support conversion functions between both formats:
Y.convertUpdateFormatV1ToV2 & Y.convertUpdateFormatV2ToV1.
Y.applyUpdate(Y.Doc, update:Uint8Array, [transactionOrigin:any])
transactionOrigin that will be stored on
transaction.origin
and ydoc.on('update', (update, origin) => ..).
Y.encodeStateAsUpdate(Y.Doc, [encodedTargetStateVector:Uint8Array]):Uint8Array
Y.encodeStateVector(Y.Doc):Uint8Array
Y.mergeUpdates(Array<Uint8Array>)
Y.encodeStateVectorFromUpdate(Uint8Array): Uint8Array
Y.diffUpdate(update: Uint8Array, stateVector: Uint8Array): Uint8Array
Y.encodeStateAsUpdate(ydoc, stateVector) but works
on updates instead.
convertUpdateFormatV1ToV2
convertUpdateFormatV2ToV1
When working with collaborative documents, we often need to work with positions. Positions may represent cursor locations, selection ranges, or even assign a comment to a range of text. Normal index-positions (expressed as integers) are not convenient to use because the index-range is invalidated as soon as a remote change manipulates the document. Relative positions give you a powerful API to express positions.
A relative position is fixated to an element in the shared document and is not
affected by remote changes. I.e. given the document "a|c", the relative
position is attached to c. When a remote user modifies the document by
inserting a character before the cursor, the cursor will stay attached to the
character c. insert(1, 'x')("a|c") = "ax|c". When the relative position is
set to the end of the document, it will stay attached to the end of the
document.
const relPos = Y.createRelativePositionFromTypeIndex(ytext, 2)
const pos = Y.createAbsolutePositionFromRelativePosition(relPos, doc)
pos.type === ytext // => true
pos.index === 2 // => true
const relPos = Y.createRelativePositionFromTypeIndex(ytext, 2)
const encodedRelPos = JSON.stringify(relPos)
// send encodedRelPos to remote client..
const parsedRelPos = JSON.parse(encodedRelPos)
const pos = Y.createAbsolutePositionFromRelativePosition(parsedRelPos, remoteDoc)
pos.type === remoteytext // => true
pos.index === 2 // => true
const relPos = Y.createRelativePositionFromTypeIndex(ytext, 2)
const encodedRelPos = Y.encodeRelativePosition(relPos)
// send encodedRelPos to remote client..
const parsedRelPos = Y.decodeRelativePosition(encodedRelPos)
const pos = Y.createAbsolutePositionFromRelativePosition(parsedRelPos, remoteDoc)
pos.type === remoteytext // => true
pos.index === 2 // => true
Y.createRelativePositionFromTypeIndex(type:Uint8Array|Y.Type, index: number
[, assoc=0])
assoc >= 0). By default, the position associates
with the character that comes after the specified index position. If
assoc < 0, then the relative position associates with the character
before the specified index position.
Y.createAbsolutePositionFromRelativePosition(RelativePosition, Y.Doc):
{ type: Y.AbstractType, index: number, assoc: number } | null
Y.encodeRelativePosition(RelativePosition):Uint8Array
Y.decodeRelativePosition(Uint8Array):RelativePosition
Yjs ships with an Undo/Redo manager for selective undo/redo of changes on a Yjs type. The changes can be optionally scoped to transaction origins.
const ytext = doc.getText('text')
const undoManager = new Y.UndoManager(ytext)
ytext.insert(0, 'abc')
undoManager.undo()
ytext.toString() // => ''
undoManager.redo()
ytext.toString() // => 'abc'
constructor(scope:Y.AbstractType|Array<Y.AbstractType>
[, {captureTimeout:number,trackedOrigins:Set<any>,deleteFilter:function(item):boolean}])
undo()
redo()
stopCapturing()
on('stack-item-added', { stackItem: { meta: Map<any,any> }, type: 'undo'
| 'redo' })
StackItem is added to the
undo- or the redo-stack.
on('stack-item-updated', { stackItem: { meta: Map<any,any> }, type: 'undo'
| 'redo' })
StackItem is updated.
This happens when two changes happen within a "captureInterval".
on('stack-item-popped', { stackItem: { meta: Map<any,any> }, type: 'undo'
| 'redo' })
StackItem is popped from
the undo- or the redo-stack.
on('stack-cleared', { undoStackCleared: boolean, redoStackCleared: boolean })
UndoManager merges Undo-StackItems if they are created within time-gap
smaller than options.captureTimeout. Call um.stopCapturing() so that the next
StackItem won't be merged.
// without stopCapturing
ytext.insert(0, 'a')
ytext.insert(1, 'b')
undoManager.undo()
ytext.toString() // => '' (note that 'ab' was removed)
// with stopCapturing
ytext.insert(0, 'a')
undoManager.stopCapturing()
ytext.insert(0, 'b')
undoManager.undo()
ytext.toString() // => 'a' (note that only 'b' was removed)
Every change on the shared document has an origin. If no origin was specified,
it defaults to null. By specifying trackedOrigins you can
selectively specify which changes should be tracked by UndoManager. The
UndoManager instance is always added to trackedOrigins.
class CustomBinding {}
const ytext = doc.getText('text')
const undoManager = new Y.UndoManager(ytext, {
trackedOrigins: new Set([42, CustomBinding])
})
ytext.insert(0, 'abc')
undoManager.undo()
ytext.toString() // => 'abc' (does not track because origin `null` and not part
// of `trackedTransactionOrigins`)
ytext.delete(0, 3) // revert change
doc.transact(() => {
ytext.insert(0, 'abc')
}, 42)
undoManager.undo()
ytext.toString() // => '' (tracked because origin is an instance of `trackedTransactionorigins`)
doc.transact(() => {
ytext.insert(0, 'abc')
}, 41)
undoManager.undo()
ytext.toString() // => 'abc' (not tracked because 41 is not an instance of
// `trackedTransactionorigins`)
ytext.delete(0, 3) // revert change
doc.transact(() => {
ytext.insert(0, 'abc')
}, new CustomBinding())
undoManager.undo()
ytext.toString() // => '' (tracked because origin is a `CustomBinding` and
// `CustomBinding` is in `trackedTransactionorigins`)
When undoing or redoing a previous action, it is often expected to restore additional meta information like the cursor location or the view on the document. You can assign meta-information to Undo-/Redo-StackItems.
const ytext = doc.getText('text')
const undoManager = new Y.UndoManager(ytext, {
trackedOrigins: new Set([42, CustomBinding])
})
undoManager.on('stack-item-added', event => {
// save the current cursor location on the stack-item
event.stackItem.meta.set('cursor-location', getRelativeCursorLocation())
})
undoManager.on('stack-item-popped', event => {
// restore the current cursor location on the stack-item
restoreCursorLocation(event.stackItem.meta.get('cursor-location'))
})
Conflict-free replicated data types (CRDT) for collaborative editing are an alternative approach to operational transformation (OT). A very simple differentiation between the two approaches is that OT attempts to transform index positions to ensure convergence (all clients end up with the same content), while CRDTs use mathematical models that usually do not involve index transformations, like linked lists. OT is currently the de-facto standard for shared editing on text. OT approaches that support shared editing without a central source of truth (a central server) require too much bookkeeping to be viable in practice. CRDTs are better suited for distributed systems, provide additional guarantees that the document can be synced with remote clients, and do not require a central source of truth.
Yjs implements a modified version of the algorithm described in this paper. This article explains a simple optimization on the CRDT model and gives more insight about the performance characteristics in Yjs. More information about the specific implementation is available in INTERNALS.md and in this walkthrough of the Yjs codebase.
CRDTs that are suitable for shared text editing suffer from the fact that they only grow in size. There are CRDTs that do not grow in size, but they do not have the characteristics that are benificial for shared text editing (like intention preservation). Yjs implements many improvements to the original algorithm that diminish the trade-off that the document only grows in size. We can't garbage collect deleted structs (tombstones) while ensuring a unique order of the structs. But we can 1. merge preceeding structs into a single struct to reduce the amount of meta information, 2. we can delete content from the struct if it is deleted, and 3. we can garbage collect tombstones if we don't care about the order of the structs anymore (e.g. if the parent was deleted).
Examples:
text.insert(0, 'a'), text.insert(1, 'b'); is
first represented as two structs ([{id: {client, clock: 0}, content: 'a'}, {id: {client, clock: 1}, content: 'b'}) and then merged into a single
struct: [{id: {client, clock: 0}, content: 'ab'}].ItemString) is deleted, the
struct will be replaced with an ItemDeleted that does not contain content
anymore.GC structs. A
GC struct only denotes the existence of a struct and that it is deleted.
GC structs can always be merged with other GC structs if the id's are
adjacent.Especially when working on structured content (e.g. shared editing on ProseMirror), these improvements yield very good results when benchmarking random document edits. In practice they show even better results, because users usually edit text in sequence, resulting in structs that can easily be merged. The benchmarks show that even in the worst case scenario that a user edits text from right to left, Yjs achieves good performance even for huge documents.
Yjs has the ability to exchange only the differences when syncing two clients.
We use lamport timestamps to identify structs and to track in which order a
client created them. Each struct has an struct.id = { client: number, clock: number} that uniquely identifies a struct. We define the next expected clock
by each client as the state vector. This data structure is similar to the
version vectors data structure.
But we use state vectors only to describe the state of the local document, so we
can compute the missing struct of the remote client. We do not use it to track
causality.
Yjs and all related projects are MIT licensed.
Yjs is based on my research as a student at the RWTH i5. Now I am working on Yjs in my spare time.
Fund this project by donating on GitHub Sponsors or hiring me as a contractor for your collaborative app.
FAQs
> A CRDT framework with a powerful abstraction of shared data
We found that @dramaorg/animi-quas-maiores demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 0 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
Socket’s threat research team has detected six malicious npm packages typosquatting popular libraries to insert SSH backdoors.
Security News
MITRE's 2024 CWE Top 25 highlights critical software vulnerabilities like XSS, SQL Injection, and CSRF, reflecting shifts due to a refined ranking methodology.
Security News
In this segment of the Risky Business podcast, Feross Aboukhadijeh and Patrick Gray discuss the challenges of tracking malware discovered in open source softare.