github.com/lni/dragonboat/v3
Package dragonboat is a multi-group Raft implementation. The NodeHost struct is the facade interface for all features provided by the dragonboat package. Each NodeHost instance usually runs on a separate host managing its CPU, storage and network resources. Each NodeHost can manage Raft nodes from many different Raft groups known as Raft clusters. Each Raft cluster is identified by its ClusterID and it usually consists of multiple nodes, each identified its NodeID value. Nodes from the same Raft cluster can be considered as replicas of the same data, they are suppose to be distributed on different NodeHost instances across the network, this brings fault tolerance to machine and network failures as application data stored in the Raft cluster will be available as long as the majority of its managing NodeHost instances (i.e. its underlying hosts) are available. User applications can leverage the power of the Raft protocol implemented in dragonboat by implementing the IStateMachine or IOnDiskStateMachine component, as defined in github.com/lni/dragonboat/v3/statemachine. Known as user state machines, each IStateMachine and IOnDiskStateMachine instance is in charge of updating, querying and snapshotting application data with minimum exposure to the complexity of the Raft protocol implementation. User applications can use NodeHost's APIs to update the state of their IStateMachine or IOnDiskStateMachine instances, this is called making proposals. Once accepted by the majority nodes of a Raft cluster, the proposal is considered as committed and it will be applied on all member nodes of the Raft cluster. Applications can also make linearizable reads to query the state of the IStateMachine or IOnDiskStateMachine instances. Dragonboat employs the ReadIndex protocol invented by Diego Ongaro for fast linearizable reads. Dragonboat guarantees the linearizability of your I/O when interacting with the IStateMachine or IOnDiskStateMachine instances. In plain English, writes (via making proposal) to your Raft cluster appears to be instantaneous, once a write is completed, all later reads (linearizable read using the ReadIndex protocol as implemented and provided in dragonboat) should return the value of that write or a later write. Once a value is returned by a linearizable read, all later reads should return the same value or the result of a later write. To strictly provide such guarantee, we need to implement the at-most-once semantic required by linearizability. For a client, when it retries the proposal that failed to complete before its deadline during the previous attempt, it has the risk to have the same proposal committed and applied twice into the user state machine. Dragonboat prevents this by implementing the client session concept described in Diego Ongaro's PhD thesis. Arbitrary number of Raft clusters can be launched across the network simultaneously to aggregate distributed processing and storage capacities. Users can also make membership change requests to add or remove nodes from any interested Raft cluster. NodeHost APIs for making the above mentioned requests can be loosely classified into two categories, synchronous and asynchronous APIs. Synchronous APIs will not return until the completion of the requested operation. Their method names all start with Sync*. The asynchronous counterparts of such synchronous APIs, on the other hand, usually return immediately. This allows users to concurrently initiate multiple such asynchronous operations to save the total amount of time required to complete all of them. Dragonboat is a feature complete Multi-Group Raft implementation - snapshotting, membership change, leadership transfer, non-voting members and disk based state machine are all provided. Dragonboat is also extensively optimized. The Raft protocol implementation is fully pipelined, meaning proposals can start before the completion of previous proposals. This is critical for system throughput in high latency environment. Dragonboat is also fully batched, internal operations are batched whenever possible to maximize the overall throughput.
Readme
Dragonboat is a high performance multi-group Raft consensus library in pure Go.
Consensus algorithms such as Raft provides fault-tolerance by alllowing a system continue to operate as long as the majority member servers are available. For example, a Raft cluster of 5 servers can make progress even if 2 servers fail. It also appears to clients as a single entity with strong data consistency always provided. All Raft replicas can be used to handle read requests for aggregated read throughput.
Dragonboat handles all technical difficulties associated with Raft to allow users to just focus on their application domains. It is also very easy to use, our step-by-step examples can help new users to master it in half an hour.
Most features covered in Diego Ongaro's Raft thesis have been supported -
Dragonboat is the fastest open source multi-group Raft implementation on Github.
For 3-nodes system using mid-range hardware (details here) and in-memory state machine, when RocksDB is used as the storage engine, Dragonboat can sustain at 9 million writes per second when the payload is 16bytes each or 11 million mixed I/O per second at 9:1 read:write ratio. High throughput is maintained in geographically distributed environment. When the RTT between nodes is 30ms, 2 million I/O per second can still be achieved using a much larger number of clients.
The number of concurrent active Raft groups affects the overall throughput as requests become harder to be batched. On the other hand, having thousands of idle Raft groups has a much smaller impact on throughput.
Table below shows write latencies in millisecond, Dragonboat has <5ms P99 write latency when handling 8 million writes per second at 16 bytes each. Read latency is lower than writes as the ReadIndex protocol employed for linearizable reads doesn't require fsync-ed disk I/O.
| Ops | Payload Size | 99.9% percentile | 99% percentile | AVG |
|---|---|---|---|---|
| 1m | 16 | 2.24 | 1.19 | 0.79 |
| 1m | 128 | 11.11 | 1.37 | 0.92 |
| 1m | 1024 | 71.61 | 25.91 | 3.75 |
| 5m | 16 | 4.64 | 1.95 | 1.16 |
| 5m | 128 | 36.61 | 6.55 | 1.96 |
| 8m | 16 | 12.01 | 4.65 | 2.13 |
When tested on a single Raft group, Dragonboat can sustain writes at 1.25 million per second when payload is 16 bytes each, average latency is 1.3ms and the P99 latency is 2.6ms. This is achieved when using an average of 3 cores (2.8GHz) on each server.
As visualized below, Stop-the-World pauses caused by Go1.11's GC are sub-millisecond on highly loaded systems. Such very short Stop-the-World pause time is further significantly reduced in Go 1.12. Golang's runtime.ReadMemStats reports that less than 1% of the available CPU time is used by GC on highly loaded system.
Master is our unstable branch for development. Please use the latest released versions for any production purposes. For Dragonboat v3.3.x, please follow the instructions in v3.3.x's README.md.
Go 1.14 or above with Go module support is required.
To use Dragonboat, make sure to import the package github.com/lni/dragonboat/v3. Also add "github.com/lni/dragonboat/v3 v3.3.0" to the require section of your project's go.mod file.
By default, Pebble is used for storing Raft Logs in Dragonboat. RocksDB and other storage engines are also supported, more info here.
You can also follow our examples on how to use Dragonboat.
FAQ, docs, step-by-step examples, DevOps doc, CHANGELOG and online chat are available.
Dragonboat examples are here.
Dragonboat is production ready.
For reporting bugs, please open an issue. For contributing improvements or new features, please send in the pull request.
Dragonboat is licensed under the Apache License Version 2.0. See LICENSE for details.
Third party code used in Dragonboat and their licenses is summarized here.
FAQs
Package dragonboat is a multi-group Raft implementation. The NodeHost struct is the facade interface for all features provided by the dragonboat package. Each NodeHost instance usually runs on a separate host managing its CPU, storage and network resources. Each NodeHost can manage Raft nodes from many different Raft groups known as Raft clusters. Each Raft cluster is identified by its ClusterID and it usually consists of multiple nodes, each identified its NodeID value. Nodes from the same Raft cluster can be considered as replicas of the same data, they are suppose to be distributed on different NodeHost instances across the network, this brings fault tolerance to machine and network failures as application data stored in the Raft cluster will be available as long as the majority of its managing NodeHost instances (i.e. its underlying hosts) are available. User applications can leverage the power of the Raft protocol implemented in dragonboat by implementing the IStateMachine or IOnDiskStateMachine component, as defined in github.com/lni/dragonboat/v3/statemachine. Known as user state machines, each IStateMachine and IOnDiskStateMachine instance is in charge of updating, querying and snapshotting application data with minimum exposure to the complexity of the Raft protocol implementation. User applications can use NodeHost's APIs to update the state of their IStateMachine or IOnDiskStateMachine instances, this is called making proposals. Once accepted by the majority nodes of a Raft cluster, the proposal is considered as committed and it will be applied on all member nodes of the Raft cluster. Applications can also make linearizable reads to query the state of the IStateMachine or IOnDiskStateMachine instances. Dragonboat employs the ReadIndex protocol invented by Diego Ongaro for fast linearizable reads. Dragonboat guarantees the linearizability of your I/O when interacting with the IStateMachine or IOnDiskStateMachine instances. In plain English, writes (via making proposal) to your Raft cluster appears to be instantaneous, once a write is completed, all later reads (linearizable read using the ReadIndex protocol as implemented and provided in dragonboat) should return the value of that write or a later write. Once a value is returned by a linearizable read, all later reads should return the same value or the result of a later write. To strictly provide such guarantee, we need to implement the at-most-once semantic required by linearizability. For a client, when it retries the proposal that failed to complete before its deadline during the previous attempt, it has the risk to have the same proposal committed and applied twice into the user state machine. Dragonboat prevents this by implementing the client session concept described in Diego Ongaro's PhD thesis. Arbitrary number of Raft clusters can be launched across the network simultaneously to aggregate distributed processing and storage capacities. Users can also make membership change requests to add or remove nodes from any interested Raft cluster. NodeHost APIs for making the above mentioned requests can be loosely classified into two categories, synchronous and asynchronous APIs. Synchronous APIs will not return until the completion of the requested operation. Their method names all start with Sync*. The asynchronous counterparts of such synchronous APIs, on the other hand, usually return immediately. This allows users to concurrently initiate multiple such asynchronous operations to save the total amount of time required to complete all of them. Dragonboat is a feature complete Multi-Group Raft implementation - snapshotting, membership change, leadership transfer, non-voting members and disk based state machine are all provided. Dragonboat is also extensively optimized. The Raft protocol implementation is fully pipelined, meaning proposals can start before the completion of previous proposals. This is critical for system throughput in high latency environment. Dragonboat is also fully batched, internal operations are batched whenever possible to maximize the overall throughput.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Cyber extortion in the US and Canada hit record levels in 2023, with ransomware attacks surging and median ransom demands skyrocketing, though fewer companies are choosing to pay ransoms.
Security News
Ecma TC39 is meeting this week and has moved key ECMAScript proposals forward, advancing Deferred Import Evaluation, Error.isError(), RegExp Escaping, and Promise.try to the next stages.
Security News
Researchers have demonstrated that teams of LLM agents can exploit zero-day vulnerabilities with a 53% success rate, and the costs of using AI to do so are rapidly becoming more affordable than hiring a human penetration tester.
