Package kademlia implements a configurable Kademlia Distributed Hash Table.
Package dht implements a distributed hash table that satisfies the ipfs routing interface.
Package dht implements a Distributed Hash Table (DHT) part of the BitTorrent protocol, as specified by BEP 5: http://www.bittorrent.org/beps/bep_0005.html
Package dht implements a distributed hash table that satisfies the ipfs routing interface.
Offheap An off-heap hash-table for Go (golang).
Package dht implements a distributed hash table that satisfies the ipfs routing interface.
Package dht implements a Distributed Hash Table (DHT) part of the BitTorrent protocol, as specified by BEP 5: http://www.bittorrent.org/beps/bep_0005.html BitTorrent uses a "distributed hash table" (DHT) for storing peer contact information for "trackerless" torrents.
Package dht implements a distributed hash table that satisfies the ipfs routing interface.
Package cuckoo implements d-ary bucketized cuckoo hashing with stash (bucketized cuckoo hashing is also known as splash tables).
Package merkletree is an implementation of a Merkle tree (https://en.wikipedia.org/wiki/Merkle_tree). It provides methods to create a tree and generate and verify proofs. The hashing algorithm for the tree is selectable between BLAKE2b and Keccak256, or you can supply your own. This implementation includes advanced features salting and pollarding. Salting is the act of adding a piece of data to each value in the Merkle tree as it is initially hashed to form the leaves, which helps avoid rainbow table attacks on leaf hashes presented as part of proofs. Pollarding is the act of providing the root plus all branches to a certain height which can be used to reduce the size of proofs. This is useful when multiple proofs are presented against the same tree as it can reduce the overall size. Creating a Merkle tree requires a list of values that are each byte arrays. Once a tree has been created proofs can be generated using the tree's GenerateProof() function. The package includes a function VerifyProof() to verify a generated proof given only the data to prove, proof and the pollard of the relevant Merkle tree. This allows for efficient verification of proofs without requiring the entire Merkle tree to be stored or recreated. The tree pads its values to the next highest power of 2; values not supplied are treated as null with a value hash of 0. This can be seen graphically by generating a DOT representation of the graph with DOT(). If salting is enabled it appends an 4-byte value to each piece of data. The value is the binary representation of the index in big-endian form. Note that if there are more than 2^32 values in the tree the salt will wrap, being modulo 2^32 Package merkletree is an implementation of a Merkle tree (https://en.wikipedia.org/wiki/Merkle_tree). It provides methods to create a tree and generate and verify proofs. The hashing algorithm for the tree is selectable between BLAKE2b and Keccak256, or you can supply your own. This implementation includes advanced features salting and pollarding. Salting is the act of adding a piece of data to each value in the Merkle tree as it is initially hashed to form the leaves, which helps avoid rainbow table attacks on leaf hashes presented as part of proofs. Pollarding is the act of providing the root plus all branches to a certain height which can be used to reduce the size of proofs. This is useful when multiple proofs are presented against the same tree as it can reduce the overall size. Creating a Merkle tree requires a list of values that are each byte arrays. Once a tree has been created proofs can be generated using the tree's GenerateProof() function. The package includes a function VerifyProof() to verify a generated proof given only the data to prove, proof and the pollard of the relevant Merkle tree. This allows for efficient verification of proofs without requiring the entire Merkle tree to be stored or recreated. The tree pads its values to the next highest power of 2; values not supplied are treated as null with a value hash of 0. This can be seen graphically by generating a DOT representation of the graph with DOT(). If salting is enabled it appends an 4-byte value to each piece of data. The value is the binary representation of the index in big-endian form. Note that if there are more than 2^32 values in the tree the salt will wrap, being modulo 2^32
Package dht implements a distributed hash table that satisfies the ipfs routing interface.
Package dht implements a distributed hash table that satisfies the ipfs routing interface.
Package dht implements a distributed hash table that satisfies the ipfs routing interface.
Package hamt is the front-end for a 32bit and 64bit implementations of a functional Hash Array Mapped Trie (HAMT) datastructure. In this case, functional means immutable and persistent. The term "immutable" means that the datastructure is is never changed after construction. Where "persistent" means that when that immutable datastructure is modified, is is based on the previous datastructure. In other words the new datastructure is not a copy with the modification applied. In stead, that new datastructure shares all un-modified parts of the previouse datastructure and only the changed parts are copied and modified; unchanged parts of the datastructure are shared between the old and new version. A HAMT structure is a tree with a fixed & wide branching factor. Trees make and excellent datastructure to be immutable and persistent. Our HAMT starts with a root branch. Branches are called tables, because they are represented as tables with the "branching factor" number of entries. These entries may be one of three types of nodes: further branches (aka tables) or key/val entries (aka leafs) or emtpy (aka nil). A HAMT is a key/val indexing datastructure. Rather than indexing the datastructure on the key directly, which could result in a rather deep tree datastructure. We generate a hash value of the key, and split the hash value into a fixed number of indexes into fixed size arrays. This results in a tree with a maximum depth and a wide branching factor. For example, we can use a key type of a string. Hash that string into a 32 bit hash value. Coerce that 32 bit value into a 30 bit value. Then split that 30 bit hash value into six 5 bit values. Those 5 bit values will index perfectly into tree nodes with 32 wide branching factor. Now we have tree with a string for the key that is AT MOST six levels deep, in other words O(1) lookup and modification operations. Lets call the number of hash bits H (for hash value). The number of parts the hash value can be split into we'll call D (for depth). The width of each table (aka branching factor) is 2^B; I think of B as "bits per level". The relationship of H:D:B is given by H/B = D. I've implemented in hamt32 (H=30, D=6, B=5) and hamt64 is (H=60, D=10, B=6). We could call the branching factor W for "width" of each tree node. However W is superfluous, because it can be derived from B (aka W=2^B). The number in hamt32 is the branching factor W=2^B=32; from H=30,D=6,B=5 . The number in hamt64 is the branching factor W=2^B=64; from H=60,D=10,B=6 . HAMTs are [Tries](https://en.wikipedia.org/wiki/Trie), because when we are trying to find a location to Get, Put, or Delete a key/value pair we mearly have to walk the "hash path" till we find a non-branching node. The HashPath is the H bit hash value, split into a ordered sequence of B bit integer values that is, at most, D entry tries long. Lets start with a concrete example of a hamt32 (aka H=30,D=6,B=5). Given the string "ewyx" the Hash30() HashVal30 is 0x11a01c5e. Converted into six descreet 5 bit values (from lowest bit to highest) you get 30, 2, 7, 0 26, and 8. This library prints them out from HashVal30.String() as "/30/02/07/00/26/08"; The hash path from lowest to highest bit. That string, "/30/02/07/00/26/08", is the "hash path". Looking up where to find this entry we look up the 30th index of the root of the tree, if that entry is another branch we look up the 2nd index of that next branch. We continue (7th, 0th et al) until we find a non-branch entry. The non-branch entry can be a leaf or empty. Just to be pedantic the go-hamt-key API calculates the indexes by depth as follows: Now we know how to find the candidate location or entry for our operation. That operation can be either a straight lookup, called with h.Get(k); or it can be an insertion of a key/value pair, called with h.Put(k,v); or lastly it can be a deletion operation, called with h.Del(k). For either hamt32.Hamt or hamt64.Hamt value we have three primary operations: h.Get(), h.Put(), and h.Del(). Only h.Put() and h.Del() modify the HAMT. When they modify a table, first the table is copied, then the modification is made to the copy. Next the parent table must be copied so that the new table's entry in the copied parent may be modified. This is continued to the root table and the HAMT structure itself is copied. This is the h.persist() call. Hence, h.Put() and h.Del() return the new HAMT structure as well as any other return values specific to h.Put() or h.Del(). Given that Get() makes no modification of the HAMT structure, it only returns a boolean indicating the key was found in the HAMT and the key's value. Put() returns a copy of the HAMT and a boolean indicating whether a new entry was added (true) or a current entry was updated (false). Del() returns a boolean value indicating if the key was found, and if true what the value of the deleted key was, and the new HAMT structure. If the Del() didn't find the key (a false return value) key's value data is nil and the HAMT value is the current HAMT.
Package cuckoo implements d-ary bucketized cuckoo hashing with stash (bucketized cuckoo hashing is also known as splash tables).
Package consistent provides a consistent hashing function. Consistent hashing is often used to distribute requests to a changing set of servers. For example, say you have some cache servers cacheA, cacheB, and cacheC. You want to decide which cache server to use to look up information on a user. You could use a typical hash table and hash the user id to one of cacheA, cacheB, or cacheC. But with a typical hash table, if you add or remove a server, almost all keys will get remapped to different results, which basically could bring your service to a grinding halt while the caches get rebuilt. With a consistent hash, adding or removing a server drastically reduces the number of keys that get remapped. Read more about consistent hashing on wikipedia: http://en.wikipedia.org/wiki/Consistent_hashing
Package dht implements a distributed hash table that satisfies the ipfs routing interface.
Package dht implements a distributed hash table that satisfies the ipfs routing interface.
Package dht implements a distributed hash table that satisfies the ipfs routing interface.
Package tview implements rich widgets for terminal based user interfaces. The widgets provided with this package are useful for data exploration and data entry. The package implements the following widgets: The package also provides Application which is used to poll the event queue and draw widgets on screen. The following is a very basic example showing a box with the title "Hello, world!": First, we create a box primitive with a border and a title. Then we create an application, set the box as its root primitive, and run the event loop. The application exits when the application's Stop() function is called or when Ctrl-C is pressed. If we have a primitive which consumes key presses, we call the application's SetFocus() function to redirect all key presses to that primitive. Most primitives then offer ways to install handlers that allow you to react to any actions performed on them. You will find more demos in the "demos" subdirectory. It also contains a presentation (written using tview) which gives an overview of the different widgets and how they can be used. Throughout this package, colors are specified using the tcell.Color type. Functions such as tcell.GetColor(), tcell.NewHexColor(), and tcell.NewRGBColor() can be used to create colors from W3C color names or RGB values. Almost all strings which are displayed can contain color tags. Color tags are W3C color names or six hexadecimal digits following a hash tag, wrapped in square brackets. Examples: A color tag changes the color of the characters following that color tag. This applies to almost everything from box titles, list text, form item labels, to table cells. In a TextView, this functionality has to be switched on explicitly. See the TextView documentation for more information. Color tags may contain not just the foreground (text) color but also the background color and additional flags. In fact, the full definition of a color tag is as follows: Each of the three fields can be left blank and trailing fields can be omitted. (Empty square brackets "[]", however, are not considered color tags.) Colors that are not specified will be left unchanged. A field with just a dash ("-") means "reset to default". You can specify the following flags (some flags may not be supported by your terminal): Examples: In the rare event that you want to display a string such as "[red]" or "[#00ff1a]" without applying its effect, you need to put an opening square bracket before the closing square bracket. Note that the text inside the brackets will be matched less strictly than region or colors tags. I.e. any character that may be used in color or region tags will be recognized. Examples: You can use the Escape() function to insert brackets automatically where needed. When primitives are instantiated, they are initialized with colors taken from the global Styles variable. You may change this variable to adapt the look and feel of the primitives to your preferred style. This package supports unicode characters including wide characters. Many functions in this package are not thread-safe. For many applications, this may not be an issue: If your code makes changes in response to key events, it will execute in the main goroutine and thus will not cause any race conditions. If you access your primitives from other goroutines, however, you will need to synchronize execution. The easiest way to do this is to call Application.QueueUpdate() or Application.QueueUpdateDraw() (see the function documentation for details): One exception to this is the io.Writer interface implemented by TextView. You can safely write to a TextView from any goroutine. See the TextView documentation for details. You can also call Application.Draw() from any goroutine without having to wrap it in QueueUpdate(). And, as mentioned above, key event callbacks are executed in the main goroutine and thus should not use QueueUpdate() as that may lead to deadlocks. All widgets listed above contain the Box type. All of Box's functions are therefore available for all widgets, too. All widgets also implement the Primitive interface. There is also the Focusable interface which is used to override functions in subclassing types. The tview package is based on https://github.com/gdamore/tcell. It uses types and constants from that package (e.g. colors and keyboard values). This package does not process mouse input (yet).
Package dht implements a distributed hash table that satisfies the ipfs routing interface. This DHT is modeled after kademlia with S/Kademlia modifications. Package dht implements a distributed hash table that satisfies the ipfs routing interface. This DHT is modeled after Kademlia with S/Kademlia modifications. package query implement a query manager to drive concurrent workers to query the DHT. A query is setup with a target key, a queryFunc tasked to communicate with a peer, and a set of initial peers. As the query progress, queryFunc can return closer peers that will be used to navigate closer to the target key in the DHT until an answer is reached.
Package tview implements rich widgets for terminal based user interfaces. The widgets provided with this package are useful for data exploration and data entry. The package implements the following widgets: The package also provides Application which is used to poll the event queue and draw widgets on screen. The following is a very basic example showing a box with the title "Hello, world!": First, we create a box primitive with a border and a title. Then we create an application, set the box as its root primitive, and run the event loop. The application exits when the application's Stop() function is called or when Ctrl-C is pressed. If we have a primitive which consumes key presses, we call the application's SetFocus() function to redirect all key presses to that primitive. Most primitives then offer ways to install handlers that allow you to react to any actions performed on them. You will find more demos in the "demos" subdirectory. It also contains a presentation (written using tview) which gives an overview of the different widgets and how they can be used. Throughout this package, colors are specified using the tcell.Color type. Functions such as tcell.GetColor(), tcell.NewHexColor(), and tcell.NewRGBColor() can be used to create colors from W3C color names or RGB values. Almost all strings which are displayed can contain color tags. Color tags are W3C color names or six hexadecimal digits following a hash tag, wrapped in square brackets. Examples: A color tag changes the color of the characters following that color tag. This applies to almost everything from box titles, list text, form item labels, to table cells. In a TextView, this functionality has to be switched on explicitly. See the TextView documentation for more information. Color tags may contain not just the foreground (text) color but also the background color and additional flags. In fact, the full definition of a color tag is as follows: Each of the three fields can be left blank and trailing fields can be omitted. (Empty square brackets "[]", however, are not considered color tags.) Colors that are not specified will be left unchanged. A field with just a dash ("-") means "reset to default". You can specify the following flags (some flags may not be supported by your terminal): Examples: In the rare event that you want to display a string such as "[red]" or "[#00ff1a]" without applying its effect, you need to put an opening square bracket before the closing square bracket. Note that the text inside the brackets will be matched less strictly than region or colors tags. I.e. any character that may be used in color or region tags will be recognized. Examples: You can use the Escape() function to insert brackets automatically where needed. When primitives are instantiated, they are initialized with colors taken from the global Styles variable. You may change this variable to adapt the look and feel of the primitives to your preferred style. This package supports unicode characters including wide characters. Many functions in this package are not thread-safe. For many applications, this may not be an issue: If your code makes changes in response to key events, it will execute in the main goroutine and thus will not cause any race conditions. If you access your primitives from other goroutines, however, you will need to synchronize execution. The easiest way to do this is to call Application.QueueUpdate() or Application.QueueUpdateDraw() (see the function documentation for details): One exception to this is the io.Writer interface implemented by TextView. You can safely write to a TextView from any goroutine. See the TextView documentation for details. You can also call Application.Draw() from any goroutine without having to wrap it in QueueUpdate(). And, as mentioned above, key event callbacks are executed in the main goroutine and thus should not use QueueUpdate() as that may lead to deadlocks. All widgets listed above contain the Box type. All of Box's functions are therefore available for all widgets, too. All widgets also implement the Primitive interface. There is also the Focusable interface which is used to override functions in subclassing types. The tview package is based on https://github.com/gdamore/tcell. It uses types and constants from that package (e.g. colors and keyboard values). This package does not process mouse input (yet).
Package dht implements a distributed hash table that satisfies the ipfs routing interface. This DHT is modeled after kademlia with S/Kademlia modifications.
Package dht implements a distributed hash table that satisfies the ipfs routing interface.
Package dht implements a distributed hash table that satisfies the ipfs routing interface.
Package dht implements a distributed hash table that satisfies the ipfs routing interface.
Package dht implements a distributed hash table that satisfies the ipfs routing interface.
Package cuckoo implements d-ary bucketized cuckoo hashing with stash (bucketized cuckoo hashing is also known as splash tables).
Package dht implements a distributed hash table that satisfies the ipfs routing interface.
Package dht implements a distributed hash table that satisfies the ipfs routing interface.
File Structures 2 This is a follow up to my crufty http://github.com/timtadh/file-structures work. That system has some endemic problems: 1. It uses the read/write interface to files. This means that it needs to do block management and cache management. In theory this can be very fast but it is also very challenging in Go. 2. Because it uses read/write it has to do buffer management. Largely, the system punts on this problem and allows go to handle the buffer management through the normal memory management system. This doesn't work especially well for the use case of file-structures. File Structures 2 is an experiment to bring Memory Mapped IO to the world of Go. The hypotheses are: 1. The operating system is good at page management generally. While, we know more about how to manage the structure of B+Trees, VarChar stores, and Linear Hash tables than the OS there is no indication that from Go you can acheive better performance. Therefore, I hypothesize that leaving it to the OS will lead to a smaller working set and a faster data structure in general. 2. You can make Memory Mapping performant in Go. There are many challenges here. The biggest of which is that there are no dynamically size array TYPES in go. The size of the array is part of the type, you have to use slices. This creates complications when hooking up structures which contain slices to mmap allocated blocks of memory. I hypothesize that this repository can acheive good (enough) performance here. The major components of this project: 1. fmap - a memory mapped file inteface. Part C part Go. Uses cgo. 2. bptree - a B+ Tree with duplicate key support (fixed size keys, variable length values) written on top of fmap. 3. slice - used by fmap and bptree to completely violate memory and type safety of Go. 4. errors - just a simple error package which maintains a stack trace with every error.
Package cuckoo implements d-ary bucketized cuckoo hashing with stash (bucketized cuckoo hashing is also known as splash tables).
Package dht implements a distributed hash table that satisfies the ipfs routing interface.
Package dht implements a distributed hash table that satisfies the ipfs routing interface.
Package dht implements a distributed hash table that satisfies the ipfs routing interface.
Package algorithms package demonstrates some basic algorithms Binary Sort , Select Sort , Recursive , Quick Sort , Hash table
Package cmap introduces a thead-safe capacity-constrained hash table that evicts key/value pairs according to the LRU (least recently used) policy.
Package consistent provides a consistent hashing function. Consistent hashing is often used to distribute requests to a changing set of servers. For example, say you have some cache servers cacheA, cacheB, and cacheC. You want to decide which cache server to use to look up information on a user. You could use a typical hash table and hash the user id to one of cacheA, cacheB, or cacheC. But with a typical hash table, if you add or remove a server, almost all keys will get remapped to different results, which basically could bring your service to a grinding halt while the caches get rebuilt. With a consistent hash, adding or removing a server drastically reduces the number of keys that get remapped. Read more about consistent hashing on wikipedia: http://en.wikipedia.org/wiki/Consistent_hashing
Package dht implements a distributed hash table that satisfies the ipfs routing interface.
Package dht implements a distributed hash table that satisfies the ipfs routing interface.
Package dht implements a distributed hash table that satisfies the ipfs routing interface.
Package dht implements a distributed hash table that satisfies the ipfs routing interface.
Package dht implements a distributed hash table that satisfies the ipfs routing interface.
Package cuckoo implements d-ary bucketized cuckoo hashing with stash (bucketized cuckoo hashing is also known as splash tables).
Package dht implements a distributed hash table that satisfies the ipfs routing interface.
Package dht implements a distributed hash table that satisfies the ipfs routing interface.
Package topk implements the Filtered Space-Saving TopK streaming algorithm The original Space-Saving algorithm: https://icmi.cs.ucsb.edu/research/tech_reports/reports/2005-23.pdf The Filtered Space-Saving enhancement: http://www.l2f.inesc-id.pt/~fmmb/wiki/uploads/Work/misnis.ref0a.pdf This implementation follows the algorithm of the FSS paper, but not the suggested implementation. Specifically, we use a heap instead of a sorted list of monitored items, and since we are also using a map to provide O(1) access on update also don't need the c_i counters in the hash table. Licensed under the MIT license.
Package dht implements a distributed hash table that satisfies the ipfs routing interface.
Package dht implements a distributed hash table that satisfies the ipfs routing interface.
Package mph implements a minimal perfect hash table over strings.