Streaming

Package graw is a high level, easy to use, Reddit bot library. graw will take a low level handle from the graw/reddit package and manage everything for you. You just specify in a config what events you want to listen to on Reddit, and graw will take care of maintaining the event stream and calling the handler methods of your bot whenever new events occur. Announcing new posts in /r/self is as simple as: graw can handle many event sources on Reddit; see the Config struct for the complete set of offerings. graw has one other function that behaves like Scan(), which is Run(). Scan() is for logged-out bots (what Reddit calls "scripts"). Run() handles logged in bots, which can subscribe to logged-in event sources in the bot's account inbox like mentions and private messages.

Package mp4ff - MP4 media file parser and writer for AVC and HEVC video, AAC audio and stpp/wvtt subtitles. Focused on fragmented files as used for streaming in DASH, MSS and HLS fMP4. The mp4 library has functions for parsing (called Decode) and writing (called Encode). It is focused on fragmented files as used for streaming in DASH, MSS and HLS fMP4. mp4.File is a representation of a "File" which can be more or less complete, but should have some top layer boxes. It can include The typical child boxes are exported so that one can write paths such as to access the (only) trun box of a moof box. Some simple command line tools are available in cmd directory. Example code is available in the examples directory.

Package redisqueue provides a producer and consumer of a queue that uses Redis streams (https://redis.io/topics/streams-intro). The features of this package include: Here's an example of a producer that inserts 1000 messages into a queue: And here's an example of a consumer that reads the messages off of that queue:

Package multistream implements a simple stream router for the multistream-select protocoli. The protocol is defined at https://github.com/multiformats/multistream-select

Module mp4ff implements MP4 media file parsing and writing for AVC and HEVC video, AAC and AC-3 audio, stpp and wvtt subtitles, and timed metadata tracks. It is focused on fragmented files as used for streaming in MPEG-DASH, MSS and HLS fMP4, but can also decode and encode all boxes needed for progressive MP4 files. Some useful command line tools are available in [cmd](cmd) directory. You can install these tools by going to their respective directory and run `go install .` or directly from the repo with for each individual tool. Example code for some common use cases is available in the [examples](examples) directory. The examples and their functions are: The top-level packages in the mp4ff module are The main specification for the MP4 file format is the ISO Base Media File Format (ISOBMFF) standard ISO/IEC 14496-12 7th edition 2021. Some boxes are specified in other standards, as should be commented in the code.

Package stream provides filters that can be chained together in a manner similar to Unix pipelines. A simple example that prints all go files under the current directory: stream.Run is passed a list of filters that are chained together (stream.Find, stream.Grep, stream.WriteLines are filters). Each filter takes as input a sequence of strings and produces a sequence of strings. The empty sequence is passed as input to the first filter. The output of one filter is fed as input to the next filter. stream.Run is just one way to execute filters. Others are stream.Contents (returns the output of the last filter as a []string), and stream.ForEach (executes a supplied function for every output item). Filter execution can result in errors. These are returned from stream functions normally. For example, the following call will return a non-nil error. Each filter takes as input a sequence of strings (read from a channel) and produces as output a sequence of strings (written to a channel). The stream package provides a bunch of useful filters. Applications can define their own filters easily. For example, here is a filter that repeats every input n times: The output will be: Note that Repeat returns a FilterFunc, a function type that implements the Filter interface. This is a common implementation pattern: many simple filters can be expressed as a single function of type FilterFunc. FilterFunc is an appropriate type to use for most filters like Repeat above. However for some filters, dynamic customization is appropriate. Such filters provide their own implementation of the Filter interface with extra methods. For example, stream.Sort provides extra methods that can be used to control how items are sorted: The interface of this package is inspired by the http://labix.org/pipe package. Users may wish to consider that package in case it fits their needs better.

MIDI package is a midi format parser that parses the content of a midi stream and asyncronously send the data to a provided channel or collect the data into an accessible struct,

Package vssh is a Go library to handle tens of thousands SSH connections and execute the command with higher-level API for building network device / server automation. By calling the run method vssh sends the given command to all available clients or based on your query it runs the command on the specific clients and the results of the ran command can be received in two options, streaming or final result.In streaming you can get line by line from command’s stdout / stderr in real time or in case of non-real time you can get the whole of the lines together. This example demonstrates integration vSSH with AWS EC2

Package p2phttp allows to serve HTTP endpoints and make HTTP requests through LibP2P (https://github.com/libp2p/libp2p) using Go's standard "http" and "net" stacks. Instead of the regular "host:port" addressing, `p2phttp` uses a Peer ID and lets LibP2P take care of the routing, thus taking advantage of features like multi-routes, NAT transversal and stream multiplexing over a single connection. When already running a LibP2P facility, this package allows to expose existing HTTP-based services (like REST APIs) through LibP2P and to use those services with minimal changes to the code-base. For example, a simple http.Server on LibP2P works as: As shown above, a Server only needs a "github.com/libp2p/go-libp2p-gostream" listener. This listener will use a libP2P host to watch for stream tagged with our Protocol. On the other side, a client just needs to be initialized with a custom LibP2P host-based transport to perform requests to such server: In the example above, the client registers a "libp2p" protocol for which the custom transport is used. It can still perform regular "http" requests. The protocol name used is arbitraty and non standard. Note that LibP2P hosts cannot dial to themselves, so there is no possibility of using the same host as server and as client.

Package grpchan provides an abstraction for a gRPC transport, called a Channel. The channel is more general than the concrete *grpc.ClientConn and *grpc.Server types provided by gRPC. It allows gRPC over alternate substrates and includes sub-packages that provide two such alternatives: in-process and HTTP 1.1. The key type in this package is an alternate implementation of grpc.ServiceRegistrar interface that allows you to accumulate service registrations, for use with an implementation other than *grpc.Server. This repo also includes a deprecated protoc plugin. This is no longer needed now that the standard protoc-gen-go-grpc plugin generates code that uses interfaces: grpc.ClientConnInterface and grpc.ServiceRegistrar. In older versions, the generated code only supported concrete types (*grpc.ClientConn and *grpc.Server) so this repo's protoc plugin would generate alternate code that used interfaces (and thus supported other concrete implementations). Continued use of the plugin is only to continue supporting code that uses the functions generated by it. To use the protoc plugin, you need to first build it and make sure its location is in your PATH. When you invoke protoc, include a --grpchan_out parameter that indicates the same output directory as used for your --go_out parameter. Alongside the *.pb.go files generated, the grpchan plugin will also create *.pb.grpchan.go files. In older versions of the Go plugin (when emitting gRPC code), a server registration function for each RPC service defined in the proto source files was generated that looked like so: The grpchan plugin produces a similarly named method that accepts the ServiceRegistry interface: A new transport can then be implemented by just implementing two interfaces: grpc.ClientConnInterface for the client side and grpchan.ServiceRegistry for the server side. The alternate method also works just fine with *grpc.Server as it implements the ServiceRegistry interface. NOTE: If your have code relying on New<ServiceName>ChannelClient methods that earlier versions of this package produced, they can still be generated by passing a "legacy_stubs" option to the plugin. Example: The client-side implementation of a transport is done with just the two methods in grpc.ClientConnInterface: one for unary RPCs and the other for streaming RPCs. Note that when a unary interceptor is invoked for an RPC on a channel that is *not* a *grpc.ClientConn, the parameter of that type will be nil. Not all client call options will make sense for all transports. This repo chooses to ignore call options that do not apply (as opposed to failing the RPC or panicking). However, several call options are likely important to support: those for accessing header and trailer metadata. The peer, per-RPC credentials, and message size limits are other options that are reasonably straight-forward to apply to other transports. But the other options (dealing with on-the-wire encoding, compression, etc) may not be applicable. The server-side implementation of a transport must be able to invoke method and stream handlers for a given service implementation. This is done by implementing the grpc.ServiceRegistrar interface. When a service is registered, a service description is provided that includes access to method and stream handlers. When the transport receives requests for RPC operations, it in turn invokes these handlers. For streaming operations, it must also supply a grpc.ServerStream implementation, for exchanging messages on the stream. Note that the server stream's context will need a custom implementation of the grpc.ServerTransportStream in it, too. Sadly, this interface is just different enough from grpc.ServerStream that they cannot be implemented by the same type. This is particularly necessary for unary calls since this is how a unary handler indicates what headers and trailers to send back to the client.

Package nrstan instruments https://github.com/nats-io/stan.go. This package can be used to simplify instrumenting NATS Streaming subscribers. Currently due to the nature of the NATS Streaming framework we are limited to two integration points: `StartPublishSegment` for publishers, and `SubWrapper` for subscribers. `nrstan.StreamingSubWrapper` can be used to wrap the function for STREAMING stan.Subscribe and stan.QueueSubscribe (https://godoc.org/github.com/nats-io/stan.go#Conn) If the `newrelic.Application` parameter is non-nil, it will create a `newrelic.Transaction` and end the transaction when the passed function is complete. Example: You can use `nrnats.StartPublishSegment` from the `nrnats` package (https://godoc.org/github.com/newrelic/go-agent/v3/integrations/nrnats/#StartPublishSegment) to start an external segment when doing a streaming publish, which must be ended after publishing is complete. Example: Full Publisher/Subscriber example: https://github.com/newrelic/go-agent/blob/master/v3/integrations/nrstan/examples/main.go

muxado implements a general purpose stream-multiplexing protocol. muxado allows clients applications to multiplex any io.ReadWriteCloser (like a net.Conn) into multiple, independent full-duplex streams. muxado is a useful protocol for any two communicating processes. It is an excellent base protocol for implementing lightweight RPC. It eliminates the need for custom async/pipeling code from your peers in order to support multiple simultaneous inflight requests between peers. For the same reason, it also eliminates the need to build connection pools for your clients. It enables servers to initiate streams to clients without building any NAT traversal. muxado can also yield performance improvements (especially latency) for protocols that require rapidly opening many concurrent connections. Here's an example client which responds to simple JSON requests from a server. Maybe the client wants to make a request to the server instead of just responding. This is easy as well: muxado defines the following terms for clarity of the documentation: A "Transport" is an underlying stream (typically TCP) that is multiplexed by sending frames between muxado peers over this transport. A "Stream" is any of the full-duplex byte-streams multiplexed over the transport A "Session" is two peers running the muxado protocol over a single transport muxado's design is influenced heavily by the framing layer of HTTP2 and SPDY. However, instead of being specialized for a higher-level protocol, muxado is designed in a protocol agnostic way with simplicity and speed in mind. More advanced features are left to higher-level libraries and protocols. muxado's API is designed to make it seamless to integrate into existing Go programs. muxado.Session implements the net.Listener interface and muxado.Stream implements net.Conn. muxado ships with two wrappers that add commonly used functionality. The first is a TypedStreamSession which allows a client application to open streams with a type identifier so that the remote peer can identify the protocol that will be communicated on that stream. The second wrapper is a simple Heartbeat which issues a callback to the application informing it of round-trip latency and heartbeat failure.

Package edn implements encoding and decoding of EDN values as defined in https://github.com/edn-format/edn. For a full introduction on how to use go-edn, see https://github.com/go-edn/edn/blob/v1/docs/introduction.md. Fully self-contained examples of go-edn can be found at https://github.com/go-edn/edn/tree/v1/examples. Note that the small examples in this package is not checking errors as persively as you should do when you use this package. This is done because I'd like the examples to be easily readable and understandable. The bigger examples provide proper error handling. EDN, in contrast to JSON, supports arbitrary values as keys. This example shows how one can implement enums and sets, and how to support multiple different forms for a specific value type. The set implemented here supports the notation `:all` for all values. This example shows how to read and write basic EDN tags, and how this can be utilised: In contrast to encoding/json, you can read in data where you only know that the input satisfies some sort of interface, provided the value is tagged. This example shows how one can do streaming with the decoder, and how to properly know when the stream has no elements left.

Package chunk implements streaming block splitters. Splitters read data from a reader and provide byte slices (chunks) The size and contents of these slices depend on the splitting method used.

hllpp implements the HyperLogLog++ cardinality estimator as specified in the HyperLogLog++ paper http://goo.gl/Z5Sqgu. hllpp uses a built-in non-streaming implementation of murmur3 to hash data as you add it to the estimator.

Package gchalk is terminal string styling for go done right, with full Linux, MacOS, and painless Windows 10 support. GChalk is a library heavily inspired by https://github.com/chalk/chalk, the popular Node.js terminal color library, and using golang ports of supports-color (https://github.com/jwalton/go-supportscolor) and ansi-styles (https://github.com/jwalton/gchalk/pkg/ansistyles). A very simple usage example would be: Note that this works on all platforms - there's no need to write to a special stream or use a special print function to get color on Windows 10. Some examples: See the README.md for more details.

Package signalr contains a SignalR client and a SignalR server. Both support the transport types Websockets and Server-Sent Events and the transfer formats Text (JSON) and Binary (MessagePack). The SignalR Protocol is a protocol for two-way RPC over any stream- or message-based transport. Either party in the connection may invoke procedures on the other party, and procedures can return zero or more results or an error. Typically, SignalR connections are HTTP-based, but it is dead simple to implement a signalr.Connection on any transport that supports io.Reader and io.Writer. A Client can be used in client side code to access server methods. From an existing connection, it can be created with NewClient(). A special case is NewHTTPClient(), which creates a Client from a server address and negotiates with the server which kind of connection (Websockets, Server-Sent Events) will be used. The object which will receive server callbacks is passed to NewClient() by using the WithReceiver option. After calling client.Start(), the client is ready to call server methods or to receive callbacks. A Server provides the public methods of a server side class over signalr to the client. Such a server side class is called a hub and must implement HubInterface. It is reasonable to derive your hubs from the Hub struct type, which already implements HubInterface. Servers for arbitrary connection types can be created with NewServer(). To serve a connection, call server.Serve(connection) in a goroutine. Serve ends when the connection is closed or the servers context is canceled. To serve a HTTP connection, use server.MapHTTP(), which connects the server with a path in an http.ServeMux. The server then automatically negotiates which kind of connection (Websockets, Server-Sent Events) will be used. The SignalR protocol constrains the signature of hub or receiver methods that can be used over SignalR. All methods with serializable types as parameters and return types are supported. Methods with multiple return values are not generally supported, but returning one or no value and an optional error is supported. Methods which return a single sending channel (<-chan), and optionally an error, are used to initiate callee side streaming. The caller will receive the contents of the channel as stream. When the returned channel is closed, the stream will be completed. Methods with one or multiple receiving channels (chan<-) as parameters are used as receivers for caller side streaming. The caller invokes this method and pushes one or multiple streams to the callee. The method should end when all channels are closed. A channel is closed by the server when the assigned stream is completed. The methods which return a channel are not supported. In most cases, the caller will be the client and the callee the server. But the vice versa case is also possible.

Package mpegts proves utility functions for mpegts streams

Package irc allows your application to speak the IRC protocol. The Message and Prefix structs provide translation to and from raw IRC messages: Decoder and Encoder can be used to decode and encode messages in a stream: The Conn type combines an Encoder and Decoder for a duplex connection.

Package wire implements the Decred wire protocol. For the complete details of the Decred protocol, see the official wiki entry at https://en.bitcoin.it/wiki/Protocol_specification. The following only serves as a quick overview to provide information on how to use the package. At a high level, this package provides support for marshalling and unmarshalling supported Decred messages to and from the wire. This package does not deal with the specifics of message handling such as what to do when a message is received. This provides the caller with a high level of flexibility. The Decred protocol consists of exchanging messages between peers. Each message is preceded by a header which identifies information about it such as which Decred network it is a part of, its type, how big it is, and a checksum to verify validity. All encoding and decoding of message headers is handled by this package. To accomplish this, there is a generic interface for Decred messages named Message which allows messages of any type to be read, written, or passed around through channels, functions, etc. In addition, concrete implementations of most of the currently supported Decred messages are provided. For these supported messages, all of the details of marshalling and unmarshalling to and from the wire using Decred encoding are handled so the caller doesn't have to concern themselves with the specifics. The following provides a quick summary of how the Decred messages are intended to interact with one another. As stated above, these interactions are not directly handled by this package. For more in-depth details about the appropriate interactions, see the official Decred protocol wiki entry at https://en.bitcoin.it/wiki/Protocol_specification. The initial handshake consists of two peers sending each other a version message (MsgVersion) followed by responding with a verack message (MsgVerAck). Both peers use the information in the version message (MsgVersion) to negotiate things such as protocol version and supported services with each other. Once the initial handshake is complete, the following chart indicates message interactions in no particular order. There are several common parameters that arise when using this package to read and write Decred messages. The following sections provide a quick overview of these parameters so the next sections can build on them. The protocol version should be negotiated with the remote peer at a higher level than this package via the version (MsgVersion) message exchange, however, this package provides the wire.ProtocolVersion constant which indicates the latest protocol version this package supports and is typically the value to use for all outbound connections before a potentially lower protocol version is negotiated. The Decred network is a magic number which is used to identify the start of a message and which Decred network the message applies to. This package provides the following constants: As discussed in the Decred message overview section, this package reads and writes Decred messages using a generic interface named Message. In order to determine the actual concrete type of the message, use a type switch or type assertion. An example of a type switch follows: In order to unmarshall Decred messages from the wire, use the ReadMessage function. It accepts any io.Reader, but typically this will be a net.Conn to a remote node running a Decred peer. Example syntax is: In order to marshall Decred messages to the wire, use the WriteMessage function. It accepts any io.Writer, but typically this will be a net.Conn to a remote node running a Decred peer. Example syntax to request addresses from a remote peer is: Errors returned by this package are either the raw errors provided by underlying calls to read/write from streams such as io.EOF, io.ErrUnexpectedEOF, and io.ErrShortWrite, or of type wire.MessageError. This allows the caller to differentiate between general IO errors and malformed messages through type assertions. This package includes spec changes outlined by the following BIPs:

Package sqlite provides a Go interface to SQLite 3. The semantics of this package are deliberately close to the SQLite3 C API. See the official C API introduction for an overview of the basics. An SQLite connection is represented by a *Conn. Connections cannot be used concurrently. A typical Go program will create a pool of connections (e.g. by using zombiezen.com/go/sqlite/sqlitex.NewPool to create a *zombiezen.com/go/sqlite/sqlitex.Pool) so goroutines can borrow a connection while they need to talk to the database. This package assumes SQLite will be used concurrently by the process through several connections, so the build options for SQLite enable multi-threading and the shared cache. The implementation automatically handles shared cache locking, see the documentation on Stmt.Step for details. The optional SQLite 3 extensions compiled in are: session, FTS5, RTree, JSON1, and GeoPoly. This is not a database/sql driver. For helper functions to make it easier to execute statements, see the zombiezen.com/go/sqlite/sqlitex package. Statements are prepared with the Conn.Prepare and Conn.PrepareTransient methods. When using Conn.Prepare, statements are keyed inside a connection by the original query string used to create them. This means long-running high-performance code paths can write: After all the connections in a pool have been warmed up by passing through one of these Prepare calls, subsequent calls are simply a map lookup that returns an existing statement. SQLite transactions can be managed manually with this package by directly executing BEGIN / COMMIT / ROLLBACK or SAVEPOINT / RELEASE / ROLLBACK statements, but there are also helper functions available in zombiezen.com/go/sqlite/sqlitex: For simple schema migration needs, see the zombiezen.com/go/sqlite/sqlitemigration package. Use Conn.CreateFunction to register Go functions for use as SQL functions. The sqlite package supports the SQLite incremental I/O interface for streaming blob data into and out of the the database without loading the entire blob into a single []byte. (This is important when working either with very large blobs, or more commonly, a large number of moderate-sized blobs concurrently.) See Conn.OpenBlob for more details. Every connection can have a done channel associated with it using the Conn.SetInterrupt method. This is typically the channel returned by a context.Context.Done method. As database connections are long-lived, the Conn.SetInterrupt method can be called multiple times to reset the associated lifetime. Using a Pool to execute SQL in a concurrent HTTP handler. This is the same as the main package example, but uses the SQLite statement API instead of sqlitex.

Package rpc is heavily inspired by Go standard net/rpc package. It aims to do the same thing, except it uses libp2p for communication and provides context support for cancelling operations. A server registers an object, making it visible as a service with the name of the type of the object. After registration, exported methods of the object will be accessible remotely. A server may register multiple objects (services) of different types but it is an error to register multiple objects of the same type. Only methods that satisfy these criteria will be made available for remote access; other methods will be ignored: In effect, the method must look schematically like where T1 and T2 can be marshaled by github.com/ugorji/go/codec. In normal calls, the method's second argument represents the arguments provided by the caller; the third argument represents the result parameters to be returned to the caller. The function error response is passed to the client accordingly. In streaming calls, the method's second and third arguments are argument and replies channels. The method is expected to read from the argument channel until it is closed. The method is expected to send responses on the replies channel and close it when done. Both channels are transparently and asynchronously streamed on the wire between remote hosts. In order to use this package, a ready-to-go libp2p Host must be provided to clients and servers, along with a protocol.ID. rpc will add a stream handler for the given protocol. Contexts are supported and honored when provided. On the server side, methods must take a context. A closure or reset of the libp2p stream will trigger a cancellation of the context received by the functions. On the client side, the user can optionally provide a context. Cancelling the client's context will cancel the operation both on the client and on the server side (by closing the associated stream).

Package minhash implements the bottom-k sketch for streaming set similarity. For more information, This package works best when provided with a strong 64-bit hash function, such as CityHash, Spooky, Murmur3, or SipHash.