Real-time

Package lunk provides a set of tools for structured logging in the style of Google's Dapper or Twitter's Zipkin. When we consider a complex event in a distributed system, we're actually considering a partially-ordered tree of events from various services, libraries, and modules. Consider a user-initiated web request. Their browser sends an HTTP request to an edge server, which extracts the credentials (e.g., OAuth token) and authenticates the request by communicating with an internal authentication service, which returns a signed set of internal credentials (e.g., signed user ID). The edge web server then proxies the request to a cluster of web servers, each running a PHP application. The PHP application loads some data from several databases, places the user in a number of treatment groups for running A/B experiments, writes some data to a Dynamo-style distributed database, and returns an HTML response. The edge server receives this response and proxies it to the user's browser. In this scenario we have a number of infrastructure-specific events: This scenario also involves a number of events which have little to do with the infrastructure, but are still critical information for the business the system supports: There are a number of different teams all trying to monitor and improve aspects of this system. Operational staff need to know if a particular host or service is experiencing a latency spike or drop in throughput. Development staff need to know if their application's response times have gone down as a result of a recent deploy. Customer support staff need to know if the system is operating nominally as a whole, and for customers in particular. Product designers and managers need to know the effect of an A/B test on user behavior. But the fact that these teams will be consuming the data in different ways for different purposes does mean that they are working on different systems. In order to instrument the various components of the system, we need a common data model. We adopt Dapper's notion of a tree to mean a partially-ordered tree of events from a distributed system. A tree in Lunk is identified by its root ID, which is the unique ID of its root event. All events in a common tree share a root ID. In our photo example, we would assign a unique root ID as soon as the edge server received the request. Events inside a tree are causally ordered: each event has a unique ID, and an optional parent ID. By passing the IDs across systems, we establish causal ordering between events. In our photo example, the two database queries from the app would share the same parent ID--the ID of the event corresponding to the app handling the request which caused those queries. Each event has a schema of properties, which allow us to record specific pieces of information about each event. For HTTP requests, we can record the method, the request URI, the elapsed time to handle the request, etc. Lunk is agnostic in terms of aggregation technologies, but two use cases seem clear: real-time process monitoring and offline causational analysis. For real-time process monitoring, events can be streamed to a aggregation service like Riemann (http://riemann.io) or Storm (http://storm.incubator.apache.org), which can calculate process statistics (e.g., the 95th percentile latency for the edge server responses) in real-time. This allows for adaptive monitoring of all services, with the option of including example root IDs in the alerts (e.g., 95th percentile latency is over 300ms, mostly as a result of requests like those in tree XXXXX). For offline causational analysis, events can be written in batches to batch processing systems like Hadoop or OLAP databases like Vertica. These aggregates can be queried to answer questions traditionally reserved for A/B testing systems. "Did users who were show the new navbar view more photos?" "Did the new image optimization algorithm we enabled for 1% of views run faster? Did it produce smaller images? Did it have any effect on user engagement?" "Did any services have increased exception rates after any recent deploys?" &tc &tc By capturing the root ID of a particular web request, we can assemble a partially-ordered tree of events which were involved in the handling of that request. All events with a common root ID are in a common tree, which allows for O(M) retrieval for a tree of M events. To send a request with a root ID and a parent ID, use the Event-ID HTTP header: The header value is simply the root ID and event ID, hex-encoded and separated with a slash. If the event has a parent ID, that may be included as an optional third parameter. A server that receives a request with this header can use this to properly parent its own events. Each event has a set of named properties, the keys and values of which are strings. This allows aggregation layers to take advantage of simplifying assumptions and either store events in normalized form (with event data separate from property data) or in denormalized form (essentially pre-materializing an outer join of the normalized relations). Durations are always recorded as fractional milliseconds. Lunk currently provides two formats for log entries: text and JSON. Text-based logs encode each entry as a single line of text, using key="value" formatting for all properties. Event property keys are scoped to avoid collisions. JSON logs encode each entry as a single JSON object.

shortinette is the core framework for managing and automating the process of grading coding bootcamps (Shorts). It provides a comprehensive set of tools for running and testing student submissions across various programming languages. The shortinette package is composed of several sub-packages, each responsible for a specific aspect of the grading pipeline: `logger`: Handles logging for the framework, including general informational messages, error reporting, and trace logging for feedback on individual submissions. This package ensures that all important events and errors are captured for debugging and auditing purposes. `requirements`: Validates the necessary environment variables and dependencies required by the framework. This includes checking for essential configuration values in a `.env` file and ensuring that all necessary tools (e.g., Docker images) are available before grading begins. `testutils`: Provides utility functions for compiling and running code submissions. This includes functions for compiling Rust code, running executables with various options (such as timeouts and real-time output), and manipulating files. The utility functions are designed to handle the intricacies of running untrusted student code safely and efficiently. `git`: Manages interactions with GitHub, including cloning repositories, managing collaborators, and uploading files. This package abstracts the GitHub API to simplify common tasks such as adding collaborators to repositories, creating branches, and pushing code or data to specific locations in a repository. `exercise`: Defines the structure and behavior of individual coding exercises. This includes specifying the files that students are allowed to submit, the expected output, and the functions to be tested. The `exercise` package provides the framework for setting up exercises, running tests, and reporting results. `module`: Organizes exercises into modules, allowing for the grouping of related exercises into a coherent curriculum. The `module` package handles the execution of all exercises within a module, aggregates results, and manages the overall grading process. `webhook`: Enables automatic grading triggered by GitHub webhooks. This allows for a fully automated workflow where student submissions are graded as soon as they are pushed to a specific branch in a GitHub repository. `short`: The central orchestrator of the grading process, integrating all sub-packages into a cohesive system. The `short` package handles the setup and teardown of grading environments, manages the execution of modules and exercises, and ensures that all results are properly recorded and reported.

Gor is simple http traffic replication tool written in Go. Its main goal to replay traffic from production servers to staging and dev environments. Now you can test your code on real user sessions in an automated and repeatable fashion. Gor consists of 2 parts: listener and replay servers. Listener catch http traffic from given port in real-time and send it to replay server via UDP. Replay server forwards traffic to given address.

Package influxdb is the root package of InfluxDB, the scalable datastore for metrics, events, and real-time analytics. If you're looking for the Go HTTP client for InfluxDB, see package github.com/influxdata/influxdb/client/v2.

Package influxdb is the root package of InfluxDB, the scalable datastore for metrics, events, and real-time analytics. If you're looking for the Go HTTP client for InfluxDB, see package github.com/influxdata/influxdb/client/v2.

Speech recognition with Next-gen Kaldi. sherpa-onnx is an open-source speech recognition framework for Next-gen Kaldi. It depends only on onnxruntime, supporting both streaming and non-streaming speech recognition. It does not need to access the network during recognition and everything runs locally. It supports a variety of platforms, such as Linux (x86_64, aarch64, arm), Windows (x86_64, x86), macOS (x86_64, arm64), etc. Usage examples: Real-time speech recognition from a microphone Please see https://github.com/k2-fsa/sherpa-onnx/tree/master/go-api-examples/real-time-speech-recognition-from-microphone Decode files using a non-streaming model Please see https://github.com/k2-fsa/sherpa-onnx/tree/master/go-api-examples/non-streaming-decode-files Decode files using a streaming model Please see https://github.com/k2-fsa/sherpa-onnx/tree/master/go-api-examples/streaming-decode-files Convert text to speech using a non-streaming model Please see https://github.com/k2-fsa/sherpa-onnx/tree/master/go-api-examples/non-streaming-tts

Package influxdb is the root package of InfluxDB, the scalable datastore for metrics, events, and real-time analytics. If you're looking for the Go HTTP client for InfluxDB, see package github.com/influxdata/influxdb/client/v2.

Package applicationsignals provides the API client, operations, and parameter types for Amazon CloudWatch Application Signals. Use CloudWatch Application Signals for comprehensive observability of your cloud-based applications. It enables real-time service health dashboards and helps you track long-term performance trends against your business goals. The application-centric view provides you with unified visibility across your applications, services, and dependencies, so you can proactively monitor and efficiently triage any issues that may arise, ensuring optimal customer experience. Application Signals provides the following benefits: Automatically collect metrics and traces from your applications, and display key metrics such as call volume, availability, latency, faults, and errors. Create and monitor service level objectives (SLOs). See a map of your application topology that Application Signals automatically discovers, that gives you a visual representation of your applications, dependencies, and their connectivity. Application Signals works with CloudWatch RUM, CloudWatch Synthetics canaries, and Amazon Web Services Service Catalog AppRegistry, to display your client pages, Synthetics canaries, and application names within dashboards and maps.

Copyright 2015 Realtime Framework. The ortc package implements the Go lang version of the Realtime Messaging protocol, If your application has data that needs to be updated in the user’s interface as it changes (e.g. real-time stock quotes or ever changing social news feed) Realtime Messaging is the reliable, easy, unbelievably fast, “works everywhere” solution. Installation: Below are examples of use of the ortc package: - Create a new instance of ortc client: client, onConnected, onDisconnected, onException, onMessage, onReconnected, onReconnecting, onSubscribed, onUnsubscribed := ortc.NewOrtcClient() - Using the channels received on the ortc client: - Connect to a ortc server: client.Connect("YOUR_APPLICATION_KEY", "myToken", "GoApp", "http://ortc-developers.realtime.co/server/2.1", true, false) - Disconnect from ortc server: client.Disconnect() - Disable presence on a channel: ch := make(chan ortc.PresenceType) - Enable presence on a channel: - Get presence on channel: - Save Authentication: permissions := make(map[string][]authentication.ChannelPermissions) yellowPermissions := []authentication.ChannelPermissions{} yellowPermissions = append(yellowPermissions, authentication.Write) yellowPermissions = append(yellowPermissions, authentication.Presence) testPermissions := []authentication.ChannelPermissions{} testPermissions = append(testPermissions, authentication.Read) testPermissions = append(testPermissions, authentication.Presence) permissions["yellow:*"] = yellowPermissions permissions["test:*"] = testPermissions - Send message to a channel: client.Send("my_channel", "Hello World!") - Subscribe to a channel: client.Subscribe("my_channel", true) - Unsubscribe from a channel: client.Unsubscribe("my_channel") More documentation about the Realtime Messaging service (ORTC) can be found at: http://messaging-public.realtime.co/documentation/starting-guide/overview.html

Package bugsnag captures errors in real-time and reports them to Bugsnag (http://bugsnag.com). Using bugsnag-go is a three-step process. 1. As early as possible in your program configure the notifier with your APIKey. This sets up handling of panics that would otherwise crash your app. 2. Add bugsnag to places that already catch panics. For example you should add it to the HTTP server when you call ListenAndServer: If that's not possible, for example because you're using Google App Engine, you can also wrap each HTTP handler manually: 3. To notify Bugsnag of an error that is not a panic, pass it to bugsnag.Notify. This will also log the error message using the configured Logger. For detailed integration instructions see https://bugsnag.com/docs/notifiers/go. The only required configuration is the Bugsnag API key which can be obtained by clicking "Settings" on the top of https://bugsnag.com/ after signing up. We also recommend you set the ReleaseStage, AppType, and AppVersion if these make sense for your deployment workflow. If you need to attach extra data to Bugsnag notifications you can do that using the rawData mechanism. Most of the functions that send errors to Bugsnag allow you to pass in any number of interface{} values as rawData. The rawData can consist of the Severity, Context, User or MetaData types listed below, and there is also builtin support for *http.Requests. If you want to add custom tabs to your bugsnag dashboard you can pass any value in as rawData, and then process it into the event's metadata using a bugsnag.OnBeforeNotify() hook. If necessary you can pass Configuration in as rawData, or modify the Configuration object passed into OnBeforeNotify hooks. Configuration passed in this way only affects the current notification.

gokaf is a robust in-memory pubsub engine meticulously crafted to provide seamless and nearly real-time data streams.

Overview Monresql is a specialized library designed to facilitate efficient data replication, transfer, and synchronization from MongoDB to PostgreSQL databases. Inspired by similar tools like Moresql, Monresql focuses on unidirectional data movement, ensuring seamless integration and synchronization between MongoDB documents and PostgreSQL tables. Key Features Data Replication: Efficiently replicate data from MongoDB collections to corresponding PostgreSQL tables. Incremental Updates: Support for incremental updates to keep PostgreSQL data up-to-date without full data reloads. Performance Optimization: Optimizes data transfer processes for minimal latency and optimal resource utilization. API Reference LoadFieldsMap() Loads a mapping file to define how MongoDB documents should be mapped to PostgreSQL tables. ValidateOrCreatePostgresTable() Validates the existence of a PostgreSQL table to ensure it's ready for data replication. Replicate() Initiates the data replication process from MongoDB to PostgreSQL based on the loaded mapping. Sync() Starts the synchronization process, ensuring that changes in MongoDB are reflected in PostgreSQL in real-time and also save the marker to sync from the last stopped mark if the service stopped NewSyncOptions() NewSyncOptions will return the pointer of the syncoptions struct with default values of &syncOptions{checkpoint: true, checkPointPeriod: time.Minute * 1, lastEpoch: 0, reportPeriod: time.Minute * 1} then you can edit and change the values by set methods

Package ecpush is a package for subscribing to real-time meteorological data feeds from Environment Canada. The main goal of ecpush is to provide a simple and lightweight client that can be used for receiving real-time data events directly from Environment Canada's meteorological product feed. The client can directly fetch the published products, or it can just provide a notification channel containing the product location (HTTP URL to Environment Canada's Datamart). The client has also been designed to automatically recover from any connection or channel interruptions. To create a new client, create a Client struct. The only required field is the Subtopics array. Default values for other fields are listed in the struct definition. An example configuration is shown below (subscribing text bulletins, citypage XML and CAP alert files). Please see https://github.com/MetPX/sarracenia/blob/master/doc/sr_subscribe.1.rst#subtopic-amqp-pattern-subtopic-need-to-be-set for formatting subtopics. Calling Connect(ctx) will return an error if no subtopics are provided. The function will block until the initial connection with the remote server is established. When the client is provisioned, an internal Goroutine is created to consume the feed. To consume the events, call Consume() on the client. This returns an Event and an indicator if the client is still actively consuming from the remote server. To close the client, call the cancel function on the context provided to the client. This will gracefully close the active channels and connection to the remote server. A fully functioning client can be found in the example directory. I would like to thank Sean Treadway for his Go RabbitMQ client package. I would also like to thank Environment Canada and the awesome people at Shared Services Canada for their developments and "openness" of MetPX and sarracenia. Copyright (c) 2019 Tanner Ryan. All rights reserved. Use of this source code is governed by a BSD-style license that can be found in the LICENSE file. Sean Treadway's Go RabbitMQ client package is under a BSD 2-clause license. Cenk Alti's Go exponential backoff package is under an MIT license. Once again, all rights reserved.

Package influxdb is the root package of InfluxDB, the scalable datastore for metrics, events, and real-time analytics. If you're looking for the Go HTTP client for InfluxDB, see package github.com/influxdata/influxdb/client/v2.

Package influxdb is the root package of InfluxDB, the scalable datastore for metrics, events, and real-time analytics. If you're looking for the Go HTTP client for InfluxDB, see package github.com/influxdata/influxdb/client/v2.

Package livefile implements atomic file update operations, as well as consistent, safe real-time notifications for file changes.

Package influxdb is the root package of InfluxDB, the scalable datastore for metrics, events, and real-time analytics. If you're looking for the Go HTTP client for InfluxDB, see package github.com/ivopetiz/influxdb/client/v2.

Package influxdb is the root package of InfluxDB, the scalable datastore for metrics, events, and real-time analytics. If you're looking for the Go HTTP client for InfluxDB, see package github.com/influxdata/influxdb/client/v2.

Package gafka is a full ecosystem built for real-time cloud PubSub system.

Package gocent is a Go language API client for Centrifugo real-time messaging server. In example below we initialize new client with server URL address, project secret and request timeout. Then publish data into channel, call presence and history for channel and finally show how to publish several messages in one POST request to API endpoint using internal command buffer.

Banshee is a real-time anomalies(outliers) detection system for periodic metrics. We are using it to monitor our website and rpc services intefaces, including called frequency, response time and exception calls. Our services send statistics to statsd, statsd aggregates them every 10 seconds and broadcasts the results to its backends including banshee, banshee analyzes current metrics with history data, calculates the trending and alerts us if the trending behaves anomalous. For example, we have an api named get_user, this api's response time (in milliseconds) is reported to banshee from statsd every 10 seconds: Banshee will catch the latest metric 300 and report it as an anomaly. Why don't we just set a fixed threshold instead (i.e. 200ms)? This may also works but it is boring and hard to maintain a lot of thresholds. Banshee will analyze metric trendings automatically, it will find the "thresholds" automatically. 1. Designed for periodic metrics. Reality metrics are always with periodicity, banshee only peeks metrics with the same "phase" to detect. 2. Multiple alerting rule configuration options, to alert via fixed-thresholds or via anomalous trendings. 3. Coming with anomalies visualization webapp and alerting rules admin panels. 4. Require no extra storage services, banshee handles storage on disk by itself. 1. Go >= 1.4 and godep. 2. Node and gulp. 1. Clone the repo. 2. Build binary via `make`. 3. Build static files via `make static`. Usage: Flags: See package config. In order to forward metrics to banshee from statsd, we need to add the npm module statsd-banshee to statsd's banckends: 1. Install statsd-banshee on your statsd servers: 2. Add module statsd-banshee to statsd's backends in config.js: Require bell.js v2.0+ and banshee v0.0.7+: Banshee have 4 compontents and they are running in the same process: 1. Detector is to detect incoming metrics with history data and store the results. 2. Webapp is to visualize the detection results and provides panels to manage alerting rules, projects and users. 3. Alerter is to send sms and emails once anomalies are found. 4. Cleaner is to clean outdated metrics from storage. See package alerter and alerter/exampleCommand. 1. Detection algorithms, see package detector. 2. Detector input net protocol, see package detector. 3. Storage, see package storage. 4. Filter, see package filter. MIT (c) eleme, inc.

Package movingminmax provides an efficient O(1) moving minimum-maximum filter that can be used in real-time contexts. It uses the algorithm from: Daniel Lemire, Streaming Maximum-Minimum Filter Using No More than Three Comparisons per Element. Nordic Journal of Computing, 13 (4), pages 328-339, 2006. http://arxiv.org/abs/cs/0610046 This implementation uses a fixed amount of memory and makes no dynamic allocations during updates.

Banshee is a real-time anomalies(outliers) detection system for periodic metrics. We are using it to monitor our website and rpc services intefaces, including called frequency, response time and exception calls. Our services send statistics to statsd, statsd aggregates them every 10 seconds and broadcasts the results to its backends including banshee, banshee analyzes current metrics with history data, calculates the trending and alerts us if the trending behaves anomalous. For example, we have an api named get_user, this api's response time (in milliseconds) is reported to banshee from statsd every 10 seconds: Banshee will catch the latest metric 300 and report it as an anomaly. Why don't we just set a fixed threshold instead (i.e. 200ms)? This may also works but it is boring and hard to maintain a lot of thresholds. Banshee will analyze metric trendings automatically, it will find the "thresholds" automatically. 1. Designed for periodic metrics. Reality metrics are always with periodicity, banshee only peeks metrics with the same "phase" to detect. 2. Multiple alerting rule configuration options, to alert via fixed-thresholds or via anomalous trendings. 3. Coming with anomalies visualization webapp and alerting rules admin panels. 4. Require no extra storage services, banshee handles storage on disk by itself. 1. Go >= 1.5. 2. Node and gulp. 1. Clone the repo. 2. Build binary via `make`. 3. Build static files via `make static`. Usage: Flags: See package config. In order to forward metrics to banshee from statsd, we need to add the npm module statsd-banshee to statsd's banckends: 1. Install statsd-banshee on your statsd servers: 2. Add module statsd-banshee to statsd's backends in config.js: Require bell.js v2.0+ and banshee v0.0.7+: Banshee have 4 compontents and they are running in the same process: 1. Detector is to detect incoming metrics with history data and store the results. 2. Webapp is to visualize the detection results and provides panels to manage alerting rules, projects and users. 3. Alerter is to send sms and emails once anomalies are found. 4. Cleaner is to clean outdated metrics from storage. See package alerter and alerter/exampleCommand. Via fabric(http://www.fabfile.org/): See deploy.py docs for more. Just pull the latest code: Note that the admin storage sqlite3 schema will be auto-migrated. 1. Detection algorithms, see package detector. 2. Detector input net protocol, see package detector. 3. Storage, see package storage. 4. Filter, see package filter. MIT (c) eleme, inc.

Package track provides a beep.Streamer with real-time stream insertion.

`zkclient` is a encapsulation utility of zookeeper based on [go-zookeeper](github.com/samuel/go-zookeeper), supports the following features: ## Features: - auto reconnect - set/get/delete value - support string/json codec, and you can implement your own - real-time synchronize data from zookeeper to memory ## Modules - `Codec`: value encode/decode - `Watcher`: loop watch control - `Handler`: include `valueHandler` and `mapHandler`, set/get/delete value, handle event, synchronize value, trigger listener - `Listener`: include `ValueListener` and `ChildListener`, listen value updated/deleted ## API - `Sync*`: synchronize data into memory - `SyncWatch*`: synchronize data into memory, and listen value change - `Decode*`: set value into zookeeper - `Encode*`: get value from zookeeper

Banshee is a real-time anomalies(outliers) detection system for periodic metrics. We are using it to monitor our website and rpc services intefaces, including called frequency, response time and exception calls. Our services send statistics to statsd, statsd aggregates them every 10 seconds and broadcasts the results to its backends including banshee, banshee analyzes current metrics with history data, calculates the trending and alerts us if the trending behaves anomalous. For example, we have an api named get_user, this api's response time (in milliseconds) is reported to banshee from statsd every 10 seconds: Banshee will catch the latest metric 300 and report it as an anomaly. Why don't we just set a fixed threshold instead (i.e. 200ms)? This may also works but it is boring and hard to maintain a lot of thresholds. Banshee will analyze metric trendings automatically, it will find the "thresholds" automatically. 1. Designed for periodic metrics. Reality metrics are always with periodicity, banshee only peeks metrics with the same "phase" to detect. 2. Multiple alerting rule configuration options, to alert via fixed-thresholds or via anomalous trendings. 3. Coming with anomalies visualization webapp and alerting rules admin panels. 4. Require no extra storage services, banshee handles storage on disk by itself. 1. Go >= 1.5. 2. Node and gulp. 1. Clone the repo. 2. Build binary via `make`. 3. Build static files via `make static`. Usage: Flags: See package config. In order to forward metrics to banshee from statsd, we need to add the npm module statsd-banshee to statsd's banckends: 1. Install statsd-banshee on your statsd servers: 2. Add module statsd-banshee to statsd's backends in config.js: Require bell.js v2.0+ and banshee v0.0.7+: Banshee have 4 compontents and they are running in the same process: 1. Detector is to detect incoming metrics with history data and store the results. 2. Webapp is to visualize the detection results and provides panels to manage alerting rules, projects and users. 3. Alerter is to send sms and emails once anomalies are found. 4. Cleaner is to clean outdated metrics from storage. See package alerter and alerter/exampleCommand. Via fabric(http://www.fabfile.org/): See deploy.py docs for more. Just pull the latest code: Note that the admin storage sqlite3 schema will be auto-migrated. 1. Detection algorithms, see package detector. 2. Detector input net protocol, see package detector. 3. Storage, see package storage. 4. Filter, see package filter. Reference: https://github.com/eleme/banshee/blob/master/intro.md MIT (c) eleme, inc.

Package godnsbl lets you perform RBL (Real-time Blackhole List - https://en.wikipedia.org/wiki/DNSBL) lookups using Golang JSON annotations on the types are provided as a convenience.

Package influxdb is the root package of InfluxDB, the scalable datastore for metrics, events, and real-time analytics. If you're looking for the Go HTTP client for InfluxDB, see package github.com/influxdata/influxdb/client/v2.

Package influxdb is the root package of InfluxDB, the scalable datastore for metrics, events, and real-time analytics. If you're looking for the Go HTTP client for InfluxDB, see package github.com/influxdata/influxdb/client/v2.