Package dawg is an implemention of a Directed Acyclic Word Graph, as described on my blog at http://stevehanov.ca/blog/?id=115 A DAWG provides fast lookup of all possible prefixes of words in a dictionary, as well as the ability to get the index number of any word. This particular implementation may be different from others because it is very memory efficient, and it also works fast with large character sets. It can deal with thousands of branches out of a single node without needing to go through each one. The storage format is as small as possible. Bits are used instead of bytes so that no space is wasted as padding, and there are no practical limitations to the number of nodes or characters. A summary of the data format is found at the top of disk.go. In general, to use it you first create a builder using dawg.New(). You can then add words to the Dawg. The two restrictions are that you cannot repeat a word, and they must be in strictly increasing alphabetical order. After all the words are added, call Finish() which returns a dawg.Finder interface. You can perform queries with this interface, such as finding all prefixes of a given string which are also words, or looking up a word's index that you have previously added. After you have called Finish() on a Builder, you may choose to write it to disk using the Save() function. The DAWG can then be opened again later using the Load() function. When opened from disk, no memory is used. The structure is accessed in-place on disk.
Package gofpdf implements a PDF document generator with high level support for text, drawing and images. - UTF-8 support - Choice of measurement unit, page format and margins - Page header and footer management - Automatic page breaks, line breaks, and text justification - Inclusion of JPEG, PNG, GIF, TIFF and basic path-only SVG images - Colors, gradients and alpha channel transparency - Outline bookmarks - Internal and external links - TrueType, Type1 and encoding support - Page compression - Lines, Bézier curves, arcs, and ellipses - Rotation, scaling, skewing, translation, and mirroring - Clipping - Document protection - Layers - Templates - Barcodes - Charting facility - Import PDFs as templates gofpdf has no dependencies other than the Go standard library. All tests pass on Linux, Mac and Windows platforms. gofpdf supports UTF-8 TrueType fonts and “right-to-left” languages. Note that Chinese, Japanese, and Korean characters may not be included in many general purpose fonts. For these languages, a specialized font (for example, NotoSansSC for simplified Chinese) can be used. Also, support is provided to automatically translate UTF-8 runes to code page encodings for languages that have fewer than 256 glyphs. This repository will not be maintained, at least for some unknown duration. But it is hoped that gofpdf has a bright future in the open source world. Due to Go’s promise of compatibility, gofpdf should continue to function without modification for a longer time than would be the case with many other languages. Forks should be based on the last viable commit. Tools such as active-forks can be used to select a fork that looks promising for your needs. If a particular fork looks like it has taken the lead in attracting followers, this README will be updated to point people in that direction. The efforts of all contributors to this project have been deeply appreciated. Best wishes to all of you. To install the package on your system, run Later, to receive updates, run The following Go code generates a simple PDF file. See the functions in the fpdf_test.go file (shown as examples in this documentation) for more advanced PDF examples. If an error occurs in an Fpdf method, an internal error field is set. After this occurs, Fpdf method calls typically return without performing any operations and the error state is retained. This error management scheme facilitates PDF generation since individual method calls do not need to be examined for failure; it is generally sufficient to wait until after Output() is called. For the same reason, if an error occurs in the calling application during PDF generation, it may be desirable for the application to transfer the error to the Fpdf instance by calling the SetError() method or the SetErrorf() method. At any time during the life cycle of the Fpdf instance, the error state can be determined with a call to Ok() or Err(). The error itself can be retrieved with a call to Error(). This package is a relatively straightforward translation from the original FPDF library written in PHP (despite the caveat in the introduction to Effective Go). The API names have been retained even though the Go idiom would suggest otherwise (for example, pdf.GetX() is used rather than simply pdf.X()). The similarity of the two libraries makes the original FPDF website a good source of information. It includes a forum and FAQ. However, some internal changes have been made. Page content is built up using buffers (of type bytes.Buffer) rather than repeated string concatenation. Errors are handled as explained above rather than panicking. Output is generated through an interface of type io.Writer or io.WriteCloser. A number of the original PHP methods behave differently based on the type of the arguments that are passed to them; in these cases additional methods have been exported to provide similar functionality. Font definition files are produced in JSON rather than PHP. A side effect of running go test ./... is the production of a number of example PDFs. These can be found in the gofpdf/pdf directory after the tests complete. Please note that these examples run in the context of a test. In order run an example as a standalone application, you’ll need to examine fpdf_test.go for some helper routines, for example exampleFilename() and summary(). Example PDFs can be compared with reference copies in order to verify that they have been generated as expected. This comparison will be performed if a PDF with the same name as the example PDF is placed in the gofpdf/pdf/reference directory and if the third argument to ComparePDFFiles() in internal/example/example.go is true. (By default it is false.) The routine that summarizes an example will look for this file and, if found, will call ComparePDFFiles() to check the example PDF for equality with its reference PDF. If differences exist between the two files they will be printed to standard output and the test will fail. If the reference file is missing, the comparison is considered to succeed. In order to successfully compare two PDFs, the placement of internal resources must be consistent and the internal creation timestamps must be the same. To do this, the methods SetCatalogSort() and SetCreationDate() need to be called for both files. This is done automatically for all examples. Nothing special is required to use the standard PDF fonts (courier, helvetica, times, zapfdingbats) in your documents other than calling SetFont(). You should use AddUTF8Font() or AddUTF8FontFromBytes() to add a TrueType UTF-8 encoded font. Use RTL() and LTR() methods switch between “right-to-left” and “left-to-right” mode. In order to use a different non-UTF-8 TrueType or Type1 font, you will need to generate a font definition file and, if the font will be embedded into PDFs, a compressed version of the font file. This is done by calling the MakeFont function or using the included makefont command line utility. To create the utility, cd into the makefont subdirectory and run “go build”. This will produce a standalone executable named makefont. Select the appropriate encoding file from the font subdirectory and run the command as in the following example. In your PDF generation code, call AddFont() to load the font and, as with the standard fonts, SetFont() to begin using it. Most examples, including the package example, demonstrate this method. Good sources of free, open-source fonts include Google Fonts and DejaVu Fonts. The draw2d package is a two dimensional vector graphics library that can generate output in different forms. It uses gofpdf for its document production mode. gofpdf is a global community effort and you are invited to make it even better. If you have implemented a new feature or corrected a problem, please consider contributing your change to the project. A contribution that does not directly pertain to the core functionality of gofpdf should be placed in its own directory directly beneath the contrib directory. Here are guidelines for making submissions. Your change should - be compatible with the MIT License - be properly documented - be formatted with go fmt - include an example in fpdf_test.go if appropriate - conform to the standards of golint and go vet, that is, golint . and go vet . should not generate any warnings - not diminish test coverage Pull requests are the preferred means of accepting your changes. gofpdf is released under the MIT License. It is copyrighted by Kurt Jung and the contributors acknowledged below. This package’s code and documentation are closely derived from the FPDF library created by Olivier Plathey, and a number of font and image resources are copied directly from it. Bruno Michel has provided valuable assistance with the code. Drawing support is adapted from the FPDF geometric figures script by David Hernández Sanz. Transparency support is adapted from the FPDF transparency script by Martin Hall-May. Support for gradients and clipping is adapted from FPDF scripts by Andreas Würmser. Support for outline bookmarks is adapted from Olivier Plathey by Manuel Cornes. Layer support is adapted from Olivier Plathey. Support for transformations is adapted from the FPDF transformation script by Moritz Wagner and Andreas Würmser. PDF protection is adapted from the work of Klemen Vodopivec for the FPDF product. Lawrence Kesteloot provided code to allow an image’s extent to be determined prior to placement. Support for vertical alignment within a cell was provided by Stefan Schroeder. Ivan Daniluk generalized the font and image loading code to use the Reader interface while maintaining backward compatibility. Anthony Starks provided code for the Polygon function. Robert Lillack provided the Beziergon function and corrected some naming issues with the internal curve function. Claudio Felber provided implementations for dashed line drawing and generalized font loading. Stani Michiels provided support for multi-segment path drawing with smooth line joins, line join styles, enhanced fill modes, and has helped greatly with package presentation and tests. Templating is adapted by Marcus Downing from the FPDF_Tpl library created by Jan Slabon and Setasign. Jelmer Snoeck contributed packages that generate a variety of barcodes and help with registering images on the web. Jelmer Snoek and Guillermo Pascual augmented the basic HTML functionality with aligned text. Kent Quirk implemented backwards-compatible support for reading DPI from images that support it, and for setting DPI manually and then having it properly taken into account when calculating image size. Paulo Coutinho provided support for static embedded fonts. Dan Meyers added support for embedded JavaScript. David Fish added a generic alias-replacement function to enable, among other things, table of contents functionality. Andy Bakun identified and corrected a problem in which the internal catalogs were not sorted stably. Paul Montag added encoding and decoding functionality for templates, including images that are embedded in templates; this allows templates to be stored independently of gofpdf. Paul also added support for page boxes used in printing PDF documents. Wojciech Matusiak added supported for word spacing. Artem Korotkiy added support of UTF-8 fonts. Dave Barnes added support for imported objects and templates. Brigham Thompson added support for rounded rectangles. Joe Westcott added underline functionality and optimized image storage. Benoit KUGLER contributed support for rectangles with corners of unequal radius, modification times, and for file attachments and annotations. - Remove all legacy code page font support; use UTF-8 exclusively - Improve test coverage as reported by the coverage tool. Example demonstrates the generation of a simple PDF document. Note that since only core fonts are used (in this case Arial, a synonym for Helvetica), an empty string can be specified for the font directory in the call to New(). Note also that the example.Filename() and example.Summary() functions belong to a separate, internal package and are not part of the gofpdf library. If an error occurs at some point during the construction of the document, subsequent method calls exit immediately and the error is finally retrieved with the output call where it can be handled by the application.
Goyacc is a version of yacc generating Go parsers. Note: If no non flag arguments are given, goyacc reads standard input. 2018-03-23: The new option -pool enables using sync.Pool to recycle parser stacks. 2017-08-01: New option -fs emits a table of the follow sets. Index is the state number. 2016-03-17: Error messages now use the last token literal string, if any, to produce nicer text like "unexpected integer constant". If using xerrors the message could be, for example, something like "unexpected integer constant, expected '{'"- 2015-03-24: The search for a custom error message is now extended to include also the last state that was shifted into, if any. This change resolves a problem in which a lookahead symbol is valid for a reduce action in state A, but the same symbol is later never accepted by any shift action in some state B which is popped from the state stack after the reduction is performed. The computed from example state is A but when the error is actually detected, the state is now B and the custom error was thus not used. 2015-02-23: Added -xegen flag. It can be used to automagically generate a skeleton errors by example file which can be, for example, edited and/or submited later as an argument of the -xe option. 2014-12-18: Support %precedence for better bison compatibility[3]. The actual changes are in packages goyacc is dependent on. Goyacc users should rebuild the binary: 2014-12-02: Added support for the optional yyLexerEx interface. The Reduced method can be useful for debugging and/or automatically producing examples by parsing code fragments. If it returns true the parser exits immediately with return value -1. The generated parser is reentrant and mostly backwards compatible with parsers generated by go tool yacc[0]. yyParse expects to be given an argument that conforms to the following interface: Optionally the argument to yyParse may implement the following interface: Lex should return the token identifier, and place other token information in lval (which replaces the usual yylval). Error is equivalent to yyerror in the original yacc. Code inside the parser may refer to the variable yylex, which holds the yyLexer passed to Parse. Multiple grammars compiled into a single program should be placed in distinct packages. If that is impossible, the "-p prefix" flag to yacc sets the prefix, by default yy, that begins the names of symbols, including types, the parser, and the lexer, generated and referenced by yacc's generated code. Setting it to distinct values allows multiple grammars to be placed in a single package. - goyacc implements ideas from "Generating LR Syntax Error Messages from Examples"[1]. Use the -xe flag to pass a name of the example file. For more details about the example format please see [2]. - The grammar report includes example token sequences leading to the particular state. Can help understanding conflicts. - Minor changes in parser debug output.
Package protobuf implements Protocol Buffers reflectively using Go types to define message formats. This approach provides convenience similar to Gob encoding, but with a widely-used and language-neutral wire format. For general information on Protocol buffers see http://protobuf.googlecode.com. In contrast with goprotobuf, this package does not require users to write or compile .proto files; you just define the message formats you want as Go struct types. Consider for example this example message format definition from the Protocol Buffers overview: The following Go type and const definitions express exactly the same format, for purposes of encoding and decoding with this protobuf package: To encode a message, you simply call the Encode() function with a pointer to the struct you wish to encode, and Encode() returns a []byte slice containing the protobuf-encoded struct: To decode an encoded message, simply call Decode() on the byte-slice: If you want to interoperate with code in other languages using the same message formats, you may of course still end up writing .proto files for the code in those other languages. However, defining message formats with native Go types enables these types to be tailored to the code using them without affecting wire compatibility, such as by attaching useful methods to these struct types. The translation between a Go struct definition and a basic Protocol Buffers message format definition is straightforward; the rules are as follows. A message definition in a .proto file translates to a Go struct, whose fields are implicitly assigned consecutive numbers starting from 1. If you need to leave gaps in the field number sequence (e.g., to delete an obsolete field without breaking wire compatibility), then you can skip that field number using a blank Go field, like this: A 'required' protobuf field translates to a plain field of a corresponding type in the Go struct. The following table summarizes the correspondence between .proto definition types and Go field types: An 'optional' protobuf field is expressed as a pointer field in Go. Encode() will transmit the field only if the pointer is non-nil. Decode() will instantiate the pointed-to type and fill in the pointer if the field is present in the message being decoded, leaving the pointer unmodified (usually nil) if the field is not present. A 'repeated' protobuf field translates to a slice field in Go. Slices of primitive bool, integer, and float types are encoded and decoded in packed format, as if the [packed=true] option was declared for the field in the .proto file. For flexibility and convenience, struct fields may have interface types, which this package interprets as having dynamic types to be bound at runtime. Encode() follows the interface's implicit pointer and uses reflection to determine the referred-to object's actual type for encoding Decode() takes an optional map of interface types to constructor functions, which it uses to instantiate concrete types for interfaces while decoding. Furthermore, if the instantiated types support the Encoding interface, Encode() and Decode() will invoke the methods of that interface, allowing objects to implement their own custom encoding/decoding methods. This package does not try to support all possible protobuf formats. It currently does not support nonzero default value declarations for enums, the legacy unpacked formats for repeated numeric fields, messages with extremely sparse field numbering, or other more exotic features like extensions or oneof. If you need to interoperate with existing protobuf code using these features, then you should probably use goprotobuf, at least for those particular message formats. Another downside of this reflective approach to protobuf implementation is that reflective code is generally less efficient than statically generated code, as gogoprotobuf produces for example. If we decide we want the convenience of format definitions in Go with the runtime performance of static code generation, we could in principle achieve that by adding a "Go-format" message format compiler frontend to goprotobuf or gogoprotobuf - but we leave this as an exercise for the reader. This example defines, encodes, and decodes a Person message format equivalent to the example used in the Protocol Buffers overview.
Package protobuf implements Protocol Buffers reflectively using Go types to define message formats. This approach provides convenience similar to Gob encoding, but with a widely-used and language-neutral wire format. For general information on Protocol buffers see http://protobuf.googlecode.com. In contrast with goprotobuf, this package does not require users to write or compile .proto files; you just define the message formats you want as Go struct types. Consider for example this example message format definition from the Protocol Buffers overview: The following Go type and const definitions express exactly the same format, for purposes of encoding and decoding with this protobuf package: To encode a message, you simply call the Encode() function with a pointer to the struct you wish to encode, and Encode() returns a []byte slice containing the protobuf-encoded struct: To decode an encoded message, simply call Decode() on the byte-slice: If you want to interoperate with code in other languages using the same message formats, you may of course still end up writing .proto files for the code in those other languages. However, defining message formats with native Go types enables these types to be tailored to the code using them without affecting wire compatibility, such as by attaching useful methods to these struct types. The translation between a Go struct definition and a basic Protocol Buffers message format definition is straightforward; the rules are as follows. A message definition in a .proto file translates to a Go struct, whose fields are implicitly assigned consecutive numbers starting from 1. If you need to leave gaps in the field number sequence (e.g., to delete an obsolete field without breaking wire compatibility), then you can skip that field number using a blank Go field, like this: A 'required' protobuf field translates to a plain field of a corresponding type in the Go struct. The following table summarizes the correspondence between .proto definition types and Go field types: An 'optional' protobuf field is expressed as a pointer field in Go. Encode() will transmit the field only if the pointer is non-nil. Decode() will instantiate the pointed-to type and fill in the pointer if the field is present in the message being decoded, leaving the pointer unmodified (usually nil) if the field is not present. A 'repeated' protobuf field translates to a slice field in Go. Slices of primitive bool, integer, and float types are encoded and decoded in packed format, as if the [packed=true] option was declared for the field in the .proto file. For flexibility and convenience, struct fields may have interface types, which this package interprets as having dynamic types to be bound at runtime. Encode() follows the interface's implicit pointer and uses reflection to determine the referred-to object's actual type for encoding Decode() takes an optional map of interface types to constructor functions, which it uses to instantiate concrete types for interfaces while decoding. Furthermore, if the instantiated types support the Encoding interface, Encode() and Decode() will invoke the methods of that interface, allowing objects to implement their own custom encoding/decoding methods. This package does not try to support all possible protobuf formats. It currently does not support nonzero default value declarations for enums, the legacy unpacked formats for repeated numeric fields, messages with extremely sparse field numbering, or other more exotic features like extensions or oneof. If you need to interoperate with existing protobuf code using these features, then you should probably use goprotobuf, at least for those particular message formats. Another downside of this reflective approach to protobuf implementation is that reflective code is generally less efficient than statically generated code, as gogoprotobuf produces for example. If we decide we want the convenience of format definitions in Go with the runtime performance of static code generation, we could in principle achieve that by adding a "Go-format" message format compiler frontend to goprotobuf or gogoprotobuf - but we leave this as an exercise for the reader. This example defines, encodes, and decodes a Person message format equivalent to the example used in the Protocol Buffers overview.
Package gcfg reads "INI-style" text-based configuration files with "name=value" pairs grouped into sections (gcfg files). This package is still a work in progress; see the sections below for planned changes. The syntax is based on that used by git config: http://git-scm.com/docs/git-config#_syntax . There are some (planned) differences compared to the git config format: The functions in this package read values into a user-defined struct. Each section corresponds to a struct field in the config struct, and each variable in a section corresponds to a data field in the section struct. The mapping of each section or variable name to fields is done either based on the "gcfg" struct tag or by matching the name of the section or variable, ignoring case. In the latter case, hyphens '-' in section and variable names correspond to underscores '_' in field names. Fields must be exported; to use a section or variable name starting with a letter that is neither upper- or lower-case, prefix the field name with 'X'. (See https://code.google.com/p/go/issues/detail?id=5763#c4 .) For sections with subsections, the corresponding field in config must be a map, rather than a struct, with string keys and pointer-to-struct values. Values for subsection variables are stored in the map with the subsection name used as the map key. (Note that unlike section and variable names, subsection names are case sensitive.) When using a map, and there is a section with the same section name but without a subsection name, its values are stored with the empty string used as the key. The functions in this package panic if config is not a pointer to a struct, or when a field is not of a suitable type (either a struct or a map with string keys and pointer-to-struct values). The section structs in the config struct may contain single-valued or multi-valued variables. Variables of unnamed slice type (that is, a type starting with `[]`) are treated as multi-value; all others (including named slice types) are treated as single-valued variables. Single-valued variables are handled based on the type as follows. Unnamed pointer types (that is, types starting with `*`) are dereferenced, and if necessary, a new instance is allocated. For types implementing the encoding.TextUnmarshaler interface, the UnmarshalText method is used to set the value. Implementing this method is the recommended way for parsing user-defined types. For fields of string kind, the value string is assigned to the field, after unquoting and unescaping as needed. For fields of bool kind, the field is set to true if the value is "true", "yes", "on" or "1", and set to false if the value is "false", "no", "off" or "0", ignoring case. In addition, single-valued bool fields can be specified with a "blank" value (variable name without equals sign and value); in such case the value is set to true. Predefined integer types [u]int(|8|16|32|64) and big.Int are parsed as decimal or hexadecimal (if having '0x' prefix). (This is to prevent unintuitively handling zero-padded numbers as octal.) Other types having [u]int* as the underlying type, such as os.FileMode and uintptr allow decimal, hexadecimal, or octal values. Parsing mode for integer types can be overridden using the struct tag option ",int=mode" where mode is a combination of the 'd', 'h', and 'o' characters (each standing for decimal, hexadecimal, and octal, respectively.) All other types are parsed using fmt.Sscanf with the "%v" verb. For multi-valued variables, each individual value is parsed as above and appended to the slice. If the first value is specified as a "blank" value (variable name without equals sign and value), a new slice is allocated; that is any values previously set in the slice will be ignored. The types subpackage for provides helpers for parsing "enum-like" and integer types. The following is a list of changes under consideration:
Package protobuf implements Protocol Buffers reflectively using Go types to define message formats. This approach provides convenience similar to Gob encoding, but with a widely-used and language-neutral wire format. For general information on Protocol buffers see http://protobuf.googlecode.com. In contrast with goprotobuf, this package does not require users to write or compile .proto files; you just define the message formats you want as Go struct types. Consider for example this example message format definition from the Protocol Buffers overview: The following Go type and const definitions express exactly the same format, for purposes of encoding and decoding with this protobuf package: To encode a message, you simply call the Encode() function with a pointer to the struct you wish to encode, and Encode() returns a []byte slice containing the protobuf-encoded struct: To decode an encoded message, simply call Decode() on the byte-slice: If you want to interoperate with code in other languages using the same message formats, you may of course still end up writing .proto files for the code in those other languages. However, defining message formats with native Go types enables these types to be tailored to the code using them without affecting wire compatibility, such as by attaching useful methods to these struct types. The translation between a Go struct definition and a basic Protocol Buffers message format definition is straightforward; the rules are as follows. A message definition in a .proto file translates to a Go struct, whose fields are implicitly assigned consecutive numbers starting from 1. If you need to leave gaps in the field number sequence (e.g., to delete an obsolete field without breaking wire compatibility), then you can skip that field number using a blank Go field, like this: A 'required' protobuf field translates to a plain field of a corresponding type in the Go struct. The following table summarizes the correspondence between .proto definition types and Go field types: An 'optional' protobuf field is expressed as a pointer field in Go. Encode() will transmit the field only if the pointer is non-nil. Decode() will instantiate the pointed-to type and fill in the pointer if the field is present in the message being decoded, leaving the pointer unmodified (usually nil) if the field is not present. A 'repeated' protobuf field translates to a slice field in Go. Slices of primitive bool, integer, and float types are encoded and decoded in packed format, as if the [packed=true] option was declared for the field in the .proto file. For flexibility and convenience, struct fields may have interface types, which this package interprets as having dynamic types to be bound at runtime. Encode() follows the interface's implicit pointer and uses reflection to determine the referred-to object's actual type for encoding Decode() takes an optional map of interface types to constructor functions, which it uses to instantiate concrete types for interfaces while decoding. Furthermore, if the instantiated types support the Encoding interface, Encode() and Decode() will invoke the methods of that interface, allowing objects to implement their own custom encoding/decoding methods. This package does not try to support all possible protobuf formats. It currently does not support nonzero default value declarations for enums, the legacy unpacked formats for repeated numeric fields, messages with extremely sparse field numbering, or other more exotic features like extensions or oneof. If you need to interoperate with existing protobuf code using these features, then you should probably use goprotobuf, at least for those particular message formats. Another downside of this reflective approach to protobuf implementation is that reflective code is generally less efficient than statically generated code, as gogoprotobuf produces for example. If we decide we want the convenience of format definitions in Go with the runtime performance of static code generation, we could in principle achieve that by adding a "Go-format" message format compiler frontend to goprotobuf or gogoprotobuf - but we leave this as an exercise for the reader. This example defines, encodes, and decodes a Person message format equivalent to the example used in the Protocol Buffers overview.
Package spreak provides a simple translation facility based on the concepts of gettext. Domain: A message domain is a set of translatable messages. Usually, every software package has its own message domain. The domain name is used to determine the message catalog where the translation is looked up. Default domain: The default domain is used if a domain is not explicitly specified for a requested translation. If no default domain is specified, the default domain of the bundle is used. If this was not specified either, the domain is NoDomain (an empty string). Context: Context can be added to strings to be translated. A context dependent translation lookup is when a translation for a given string is searched, that is limited to a given context. The translation for the same string in a different context can be different. The different translations of the same string in different contexts can be stored in the same MO file, and can be edited by the translator in the same PO file. The Context string is visible in the PO file to the translator. You should try to make it somehow canonical and never changing. Because every time you change an Context, the translator will have to review the translation of msgid. For JSON files only the CLDR plural rules are supported. For po and mo files both gettext plural forms and CLDR plural rules are supported. The CLDR rules provides better support when floating point numbers are used. When using the CLDR plural rules with po files, a notation increasing from "Zero" to "Other" should be used. For example, if the used language supports "Zero", "Few" and "Other", Zero should be notated as entry 0, Few as entry 1 and Other as entry 2. It is also recommended to define a gettext compatible plural rule. On the website https://php-gettext.github.io/Languages/ you can find a list of gettext plural rules which are compatible to the CLDR plural rules. To use the CLDR rules in po/mo files you can either add a header "X-spreak-use-CLDR: true" or create a decoder with catalog.NewPoCLDRDecoder() / catalog.NewMoCLDRDecoder(). For Polish with One, Few and Other, the structure of a Po file according to this convention could look like this: If floating point numbers are used, it is recommended to pass them formatted as strings as they will be displayed later. For example, if the number n is to be displayed with two numbers after the decimal point, it should be formatted with fmt.Sprintf("%.2f", n).
Package log15 provides an opinionated, simple toolkit for best-practice logging that is both human and machine readable. It is modeled after the standard library's io and net/http packages. This package enforces you to only log key/value pairs. Keys must be strings. Values may be any type that you like. The default output format is logfmt, but you may also choose to use JSON instead if that suits you. Here's how you log: This will output a line that looks like: To get started, you'll want to import the library: Now you're ready to start logging: Because recording a human-meaningful message is common and good practice, the first argument to every logging method is the value to the *implicit* key 'msg'. Additionally, the level you choose for a message will be automatically added with the key 'lvl', and so will the current timestamp with key 't'. You may supply any additional context as a set of key/value pairs to the logging function. log15 allows you to favor terseness, ordering, and speed over safety. This is a reasonable tradeoff for logging functions. You don't need to explicitly state keys/values, log15 understands that they alternate in the variadic argument list: If you really do favor your type-safety, you may choose to pass a log.Ctx instead: Frequently, you want to add context to a logger so that you can track actions associated with it. An http request is a good example. You can easily create new loggers that have context that is automatically included with each log line: This will output a log line that includes the path context that is attached to the logger: The Handler interface defines where log lines are printed to and how they are formated. Handler is a single interface that is inspired by net/http's handler interface: Handlers can filter records, format them, or dispatch to multiple other Handlers. This package implements a number of Handlers for common logging patterns that are easily composed to create flexible, custom logging structures. Here's an example handler that prints logfmt output to Stdout: Here's an example handler that defers to two other handlers. One handler only prints records from the rpc package in logfmt to standard out. The other prints records at Error level or above in JSON formatted output to the file /var/log/service.json This package implements three Handlers that add debugging information to the context, CallerFileHandler, CallerFuncHandler and CallerStackHandler. Here's an example that adds the source file and line number of each logging call to the context. This will output a line that looks like: Here's an example that logs the call stack rather than just the call site. This will output a line that looks like: The "%+v" format instructs the handler to include the path of the source file relative to the compile time GOPATH. The github.com/go-stack/stack package documents the full list of formatting verbs and modifiers available. The Handler interface is so simple that it's also trivial to write your own. Let's create an example handler which tries to write to one handler, but if that fails it falls back to writing to another handler and includes the error that it encountered when trying to write to the primary. This might be useful when trying to log over a network socket, but if that fails you want to log those records to a file on disk. This pattern is so useful that a generic version that handles an arbitrary number of Handlers is included as part of this library called FailoverHandler. Sometimes, you want to log values that are extremely expensive to compute, but you don't want to pay the price of computing them if you haven't turned up your logging level to a high level of detail. This package provides a simple type to annotate a logging operation that you want to be evaluated lazily, just when it is about to be logged, so that it would not be evaluated if an upstream Handler filters it out. Just wrap any function which takes no arguments with the log.Lazy type. For example: If this message is not logged for any reason (like logging at the Error level), then factorRSAKey is never evaluated. The same log.Lazy mechanism can be used to attach context to a logger which you want to be evaluated when the message is logged, but not when the logger is created. For example, let's imagine a game where you have Player objects: You always want to log a player's name and whether they're alive or dead, so when you create the player object, you might do: Only now, even after a player has died, the logger will still report they are alive because the logging context is evaluated when the logger was created. By using the Lazy wrapper, we can defer the evaluation of whether the player is alive or not to each log message, so that the log records will reflect the player's current state no matter when the log message is written: If log15 detects that stdout is a terminal, it will configure the default handler for it (which is log.StdoutHandler) to use TerminalFormat. This format logs records nicely for your terminal, including color-coded output based on log level. Becasuse log15 allows you to step around the type system, there are a few ways you can specify invalid arguments to the logging functions. You could, for example, wrap something that is not a zero-argument function with log.Lazy or pass a context key that is not a string. Since logging libraries are typically the mechanism by which errors are reported, it would be onerous for the logging functions to return errors. Instead, log15 handles errors by making these guarantees to you: - Any log record containing an error will still be printed with the error explained to you as part of the log record. - Any log record containing an error will include the context key LOG15_ERROR, enabling you to easily (and if you like, automatically) detect if any of your logging calls are passing bad values. Understanding this, you might wonder why the Handler interface can return an error value in its Log method. Handlers are encouraged to return errors only if they fail to write their log records out to an external source like if the syslog daemon is not responding. This allows the construction of useful handlers which cope with those failures like the FailoverHandler. log15 is intended to be useful for library authors as a way to provide configurable logging to users of their library. Best practice for use in a library is to always disable all output for your logger by default and to provide a public Logger instance that consumers of your library can configure. Like so: Users of your library may then enable it if they like: The ability to attach context to a logger is a powerful one. Where should you do it and why? I favor embedding a Logger directly into any persistent object in my application and adding unique, tracing context keys to it. For instance, imagine I am writing a web browser: When a new tab is created, I assign a logger to it with the url of the tab as context so it can easily be traced through the logs. Now, whenever we perform any operation with the tab, we'll log with its embedded logger and it will include the tab title automatically: There's only one problem. What if the tab url changes? We could use log.Lazy to make sure the current url is always written, but that would mean that we couldn't trace a tab's full lifetime through our logs after the user navigate to a new URL. Instead, think about what values to attach to your loggers the same way you think about what to use as a key in a SQL database schema. If it's possible to use a natural key that is unique for the lifetime of the object, do so. But otherwise, log15's ext package has a handy RandId function to let you generate what you might call "surrogate keys" They're just random hex identifiers to use for tracing. Back to our Tab example, we would prefer to set up our Logger like so: Now we'll have a unique traceable identifier even across loading new urls, but we'll still be able to see the tab's current url in the log messages. For all Handler functions which can return an error, there is a version of that function which will return no error but panics on failure. They are all available on the Must object. For example: All of the following excellent projects inspired the design of this library: code.google.com/p/log4go github.com/op/go-logging github.com/technoweenie/grohl github.com/Sirupsen/logrus github.com/kr/logfmt github.com/spacemonkeygo/spacelog golang's stdlib, notably io and net/http https://xkcd.com/927/
Package promptui is a library providing a simple interface to create command-line prompts for go. It can be easily integrated into spf13/cobra, urfave/cli or any cli go application. promptui has two main input modes: Prompt provides a single line for user input. It supports optional live validation, confirmation and masking the input. Select provides a list of options to choose from. It supports pagination, search, detailed view and custom templates. This is an example for the Prompt mode of promptui. In this example, a prompt is created with a validator function that validates the given value to make sure its a number. If successful, it will output the chosen number in a formatted message. This is an example for the Select mode of promptui. In this example, a select is created with the days of the week as its items. When an item is selected, the selected day will be displayed in a formatted message.
Command goat provides an implementation of a BitTorrent tracker, written in Go. goat can be built using Go 1.1+. It can be downloaded, built, and installed, simply by running: In addition, goat depends on a MySQL server for data storage. After creating a database and user for goat, its database schema may be imported from the SQL files located in 'res/'. goat will not run unless MySQL is installed, and a database and user are properly configured for its use. Optionally, goat can be built to use ql (https://github.com/cznic/ql) as its storage backend. This is done by supplying the 'ql' tag in the go get command: A blank ql database file is located under 'res/ql/goat.db', and will be copied to '~/.config/goat/goat.db' on UNIX systems. goat is now able to use ql as its storage backend, for those who do not wish to use an external, MySQL backend. goat is capable of listening for torrent traffic in three modes: HTTP, HTTPS, and UDP. HTTP/HTTPS are the recommended methods, and are required in order for goat to serve its API, and to allow use of private tracker passkeys. HTTP is considered the standard mode of operation for goat. HTTP allows gathering a great number of metrics, use of passkeys, use of a client whitelist, and access to goat's RESTful API, when configured. For most trackers, this will be the only listener which is necessary in order for goat to function properly. The HTTPS listener provides a method to encrypt traffic to the tracker, but must be used with caution. Unless the SSL certificate in use is signed by a proper certificate authority, it will distress most clients, and they may outright refuse to announce to it. If you are in possession of a certificate signed by a certificate authority, this mode may be more ideal, as it provides added security for your clients. The UDP listener is the most unusual method of the three, and should only be used for public trackers. The BitTorrent UDP tracker protocol specifies a very specific packet format, meaning that additional information or parameters cannot be packed into a UDP datagram in a standard way. The UDP tracker may be the fastest and least bandwidth-intensive, but as stated, should only be used for public trackers. A new feature goat added to goat in order to allow better interoperability with many languages is a RESTful API, which is served using the HTTP or HTTPS listeners. This API enables easy retrieval of tracker statistics, while allowing goat to run as a completely independent process. It should be noted that the API is only enabled when configured, and when a HTTP or HTTPS listener is enabled. Without a transport mechanism, the API will be inaccessible. The API features several modes of authentication, including HTTP Basic for login and HMAC-SHA1 other calls. Upon logging into the API using HTTP Basic with a username and password pair, an API public key and secret will be generated. The public key is used as the username for HTTP Basic authentication, and the secret key is used to calculate a HMAC-SHA1 signature for the password. As part of API signature generation, a random nonce value must be generated and added to the request. It is added to the password portion of the HTTP Basic request, and also to the string which is used to create the signature. Nonce values must be changed on every request, or the request will fail. The current pseudocode format of the HMAC-SHA1 signature is as follows: The proper format for a HTTP Basic request is as follows: When the public key, nonce, and API signature are sent via HTTP Basic, the server will verify the signature. Successful authentication will allow access to the API. This list contains all API calls currently recognized by goat. Each call must be authenticated using the aforementioned methods. Request an API public key and secret key for this user. The public key, user ID, and secret key are used to authenticate further API calls. The expire time indicates when this key is set to expire. Further API calls will extend the expiration time. Retrieve a list of all files tracked by goat. Some extended attributes are not added to reduce strain on database, and to provide a more general overview. Retrieve extended attributes about a specific file with matching ID. This provides counts for number of completions, seeders, leechers, and a list of fileUser relationships associated with a given file. Retrieve a variety of metrics about the current status of goat, including its PID, hostname, memory usage, number of HTTP/UDP hits, etc. Create a user with the specified username, password, and torrent limit. Reterieve a list of all users registered to goat, including their ID, torrent limit, and username. Retrieve information about a single user with matching ID, including their ID, torrent limit, and username. goat is configured using a JSON file, which will be created under '~/.config/goat/config.json' on UNIX systems. Here is an example configuration, describing the settings available to the user.
Package goBolt implements drivers for the Neo4J Bolt Protocol Versions 1-4. There are some limitations to the types of collections the internalDriver supports. Specifically, maps should always be of type map[string]interface{} and lists should always be of type []interface{}. It doesn't seem that the Bolt protocol supports uint64 either, so the biggest number it can send right now is the int64 max. The URL format is: `bolt://(user):(password)@(host):(port)` Schema must be `bolt`. User and password is only necessary if you are authenticating. TLS is supported by using query parameters on the connection string, like so: `bolt://host:port?tls=true&tls_no_verify=false` The supported query params are: * timeout - the number of seconds to set the connection timeout to. Defaults to 60 seconds. * tls - Set to 'true' or '1' if you want to use TLS encryption * tls_no_verify - Set to 'true' or '1' if you want to accept any server certificate (for testing, not secure) * tls_ca_cert_file - path to a custom ca cert for a self-signed TLS cert * tls_cert_file - path to a cert file for this client (need to verify this is processed by Neo4j) * tls_key_file - path to a key file for this client (need to verify this is processed by Neo4j) Errors returned from the API support wrapping, so if you receive an error from the library, it might be wrapping other errors. You can get the innermost error by using the `InnerMost` method. Failure messages from Neo4J are reported, along with their metadata, as an error. In order to get the failure message metadata from a wrapped error, you can do so by calling `err.(*errors.Error).InnerMost().(messages.FailureMessage).Metadata` If there is an error with the database connection, you should get a sql/internalDriver ErrBadConn as per the best practice recommendations of the Golang SQL Driver. However, this error may be wrapped, so you might have to call `InnerMost` to get it, as specified above.
Package peer provides a common base for creating and managing Bitcoin network peers. This package builds upon the wire package, which provides the fundamental primitives necessary to speak the bitcoin wire protocol, in order to simplify the process of creating fully functional peers. In essence, it provides a common base for creating concurrent safe fully validating nodes, Simplified Payment Verification (SPV) nodes, proxies, etc. A quick overview of the major features peer provides are as follows: Provides a basic concurrent safe bitcoin peer for handling bitcoin communications via the peer-to-peer protocol Full duplex reading and writing of bitcoin protocol messages Automatic handling of the initial handshake process including protocol version negotiation Asynchronous message queuing of outbound messages with optional channel for notification when the message is actually sent Flexible peer configuration Caller is responsible for creating outgoing connections and listening for incoming connections so they have flexibility to establish connections asthey see fit (proxies, etc) User agent name and version Bitcoin network Service support signalling (full nodes, bloom filters, etc) Maximum supported protocol version Ability to register callbacks for handling bitcoin protocol messages Inventory message batching and send trickling with known inventory detection and avoidance Automatic periodic keep-alive pinging and pong responses Random Nonce generation and self connection detection Proper handling of bloom filter related commands when the caller does not specify the related flag to signal support Disconnects the peer when the protocol version is high enough Does not invoke the related callbacks for older protocol versions Snapshottable peer statistics such as the total number of bytes read and written, the remote address, user agent, and negotiated protocol version Helper functions pushing addresses, getblocks, getheaders, and reject messages These could all be sent manually via the standard message output function, but the helpers provide additional nice functionality such as duplicate filtering and address randomization Ability to wait for shutdown/disconnect Comprehensive test coverage All peer configuration is handled with the Config struct. This allows the caller to specify things such as the user agent name and version, the bitcoin network to use, which services it supports, and callbacks to invoke when bitcoin messages are received. See the documentation for each field of the Config struct for more details. A peer can either be inbound or outbound. The caller is responsible for establishing the connection to remote peers and listening for incoming peers. This provides high flexibility for things such as connecting via proxies, acting as a proxy, creating bridge peers, choosing whether to listen for inbound peers, etc. NewOutboundPeer and NewInboundPeer functions must be followed by calling Connect with a net.Conn instance to the peer. This will start all async I/O goroutines and initiate the protocol negotiation process. Once finished with the peer call Disconnect to disconnect from the peer and clean up all resources. WaitForDisconnect can be used to block until peer disconnection and resource cleanup has completed. In order to do anything useful with a peer, it is necessary to react to bitcoin messages. This is accomplished by creating an instance of the MessageListeners struct with the callbacks to be invoke specified and setting the Listeners field of the Config struct specified when creating a peer to it. For convenience, a callback hook for all of the currently supported bitcoin messages is exposed which receives the peer instance and the concrete message type. In addition, a hook for OnRead is provided so even custom messages types for which this package does not directly provide a hook, as long as they implement the wire.Message interface, can be used. Finally, the OnWrite hook is provided, which in conjunction with OnRead, can be used to track server-wide byte counts. It is often useful to use closures which encapsulate state when specifying the callback handlers. This provides a clean method for accessing that state when callbacks are invoked. The QueueMessage function provides the fundamental means to send messages to the remote peer. As the name implies, this employs a non-blocking queue. A done channel which will be notified when the message is actually sent can optionally be specified. There are certain message types which are better sent using other functions which provide additional functionality. Of special interest are inventory messages. Rather than manually sending MsgInv messages via Queuemessage, the inventory vectors should be queued using the QueueInventory function. It employs batching and trickling along with intelligent known remote peer inventory detection and avoidance through the use of a most-recently used algorithm. In addition to the bare QueueMessage function previously described, the PushAddrMsg, PushGetBlocksMsg, PushGetHeadersMsg, and PushRejectMsg functions are provided as a convenience. While it is of course possible to create and send these message manually via QueueMessage, these helper functions provided additional useful functionality that is typically desired. For example, the PushAddrMsg function automatically limits the addresses to the maximum number allowed by the message and randomizes the chosen addresses when there are too many. This allows the caller to simply provide a slice of known addresses, such as that returned by the addrmgr package, without having to worry about the details. Next, the PushGetBlocksMsg and PushGetHeadersMsg functions will construct proper messages using a block locator and ignore back to back duplicate requests. Finally, the PushRejectMsg function can be used to easily create and send an appropriate reject message based on the provided parameters as well as optionally provides a flag to cause it to block until the message is actually sent. A snapshot of the current peer statistics can be obtained with the StatsSnapshot function. This includes statistics such as the total number of bytes read and written, the remote address, user agent, and negotiated protocol version. This package provides extensive logging capabilities through the UseLogger function which allows a btclog.Logger to be specified. For example, logging at the debug level provides summaries of every message sent and received, and logging at the trace level provides full dumps of parsed messages as well as the raw message bytes using a format similar to hexdump -C. This package supports all BIPS supported by the wire package. (https://godoc.org/github.com/p9c/pod/wire#hdr-Bitcoin_Improvement_Proposals) This example demonstrates the basic process for initializing and creating an outbound peer. Peers negotiate by exchanging version and verack messages. For demonstration, a simple handler for version message is attached to the peer.
Package spew implements a deep pretty printer for Go data structures to aid in debugging. A quick overview of the additional features spew provides over the built-in printing facilities for Go data types are as follows: There are two different approaches spew allows for dumping Go data structures: This section demonstrates how to quickly get started with spew. See the sections below for further details on formatting and configuration options. To dump a variable with full newlines, indentation, type, and pointer information use Dump, Fdump, or Sdump: Alternatively, if you would prefer to use format strings with a compacted inline printing style, use the convenience wrappers Printf, Fprintf, etc with %v (most compact), %+v (adds pointer addresses), %#v (adds types), or %#+v (adds types and pointer addresses): Configuration of spew is handled by fields in the ConfigState type. For convenience, all of the top-level functions use a global state available via the spew.Config global. It is also possible to create a ConfigState instance that provides methods equivalent to the top-level functions. This allows concurrent configuration options. See the ConfigState documentation for more details. The following configuration options are available: Indent String to use for each indentation level for Dump functions. It is a single space by default. A popular alternative is "\t". MaxDepth Maximum number of levels to descend into nested data structures. There is no limit by default. DisableMethods Disables invocation of error and Stringer interface methods. Method invocation is enabled by default. DisablePointerMethods Disables invocation of error and Stringer interface methods on types which only accept pointer receivers from non-pointer variables. Pointer method invocation is enabled by default. DisablePointerAddresses DisablePointerAddresses specifies whether to disable the printing of pointer addresses. This is useful when diffing data structures in tests. DisableCapacities DisableCapacities specifies whether to disable the printing of capacities for arrays, slices, maps and channels. This is useful when diffing data structures in tests. ContinueOnMethod Enables recursion into types after invoking error and Stringer interface methods. Recursion after method invocation is disabled by default. SortKeys Specifies map keys should be sorted before being printed. Use this to have a more deterministic, diffable output. Note that only native types (bool, int, uint, floats, uintptr and string) and types which implement error or Stringer interfaces are supported with other types sorted according to the reflect.Value.String() output which guarantees display stability. Natural map order is used by default. SpewKeys Specifies that, as a last resort attempt, map keys should be spewed to strings and sorted by those strings. This is only considered if SortKeys is true. Simply call spew.Dump with a list of variables you want to dump: You may also call spew.Fdump if you would prefer to output to an arbitrary io.Writer. For example, to dump to standard error: A third option is to call spew.Sdump to get the formatted output as a string: See the Dump example for details on the setup of the types and variables being shown here. Byte (and uint8) arrays and slices are displayed uniquely like the hexdump -C command as shown. Spew provides a custom formatter that implements the fmt.Formatter interface so that it integrates cleanly with standard fmt package printing functions. The formatter is useful for inline printing of smaller data types similar to the standard %v format specifier. The custom formatter only responds to the %v (most compact), %+v (adds pointer addresses), %#v (adds types), or %#+v (adds types and pointer addresses) verb combinations. Any other verbs such as %x and %q will be sent to the the standard fmt package for formatting. In addition, the custom formatter ignores the width and precision arguments (however they will still work on the format specifiers not handled by the custom formatter). The simplest way to make use of the spew custom formatter is to call one of the convenience functions such as spew.Printf, spew.Println, or spew.Printf. The functions have syntax you are most likely already familiar with: See the Index for the full list convenience functions. Double pointer to a uint8: Pointer to circular struct with a uint8 field and a pointer to itself: See the Printf example for details on the setup of variables being shown here. Since it is possible for custom Stringer/error interfaces to panic, spew detects them and handles them internally by printing the panic information inline with the output. Since spew is intended to provide deep pretty printing capabilities on structures, it intentionally does not return any errors.
Package log15 provides an opinionated, simple toolkit for best-practice logging that is both human and machine readable. It is modeled after the standard library's io and net/http packages. This package enforces you to only log key/value pairs. Keys must be strings. Values may be any type that you like. The default output format is logfmt, but you may also choose to use JSON instead if that suits you. Here's how you log: This will output a line that looks like: To get started, you'll want to import the library: Now you're ready to start logging: Because recording a human-meaningful message is common and good practice, the first argument to every logging method is the value to the *implicit* key 'msg'. Additionally, the level you choose for a message will be automatically added with the key 'lvl', and so will the current timestamp with key 't'. You may supply any additional context as a set of key/value pairs to the logging function. log15 allows you to favor terseness, ordering, and speed over safety. This is a reasonable tradeoff for logging functions. You don't need to explicitly state keys/values, log15 understands that they alternate in the variadic argument list: If you really do favor your type-safety, you may choose to pass a log.Ctx instead: Frequently, you want to add context to a logger so that you can track actions associated with it. An http request is a good example. You can easily create new loggers that have context that is automatically included with each log line: This will output a log line that includes the path context that is attached to the logger: The Handler interface defines where log lines are printed to and how they are formated. Handler is a single interface that is inspired by net/http's handler interface: Handlers can filter records, format them, or dispatch to multiple other Handlers. This package implements a number of Handlers for common logging patterns that are easily composed to create flexible, custom logging structures. Here's an example handler that prints logfmt output to Stdout: Here's an example handler that defers to two other handlers. One handler only prints records from the rpc package in logfmt to standard out. The other prints records at Error level or above in JSON formatted output to the file /var/log/service.json This package implements three Handlers that add debugging information to the context, CallerFileHandler, CallerFuncHandler and CallerStackHandler. Here's an example that adds the source file and line number of each logging call to the context. This will output a line that looks like: Here's an example that logs the call stack rather than just the call site. This will output a line that looks like: The "%+v" format instructs the handler to include the path of the source file relative to the compile time GOPATH. The github.com/go-stack/stack package documents the full list of formatting verbs and modifiers available. The Handler interface is so simple that it's also trivial to write your own. Let's create an example handler which tries to write to one handler, but if that fails it falls back to writing to another handler and includes the error that it encountered when trying to write to the primary. This might be useful when trying to log over a network socket, but if that fails you want to log those records to a file on disk. This pattern is so useful that a generic version that handles an arbitrary number of Handlers is included as part of this library called FailoverHandler. Sometimes, you want to log values that are extremely expensive to compute, but you don't want to pay the price of computing them if you haven't turned up your logging level to a high level of detail. This package provides a simple type to annotate a logging operation that you want to be evaluated lazily, just when it is about to be logged, so that it would not be evaluated if an upstream Handler filters it out. Just wrap any function which takes no arguments with the log.Lazy type. For example: If this message is not logged for any reason (like logging at the Error level), then factorRSAKey is never evaluated. The same log.Lazy mechanism can be used to attach context to a logger which you want to be evaluated when the message is logged, but not when the logger is created. For example, let's imagine a game where you have Player objects: You always want to log a player's name and whether they're alive or dead, so when you create the player object, you might do: Only now, even after a player has died, the logger will still report they are alive because the logging context is evaluated when the logger was created. By using the Lazy wrapper, we can defer the evaluation of whether the player is alive or not to each log message, so that the log records will reflect the player's current state no matter when the log message is written: If log15 detects that stdout is a terminal, it will configure the default handler for it (which is log.StdoutHandler) to use TerminalFormat. This format logs records nicely for your terminal, including color-coded output based on log level. Becasuse log15 allows you to step around the type system, there are a few ways you can specify invalid arguments to the logging functions. You could, for example, wrap something that is not a zero-argument function with log.Lazy or pass a context key that is not a string. Since logging libraries are typically the mechanism by which errors are reported, it would be onerous for the logging functions to return errors. Instead, log15 handles errors by making these guarantees to you: - Any log record containing an error will still be printed with the error explained to you as part of the log record. - Any log record containing an error will include the context key LOG15_ERROR, enabling you to easily (and if you like, automatically) detect if any of your logging calls are passing bad values. Understanding this, you might wonder why the Handler interface can return an error value in its Log method. Handlers are encouraged to return errors only if they fail to write their log records out to an external source like if the syslog daemon is not responding. This allows the construction of useful handlers which cope with those failures like the FailoverHandler. log15 is intended to be useful for library authors as a way to provide configurable logging to users of their library. Best practice for use in a library is to always disable all output for your logger by default and to provide a public Logger instance that consumers of your library can configure. Like so: Users of your library may then enable it if they like: The ability to attach context to a logger is a powerful one. Where should you do it and why? I favor embedding a Logger directly into any persistent object in my application and adding unique, tracing context keys to it. For instance, imagine I am writing a web browser: When a new tab is created, I assign a logger to it with the url of the tab as context so it can easily be traced through the logs. Now, whenever we perform any operation with the tab, we'll log with its embedded logger and it will include the tab title automatically: There's only one problem. What if the tab url changes? We could use log.Lazy to make sure the current url is always written, but that would mean that we couldn't trace a tab's full lifetime through our logs after the user navigate to a new URL. Instead, think about what values to attach to your loggers the same way you think about what to use as a key in a SQL database schema. If it's possible to use a natural key that is unique for the lifetime of the object, do so. But otherwise, log15's ext package has a handy RandId function to let you generate what you might call "surrogate keys" They're just random hex identifiers to use for tracing. Back to our Tab example, we would prefer to set up our Logger like so: Now we'll have a unique traceable identifier even across loading new urls, but we'll still be able to see the tab's current url in the log messages. For all Handler functions which can return an error, there is a version of that function which will return no error but panics on failure. They are all available on the Must object. For example: All of the following excellent projects inspired the design of this library: code.google.com/p/log4go github.com/op/go-logging github.com/technoweenie/grohl github.com/Sirupsen/logrus github.com/kr/logfmt github.com/spacemonkeygo/spacelog golang's stdlib, notably io and net/http https://xkcd.com/927/
Package ogdl is used to process OGDL, the Ordered Graph Data Language. OGDL is a textual format to write trees or graphs of text, where indentation and spaces define the structure. Here is an example: The languange is simple, either in its textual representation or its number of productions (the specification rules), allowing for compact implementations. OGDL character streams are normally formed by Unicode characters, and encoded as UTF-8 strings, but any encoding that is ASCII transparent is compatible with the specification. See the full spec at http://ogdl.org. To install this package just do: If we have a text file 'config.g' containing: then, will print If the timeout parameter was not present, then the default value (60) will be assigned to 'to'. The default value is optional, but be aware that Int64() will return 0 in case that the parameter doesn't exist. The configuration file can be written in a conciser way: The package includes a template processor. It takes an arbitrary input stream with some variables in it, and produces an output stream with the variables resolved out of a Graph object which acts as context. For example (given the previous config file): string(b) is then: Some rules are followed:
Package ogdl is used to process OGDL, the Ordered Graph Data Language. OGDL is a textual format to write trees or graphs of text, where indentation and spaces define the structure. Here is an example: The languange is simple, either in its textual representation or its number of productions (the specification rules), allowing for compact implementations. OGDL character streams are normally formed by Unicode characters, and encoded as UTF-8 strings, but any encoding that is ASCII transparent is compatible with the specification. See the full spec at http://ogdl.org. To install this package just do: If we have a text file 'config.ogdl' containing: then, will print If the timeout parameter was not present, then the default value (60) will be assigned to 'to'. The default value is optional, but be aware that Int64() will return 0 in case that the parameter doesn't exist. The configuration file can be written in a conciser way: The package includes a template processor. It takes an arbitrary input stream with some variables in it, and produces an output stream with the variables resolved out of a Graph object which acts as context. For example (given the previous config file): string(b) is then: Some rules are followed:
Package dcc implements the DCC protocol for controlling model trains. It can support a number of different encoders, which are in charge of translating DCC packages into electrical signals. By default, a Raspberry Pi driver is provided. The implementation follows the S-91 Electrical Standard (http://www.nmra.org/sites/default/files/standards/sandrp/pdf/s-9.1_electrical_standards_2006.pdf), the S-92 DCC Communications Standard (http://www.nmra.org/sites/default/files/s-92-2004-07.pdf) and the S-9.2.1 Extended Packet Formats for Digital Command Control standard (http://www.nmra.org/sites/default/files/s-9.2.1_2012_07.pdf).
Package couchdb is a driver for connecting with a CouchDB server over HTTP. Use the `couch` driver name when using this driver. The DSN should be a full URL, likely with login credentials: The CouchDB driver generally interprets kivik.Options keys and values as URL query parameters. Values of the following types will be converted to their appropriate string representation when URL-encoded: Passing any other type will return an error. The only exceptions to the above rule are: The CouchDB driver supports a number of authentication methods. For most uses, you don't need to worry about authentication at all--just include authentication credentials in your connection DSN: This will use Cookie authentication by default. To use one of the explicit authentication mechanisms, you'll need to use kivik's Authenticate method. For example: Normally, to include an attachment in a CouchDB document, it must be base-64 encoded, which leads to increased network traffic and higher CPU load. CouchDB also supports the option to upload multiple attachments in a single request using the 'multipart/related' content type. See http://docs.couchdb.org/en/stable/api/document/common.html#creating-multiple-attachments As an experimental feature, this is now supported by the Kivik CouchDB driver as well. To take advantage of this capability, the `doc` argument to the Put() method must be either: With this in place, the CouchDB driver will switch to `multipart/related` mode, sending each attachment in binary format, rather than base-64 encoding it. To function properly, each attachment must have an accurate Size value. If the Size value is unset, the entirely attachment may be read to determine its size, prior to sending it over the network, leading to delays and unnecessary I/O and CPU usage. The simplest way to ensure efficiency is to use the NewAttachment() method, provided by this package. See the documentation on that method for proper usage. Example: To disable the `multipart/related` capabilities entirely, you may pass the `NoMultipartPut` option, with any value. This will fallback to the default of inline base-64 encoding the attachments. Example: If you find yourself wanting to disable this feature, due to bugs or performance, please consider filing a bug report against Kivik as well, so we can look for a solution that will allow using this optimization.
Package signal provides primitives for digital signal manipulations. Floating-point, fixed signed and unsigned signal types are supported. This package focuses on allocation optimisations as it's one of the most important aspects for DSP applications. Digital signal is a representation of a physical signal that is a sampled and quantized. It is discrete in time and amplitude. When analog signal is converted to digital, it goes through two steps: discretization and quantization. Discretization means that the signal is divided into equal intervals of time, and each interval is represented by a single measurement of amplitude. Quantization means each amplitude measurement is approximated by a value from a finite set. The finite set of quantization values determines the digital signal representation. It differs from type to type: Floating-point signal can exceed this range without loosing signal data, but fixed-point signals will be clipped and meaningful signal values will be lost. In order to allocate any of signal buffers, an allocator should be used. Allocator defines what number of channels and capacity per channel allocated buffers will have: This package offers types that represent floating-point and both signed/unsigned fixed-point signal buffers. They implement Floating, Signed and Unsigned interfaces respectively. Internally, signal buffers use a slice of built-in type to hold the data. Fixed-point buffers require a bit depth to be provided at allocation time. It allows to use the same type to hold values of various bit depths: Signal buffers have semantics of golang-slices - they can be sliced or apended one to another. All operations respect number of channels within buffer, so slicing and appending always happens for all channels. Write/Read values There are multiple ways to write/read values from the signal buffers. WriteT/ReadT functions allows to write or read the data in the format of single slice, where samples for different channels are interleaved: WriteStripedT/ReadStripedT functions, on the other hand, can use slice of slices, where each nested slice represents a single channel of signal: It's possible to append samples to the buffers using AppendSample fucntion. However, in order to have more control over allocations, this function won't let the buffer grow beyond it's initial capacity. To achieve this, another buffer needs to be explicitly allocated. The one can also iterate over signal buffers. Please, refer to examples for more details. This package also provides a pool-backed allocator. It contains a sync.Pool per signal type. It serves the purpose of decreasing a GC during the runtime when many buffers are allocated. The pool operates with the interface types (Signed, Unsigned, Floating) for convenience and alignment with other functions of this package. It is safe for concurrent use by multiple goroutines. This example demonstrates how to iterate over the buffer. This example demonstrates how to use pool to allocate buffers. This example demonstrates how read and write data to the buffer.
The Escher HTTP request signing framework is intended to provide a secure way for clients to sign HTTP requests, and servers to check the integrity of these messages. The goal of the protocol is to introduce an authentication solution for REST API services, that is more secure than the currently available protocols. RFC 2617 (HTTP Authentication) defines Basic and Digest Access Authentication. They’re widely used, but Basic Access Authentication doesn’t encrypt the secret and doesn’t add integrity checks to the requests. Digest Access Authentication sends the secret encrypted, but the algorithm with creating a checksum with a nonce and using md5 should not be considered highly secure these days, and as with Basic Access Authentication, there’s no way to check the integrity of the message. RFC 6749 (OAuth 2.0 Authorization) enables a third-party application to obtain limited access to an HTTP service, either on behalf of a resource owner by orchestrating an approval interaction between the resource owner and the HTTP service, or by allowing the third-party application to obtain access on its own behalf. This is not helpful for a machine-to-machine communication situation, like a REST API authentication, because typically there’s no third-party user involved. Additionally, after a token is obtained from the authorization endpoint, it is used with no encryption, and doesn’t provide integration checking, or prevent repeating messages. OAuth 2.0 is a stateful protocol which needs a database to store the tokens for client sessions. Amazon and other service providers created protocols addressing these issues, however there is no public standard with open source implementations available from them. As Escher is based on a publicly documented, widely, in-the-wild used protocol, the specification does not include novelty techniques. 2. Signing an HTTP Request Escher defines a stateless signature generation mechanism. The signature is calculated from the key parts of the HTTP request, a service identifier string called credential scope, and a client key and client secret. The signature generation steps are: canonicalizing the request, creating a string to calculate the signature, and adding the signature to the original request. Escher supports two hash algorithms: SHA256 and SHA512 designed by the NSA (U.S. National Security Agency). 2.1. Canonicalizing the Request In order to calculate a checksum from the key HTTP request parts, the HTTP request method, the request URI, the query parts, the headers, and the request body have to be canonicalized. The output of the canonicalization step will be a string including the request parts separated by LF (line feed, “n”) characters. The string will be used to calculate a checksum for the request. 2.1.1. The HTTP method The HTTP method defined by RFC2616 (Hypertext Transfer Protocol) is case sensitive, and must be available in upper case, no transformation has to be applied: POST 2.1.2. The Path The path is the absolute path of the URL. Starts with a slash (/) character, and does not include the query part (and the question mark). Escher follows the rules defined by RFC3986 (Uniform Resource Identifier) to normalize the path. Basically it means: Convert relative paths to absolute, remove redundant path components. URI-encode each path components: the “reserved characters” defined by RFC3986 (Uniform Resource Identifier) have to be kept as they are (no encoding applied) all other characters have to be percent encoded, including SPACE (to %20, instead of +) non-ASCII, UTF-8 characters should be percent encoded to 2 or more pieces (á to %C3%A1) percent encoded hexadecimal numbers have to be upper cased (eg: a%c2%b1b to a%C2%B1b) Normalize empty paths to /. For example: 2.1.3. The Query String RFC3986 (Uniform Resource Identifier) should provide guidance for canonicalization of the query string, but here’s the complete list of the rules to be applied: URI-encode each query parameter names and values the “reserved characters” defined by RFC3986 (Uniform Resource Identifier) have to be kept as they are (no encoding applied) all other characters have to be percent encoded, including SPACE (to %20, instead of +) non-ASCII, UTF-8 characters should be percent encoded to 2 or more pieces (á to %C3%A1) percent encoded hexadecimal numbers have to be upper cased (eg: a%c2%b1b to a%C2%B1b) Normalize empty query strings to empty string. Sort query parameters by the encoded parameter names (ASCII order). Do not shorten parameter values if their parameter name is the same (key=B&key=A is a valid output), the order of parameters in a URL may be significant (this is not defined by the HTTP standard). Separate parameter names and values by = signs, include = for empty values, too Separate parameters by & For example: To canonicalize the headers, the following rules have to be followed: Lower case the header names. Separate header names and values by a :, with no spaces. Sort header names to alphabetical order (ASCII). Group headers with the same names into a header, and separate their values by commas, without sorting. Trim header values, keep all the spaces between quote characters ("). For example: 2.1.5. Signed Headers The list of headers to include when calculating the signature. Lower cased value of header names, separated by ;, like this: date;host 2.1.6. Body Checksum A checksum for the request body, aka the payload has to be calculated. Escher supports SHA-256 and SHA-512 algorithms for checksum calculation. If the request contains no body, an empty string has to be used as the input for the hash algorithm. The selected algorithm will be added later to the authorization header, so the server will be able to use the same algorithm for validation. The checksum of the body has to be presented as a lower cased hexadecimal string, for example: 2.1.7. Concatenating the Canonicalized Parts All the steps above produce a row of data, except the headers canonicalization, as it creates one row per headers. These have to be concatenated with LF (line feed, “n”) characters into a string. An example: 2.2. Creating the Signature The next step is creating another string which will be directly used to calculate the signature. 2.2.1. Algorithm ID The algorithm ID comes from the algo_prefix (default value is ESR) and the algorithm used to calculate checksums during the signing process. The string algo_prefix, “HMAC”, and the algorithm name should be concatenated with dashes, like this: 2.2.2. Long Date The long date is the request date in the ISO 8601 basic format, like YYYYMMDD + T + HHMMSS + Z. Note that the basic format uses no punctuation. Example is: This date has to be added later, too, as a date header (default header name is X-Escher-Date). 2.2.3. Date and Credential Scope Next information is the short date, and the credential scope concatenated with a / character. The short date is the request date’s date part, an ISO 8601 basic formatted representation, the credential scope is defined by the service. Example: This will be added later, too, as part of the authorization header (default header name is X-Escher-Auth). 2.2.4. Checksum of the Canonicalized Request Take the output of step 2.1.7., and create a checksum from the canonicalized checksum string. This checksum has to be represented as a lower cased hexadecimal string, too. Something like this will be an output: 2.2.5. Concatenating the Signing String Concatenate the outputs of steps 2.2. with LF characters. Example output: 2.2.6. The Signing Key The signing key is based on the algo_prefix, the client secret, the parts of the credential scope, and the request date. Take the algo_prefix, concatenate the client secret to it. First apply the HMAC algorithm to the request date, then apply the actual value on each of the credential scope parts (splitted at /). The end result is a binary signing key. Pseudo code: 2.2.7. Create the Signature The signature is created from the output of steps 2.2.5. (Signing String) and 2.2.6. (Signing Key). With the selected algorithm, create a checksum. It has to be represented as a lower cased hexadecimal string. Something like this will be an output: 2.3. Adding the Signature to the Request The final step of the Escher signing process is adding the Signature to the request. Escher adds a new header to the request, by default, the header name is X-Escher-Auth. The header value will include the algorithm ID (see 2.2.1.), the client key, the short date and the credential scope (see 2.2.3.), the signed headers string (see 2.1.5.) and finally the signature (see 2.2.7.). The values of this inputs have to be concatenated like this: 3. Presigning a URL The URL presigning process is very similar to the request signing procedure. But for a URL, there are no headers, no request body, so the calculation of the Signature is different. Also, the Signature cannot be added to the headers, but is included as query parameters. A significant difference is that the presigning allows defining an expiration time. By default, it is 86400 secs, 24 hours. The current time and the expiration time will be included in the URL, and the server has to check if the URL is expired. 3.1. Canonicalizing the URL to Presign The canonicalization for URL presigning is the same process as for HTTP requests, in this section we will cover the differences only. 3.1.1. The HTTP method The HTTP method for presigned URLs is fixed to: For example: 3.1.3. The Query String The query is coming from the URL, but the algorithm, credentials, date, expiration time, and signed headers have to be added to the query parts. 3.1.4. The Headers A URL has no headers, Escher creates the Host header based on the URL’s domain information, and adds it to the canonicalized request. For example: 3.1.5. Signed Headers It will be host, as that’s the only header included. Example:
Package zap provides fast, structured, leveled logging. For applications that log in the hot path, reflection-based serialization and string formatting are prohibitively expensive - they're CPU-intensive and make many small allocations. Put differently, using json.Marshal and fmt.Fprintf to log tons of interface{} makes your application slow. Zap takes a different approach. It includes a reflection-free, zero-allocation JSON encoder, and the base Logger strives to avoid serialization overhead and allocations wherever possible. By building the high-level SugaredLogger on that foundation, zap lets users choose when they need to count every allocation and when they'd prefer a more familiar, loosely typed API. In contexts where performance is nice, but not critical, use the SugaredLogger. It's 4-10x faster than other structured logging packages and supports both structured and printf-style logging. Like log15 and go-kit, the SugaredLogger's structured logging APIs are loosely typed and accept a variadic number of key-value pairs. (For more advanced use cases, they also accept strongly typed fields - see the SugaredLogger.With documentation for details.) By default, loggers are unbuffered. However, since zap's low-level APIs allow buffering, calling Sync before letting your process exit is a good habit. In the rare contexts where every microsecond and every allocation matter, use the Logger. It's even faster than the SugaredLogger and allocates far less, but it only supports strongly-typed, structured logging. Choosing between the Logger and SugaredLogger doesn't need to be an application-wide decision: converting between the two is simple and inexpensive. The simplest way to build a Logger is to use zap's opinionated presets: NewExample, NewProduction, and NewDevelopment. These presets build a logger with a single function call: Presets are fine for small projects, but larger projects and organizations naturally require a bit more customization. For most users, zap's Config struct strikes the right balance between flexibility and convenience. See the package-level BasicConfiguration example for sample code. More unusual configurations (splitting output between files, sending logs to a message queue, etc.) are possible, but require direct use of github.com/kunnos/zap/zapcore. See the package-level AdvancedConfiguration example for sample code. The zap package itself is a relatively thin wrapper around the interfaces in github.com/kunnos/zap/zapcore. Extending zap to support a new encoding (e.g., BSON), a new log sink (e.g., Kafka), or something more exotic (perhaps an exception aggregation service, like Sentry or Rollbar) typically requires implementing the zapcore.Encoder, zapcore.WriteSyncer, or zapcore.Core interfaces. See the zapcore documentation for details. Similarly, package authors can use the high-performance Encoder and Core implementations in the zapcore package to build their own loggers. An FAQ covering everything from installation errors to design decisions is available at https://github.com/uber-go/zap/blob/master/FAQ.md.
Package jibby is a high-performance, streaming JSON-to-BSON decoder. It decodes successive JSON objects into BSON documents from a buffered input byte stream while minimizing memory copies. Only UTF-8 encoding is supported and input text is expected to be well-formed. Jibby optionally supports the MongoDB Extended JSON v2 format (https://docs.mongodb.com/manual/reference/mongodb-extended-json/index.html). There is limited support for the v1 format -- specifically, the `$type` and `$regex` keys use heuristics to determine whether these are extended JSON or MongoDB query operators. Escape sequences are not supported in Extended JSON keys or number formats, only in naturally textual fields like `$symbol`, `$code`, etc. In practice, MongoDB Extended JSON generators should never output escape sequences in keys and number fields anyway. Jibby is extensively tested. Jibby's JSON-to-BSON output is compared against reference output from the MongoDB Go driver. Extended JSON conversion is tested against the MongoDB BSON Corpus: https://github.com/mongodb/specifications/tree/master/source/bson-corpus. JSON parsing support is tested against data sets from Nicholas Seriot's Parsing JSON is a Minefield article (http://seriot.ch/parsing_json.php). It behaves correctly against all "y" (must support) tests and "n" (must error) tests. It passes all "i" (implementation defined) tests except for cases exceeding Go's numerical precision or with invalid/unsupported Unicode encoding.
Package cgi implements the common gateway interface (CGI) for Caddy 2, a modern, full-featured, easy-to-use web server. It has been forked from the fantastic work of Kurt Jung who wrote that plugin for Caddy 1. This plugin lets you generate dynamic content on your website by means of command line scripts. To collect information about the inbound HTTP request, your script examines certain environment variables such as PATH_INFO and QUERY_STRING. Then, to return a dynamically generated web page to the client, your script simply writes content to standard output. In the case of POST requests, your script reads additional inbound content from standard input. The advantage of CGI is that you do not need to fuss with server startup and persistence, long term memory management, sockets, and crash recovery. Your script is called when a request matches one of the patterns that you specify in your Caddyfile. As soon as your script completes its response, it terminates. This simplicity makes CGI a perfect complement to the straightforward operation and configuration of Caddy. The benefits of Caddy, including HTTPS by default, basic access authentication, and lots of middleware options extend easily to your CGI scripts. CGI has some disadvantages. For one, Caddy needs to start a new process for each request. This can adversely impact performance and, if resources are shared between CGI applications, may require the use of some interprocess synchronization mechanism such as a file lock. Your server's responsiveness could in some circumstances be affected, such as when your web server is hit with very high demand, when your script's dependencies require a long startup, or when concurrently running scripts take a long time to respond. However, in many cases, such as using a pre-compiled CGI application like fossil or a Lua script, the impact will generally be insignificant. Another restriction of CGI is that scripts will be run with the same permissions as Caddy itself. This can sometimes be less than ideal, for example when your script needs to read or write files associated with a different owner. Serving dynamic content exposes your server to more potential threats than serving static pages. There are a number of considerations of which you should be aware when using CGI applications. CGI scripts should be located outside of Caddy's document root. Otherwise, an inadvertent misconfiguration could result in Caddy delivering the script as an ordinary static resource. At best, this could merely confuse the site visitor. At worst, it could expose sensitive internal information that should not leave the server. Mistrust the contents of PATH_INFO, QUERY_STRING and standard input. Most of the environment variables available to your CGI program are inherently safe because they originate with Caddy and cannot be modified by external users. This is not the case with PATH_INFO, QUERY_STRING and, in the case of POST actions, the contents of standard input. Be sure to validate and sanitize all inbound content. If you use a CGI library or framework to process your scripts, make sure you understand its limitations. An error in a CGI application is generally handled within the application itself and reported in the headers it returns. Your CGI application can be executed directly or indirectly. In the direct case, the application can be a compiled native executable or it can be a shell script that contains as its first line a shebang that identifies the interpreter to which the file's name should be passed. Caddy must have permission to execute the application. On Posix systems this will mean making sure the application's ownership and permission bits are set appropriately; on Windows, this may involve properly setting up the filename extension association. In the indirect case, the name of the CGI script is passed to an interpreter such as lua, perl or python. - This module needs to be installed (obviously). - The directive needs to be registered in the Caddyfile: The basic cgi directive lets you add a handler in the current caddy router location with a given script and optional arguments. The matcher is a default caddy matcher that is used to restrict the scope of this directive. The directive can be repeated any reasonable number of times. Here is the basic syntax: For example: When a request such as https://example.com/report or https://example.com/report/weekly arrives, the cgi middleware will detect the match and invoke the script named /usr/local/cgi-bin/report. The current working directory will be the same as Caddy itself. Here, it is assumed that the script is self-contained, for example a pre-compiled CGI application or a shell script. Here is an example of a standalone script, similar to one used in the cgi plugin's test suite: The environment variables PATH_INFO and QUERY_STRING are populated and passed to the script automatically. There are a number of other standard CGI variables included that are described below. If you need to pass any special environment variables or allow any environment variables that are part of Caddy's process to pass to your script, you will need to use the advanced directive syntax described below. Beware that in Caddy v2 it is (currently) not possible to separate the path left of the matcher from the full URL. Therefore if you require your CGI program to know the SCRIPT_NAME, make sure to pass that explicitly: In order to specify custom environment variables, pass along one or more environment variables known to Caddy, or specify more than one match pattern for a given rule, you will need to use the advanced directive syntax. That looks like this: For example, The script_name subdirective helps the cgi module to separate the path to the script from the (virtual) path afterwards (which shall be passed to the script). env can be used to define a list of key=value environment variable pairs that shall be passed to the script. pass_env can be used to define a list of environment variables of the Caddy process that shall be passed to the script. If your CGI application runs properly at the command line but fails to run from Caddy it is possible that certain environment variables may be missing. For example, the ruby gem loader evidently requires the HOME environment variable to be set; you can do this with the subdirective pass_env HOME. Another class of problematic applications require the COMPUTERNAME variable. The pass_all_env subdirective instructs Caddy to pass each environment variable it knows about to the CGI excutable. This addresses a common frustration that is caused when an executable requires an environment variable and fails without a descriptive error message when the variable cannot be found. These applications often run fine from the command prompt but fail when invoked with CGI. The risk with this subdirective is that a lot of server information is shared with the CGI executable. Use this subdirective only with CGI applications that you trust not to leak this information. buffer_limit is used when a http request has Transfer-Endcoding: chunked. The Go CGI Handler refused to handle these kinds of requests, see https://github.com/golang/go/issues/5613. In order to work around this the chunked request is buffered by caddy and sent to the CGI application as a whole with the correct CONTENT_LENGTH set. The buffer_limit setting marks a threshold between buffering in memory and using a temporary file. Every request body smaller than the buffer_limit is buffered in-memory. It accepts all formats supported by go-humanize. Default: 4MiB. (An example of this is git push if the objects to push are larger than the http.postBuffer) With the unbuffered_output subdirective it is possible to instruct the CGI handler to flush output from the CGI script as soon as possible. By default, the output is buffered into chunks before it is being written to optimize the network usage and allow to determine the Content-Length. When unbuffered, bytes will be written as soon as possible. This will also force the response to be written in chunked encoding. If you run into unexpected results with the CGI plugin, you are able to examine the environment in which your CGI application runs. To enter inspection mode, add the subdirective inspect to your CGI configuration block. This is a development option that should not be used in production. When in inspection mode, the plugin will respond to matching requests with a page that displays variables of interest. In particular, it will show the replacement value of {match} and the environment variables to which your CGI application has access. For example, consider this example CGI block: When you request a matching URL, for example, the Caddy server will deliver a text page similar to the following. The CGI application (in this case, wapptclsh) will not be called. This information can be used to diagnose problems with how a CGI application is called. To return to operation mode, remove or comment out the inspect subdirective. In this example, the Caddyfile looks like this: Note that a request for /show gets mapped to a script named /usr/local/cgi-bin/report/gen. There is no need for any element of the script name to match any element of the match pattern. The contents of /usr/local/cgi-bin/report/gen are: The purpose of this script is to show how request information gets communicated to a CGI script. Note that POST data must be read from standard input. In this particular case, posted data gets stored in the variable POST_DATA. Your script may use a different method to read POST content. Secondly, the SCRIPT_EXEC variable is not a CGI standard. It is provided by this middleware and contains the entire command line, including all arguments, with which the CGI script was executed. When a browser requests the response looks like When a client makes a POST request, such as with the following command the response looks the same except for the following lines: This small example demonstrates how to write a CGI program in Go. The use of a bytes.Buffer makes it easy to report the content length in the CGI header. When this program is compiled and installed as /usr/local/bin/servertime, the following directive in your Caddy file will make it available:
Package imap implements an IMAP4rev1 client, as defined in RFC 3501. The implementation provides low-level access to all protocol features described in the relevant RFCs (see list below), and assumes that the developer is familiar with the basic rules governing connection states, command execution, server responses, and IMAP data types. Reading this documentation alone is not sufficient for writing a working IMAP client. As a starting point, you should read RFC 2683 to understand some of the nuances of the protocol operation. The rest of the documentation deals with the implementation of this package and not the protocol in general. The package provides three main objects for interacting with an IMAP4 server: Client, Command, and Response. The client sends commands to the server and receives responses. The Response object is capable of representing all possible server responses and provides helper methods for decoding common data formats, such as LIST, FETCH, SEARCH, etc. The client has two interfaces for issuing commands. The Send method is the raw command interface that can be used for implementing new commands, which are not already supported by the package. All standard commands, as well as those from a number of popular extensions, have dedicated methods that perform capability and field type checks, and properly encode the command arguments before passing them to Send. To support execution of multiple concurrent commands, each server response goes through a filtering process to identify its owner. Each command in progress has an associated ResponseFilter function for this purpose. The Client calls all active filters in the command-issue order until one of the filters "claims" the response. Claimed responses are appended to the Command.Data queue of the claimer. Responses rejected by all filters are referred to as "unilateral server data" and are appended to the Client.Data queue. Commands documented as expecting "no specific responses" use a nil ResponseFilter, which never claims anything. Thus, responses for commands such as NOOP are always delivered to Client.Data queue. The Client/Command state can only be updated by a call to Client.Recv. Each call receives and delivers at most one response, but these calls are often implicit, such as when using the Wait helper function (see below). Be sure to inspect and clear out the Client data queue after all receive operations to avoid missing important server updates. The Client example below demonstrates correct response handling. The Client and its Command objects cannot be used concurrently from multiple goroutines. It is safe to pass Response objects to other goroutines for processing, but the Client assumes "single-threaded" model of operation, so all method calls for the same connection must be serialized with sync.Mutex or some other synchronization mechanism. Likewise, it is not safe to access Client.Data and Command.Data in parallel with a call that can append new responses to these fields. Unless a command is marked as being "synchronous", which is usually those commands that change the connection state, the associated method returns as soon as the command is sent to the server, without waiting for completion. This allows the client to issue multiple concurrent commands, and then process the responses and command completions as they arrive. A call to Command.Result on a command that is "in progress" will block until that command is finished. There is also a convenience function that turns any asynchronous command into a synchronous one: If err is nil when the call returns, the command was completed with the OK status and all data responses (if any) are queued in cmd.Data. The Client launches a goroutine to support receive operations with timeouts. The user must call Client.Logout to close the connection and stop the goroutine. The only time it is unnecessary to call Client.Logout is when the server closes the connection first and Client.Recv returns io.EOF error. The following RFCs are implemented by this package: The following RFCs are either informational, not fully implemented, or place no implementation requirements on the package, but may be relevant to other parts of a client application: Copyright 2014 The imapsrv Authors. All rights reserved. Use of this source code is governed by a BSD-style license that can be found in the imapsrv.LICENSE file. Copyright 2014 The imapsrv Authors. All rights reserved. Use of this source code is governed by a BSD-style license that can be found in the imapsrv.LICENSE file. Copyright 2014 The imapsrv Authors. All rights reserved. Use of this source code is governed by a BSD-style license that can be found in the imapsrv.LICENSE file.
Package log15 provides an opinionated, simple toolkit for best-practice logging that is both human and machine readable. It is modeled after the standard library's io and net/http packages. This package enforces you to only log key/value pairs. Keys must be strings. Values may be any type that you like. The default output format is logfmt, but you may also choose to use JSON instead if that suits you. Here's how you log: This will output a line that looks like: To get started, you'll want to import the library: Now you're ready to start logging: Because recording a human-meaningful message is common and good practice, the first argument to every logging method is the value to the *implicit* key 'msg'. Additionally, the level you choose for a message will be automatically added with the key 'lvl', and so will the current timestamp with key 't'. You may supply any additional context as a set of key/value pairs to the logging function. log15 allows you to favor terseness, ordering, and speed over safety. This is a reasonable tradeoff for logging functions. You don't need to explicitly state keys/values, log15 understands that they alternate in the variadic argument list: If you really do favor your type-safety, you may choose to pass a log.Ctx instead: Frequently, you want to add context to a logger so that you can track actions associated with it. An http request is a good example. You can easily create new loggers that have context that is automatically included with each log line: This will output a log line that includes the path context that is attached to the logger: The Handler interface defines where log lines are printed to and how they are formated. Handler is a single interface that is inspired by net/http's handler interface: Handlers can filter records, format them, or dispatch to multiple other Handlers. This package implements a number of Handlers for common logging patterns that are easily composed to create flexible, custom logging structures. Here's an example handler that prints logfmt output to Stdout: Here's an example handler that defers to two other handlers. One handler only prints records from the rpc package in logfmt to standard out. The other prints records at Error level or above in JSON formatted output to the file /var/log/service.json This package implements three Handlers that add debugging information to the context, CallerFileHandler, CallerFuncHandler and CallerStackHandler. Here's an example that adds the source file and line number of each logging call to the context. This will output a line that looks like: Here's an example that logs the call stack rather than just the call site. This will output a line that looks like: The "%+v" format instructs the handler to include the path of the source file relative to the compile time GOPATH. The github.com/go-stack/stack package documents the full list of formatting verbs and modifiers available. The Handler interface is so simple that it's also trivial to write your own. Let's create an example handler which tries to write to one handler, but if that fails it falls back to writing to another handler and includes the error that it encountered when trying to write to the primary. This might be useful when trying to log over a network socket, but if that fails you want to log those records to a file on disk. This pattern is so useful that a generic version that handles an arbitrary number of Handlers is included as part of this library called FailoverHandler. Sometimes, you want to log values that are extremely expensive to compute, but you don't want to pay the price of computing them if you haven't turned up your logging level to a high level of detail. This package provides a simple type to annotate a logging operation that you want to be evaluated lazily, just when it is about to be logged, so that it would not be evaluated if an upstream Handler filters it out. Just wrap any function which takes no arguments with the log.Lazy type. For example: If this message is not logged for any reason (like logging at the Error level), then factorRSAKey is never evaluated. The same log.Lazy mechanism can be used to attach context to a logger which you want to be evaluated when the message is logged, but not when the logger is created. For example, let's imagine a game where you have Player objects: You always want to log a player's name and whether they're alive or dead, so when you create the player object, you might do: Only now, even after a player has died, the logger will still report they are alive because the logging context is evaluated when the logger was created. By using the Lazy wrapper, we can defer the evaluation of whether the player is alive or not to each log message, so that the log records will reflect the player's current state no matter when the log message is written: If log15 detects that stdout is a terminal, it will configure the default handler for it (which is log.StdoutHandler) to use TerminalFormat. This format logs records nicely for your terminal, including color-coded output based on log level. Becasuse log15 allows you to step around the type system, there are a few ways you can specify invalid arguments to the logging functions. You could, for example, wrap something that is not a zero-argument function with log.Lazy or pass a context key that is not a string. Since logging libraries are typically the mechanism by which errors are reported, it would be onerous for the logging functions to return errors. Instead, log15 handles errors by making these guarantees to you: - Any log record containing an error will still be printed with the error explained to you as part of the log record. - Any log record containing an error will include the context key LOG15_ERROR, enabling you to easily (and if you like, automatically) detect if any of your logging calls are passing bad values. Understanding this, you might wonder why the Handler interface can return an error value in its Log method. Handlers are encouraged to return errors only if they fail to write their log records out to an external source like if the syslog daemon is not responding. This allows the construction of useful handlers which cope with those failures like the FailoverHandler. log15 is intended to be useful for library authors as a way to provide configurable logging to users of their library. Best practice for use in a library is to always disable all output for your logger by default and to provide a public Logger instance that consumers of your library can configure. Like so: Users of your library may then enable it if they like: The ability to attach context to a logger is a powerful one. Where should you do it and why? I favor embedding a Logger directly into any persistent object in my application and adding unique, tracing context keys to it. For instance, imagine I am writing a web browser: When a new tab is created, I assign a logger to it with the url of the tab as context so it can easily be traced through the logs. Now, whenever we perform any operation with the tab, we'll log with its embedded logger and it will include the tab title automatically: There's only one problem. What if the tab url changes? We could use log.Lazy to make sure the current url is always written, but that would mean that we couldn't trace a tab's full lifetime through our logs after the user navigate to a new URL. Instead, think about what values to attach to your loggers the same way you think about what to use as a key in a SQL database schema. If it's possible to use a natural key that is unique for the lifetime of the object, do so. But otherwise, log15's ext package has a handy RandId function to let you generate what you might call "surrogate keys" They're just random hex identifiers to use for tracing. Back to our Tab example, we would prefer to set up our Logger like so: Now we'll have a unique traceable identifier even across loading new urls, but we'll still be able to see the tab's current url in the log messages. For all Handler functions which can return an error, there is a version of that function which will return no error but panics on failure. They are all available on the Must object. For example: All of the following excellent projects inspired the design of this library: code.google.com/p/log4go github.com/op/go-logging github.com/technoweenie/grohl github.com/Sirupsen/logrus github.com/kr/logfmt github.com/spacemonkeygo/spacelog golang's stdlib, notably io and net/http https://xkcd.com/927/
Package couchdb is a driver for connecting with a CouchDB server over HTTP. Use the `couch` driver name when using this driver. The DSN should be a full URL, likely with login credentials: The CouchDB driver generally interprets kivik.Options keys and values as URL query parameters. Values of the following types will be converted to their appropriate string representation when URL-encoded: Passing any other type will return an error. The only exceptions to the above rule are: The CouchDB driver supports a number of authentication methods. For most uses, you don't need to worry about authentication at all--just include authentication credentials in your connection DSN: This will use Cookie authentication by default. To use one of the explicit authentication mechanisms, you'll need to use kivik's Authenticate method. For example: Normally, to include an attachment in a CouchDB document, it must be base-64 encoded, which leads to increased network traffic and higher CPU load. CouchDB also supports the option to upload multiple attachments in a single request using the 'multipart/related' content type. See http://docs.couchdb.org/en/stable/api/document/common.html#creating-multiple-attachments As an experimental feature, this is now supported by the Kivik CouchDB driver as well. To take advantage of this capability, the `doc` argument to the Put() method must be either: With this in place, the CouchDB driver will switch to `multipart/related` mode, sending each attachment in binary format, rather than base-64 encoding it. To function properly, each attachment must have an accurate Size value. If the Size value is unset, the entirely attachment may be read to determine its size, prior to sending it over the network, leading to delays and unnecessary I/O and CPU usage. The simplest way to ensure efficiency is to use the NewAttachment() method, provided by this package. See the documentation on that method for proper usage. Example: To disable the `multipart/related` capabilities entirely, you may pass the `NoMultipartPut` option, with any value. This will fallback to the default of inline base-64 encoding the attachments. Example: If you find yourself wanting to disable this feature, due to bugs or performance, please consider filing a bug report against Kivik as well, so we can look for a solution that will allow using this optimization.
Package blackfriday is a markdown processor. It translates plain text with simple formatting rules into an AST, which can then be further processed to HTML (provided by Blackfriday itself) or other formats (provided by the community). The simplest way to invoke Blackfriday is to call the Run function. It will take a text input and produce a text output in HTML (or other format). A slightly more sophisticated way to use Blackfriday is to create a Markdown processor and to call Parse, which returns a syntax tree for the input document. You can leverage Blackfriday's parsing for content extraction from markdown documents. You can assign a custom renderer and set various options to the Markdown processor. If you're interested in calling Blackfriday from command line, see https://github.com/russross/blackfriday-tool. Blackfriday includes an algorithm for creating sanitized anchor names corresponding to a given input text. This algorithm is used to create anchors for headings when AutoHeadingIDs extension is enabled. The algorithm is specified below, so that other packages can create compatible anchor names and links to those anchors. The algorithm iterates over the input text, interpreted as UTF-8, one Unicode code point (rune) at a time. All runes that are letters (category L) or numbers (category N) are considered valid characters. They are mapped to lower case, and included in the output. All other runes are considered invalid characters. Invalid characters that precede the first valid character, as well as invalid character that follow the last valid character are dropped completely. All other sequences of invalid characters between two valid characters are replaced with a single dash character '-'. SanitizedAnchorName exposes this functionality, and can be used to create compatible links to the anchor names generated by blackfriday. This algorithm is also implemented in a small standalone package at github.com/shurcooL/sanitized_anchor_name. It can be useful for clients that want a small package and don't need full functionality of blackfriday.
Package log15 provides an opinionated, simple toolkit for best-practice logging that is both human and machine readable. It is modeled after the standard library's io and net/http packages. This package enforces you to only log key/value pairs. Keys must be strings. Values may be any type that you like. The default output format is logfmt, but you may also choose to use JSON instead if that suits you. Here's how you log: This will output a line that looks like: To get started, you'll want to import the library: Now you're ready to start logging: Because recording a human-meaningful message is common and good practice, the first argument to every logging method is the value to the *implicit* key 'msg'. Additionally, the level you choose for a message will be automatically added with the key 'lvl', and so will the current timestamp with key 't'. You may supply any additional context as a set of key/value pairs to the logging function. log15 allows you to favor terseness, ordering, and speed over safety. This is a reasonable tradeoff for logging functions. You don't need to explicitly state keys/values, log15 understands that they alternate in the variadic argument list: If you really do favor your type-safety, you may choose to pass a log.Ctx instead: Frequently, you want to add context to a logger so that you can track actions associated with it. An http request is a good example. You can easily create new loggers that have context that is automatically included with each log line: This will output a log line that includes the path context that is attached to the logger: The Handler interface defines where log lines are printed to and how they are formated. Handler is a single interface that is inspired by net/http's handler interface: Handlers can filter records, format them, or dispatch to multiple other Handlers. This package implements a number of Handlers for common logging patterns that are easily composed to create flexible, custom logging structures. Here's an example handler that prints logfmt output to Stdout: Here's an example handler that defers to two other handlers. One handler only prints records from the rpc package in logfmt to standard out. The other prints records at Error level or above in JSON formatted output to the file /var/log/service.json This package implements three Handlers that add debugging information to the context, CallerFileHandler, CallerFuncHandler and CallerStackHandler. Here's an example that adds the source file and line number of each logging call to the context. This will output a line that looks like: Here's an example that logs the call stack rather than just the call site. This will output a line that looks like: The "%+v" format instructs the handler to include the path of the source file relative to the compile time GOPATH. The github.com/go-stack/stack package documents the full list of formatting verbs and modifiers available. The Handler interface is so simple that it's also trivial to write your own. Let's create an example handler which tries to write to one handler, but if that fails it falls back to writing to another handler and includes the error that it encountered when trying to write to the primary. This might be useful when trying to log over a network socket, but if that fails you want to log those records to a file on disk. This pattern is so useful that a generic version that handles an arbitrary number of Handlers is included as part of this library called FailoverHandler. Sometimes, you want to log values that are extremely expensive to compute, but you don't want to pay the price of computing them if you haven't turned up your logging level to a high level of detail. This package provides a simple type to annotate a logging operation that you want to be evaluated lazily, just when it is about to be logged, so that it would not be evaluated if an upstream Handler filters it out. Just wrap any function which takes no arguments with the log.Lazy type. For example: If this message is not logged for any reason (like logging at the Error level), then factorRSAKey is never evaluated. The same log.Lazy mechanism can be used to attach context to a logger which you want to be evaluated when the message is logged, but not when the logger is created. For example, let's imagine a game where you have Player objects: You always want to log a player's name and whether they're alive or dead, so when you create the player object, you might do: Only now, even after a player has died, the logger will still report they are alive because the logging context is evaluated when the logger was created. By using the Lazy wrapper, we can defer the evaluation of whether the player is alive or not to each log message, so that the log records will reflect the player's current state no matter when the log message is written: If log15 detects that stdout is a terminal, it will configure the default handler for it (which is log.StdoutHandler) to use TerminalFormat. This format logs records nicely for your terminal, including color-coded output based on log level. Becasuse log15 allows you to step around the type system, there are a few ways you can specify invalid arguments to the logging functions. You could, for example, wrap something that is not a zero-argument function with log.Lazy or pass a context key that is not a string. Since logging libraries are typically the mechanism by which errors are reported, it would be onerous for the logging functions to return errors. Instead, log15 handles errors by making these guarantees to you: - Any log record containing an error will still be printed with the error explained to you as part of the log record. - Any log record containing an error will include the context key LOG15_ERROR, enabling you to easily (and if you like, automatically) detect if any of your logging calls are passing bad values. Understanding this, you might wonder why the Handler interface can return an error value in its Log method. Handlers are encouraged to return errors only if they fail to write their log records out to an external source like if the syslog daemon is not responding. This allows the construction of useful handlers which cope with those failures like the FailoverHandler. log15 is intended to be useful for library authors as a way to provide configurable logging to users of their library. Best practice for use in a library is to always disable all output for your logger by default and to provide a public Logger instance that consumers of your library can configure. Like so: Users of your library may then enable it if they like: The ability to attach context to a logger is a powerful one. Where should you do it and why? I favor embedding a Logger directly into any persistent object in my application and adding unique, tracing context keys to it. For instance, imagine I am writing a web browser: When a new tab is created, I assign a logger to it with the url of the tab as context so it can easily be traced through the logs. Now, whenever we perform any operation with the tab, we'll log with its embedded logger and it will include the tab title automatically: There's only one problem. What if the tab url changes? We could use log.Lazy to make sure the current url is always written, but that would mean that we couldn't trace a tab's full lifetime through our logs after the user navigate to a new URL. Instead, think about what values to attach to your loggers the same way you think about what to use as a key in a SQL database schema. If it's possible to use a natural key that is unique for the lifetime of the object, do so. But otherwise, log15's ext package has a handy RandId function to let you generate what you might call "surrogate keys" They're just random hex identifiers to use for tracing. Back to our Tab example, we would prefer to set up our Logger like so: Now we'll have a unique traceable identifier even across loading new urls, but we'll still be able to see the tab's current url in the log messages. For all Handler functions which can return an error, there is a version of that function which will return no error but panics on failure. They are all available on the Must object. For example: All of the following excellent projects inspired the design of this library: code.google.com/p/log4go github.com/op/go-logging github.com/technoweenie/grohl github.com/Sirupsen/logrus github.com/kr/logfmt github.com/spacemonkeygo/spacelog golang's stdlib, notably io and net/http https://xkcd.com/927/
Package log15 provides an opinionated, simple toolkit for best-practice logging that is both human and machine readable. It is modeled after the standard library's io and net/http packages. This package enforces you to only log key/value pairs. Keys must be strings. Values may be any type that you like. The default output format is logfmt, but you may also choose to use JSON instead if that suits you. Here's how you log: This will output a line that looks like: To get started, you'll want to import the library: Now you're ready to start logging: Because recording a human-meaningful message is common and good practice, the first argument to every logging method is the value to the *implicit* key 'msg'. Additionally, the level you choose for a message will be automatically added with the key 'lvl', and so will the current timestamp with key 't'. You may supply any additional context as a set of key/value pairs to the logging function. log15 allows you to favor terseness, ordering, and speed over safety. This is a reasonable tradeoff for logging functions. You don't need to explicitly state keys/values, log15 understands that they alternate in the variadic argument list: If you really do favor your type-safety, you may choose to pass a log.Ctx instead: Frequently, you want to add context to a logger so that you can track actions associated with it. An http request is a good example. You can easily create new loggers that have context that is automatically included with each log line: This will output a log line that includes the path context that is attached to the logger: The Handler interface defines where log lines are printed to and how they are formated. Handler is a single interface that is inspired by net/http's handler interface: Handlers can filter records, format them, or dispatch to multiple other Handlers. This package implements a number of Handlers for common logging patterns that are easily composed to create flexible, custom logging structures. Here's an example handler that prints logfmt output to Stdout: Here's an example handler that defers to two other handlers. One handler only prints records from the rpc package in logfmt to standard out. The other prints records at Error level or above in JSON formatted output to the file /var/log/service.json This package implements three Handlers that add debugging information to the context, CallerFileHandler, CallerFuncHandler and CallerStackHandler. Here's an example that adds the source file and line number of each logging call to the context. This will output a line that looks like: Here's an example that logs the call stack rather than just the call site. This will output a line that looks like: The "%+v" format instructs the handler to include the path of the source file relative to the compile time GOPATH. The github.com/go-stack/stack package documents the full list of formatting verbs and modifiers available. The Handler interface is so simple that it's also trivial to write your own. Let's create an example handler which tries to write to one handler, but if that fails it falls back to writing to another handler and includes the error that it encountered when trying to write to the primary. This might be useful when trying to log over a network socket, but if that fails you want to log those records to a file on disk. This pattern is so useful that a generic version that handles an arbitrary number of Handlers is included as part of this library called FailoverHandler. Sometimes, you want to log values that are extremely expensive to compute, but you don't want to pay the price of computing them if you haven't turned up your logging level to a high level of detail. This package provides a simple type to annotate a logging operation that you want to be evaluated lazily, just when it is about to be logged, so that it would not be evaluated if an upstream Handler filters it out. Just wrap any function which takes no arguments with the log.Lazy type. For example: If this message is not logged for any reason (like logging at the Error level), then factorRSAKey is never evaluated. The same log.Lazy mechanism can be used to attach context to a logger which you want to be evaluated when the message is logged, but not when the logger is created. For example, let's imagine a game where you have Player objects: You always want to log a player's name and whether they're alive or dead, so when you create the player object, you might do: Only now, even after a player has died, the logger will still report they are alive because the logging context is evaluated when the logger was created. By using the Lazy wrapper, we can defer the evaluation of whether the player is alive or not to each log message, so that the log records will reflect the player's current state no matter when the log message is written: If log15 detects that stdout is a terminal, it will configure the default handler for it (which is log.StdoutHandler) to use TerminalFormat. This format logs records nicely for your terminal, including color-coded output based on log level. Becasuse log15 allows you to step around the type system, there are a few ways you can specify invalid arguments to the logging functions. You could, for example, wrap something that is not a zero-argument function with log.Lazy or pass a context key that is not a string. Since logging libraries are typically the mechanism by which errors are reported, it would be onerous for the logging functions to return errors. Instead, log15 handles errors by making these guarantees to you: - Any log record containing an error will still be printed with the error explained to you as part of the log record. - Any log record containing an error will include the context key LOG15_ERROR, enabling you to easily (and if you like, automatically) detect if any of your logging calls are passing bad values. Understanding this, you might wonder why the Handler interface can return an error value in its Log method. Handlers are encouraged to return errors only if they fail to write their log records out to an external source like if the syslog daemon is not responding. This allows the construction of useful handlers which cope with those failures like the FailoverHandler. log15 is intended to be useful for library authors as a way to provide configurable logging to users of their library. Best practice for use in a library is to always disable all output for your logger by default and to provide a public Logger instance that consumers of your library can configure. Like so: Users of your library may then enable it if they like: The ability to attach context to a logger is a powerful one. Where should you do it and why? I favor embedding a Logger directly into any persistent object in my application and adding unique, tracing context keys to it. For instance, imagine I am writing a web browser: When a new tab is created, I assign a logger to it with the url of the tab as context so it can easily be traced through the logs. Now, whenever we perform any operation with the tab, we'll log with its embedded logger and it will include the tab title automatically: There's only one problem. What if the tab url changes? We could use log.Lazy to make sure the current url is always written, but that would mean that we couldn't trace a tab's full lifetime through our logs after the user navigate to a new URL. Instead, think about what values to attach to your loggers the same way you think about what to use as a key in a SQL database schema. If it's possible to use a natural key that is unique for the lifetime of the object, do so. But otherwise, log15's ext package has a handy RandId function to let you generate what you might call "surrogate keys" They're just random hex identifiers to use for tracing. Back to our Tab example, we would prefer to set up our Logger like so: Now we'll have a unique traceable identifier even across loading new urls, but we'll still be able to see the tab's current url in the log messages. For all Handler functions which can return an error, there is a version of that function which will return no error but panics on failure. They are all available on the Must object. For example: All of the following excellent projects inspired the design of this library: code.google.com/p/log4go github.com/op/go-logging github.com/technoweenie/grohl github.com/Sirupsen/logrus github.com/kr/logfmt github.com/spacemonkeygo/spacelog golang's stdlib, notably io and net/http https://xkcd.com/927/
This is a logger implementation that supports multiple log levels, multiple output destinations with configurable formats and levels for each. It also supports granular output configuration to get more detailed logging for specific files/packages. Timber includes support for standard XML or JSON config files to get you started quickly. It's also easy to configure in code if you want to DIY. Basic use: IMPORTANT: timber has not default destination configured so log messages will be dropped until a destination is configured It can be used as a drop-in replacement for the standard logger by changing the log import statement from: to It can also be used as the output of the standard logger with Configuration in code is also simple: XML Config file: The <tag> is ignored. To configure the pattern formatter all filters accept: Pattern format specifiers (not the same as log4go!): the string number prefixes are allowed e.g.: %10s will pad the source field to 10 spaces pattern defaults to %M Both log4go synatax of <property name="format"> and new <format name=type> are supported the property syntax will only ever support the pattern formatter To configure granulars: Code Architecture: A MultiLogger <logging> which consists of many ConfigLoggers <filter>. ConfigLoggers have three properties: LogWriter <type>, Level (as a threshold) <level> and LogFormatter <format>. In practice, this means that you define ConfigLoggers with a LogWriter (where the log prints to eg. socket, file, stdio etc), the Level threshold, and a LogFormatter which formats the message before writing. Because the LogFormatters and LogWriters are simple interfaces, it is easy to write your own custom implementations. Once configured, you only deal with the "Logger" interface and use the log methods in your code The motivation for this package grew from a need to make some changes to the functionality of log4go (which had already been integrated into a larger project). I tried to maintain compatiblity with log4go for the interface and configuration. The main issue I had with log4go was that each of logger types had incisistent and incompatible configuration. I looked at contributing changes to log4go, but I would have needed to break existing use cases so I decided to do a rewrite from scratch.
Package promptui is a library providing a simple interface to create command-line prompts for go. It can be easily integrated into spf13/cobra, urfave/cli or any cli go application. promptui has two main input modes: Prompt provides a single line for user input. It supports optional live validation, confirmation and masking the input. Select provides a list of options to choose from. It supports pagination, search, detailed view and custom templates. This is an example for the Prompt mode of promptui. In this example, a prompt is created with a validator function that validates the given value to make sure its a number. If successful, it will output the chosen number in a formatted message. This is an example for the Select mode of promptui. In this example, a select is created with the days of the week as its items. When an item is selected, the selected day will be displayed in a formatted message.
Package log15 provides an opinionated, simple toolkit for best-practice logging that is both human and machine readable. It is modeled after the standard library's io and net/http packages. This package enforces you to only log key/value pairs. Keys must be strings. Values may be any type that you like. The default output format is logfmt, but you may also choose to use JSON instead if that suits you. Here's how you log: This will output a line that looks like: To get started, you'll want to import the library: Now you're ready to start logging: Because recording a human-meaningful message is common and good practice, the first argument to every logging method is the value to the *implicit* key 'msg'. Additionally, the level you choose for a message will be automatically added with the key 'lvl', and so will the current timestamp with key 't'. You may supply any additional context as a set of key/value pairs to the logging function. log15 allows you to favor terseness, ordering, and speed over safety. This is a reasonable tradeoff for logging functions. You don't need to explicitly state keys/values, log15 understands that they alternate in the variadic argument list: If you really do favor your type-safety, you may choose to pass a log.Ctx instead: Frequently, you want to add context to a logger so that you can track actions associated with it. An http request is a good example. You can easily create new loggers that have context that is automatically included with each log line: This will output a log line that includes the path context that is attached to the logger: The Handler interface defines where log lines are printed to and how they are formated. Handler is a single interface that is inspired by net/http's handler interface: Handlers can filter records, format them, or dispatch to multiple other Handlers. This package implements a number of Handlers for common logging patterns that are easily composed to create flexible, custom logging structures. Here's an example handler that prints logfmt output to Stdout: Here's an example handler that defers to two other handlers. One handler only prints records from the rpc package in logfmt to standard out. The other prints records at Error level or above in JSON formatted output to the file /var/log/service.json This package implements three Handlers that add debugging information to the context, CallerFileHandler, CallerFuncHandler and CallerStackHandler. Here's an example that adds the source file and line number of each logging call to the context. This will output a line that looks like: Here's an example that logs the call stack rather than just the call site. This will output a line that looks like: The "%+v" format instructs the handler to include the path of the source file relative to the compile time GOPATH. The github.com/go-stack/stack package documents the full list of formatting verbs and modifiers available. The Handler interface is so simple that it's also trivial to write your own. Let's create an example handler which tries to write to one handler, but if that fails it falls back to writing to another handler and includes the error that it encountered when trying to write to the primary. This might be useful when trying to log over a network socket, but if that fails you want to log those records to a file on disk. This pattern is so useful that a generic version that handles an arbitrary number of Handlers is included as part of this library called FailoverHandler. Sometimes, you want to log values that are extremely expensive to compute, but you don't want to pay the price of computing them if you haven't turned up your logging level to a high level of detail. This package provides a simple type to annotate a logging operation that you want to be evaluated lazily, just when it is about to be logged, so that it would not be evaluated if an upstream Handler filters it out. Just wrap any function which takes no arguments with the log.Lazy type. For example: If this message is not logged for any reason (like logging at the Error level), then factorRSAKey is never evaluated. The same log.Lazy mechanism can be used to attach context to a logger which you want to be evaluated when the message is logged, but not when the logger is created. For example, let's imagine a game where you have Player objects: You always want to log a player's name and whether they're alive or dead, so when you create the player object, you might do: Only now, even after a player has died, the logger will still report they are alive because the logging context is evaluated when the logger was created. By using the Lazy wrapper, we can defer the evaluation of whether the player is alive or not to each log message, so that the log records will reflect the player's current state no matter when the log message is written: If log15 detects that stdout is a terminal, it will configure the default handler for it (which is log.StdoutHandler) to use TerminalFormat. This format logs records nicely for your terminal, including color-coded output based on log level. Becasuse log15 allows you to step around the type system, there are a few ways you can specify invalid arguments to the logging functions. You could, for example, wrap something that is not a zero-argument function with log.Lazy or pass a context key that is not a string. Since logging libraries are typically the mechanism by which errors are reported, it would be onerous for the logging functions to return errors. Instead, log15 handles errors by making these guarantees to you: - Any log record containing an error will still be printed with the error explained to you as part of the log record. - Any log record containing an error will include the context key LOG15_ERROR, enabling you to easily (and if you like, automatically) detect if any of your logging calls are passing bad values. Understanding this, you might wonder why the Handler interface can return an error value in its Log method. Handlers are encouraged to return errors only if they fail to write their log records out to an external source like if the syslog daemon is not responding. This allows the construction of useful handlers which cope with those failures like the FailoverHandler. log15 is intended to be useful for library authors as a way to provide configurable logging to users of their library. Best practice for use in a library is to always disable all output for your logger by default and to provide a public Logger instance that consumers of your library can configure. Like so: Users of your library may then enable it if they like: The ability to attach context to a logger is a powerful one. Where should you do it and why? I favor embedding a Logger directly into any persistent object in my application and adding unique, tracing context keys to it. For instance, imagine I am writing a web browser: When a new tab is created, I assign a logger to it with the url of the tab as context so it can easily be traced through the logs. Now, whenever we perform any operation with the tab, we'll log with its embedded logger and it will include the tab title automatically: There's only one problem. What if the tab url changes? We could use log.Lazy to make sure the current url is always written, but that would mean that we couldn't trace a tab's full lifetime through our logs after the user navigate to a new URL. Instead, think about what values to attach to your loggers the same way you think about what to use as a key in a SQL database schema. If it's possible to use a natural key that is unique for the lifetime of the object, do so. But otherwise, log15's ext package has a handy RandId function to let you generate what you might call "surrogate keys" They're just random hex identifiers to use for tracing. Back to our Tab example, we would prefer to set up our Logger like so: Now we'll have a unique traceable identifier even across loading new urls, but we'll still be able to see the tab's current url in the log messages. For all Handler functions which can return an error, there is a version of that function which will return no error but panics on failure. They are all available on the Must object. For example: All of the following excellent projects inspired the design of this library: code.google.com/p/log4go github.com/op/go-logging github.com/technoweenie/grohl github.com/Sirupsen/logrus github.com/kr/logfmt github.com/spacemonkeygo/spacelog golang's stdlib, notably io and net/http https://xkcd.com/927/
Package lunk provides a set of tools for structured logging in the style of Google's Dapper or Twitter's Zipkin. When we consider a complex event in a distributed system, we're actually considering a partially-ordered tree of events from various services, libraries, and modules. Consider a user-initiated web request. Their browser sends an HTTP request to an edge server, which extracts the credentials (e.g., OAuth token) and authenticates the request by communicating with an internal authentication service, which returns a signed set of internal credentials (e.g., signed user ID). The edge web server then proxies the request to a cluster of web servers, each running a PHP application. The PHP application loads some data from several databases, places the user in a number of treatment groups for running A/B experiments, writes some data to a Dynamo-style distributed database, and returns an HTML response. The edge server receives this response and proxies it to the user's browser. In this scenario we have a number of infrastructure-specific events: This scenario also involves a number of events which have little to do with the infrastructure, but are still critical information for the business the system supports: There are a number of different teams all trying to monitor and improve aspects of this system. Operational staff need to know if a particular host or service is experiencing a latency spike or drop in throughput. Development staff need to know if their application's response times have gone down as a result of a recent deploy. Customer support staff need to know if the system is operating nominally as a whole, and for customers in particular. Product designers and managers need to know the effect of an A/B test on user behavior. But the fact that these teams will be consuming the data in different ways for different purposes does mean that they are working on different systems. In order to instrument the various components of the system, we need a common data model. We adopt Dapper's notion of a tree to mean a partially-ordered tree of events from a distributed system. A tree in Lunk is identified by its root ID, which is the unique ID of its root event. All events in a common tree share a root ID. In our photo example, we would assign a unique root ID as soon as the edge server received the request. Events inside a tree are causally ordered: each event has a unique ID, and an optional parent ID. By passing the IDs across systems, we establish causal ordering between events. In our photo example, the two database queries from the app would share the same parent ID--the ID of the event corresponding to the app handling the request which caused those queries. Each event has a schema of properties, which allow us to record specific pieces of information about each event. For HTTP requests, we can record the method, the request URI, the elapsed time to handle the request, etc. Lunk is agnostic in terms of aggregation technologies, but two use cases seem clear: real-time process monitoring and offline causational analysis. For real-time process monitoring, events can be streamed to a aggregation service like Riemann (http://riemann.io) or Storm (http://storm.incubator.apache.org), which can calculate process statistics (e.g., the 95th percentile latency for the edge server responses) in real-time. This allows for adaptive monitoring of all services, with the option of including example root IDs in the alerts (e.g., 95th percentile latency is over 300ms, mostly as a result of requests like those in tree XXXXX). For offline causational analysis, events can be written in batches to batch processing systems like Hadoop or OLAP databases like Vertica. These aggregates can be queried to answer questions traditionally reserved for A/B testing systems. "Did users who were show the new navbar view more photos?" "Did the new image optimization algorithm we enabled for 1% of views run faster? Did it produce smaller images? Did it have any effect on user engagement?" "Did any services have increased exception rates after any recent deploys?" &tc &tc By capturing the root ID of a particular web request, we can assemble a partially-ordered tree of events which were involved in the handling of that request. All events with a common root ID are in a common tree, which allows for O(M) retrieval for a tree of M events. To send a request with a root ID and a parent ID, use the Event-ID HTTP header: The header value is simply the root ID and event ID, hex-encoded and separated with a slash. If the event has a parent ID, that may be included as an optional third parameter. A server that receives a request with this header can use this to properly parent its own events. Each event has a set of named properties, the keys and values of which are strings. This allows aggregation layers to take advantage of simplifying assumptions and either store events in normalized form (with event data separate from property data) or in denormalized form (essentially pre-materializing an outer join of the normalized relations). Durations are always recorded as fractional milliseconds. Lunk currently provides two formats for log entries: text and JSON. Text-based logs encode each entry as a single line of text, using key="value" formatting for all properties. Event property keys are scoped to avoid collisions. JSON logs encode each entry as a single JSON object.
Package glick provides a simple plug-in environment. The central feature of glick is the Library which contains example types for the input and output of each API on the system. Each of these APIs can have a number of "actions" upon them, for example a file conversion API may have one action for each of the file formats to be convereted. Using the Run() method of glick.Library, a given API/Action combination runs the code in a function of Go type Plugin. Although it is easy to create your own plugins, there are three types built-in: Remote Procedure Calls (RPC), simple URL fetch (URL) and OS commands (CMD). A number of sub-packages simplify the use of third-party libraries when providing further types of plugin. The mapping of which plugin code to run occurs at three levels: 1) Intialisation and set-up code for the application will establish the glick.Library using glick.New(), then add API specifications using RegAPI(), it may also add the application's base plugins using RegPlugin(). 2) The base set-up can be extended and overloaded using a JSON format configuration description (probaly held in a file) by calling the Config() method of glick.Library. This configuration process is extensible, using the AddConfigurator() method - see the glick/glpie or glick/glkit sub-pakages for examples. 3) Which plugin to use can also be set-up or overloaded at runtime within Run(). Each call to a plugin includes a Context (as described in https://blog.golang.org/context). This context can contain for example user details, which could be matched against a database to see if that user should be directed to one plugin for a given action, rather than another. It could also be used to wrap every plugin call by a particular user with some other code, for example to log or meter activity.
Package csv decodes and encodes comma-separated values (CSV) files to and from arbitrary Go types. Because there are many different kinds of CSV files, this package implements the format described in RFC 4180. A CSV file may contain an optional header and zero or more records of one or more fields per record. The number of fields must be the same for each record and the optional header. The field separator is configurable and defaults to comma ',' (0x2C). Empty lines and lines starting with a comment character are ignored. The comment character is configurable as well and defaults to the number sign '#' (0x23). Records are separated by the newline character '\n' (0x0A) and the final record may or may not be followed by a newline. Carriage returns '\r' (0x0D) before newline characters are silently removed. White space is considered part of a field. Leading or trailing whitespace can optionally be trimmed when parsing a value. Fields may optionally be quoted in which case the surrounding double quotes '"' (0x22) are removed before processing. Inside a quoted field a double quote may be escaped by a preceeding second double quote which will be removed during parsing.
Package kvlog implement a key=value log formatter for the logrus logging package. It can optionally include the calling function name and line number, include constant values in every line and promote certain primary keys to the beginning of each line. All other keys are sorted into alphabetical order for easy scanning, with the human-readnable description at the end of each line. eg. This provides a format that's human-readable, yet automatically extracted by tools such as Splunk.
Package dcrjson provides infrastructure for working with Decred JSON-RPC APIs. When communicating via the JSON-RPC protocol, all requests and responses must be marshalled to and from the wire in the appropriate format. This package provides infrastructure and primitives to ease this process. This information is not necessary in order to use this package, but it does provide some intuition into what the marshalling and unmarshalling that is discussed below is doing under the hood. As defined by the JSON-RPC spec, there are effectively two forms of messages on the wire: Request Objects {"jsonrpc":"1.0","id":"SOMEID","method":"SOMEMETHOD","params":[SOMEPARAMS]} NOTE: Notifications are the same format except the id field is null. Response Objects {"result":SOMETHING,"error":null,"id":"SOMEID"} {"result":null,"error":{"code":SOMEINT,"message":SOMESTRING},"id":"SOMEID"} For requests, the params field can vary in what it contains depending on the method (a.k.a. command) being sent. Each parameter can be as simple as an int or a complex structure containing many nested fields. The id field is used to identify a request and will be included in the associated response. When working with streamed RPC transports, such as websockets, spontaneous notifications are also possible. As indicated, they are the same as a request object, except they have the id field set to null. Therefore, servers will ignore requests with the id field set to null, while clients can choose to consume or ignore them. Unfortunately, the original Bitcoin JSON-RPC API (and hence anything compatible with it) doesn't always follow the spec and will sometimes return an error string in the result field with a null error for certain commands. However, for the most part, the error field will be set as described on failure. To simplify the marshalling of the requests and responses, the MarshalCmd and MarshalResponse functions are provided. They return the raw bytes ready to be sent across the wire. Unmarshalling a received Request object is a two step process: This approach is used since it provides the caller with access to the additional fields in the request that are not part of the command such as the ID. Unmarshalling a received Response object is also a two step process: As above, this approach is used since it provides the caller with access to the fields in the response such as the ID and Error. This package provides the NewCmd function which takes a method (command) name and variable arguments. The function includes full checking to ensure the parameters are accurate according to provided method, however these checks are, obviously, run-time which means any mistakes won't be found until the code is actually executed. However, it is quite useful for user-supplied commands that are intentionally dynamic. External packages can and should implement types implementing Command for use with MarshalCmd/ParseParams. The command handling of this package is built around the concept of registered commands. This is true for the wide variety of commands already provided by the package, but it also means caller can easily provide custom commands with all of the same functionality as the built-in commands. Use the RegisterCmd function for this purpose. A list of all registered methods can be obtained with the RegisteredCmdMethods function. All registered commands are registered with flags that identify information such as whether the command applies to a chain server, wallet server, or is a notification along with the method name to use. These flags can be obtained with the MethodUsageFlags flags, and the method can be obtained with the CmdMethod function. To facilitate providing consistent help to users of the RPC server, this package exposes the GenerateHelp and function which uses reflection on registered commands or notifications to generate the final help text. In addition, the MethodUsageText function is provided to generate consistent one-line usage for registered commands and notifications using reflection. There are 2 distinct type of errors supported by this package: The first category of errors (type Error) typically indicates a programmer error and can be avoided by properly using the API. Errors of this type will be returned from the various functions available in this package. They identify issues such as unsupported field types, attempts to register malformed commands, and attempting to create a new command with an improper number of parameters. The specific reason for the error can be detected by type asserting it to a *dcrjson.Error and accessing the ErrorKind field. The second category of errors (type RPCError), on the other hand, are useful for returning errors to RPC clients. Consequently, they are used in the previously described Response type. This example demonstrates how to unmarshal a JSON-RPC response and then unmarshal the result field in the response to a concrete type.
Package log implements a std log compatible logging system that draws some inspiration from the Python logging module from Python's standard library. It supports multiple handlers, log levels, zero-allocation, scopes, custom formatting, and environment and runtime configuration. When not used to replace std log, the import should use the package name "analog" as in: Each Logger has a sequence of names that are used for filtering and context. Names are commonly attached as Loggers are passed into code of deeper context. The full import path of the package where a message is generated, and the short source file name and line number are added as the last 2 names for each message (applying any Msg.Skip in finding the right frame) when filtering is applied. The names are included at the end of each logging line. A sequence of rules are parsed from the environment variable with the key of EnvRules. Rules are separated by ",". Each rule is a substring of a log message name that or "*" to match any name. If there is no "=" in the rule, then all messages that match will be logged. If there is a "=", then a message must have the level following the "=", as parsed by Level.UnmarshalText or higher to be logged. Each rule is checked in order, and the last match takes precedence. This helps when you want to chain new rules on existing ones, you can always append to the end to override earlier rules. Some examples: GO_LOG=* Log everything, no matter the level GO_LOG=*=,*,hello=debug Log everything, except for any message containing a name with the substring "hello", which must be at least debug level. GO_LOG=something=info Handle messages at the info level or greater if they have a name containing "something". If no rule matches, the Logger's filter level is checked. The Default filter level is Warning. This means only messages with the level of Warning or higher will be logged, unless overridden by the specific Logger in use, or a rule from the environment matches. If the environment variable with the key EnvReportRules is not the empty string, each message logged with a previously unseen permutation of names will output a message to a standard library logger with the minimum level required to log that permutation. The message itself is then handled as usual. The same permutation will not be reported on again. This is useful to determine what logging names are in use, and to debug their reporting level thresholds.
Package couch implements a client for a CouchDB database. Version 0.1 focuses on basic operations, proper conflict management, error handling and replication. Not part of this version are attachment handling, general statistics and optimizations, change detection and creating views. Most of the features are accessible using the generic Do() function, though. Getting started: Every document in CouchDB has to be identifiable by a document id and a revision id. Two types already implement this interface called Identifiable: Doc and DynamicDoc. Doc can be used as an anonymous field in your own struct. DynamicDoc is a type alias for map[string]interface{}, use it when your documents have no implicit schema at all. To make code examples easier to follow, there will be no explicit error handling in these examples even though it's fully supported throughout the API. Insert() will create a new document if it doesn't have an id yet: After the operation the final id and revision id will be written back to p. That's why you can now just edit p and call Insert() again which will save the same document under a new revision. After this edit, p will contain the latest revision id. Note that it is possible that this second edit fails because someone else edited and saved the same document in the meantime. You will be notified of this in form of an error and you should then first retrieve the latest document revision to see the changes of this lost update: CouchDB doesn't edit documents in-place but adds a complete revision for each edit. That's why you will be correctly informed of any lost update. Because CouchDB supports multi-master replication of databases, it is possible that conflicts like the one described above can't be avoided. CouchDB is not going to interrupt replication because of a lost update. Let's say you have two instances running, maybe a central one and a mobile one and both are kept in sync by replication. Now let's assume you edit a document on your mobile DB and someone else edits the same document on the central DB. After you've come online again, you use bi-directional replication to sync the databases. CouchDB will now create a branch structure for your document, similar to version control systems. Your document has two conflicting revisions and in this case they can't necessarily be resolved automatically. This client helps you with a number of methods to resolve such an issue quickly. Read more about the conflict model http://docs.couchdb.org/en/latest/replication/conflicts.html Continuing with above example, replicate the database: Now, on the other database, edit the document (note that it has the same id there): Now edit the document on the first database. Retrieve it first to make sure it has the correct revision id: Now replicate anotherDB back to our first database: Now we have two conflicting versions of a document. Only you as the editor can decide whether "LatestAnna" or "AnotherAnna" is correct. To detect this conflict there are a number of methods. First, you can just ask a document: You probably want to have a look at the revisions in your preferred format, use Revisions() to unmarshal the revision data into a slice of a custom data type: Pick one of the revisions or create a new document to solve the conflict: That's it. You can detect conflicts like these throughout your database using: Errors returned by CouchDB will be converted into a Go error. Its regular Error() method will then return a combination of the shortform (e.g. bad_request) as well as the longer and more specific description. To be able to identify a specific error within your application, use ErrorType() to get the shortform only.
Package wav decodes and encodes wav audio files. The decoder is able to decode all wav audio formats (except extensible WAV formats), with any number of channels. These formats are: The encoder is capable of encoding any audio data -- but it currently will convert all data to 16-bit signed PCM on-the-fly before writing to a file. Ultimately this means regardless of what type of audio data you encode, it ends up as a 16-bit WAV file in the end. Future versions of this package will allow the encoder to output the same types as the decoder. Please refer to the WAV specification for in-depth details about its file format:
Package zappy implements the zappy block-based compression format. It aims for a combination of good speed and reasonable compression. Zappy is a format incompatible, API compatible fork of snappy[1]. The C++ snappy implementation is at [2]. The snappy compression is pretty good. Yet it has one problem built into its format definition[3] - the maximum length of a copy "instruction" is 64 bytes. For some specific usage patterns with long runs of repeated data, it turns out the compression is suboptimal. For example a 1:1000 "sparseness" 64kB bit index with only few set bits is compressed to about 3kB (about 1000 of 64B copy, 3 byte "instructions"). Zappy uses much less complicated format than snappy. Each encoded block begins with the uvarint-encoded[4] length of the decoded data, followed by a sequence of chunks. Chunks begin and end on byte boundaries. The chunk starts with a varint encoded number N: Compression rate is roughly the same as of snappy for the reference data set: Zappy has better RLE handling (1/1000+1 non zero bytes in each index): When compiled with CGO_ENABLED=1, zappy is now faster than Go snappy. Old=Go snappy, new=zappy: The package builds with CGO_ENABLED=0 as well, but the performance is worse. If a constraint 'purego' appears in the build constraints [5] then a pure Go version is built regardless of the $CGO_ENABLED value. ... referenced from the above documentation.
Package Authaus is an authentication and authorization system. Authaus brings together the following pluggable components: Any of these five components can be swapped out, and in fact the fourth, and fifth ones (Role Groups and User Store) are entirely optional. A typical setup is to use LDAP as an Authenticator, and Postgres as a Session, Permit, and Role Groups database. Your session database does not need to be particularly performant, since Authaus maintains an in-process cache of session keys and their associated tokens. Authaus was NOT designed to be a "Facebook Scale" system. The target audience is a system of perhaps 100,000 users. There is nothing fundamentally limiting about the API of Authaus, but the internals certainly have not been built with millions of users in mind. The intended usage model is this: Authaus is intended to be embedded inside your security system, and run as a standalone HTTP service (aka a REST service). This HTTP service CAN be open to the wide world, but it's also completely OK to let it listen only to servers inside your DMZ. Authaus only gives you the skeleton and some examples of HTTP responders. It's up to you to flesh out the details of your authentication HTTP interface, and whether you'd like that to face the world, or whether it should only be accessible via other services that you control. At startup, your services open an HTTP connection to the Authaus service. This connection will typically live for the duration of the service. For every incoming request, you peel off whatever authentication information is associated with that request. This is either a session key, or a username/password combination. Let's call it the authorization information. You then ask Authaus to tell you WHO this authorization information belongs to, as well as WHAT this authorization information allows the requester to do (ie Authentication and Authorization). Authaus responds either with a 401 (Unauthorized), 403 (Forbidden), or a 200 (OK) and a JSON object that tells you the identity of the agent submitting this request, as well the permissions that this agent posesses. It's up to your individual services to decide what to do with that information. It should be very easy to expose Authaus over a protocol other than HTTP, since Authaus is intended to be easy to embed. The HTTP API is merely an illustrative example. A `Session Key` is the long random number that is typically stored as a cookie. A `Permit` is a set of roles that has been granted to a user. Authaus knows nothing about the contents of a permit. It simply treats it as a binary blob, and when writing it to an SQL database, encodes it as base64. The interpretation of the permit is application dependent. Typically, a Permit will hold information such as "Allowed to view billing information", or "Allowed to paint your bathroom yellow". Authaus does have a built-in module called RoleGroupDB, which has its own interpretation of what a Permit is, but you do not need to use this. A `Token` is the result of a successful authentication. It stores the identity of a user, an expiry date, and a Permit. A token will usually be retrieved by a session key. However, you can also perform a once-off authentication, which also yields you a token, which you will typically throw away when you are finished with it. All public methods of the `Central` object are callable from multiple threads. Reader-Writer locks are used in all of the caching systems. The number of concurrent connections is limited only by the limits of the Go runtime, and the performance limits that are inherent to the simple reader-writer locks used to protect shared state. Authaus must be deployed as a single process (which implies running on a single logical machine). The sole reason why it must run on only one process and not more, is because of the state that lives inside the various Authaus caches. Were it not for these caches, then there would be nothing preventing you from running Authaus on as many machines as necessary. The cached state stored inside the Authaus server is: If you wanted to make Authaus runnable across multiple processes, then you would need to implement a cache invalidation system for these caches. Authaus makes no attempt to mitigate DOS attacks. The most sane approach in this domain seems to be this (http://security.stackexchange.com/questions/12101/prevent-denial-of-service-attacks-against-slow-hashing-functions). The password database (created via NewAuthenticationDB_SQL) stores password hashes using the scrypt key derivation system (http://www.tarsnap.com/scrypt.html). Internally, we store our hash in a format that can later be extended, should we wish to double-hash the passwords, etc. The hash is 65 bytes and looks like this: The first byte of the hash is a version number of the hash. The remaining 64 bytes are the salt and the hash itself. At present, only one version is supported, which is version 1. It consists of 32 bytes of salt, and 32 bytes of scrypt'ed hash, with scrypt parameters N=256 r=8 p=1. Note that the parameter N=256 is quite low, meaning that it is possible to compute this in approximately 1 millisecond (1,000,000 nanoseconds) on a 2009-era Intel Core i7. This is a deliberate tradeoff. On the same CPU, a SHA256 hash takes about 500 nanoseconds to compute, so we are still making it 2000 times harder to brute force the passwords than an equivalent system storing only a SHA256 salted hash. This discussion is only of relevance in the event that the password table is compromised. No cookie signing mechanism is implemented. Cookies are not presently transmitted with Secure:true. This must change. The LDAP Authenticator is extremely simple, and provides only one function: Authenticate a user against an LDAP system (often this means Active Directory, AKA a Windows Domain). It calls the LDAP "Bind" method, and if that succeeds for the given identity/password, then the user is considered authenticated. We take care not to allow an "anonymous bind", which many LDAP servers allow when the password is blank. The Session Database runs on Postgres. It stores a table of sessions, where each row contains the following information: When a permit is altered with Authaus, then all existing sessions have their permits altered transparently. For example, imagine User X is logged in, and his administrator grants him a new permission. User X does not need to log out and log back in again in order for his new permissions to be reflected. His new permissions will be available immediately. Similarly, if a password is changed with Authaus, then all sessions are invalidated. Do take note though, that if a password is changed through an external mechanism (such as with LDAP), then Authaus will have no way of knowing this, and will continue to serve up sessions that were authenticated with the old password. This is a problem that needs addressing. You can limit the number of concurrent sessions per user to 1, by setting MaxActiveSessions.ConfigSessionDB to 1. This setting may only be zero or one. Zero, which is the default, means an unlimited number of concurrent sessions per user. Authaus will always place your Session Database behind its own Session Cache. This session cache is a very simple single-process in-memory cache of recent sessions. The limit on the number of entries in this cache is hard-coded, and that should probably change. The Permit database runs on Postgres. It stores a table of permits, which is simply a 1:1 mapping from Identity -> Permit. The Permit is just an array of bytes, which we store base64 encoded, inside a text field. This part of the system doesn't care how you interpret that blob. The Role Group Database is an entirely optional component of Authaus. The other components of Authaus (Authenticator, PermitDB, SessionDB) do not understand your Permits. To them, a Permit is simply an arbitrary array of bytes. The Role Group Database is a component that adds a specific meaning to a permit blob. Let's see what that specific meaning looks like... The built-in Role Group Database interprets a permit blob as a string of 32-bit integer IDs: These 32-bit integer IDs refer to "role groups" inside a database table. The "role groups" table might look like this: The Role Group IDs use 32-bit indices, because we assume that you are not going to create more than 2^32 different role groups. The worst case we assume here is that of an automated system that creates 100,000 roles per day. Such a system would run for more than 100 years, given a 32-bit ID. These constraints are extraordinary, suggesting that we do not even need 32 bits, but could even get away with just a 16-bit group ID. However, we expect the number of groups to be relatively small. Our aim here, arbitrary though it may be, is to fit the permit and identity into a single ethernet packet, which one can reasonably peg at 1500 bytes. 1500 / 4 = 375. We assume that no sane human administrator will assign 375 security groups to any individual. We expect the number of groups assigned to any individual to be in the range of 1 to 20. This makes 375 a gigantic buffer. OAuth support in Authaus is limited to a very simple scenario: * You wish to allow your users to login using an OAuth service - thereby outsourcing the Authentication to that external service, and using it to populate the email address of your users. OAuth was developed in order to work with Microsoft Azure Active Directory, however it should be fairly easy to extend the code to be able to handle other OAuth providers. Inside the database are two tables related to OAuth: oauthchallenge: The challenge table holds OAuth sessions which have been started, and which are expected to either succeed or fail within the next few minutes. The default timeout for a challenge is 5 minutes. A challenge record is usually created the moment the user clicks on the "Sign in with Microsoft" button on your site, and it tracks that authentication attempt. oauthsession: The session table holds OAuth sessions which have successfully authenticated, and also the token that was retrieved by a successful authorization. If a token has expired, then it is refreshed and updated in-place, inside the oauthsession table. An OAuth login follows this sequence of events: 1. User clicks on a "Signin with X" button on your login page 2. A record is created in the oauthchallenge table, with a unique ID. This ID is a secret known only to the authaus server and the OAuth server. It is used as the `state` parameter in the OAuth login mechanism. 3. The HTTP call which prompts #2 return a redirect URL (eg via an HTTP 302 response), which redirects the user's browser to the OAuth website, so that the user can either grant or refuse access. If the user refuses, or fails to login, then the login sequence ends here. 4. Upon successful authorization with the OAuth system, the OAuth website redirects the user back to your website, to a URL such as example.com/auth/oauth/finish, and you'll typically want Authaus to handle this request directly (via HttpHandlerOAuthFinish). Authaus will extract the secrets from the URL, perform any validations necessary, and then move the record from the oauthchallenge table, into the oauthsession table. While 'moving' the record over, it will also add any additional information that was provided by the successful authentication, such as the token provided by the OAuth provider. 5. Authaus makes an API call to the OAuth system, to retrieve the email address and name of the person that just logged in, using the token just received. 6. If that email address does not exist inside authuserstore, then create a new user record for this identity. 7. Log the user into Authaus, by creating a record inside authsession, for the relevant identity. Inside the authsession table, store a link to the oauthsession record, so that there is a 1:1 link from the authsession table, to the oauthsession table (ie Authaus Session to OAuth Token). 8. Return an Authaus session cookie to the browser, thereby completing the login. Although we only use our OAuth token a single time, during login, to retrieve the user's email address and name, we retain the OAuth token, and so we maintain the ability to make other API calls on behalf of that user. This hasn't proven necessary yet, but it seems like a reasonable bit of future-proofing. See the guidelines at the top of all_test.go for testing instructions.