github.com/jimsmart/peanut
Package peanut writes tagged data structs to disk in a variety of formats. Its primary purpose is to provide a single consistent interface for easy, ceremony-free persistence of record-based struct data. Each distinct struct type is written to an individual file (or table), automatically created, each named according to the name of the struct. Field/column names in each file/table are derived from struct tags. All writers use the same tags. Currently supported formats are CSV, TSV, Excel (.xlsx), JSON Lines (JSONL), and SQLite. Additional writers are also provided to assist with testing and debugging. Mutiple writers can be combined using MultiWriter. All writers have the same basic interface: a Write method, that can take any appropriately tagged struct; a Close method, which should be called to successfully complete writing; and a Cancel method, which should be called to abort writing and clean-up, in the event of an error or cancellation. It is safe to make mulltiple calls to Cancel, and it is safe to call Close after having previously called Cancel. All writers output their files atomically — that is to say: all output is written to a temporary location and only moved to the final output location when Close is called, meaning the output folder never contains any partially written files. Structs to be used with peanut must have appropriately tagged fields, for example: Fields without tags do not get written as output. First create a writer, for example: Next, write some records to it: When successfully completed: Or, to abort the whole operation in the event of an error or cancellation while writing records: Multiple writers can be combined using MultiWriter: Here w will write records to CSV files, Excel files, and a logger. Behaviour is undefined for types with the same name but in different packages, such as package1.Foo and package2.Foo. Supported datatypes for struct fields: string, bool, float32, float64, int, int8, int16, int32, int64, uint, uint8, uint16, uint32, uint64. Pointer following and nested structs are currently unsupported. Tagging a field that has an unsupported datatype will result in a error when Write is called.
Readme
peanut is a Go package to write tagged data structs to disk in a variety of formats.
Its primary purpose is to provide a single consistent interface for easy, ceremony-free persistence of record-based struct data.
Each distinct struct type is written to an individual file (or table), automatically created, each named according to the name of the struct. Field/column names in each file/table are derived from struct tags. All writers use the same tags.
Currently supported formats are CSV, TSV, Excel (.xlsx), JSON Lines (JSONL), and SQLite. Additional writers are also provided to assist with testing and debugging. Mutiple writers can be combined using MultiWriter.
All writers perform atomic file operations, writing data to a temporary location and moving it to the final output location when Close is called.
When building an app or tool that needs to output data consisting of multiple different record types to disk, perhaps with requirements that change over time (whether during development or after initial deployment), perhaps requiring multiple output formats (during development/testing, or as final output) — is where peanut might be 'the right tool for the job'.
Ideal for use as an output solution for, e.g. data conversion tools, part of an ETL pipeline, data-acquistion or extraction tools/apps, web-scrapers, structured logging, persistence of captured data/metadata/events, job reporting, etc. Whether building an ad-hoc tool as a quick hack, or as part of a bigger, more serious project.
peanut initially evolved as part of a larger closed-source project, is tried and tested, and production-ready.
Get the package:
go get github.com/jimsmart/peanut
Use the package within your code:
import "github.com/jimsmart/peanut"
All peanut writers implement this interface:
type Writer interface {
Write(r interface{}) error
Close() error
Cancel() error
}
peanut.Writer to use.Write() to write records, repeating until done.Close() to finish.See GoDocs.
GoDocs https://godoc.org/github.com/jimsmart/peanut
To run the tests execute go test inside the project folder.
For a full coverage report, try:
go test -coverprofile=coverage.out && go tool cover -html=coverage.out
Package peanut is copyright 2020-2022 by Jim Smart and released under the BSD 3-Clause License.
FAQs
Package peanut writes tagged data structs to disk in a variety of formats. Its primary purpose is to provide a single consistent interface for easy, ceremony-free persistence of record-based struct data. Each distinct struct type is written to an individual file (or table), automatically created, each named according to the name of the struct. Field/column names in each file/table are derived from struct tags. All writers use the same tags. Currently supported formats are CSV, TSV, Excel (.xlsx), JSON Lines (JSONL), and SQLite. Additional writers are also provided to assist with testing and debugging. Mutiple writers can be combined using MultiWriter. All writers have the same basic interface: a Write method, that can take any appropriately tagged struct; a Close method, which should be called to successfully complete writing; and a Cancel method, which should be called to abort writing and clean-up, in the event of an error or cancellation. It is safe to make mulltiple calls to Cancel, and it is safe to call Close after having previously called Cancel. All writers output their files atomically — that is to say: all output is written to a temporary location and only moved to the final output location when Close is called, meaning the output folder never contains any partially written files. Structs to be used with peanut must have appropriately tagged fields, for example: Fields without tags do not get written as output. First create a writer, for example: Next, write some records to it: When successfully completed: Or, to abort the whole operation in the event of an error or cancellation while writing records: Multiple writers can be combined using MultiWriter: Here w will write records to CSV files, Excel files, and a logger. Behaviour is undefined for types with the same name but in different packages, such as package1.Foo and package2.Foo. Supported datatypes for struct fields: string, bool, float32, float64, int, int8, int16, int32, int64, uint, uint8, uint16, uint32, uint64. Pointer following and nested structs are currently unsupported. Tagging a field that has an unsupported datatype will result in a error when Write is called.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
The June TC39 meeting wrapped up this week with eight proposals moving on to the next stage. Here's a quick roundup of the features that the committee approved to advance.
Security News
Cyber extortion in the US and Canada hit record levels in 2023, with ransomware attacks surging and median ransom demands skyrocketing, though fewer companies are choosing to pay ransoms.
Security News
Ecma TC39 is meeting this week and has moved key ECMAScript proposals forward, advancing Deferred Import Evaluation, Error.isError(), RegExp Escaping, and Promise.try to the next stages.