crawshaw.io/iox
Package iox contains I/O utilities. The primary concern of the package is managing and minimizing the use of file descriptors, an operating system resource which is often in short supply in high-concurrency servers. The two objects that help in this are the Filer and BufferFile. A filer manages a allotment of file descriptors, blocking on file creation until an old file closes and frees up a descriptor allotment. A BufferFile keeps a fraction of its contents in memory. If the number of bytes stored in a BufferFile is small, no file descriptor is ever used.
Readme
Package iox contains two Go objects of note: Filer and BufferFile.
https://godoc.org/crawshaw.io/iox
Managing file resources in highly-concurrent programs gets tricky. A process easily, even typically, has more in-flight goroutines than allowed file descriptors from the operating system. This requires programmers limit the number of open descriptors with some kind of throttle object.
An iox.Filer wraps the functions used to open file descriptors and makes sure it never opens more than some maximum (typically derived from the processes rlimit).
It wraps *os.File pointers in a new object which returns the file descriptor allotment to the Filer pool when Close is called.
A BufferFile is a file-like object that stores its first N bytes in memory, and the rest in a temporary file on disk.
It is designed for loads where the typical case fits in some small amount of memory, but the worst case requires more space than can be provisioned in RAM. (This usually means a server is handling tens to hundreds of thousands of simultaneous requests.)
BufferFile does not create its temporary backing file until its contents exceed the memory buffer, so the typical case does not require any file descriptors. Programs can begin (and usually complete) processing a request without ever blocking on file descriptors, meaning a server never runs into file descriptors as a bottleneck when processing a typical workload.
Install with:
go get crawshaw.io/iox
There are no version numbers yet, this package needs some time to bake.
FAQs
Package iox contains I/O utilities. The primary concern of the package is managing and minimizing the use of file descriptors, an operating system resource which is often in short supply in high-concurrency servers. The two objects that help in this are the Filer and BufferFile. A filer manages a allotment of file descriptors, blocking on file creation until an old file closes and frees up a descriptor allotment. A BufferFile keeps a fraction of its contents in memory. If the number of bytes stored in a BufferFile is small, no file descriptor is ever used.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
A hospital in Mobile, Alabama, agreed to a settlement in a landmark ransomware death lawsuit, but is now reportedly reconsidering the agreement and refusing to pay.
Security News
A new report explores how advancements in LLMs are enhancing cyber threats, including polymorphic malware, personalized spearphishing, and the risk of hijacking customer service bots.
Security News
ESLint has approved an RFC that adds support for TypeScript configuration files, which is aimed at improving the developer experience and recognizing changes in the evolving JavaScript ecosystem.