go-storage
![go storage dev](https://img.shields.io/matrix/beyondstorage@go-storage:matrix.org.svg?label=go-storage&logo=matrix)
A vendor-neutral storage library for Golang.
Vision
Write once, run on every storage service.
Goal
- Vendor agnostic
- Production ready
- High performance
Examples
package main
import (
"log"
"github.com/beyondstorage/go-storage/v4/services"
"github.com/beyondstorage/go-storage/v4/types"
_ "github.com/beyondstorage/go-service-fs/v3"
_ "github.com/beyondstorage/go-service-s3/v2"
_ "github.com/beyondstorage/go-service-gcs/v2"
_ "github.com/beyondstorage/go-service-azblob/v2"
_ "github.com/beyondstorage/go-service-xxx"
)
func main() {
store, err := services.NewStoragerFromString("s3://bucket_name/path/to/workdir")
if err != nil {
log.Fatalf("service init failed: %v", err)
}
n, err := store.Write("hello.txt", r, length)
n, err := store.Read("hello.txt", w)
o, err := store.Stat("hello.txt")
length, ok := o.GetContentLength()
it, err := store.List("path")
for {
o, err := it.Next()
if errors.Is(err, types.IteraoorDone) {
break
}
}
err = store.Delete("hello.txt")
}
More examples could be found at go-storage-example.
Features
Widely native services support
16 stable services that have passed all integration tests.
3 beta services that implemented required functions, but not passed integration tests.
4 alpha services that still under development.
More service ideas could be found at Service Integration Tracking.
Complete and easily extensible interface
Basic operations
- Metadata: get
Storager
metadata
meta := store.Metadata()
_ := meta.GetWorkDir()
_, ok := meta.GetWriteSizeMaximum()
- Read: read
Object
content
n, err := store.Read("path", w, pairs.WithOffset(1024), pairs.WithSize(2048))
- Write: write content into
Object
n, err := store.Write("path", r, 2048)
- Stat: get
Object
metadata or check existences
o, err := store.Stat("path")
if errors.Is(err, services.ErrObjectNotExist) {
}
length, ok := o.GetContentLength()
err := store.Delete("path")
- List: list
Object
in given prefix or dir
it, err := store.List("path")
for {
o, err := it.Next()
if err != nil && errors.Is(err, types.IteratorDone) {
}
length, ok := o.GetContentLength()
}
Extended operations
- Copy: copy a
Object
inside storager
err := store.(Copier).Copy(src, dst)
- Move: move a
Object
inside storager
err := store.(Mover).Move(src, dst)
- Reach: generate a public accessible url to an
Object
url, err := store.(Reacher).Reach("path")
o, err := store.(Direr).CreateDir("path")
Large file manipulation
- Multipart: allow doing multipart uploads
ms := store.(Multiparter)
o, err := ms.CreateMultipart("path")
n, part, err := ms.WriteMultipart(o, r, 1024, 1)
err := ms.CompleteMultipart(o, []*Part{part})
- Append: allow appending to an object
as := store.(Appender)
o, err := as.CreateAppend("path")
n, err := as.WriteAppend(o, r, 1024)
err = as.CommitAppend(o)
- Block: allow combining an object with block ids
bs := store.(Blocker)
o, err := bs.CreateBlock("path")
n, err := bs.WriteBlock(o, r, 1024, "id-abc")
err := bs.CombineBlock(o, []string{"id-abc"})
- Page: allow doing random writes
ps := store.(Pager)
o, err := ps.CreatePage("path")
n, err := ps.WritePage(o, r, 1024, 2048)
Comprehensive metadata
Global object metadata
id
: unique key in servicename
: relative path towards service's work dirmode
: object mode can be a combination of read
, dir
, part
and moreetag
: entity tag as defined in rfc2616content-length
: object's content size.content-md5
: md5 digest as defined in rfc2616content-type
: media type as defined in rfc2616last-modified
: object's last updated time.
System object metadata
Service system object metadata like storage-class
and so on.
o, err := store.Stat("path")
om := s3.GetObjectSystemMetadata(o)
_ = om.StorageClass
_ = om.ServerSideEncryptionCustomerAlgorithm
Strong Typing Everywhere
Self maintained codegen definitions helps to generate all our APIs, pairs and metadata.
Generated pairs which can be used as API optional arguments.
func WithContentMd5(v string) Pair {
return Pair{
Key: "content_md5",
Value: v,
}
}
Generated object metadata which can be used to get content md5 from object.
func (o *Object) GetContentMd5() (string, bool) {
o.stat()
if o.bit&objectIndexContentMd5 != 0 {
return o.contentMd5, true
}
return "", false
}
Server-Side Encrypt
Server-Side Encrypt supports via system pair and system metadata, and we can use Default Pairs to simplify the job.
func NewS3SseC(key []byte) (types.Storager, error) {
defaultPairs := s3.DefaultStoragePairs{
Write: []types.Pair{
s3.WithServerSideEncryptionCustomerAlgorithm(s3.ServerSideEncryptionAes256),
s3.WithServerSideEncryptionCustomerKey(key),
},
Read: []types.Pair{
s3.WithServerSideEncryptionCustomerAlgorithm(s3.ServerSideEncryptionAes256),
s3.WithServerSideEncryptionCustomerKey(key),
}}
return s3.NewStorager(..., s3.WithDefaultStoragePairs(defaultPairs))
}