github.com/jochasinga/requests

requests

The simplest and functional HTTP requests in Go.

Introduction

Ever find yourself going back and forth between net/http and io docs and your code while making HTTP calls? Requests takes care of that by abstracting several types on both ends to just Request and Response--just the way it should be. Find out how or jump right in to examples.

Purpose

I need an HTTP request package that:

• is atomic--All HTTP request configurations should be set, intuitively, on the request only.
• wraps useful channels-ridden asynchronous patterns.
• has helper functions like marshaling and posting JSON.
• stays true to net/http APIs.
• is idiomatic Go.

Usage

the following are the core differences from the standard net/http package.

Functional Options

requests employs functional options as optional parameters, this approach being idiomatic, clean, and makes a friendly, extensible API. This pattern is adopted after feedback from the Go community.

jsontype := func(r *requests.Request) {
        r.Header.Add("content-type", "application/json")
}
res, err := requests.Get("http://example.com", jsontype)

Embedded Standard Types

requests uses custom Request and Response types to embed standard http.Request, http.Response, and http.Client in order to insert helper methods, make configuring options atomic, and handle asynchronous errors.

The principle is, the caller should be able to set all the configurations on the "Request" instead of doing it on the client, transport, vice versa. For instance, Timeout can be set on the Request.

timeout := func(r *requests.Request) {

        // Set Timeout on *Request instead of *http.Client
        r.Timeout = time.Duration(5) * time.Second
}
res, err := requests.Get("http://example.com", timeout)
if err != nil {
        panic(err)
}

// Helper method
htmlStr := res.String()

Also, where http.Transport was normally needed to set control over proxies, TLS configuration, keep-alives, compression, and other settings, now everything is handled by Request.

tlsConfig := func(r *requests.Request) {
        r.TLSClientConfig = &tls.Config{RootCAs: x509.NewCertPool()}
        r.DisableCompression = true
}
res, _ := requests.Get("http://example.com", tlsConfig)

See Types and Methods for more information.

Asynchronous APIs

requests provides the following abstractions around sending HTTP requests in goroutines:

• requests.GetAsync
• requests.PostAsync
• requests.Pool

All return a receive-only channel on which *requests.Response can be waited on.

rc, err := requests.GetAsync("http://httpbin.org/get")
if err != nil {
        panic(err)
}
res := <-rc

// Handle connection errors.
if res.Error != nil {
        panic(res.Error)
}

// Helper method
content := res.Bytes()

See Async and Handling Async Errors for more usage information.

Install

go get github.com/jochasinga/requests

Testing

requests uses Go standard testing package. Run this in the project's directory:

go test -v -cover

Examples

requests.Get

Sending a basic GET request is straightforward.

res, err := requests.Get("http://httpbin.org/get")
if err != nil {
        panic(err)
}

fmt.Println(res.StatusCode)  // 200

To send additional data, such as a query parameter, or set basic authorization header or content type, use functional options.

// Add a query parameter.
addFoo := func(r *requests.Request) {
        r.Params.Add("foo", "bar")
}

// Set basic username and password.
setAuth := func(r *requests.Request) {
        r.SetBasicAuth("user", "pass")
}

// Set the Content-Type.
setMime := func(r *requests.Request) {
        r.Header.Add("content-type", "application/json")
}

// Pass as parameters to the function.
res, err := requests.Get("http://httpbin.org/get", addFoo, setAuth, setMime)

Or configure everything in one function.

opts := func(r *requests.Request) {
        r.Params.Add("foo", "bar")
        r.SetBasicAuth("user", "pass")
        r.Header.Add("content-type", "application/json")
}
res, err := requests.Get("http://httpbin.org/get", opts)

requests.Post

Send POST requests with specified bodyType and body.

res, err := requests.Post("https://httpbin.org/post", "image/jpeg", &buf)

It also accepts variadic number of functional options:

notimeout := func(r *requests.Request) {
        r.Timeout = 0
}
res, err := requests.Post("https://httpbin.org/post", "application/json", &buf, notimeout)

requests.PostJSON

Encode your map or struct data as JSON and set bodyType to application/json implicitly.

first := map[string][]string{
        "foo": []string{"bar", "baz"},
}
second := struct {
        Foo []string `json:"foo"`
}{[]string{"bar", "baz"}}

payload := map[string][]interface{}{
        "twins": {first, second}
}

res, err := requests.PostJSON("https://httpbin.org/post", payload)

Other Verbs

HEAD, PUT, PATCH, DELETE, and OPTIONS are supported. See the doc for more info.

Async

requests.GetAsync

After parsing all the options, GetAsync spawns a goroutine to send a GET request and return <-chan *Response immediately on which *Response can be waited.

timeout := func(r *requests.Request) {
        r.Timeout = time.Duration(5) * time.Second
}

rc, err := requests.GetAsync("http://golang.org", timeout)
if err != nil {
        panic(err)
}

// Do other things...

// Block and wait
res := <-rc

// Handle a "reject" with Error field.
if res.Error != nil {
	panic(res.Error)
}

fmt.Println(res.StatusCode)  // 200

select can be used to poll many channels asynchronously like normal.

res1, _ := requests.GetAsync("http://google.com")
res2, _ := requests.GetAsync("http://facebook.com")
res3, _ := requests.GetAsync("http://docker.com")

for i := 0; i < 3; i++ {
        select {
    	case r1 := <-res1:
    		fmt.Println(r1.StatusCode)
    	case r2 := <-res2:
    		fmt.Println(r2.StatusCode)
    	case r3 := <-res3:
    		fmt.Println(r3.StatusCode)
    	}
}

requests.Pool is recommended for collecting concurrent responses from multiple requests.

requests.PostAsync

An asynchronous counterpart of requests.Post.

query := bytes.NewBufferString(`{
        "query" : {
            "term" : { "user" : "poco" }
        }
}`)

// Sending query to Elasticsearch server
rc, err := PostAsync("http://localhost:9200/users/_search", "application/json", query)
if err != nil {
        panic(err)
}
resp := <-rc
if resp.Error != nil {
        panic(resp.Error)
}
result := resp.JSON()

requests.Pool

Contains a Responses field of type chan *Response with variable-sized buffer specified in the constructor. Pool is used to collect in-bound responses sent from numbers of goroutines corresponding to the number of URLs provided in the slice.

urls := []string{
        "http://golang.org",
        "http://google.com",
        "http://docker.com",
        "http://medium.com",
        "http://example.com",
        "http://httpbin.org/get",
        "https://en.wikipedia.org",
}

// Create a pool with the maximum buffer size.
p := requests.NewPool(10)
opts := func(r *requests.Request) {
        r.Header.Set("user-agent", "GoBot(http://example.org/"))
        r.Timeout = time.Duration(10) * time.Second
}

results, err := p.Get(urls, opts)

// An error is returned when an attempt to construct a
// request fails, probably from a malformed URL.
if err != nil {
        panic(err)
}
for res := range results {
        if res.Error != nil {
                panic(res.Error)
        }
        fmt.Println(res.StatusCode)
}

You may want to ignore errors from malformed URLs instead of handling each of them, for instance, when crawling mass URLs.

To suppress the errors from being returned, either thrown away the error with _ or set the IgnoreBadURL field to true, which suppress all internal errors from crashing the pool:

results, err := p.Get(urls, func(r *requests.Request) {
        r.IgnoreBadURL = true
})

Pool.Responses channel is closed internally when all the responses are sent.

Types and Methods

requests.Request

It has embedded types *http.Request and *http.Client, making it an atomic type to pass into a functional option. It also contains field Params, which has the type url.Values. Use this field to add query parameters to your URL. Currently, parameters in Params will replace all the existing query string in the URL.

addParams := func(r *requests.Request) {
        r.Params = url.Values{
	        "name" : { "Ava", "Sanchez", "Poco" },
        }
}

// "q=cats" will be replaced by the new query string
res, err := requests.Get("https://httpbin.org/get?q=cats", addParams)

requests.Response

It has embedded type *http.Response and provides extra byte-like helper methods such as:

• Len() int
• String() string
• Bytes() []byte
• JSON() []byte

These methods will return an equivalent of nil for each return type if a certain condition isn't met. For instance:

res, _ := requests.Get("http://somecoolsite.io")
fmt.Println(res.JSON())

If the response from the server does not specify Content-Type as "application/json", res.JSON() will return an empty bytes slice. It does not panic if the content type is empty.

These methods close the response's body automatically.

Another helper method, ContentType, is used to get the media type in the response's header, and can be used with the helper methods to determine the type before reading the output.

mime, _, err := res.ContentType()
if err != nil {
        panic(err)
}

switch mime {
case "application/json":
        fmt.Println(res.JSON())
case "text/html", "text/plain":
        fmt.Println(res.String())
default:
        fmt.Println(res.Bytes())
}

Handling Async Errors

requests.Response also has an Error field which will contain any error caused in the goroutine within requests.GetAsync and carries it downstream for proper handling (Think reject in Promise but more straightforward in Go-style).

rc, err := requests.GetAsync("http://www.docker.io")

// This error is returned before the goroutine i.e. malformed URL.
if err != nil {
        panic(err)
}

res := <-rc

// This connection error is "attached" to the response.
if res.Error != nil {
	panic(res.Error)
}

fmt.Println(res.StatusCode)

Response.Error is default to nil when there is no error or when the response is being retrieved from a synchronous function.

HTTP Test Servers

Check out my other project relay, useful test proxies and round-robin switchers for end-to-end HTTP tests.

Contribution

Yes, please fork away.

Disclaimer

To support my ends in NYC and help me push commits, please consider to fuel me with quality 🍵 or 🌟 this repo for spiritual octane.
Reach me at @jochasinga.