github.com/ClickHouse/clickhouse-go/v2
Golang SQL database client for ClickHouse.
database/sql (slower than native interface!)database/sql supports http protocol for transport. (Experimental)database/sql use begin->prepare->(in loop exec)->commit)Support for the ClickHouse protocol advanced features using Context:
The client is tested against the currently supported versions of ClickHouse
| Client Version | Golang Versions |
|---|---|
| => 2.0 <= 2.2 | 1.17, 1.18 |
| >= 2.3 | 1.18.4+, 1.19 |
| >= 2.14 | 1.20, 1.21 |
| >= 2.19 | 1.21, 1.22 |
| >= 2.28 | 1.22, 1.23 |
| >= 2.29 | 1.21, 1.22, 1.23 |
https://clickhouse.com/docs/en/integrations/go
clickhouse interface (formally native interface) conn, err := clickhouse.Open(&clickhouse.Options{
Addr: []string{"127.0.0.1:9000"},
Auth: clickhouse.Auth{
Database: "default",
Username: "default",
Password: "",
},
DialContext: func(ctx context.Context, addr string) (net.Conn, error) {
dialCount++
var d net.Dialer
return d.DialContext(ctx, "tcp", addr)
},
Debug: true,
Debugf: func(format string, v ...any) {
fmt.Printf(format+"\n", v...)
},
Settings: clickhouse.Settings{
"max_execution_time": 60,
},
Compression: &clickhouse.Compression{
Method: clickhouse.CompressionLZ4,
},
DialTimeout: time.Second * 30,
MaxOpenConns: 5,
MaxIdleConns: 5,
ConnMaxLifetime: time.Duration(10) * time.Minute,
ConnOpenStrategy: clickhouse.ConnOpenInOrder,
BlockBufferSize: 10,
MaxCompressionBuffer: 10240,
ClientInfo: clickhouse.ClientInfo{ // optional, please see Client info section in the README.md
Products: []struct {
Name string
Version string
}{
{Name: "my-app", Version: "0.1"},
},
},
})
if err != nil {
return err
}
return conn.Ping(context.Background())
database/sql interfaceconn := clickhouse.OpenDB(&clickhouse.Options{
Addr: []string{"127.0.0.1:9999"},
Auth: clickhouse.Auth{
Database: "default",
Username: "default",
Password: "",
},
TLS: &tls.Config{
InsecureSkipVerify: true,
},
Settings: clickhouse.Settings{
"max_execution_time": 60,
},
DialTimeout: time.Second * 30,
Compression: &clickhouse.Compression{
Method: clickhouse.CompressionLZ4,
},
Debug: true,
BlockBufferSize: 10,
MaxCompressionBuffer: 10240,
ClientInfo: clickhouse.ClientInfo{ // optional, please see Client info section in the README.md
Products: []struct {
Name string
Version string
}{
{Name: "my-app", Version: "0.1"},
},
},
})
conn.SetMaxIdleConns(5)
conn.SetMaxOpenConns(10)
conn.SetConnMaxLifetime(time.Hour)
zstd, lz4, gzip, deflate, br. If set to true, lz4 will be used.gzip - -2 (Best Speed) to 9 (Best Compression)deflate - -2 (Best Speed) to 9 (Best Compression)br - 0 (Best Speed) to 11 (Best Compression)zstd, lz4 - ignored/. This value will be pass a part of client info. e.g. client_info_product=my_app/1.0,my_module/0.1 More details in Client info section.SSL/TLS parameters:
Example:
clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60
Note: using HTTP protocol is possible only with database/sql interface.
The native format can be used over the HTTP protocol. This is useful in scenarios where users need to proxy traffic e.g. using ChProxy or via load balancers.
This can be achieved by modifying the DSN to specify the HTTP protocol.
http://host1:8123,host2:8123/database?dial_timeout=200ms&max_execution_time=60
Alternatively, use OpenDB and specify the interface type.
conn := clickhouse.OpenDB(&clickhouse.Options{
Addr: []string{"127.0.0.1:8123"},
Auth: clickhouse.Auth{
Database: "default",
Username: "default",
Password: "",
},
Settings: clickhouse.Settings{
"max_execution_time": 60,
},
DialTimeout: 30 * time.Second,
Compression: &clickhouse.Compression{
Method: clickhouse.CompressionLZ4,
},
Protocol: clickhouse.HTTP,
})
HTTP proxy can be set in the DSN string by specifying the http_proxy parameter.
(make sure to URL encode the proxy address)
http://host1:8123,host2:8123/database?dial_timeout=200ms&max_execution_time=60&http_proxy=http%3A%2F%2Fproxy%3A8080
If you are using clickhouse.OpenDB, set the HTTProxy field in the clickhouse.Options.
An alternative way is to enable proxy by setting the HTTP_PROXY (for HTTP) or HTTPS_PROXY (for HTTPS) environment variables.
See more details in the Go documentation.
ZSTD/LZ4 compression is supported over native and http protocols. This is performed column by column at a block level and is only used for inserts. Compression buffer size is set as MaxCompressionBuffer option.
If using Open via the std interface and specifying a DSN, compression can be enabled via the compress flag. Currently, this is a boolean flag which enables LZ4 compression.
Other compression methods will be added in future PRs.
At a low level all client connect methods (DSN/OpenDB/Open) will use the Go tls package to establish a secure connection. The client knows to use TLS if the Options struct contains a non-nil tls.Config pointer.
Setting secure in the DSN creates a minimal tls.Config struct with only the InsecureSkipVerify field set (either true or false). It is equivalent to this code:
conn := clickhouse.OpenDB(&clickhouse.Options{
...
TLS: &tls.Config{
InsecureSkipVerify: false
}
...
})
This minimal tls.Config is normally all that is necessary to connect to the secure native port (normally 9440) on a ClickHouse server. If the ClickHouse server does not have a valid certificate (expired, wrong host name, not signed by a publicly recognized root Certificate Authority), InsecureSkipVerify can be to true, but that is strongly discouraged.
If additional TLS parameters are necessary the application code should set the desired fields in the tls.Config struct. That can include specific cipher suites, forcing a particular TLS version (like 1.2 or 1.3), adding an internal CA certificate chain, adding a client certificate (and private key) if required by the ClickHouse server, and most of the other options that come with a more specialized security setup.
To connect using HTTPS either:
Use https in your dsn string e.g.
https://host1:8443,host2:8443/database?dial_timeout=200ms&max_execution_time=60
Specify the interface type as HttpsInterface e.g.
conn := clickhouse.OpenDB(&clickhouse.Options{
Addr: []string{"127.0.0.1:8443"},
Auth: clickhouse.Auth{
Database: "default",
Username: "default",
Password: "",
},
Protocol: clickhouse.HTTP,
})
Clickhouse-go implements client info as a part of language client specification. client_name for native protocol and HTTP User-Agent header values are provided with the exact client info string.
Users can extend client options with additional product information included in client info. This might be useful for analysis on a server side.
Order is the highest abstraction to the lowest level implementation left to right.
Usage examples for native API and database/sql are provided.
Asynchronous insert is supported via dedicated AsyncInsert method. This allows to insert data with a non-blocking call.
Effectively, it controls a async_insert setting for the query.
Using native protocol, asynchronous insert does not support batching. It means, only inline query data is supported. Please see an example here.
HTTP protocol supports batching. It can be enabled by setting async_insert when using standard Prepare method.
For more details please see asynchronous inserts documentation.
Available options:
| V1 (READ) | V2 (READ) std | V2 (READ) clickhouse API |
|---|---|---|
| 1.218s | 924.390ms | 675.721ms |
| V1 (WRITE) | V2 (WRITE) std | V2 (WRITE) clickhouse API | V2 (WRITE) by column |
|---|---|---|---|
| 1.899s | 1.177s | 699.203ms | 661.973ms |
go get -u github.com/ClickHouse/clickhouse-go/v2
database/sql interfaceVersions of this client >=2.3.x utilise ch-go for their low level encoding/decoding. This low lever client provides a high performance columnar interface and should be used in performance critical use cases. This client provides more familar row orientated and database/sql semantics at the cost of some performance.
Both clients are supported by ClickHouse.
Database client/clients:
database/sql-like API)Insert collectors:
FAQs
Unknown package
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
In this segment of the Risky Business podcast, Feross Aboukhadijeh and Patrick Gray discuss the challenges of tracking malware discovered in open source softare.
Research
Security News
A threat actor's playbook for exploiting the npm ecosystem was exposed on the dark web, detailing how to build a blockchain-powered botnet.
Security News
NVD’s backlog surpasses 20,000 CVEs as analysis slows and NIST announces new system updates to address ongoing delays.