github.com/databricks/databricks-sql-go
Package dbsql implements the go driver to Databricks SQL Clients should use the database/sql package in conjunction with the driver: Use sql.Open() to create a database handle via a data source name string: The DSN format is: Supported optional connection parameters can be specified in param=value and include: Supported optional session parameters can be specified in param=value and include: Use sql.OpenDB() to create a database handle via a new connector object created with dbsql.NewConnector(): Supported functional options include: Cancelling a query via context cancellation or timeout is supported. Use the driverctx package under driverctx/ctx.go to add CorrelationId and ConnId to the context. CorrelationId and ConnId makes it convenient to parse and create metrics in logging. **Connection Id** Internal id to track what happens under a connection. Connections can be reused so this would track across queries. **Query Id** Internal id to track what happens under a query. Useful because the same query can be used with multiple connections. **Correlation Id** External id, such as request ID, to track what happens under a request. Useful to track multiple connections in the same request. Use the logger package under logger.go to set up logging (from zerolog). By default, logging level is `warn`. If you want to disable logging, use `disabled`. The user can also utilize Track() and Duration() to custom log the elapsed time of anything tracked. The result log may look like this: Use the driverctx package under driverctx/ctx.go to add callbacks to the query context to receive the connection id and query id. Passing parameters to a query is supported when run against servers with version DBR 14.1. For complex types, you can specify the SQL type using the dbsql.Parameter type field. If this field is set, the value field MUST be set to a string. The Go driver now supports staging operations. In order to use a staging operation, you first must update the context with a list of folders that you are allowing the driver to access. After doing so, you can execute staging operations using this context using the exec context. There are three error types exposed via dbsql/errors Each type has a corresponding sentinel value which can be used with errors.Is() to determine if one of the types is present in an error chain. Example usage: See the documentation for dbsql/errors for more information. The driver supports the ability to retrieve Apache Arrow record batches. To work with record batches it is necessary to use sql.Conn.Raw() to access the underlying driver connection to retrieve a driver.Rows instance. The driver exposes two public interfaces for working with record batches from the rows sub-package: The driver.Rows instance retrieved using Conn.Raw() can be converted to a Databricks Rows instance via a type assertion, then use GetArrowBatches() to retrieve a batch iterator. If the ArrowBatchIterator is not closed it will leak resources, such as the underlying connection. Calling code must call Release() on records returned by DBSQLArrowBatchIterator.Next(). Example usage: ================================== Databricks Type --> Golang Type ================================== BOOLEAN --> bool TINYINT --> int8 SMALLINT --> int16 INT --> int32 BIGINT --> int64 FLOAT --> float32 DOUBLE --> float64 VOID --> nil STRING --> string DATE --> time.Time TIMESTAMP --> time.Time DECIMAL(p,s) --> sql.RawBytes BINARY --> sql.RawBytes ARRAY<elementType> --> sql.RawBytes STRUCT --> sql.RawBytes MAP<keyType, valueType> --> sql.RawBytes INTERVAL (year-month) --> string INTERVAL (day-time) --> string For ARRAY, STRUCT, and MAP types, sql.Scan can cast sql.RawBytes to JSON string, which can be unmarshalled to Golang arrays, maps, and structs. For example: May generate the following row:
Readme
This repo contains a Databricks SQL Driver for Go's database/sql package. It can be used to connect and query Databricks clusters and SQL Warehouses.
See doc.go for full documentation or the Databrick's documentation for SQL Driver for Go.
import (
"context"
"database/sql"
_ "github.com/databricks/databricks-sql-go"
)
db, err := sql.Open("databricks", "token:********@********.databricks.com:443/sql/1.0/endpoints/********")
if err != nil {
panic(err)
}
defer db.Close()
rows, err := db.QueryContext(context.Background(), "SELECT 1")
defer rows.Close()
Additional usage examples are available here.
The DSN format is:
token:[your token]@[Workspace hostname]:[Port number][Endpoint HTTP Path]?param=value
You can set query timeout value by appending a timeout query parameter (in seconds) and you can set max rows to retrieve per network request by setting the maxRows query parameter:
token:[your token]@[Workspace hostname]:[Port number][Endpoint HTTP Path]?timeout=1000&maxRows=1000
You can turn on Cloud Fetch to increase the performance of extracting large query results by fetching data in parallel via cloud storage (more info here). To turn on Cloud Fetch, append useCloudFetch=true. You can also set the number of concurrently fetching goroutines by setting the maxDownloadThreads query parameter (default is 10):
token:[your token]@[Workspace hostname]:[Port number][Endpoint HTTP Path]?useCloudFetch=true&maxDownloadThreads=3
You can also connect with a new connector object. For example:
import (
"database/sql"
_ "github.com/databricks/databricks-sql-go"
)
connector, err := dbsql.NewConnector(
dbsql.WithServerHostname(<Workspace hostname>),
dbsql.WithPort(<Port number>),
dbsql.WithHTTPPath(<Endpoint HTTP Path>),
dbsql.WithAccessToken(<your token>)
)
if err != nil {
log.Fatal(err)
}
db := sql.OpenDB(connector)
defer db.Close()
View doc.go or connector.go to understand all the functional options available when creating a new connector object.
We use golangci-lint as the lint tool. If you use vs code, just add the following settings:
{
"go.lintTool": "golangci-lint",
"go.lintFlags": [
"--fast"
]
}
go test
If you find any issues, feel free to create an issue or send a pull request directly.
See CONTRIBUTING.md
FAQs
Package dbsql implements the go driver to Databricks SQL Clients should use the database/sql package in conjunction with the driver: Use sql.Open() to create a database handle via a data source name string: The DSN format is: Supported optional connection parameters can be specified in param=value and include: Supported optional session parameters can be specified in param=value and include: Use sql.OpenDB() to create a database handle via a new connector object created with dbsql.NewConnector(): Supported functional options include: Cancelling a query via context cancellation or timeout is supported. Use the driverctx package under driverctx/ctx.go to add CorrelationId and ConnId to the context. CorrelationId and ConnId makes it convenient to parse and create metrics in logging. **Connection Id** Internal id to track what happens under a connection. Connections can be reused so this would track across queries. **Query Id** Internal id to track what happens under a query. Useful because the same query can be used with multiple connections. **Correlation Id** External id, such as request ID, to track what happens under a request. Useful to track multiple connections in the same request. Use the logger package under logger.go to set up logging (from zerolog). By default, logging level is `warn`. If you want to disable logging, use `disabled`. The user can also utilize Track() and Duration() to custom log the elapsed time of anything tracked. The result log may look like this: Use the driverctx package under driverctx/ctx.go to add callbacks to the query context to receive the connection id and query id. Passing parameters to a query is supported when run against servers with version DBR 14.1. For complex types, you can specify the SQL type using the dbsql.Parameter type field. If this field is set, the value field MUST be set to a string. The Go driver now supports staging operations. In order to use a staging operation, you first must update the context with a list of folders that you are allowing the driver to access. After doing so, you can execute staging operations using this context using the exec context. There are three error types exposed via dbsql/errors Each type has a corresponding sentinel value which can be used with errors.Is() to determine if one of the types is present in an error chain. Example usage: See the documentation for dbsql/errors for more information. The driver supports the ability to retrieve Apache Arrow record batches. To work with record batches it is necessary to use sql.Conn.Raw() to access the underlying driver connection to retrieve a driver.Rows instance. The driver exposes two public interfaces for working with record batches from the rows sub-package: The driver.Rows instance retrieved using Conn.Raw() can be converted to a Databricks Rows instance via a type assertion, then use GetArrowBatches() to retrieve a batch iterator. If the ArrowBatchIterator is not closed it will leak resources, such as the underlying connection. Calling code must call Release() on records returned by DBSQLArrowBatchIterator.Next(). Example usage: ================================== Databricks Type --> Golang Type ================================== BOOLEAN --> bool TINYINT --> int8 SMALLINT --> int16 INT --> int32 BIGINT --> int64 FLOAT --> float32 DOUBLE --> float64 VOID --> nil STRING --> string DATE --> time.Time TIMESTAMP --> time.Time DECIMAL(p,s) --> sql.RawBytes BINARY --> sql.RawBytes ARRAY<elementType> --> sql.RawBytes STRUCT --> sql.RawBytes MAP<keyType, valueType> --> sql.RawBytes INTERVAL (year-month) --> string INTERVAL (day-time) --> string For ARRAY, STRUCT, and MAP types, sql.Scan can cast sql.RawBytes to JSON string, which can be unmarshalled to Golang arrays, maps, and structs. For example: May generate the following row:
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Cyber extortion in the US and Canada hit record levels in 2023, with ransomware attacks surging and median ransom demands skyrocketing, though fewer companies are choosing to pay ransoms.
Security News
Ecma TC39 is meeting this week and has moved key ECMAScript proposals forward, advancing Deferred Import Evaluation, Error.isError(), RegExp Escaping, and Promise.try to the next stages.
Security News
Researchers have demonstrated that teams of LLM agents can exploit zero-day vulnerabilities with a 53% success rate, and the costs of using AI to do so are rapidly becoming more affordable than hiring a human penetration tester.