Socket
Socket
Sign inDemoInstall

streamxfer

Package Overview
Dependencies
7
Maintainers
1
Alerts
File Explorer

Install Socket

Detect and block malicious and high-risk dependencies

Install

streamxfer

StreamXfer is a powerful tool for streaming data from SQL Server to object storage for seamless transfer using UNIX pipe, supporting various general data formats(CSV, TSV, JSON).

    0.0.1

Maintainers
1

Readme

StreamXfer

StreamXfer is a powerful tool for streaming data from SQL Server to object storage for seamless transfer using UNIX pipe, supporting various general data formats(CSV, TSV, JSON).

Demo

asciicast

Installation

Before installing StreamXfer, you need to install the following dependencies:

  • mssql-tools: SQL Docs - bcp Utility
  • lzop: yum install lzop

Then, install StreamXfer from PyPI:

$ python3 -m pip install 

Alternatively, install from source:

$ git clone https://github.com/zhiweio/StreamXfer.git && cd StreamXfer/
$ python3 setup.py install

Usage

StreamXfer can be used as a command-line tool or as a library in Python.

Command-line Usage

$ stx [OPTIONS] PYMSSQL_URL TABLE OUTPUT_PATH

Here is an example command:

$ stx 'mssql+pymssql:://user:pass@host:port/db' '[dbo].[test]' /local/path/to/dir/

You can also use the following options:

  • -F, --format: The data format (CSV, TSV, or JSON).
  • --compress-type: The compression type (LZOP or GZIP).
  • --no-compress: Disables compression.

For more information on the options, run stx --help.

$ stx --help
Usage: stx [OPTIONS] PYMSSQL_URL TABLE OUTPUT_PATH

  StreamXfer is a powerful tool for streaming data from SQL Server to object
  storage for seamless transfer using UNIX pipe, supporting various general
  data formats(CSV, TSV, JSON).

  Examples:
      stx 'mssql+pymssql:://user:pass@host:port/db' '[dbo].[test]' /local/path/to/dir/
      stx 'mssql+pymssql:://user:pass@host:port/db' '[dbo].[test]' s3://bucket/path/to/dir/

Options:
  -F, --format [CSV|TSV|JSON]  [default: JSON]
  --compress-type [LZOP|GZIP]  [default: LZOP]
  --no-compress
  --redshift-escape
  --help                       Show this message and exit.

Library Usage

To use StreamXfer as a library in Python, you can import the StreamXfer class and the sink classes (such as S3Sink), and use them to build and pump the data stream.

Here is an example code snippet:

from streamxfer import StreamXfer
from streamxfer.cmd import S3Sink

sx = StreamXfer(
    "mssql+pymssql:://user:pass@host:port/db",
    format="TSV",
    enable_compress=True,
    compress_type="LZOP",
)
sx.build("[dbo].[test]", path="s3://bucket/path/to/dir/", sink="s3")
sx.pump()

Here are some related articles

Authors

License

GPL-3.0

FAQs


Did you know?

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install

Related posts

SocketSocket SOC 2 Logo

Product

  • Package Alerts
  • Integrations
  • Docs
  • Pricing
  • FAQ
  • Roadmap

Stay in touch

Get open source security insights delivered straight into your inbox.


  • Terms
  • Privacy
  • Security

Made with ⚡️ by Socket Inc