
Product
Introducing Scala and Kotlin Support in Socket
Socket now supports Scala and Kotlin, bringing AI-powered threat detection to JVM projects with easy manifest generation and fast, accurate scans.
This repository provides a collection of utility functions and classes for data cleaning, SQL query generation, and data analysis. The code is written in Python and uses libraries such as `pandas`, `numpy`, and `dateutil`.
This project provides a comprehensive Data ETL (Extract, Transform, Load) and data manipulation framework using Python. It integrates with databases using SQLAlchemy and provides tools for data parsing, cleaning, loading, validating, and more. The project is structured with classes that encapsulate different functionalities.
The Connector
class handles creating connections to various types of databases (MSSQL, PostgreSQL, MySQL) using SQLAlchemy. It provides static methods for obtaining both trusted and user connections.
Key Methods:
get_mssql_trusted_connection
get_mssql_user_connection
get_postgres_user_connection
get_mysql_user_connection
The Loader
class is responsible for loading data from a Pandas DataFrame into a database. It manages the insertion process, ensuring data is inserted efficiently and effectively with the use of SQLAlchemy and custom logging.
A slight extension of the Loader
class specifically for MySQL databases. It provides overrides to manage MySQL-specific data types and query formatting.
A specialized loader for loading data into MSSQL databases with additional functionalities like fast insertions using bulk methods.
The Parser
class consists of a series of static methods dedicated to parsing various data types—boolean, float, date, and integer. These methods are essential for data type conversion and consistency across the application.
The Cleaner
class provides methods for sanitizing and formatting data in a DataFrame. It includes functions for setting column name casing conventions, cleaning various types of data, and preparing data for reliable analysis and insertion.
This class deals with generating SQL CREATE TABLE
statements for different databases like MSSQL and MariaDB. The query generation considers data types deduced from DataFrame content.
The Analyzer
class assesses DataFrame characteristics and helps identify unique columns, column pairs, empty columns, and more. It aids in generating metadata for data types, which is crucial for creating or validating schemas.
The Validator
class ensures DataFrame compatibility with the target database table structure by checking for extra columns, validating data types, and ensuring that no data truncation will occur during upload.
A class designed for constructing SQL statements for operations like mergers, updates, inserts, and appends to manage data transitions between tables efficiently.
The project uses a singleton Logger
class with colored output format for console logging. This helps in debugging and understanding the flow by logging messages at various severity levels.
FAQs
This repository provides a collection of utility functions and classes for data cleaning, SQL query generation, and data analysis. The code is written in Python and uses libraries such as `pandas`, `numpy`, and `dateutil`.
We found that etl-utilities demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Product
Socket now supports Scala and Kotlin, bringing AI-powered threat detection to JVM projects with easy manifest generation and fast, accurate scans.
Application Security
/Security News
Socket CEO Feross Aboukhadijeh and a16z partner Joel de la Garza discuss vibe coding, AI-driven software development, and how the rise of LLMs, despite their risks, still points toward a more secure and innovative future.
Research
/Security News
Threat actors hijacked Toptal’s GitHub org, publishing npm packages with malicious payloads that steal tokens and attempt to wipe victim systems.