Easily find and add relevant features to your ML & AI pipeline from
hundreds of public, community and premium external data sources,
including open & commercial LLMs
Quick Start in Colab » |
Register / Sign In |
Slack Community |
Propose new Data source
Upgini is an intelligent data search engine with a Python library that helps you find and add relevant features to your ML pipeline from hundreds of public, community, and premium external data sources. Under the hood, Upgini automatically optimizes all connected data sources by generating an optimal set of machine ML features using large language models (LLMs), GraphNNs and recurrent neural networks (RNNs).
Motivation: for most supervised ML models external data & features boost accuracy significantly better than any hyperparameters tuning. But lack of automated and time-efficient enrichment tools for external data blocks massive adoption of external features in ML pipelines. We want radically simplify features search and enrichment to make external data a standard approach. Like a hyperparameter tuning for machine learning nowadays.
Mission: Democratize access to data sources for data science community.
⭐️ Automatically find only relevant features that give accuracy improvement for ML model. Not just correlated with target variable, what 9 out of 10 cases gives zero accuracy improvement
⭐️ Automated feature generation from the sources: feature generation with Large Language Models' data augmentation, RNNs, GraphNN; multiple data source ensembling
⭐️ Automatic search key augmentation from all connected sources. If you do not have all search keys in your search request, such as postal/zip code, Upgini will try to add those keys based on the provided set of search keys. This will broaden the search across all available data sources
⭐️ Calculate accuracy metrics and uplifts after enrichment existing ML model with external features
⭐️ Check the stability of accuracy gain from external data on out-of-time intervals and verification datasets. Mitigate risks of unstable external data dependencies in ML pipeline
⭐️ Easy to use - single request to enrich training dataset with all of the keys at once:
| date / datetime | phone number |
| postal / ZIP code | hashed email / HEM |
| country | IP-address |
⭐️ Scikit-learn compatible interface for quick data integration with existing ML pipelines
⭐️ Support for most common supervised ML tasks on tabular data:
| ☑️ binary classification | ☑️ multiclass classification |
| ☑️ regression | ☑️ time series prediction |
⭐️ Simple Drag & Drop Search UI:
👉 Details on datasets and features
| Data sources | Countries | History, years | # sources for ensemble | Update | Search keys | API Key required |
|---|---|---|---|---|---|---|
| Historical weather & Climate normals | 68 | 22 | - | Monthly | date, country, postal/ZIP code | No |
| Location/Places/POI/Area/Proximity information from OpenStreetMap | 221 | 2 | - | Monthly | date, country, postal/ZIP code | No |
| International holidays & events, Workweek calendar | 232 | 22 | - | Monthly | date, country | No |
| Consumer Confidence index | 44 | 22 | - | Monthly | date, country | No |
| World economic indicators | 191 | 41 | - | Monthly | date, country | No |
| Markets data | - | 17 | - | Monthly | date, datetime | No |
| World mobile & fixed broadband network coverage and performance | 167 | - | 3 | Monthly | country, postal/ZIP code | No |
| World demographic data | 90 | - | 2 | Annual | country, postal/ZIP code | No |
| World house prices | 44 | - | 3 | Annual | country, postal/ZIP code | No |
| Public social media profile data | 104 | - | - | Monthly | date, email/HEM, phone | Yes |
| Car ownership data and Parking statistics | 3 | - | - | Annual | country, postal/ZIP code, email/HEM, phone | Yes |
| Geolocation profile for phone & IPv4 & email | 239 | - | 6 | Monthly | date, email/HEM, phone, IPv4 | Yes |
| 🔜 Email/WWW domain profile | - | - | - | - |
❓Know other useful data sources for machine learning? Give us a hint and we'll add it for free.
Run Feature search & generation notebook inside your browser:
%pip install upgini
You can use your labeled training datasets "as is" to initiate the search. Under the hood, we'll search for relevant data using:
Load training dataset into pandas dataframe and separate features' columns from label column in a Scikit-learn way:
import pandas as pd
# labeled training dataset - customer_churn_prediction_train.csv
train_df = pd.read_csv("customer_churn_prediction_train.csv")
X = train_df.drop(columns="churn_flag")
y = train_df["churn_flag"]
|
⚠️ Requirements for search initialization dataset
We do dataset verification and cleaning under the hood, but still there are some requirements to follow: 1. pandas.DataFrame, pandas.Series or numpy.ndarray representation; 2. correct label column types: boolean/integers/strings for binary and multiclass labels, floats for regression; 3. at least one column selected as a search key; 4. min size after deduplication by search key column and NaNs removal: 100 records |
Search keys columns will be used to match records from all potential external data sources / features.
Define one or multiple columns as a search keys with FeaturesEnricher class initialization.
from upgini.features_enricher import FeaturesEnricher
from upgini.metadata import SearchKey
enricher = FeaturesEnricher(
search_keys={
"subscription_activation_date": SearchKey.DATE,
"country": SearchKey.COUNTRY,
"zip_code": SearchKey.POSTAL_CODE,
"hashed_email": SearchKey.HEM,
"last_visit_ip_address": SearchKey.IP,
"registered_with_phone": SearchKey.PHONE
})
| Search Key Meaning Type | Description | Allowed pandas dtypes (python types) | Example |
|---|---|---|---|
| SearchKey.EMAIL | object(str) string | support@upgini.com | |
| SearchKey.HEM | sha256(lowercase(email)) | object(str) string | 0e2dfefcddc929933dcec9a5c7db7b172482814e63c80b8460b36a791384e955 |
| SearchKey.IP | IP address (version 4) | object(str, ipaddress.IPv4Address) string int64 | 192.168.0.1 |
| SearchKey.PHONE | phone number, E.164 standard | object(str) string int64 float64 | 443451925138 |
| SearchKey.DATE | date |
object(str) string datetime64[ns] period[D] |
2020-02-12 (ISO-8601 standard)
12.02.2020 (non standard notation) |
| SearchKey.DATETIME | datetime |
object(str) string datetime64[ns] period[D] | 2020-02-12 12:46:18 12:46:18 12.02.2020 |
| SearchKey.COUNTRY | Country ISO-3166 code, Country name | object(str) string | GB US IN |
| SearchKey.POSTAL_CODE | Postal code a.k.a. ZIP code. Could be used only with SearchKey.COUNTRY | object(str) string | 21174 061107 SE-999-99 |
For the meaning types SearchKey.DATE/SearchKey.DATETIME with dtypes object or string you have to clarify date/datetime format by passing date_format parameter to FeaturesEnricher. For example:
from upgini.features_enricher import FeaturesEnricher
from upgini.metadata import SearchKey
enricher = FeaturesEnricher(
search_keys={
"subscription_activation_date": SearchKey.DATE,
"country": SearchKey.COUNTRY,
"zip_code": SearchKey.POSTAL_CODE,
"hashed_email": SearchKey.HEM,
"last_visit_ip_address": SearchKey.IP,
"registered_with_phone": SearchKey.PHONE
},
date_format = "%Y-%d-%m"
)
To use datetime not in UTC timezone, you can cast datetime column explicitly to your timezone (example for Warsaw):
df["date"] = df.date.astype("datetime64").dt.tz_localize("Europe/Warsaw")
Single country for the whole training dataset can be passed with country_code parameter:
from upgini.features_enricher import FeaturesEnricher
from upgini.metadata import SearchKey
enricher = FeaturesEnricher(
search_keys={
"subscription_activation_date": SearchKey.DATE,
"zip_code": SearchKey.POSTAL_CODE,
},
country_code = "US",
date_format = "%Y-%d-%m"
)
The main abstraction you interact is FeaturesEnricher, a Scikit-learn compatible estimator. You can easily add it into your existing ML pipelines.
Create instance of the FeaturesEnricher class and call:
fit to search relevant datasets & featurestransform to enrich your dataset with features from search resultLet's try it out!
import pandas as pd
from upgini.features_enricher import FeaturesEnricher
from upgini.metadata import SearchKey
# load labeled training dataset to initiate search
train_df = pd.read_csv("customer_churn_prediction_train.csv")
X = train_df.drop(columns="churn_flag")
y = train_df["churn_flag"]
# now we're going to create `FeaturesEnricher` class
enricher = FeaturesEnricher(
search_keys={
"subscription_activation_date": SearchKey.DATE,
"country": SearchKey.COUNTRY,
"zip_code": SearchKey.POSTAL_CODE
})
# everything is ready to fit! For 200к records fitting should take around 10 minutes,
# we send email notification, just register on profile.upgini.com
enricher.fit(X, y)
That's all). We've fitted FeaturesEnricher.
FeaturesEnricher class has two properties for feature importances, which will be filled after fit - feature_names_ and feature_importances_:
feature_names_ - feature names from the search result, and if parameter keep_input=True was used, initial columns from search dataset as wellfeature_importances_ - SHAP values for features from the search result, same order as in feature_names_Method get_features_info() returns pandas dataframe with features and full statistics after fit, including SHAP values and match rates:
enricher.get_features_info()
Get more details about FeaturesEnricher at runtime using docstrings via help(FeaturesEnricher) or help(FeaturesEnricher.fit).
FeaturesEnricher is a Scikit-learn compatible estimator, so any pandas dataframe can be enriched with external features from a search result (after fit ).
Use transform method of FeaturesEnricher , and let magic to do the rest 🪄
# load dataset for enrichment
test_x = pd.read_csv("test.csv")
# enrich it!
enriched_test_features = enricher.transform(test_x)
FeaturesEnricher can be initiated with a search_id parameter from completed search after fit method call.
Just use enricher.get_search_id() or copy search id string from the fit() output.
Search keys and features in X should be the same as for fit()
enricher = FeaturesEnricher(
#same set of a search keys as for the fit step
search_keys={"date": SearchKey.DATE},
api_key="<YOUR API_KEY>", # if you fit enricher with api_key then you should use it here
search_id = "abcdef00-0000-0000-0000-999999999999"
)
enriched_prod_dataframe=enricher.transform(input_dataframe)
For most of the ML cases, training step requires labeled dataset with a historical observations from the past. But for production step you'll need an updated and actual data sources and features for the present time, to calculate a prediction.
FeaturesEnricher, when initiated with set of search keys which includes SearchKey.DATE, will match records from all potential external data sources exactly on a the specific date/datetime based on SearchKey.DATE. To avoid enrichment with features "form the future" for the fit step.
And then, for transform in a production ML pipeline, you'll get enrichment with relevant features, actual for the present date.
⚠️ Initiate FeaturesEnricher with SearchKey.DATE search key in a key set to get actual features for production and avoid features from the future for the training:
enricher = FeaturesEnricher(
search_keys={
"subscription_activation_date": SearchKey.DATE,
"country": SearchKey.COUNTRY,
"zip_code": SearchKey.POSTAL_CODE,
},
)
We validate and clean search initialization dataset under the hood:
We detect ML task under the hood based on label column values. Currently we support:
But for certain search datasets you can pass parameter to FeaturesEnricher with correct ML taks type:
from upgini.features_enricher import FeaturesEnricher
from upgini.metadata import SearchKey, ModelTaskType
enricher = FeaturesEnricher(
search_keys={"subscription_activation_date": SearchKey.DATE},
model_task_type=ModelTaskType.REGRESSION
)
Time series prediction supported as ModelTaskType.REGRESSION or ModelTaskType.BINARY tasks with time series specific cross-validation split:
CVType.time_series parameterCVType.blocked_time_series parameterTo initiate feature search you can pass cross-validation type parameter to FeaturesEnricher with time series specific CV type:
from upgini.features_enricher import FeaturesEnricher
from upgini.metadata import SearchKey, CVType
enricher = FeaturesEnricher(
search_keys={"sales_date": SearchKey.DATE},
cv=CVType.time_series
)
⚠️ Pre-process search dataset in case of time series prediction:
sort rows in dataset according to observation order, in most cases - ascending order by date/datetime.
FeaturesEnricher automaticaly calculates model metrics and uplift from new relevant features either using calculate_metrics() method or calculate_metrics=True parameter in fit or fit_transform methods (example below).
You can use any model estimator with scikit-learn compartible interface, some examples are:
| Metric | Description |
|---|---|
| explained_variance | Explained variance regression score function |
| r2 | R2 (coefficient of determination) regression score function |
| max_error | Calculates the maximum residual error (negative - greater is better) |
| median_absolute_error | Median absolute error regression loss |
| mean_absolute_error | Mean absolute error regression loss |
| mean_absolute_percentage_error | Mean absolute percentage error regression loss |
| mean_squared_error | Mean squared error regression loss |
| mean_squared_log_error (or aliases: msle, MSLE) | Mean squared logarithmic error regression loss |
| root_mean_squared_log_error (or aliases: rmsle, RMSLE) | Root mean squared logarithmic error regression loss |
| root_mean_squared_error | Root mean squared error regression loss |
| mean_poisson_deviance | Mean Poisson deviance regression loss |
| mean_gamma_deviance | Mean Gamma deviance regression loss |
| accuracy | Accuracy classification score |
| top_k_accuracy | Top-k Accuracy classification score |
| roc_auc | Area Under the Receiver Operating Characteristic Curve (ROC AUC) from prediction scores |
| roc_auc_ovr | Area Under the Receiver Operating Characteristic Curve (ROC AUC) from prediction scores (multi_class="ovr") |
| roc_auc_ovo | Area Under the Receiver Operating Characteristic Curve (ROC AUC) from prediction scores (multi_class="ovo") |
| roc_auc_ovr_weighted | Area Under the Receiver Operating Characteristic Curve (ROC AUC) from prediction scores (multi_class="ovr", average="weighted") |
| roc_auc_ovo_weighted | Area Under the Receiver Operating Characteristic Curve (ROC AUC) from prediction scores (multi_class="ovo", average="weighted") |
| balanced_accuracy | Compute the balanced accuracy |
| average_precision | Compute average precision (AP) from prediction scores |
| log_loss | Log loss, aka logistic loss or cross-entropy loss |
| brier_score | Compute the Brier score loss |
In addition to that list, you can define custom evaluation metric function using scikit-learn make_scorer, for example SMAPE.
By default, calculate_metrics() method calculates evaluation metric with the same cross-validation split as selected for FeaturesEnricher.fit() by parameter cv = CVType.<cross-validation-split>.
But you can easily define new split by passing child of BaseCrossValidator to parameter cv in calculate_metrics().
Example with more tips-and-tricks:
from upgini.features_enricher import FeaturesEnricher
from upgini.metadata import SearchKey
enricher = FeaturesEnricher(search_keys={"registration_date": SearchKey.DATE})
# Fit with default setup for metrics calculation
# CatBoost will be used
enricher.fit(X, y, eval_set=eval_set, calculate_metrics=True)
# LightGBM estimator for metrics
custom_estimator = LGBMRegressor()
enricher.calculate_metrics(estimator=custom_estimator)
# Custom metric function to scoring param (callable or name)
custom_scoring = "RMSLE"
enricher.calculate_metrics(scoring=custom_scoring)
# Custom cross validator
custom_cv = TimeSeriesSplit(n_splits=5)
enricher.calculate_metrics(cv=custom_cv)
# All this custom parameters could be combined in both methods: fit, fit_transform and calculate_metrics:
enricher.fit(X, y, eval_set, calculate_metrics=True, estimator=custom_estimator, scoring=custom_scoring, cv=custom_cv)
If a training dataset has a text column, you can generate additional embeddings from it using instructed embeddings generation with LLMs and data augmentation from external sources, just like Upgini does for all records from connected data sources.
For most cases, this gives better results than direct embeddings generation from a text field. Currently, Upgini has two LLMs connected to a search engine - GPT-3.5 from OpenAI and GPT-J.
To use this feature, pass the column names as arguments to the generate_features parameter. You can use up to 2 columns.
Here's an example for generating features from the "description" and "summary" columns:
enricher = FeaturesEnricher(
search_keys={"date": SearchKey.DATE},
generate_features=["description", "summary"]
)
With this code, Upgini will generate LLM embeddings from text columns and then check them for predictive power for your ML task.
Finally, Upgini will return a dataset enriched by only relevant components of LLM embeddings.
If you already have features or other external data sources, you can specifically search new datasets & features only give accuracy gain "on top" of them.
Just leave all these existing features in the labeled training dataset and Upgini library automatically use them during feature search process and as a baseline ML model to calculate accuracy metric uplift. Only features which improve accuracy will return.
You can validate external features robustness on out-of-time dataset using eval_set parameter:
# load train dataset
train_df = pd.read_csv("train.csv")
train_ids_and_features = train_df.drop(columns="label")
train_label = train_df["label"]
# load out-of-time validation dataset
eval_df = pd.read_csv("validation.csv")
eval_ids_and_features = eval_df.drop(columns="label")
eval_label = eval_df["label"]
# create FeaturesEnricher
enricher = FeaturesEnricher(search_keys={"registration_date": SearchKey.DATE})
# now we fit WITH eval_set parameter to calculate accuracy metrics on Out-of-time dataset.
# the output will contain quality metrics for both the training data set and
# the eval set (validation OOT data set)
enricher.fit(
train_ids_and_features,
train_label,
eval_set = [(eval_ids_and_features, eval_label)]
)
FeaturesEnricher can be initialized with additional string parameter loss.
Depending on ML-task, you can use the following loss functions:
regression: regression, regression_l1, huber, poisson, quantile, mape, gamma, tweedie;binary: binary;multiclass: multiclass, multiclassova.For instance, if your target variable has a Poisson distribution (count of events, number of customers in the shop and so on), you should try to use loss="poisson" to improve quality of feature selection and get better evaluation metrics.
Usage example:
enricher = FeaturesEnricher(
search_keys={"date": SearchKey.DATE},
loss="poisson",
model_task_type=ModelTaskType.REGRESSION
)
enriched_dataframe.fit(X, y)
transform and fit_transform methods of FeaturesEnricher can be used with two additional parameters:
importance_threshold: float = 0 - only features with importance >= threshold will be added to the output dataframemax_features: int - only first TOP N features by importance will be returned, where N = max_featuresAnd keep_input=True will keep all initial columns from search dataset X:
enricher = FeaturesEnricher(
search_keys={"subscription_activation_date": SearchKey.DATE}
)
enriched_dataframe.fit_transform(X, y, keep_input=True, max_features=2)
fit, fit_transform, transform and calculate_metrics methods of FeaturesEnricher can be used with parameter exclude_features_sources that allows to exclude Trial or Paid features from Premium data sources:
enricher = FeaturesEnricher(
search_keys={"subscription_activation_date": SearchKey.DATE}
)
enricher.fit(X, y, calculate_metrics=False)
trial_features = enricher.get_features_info()[enricher.get_features_info()["Feature type"] == "Trial"]["Feature name"].values.tolist()
paid_features = enricher.get_features_info()[enricher.get_features_info()["Feature type"] == "Paid"]["Feature name"].values.tolist()
enricher.calculate_metrics(exclude_features_sources=(trial_features + paid_features))
enricher.transform(X, exclude_features_sources=(trial_features + paid_features))
Upgini has autodetection of search keys on by default.
To turn off use detect_missing_search_keys=False:
enricher = FeaturesEnricher(
search_keys={"date": SearchKey.DATE},
detect_missing_search_keys=False,
)
enricher.fit(X, y)
Upgini detect rows with target outlier for regression tasks. By default such rows are dropped on metrics calculation. To turn off removing of target outlier rows use parameter remove_outliers_calc_metrics=False in fit, fit_transform or calculate_metrics methods:
enricher = FeaturesEnricher(
search_keys={"date": SearchKey.DATE},
)
enricher.fit(X, y, remove_outliers_calc_metrics=False)
Register and get a free API key for exclusive data sources and features: 600 mln+ phone numbers, 350 mln+ emails, 2^32 IP addresses
| Benefit | No Sign-up | Registered user |
|---|---|---|
| Enrichment with date/datetime, postal/ZIP code and country keys | Yes | Yes |
| Enrichment with phone number, hashed email/HEM and IP-address keys | No | Yes |
| Email notification on search task completion | No | Yes |
| Automated feature generation with LLMs from columns in a search dataset | Yes, till 12/05/23 | Yes |
| Email notification on new data source activation 🔜 | No | Yes |
You may publish ANY data which you consider as royalty / license free (Open Data) and potentially valuable for ML applications for community usage:
import pandas as pd
from upgini.metadata import SearchKey
from upgini.ads import upload_user_ads
import os
os.environ["UPGINI_API_KEY"] = "your_long_string_api_key_goes_here"
#you can define custom search key which might not be supported yet, just use SearchKey.CUSTOM_KEY type
sample_df = pd.read_csv("path_to_data_sample_file")
upload_user_ads("test", sample_df, {
"city": SearchKey.CUSTOM_KEY,
"stats_date": SearchKey.DATE
})
Please note, that we are still in a beta stage.
Requests and support, in preferred order
❗Please try to create bug reports that are:
We are not a large team, so we probably won't be able to:
So, we'll be happy about every pull request you open and issue you find to make this library more incredible. Please note that it might sometimes take us a while to get back to you. For major changes, please open an issue first to discuss what you would like to change
Some convenient ways to start contributing are:
⚙️ Open in Visual Studio Code You can remotely open this repo in VS Code without cloning or automatically clone and open it inside a docker container.
⚙️ Gitpod 
😔 Found mistype or a bug in code snippet? Our bad! Please report it here
FAQs
Intelligent data search & enrichment for Machine Learning
We found that upgini demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 3 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.
Security News
The Ultralytics' PyPI Package was compromised four times in one weekend through GitHub Actions cache poisoning and failure to rotate previously compromised API tokens.
