Latest Threat Research:SANDWORM_MODE: Shai-Hulud-Style npm Worm Hijacks CI Workflows and Poisons AI Toolchains.Details
Socket
Book a DemoInstallSign in
Socket

deepchecks

Package Overview
Dependencies
Maintainers
1
Versions
59
Alerts
File Explorer

Advanced tools

Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

deepchecks - npm Package Compare versions

Comparing version
0.17.2
to
0.17.3
+2
-2
deepchecks.egg-info/PKG-INFO
Metadata-Version: 2.1
Name: deepchecks
Version: 0.17.2
Version: 0.17.3
Summary: Package for validating your machine learning model and data

@@ -9,3 +9,3 @@ Home-page: https://github.com/deepchecks/deepchecks

License: UNKNOWN
Download-URL: https://github.com/deepchecks/deepchecks/releases/download/0.17.2/deepchecks-0.17.2.tar.gz
Download-URL: https://github.com/deepchecks/deepchecks/releases/download/0.17.3/deepchecks-0.17.3.tar.gz
Project-URL: Documentation, https://docs.deepchecks.com

@@ -12,0 +12,0 @@ Project-URL: Bug Reports, https://github.com/deepchecks/deepchecks

@@ -20,3 +20,3 @@ # ----------------------------------------------------------------------------

from deepchecks.core import CheckResult, DatasetKind
from deepchecks.core.errors import NotEnoughSamplesError
from deepchecks.core.errors import DeepchecksValueError, NotEnoughSamplesError
from deepchecks.nlp import Context, SingleDatasetCheck

@@ -50,3 +50,3 @@ from deepchecks.nlp.utils.nlp_plot import get_text_outliers_graph

min_samples : int , default : 10
Minimum number of samples required to calculate IQR. If there are not enough non-null samples a specific
Minimum number of samples required to calculate IQR. If there are not enough non-null samples for a specific
property, the check will skip it. If all properties are skipped, the check will raise a NotEnoughSamplesError.

@@ -78,61 +78,67 @@ """

if all(len(np.hstack(v).squeeze()) < self.min_samples for v in properties.values()):
raise NotEnoughSamplesError(f'Need at least {self.min_samples} non-null samples to calculate outliers.')
# The values are in the same order as the batch order, so always keeps the same order in order to access
# the original sample at this index location
for name, values in properties.items():
# If the property is single value per sample, then wrap the values in list in order to work on fixed
# structure
if not isinstance(values[0], list):
values = [[x] for x in values]
is_numeric = name not in cat_properties
if is_numeric:
values_arr = np.hstack(values).astype(float).squeeze()
values_arr = np.array([x for x in values_arr if pd.notnull(x)])
else:
values_arr = np.hstack(values).astype(str).squeeze()
try:
if not isinstance(values[0], list):
if is_numeric:
# Check for non numeric data in the column
curr_nan_count = pd.isnull(values).sum()
values = pd.to_numeric(values, errors='coerce')
updated_nan_count = pd.isnull(values).sum()
if updated_nan_count > curr_nan_count:
raise DeepchecksValueError('Numeric property contains non-numeric values.')
# If the property is single value per sample, then wrap the values in list in order to
# work on fixed structure
values = [[x] for x in values]
if len(values_arr) < self.min_samples:
result[name] = 'Not enough non-null samples to calculate outliers.'
continue
if is_numeric:
values_arr = np.hstack(values).astype(float).squeeze()
values_arr = np.array([x for x in values_arr if pd.notnull(x)])
else:
values_arr = np.hstack(values).astype(str).squeeze()
if is_numeric:
lower_limit, upper_limit = iqr_outliers_range(values_arr, self.iqr_percentiles,
self.iqr_scale, self.sharp_drop_ratio)
else:
# Counting the frequency of each category. Normalizing because distribution graph shows the percentage.
counts_map = pd.Series(values_arr.astype(str)).value_counts(normalize=True).to_dict()
lower_limit = sharp_drop_outliers_range(sorted(list(counts_map.values()), reverse=True),
self.sharp_drop_ratio) or 0
upper_limit = len(values_arr) # No upper limit for categorical properties
values_arr = np.array([counts_map[x] for x in values_arr]) # replace the values with the counts
if len(values_arr) < self.min_samples:
raise NotEnoughSamplesError(f'Not enough non-null samples to calculate outliers'
f'(min_samples={self.min_samples}).')
# Get the indices of the outliers
top_outliers = np.argwhere(values_arr > upper_limit).squeeze(axis=1)
# Sort the indices of the outliers by the original values
top_outliers = top_outliers[
np.apply_along_axis(lambda i, sort_arr=values_arr: sort_arr[i], axis=0, arr=top_outliers).argsort()
]
if is_numeric:
lower_limit, upper_limit = iqr_outliers_range(values_arr, self.iqr_percentiles,
self.iqr_scale, self.sharp_drop_ratio)
else:
# Counting the frequency of each category. Normalizing because distribution graph shows percentage.
counts_map = pd.Series(values_arr.astype(str)).value_counts(normalize=True).to_dict()
lower_limit = sharp_drop_outliers_range(sorted(list(counts_map.values()), reverse=True),
self.sharp_drop_ratio) or 0
upper_limit = len(values_arr) # No upper limit for categorical properties
values_arr = np.array([counts_map[x] for x in values_arr]) # replace the values with the counts
# Doing the same for bottom outliers
bottom_outliers = np.argwhere(values_arr < lower_limit).squeeze(axis=1)
# Sort the indices of the outliers by the original values
bottom_outliers = bottom_outliers[
np.apply_along_axis(lambda i, sort_arr=values_arr: sort_arr[i], axis=0, arr=bottom_outliers).argsort()
]
# Get the indices of the outliers
top_outliers = np.argwhere(values_arr > upper_limit).squeeze(axis=1)
# Sort the indices of the outliers by the original values
top_outliers = top_outliers[
np.apply_along_axis(lambda i, sort_arr=values_arr: sort_arr[i], axis=0, arr=top_outliers).argsort()
]
text_outliers = np.concatenate([bottom_outliers, top_outliers])
# Doing the same for bottom outliers
bottom_outliers = np.argwhere(values_arr < lower_limit).squeeze(axis=1)
# Sort the indices of the outliers by the original values
bottom_outliers = bottom_outliers[
np.apply_along_axis(lambda i, sort_arr=values_arr: sort_arr[i],
axis=0, arr=bottom_outliers).argsort()
]
result[name] = {
'indices': [dataset.get_original_text_indexes()[i] for i in text_outliers],
# For the upper and lower limits doesn't show values that are smaller/larger than the actual values
# we have in the data
'lower_limit': max(lower_limit, min(values_arr)),
'upper_limit': min(upper_limit, max(values_arr)) if is_numeric else None,
'outlier_ratio': len(text_outliers) / len(values_arr)
}
text_outliers = np.concatenate([bottom_outliers, top_outliers])
result[name] = {
'indices': [dataset.get_original_text_indexes()[i] for i in text_outliers],
# For the upper and lower limits doesn't show values that are smaller/larger than the actual values
# we have in the data
'lower_limit': max(lower_limit, min(values_arr)),
'upper_limit': min(upper_limit, max(values_arr)) if is_numeric else None,
'outlier_ratio': len(text_outliers) / len(values_arr),
}
except Exception as exp: # pylint: disable=broad-except
result[name] = f'{exp}'
# Create display

@@ -143,5 +149,10 @@ if context.with_display:

sorted_result_items = sorted(result.items(), key=lambda x: len(x[1]['indices']), reverse=True)
# Sort the result map based on the length of indices and if there are any error message associated to
# any property, keep that property at the very end.
sorted_result_items = sorted(result.items(),
key=lambda x: len(x[1].get('indices', [])) if isinstance(x[1], dict) else 0,
reverse=True)
for property_name, info in sorted_result_items:
# If info is string it means there was error

@@ -154,7 +165,10 @@ if isinstance(info, str):

if len(display) < self.n_show_top:
dist = df_properties[property_name]
if len(dist[~pd.isnull(dist)]) >= self.min_samples:
lower_limit = info['lower_limit']
upper_limit = info['upper_limit']
if property_name not in cat_properties:
dist = df_properties[property_name].astype(float)
else:
dist = df_properties[property_name]
lower_limit = info['lower_limit']
upper_limit = info['upper_limit']
try:
fig = get_text_outliers_graph(

@@ -170,9 +184,5 @@ dist=dist,

display.append(fig)
else:
no_outliers = pd.concat(
[no_outliers, pd.Series(property_name, index=[
f'Not enough non-null samples to compute'
f' properties (min_samples={self.min_samples}).'
])]
)
except Exception as exp: # pylint: disable=broad-except
result[property_name] = f'{exp}'
no_outliers = pd.concat([no_outliers, pd.Series(property_name, index=[exp])])
else:

@@ -179,0 +189,0 @@ no_outliers = pd.concat([no_outliers, pd.Series(property_name, index=[

@@ -82,5 +82,5 @@ # ----------------------------------------------------------------------------

"""
import math
import time
import typing as t
from typing import Tuple

@@ -92,8 +92,6 @@ import numpy as np

__all__ = ['load_data', 'load_pre_calculated_prediction', 'load_pre_calculated_feature_importance']
__all__ = ['load_data_and_predictions', 'load_pre_calculated_feature_importance']
from numpy import ndarray
_TRAIN_DATA_URL = 'https://drive.google.com/uc?export=download&id=1UWkr1BQlyyUkbsW5hHIFTr-x0evZE3Ie'
_TEST_DATA_URL = 'https://drive.google.com/uc?export=download&id=1v_0ZyyycoFfltJ6wj_riGZoXhtPzrnqR'
_TEST_DATA_URL = 'https://drive.google.com/uc?export=download&id=1lfpWVtDktrnsLUzCN1tkRc1jRbguEz3a'
_target = 'price'

@@ -108,4 +106,5 @@ _predictions = 'predictions'

def load_data(data_format: str = 'Dataset', as_train_test: bool = True, modify_timestamps: bool = True,
data_size: t.Optional[int] = 15000) -> t.Union[t.Tuple, t.Union[Dataset, pd.DataFrame]]:
def load_data_and_predictions(data_format: str = 'Dataset', load_train: bool = True, modify_timestamps: bool = True,
data_size: t.Optional[int] = 15000, random_state: int = 42) \
-> t.Tuple[t.Union[Dataset, pd.DataFrame], np.ndarray]:
"""Load and returns the Airbnb NYC 2019 dataset (regression).

@@ -119,7 +118,4 @@

'Dataframe' will return the data as a pandas Dataframe object
as_train_test : bool , default: True
If True, the returned data is split into train and test exactly like the toy model
was trained. The first return value is the train data and the second is the test data.
In order to get this model, call the load_fitted_model() function.
Otherwise, returns a single object.
load_train : bool , default: True
If True, the returned data is the train data. otherwise the test dataset.
modify_timestamps : bool , default: True

@@ -130,64 +126,39 @@ If True, the returned data timestamp column will be for the last 30 days.

The number of samples to return. If None, returns all the data.
random_state : int , default 42
The random state to use for sampling.
Returns
-------
dataset : Union[deepchecks.Dataset, pd.DataFrame]
the data object, corresponding to the data_format attribute.
train_data, test_data : Tuple[Union[deepchecks.Dataset, pd.DataFrame],Union[deepchecks.Dataset, pd.DataFrame]
tuple if as_train_test = True. Tuple of two objects represents the dataset split to train and test sets.
dataset, predictions : Tuple[Union[deepchecks.Dataset, pd.DataFrame], np.ndarray]
Tuple of the deepchecks dataset or dataframe and the predictions.
"""
train = pd.read_csv(_TRAIN_DATA_URL, index_col=0).drop(_predictions, axis=1)
test = pd.read_csv(_TEST_DATA_URL, index_col=0).drop(_predictions, axis=1)
if load_train:
dataset = pd.read_csv(_TRAIN_DATA_URL)
else:
dataset = pd.read_csv(_TEST_DATA_URL)
if data_size is not None:
if data_size < len(train):
train = train.sample(data_size, random_state=42)
if data_size < len(test):
test = test.sample(data_size, random_state=42)
if data_size < len(dataset):
dataset = dataset.sample(data_size, random_state=random_state)
elif data_size > len(dataset):
dataset = pd.concat([dataset] * math.ceil(data_size / len(dataset)), axis=0, ignore_index=True)
dataset = dataset.sample(data_size, random_state=random_state)
if not load_train:
dataset = dataset.sort_values(_datetime)
if modify_timestamps:
if modify_timestamps and not load_train:
current_time = int(time.time())
time_test_start = current_time - 86400 * 30 # Span data for 30 days
test[_datetime] = np.sort((np.random.rand(len(test)) * (current_time - time_test_start)) + time_test_start)
test[_datetime] = test[_datetime].apply(lambda x: pd.Timestamp(x, unit='s'))
dataset[_datetime] = np.sort(
(np.random.rand(len(dataset)) * (current_time - time_test_start)) + time_test_start
)
dataset[_datetime] = dataset[_datetime].apply(lambda x: pd.Timestamp(x, unit='s'))
if not as_train_test:
dataset = pd.concat([train, test.drop(_datetime, axis=1)], axis=0, ignore_index=True)
if data_format == 'Dataset':
dataset = Dataset(dataset, label=_target, cat_features=_CAT_FEATURES, features=_FEATURES)
return dataset
else:
if data_format == 'Dataset':
train = Dataset(train, label=_target, cat_features=_CAT_FEATURES,
features=_FEATURES)
test = Dataset(test, label=_target, cat_features=_CAT_FEATURES,
datetime_name=_datetime, features=_FEATURES)
return train, test
predictions = np.asarray(dataset[_predictions])
dataset.drop(_predictions, axis=1, inplace=True)
if data_format == 'Dataset':
dataset = Dataset(dataset, label=_target, cat_features=_CAT_FEATURES,
features=_FEATURES)
return dataset, predictions
def load_pre_calculated_prediction(data_size: t.Optional[int] = 15000) -> Tuple[ndarray, ndarray]:
"""Load the pre-calculated prediction for the Airbnb NYC 2019 dataset.
Parameters
----------
data_size : t.Optional[int] , default: 15000
The number of samples to return. If None, returns all the data.
Returns
-------
predictions : Tuple(np.ndarray, np.ndarray)
The first element is the pre-calculated prediction for the train set.
The second element is the pre-calculated prediction for the test set.
"""
usable_columns = [_target, _predictions]
train = pd.read_csv(_TRAIN_DATA_URL, usecols=usable_columns)
test = pd.read_csv(_TEST_DATA_URL, usecols=usable_columns)
if data_size is not None:
if data_size < len(train):
train = train.sample(data_size, random_state=42)
if data_size < len(test):
test = test.sample(data_size, random_state=42)
return np.asarray(train[_predictions]), np.asarray(test[_predictions])
def load_pre_calculated_feature_importance() -> pd.Series:

@@ -194,0 +165,0 @@ """Load the pre-calculated feature importance for the Airbnb NYC 2019 dataset.

Metadata-Version: 2.1
Name: deepchecks
Version: 0.17.2
Version: 0.17.3
Summary: Package for validating your machine learning model and data

@@ -9,3 +9,3 @@ Home-page: https://github.com/deepchecks/deepchecks

License: UNKNOWN
Download-URL: https://github.com/deepchecks/deepchecks/releases/download/0.17.2/deepchecks-0.17.2.tar.gz
Download-URL: https://github.com/deepchecks/deepchecks/releases/download/0.17.3/deepchecks-0.17.3.tar.gz
Project-URL: Documentation, https://docs.deepchecks.com

@@ -12,0 +12,0 @@ Project-URL: Bug Reports, https://github.com/deepchecks/deepchecks

+44
-36

@@ -38,7 +38,7 @@ <!--

<a target="_blank" href="https://deepchecks.com/?utm_source=github.com&utm_medium=referral&utm_campaign=readme&utm_content=logo">
<a target="_blank" href="https://docs.deepchecks.com/?utm_source=github.com&utm_medium=referral&utm_campaign=readme&utm_content=logo">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="docs/source/_static/images/readme/cont_validation_dark.png">
<source media="(prefers-color-scheme: light)" srcset="docs/source/_static/images/readme/cont_validation_light.png">
<img alt="Deepchecks continuous validation parts." src="docs/source/_static/images//readme/cont_validation_light.png">
<source media="(prefers-color-scheme: dark)" srcset="docs/source/_static/images/readme/deepchecks_continuous_validation_dark.png">
<source media="(prefers-color-scheme: light)" srcset="docs/source/_static/images/readme/deepchecks_continuous_validation_light.png">
<img alt="Deepchecks continuous validation parts." src="docs/source/_static/images//readme/deepchecks_continuous_validation_light.png">
</picture>

@@ -60,25 +60,2 @@ </a>

## 🧮 How does it work?
At its core, deepchecks includes a wide variety of built-in Checks,
for testing all types of data and model related issues.
These checks are implemented for various models and data types (Tabular, NLP, Vision),
and can easily be customized and expanded.
The check results can be used to automatically make informed decisions
about your model's production-readiness, and for monitoring it over time in production.
The check results can be examined with visual reports (by saving them to an HTML file, or seeing them in Jupyter),
processed with code (using their pythonic / json output), and inspected and collaborated on with Deepchecks' dynamic UI
(for examining test results and for production monitoring).
<!---
At its core, Deepchecks has a wide variety of built-in Checks and Suites (lists of checks)
for all data types (Tabular, NLP, Vision),
These includes checks for validating your model's performance (e.g. identify weak segments), the data's
distribution (e.g. detect drifts or leakages), data integrity (e.g. find conflicting labels) and more.
These checks results can be run manually (e.g. during research) or trigerred automatically (e.g. during CI
and production monitoring) and enable automatically making informed decisions regarding your model pipelines'
production-readiness, and behavior over time.
--->
## 🧩 Components

@@ -131,14 +108,19 @@

To use deepchecks for production monitoring, you can either use our SaaS service, or deploy a local instance in one line on Linux/MacOS (Windows is WIP!) with Docker:
To use deepchecks for production monitoring, you can either use our SaaS service, or deploy a local instance in one line on Linux/MacOS (Windows is WIP!) with Docker.
Create a new directory for the installation files, open a terminal within that directory and run the following:
```
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/deepchecks/monitoring/main/deploy/deploy-oss.sh)"
pip install deepchecks-installer
deepchecks-installer monitoring-install
```
This will automatically download the necessary dependencies and start the application locally.
This will automatically download the necessary dependencies, run the installation prcoess
and then start the application locally.
The installation will take a few minutes. Then you can open the deployment url (default is http://localhost),
and start the system onboarding. Check out the full monitoring [open source installation & quickstart](https://docs.deepchecks.com/monitoring/stable/getting-started/deploy_self_host_open_source.html).
Note that the open source product is built such that each deployment supports monitoring of
a single model.
Check out the full installation instructions for deepchecks monitoring [here](https://docs.deepchecks.com/monitoring/stable/installation/index.html).
</details>

@@ -179,3 +161,3 @@

<p align="center">
<img src="docs/source/_static/images/general/model_evaluation_suite.gif" width="800">
<img src="docs/source/_static/images/readme/model-evaluation-suite.gif" width="600">
</p>

@@ -193,3 +175,3 @@

Jump right into the
[monitoring quickstart docs](https://docs.deepchecks.com/monitoring/stable/user-guide/tabular/auto_quickstarts/plot_quickstart.html)
[open source monitoring quickstart docs](https://docs.deepchecks.com/monitoring/stable/getting-started/deploy_self_host_open_source.html)
to have it up and running on your data.

@@ -200,3 +182,3 @@ You'll then be able to see the checks results over time, set alerts, and interact

<p align="center">
<img src="docs/source/_static/images/general/monitoring-app-ui.gif" width="800">
<img src="docs/source/_static/images/general/monitoring-app-ui.gif" width="600">
</p>

@@ -217,3 +199,3 @@

<p align="center">
<img src="docs/source/_static/images/general/deepchecks-ci-checks.png" width="800">
<img src="docs/source/_static/images/general/deepchecks-ci-checks.png" width="600">
</p>

@@ -226,2 +208,27 @@

## 🧮 How does it work?
At its core, deepchecks includes a wide variety of built-in Checks,
for testing all types of data and model related issues.
These checks are implemented for various models and data types (Tabular, NLP, Vision),
and can easily be customized and expanded.
The check results can be used to automatically make informed decisions
about your model's production-readiness, and for monitoring it over time in production.
The check results can be examined with visual reports (by saving them to an HTML file, or seeing them in Jupyter),
processed with code (using their pythonic / json output), and inspected and collaborated on with Deepchecks' dynamic UI
(for examining test results and for production monitoring).
<!---
At its core, Deepchecks has a wide variety of built-in Checks and Suites (lists of checks)
for all data types (Tabular, NLP, Vision),
These includes checks for validating your model's performance (e.g. identify weak segments), the data's
distribution (e.g. detect drifts or leakages), data integrity (e.g. find conflicting labels) and more.
These checks results can be run manually (e.g. during research) or trigerred automatically (e.g. during CI
and production monitoring) and enable automatically making informed decisions regarding your model pipelines'
production-readiness, and behavior over time.
--->
<details open>

@@ -261,2 +268,3 @@ <summary>

## 📜 Open Source vs Paid

@@ -263,0 +271,0 @@

@@ -44,3 +44,2 @@ twine

wandb>=0.12.15,<0.13.0
protobuf>=3.12.0,<4.0dev # wandb does not work if version is higer that 4.0
beautifulsoup4>=4.11.1

@@ -47,0 +46,0 @@

@@ -1,1 +0,1 @@

0.17.2
0.17.3