Huge News!Announcing our $40M Series B led by Abstract Ventures.Learn More
Socket
Sign inDemoInstall
Socket

flytekitplugins-whylogs

Package Overview
Dependencies
Maintainers
1
Alerts
File Explorer

Advanced tools

Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

flytekitplugins-whylogs

Enable the use of whylogs profiles to be used in flyte tasks to get aggregate statistics about data.

  • 1.14.0
  • PyPI
  • Socket score

Maintainers
1

Flytekit whylogs Plugin

whylogs is an open source library for logging any kind of data. With whylogs, you are able to generate summaries of datasets (called whylogs profiles) which can be used to:

  • Create data constraints to know whether your data looks the way it should
  • Quickly visualize key summary statistics about a dataset
  • Track changes in a dataset over time
pip install flytekitplugins-whylogs

To generate profiles, you can add a task like the following:

import whylogs as why
from whylogs.core import DatasetProfileView

import pandas as pd

from flytekit import task

@task
def profile(df: pd.DataFrame) -> DatasetProfileView:
    result = why.log(df) # Various overloads for different common data types exist
    profile_view = result.view()
    return profile

NOTE: You'll be passing around DatasetProfileView from tasks, not DatasetProfile.

Validating Data

A common step in data pipelines is data validation. This can be done in whylogs through the constraint feature. You'll be able to create failure tasks if the data in the workflow doesn't conform to some configured constraints, like min/max values on features, data types on features, etc.

from whylogs.core.constraints.factories import greater_than_number, mean_between_range

@task
def validate_data(profile_view: DatasetProfileView):
    builder = ConstraintsBuilder(dataset_profile_view=profile_view)
    builder.add_constraint(greater_than_number(column_name="my_column", number=0.14))
    builder.add_constraint(mean_between_range(column_name="my_other_column", lower=2, upper=3))
    constraint = builder.build()
    valid = constraint.validate()

    if valid is False:
        print(constraint.report())
        raise Exception("Invalid data found")

If you want to learn more about whylogs, check out our example notebooks.

FAQs


Did you know?

Socket

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install

Related posts

SocketSocket SOC 2 Logo

Product

  • Package Alerts
  • Integrations
  • Docs
  • Pricing
  • FAQ
  • Roadmap
  • Changelog

Packages

npm

Stay in touch

Get open source security insights delivered straight into your inbox.


  • Terms
  • Privacy
  • Security

Made with ⚡️ by Socket Inc