
Research
/Security News
Contagious Interview Campaign Escalates With 67 Malicious npm Packages and New Malware Loader
North Korean threat actors deploy 67 malicious npm packages using the newly discovered XORIndex malware loader.
AutoEvaluator is a Python library that speeds up the large language models (LLMs) output generation QC work. It provides a simple, transparent, and user-friendly API to identify the True Positives (TP), False Positives (FP), and False Negatives (FN) statements based the generated statement and ground truth provided. Get ready to turbocharge your LLM evaluations!
Features:
Autoevaluator requires Python 3.9
and several dependencies. You can install autoevaluator:
pip install autoevaluator
Prepare your data:
setup environment variables
import os
os.environ["OPENAI_API_KEY"] = "<OPENAI_API_KEY>"
os.environ["AZURE_OPENAI_API_KEY"] = "<AZURE_OPENAI_API_KEY>"
os.environ["AZURE_OPENAI_ENDPOINT"] = "<AZURE_OPENAI_ENDPOINT>"
os.environ["DEPLOYMENT"] = "<azure>/<not-azure>"
# Import the evaluate function from the autoevaluator module
from autoevaluator import evaluate, setup_client
# setup openai client
client, model = setup_client()
# Define the claim to be evaluated
claim = 'Feynmann was born in 1918 in Malaysia'
# Define the ground truth statement
ground_truth = 'Feynmann was born in 1918 in America.'
# Evaluate the claim against the ground truth
evaluate(claim, ground_truth, client=client, model_name = model)
# output
{'TP': ['Feynmann was born in 1918.'],
'FP': ['Feynmann was born in Malaysia.'],
'FN': ['Feynmann was born in America.'],
'recall': 0.5,
'precision': 0.5,
'f1_score': 0.5}
License:
This project is licensed under the MIT License. See the LICENSE
file for details.
FAQs
Fully automated LLM evaluator
We found that autoevaluator demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
/Security News
North Korean threat actors deploy 67 malicious npm packages using the newly discovered XORIndex malware loader.
Security News
Meet Socket at Black Hat & DEF CON 2025 for 1:1s, insider security talks at Allegiant Stadium, and a private dinner with top minds in software supply chain security.
Security News
CAI is a new open source AI framework that automates penetration testing tasks like scanning and exploitation up to 3,600× faster than humans.