vecman
Advanced tools
VECMAN (Vector Manager) - A high-performance VQ-VAE based vector database for efficient text embeddings and retrieval. This package provides a memory-efficient way to store and retrieve text embeddings using Vector Quantized Variational Autoencoder (VQ-VAE) with state-of-the-art performance optimizations.
pip install vecman
"""
VECMAN Example Usage
This script demonstrates how to use the VECMAN package for text embedding,
training a VQ-VAE model, and performing retrieval-augmented generation.
"""
import os
import numpy as np
from pathlib import Path
from vecman import VQVAE, train_corpus, embed_texts, save_jsonl, load_assets, retrieve, generate_answer
def main():
# Example data - more diverse examples for better training + synthetic variations
base_texts = [
"Machine learning is a subset of artificial intelligence that enables computers to learn and improve from experience.",
"Deep learning uses neural networks with multiple layers to model and understand complex patterns in data.",
"Natural language processing helps computers understand, interpret, and generate human language in a valuable way.",
"Computer vision enables machines to interpret and analyze visual information from the world around them.",
"Reinforcement learning trains agents through rewards and penalties to make sequential decisions in an environment.",
"Supervised learning uses labeled training data to learn a mapping from inputs to outputs.",
"Unsupervised learning finds hidden patterns in data without using labeled examples.",
"Transfer learning leverages pre-trained models to solve new but related problems with less data.",
"Feature engineering is the process of selecting and transforming variables for machine learning models.",
"Cross-validation is a technique to assess how well a model will generalize to independent datasets."
]
# Add synthetic variations to expand training data for better VQ-VAE learning
synthetic_texts = [
# ML variations
"Artificial intelligence includes machine learning as a key component for learning from data.",
"Machine learning algorithms improve performance through experience and data exposure.",
"AI systems use machine learning to adapt and enhance their capabilities over time.",
# Deep learning variations
"Neural networks with multiple hidden layers form the foundation of deep learning systems.",
"Deep learning models excel at pattern recognition through layered neural architectures.",
"Multi-layer neural networks enable deep learning to solve complex recognition tasks.",
# NLP variations
"Language models help computers process and understand human communication patterns.",
"Text processing and natural language understanding are core NLP capabilities.",
"Computational linguistics enables machines to work with human language effectively.",
# Computer vision variations
"Image recognition and visual analysis are primary goals of computer vision systems.",
"Visual perception algorithms help machines understand and interpret image content.",
"Computer vision systems process visual data to extract meaningful information.",
# Supervised learning variations
"Training with labeled examples enables supervised learning algorithms to make predictions.",
"Supervised algorithms learn input-output mappings from annotated training datasets.",
"Classification and regression are common supervised learning problem types.",
# Unsupervised learning variations
"Clustering and dimensionality reduction are key unsupervised learning techniques.",
"Pattern discovery in unlabeled data is the main goal of unsupervised methods.",
"Unsupervised algorithms identify hidden structures without labeled training examples."
]
# Combine base and synthetic data for richer training
texts = base_texts + synthetic_texts
print("🚀 VECMAN Example Usage")
print("=" * 50)
try:
# Step 1: Embed texts
print("📝 Step 1: Embedding texts...")
embeddings = embed_texts(texts)
print(f" Generated embeddings shape: {embeddings.shape}")
# Step 2: Save embeddings and documents
print("💾 Step 2: Saving data...")
corpus_path = "example_corpus.npy"
docs_path = "docs.jsonl"
np.save(corpus_path, embeddings)
save_jsonl(texts, docs_path)
print(f" Saved: {corpus_path}, {docs_path}")
# Step 3: Train VQ-VAE
print("🏋️ Step 3: Training VQ-VAE...")
output_dir = train_corpus(
corpus_path,
input_dim=embeddings.shape[1],
epochs=10, # Increased epochs
device="cpu", # Using CPU for compatibility
latent_bits=20, # Increased from 12 to 20 for better representation
batch_size=min(8192, len(texts) * 10), # Increased batch size, but limit for small datasets
learning_rate=1e-3, # Increased learning rate
commitment_beta=0.1 # Lower commitment loss for less quantization pressure
)
print(f" Training completed! Output dir: {output_dir}")
# Step 4: Load trained model
print("📂 Step 4: Loading trained model...")
vqvae, codes, docs = load_assets(output_dir)
print(f" Loaded model with {len(docs)} documents")
# Step 5: Perform retrieval with similarity scores - PURE VQ-VAE ONLY
print("🔍 Step 5: Testing PURE VQ-VAE retrieval (no semantic fallbacks)...")
questions = [
"What is machine learning?",
"How do neural networks work?",
"What is the difference between supervised and unsupervised learning?"
]
for question in questions:
print(f"\n Query: {question}")
# Get query embedding
q_vec = embed_texts([question])[0]
# ONLY VQ-VAE retrieval - no semantic fallbacks
print(" 🔧 VQ-VAE retrieval ONLY:")
vqvae_docs, vqvae_scores = retrieve(vqvae, codes, docs, q_vec, k=3, method="vqvae", return_scores=True)
for i, (doc, score) in enumerate(zip(vqvae_docs, vqvae_scores), 1):
print(f" {i}. [{score:.3f}] {doc[:100]}...")
# Step 6: Optional - Generate answer using ONLY VQ-VAE
api_key = os.getenv("GOOGLE_API_KEY")
if api_key:
print("\n🤖 Step 6: Generating answer with PURE VQ-VAE context...")
question = "What is machine learning?"
q_vec = embed_texts([question])[0]
# Use ONLY VQ-VAE method - no auto fallback
context_docs, context_scores = retrieve(vqvae, codes, docs, q_vec, k=3, method="vqvae", return_scores=True)
# Show which documents were selected with scores
print(f" 📄 VQ-VAE selected context (avg score: {np.mean(context_scores):.3f}):")
for i, (doc, score) in enumerate(zip(context_docs, context_scores), 1):
print(f" {i}. [{score:.3f}] {doc[:80]}...")
# Use a more flexible prompt template
custom_template = """
You are a helpful assistant. Use the following information to answer the question.
If you can find relevant information in the context, provide a clear and informative answer.
If the context doesn't contain relevant information, say 'I don't have enough relevant information.'
Context:
{context}
Question: {question}
Answer:
"""
try:
answer = generate_answer(question, context_docs, api_key=api_key, prompt_template=custom_template)
print(f" Q: {question}")
print(f" A: {answer}")
except Exception as e:
print(f" ⚠️ Could not generate answer: {e}")
else:
print("\n🤖 Step 6: Skipping answer generation (no GOOGLE_API_KEY)")
print("\n✅ VECMAN example completed successfully!")
# Cleanup example files
cleanup_files = [corpus_path, docs_path]
for file_path in cleanup_files:
if os.path.exists(file_path):
os.remove(file_path)
print(f"🧹 Cleaned up: {file_path}")
except Exception as e:
print(f"❌ Example failed: {e}")
import traceback
traceback.print_exc()
# Cleanup on error
cleanup_files = ["example_corpus.npy", "docs.jsonl"]
for file_path in cleanup_files:
if os.path.exists(file_path):
try:
os.remove(file_path)
print(f"🧹 Cleaned up: {file_path}")
except:
pass
if __name__ == "__main__":
main()
#!/usr/bin/env python3
"""
VECMAN RAGAS Evaluation Script for Web Questions Dataset
This script evaluates VECMAN's performance using the RAGAS framework
with the stanfordnlp/web_questions dataset.
"""
import os
import json
import pandas as pd
import numpy as np
from pathlib import Path
from typing import List, Dict, Any, Optional
from tqdm.auto import tqdm
# Core imports
from datasets import load_dataset, Dataset
import torch
# VECMAN imports
from vecman import (
VQVAE,
train_corpus,
embed_texts,
save_jsonl,
load_assets,
retrieve,
generate_answer
)
# RAGAS imports
from ragas import evaluate
from ragas.metrics import (
faithfulness,
answer_relevancy,
context_precision,
context_recall,
answer_correctness
)
# Optional: Enhanced evaluation setup
try:
import google.generativeai as genai
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_community.embeddings import HuggingFaceBgeEmbeddings
ENHANCED_EVAL = True
except ImportError:
print("⚠️ Enhanced evaluation dependencies not found. Using basic evaluation.")
ENHANCED_EVAL = False
class WebQuestionsEvaluator:
"""VECMAN evaluation system for Web Questions dataset with RAGAS integration."""
def __init__(self,
model_dir: str = "webquestions_models",
embedding_model: str = "all-MiniLM-L6-v2",
device: str = "cuda" if torch.cuda.is_available() else "cpu"):
"""
Initialize Web Questions Evaluator.
Args:
model_dir: Directory to store/load VECMAN models
embedding_model: Sentence transformer model for embeddings
device: Device for training ('cuda' or 'cpu')
"""
self.model_dir = Path(model_dir)
self.model_dir.mkdir(exist_ok=True)
self.embedding_model = embedding_model
self.device = device
self.vqvae = None
self.codes = None
self.docs = None
def prepare_dataset(self, dataset_name: str = "stanfordnlp/web_questions",
split: str = "train",
max_samples: Optional[int] = None) -> Dataset:
"""
Load and prepare the Web Questions dataset.
Args:
dataset_name: HuggingFace dataset name
split: Dataset split to use
max_samples: Maximum number of samples to use (None for all)
Returns:
Prepared dataset
"""
print(f"📥 Loading dataset: {dataset_name}")
dataset = load_dataset(dataset_name, split=split)
if max_samples:
dataset = dataset.select(range(min(max_samples, len(dataset))))
print(f"✅ Loaded {len(dataset)} samples")
print(f"📋 Dataset fields: {list(dataset.features.keys())}")
# Display sample for verification
if len(dataset) > 0:
sample = dataset[0]
print(f"📝 Sample question: {sample.get('question', 'N/A')}")
print(f"📝 Sample answers: {sample.get('answers', 'N/A')}")
return dataset
def build_corpus(self, dataset: Dataset, force_rebuild: bool = False) -> None:
"""
Build VECMAN corpus from Web Questions dataset.
Creates documents by combining questions and answers from the dataset.
Args:
dataset: Input Web Questions dataset
force_rebuild: Whether to rebuild even if corpus exists
"""
corpus_path = self.model_dir / "corpus.npy"
docs_path = self.model_dir / "docs.jsonl"
if not force_rebuild and corpus_path.exists() and docs_path.exists():
print("📂 Using existing corpus")
return
print("🔨 Building corpus from Web Questions dataset...")
# Create documents by combining questions and answers
documents = []
for i, row in tqdm(enumerate(dataset), desc="Processing Web Questions", total=len(dataset)):
question = row.get('question', '')
answers = row.get('answers', [])
if question and isinstance(question, str) and question.strip():
# Create a document that includes the question and its answers
doc_parts = [f"Question: {question.strip()}"]
if answers and len(answers) > 0:
# Ensure answers is a list and process each answer
if not isinstance(answers, list):
answers = [answers]
# Join all answers for this question, ensuring they're strings
valid_answers = []
for ans in answers:
if ans is not None:
ans_str = str(ans).strip()
if ans_str:
valid_answers.append(ans_str)
if valid_answers:
answers_text = "; ".join(valid_answers)
doc_parts.append(f"Answers: {answers_text}")
document = "\n".join(doc_parts)
if document.strip(): # Only add non-empty documents
documents.append(document)
print(f"📊 Created {len(documents)} documents from Web Questions")
if len(documents) == 0:
raise ValueError("❌ Could not create any documents from dataset")
# Generate embeddings for the documents
print("🔢 Generating embeddings...")
embeddings = embed_texts(documents, self.embedding_model)
# Save corpus and documents
np.save(self.model_dir / "corpus.npy", embeddings)
save_jsonl(documents, str(self.model_dir / "docs.jsonl"))
print(f"💾 Saved corpus: {embeddings.shape}")
def train_vecman(self, epochs: int = 10, force_retrain: bool = False) -> None:
"""
Train VECMAN model on the corpus.
Args:
epochs: Number of training epochs
force_retrain: Whether to retrain even if model exists
"""
model_path = self.model_dir / "vqvae.pt"
if not force_retrain and model_path.exists():
print("🤖 Using existing trained model")
return
print("🏋️ Training VECMAN model...")
# Load corpus to get input dimension
corpus = np.load(self.model_dir / "corpus.npy")
input_dim = corpus.shape[1]
# Train the model
train_corpus(
str(self.model_dir / "corpus.npy"),
input_dim=input_dim,
epochs=epochs,
device=self.device,
output_dir=str(self.model_dir)
)
print("✅ VECMAN training completed")
def load_vecman(self) -> None:
"""Load trained VECMAN model and assets."""
print("📂 Loading VECMAN model...")
self.vqvae, self.codes, self.docs = load_assets(str(self.model_dir))
print(f"✅ Loaded model with {len(self.docs)} documents")
def vecman_chat(self, question: str, k: int = 5, api_key: Optional[str] = None) -> Dict[str, Any]:
"""
VECMAN-based chat function that retrieves and generates answers using improved retrieval.
Args:
question: User question
k: Number of documents to retrieve
api_key: Google API key for answer generation
Returns:
Dictionary with answer, contexts, similarity scores, and intermediate steps
"""
# Embed the question
q_vec = embed_texts([question], self.embedding_model)[0]
# Retrieve relevant contexts using improved VQ-VAE method with scores
try:
# Get contexts and scores from improved retrieve function
retrieve_result = retrieve(
self.vqvae,
self.codes,
self.docs,
q_vec,
k=k,
method="vqvae", # Use pure VQ-VAE for evaluation
return_scores=True
)
# Handle the return format properly
if isinstance(retrieve_result, tuple):
contexts, scores = retrieve_result
else:
contexts = retrieve_result
scores = [0.0] * len(contexts)
# Ensure contexts is a list of strings
if contexts and isinstance(contexts[0], list):
# Flatten if nested lists
contexts = [str(item) for sublist in contexts for item in (sublist if isinstance(sublist, list) else [sublist])]
# Ensure all contexts are strings
contexts = [str(ctx) if ctx is not None else "" for ctx in contexts]
scores = [float(score) if score is not None else 0.0 for score in scores]
except Exception as e:
print(f"⚠️ Error in retrieval: {e}")
contexts = []
scores = []
# Generate answer if API key is available
if api_key or os.getenv("GOOGLE_API_KEY"):
try:
# Ensure contexts are valid strings before passing to generate_answer
valid_contexts = [ctx for ctx in contexts if ctx and isinstance(ctx, str) and ctx.strip()]
if valid_contexts:
# Custom template for web questions with improved prompt
custom_template = """
You are a helpful assistant. Use the following information to answer the question.
The context contains questions and their answers from a knowledge base.
Focus on providing accurate information based on the retrieved context.
Context:
{context}
Question: {question}
Please provide a clear and factual answer based on the context above.
If the context contains relevant information, synthesize it into a comprehensive answer.
If the context doesn't contain sufficient information, say 'I don't have enough information to answer this question.'
"""
answer = generate_answer(question, valid_contexts, api_key=api_key, prompt_template=custom_template)
else:
answer = "I don't have enough information to answer this question."
except Exception as e:
print(f"⚠️ Error generating answer: {e}")
import traceback
traceback.print_exc()
answer = "I don't know"
else:
answer = "I don't know (no API key provided)"
return {
"output": answer,
"contexts": contexts,
"scores": scores, # Include similarity scores
"intermediate_steps": [("retrieve", "\n---\n".join(contexts))] if contexts else []
}
def select_evaluation_questions(self, dataset: Dataset, num_questions: int = 100) -> List[Dict[str, Any]]:
"""
Select evaluation questions directly from the Web Questions dataset.
Args:
dataset: Web Questions dataset
num_questions: Number of questions to select for evaluation
Returns:
List of dictionaries with question and ground_truth
"""
print(f"📋 Selecting {num_questions} questions from Web Questions dataset...")
# Use test split if available, otherwise use a different portion of the dataset
try:
# Try to load test split
test_dataset = load_dataset("stanfordnlp/web_questions", split="train")
eval_source = test_dataset
print("📊 Using test split for evaluation")
except:
# If no test split, use the end portion of train split
eval_source = dataset
print("📊 Using portion of train split for evaluation")
# Select questions for evaluation
num_available = len(eval_source)
num_to_select = min(num_questions, num_available)
selected_questions = []
for i in range(num_to_select):
row = eval_source[i]
question = row.get('question', '')
answers = row.get('answers', [])
if question.strip() and answers:
# Use the first answer as ground truth, or join multiple answers
if len(answers) == 1:
ground_truth = str(answers[0]).strip()
else:
ground_truth = "; ".join([str(ans).strip() for ans in answers if str(ans).strip()])
selected_questions.append({
"question": question.strip(),
"ground_truth": ground_truth
})
print(f"✅ Selected {len(selected_questions)} valid question-answer pairs")
return selected_questions
def run_evaluation(self,
questions_data: List[Dict[str, Any]],
k: int = 5,
api_key: Optional[str] = None) -> pd.DataFrame:
"""
Run VECMAN evaluation on the selected questions.
Args:
questions_data: List of question-answer pairs
k: Number of documents to retrieve
api_key: Google API key
Returns:
DataFrame with evaluation results
"""
print("🔍 Running VECMAN evaluation...")
# Ensure model is loaded
if self.vqvae is None:
self.load_vecman()
# Prepare evaluation data
eval_data = []
for i, q_data in tqdm(enumerate(questions_data), total=len(questions_data), desc="Evaluating"):
question = q_data["question"]
ground_truth = q_data["ground_truth"]
try:
# Get VECMAN response
out = self.vecman_chat(question, k=k, api_key=api_key)
answer = out["output"]
contexts = out["contexts"]
except Exception as e:
print(f"⚠️ Error processing question {i}: {e}")
answer = "ERROR"
contexts = []
eval_data.append({
"question": question,
"answer": answer,
"contexts": contexts,
"ground_truth": ground_truth
})
# Convert to DataFrame
df = pd.DataFrame(eval_data)
print(f"✅ Evaluation completed on {len(df)} samples")
return df
def compute_ragas_scores(self, eval_df: pd.DataFrame) -> Dict[str, float]:
"""
Compute RAGAS scores for the evaluation results.
Args:
eval_df: DataFrame with evaluation results
Returns:
Dictionary with RAGAS scores
"""
print("📊 Computing RAGAS scores...")
# Convert to RAGAS dataset format
eval_dataset = Dataset.from_pandas(eval_df)
# Basic metrics that work without additional setup
basic_metrics = [
faithfulness,
answer_relevancy,
context_precision,
context_recall,
]
# Enhanced metrics if dependencies are available
if ENHANCED_EVAL:
try:
# Setup enhanced evaluation with custom LLM and embeddings
emb = HuggingFaceBgeEmbeddings(model_name="BAAI/bge-small-en-v1.5")
ragas_emb = LangchainEmbeddingsWrapper(emb)
result = evaluate(
dataset=eval_dataset,
metrics=basic_metrics + [answer_correctness],
embeddings=ragas_emb,
)
except Exception as e:
print(f"⚠️ Enhanced evaluation failed: {e}")
print("📊 Falling back to basic evaluation...")
result = evaluate(dataset=eval_dataset, metrics=basic_metrics)
else:
result = evaluate(dataset=eval_dataset, metrics=basic_metrics)
# Convert to pandas and get mean scores
result_df = result.to_pandas()
scores = {}
for metric in result_df.columns:
if metric not in ['question', 'answer', 'contexts', 'ground_truth']:
scores[metric] = result_df[metric].mean()
return scores
def save_results(self, eval_df: pd.DataFrame, scores: Dict[str, float],
output_file: str = "webquestions_evaluation_results.json") -> None:
"""
Save evaluation results to file.
Args:
eval_df: Evaluation DataFrame
scores: RAGAS scores
output_file: Output file path
"""
results = {
"evaluation_summary": scores,
"detailed_results": eval_df.to_dict('records'),
"total_samples": len(eval_df),
"model_config": {
"embedding_model": self.embedding_model,
"device": self.device,
"model_dir": str(self.model_dir)
}
}
with open(output_file, 'w') as f:
json.dump(results, f, indent=2, default=str)
print(f"💾 Results saved to {output_file}")
def main():
"""Main evaluation pipeline for Web Questions dataset."""
print("🚀 VECMAN RAGAS Evaluation Pipeline - Web Questions")
print("=" * 60)
# Configuration
MAX_SAMPLES = None # Use full dataset for corpus building
EPOCHS = 30
K_RETRIEVE = 50 # Top 10 relevant contexts
NUM_QUESTIONS = 10 # Use 100 questions from dataset
# Initialize evaluator
evaluator = WebQuestionsEvaluator()
# Step 1: Load Web Questions dataset
print("📥 Step 1: Loading Web Questions dataset...")
dataset = evaluator.prepare_dataset(max_samples=MAX_SAMPLES)
# Step 2: Build corpus from Web Questions
print("🔨 Step 2: Building corpus from Web Questions...")
evaluator.build_corpus(dataset, force_rebuild=True)
# Step 3: Train VECMAN
print("🏋️ Step 3: Training VECMAN...")
evaluator.train_vecman(epochs=EPOCHS, force_retrain=True)
# Step 4: Load VECMAN
print("📂 Step 4: Loading VECMAN...")
evaluator.load_vecman()
# Step 5: Select evaluation questions from dataset
print("📋 Step 5: Selecting evaluation questions from dataset...")
questions_data = evaluator.select_evaluation_questions(dataset, NUM_QUESTIONS)
# Step 6: Run evaluation
print("🔍 Step 6: Running RAGAS evaluation...")
eval_df = evaluator.run_evaluation(
questions_data,
k=K_RETRIEVE,
api_key=os.getenv("GOOGLE_API_KEY")
)
print(f"✅ Created RAGAS dataset with {len(eval_df)} samples")
print(f"📊 Each sample has: 1 question, 1 answer, {K_RETRIEVE} contexts, 1 ground_truth")
# Step 7: Compute RAGAS scores
try:
scores = evaluator.compute_ragas_scores(eval_df)
print("\n📊 RAGAS Evaluation Results:")
print("-" * 30)
for metric, score in scores.items():
print(f"{metric:20s}: {score:.4f}")
# Step 8: Save results
evaluator.save_results(eval_df, scores)
except Exception as e:
print(f"⚠️ Error computing RAGAS scores: {e}")
print("💾 Saving evaluation data without RAGAS scores...")
evaluator.save_results(eval_df, {})
print("\n✅ VECMAN-RAGAS evaluation on Web Questions completed!")
print("🎯 Check 'webquestions_evaluation_results.json' for detailed results")
if __name__ == "__main__":
main()
| Parameter | Default | Optimized | Description |
|---|---|---|---|
epochs | 5 | 20 | Training epochs for better convergence |
latent_bits | 16 | 20 | Higher resolution latent space |
learning_rate | 3e-4 | 1e-3 | Faster training convergence |
batch_size | 4096 | 8192 | Better gradient estimates |
commitment_beta | 0.25 | 0.1 | Less quantization pressure |
| Method | Use Case | Performance | Description |
|---|---|---|---|
vqvae | Primary | Fastest | Uses trained VQ-VAE encoder |
semantic | Fallback | Slower | Direct sentence transformer |
auto | Recommended | Adaptive | Smart fallback mechanism |
Low similarity scores (<0.3)?
# Increase training parameters
train_corpus(corpus_npy, latent_bits=24, epochs=30, learning_rate=1e-3)
Memory issues?
# Reduce batch size
train_corpus(corpus_npy, batch_size=4096, device="cpu")
Poor retrieval quality?
# Use more diverse training data
# Increase latent_bits for better representation
# Check embedding quality with different models
Note: Comprehensive benchmarks are currently in development. The performance of VECMAN will depend on:
Based on the architectural improvements we've implemented:
| Improvement | Expected Impact |
|---|---|
| Increased Learning Rate (3e-4 → 1e-3) | ~3x faster training convergence |
| Higher Latent Bits (16 → 20) | Better representation quality |
| Larger Batch Size (4096 → 8192) | More stable gradients |
| Encoder-to-Encoder Comparison | More accurate similarity computation |
You can benchmark VECMAN on your own data using our evaluation tools:
# Run your own benchmarks
from evaluate_webquestions_fixed import WebQuestionsEvaluator
evaluator = WebQuestionsEvaluator()
# ... setup and training ...
results = evaluator.run_evaluation(questions, k=5)
print("📊 Your Performance Metrics:")
print(f"Average Retrieval Score: {results['avg_retrieval_score'].mean():.3f}")
print(f"High Confidence Rate: {(results['max_retrieval_score'] > 0.5).mean()*100:.1f}%")
We encourage users to share their benchmark results! If you've evaluated VECMAN on your datasets, please:
Help us build a reliable benchmark database! 🤝
Thank you for catching that error. It's important to be honest about what we actually know vs. what we're speculating about. The improvements we made to VECMAN are real and documented, but specific performance numbers should come from actual testing, not fabrication.
We welcome contributions! Please see our Contributing Guidelines for details.
git clone https://github.com/yourusername/vecman.git
cd vecman
pip install -e ".[dev]"
pytest tests/
If you use VECMAN in your research, please cite:
@software{vecman2024,
title={VECMAN: High-Performance VQ-VAE Vector Database},
author={Loaii abdalslam},
year={2025},
url={https://github.com/yourusername/vecman}
}
This project is licensed under the MIT License - see the LICENSE file for details.
MIT License
Copyright (c) 2024 VECMAN
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Built with ❤️ for the ML community
FAQs
VECMAN - A VQ-VAE based vector database for text embeddings and retrieval
We found that vecman demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
OpenAI rotated macOS signing certificates after a malicious Axios package reached its CI pipeline in a broader software supply chain attack.
Security News
Open source is under attack because of how much value it creates. It has been the foundation of every major software innovation for the last three decades. This is not the time to walk away from it.
Security News
Socket CEO Feross Aboukhadijeh breaks down how North Korea hijacked Axios and what it means for the future of software supply chain security.