A Python implementation of Adaptive Momentum Gradient Descent (AMGD) for high-dimensional sparse Poisson regression with L1 and Elastic Net regularization. AMGD combines adaptive learning rates with momentum-based updates and specialized soft-thresholding to achieve superior performance in feature selection and optimization efficiency.
AMGD demonstrates effecient performance in regularized Poisson regression, as validated by comprehensive experiments on ecological count data:
| Metric | AMGD vs Adam | AMGD vs AdaGrad | AMGD vs GLMnet |
|---|---|---|---|
| MAE | 2.7% lower | 56.6% lower | 66.4% lower |
| RMSE | 2.9% lower | 49.2% lower | 59.0% lower |
| MPD | 3.0% lower | 81.0% lower | 92.4% lower |
| Sparsity | 11× higher | ∞ (from 0%) | ∞ (from 0%) |
| Runtime | 2× faster | >370× faster | 20× faster |
| Stat. Significance | p < 0.0001 (all) | ✅ Large effect sizes |
✅ Achieves 35.29% model sparsity (vs 11.76% for Adam, 0% for AdaGrad and GLMnet)
✅ Bootstrapped performance is stable across 1000 runs with tight confidence intervals
✅ Adaptive thresholding outperforms static regularization, enabling effective feature selection
✅ Handles high-dimensional data with robust convergence and minimal tuning
“AMGD provides up to 56.6% MAE reduction over AdaGrad, and outperforms Adam on every metric while selecting significantly fewer features.”
— Bakari & Özkale (2025)
pip install amgd
git clone https://github.com/elbakari01/amgd.git
cd amgd
pip install -e .
numpy>=1.21.0
scipy>=1.7.0
scikit-learn>=1.0.0
matplotlib>=3.3.0
seaborn>=0.11.0
import numpy as np
from amgd.models import PoissonRegressor
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
# Generate sample data
X, y = make_regression(n_samples=1000, n_features=50, noise=0.1, random_state=42)
y = np.abs(y.astype(int)) # Convert to count data for Poisson
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Train AMGD model
model = PoissonRegressor(optimizer='amgd', penalty='l1', lambda1=0.1)
model.fit(X_train, y_train)
# Evaluate
print(f"Test score: {model.score(X_test, y_test):.4f}")
print(f"Selected features: {np.sum(model.coef_ != 0)}/{len(model.coef_)}")
from amgd.models import PoissonRegressor
# Create and train model
model = PoissonRegressor(
optimizer='amgd',
penalty='l1',
lambda1=0.1,
max_iter=1000,
verbose=True
)
model.fit(X_train, y_train)
# Make predictions
y_pred = model.predict(X_test)
# Evaluate performance
from amgd.utils.metrics import poisson_deviance
deviance = poisson_deviance(y_test, y_pred)
print(f"Poisson deviance: {deviance:.4f}")
# Best for correlated features
model = PoissonRegressor(
optimizer='amgd',
penalty='elasticnet',
lambda1=0.05, # L1 penalty (sparsity)
lambda2=0.05, # L2 penalty (grouping)
max_iter=1000
)
model.fit(X_train, y_train)
# Analyze sparsity
sparsity = np.mean(model.coef_ == 0)
print(f"Sparsity: {sparsity:.2%}")
print(f"Non-zero coefficients: {np.sum(model.coef_ != 0)}")
from amgd.models import PoissonRegressor
from amgd.visualization import plot_convergence, plot_feature_importance
import pandas as pd
# Load your ecological data
# X: environmental features (temperature, rainfall, etc.)
# y: species counts
data = pd.read_csv('ecological_data.csv')
X = data[['temperature', 'rainfall', 'elevation', 'soil_ph']].values
y = data['species_count'].values
# Cross-validation for hyperparameter tuning
from sklearn.model_selection import cross_val_score
lambda_values = np.logspace(-4, 1, 20)
cv_scores = []
for lam in lambda_values:
model = PoissonRegressor(optimizer='amgd', penalty='l1', lambda1=lam)
scores = cross_val_score(model, X, y, cv=5, scoring='neg_mean_poisson_deviance')
cv_scores.append(-scores.mean())
best_lambda = lambda_values[np.argmin(cv_scores)]
# Train final model
model = PoissonRegressor(optimizer='amgd', penalty='l1', lambda1=best_lambda)
model.fit(X, y)
# Visualize results
plot_convergence(model.loss_history_, title="AMGD Convergence")
plot_feature_importance(model.coef_, feature_names=['Temperature', 'Rainfall', 'Elevation', 'Soil pH'])
from amgd.benchmarks import compare_optimizers
# Compare multiple optimizers
results = compare_optimizers(
X, y,
optimizers=['amgd', 'adam', 'adagrad'],
penalties=['l1', 'elasticnet'],
cv_folds=5,
test_size=0.2,
random_state=42
)
# View results
print("Cross-Validation Results:")
print(results['cv_results'])
print("\nTest Set Results:")
print(results['test_results'])
# Statistical significance testing
from amgd.benchmarks import statistical_significance_test
stats_results = statistical_significance_test(
X, y,
optimizers=['amgd', 'adam'],
n_runs=50
)
from amgd.core.penalties import PenaltyBase
import numpy as np
class AdaptiveLassoPenalty(PenaltyBase):
"""Adaptive Lasso with feature-specific weights."""
def __init__(self, lambda1, weights):
self.lambda1 = lambda1
self.weights = weights
def __call__(self, x):
return self.lambda1 * np.sum(self.weights * np.abs(x))
def gradient(self, x):
return self.lambda1 * self.weights * np.sign(x)
def proximal_operator(self, x, step_size):
threshold = self.lambda1 * self.weights * step_size
return np.sign(x) * np.maximum(np.abs(x) - threshold, 0)
# Use custom penalty
initial_weights = 1.0 / (np.abs(ridge_coef) + 0.1) # From ridge regression
penalty = AdaptiveLassoPenalty(lambda1=0.1, weights=initial_weights)
model = PoissonRegressor(optimizer='amgd', penalty=penalty)
model.fit(X_train, y_train)
AMGD combines three key innovations for superior performance:
αₜ = α₀ / (1 + ηt)
α₀: Initial learning rate (default: 0.01)η: Decay rate (default: 0.0001)t: Current iterationmₜ = β₁mₜ₋₁ + (1 - β₁)∇f(θₜ) # First moment
vₜ = β₂vₜ₋₁ + (1 - β₂)(∇f(θₜ))² # Second moment
m̂ₜ = mₜ / (1 - β₁ᵗ) # Bias correction
v̂ₜ = vₜ / (1 - β₂ᵗ) # Bias correction
θⱼ ← prox_λ(θⱼ - αₜ∇f(θⱼ))
where prox_λ(x) = sign(x) · max(|x| - λ, 0)
| Component | Traditional Methods | AMGD Innovation | Benefit |
|---|---|---|---|
| Learning Rate | Fixed or simple decay | Adaptive + exponential decay | Better convergence |
| Momentum | Standard momentum | Bias-corrected dual momentum | Faster escape from local minima |
| Regularization | Basic soft-thresholding | Adaptive thresholding | Superior feature selection |
import joblib
# Save trained model
model.fit(X_train, y_train)
joblib.dump(model, 'amgd_model.pkl')
# Load and use
loaded_model = joblib.load('amgd_model.pkl')
predictions = loaded_model.predict(X_new)
model = PoissonRegressor(optimizer='amgd', warm_start=True)
# Progressive training
for lambda_val in [0.1, 0.05, 0.01]:
model.lambda1 = lambda_val
model.fit(X_train, y_train)
print(f"Lambda {lambda_val}: Score = {model.score(X_val, y_val):.4f}")
from amgd.core.convergence import RelativeChangeCriterion
# Custom convergence
criterion = RelativeChangeCriterion(tol=1e-8, patience=10)
model = PoissonRegressor(
optimizer='amgd',
convergence_criterion=criterion,
max_iter=2000
)
from amgd.visualization import (
plot_convergence_comparison,
plot_coefficient_path,
plot_feature_importance
)
# Compare convergence of different optimizers
histories = {
'AMGD': model_amgd.loss_history_,
'Adam': model_adam.loss_history_,
'AdaGrad': model_adagrad.loss_history_
}
plot_convergence_comparison(histories)
# Coefficient regularization path
lambdas = np.logspace(-4, 1, 50)
coef_path = np.array([fit_model(lam).coef_ for lam in lambdas])
plot_coefficient_path(lambdas, coef_path, feature_names=feature_names)
PoissonRegressorMain class for Poisson regression with AMGD optimization.
PoissonRegressor(
optimizer='amgd', # Optimizer type
penalty='l1', # Regularization type
lambda1=0.0, # L1 penalty strength
lambda2=0.0, # L2 penalty strength
alpha=0.01, # Learning rate
beta1=0.9, # First moment decay
beta2=0.999, # Second moment decay
max_iter=1000, # Maximum iterations
tol=1e-6, # Convergence tolerance
fit_intercept=True, # Fit intercept term
warm_start=False, # Use previous solution
verbose=False # Print progress
)
| Method | Description | Returns |
|---|---|---|
fit(X, y) | Train the model | self |
predict(X) | Make predictions | ndarray |
score(X, y) | Model performance | float |
get_params() | Get parameters | dict |
set_params(**params) | Set parameters | self |
from amgd.utils.metrics import poisson_deviance, mean_absolute_error
from amgd.utils.validation import check_array, check_is_fitted
from amgd.benchmarks import compare_optimizers, statistical_significance_test
Run the test suite:
# Install test dependencies
pip install pytest pytest-cov
# Run all tests
pytest tests/ -v
# Run with coverage
pytest tests/ --cov=amgd --cov-report=html
# Run specific test categories
pytest tests/test_models.py -v # Model tests
pytest tests/test_optimizers.py -v # Optimizer tests
pytest tests/test_benchmarks.py -v # Benchmark tests
model = PoissonRegressor(
optimizer='amgd',
alpha=0.001, # Lower learning rate
max_iter=500, # Fewer iterations
tol=1e-4 # Relaxed tolerance
)
model = PoissonRegressor(
optimizer='amgd',
penalty='elasticnet',
lambda1=0.01, # Stronger L1 penalty
lambda2=0.001, # Light L2 penalty
beta1=0.95 # Higher momentum
)
# For memory-constrained environments
model = PoissonRegressor(
optimizer='amgd',
fit_intercept=False, # Skip intercept if not needed
warm_start=True, # Reuse computations
verbose=False # Reduce memory overhead
)
We welcome contributions! Here's how to get started:
# Fork and clone the repository
git clone https://github.com/yourusername/amgd.git
cd amgd
# Create virtual environment
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# Install in development mode
pip install -e ".[dev]"
# Install pre-commit hooks
pre-commit install
git checkout -b feature/amazing-feature)pytest tests/)flake8 src/)git commit -m 'Add amazing feature')git push origin feature/amazing-feature)This project is licensed under the MIT License - see the LICENSE file for details.
If you use AMGD in your research, please cite our paper:
@article{bakari2025amgd,
title={Adaptive Momentum Gradient Descent: A Novel Optimization Algorithm for Regularized Poisson Regression},
author={Bakari, Ibrahim and Özkale, M. Revan},
journal={arXiv preprint arXiv:2025.XXXXX},
year={2025},
note={Submitted for publication}
}
AMGD is ideal for:
✅ High-dimensional data (p > n problems)
✅ Sparse feature selection required
✅ Count/rate data modeling
✅ Correlated predictors present
✅ Interpretable models needed
Planned enhancements for future releases:
Vote for features or contribute:
⭐ Star this repository if you find it useful! ⭐
FAQs
Adaptive Momentum Gradient Descent for Regularized Poisson Regression
We found that amgd demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
/Security News
North Korean threat actors deploy 67 malicious npm packages using the newly discovered XORIndex malware loader.
Security News
Meet Socket at Black Hat & DEF CON 2025 for 1:1s, insider security talks at Allegiant Stadium, and a private dinner with top minds in software supply chain security.
Security News
CAI is a new open source AI framework that automates penetration testing tasks like scanning and exploitation up to 3,600× faster than humans.
