inference-activity-axios

inference-activity-axios

Axios interceptors for tracking inference activities on Heroku AI. This package helps you monitor and log API calls to inference endpoints while automatically redacting sensitive information from requests and responses.

Features

• Automatically tracks request/response times
• Logs API call activities to a specified endpoint
• Redacts sensitive information from:
  • Chat completion messages
  • Embedding inputs
  • Image generation prompts
• Handles errors gracefully
• Zero configuration needed beyond environment variables
• Support for streaming response

Installation

npm install inference-activity-axios

Usage

const axios = require('axios');
const { applyInterceptors } = require('inference-activity-axios');

// Create your axios instance
const api = axios.create({
    baseURL: process.env.INFERENCE_URL,
    headers: {
        'Authorization': `Bearer ${process.env.INFERENCE_KEY}`,
        'Content-Type': 'application/json'
    }
});

// Apply the interceptors to start tracking
applyInterceptors(api);

Environment Variables

The package requires the following environment variables:

• For Heroku Inference API:

  • INFERENCE_URL: Base URL for the inference API
  • INFERENCE_KEY: API key for authentication
  • INFERENCE_MODEL_ID: Model ID to use for inference

• For Activity Logging:

  • INFERENCE_ACTIVITY_URL: URL where activity logs will be sent
  • INFERENCE_ACTIVITY_KEY: API key for authentication with the activity logging service

You can set them up using:

# For inference API
export INFERENCE_URL=$(heroku config:get -a $APP_NAME INFERENCE_URL)
export INFERENCE_KEY=$(heroku config:get -a $APP_NAME INFERENCE_KEY)
export INFERENCE_MODEL_ID=$(heroku config:get -a $APP_NAME INFERENCE_MODEL_ID)

# For activity logging
export INFERENCE_ACTIVITY_URL=$(heroku config:get -a $APP_NAME INFERENCE_ACTIVITY_URL)
export INFERENCE_ACTIVITY_KEY=$(heroku config:get -a $APP_NAME INFERENCE_ACTIVITY_KEY)

Activity Logging

When activity logging is enabled (by setting INFERENCE_ACTIVITY_URL and INFERENCE_ACTIVITY_KEY), the following information is logged for each API call:

{
  timestamp: Date.now(),
  response_time: duration,      // Request duration in milliseconds
  status_code: response.status, // HTTP status code
  status_message: statusText,   // HTTP status message
  request: {
    method: 'POST',
    url: '/v1/chat/completions',
    params: {},
    body: {                     // Sensitive data is redacted
      model: 'gpt-3.5-turbo',
      messages: '[REDACTED]',
      temperature: 0.5
    }
  },
  response: {                   // Sensitive data is redacted
    headers: {...},
    data: {
      choices: [{
        message: {
          content: '[REDACTED]'
        }
      }]
    }
  }
}

Example: Chat Completion

const axios = require('axios');
const { applyInterceptors } = require('inference-activity-axios');

const api = axios.create({
    baseURL: process.env.INFERENCE_URL,
    headers: {
        'Authorization': `Bearer ${process.env.INFERENCE_KEY}`,
        'Content-Type': 'application/json'
    }
});

// Apply the interceptors to start tracking
applyInterceptors(api);

const payload = {
    model: process.env.INFERENCE_MODEL_ID,
    messages: [
        { role: "user", content: "Hello!" },
        { role: "assistant", content: "Hi there! How can I assist you today?" },
        { role: "user", content: "Why is Heroku so cool?" }
    ],
    temperature: 0.5,
    max_tokens: 100,
    stream: false
};

async function generateChatCompletion(payload) {
    try {
        const response = await api.post('/v1/chat/completions', payload);
        console.log("Chat Completion:", response.data.choices[0].message.content);
    } catch (error) {
        console.error("Error generating chat completion:", error.message);
    }
}

generateChatCompletion(payload);

Example: Chat Completion with Streaming

const axios = require('axios');
const { applyInterceptors } = require('inference-activity-axios');

const api = axios.create({
	baseURL: process.env.INFERENCE_URL,
	headers: {
		'Authorization': `Bearer ${process.env.INFERENCE_KEY}`,
		'Content-Type': 'application/json'
	}
});

// Apply the interceptors to start tracking
applyInterceptors(api);

const payload = {
	model: process.env.INFERENCE_MODEL_ID,
	messages: [
		{ role: "user", content: "Hello!" },
		{ role: "assistant", content: "Hi there! How can I assist you today?" },
		{ role: "user", content: "Why is Heroku so cool?" }
	],
	temperature: 0.5,
	max_tokens: 100,
	stream: true
};

async function generateChatCompletion(payload) {
	try {
		const response = await api.post('/v1/chat/completions', payload, { responseType: 'stream' });
		response.data.on('data', chunk => {
			process.stdout.write(chunk);
		});
	} catch (error) {
		console.error("Error generating chat completion:", error.message);
	}
}

generateChatCompletion(payload);

Redaction Rules

The package automatically redacts sensitive information:

• Chat Completions (/v1/chat/completions):

  • Request: message contents
  • Response: generated message content

• Embeddings (/v1/embeddings):

  • Request: input text
  • Response: embedding vectors

• Image Generation (/v1/images/generations):

  • Request: prompt and negative_prompt
  • Response: b64_json and revised_prompt

License

MIT