llm-quiver

LLMQuiver

English | 简体中文

The auxiliary tool is used to invoke the online LLM service API, supporting local caching, prompt rendering, and configuration management.

Supported

• Support Openai/Azure LLM service providers (or openai-compatible service providers).
• Support vllm service.
• Local caching (based on sqlite).
• Prompt rendering (based on toml file).
• Configuration management (based on toml file).

To Be Done

• Multi service provider support.

Installation

pip install llm-quiver

Basic Usage

1. Direct Call Mode

Assuming you have a configuration file path/to/gpt.toml, you need to fill in your own API_KEY, content as follows:

API_TYPE = "azure_openai"
API_BASE = "https://endpoint.openai.azure.com/"
API_VERSION = "2023-05-15"
API_KEY = "********************************"
MODEL_NAME = "gpt-4o-20240513"
temperature = 0.0
max_tokens = 4096
enable_cache = true
cache_dir = "oai_cache"

Running code:

from llm_quiver import LLMQuiver

# Initialize
llm = LLMQuiver(
    config_path="path/to/gpt.toml",
)

# Text generation mode
prompt_values = ["Who are you?"]
responses = llm.generate(prompt_values)
#   Default role is system
#   ["I am an AI assistant developed by OpenAI, designed to help answer questions, provide information, and complete various tasks. How can I help you?"]

# Chat mode
messages = [[{"role": "user", "content": "Who are you?"}]]
responses = llm.chat(messages)
#   ["I am an AI assistant developed by OpenAI, designed to help answer questions, provide information, and engage in conversations. Feel free to ask me anything!"]

2. Toml Template Call Mode

First, create a TOML template file, for example hello_world.toml:

[hello_world_template]
prompt = "Hello {name}, who are you?"

Then you can use it like this:

from llm_quiver import TomlLLMQuiver

# Specify template during initialization
llm = TomlLLMQuiver(
    config_path="path/to/gpt.toml",
    toml_prompt_name="hello_world_template",
    toml_template_file="path/to/hello_world.toml"
)

# Pass template parameters
prompt_values = [dict(name="GPT")]
responses = llm.generate(prompt_values)

Configuration Guide

There are two ways to configure API keys and other parameters:

• Through environment variables:

Configuration can be loaded by passing parameter config_path="path/to/config.toml" or setting environment variable "export LLMQUIVER_CONFIG=path/to/config.toml". Parameters like API_TYPE, API_BASE, API_VERSION, API_KEY, MODEL_NAME can also be set in environment variables.

• Directly passing configuration file path:

llm = TomlLLMQuiver(
    config_path="path/to/config.toml",
    toml_prompt_name="template_name",
    toml_template_file="path/to/template.toml"
)

Configuration file example:

API_TYPE = "azure_openai"
API_BASE = "https://endpoint.openai.azure.com/"
API_VERSION = "2023-05-15"
API_KEY = "********************************"
MODEL_NAME = "gpt-4o-20240513"
temperature = 0.0
max_tokens = 4096
enable_cache = true
cache_dir = "oai_cache"

Return Value Description

• Both generate() and chat() methods return a list of strings
• Each element corresponds to a response for one input prompt

Notes

• API key must be correctly configured before use
• Template files must comply with TOML format specifications
• Input parameters must correspond to placeholders in the template