LLMQuiver
English |
简体中文
The auxiliary tool is used to invoke the online LLM service API, supporting local caching, prompt rendering, and configuration management.
Supported
- Support Openai/Azure LLM service providers (or openai-compatible service providers).
- Support vllm service.
- Local caching (based on sqlite).
- Prompt rendering (based on toml file).
- Configuration management (based on toml file).
To Be Done
- Multi service provider support.
Installation
pip install llm-quiver
Basic Usage
1. Direct Call Mode
Assuming you have a configuration file path/to/gpt.toml
, you need to fill in your own API_KEY, content as follows:
API_TYPE = "azure_openai"
API_BASE = "https://endpoint.openai.azure.com/"
API_VERSION = "2023-05-15"
API_KEY = "********************************"
MODEL_NAME = "gpt-4o-20240513"
temperature = 0.0
max_tokens = 4096
enable_cache = true
cache_dir = "oai_cache"
Running code:
from llm_quiver import LLMQuiver
llm = LLMQuiver(
config_path="path/to/gpt.toml",
)
prompt_values = ["Who are you?"]
responses = llm.generate(prompt_values)
messages = [[{"role": "user", "content": "Who are you?"}]]
responses = llm.chat(messages)
2. Toml Template Call Mode
First, create a TOML template file, for example hello_world.toml
:
[hello_world_template]
prompt = "Hello {name}, who are you?"
Then you can use it like this:
from llm_quiver import TomlLLMQuiver
llm = TomlLLMQuiver(
config_path="path/to/gpt.toml",
toml_prompt_name="hello_world_template",
toml_template_file="path/to/hello_world.toml"
)
prompt_values = [dict(name="GPT")]
responses = llm.generate(prompt_values)
Configuration Guide
There are two ways to configure API keys and other parameters:
- Through environment variables:
Configuration can be loaded by passing parameter config_path="path/to/config.toml" or setting environment variable "export LLMQUIVER_CONFIG=path/to/config.toml". Parameters like API_TYPE, API_BASE, API_VERSION, API_KEY, MODEL_NAME can also be set in environment variables.
- Directly passing configuration file path:
llm = TomlLLMQuiver(
config_path="path/to/config.toml",
toml_prompt_name="template_name",
toml_template_file="path/to/template.toml"
)
Configuration file example:
API_TYPE = "azure_openai"
API_BASE = "https://endpoint.openai.azure.com/"
API_VERSION = "2023-05-15"
API_KEY = "********************************"
MODEL_NAME = "gpt-4o-20240513"
temperature = 0.0
max_tokens = 4096
enable_cache = true
cache_dir = "oai_cache"
Return Value Description
- Both generate() and chat() methods return a list of strings
- Each element corresponds to a response for one input prompt
Notes
- API key must be correctly configured before use
- Template files must comply with TOML format specifications
- Input parameters must correspond to placeholders in the template