feat: Implement prompt caching support for Anthropic and some Gemini models. #37
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Implement persistent storage for OpenRouter model configurations using SQLite. This allows users to save specific combinations of model options (e.g., provider settings, online mode) under custom aliases for easy reuse. Key changes: - Introduce database functions (`save_config`, `get_config`, etc.) and schema for storing configurations. - Add CLI commands `openrouter save`, `openrouter list-saved`, and `openrouter remove-saved` to manage saved aliases. - Modify model registration to load saved configurations and register them as distinct models identifiable by their alias. - Update model classes to apply saved options when building API requests, allowing prompt-specific options to override saved ones.
This commit refactors the display of the `architecture` field in the output of the `llm openrouter models` command. Previously, the architecture display was incomplete and could result in extra spaces. This change iterates through the architecture values: - If a value is a list, it's now joined into a comma-separated string. - Non-list truthy values are included as strings. - Only includes the `architecture` line if there are values to display. This provides a more accurate and readable representation of the model architecture.
Adds support for the prompt caching feature available on OpenRouter for certain models (like Anthropic and Gemini). Introduces two new options: - `cache_prompt`: Boolean to indicate whether the user prompt should be cached. - `cache_system`: Boolean to indicate whether the system prompt should be cached. Formats messages using the chat completion `content` array format with `cache_control` elements for supported models/providers based on the new options. Updates the `README.md` with documentation and examples for the new prompt caching options.
added 2 commits
May 1, 2025 16:35
Implements functionality to save frequently used OpenRouter model configurations (including specific options like `online` and `provider`) under aliases. This allows users to define custom model shortcuts with predefined options, simplifying repeated use of specific configurations. New commands added: - `llm openrouter save <base_model_id> --name <alias> -o <key> <value> ...` - `llm openrouter list-saved` - `llm openrouter remove-saved <alias>` Saved configurations are stored in an SQLite database within the user's LLM directory. Updates README.md with documentation and examples for the new saved configuration feature. Refactors the `register_models` function to load saved configurations and register them as alias models. Adjusts model initialization and option handling to incorporate saved options. Fixes minor output formatting for architecture info in `llm openrouter models`.
I resolved a merge conflict in `README.md`, integrating sections for prompt caching and saved configurations.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds support for Anthropic and Gemini prompt caching control.
Introduces two new options:
cache_prompt: Boolean to indicate whether the user prompt should be cached.cache_system: Boolean to indicate whether the system prompt should be cached.Formats messages using the chat completion
contentarray format withcache_controlelements for supported models/providers based on the new options.Updates the
README.mdwith documentation and examples for the new prompt caching options.