[OPIK-2982] [P SDK] [BE] [FE] ChatPrompt support #3987

alexkuzmik · 2025-11-07T16:46:35Z

Details

Change checklist

User facing
Documentation update

Issues

Resolves #
OPIK-

Testing

Documentation

- SDK: Add ChatPrompt class using ChatPromptTemplate, serialize to JSON with template_structure='chat' - Backend: Add template_structure field to PromptVersion model, create migration script - Frontend: Add template_structure column to prompts table, implement chat prompt creation UI with message-by-message editor, add ChatPromptView component for displaying chat prompts - Tests: Add unit and e2e tests for ChatPrompt functionality

…ructure field

- Improve ChatPromptView with modern chat interface design - Add circular avatar icons with role-specific colors - Enhance hover effects and visual hierarchy - Update PromptVersion type to include template_structure field - Add create_chat_prompt_example.py script demonstrating ChatPrompt usage

…nd property consistency - Add verify_chat_prompt_version() function to verify ChatPrompt properties - Verify template_structure field is set to 'chat' for ChatPrompt instances - Update test_chat_prompt.py to use the new verifier - Initialize internal backend properties to None in both Prompt and ChatPrompt __init__ - Import ChatPrompt in verifiers module

- Remove template_structure parameter requirement from verify_chat_prompt_version - Update all 10 chat prompt tests to use verify_chat_prompt_version - template_structure is now always verified to be 'chat' (no need to pass it) - Replace manual assertions with verifier calls for consistency

…tion logic - Remove temporary inference logic for template_structure field - Simplify comparison logic to directly use template_structure from API response - Backend now properly returns template_structure after regenerating OpenAPI code

…pt.View.Public.class) to PromptVersion fields

…xity - Remove latest_version subqueries from PromptDAO find() and findById() - Remove conditional logic from PromptVersionColumnMapper - Update retrievePromptVersion to fetch latest version directly from prompt_versions table - Update frontend to use versions[0] instead of latest_version.id This simplifies the backend and improves query performance by eliminating unnecessary JSON object construction in SQL queries.

- Add template_structure field to Prompt model in OpenAPI spec - Regenerate Python SDK REST API client code with Fern - Regenerate TypeScript SDK REST API client code with Fern This includes all the type definitions, serialization, and client methods needed to support the template_structure field in both SDKs.

@JSONVIEW

- Add TemplateStructure enum (string/chat) in backend API - Add template_structure field to Prompt model with proper @JSONVIEW annotations - Update PromptVersion to hide template_structure from Public view (internal field) - Add TemplateStructureArgumentFactory for JDBI enum handling - Add Liquibase migration to add template_structure column to prompts table This establishes the core backend support for differentiating between string and chat prompts at the Prompt level (not PromptVersion level).

…on SDK - Add template_structure property to Prompt class (always 'string') - Implement ChatPrompt class with messages array support - Update PromptClient to send template_structure when creating prompts - Add JSON comparison for chat prompts to avoid formatting differences - Add verify_chat_prompt_version() to E2E test verifiers The ChatPrompt class serializes messages to JSON for backend storage and sends template_structure='chat' during creation.

- Add template_structure field to Prompt TypeScript interface - Display template_structure column in prompts list (Chat/String) - Add type column with proper formatting and styling The frontend now displays whether each prompt is a 'Chat' or 'String' type based on the template_structure field from the backend.

…mat() - Remove supported_modalities parameter from ChatPrompt.format() method - Update unit tests to remove supported_modalities usage - Update e2e tests to remove supported_modalities usage - Rename test functions to reflect they test multimodal content, not modalities This simplifies the ChatPrompt API by removing the unused supported_modalities parameter.

- Remove create_chat_prompt_example.py from git (should not be committed) - File remains locally but is no longer tracked by git

github-actions · 2025-11-07T16:46:54Z

📋 PR Linter Failed

❌ Incomplete Details Section. The ## Details section cannot be empty.

❌ Incomplete Issues Section. You must reference at least one GitHub issue (#xxxx), Jira ticket (OPIK-xxxx), (CUST-xxxx) or DEV ticket (DEV-xxxx) under the ## Issues section.

Copilot

Pull Request Overview

This PR introduces comprehensive support for chat-style prompts across the Opik platform. The implementation adds a new "chat" template structure alongside the existing "string" structure, enabling users to create prompts with structured message arrays (system, user, assistant roles) instead of single text templates. The change spans backend schema updates, SDK implementations in both Python and TypeScript, and frontend UI enhancements.

Key changes:

Added template_structure field (enum: "string" | "chat") to prompts and prompt versions, immutable after creation
Implemented ChatPrompt class in Python SDK with JSON serialization of message arrays
Enhanced frontend with chat prompt creation/editing UI, message role visualization, and playground integration
Backend validation ensures template_structure consistency across prompt versions

Reviewed Changes

Copilot reviewed 86 out of 87 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
sdks/python/src/opik/api_objects/prompt/chat_prompt.py	New ChatPrompt class for chat-style prompts with message array support
sdks/python/tests/e2e/test_chat_prompt.py	Comprehensive E2E tests for ChatPrompt functionality
apps/opik-backend/.../TemplateStructure.java	New enum defining template_structure types
apps/opik-backend/.../PromptService.java	Backend logic for template_structure validation and enforcement
apps/opik-frontend/.../AddEditPromptDialog.tsx	UI for creating/editing chat prompts with message builder
apps/opik-frontend/.../ChatPromptView.tsx	Component for displaying chat messages with role-based styling
apps/opik-frontend/.../useLoadPlayground.ts	Playground integration for chat prompt message parsing
sdks/typescript/src/opik/rest_api/.../PromptTemplateStructure.ts	TypeScript SDK types for template_structure

Files not reviewed (1)

apps/opik-frontend/package-lock.json: Language not supported

Copilot · 2025-11-07T16:48:36Z

sdks/python/src/opik/api_objects/prompt/client.py

+            if template_structure == "chat":
+                try:
+                    existing_messages = json.loads(prompt_version.template)
+                    new_messages = json.loads(prompt)
+                    templates_equal = existing_messages == new_messages
+                except (json.JSONDecodeError, TypeError):
+                    templates_equal = prompt_version.template == prompt
+            else:
+                templates_equal = prompt_version.template == prompt


The logic for comparing templates has duplicated code. The fallback comparison prompt_version.template == prompt appears in both the exception handler and the else branch. Consider extracting this into a helper method or restructuring to eliminate the duplication. For example, initialize templates_equal = prompt_version.template == prompt before the if-else block and only override it for the chat case.

Suggested change

if template_structure == "chat":

try:

existing_messages = json.loads(prompt_version.template)

new_messages = json.loads(prompt)

templates_equal = existing_messages == new_messages

except (json.JSONDecodeError, TypeError):

templates_equal = prompt_version.template == prompt

else:

templates_equal = prompt_version.template == prompt

templates_equal = prompt_version.template == prompt

if template_structure == "chat":

try:

existing_messages = json.loads(prompt_version.template)

new_messages = json.loads(prompt)

templates_equal = existing_messages == new_messages

except (json.JSONDecodeError, TypeError):

pass # templates_equal remains as fallback comparison

Copilot · 2025-11-07T16:48:36Z

apps/opik-backend/src/main/java/com/comet/opik/domain/PromptService.java

+        // Validate that template_structure matches the prompt's template_structure
+        Prompt prompt = getById(promptVersion.promptId());
+        String expectedStructure = prompt.templateStructure().getValue();
+        String actualStructure = promptVersion.templateStructure();
+
+        if (actualStructure != null && !expectedStructure.equals(actualStructure)) {
+            throw new IllegalArgumentException(
+                    String.format("Template structure mismatch: prompt has '%s' but version has '%s'. " +
+                            "Template structure is immutable and cannot be changed after prompt creation.",
+                            expectedStructure, actualStructure));
+        }
+
+        // Ensure template_structure is set to match prompt if not provided
+        PromptVersion versionToSave = actualStructure == null
+                ? promptVersion.toBuilder().templateStructure(expectedStructure).build()
+                : promptVersion;


The getById(promptVersion.promptId()) call happens every time savePromptVersion is invoked, even though this validation is checking for an invariant. Consider if this validation could be optimized by checking template_structure only when it's explicitly provided in the version, or caching the prompt lookup if this method is called frequently in batch operations.

Suggested change

// Validate that template_structure matches the prompt's template_structure

Prompt prompt = getById(promptVersion.promptId());

String expectedStructure = prompt.templateStructure().getValue();

String actualStructure = promptVersion.templateStructure();

if (actualStructure != null && !expectedStructure.equals(actualStructure)) {

throw new IllegalArgumentException(

String.format("Template structure mismatch: prompt has '%s' but version has '%s'. " +

"Template structure is immutable and cannot be changed after prompt creation.",

expectedStructure, actualStructure));

}

// Ensure template_structure is set to match prompt if not provided

PromptVersion versionToSave = actualStructure == null

? promptVersion.toBuilder().templateStructure(expectedStructure).build()

: promptVersion;

// Only validate template_structure if it is provided in the version

String actualStructure = promptVersion.templateStructure();

String expectedStructure = null;

if (actualStructure != null) {

Prompt prompt = getById(promptVersion.promptId());

expectedStructure = prompt.templateStructure().getValue();

if (!expectedStructure.equals(actualStructure)) {

throw new IllegalArgumentException(

String.format("Template structure mismatch: prompt has '%s' but version has '%s'. " +

"Template structure is immutable and cannot be changed after prompt creation.",

expectedStructure, actualStructure));

}

}

// Ensure template_structure is set to match prompt if not provided

PromptVersion versionToSave;

if (actualStructure == null) {

Prompt prompt = getById(promptVersion.promptId());

expectedStructure = prompt.templateStructure().getValue();

versionToSave = promptVersion.toBuilder().templateStructure(expectedStructure).build();

} else {

versionToSave = promptVersion;

}

Copilot · 2025-11-07T16:48:37Z

apps/opik-frontend/src/components/pages/PromptsPage/AddEditPromptDialog.tsx

+                    id="template"
+                    className="comet-code min-h-[400px]"
+                    placeholder="Chat messages JSON"
+                    value={JSON.stringify(messages.map(m => ({ role: m.role, content: m.content })), null, 2)}


The message transformation logic messages.map(m => ({ role: m.role, content: m.content })) is duplicated in multiple places (lines 121-124, 227, 269). Consider extracting this into a helper function like serializeMessages(messages) to improve maintainability and reduce duplication.

Copilot · 2025-11-07T16:48:37Z

apps/opik-frontend/src/components/pages/PromptPage/PromptTab/EditPromptVersionDialog.tsx

+                    id="template"
+                    className="comet-code min-h-[400px]"
+                    placeholder="Chat messages JSON"
+                    value={JSON.stringify(messages.map(m => ({ role: m.role, content: m.content })), null, 2)}


The message serialization logic messages.map(m => ({ role: m.role, content: m.content })) is duplicated across multiple components (AddEditPromptDialog, EditPromptVersionDialog, and in the handleClickEditPrompt function). Extract this into a shared utility function to eliminate duplication and ensure consistency.

Copilot · 2025-11-07T16:48:37Z

.../opik-frontend/src/components/pages-shared/llm/LLMPromptMessages/LLMPromptMessageActions.tsx

+            const newMessages: LLMMessage[] = parsed.map((msg, index) => ({
+              id: `msg-${index}-${Date.now()}`,
+              role: msg.role as LLM_MESSAGE_ROLE,
+              content: typeof msg.content === "string" ? msg.content : JSON.stringify(msg.content),
+              promptId: promptData.id,
+              promptVersionId: versionId,
+            }));


The chat prompt parsing and message transformation logic (lines 234-250) is duplicated in useLoadPlayground.ts (lines 86-95). Consider extracting this into a shared utility function parseChatPromptMessages(template: string, promptId?: string, promptVersionId?: string): LLMMessage[] to eliminate code duplication and ensure consistent behavior.

Copilot · 2025-11-07T16:48:38Z

sdks/python/tests/e2e/verifiers.py

+    """
+    Verifies that a ChatPrompt has the expected properties.
+
+    This verifier checks all the same fields as verify_prompt_version but adapted for ChatPrompt:
+    - messages instead of template
+    - template_structure field is always verified to be "chat"
+    """


The docstring claims that template_structure field is always verified to be 'chat', but the actual implementation doesn't include any assertion to verify the template_structure field. Consider adding assert chat_prompt.template_structure == 'chat' to match the documented behavior.

github-actions · 2025-11-07T16:49:16Z

🌿 Preview your docs: https://opik-preview-9192b7f6-d03a-487c-a8fc-c566fcfd5b3f.docs.buildwithfern.com/docs/opik

The following broken links where found:

Page: https://opik-preview-9192b7f6-d03a-487c-a8fc-c566fcfd5b3f.docs.buildwithfern.com/docs/opik/integrations/gretel
❌ Broken link: https://docs.gretel.ai/ (404)

Page: https://opik-preview-9192b7f6-d03a-487c-a8fc-c566fcfd5b3f.docs.buildwithfern.com/docs/opik/integrations/gretel/
❌ Broken link: https://docs.gretel.ai/ (404)

Page: https://opik-preview-9192b7f6-d03a-487c-a8fc-c566fcfd5b3f.docs.buildwithfern.com/docs/opik/integrations/gretel
❌ Broken link: https://docs.gretel.ai/create-synthetic-data/gretel-data-designer/ (404)

Page: https://opik-preview-9192b7f6-d03a-487c-a8fc-c566fcfd5b3f.docs.buildwithfern.com/docs/opik/integrations/gretel/
❌ Broken link: https://docs.gretel.ai/create-synthetic-data/gretel-data-designer/ (404)

github-actions · 2025-11-07T16:49:36Z

SDK E2E Tests Results

0 tests ±0 0 ✅ ±0 0s ⏱️ ±0s
0 suites ±0 0 💤 ±0
0 files ±0 0 ❌ ±0

Results for commit 2d75e34. ± Comparison against base commit f303a9b.

♻️ This comment has been updated with latest results.

github-actions · 2025-11-07T17:16:03Z

Backend Tests Results

283 files ± 0 283 suites ±0 43m 27s ⏱️ - 5m 3s
5 421 tests ± 0 5 408 ✅ - 5 8 💤 ±0 5 ❌ +5
5 296 runs - 89 5 283 ✅ - 94 8 💤 ±0 5 ❌ +5

For more details on these failures, see this check.

Results for commit 2d75e34. ± Comparison against base commit f303a9b.

alexkuzmik added 17 commits November 6, 2025 18:15

Revision 2: Update Fern-generated OpenAPI client code for template_st…

fc339d8

…ructure field

Refactor e2e tests for chat prompt

706ec0c

Revision 3: Fix frontend display of chat prompts - add @JSONVIEW(Prom…

8026905

…pt.View.Public.class) to PromptVersion fields

Revision 8: Remove internal_sandbox file from git tracking

6f9e4d7

- Remove create_chat_prompt_example.py from git (should not be committed) - File remains locally but is no longer tracked by git

Refactor ChatPrompt and tests for it, fix python linter

2533cdd

Add import to prompt playground

2d75e34

Copilot AI review requested due to automatic review settings November 7, 2025 16:46

alexkuzmik requested review from a team as code owners November 7, 2025 16:46

alexkuzmik marked this pull request as draft November 7, 2025 16:46

github-actions bot assigned alexkuzmik Nov 7, 2025

Copilot AI reviewed Nov 7, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[OPIK-2982] [P SDK] [BE] [FE] ChatPrompt support #3987

[OPIK-2982] [P SDK] [BE] [FE] ChatPrompt support #3987

Uh oh!

alexkuzmik commented Nov 7, 2025

Uh oh!

github-actions bot commented Nov 7, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Nov 7, 2025

Uh oh!

Copilot AI Nov 7, 2025

Uh oh!

Copilot AI Nov 7, 2025

Uh oh!

Copilot AI Nov 7, 2025

Uh oh!

Copilot AI Nov 7, 2025

Uh oh!

Copilot AI Nov 7, 2025

Uh oh!

github-actions bot commented Nov 7, 2025

Uh oh!

github-actions bot commented Nov 7, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Nov 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

[OPIK-2982] [P SDK] [BE] [FE] ChatPrompt support #3987

Are you sure you want to change the base?

[OPIK-2982] [P SDK] [BE] [FE] ChatPrompt support #3987

Uh oh!

Conversation

alexkuzmik commented Nov 7, 2025

Details

Change checklist

Issues

Testing

Documentation

Uh oh!

github-actions bot commented Nov 7, 2025

📋 PR Linter Failed

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Nov 7, 2025

Uh oh!

github-actions bot commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

SDK E2E Tests Results

Uh oh!

github-actions bot commented Nov 7, 2025

Backend Tests Results

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions bot commented Nov 7, 2025 •

edited

Loading