Repository navigation
Added agents image generation docs - #559
dannyjameswilliams wants to merge 12 commits into
Conversation
![orca-security-eu[bot]](https://avatars.githubusercontent.com/in/276244?s=60&v=4){kind=small}
There was a problem hiding this comment.
Orca Security Scan Summary
| Status | Check | Issues by priority | |
|---|---|---|---|
 Passed |
Infrastructure as Code |  0  0  0  0 |
View in Orca |
| Passed |
SAST | 0 0 0 0 |
View in Orca |
| Passed |
Secrets | 0 0 0 0 |
View in Orca |
| Passed |
Vulnerabilities | 0 0 0 0 |
View in Orca |
| image_field: list[Annotated[QAImage, Field(description="<image guidance/style description here>")]] = Field( | ||
| max_length = 4 # max_length must be specified for lists of images | ||
| ) | ||
| # END AnnotateListImageExample |
There was a problem hiding this comment.
Ah, I wasn't aware of this? Does this let you give extra "system_prompt" style influence over the image generation in description ?
There was a problem hiding this comment.
Yeah precisely, so the model will generate an image prompt, but the description field attached to any image fields will always be passed down to the image model
|
|
||
| ## Image Generation | ||
|
|
||
| Ask mode allows you to also request images to be generated, which will be based on any retrieved data from the search. [See the page on image generation for more details.](../reference/image_generation.md) |
There was a problem hiding this comment.
Maybe remove "from the search". For example, I think it could calculate the average price and then display that right?
There was a problem hiding this comment.
Good point, I will adjust :)
| </TabItem> | ||
| </Tabs> | ||
|
|
||
| and display it with [PIL](https://pypi.org/project/pillow/) in Python, or save it as a PNG file in JavaScript/TypeScript: |
There was a problem hiding this comment.
Maybe an idea to also have, save it back in Weaviate? And a code block on how to do that?
There was a problem hiding this comment.
Implemented! I added two tabs, one to save/display the image and another to save/search the image in weaviate. Both are dropdowns to keep it less cluttered, let me know what you think
CShorten
left a comment
CShorten
left a comment
There was a problem hiding this comment.
Super cool, awesome work with this Danny! 🔥
Love the examples! model <> t-shirt, slide deck, series of adverts, temperature charts, ... haha, plenty to choose from! 🚀
| * `"landscape"` (default): 1536×1024 | ||
| * `"portrait"`: 1024×1536 | ||
|
|
||
| ## Non-client usage |
There was a problem hiding this comment.
We don't really support non-client usage and don't have any REST APIs documented.
We' re using clients to exactly encapsulate such implementation details, so maybe we're better off to remove this section not to confuse users?
There was a problem hiding this comment.
The existing output_format does allow plain dict[str, Any] in addition to pydantic BaseModels, so I agree we should probably document how to use image generation with raw JSON schemas. But that's not non-client usage as such (you're still using our client, just we don't constrain where you get your JSON schemas from) so maybe we want a different heading and to have a fuller example that calls our client with a dict schema?
There was a problem hiding this comment.
Updated this section now, so it's still using the client but raw json schemas instead of pydantic/zod
|
|
||
| **Images are generated independently**, meaning that if you want a consistent theme amongst your requested images, you should add a consistent description to your image field. Try specifying specific layout instructions, hex color codes and stylistic choices. | ||
|
|
||
| **Timeouts**: image generation adds latency. In Python, when your output format contains images, the client's default timeout rises to 60 seconds. If you set your own `timeout`, make sure it's long enough. |
There was a problem hiding this comment.
still wording says rises even 60s is the default I think
4 participants
What's being changed:
Support for new query agent feature - image generation via structured outputs.
Type of change:
How has this been tested?
yarn start