Establish an effective integration test suite within apollo
Integration tests are designed to test the functionality and content of a service, end-to-end, including real model calls. They should not set a high quality bar or depend too much on the model, but should broadly assert the flow of information through the request.
Here are the principles of integration testing:
- Integration tests are run against the running bun server, so they're HTTP calls, rather than direct pyhon calls
- Integration tests require LLM calls so are expensive to run
- Assertions should test the shape of the payload returned by apollo, and by running regex tests against the content. They should be a little loose in terms of content, to allow for the models to return different answers which still satisfy the test criteria
- They are designed to be run once per branch, before merging, and probably on some manual trigger. Maybe they can run on every push once a label has been added.
Establish an effective integration test suite within apollo
Integration tests are designed to test the functionality and content of a service, end-to-end, including real model calls. They should not set a high quality bar or depend too much on the model, but should broadly assert the flow of information through the request.
Here are the principles of integration testing: