Implementing "jsonMode" for LLM calls, and from the react app iframe calls #965
Conversation
|
This should fix #965 |
charm/src/iframe/static.ts
Outdated
There was a problem hiding this comment.
consider: mode: 'json'? seems more common to me
There was a problem hiding this comment.
Thoughts on adding tests for this? It's got a bunch of specific edge cases so tests would act as documentation of expected behavior etc.
Might be generateable?
There was a problem hiding this comment.
The Deno.test/t.step interface was used in basic-flow to handle some integration/astral slowness/assertions during development, but could probably be ported back to the BDD style -- if we don't have an explicit reason to use the inner t.step interface, consider the BDD wrapper around stepped tests e.g. import { describe, it, beforeEach } from "@std/testing/bdd"
toolshed/deno.json
Outdated
There was a problem hiding this comment.
is there a reason we want to skip caching for json requests?
There was a problem hiding this comment.
ahh, this should be the cacheKey earlier - don't recompute it!
* Now ensures processes get cleaned up on failure * Creates a temporary cache directory, so we can understand what the new cache files are, passes CACHE_DIR into toolshed * Copies cache files to the correct location
This PR introduces
jsonModecapability to the toolshed llm api endpoints.Unfortunately, each language model provider has a different strategy for allowing the return of json data. We primarily use Claude and Groq at the moment, claude only supports prompting (and they provide some instructions in their docs for how to do this better), and in groq you must prompt but can also require that the response is valid json.
So this PR introduces some special casing prompt additions to various models when
jsonMode: trueis set as an api request parameter.I also removed a handful of never-used language models to clean up some dependencies and surfaces area we need to support.