"Fix it" should not be able to rewrite the entire page template

[bfollington]

Fix it cannot be trusted in general right now, but it is also capable of rewriting our preamble in the process of fixing bugs... which we do not want.

Fix it should probably be smarter, a two phase process where it diagnoses, presents a solution and the user can intervene.

[bfollington]

Talking to @Gozala, perhaps we should call it 'diagnose' and the second step is 'fix it'.

It's a bit like "I'm Feeling Lucky" at the moment.

[bfollington]

In iterate we use extractUserCode and injectUserCode to manage this chunk, we need to do something similar for fix it.

[bfollington]

Issue Analysis: "Fix it" feature rewriting page template

After investigating the codebase, I've identified the root cause of this issue and can propose a solution.

Current Behavior

The "Fix it" feature currently sends the entire iframe source code (including preamble/template) to the LLM for modification. This is problematic because:

1. The LLM has full access to modify the entire page template, including critical infrastructure code like the bootstrap script, utility functions, and event handlers
2. When errors occur in user code, the "Fix it" feature can inadvertently modify the underlying template that surrounds the user code
3. There are no mechanisms to protect parts of the code that should remain unchanged

Relevant Implementation Details

The issue exists across several components:

1. The fixItCharm function in /charm/src/commands.ts (lines 32-57) takes a charm and error, then calls fixRecipePrompt with the entire iframe source code
2. The fixRecipePrompt function in /llm/src/prompts/recipe-fix.ts sends the entire code to Claude for fixing
3. The file /charm/src/iframe/static.ts already has the correct functions for handling user code extraction and injection:
   • extractUserCode (line 535) - Extracts just the user code from the HTML template
   • injectUserCode (line 528) - Adds user code back into the template between special comment markers

The iterate function in /charm/src/iterate.ts already properly uses these functions to protect the template, but the "Fix it" feature doesn't leverage this approach.

Solution Approach

Similar to what @bfollington suggested in his comment, we should implement a two-phase approach:

1. Diagnose phase: Show the error and offer analysis of what might be wrong
2. Fix phase: Only modify the user code portion, not the full template

Implementation changes needed:

// In charm/src/commands.ts, modify fixItCharm:
export async function fixItCharm(
  charmManager: CharmManager,
  charm: Cell<Charm>,
  error: Error,
  model = "anthropic:claude-3-7-sonnet-20250219-thinking",
): Promise<Cell<Charm>> {
  const iframeRecipe = getIframeRecipe(charm);
  if (!iframeRecipe.iframe) {
    throw new Error("Fixit only works for iframe charms");
  }

  // Extract just the user code portion instead of using the full source
  const userCode = extractUserCode(iframeRecipe.iframe.src);
  if (!userCode) {
    throw new Error("Could not extract user code from iframe source");
  }

  const fixedUserCode = await fixRecipePrompt(
    iframeRecipe.iframe.spec,
    userCode, // Send only the user code portion
    JSON.stringify(iframeRecipe.iframe.argumentSchema),
    error.message,
    model,
  );

  // Inject the fixed user code back into the template
  const fixedFullCode = injectUserCode(fixedUserCode);

  return generateNewRecipeVersion(
    charmManager,
    charm,
    fixedFullCode,
    iframeRecipe.iframe.spec,
  );
}

Additionally, we could update the UI component in jumble/src/components/CharmRunner.tsx to implement a two-phase approach where:

1. First show a "Diagnose" button that analyzes the error
2. Then offer a "Fix it" button only after the user has seen the diagnosis

Benefits of this Approach

1. Preserves critical template code and infrastructure
2. Improves user experience by making the process more transparent
3. Prevents accidental breaking changes to the template
4. Maintains consistency with the iterate function's approach to code modification

This implementation would resolve the current issue while also making the feature safer and more predictable.