[mwtrew]

Status

• Closes https://github.com/RaspberryPiFoundation/digital-editor-issues/issues/1944

Points for consideration:

• Security:
  • New endpoint is behind ai_friendly_errors feature flag, and requires the user to be logged-in
• Performance
  • Inference response is not streamed at the moment - this may change in future.
  • I have set reasoning: { effort: :none } on the OpenAI client, which makes the response a lot faster but potentially less accurate. This was useful for local testing, but may be less important with a different inference API.
  • The OpenAI client supports automatic retries with exponential backoff - I have disabled this to avoid holding Puma threads open for a long time.

What's changed?

• Adds a POST endpoint /friendly_error_messages that requires message, trace, runtime and code keys in the request body, and returns an AI-generated friendly error message, or a hardcoded sample message if BYPASS_FEM_INFERENCE=true.

What's not changed?

• I didn't set the temperature in the inference request - we could do so easily, but it affects different models differently, so should probably be something we benchmark/evaluate before picking a value.
• There is only one level of hint, and the system prompt isn't the complete prompt from https://github.com/RaspberryPiFoundation/slm-evaluation/blob/main/corpus/prompt.yml in order to keep things concise. Expecting to revisit this in https://github.com/RaspberryPiFoundation/digital-editor-issues/issues/1919.

Steps to perform after deploying to production

Eventually we will want to configure FEM_INFERENCE_API_URL and FEM_INFERENCE_API_KEY in other environments, but that is blocked on https://github.com/RaspberryPiFoundation/digital-editor-issues/issues/1920.

Guide to testing inference locally with Ollama

1. Get Ollama running locally: https://formulae.brew.sh/formula/ollama

2. Set the following in .env:

FEM_INFERENCE_API_URL=http://localhost:11434/v1/
FEM_INFERENCE_API_KEY=''
BYPASS_FEM_INFERENCE=false
BYPASS_OAUTH=true

3. Enable the ai_friendly_errors feature flag

$ rails c
app(dev):001> Flipper.enable(:ai_friendly_errors)

4. Start the API, then hit the endpoint with curl:

curl -i -X POST http://localhost:3009/api/friendly_error_messages \
  -H "Authorization: Bearer abc123" \
  -H "Content-Type: application/json" \
  -d '{
    "message": "NameError: name '\''prnt'\'' is not defined",
    "trace": "Traceback (most recent call last):\n  File \"main.py\", line 1, in <module>\nNameError: name '\''prnt'\'' is not defined",
    "runtime": "skulpt",
    "code": "prnt(\"Hello world\")"
  }'

You should get something like:

{"fem":"It looks like Python doesn't recognize the word `prnt` because it isn't a standard command or function known by that name. It might be worth checking if you accidentally typed \"print\" wrong, since the correct built-in function to display text is spelled with an \"n\" in that spot. Looking at line 1 of your code would likely reveal this small spelling detail."}

[github-actions]

Test coverage

94.18% line coverage reported by SimpleCov.
Run: https://github.com/RaspberryPiFoundation/editor-api/actions/runs/41470718922

[macroscopeapp]

Approvability

Verdict: Needs Human Approval

Macroscope's review found this PR requires human approval — The new endpoint sends learner code and error traces to a configured inference service, with authentication, feature gating and bounded requests. Confirm production remains disabled until the service destination and learner-data handling are acceptable; deployment settings are not established by the example configuration.

Needs human approval because:

• approvability is in preview mode for this repository, so Macroscope does not approve PRs here

^{You can add or adjust custom eligibility rules. Learn more.}

[zetter-rpf]

This is great, I've added a couple of small suggestions