AI Pulse
Helix Pulse — a closed-loop feedback system that turns user reactions to Helix chat responses into tracked, classified, actionable signal for improving model output quality.
What
A dashboard application, separate from Helix, that collects thumbs up/down feedback with a required written explanation, auto-classifies it via LLM using the full conversation context (prompt, response, tools called, active workflow), correlates it with the system prompt version that produced it, and layers in an LLM-as-a-Judge rating per trace. Reviewers can browse traces, comment on them, and flag ones needing change.
Why
Right now there's no way to answer "is Helix getting better?" Feedback arrives unstructured and disconnected from the prompt state that caused it, so prompt changes are made on intuition and nobody knows whether they helped or quietly broke something else. Pulse makes the loop closeable: a trend in the dashboard traces back to a specific conversation, and a prompt change traces forward to a measurable shift in that category.
Success Looks Like
Negative feedback rate trending down in categories where prompt changes shipped, with a low regression rate — changes that fix one thing without breaking another. Supporting signals: enough feedback volume per session to be statistically useful, high auto-classification confidence without human review, and a short cycle time from "pattern spotted" to "prompt change deployed."
Who's It For
Internal only — the Helix team doing prompt engineering and quality review. The feedback collection surface lives in Helix and touches all users, including customers, but the dashboard audience is the team. Customer and internal feedback are tracked separately, since internal users test edge cases and would skew the trend data.