Beyond “Be Socratic”: The Hidden Product Decisions Inside an AI Tutor

An early prototype for our mathematics tutor gave the model an instruction that sounded reasonable in design documents:

“Guide me through open-ended questions and hints in Socratic method, never revealing answers.”

Supporting notes instructed the model to assess what the student already understood, encourage them to explain their thinking, and focus on one question at a time. It read like solid teaching advice, but it offered no operational rules for how the software should behave across a multi-turn session.

The limits of that instruction surfaced in our test logs. In an internal session, a colleague repeatedly asked our tutor, Pi, for the answer to a geometry problem. Pi responded with the Pythagorean relation: $36+{b}^{2}=100$. When the tester asked again without writing anything, the tutor subtracted 36, wrote ${b}^{2}=64$, and asked: “What number multiplied by itself equals 64?”

The tester marked the interaction as failed. The mathematics was correct, and the model had technically closed the turn with a question. But isolating the variable was the actual exercise. By performing the setup and the subtraction, the tutor reduced a multi-step geometry problem to elementary arithmetic. Without modifying a record in our database or transitioning the client interface, a routine conversational hint had altered the cognitive task assigned to the user.

I work on full-stack development and AI service architecture at Thinkverse, where students ask for help during mathematics practice and show their work on a digital whiteboard. Our founding engineers established the initial prompts; our curriculum colleagues test the tutor and review conversation transcripts. My responsibility is to translate their teaching requirements into software behavior and build the systems that verify it.

Our goal was to leave the reasoning to the student. On each turn, that meant deciding whether a transformation was still theirs to perform, whether they had already demonstrated understanding, or whether a misconception called for a direct explanation.

Those choices become the exact sentence the student sees next. Looking back at these iterations, I have come to view them as core product logic: the model makes some decisions at runtime, and engineering has to define their boundaries.

How much less help?

One prompt revision I worked on focused on algebraic manipulation. If a student reached $x+18=2x-2$, the tutor was forbidden from simplifying it to $18=x-2$ and asking what to do next. Our system prompt already prohibited supplying reasoning the student had not completed, but that instruction was too abstract to prevent this specific failure. The model routinely carried out the algebraic transformation, handed the simplified expression back to the student, and concluded the turn with an open question.

The difficulty was that the expression $18=x-2$ could not be treated as a forbidden token. If the student had already written $18=x-2$ on the canvas, the tutor needed to reference it freely to validate their progress. Determining whether that mathematical string was acceptable required tracking who had introduced it and identifying the specific skill being practiced. String matching, regex filters, and output guards on the final response could not answer that question.

We had to protect reasoning steps the student had not yet completed and was expected to attempt. The context assembly layer needed to distinguish student contributions from the tutor’s prior outputs. A formula’s presence in the history was not enough; who introduced it mattered.

Multiple-choice questions introduced a similar boundary problem. Because all options were visible on the screen, a tutor could suggest that a student inspect a specific choice without generating any new mathematical formulas. Doing so narrowed the solution space on the student’s behalf. Our prompt instructions had to distinguish between an option being presented as an available choice and an option whose correctness had been confirmed. The tutor had to avoid steering attention toward a specific candidate option while the student was actively deciding, while retaining the ability to validate an option once the student explicitly submitted it.

During code review, our team debated where that rule belonged in our codebase. We decided to split our prompt architecture, moving the restriction into a dedicated template applied only to multiple-choice items. This was a scope decision analogous to placing conditional logic in a specific request handler rather than in a global middleware. A restriction required during open-ended problem solving should not interfere with post-submission answer validation.

Leaving work to the student could not be achieved by piling on negative constraints. If a student had an established misconception, repeatedly asking them what they thought simply prolonged their confusion. Our curriculum rules contained an explicit exception: when the tutor identified a clear misconception, it was permitted to provide a brief conceptual explanation before returning to a guiding question. Giving that concise explanation was an expected part of the teaching strategy.

The digital whiteboard presented another set of conditions. Asking a student to mark a segment or shade an area of a diagram helped when visual relationships were confusing. But if the student had already outlined a sound analytic method in the chat, forcing them to draw on the canvas added friction without improving understanding. Our prompt revisions instructed the tutor to request a concrete visual action when a diagram clarified the problem, and explicitly forbade mandating canvas work when the student was already making progress algebraically.

When curriculum designers asked for hints to be specific without disclosing the next operation, they highlighted this tension directly. When choosing the next algebraic operation is the learning objective, telling the student to subtract 36 is specific, but it performs the task for them. Asking the student to look at the relationship between the two sides of the equation provides support without taking over the step. We had to determine the expected student contribution for that specific turn before deciding how concrete a hint should be. Combining reasonable guidelines in a prompt leaves the reconciliation of those guidelines to the model at runtime.

I understand why teams are drawn to delegating these choices to a language model. Students do not describe their difficulties or approach mathematical problems in uniform ways. Hardcoding every conversational branch into a deterministic state machine would be impractical. A model dynamically composes responses from dialogue history. But relying on that flexibility also delegates the interpretation of the student’s understanding and the choice of appropriate scaffolding.

Our curriculum colleagues help resolve these boundaries against concrete logs. A specialist can explain why a hint undermines an exercise; engineering has to translate that judgment into instructions, runtime state, and validation checks.

I do not decide teaching methodology on behalf of our curriculum team. My job is to understand the requirement we are implementing, including what an earlier failed attempt changes and what counts as evidence of understanding. The same teaching intention leads to different actions depending on the student’s progress:

Scroll horizontally to compare all three columns.

A product decision inside the next sentence

When we tested the revised prompt in an internal preview environment, the results were mixed. The algebra rule held: the tutor stopped simplifying $x+18=2x-2$ on the student’s behalf. But the test case involving repeated demands for the answer failed. The model reverted to the behavior from our opening example, surrendering intermediate equations until only simple arithmetic remained. Out of fourteen targeted checks recorded in that run, eight passed and six failed.

These were targeted preview checks, not a statistical benchmark of production performance. We wanted repeated demands for answers to change the form of assistance toward conceptual descriptions or diagrams. Instead, the model made its mathematical hints progressively more specific until it completed the derivation.

Our curriculum team documented similar inconsistencies across iterative retests. In an earlier evaluation round, a curriculum designer on our team examined an exercise on segment midpoints. In the baseline version, the tutor collapsed the geometric reasoning into a bare calculation of $8.5+8.5$. An initial prompt revision appeared to resolve the issue, maintaining guidance over multiple follow-up turns without volunteering the numbers. Yet in a subsequent test round, the same premature arithmetic calculation reappeared.

Because those internal tests were conducted manually by probing the model in a staging environment, we could not isolate the exact cause of the regression. The prompt snapshot, the preceding conversation turns, the phrasing of the tester’s inputs, and the model temperature all introduced variation. Without versioned execution records that tracked the exact system prompt, inputs, and conversation state, we could not determine whether the regression came from model nondeterminism, subtle prompt adjustments, or variations in tester behavior.

This uncertainty highlighted the difference between two separate questions. Had our team clearly defined what assistance was appropriate in that scenario? And had the model complied with that definition? The first question required product and pedagogical judgment; the second required investigating the technical implementation. Incomplete conversational context in our logs often caused those two problems to look identical. Appending more emphatic prohibitions to the end of a system prompt addressed neither.

Restricting tool calls and database writes cannot enforce this boundary. An intermediate equation or a suggestion to inspect option C can take over part of the student’s task through plain text alone. Mathematical correctness does not tell us whether that assistance belongs at this point in the session.

A test log shows which steps the model generated, not why. That observation is enough to investigate ambiguous instructions, missing context, or unreliable prompting without speculating about the model’s motivations.

Some judgments remain difficult to formalize. Whether a student’s explanation demonstrates sufficient understanding is not always reducible to a boolean condition. We can state the requirement explicitly without guaranteeing that the model will interpret every response accurately.

Because those edge cases persist, evaluating an AI tutor cannot rely on subjective impressions of whether it sounds supportive or instructional. Automated evaluation must test whether specific behaviors occur under defined conditions. If the evaluation criteria are vague, the model evaluating the transcript will simply interpret the teaching intent on its own, and its interpretation may disagree with the model being tested.

The rubric expresses a product choice too

An earlier update made by our engineering team illustrates this dynamic directly. One of our primary evaluation rubrics contained an item called “Closure reinforces learning”. The original wording required the tutor to deliver a conceptual summary of the core mathematical principle after a student solved a problem, explicitly penalizing generic praise. An engineering co-founder on our team revised this rubric item to make it conditional. If a student had struggled through an incorrect attempt, a hint, or an explicit request for help, the tutor was required to reinforce the concept. But if the student answered correctly on their first attempt without assistance, standard praise was acceptable, and the summary requirement was marked not applicable.

Consider an interaction where a student solves a problem on their first attempt with clear, independent reasoning. The tutor acknowledges the correct answer and closes the session. Under the original rubric, an automated judge would flag this exchange as a failure because the conceptual summary was missing. An engineering team optimizing blindly for that evaluation score would modify the prompt to force an unprompted lecture on a student who had already mastered the material.

A higher evaluation score does not necessarily mean a better tutor. The same transcript can pass under an updated rubric without any change to the tutor’s prompt. Engineering must record why the criteria changed rather than attributing the improvement to model capability.

A teaching intention takes three forms: a curriculum discussion with implicit edge cases, a prompt that guides the next response, and a rubric that judges a completed interaction. Each can omit different preconditions. When they disagree, engineers need to know which interpretation they are debugging.

Using a secondary language model as an automated judge introduces another layer of interpretation. The judge model must parse the conversation, determine what the student accomplished, and decide whether a specific rubric item applies before issuing a verdict. Outputting structured JSON makes that verdict easier to store and query, but the schema does not resolve disagreements about what constitutes good teaching.

The evaluation infrastructure I am building is designed to make those judgments inspectable. Our original pipeline passed a full conversation transcript and a broad natural language rubric to a single evaluator model, asking for an overall numerical score and a brief justification. The system I am developing breaks requirements into structured checklist items. Each checklist item has a stable identifier, explicit applicability conditions, and concrete pass/fail criteria. The evaluator returns individual verdicts for each item, accompanied by quoted turn excerpts from the transcript as evidence. This system is currently in active development, and I treat it as an evolving internal tool rather than a proven release gate.

For an over-assistance failure, an engineer needs the exact turn where intermediate reasoning appeared and the preceding student work. A curriculum designer can review that same evidence. Stable item identifiers and quoted turns make disagreements easier to locate, even when they cannot resolve them automatically.

Evaluation runs must also persist the problem definition, prompt version, model identifier, and rubric schema used during execution. If we refine our definition of acceptable assistance, historical evaluation runs executed under the old criteria cannot be compared directly against new runs. Storing the full conversation transcript preserves what actually happened. Re-running a model produces a new observation; an improved reply in a later run should not overwrite the original trace we need to investigate.

Automated pipelines have technical boundaries of their own. Our system uses a student simulator to drive multi-turn conversations through defined scenarios. But a simulator can easily diverge from the intended path, solving a problem without triggering the edge case we set out to test. Even when an automated judge marks every observed checklist item as passing, our scoring logic keeps the overall run marked as inconclusive, returning null values for both the score and the pass status.

A dedicated unit test enforces this requirement in our scoring module:

{
    "conclusive": false,
    "score": null,
    "passed": null,
    "inconclusiveReasons": ["scenario_targets_not_reached"]
}

A run that misses its target scenario provides no evidence about behavior under those conditions. Marking it as passing creates false confidence; marking it as failing records a violation that never occurred. Inconclusive is a distinct outcome, and the scoring code must preserve it.

Reliable evaluation also requires standard backend engineering discipline. In our pipeline, distributed workers pull evaluation tasks from PostgreSQL using database leases, writing versioned checkpoints as multi-turn dialogues progress. Network retries must be strictly idempotent: if a worker commits an evaluation step to the database but loses connection before acknowledging the response, the retry handler must retrieve the committed step rather than re-invoking the model. Re-invoking the model would branch the conversation into a different trajectory, corrupting the transcript history. A dropped network socket or a timeout must be caught as an infrastructure error, never misclassified as an educational failure.

What engineering has to decide

When reviewing changes to our AI services, I now ask which decisions we are delegating to the model. Freedom to phrase an explanation is different from authority to perform a student’s next step. The distinction has to survive prompt templates, context assembly, and evaluation rubrics. Reading request-handling code alone cannot tell us which action the model will choose during a five-turn dialogue; engineering review has to follow the behavior into the conversation, alongside our curriculum colleagues.

Maintaining this infrastructure has real costs. Changes to tutoring guidelines require updates to prompts, scenarios, and rubrics; model upgrades require re-running historical suites. I do not have clean productivity metrics proving that this saves development hours. What I can point to is a broader set of engineering deliverables: application code, versioned scenarios, and evaluation software that makes behavior inspectable.

There is also a limit to what those deliverables establish. A correct step on our whiteboard does not prove conceptual mastery, and a simulated conversation that follows our rubric provides no evidence about how a real child learns. Our engineering work checks whether the tutor provides the assistance our team agreed upon. Measuring genuine learning outcomes requires long-term educational research.

The original Socratic instruction can remain in our prototype archives. Before our tutor writes out ${b}^{2}=64$, we still have to define how much of that work belonged to the student. We examine what the student wrote, identify the step the tutor supplied, and check it against that decision. As long as a model generates the next response, the product decision remains inside the dialogue. We can design for it deliberately or let the model make it for us.