Distilling Reliable Reasoning and Instruction Following into Small Local Assistant Models

Full report

On this page

Core difficulty

The central problem is not merely that small student models have fewer parameters. It is that naïve distillation often teaches them to imitate the surface appearance of helpful assistant behavior rather than the latent task structure that makes the behavior robust. Recent work on reasoning distillation shows this very clearly. A 2025 study on long chain-of-thought distillation found that models were far more sensitive to structural perturbations in reasoning traces than to content perturbations: shuffling or deleting steps sharply hurt accuracy, while incorrect local details or removed “reasoning keywords” had much smaller effects. The same paper reports that even when local reasoning steps remain fluent, models can still end up referencing nonexistent prior steps or using backtracking phrases with nothing to revise, which is exactly the kind of “looks right, fails wrong” brittleness that shows up in local assistants.

A complementary line of work identifies a small model learnability gap. Small students do not consistently benefit from long, intricate traces emitted by strong teachers; instead, they often learn better from shorter or mixed-complexity traces that are closer to their own intrinsic distribution. In one 2025 study, 3B students improved more when trained on a mix of long and short CoT data, or a mix of large-teacher and smaller-teacher traces, than when trained directly on only the strongest long-form reasoning. Another 2025 paper explains why: long rationales can cause the gradients on the truly important reasoning tokens to be oversmoothed, so the student learns rhetorical scaffolding rather than the decisive operations.

That same failure mode appears outside math-style reasoning. In counterfactual distillation for small reasoning models, the authors explicitly warn that students may learn spurious correlations, merely memorizing “keywords and patterns” from the training data and then failing on out-of-distribution cases. Their motivating example shows a student over-associating phrases like “egg kept” and “freezer” while missing the actual causal structure of the question. The remedy they propose is to add counterfactual examples and multi-view positive and negative rationales, so the student must discriminate causally relevant features instead of parroting teacher phrasing.

For local assistants, the operational consequences are familiar. Prompt phrasing changes can cause large swings in behavior: multi-prompt evaluation across 6.5 million instances found that semantically equivalent instruction paraphrases often change absolute and relative model performance substantially. Structured-output reliability is also fragile: JSONSchemaBench introduced 10,000 real-world schemas and found major differences in empirical support and compliance across decoding frameworks, with the best framework supporting roughly twice as many schemas as the worst; “LM-only” generation had the lowest compliance rate. Repetition is a separate but related symptom: recent mechanistic work documents a “Repeat Curse,” where models fall into repetitive generation patterns that reduce utility and diversity. Taken together, these results explain why a superficially aligned small assistant may greet politely, yet still leak templates, loop tokens, emit malformed JSON, produce weak summaries outside its narrow prompt band, and generalize poorly to unseen instructions.

The design implication is straightforward: do not distill a tiny local assistant as if the job were to memorize a prompt style. Distill it as if the job were to recognize a task family, satisfy a contract, and remain stable under paraphrase, formatting variation, and mild distribution shift. That framing is also consistent with verifiable instruction-following work such as IFEval, which focuses on explicit instruction contracts rather than subjective “assistant vibes.”

Distillation regimes

The most useful distinction is not “reasoning model” versus “chat model,” but what supervision signal the student receives and what it is allowed to emit at runtime. Distilling step-by-step showed that teacher rationales can make distillation more data-efficient than standard answer-only training, allowing a 770M T5 model to outperform few-shot 540B PaLM on some benchmarks while using less data. The mechanism matters: the student is trained on both label prediction and rationale generation, not just final answers. This is compelling evidence that rationale supervision can transfer useful intermediate task knowledge, especially for explain-code, short-reasoning, and some diagnostics tasks.

At the same time, answer-first or answer-only supervision remains valuable for small local assistants because natural-language rationales can create a rationale-sensitive pipeline. In adaptive-thinking distillation, researchers found that when students generate rationale before answer, minor errors in the rationale can cascade into wrong answers. Their answer-first “post-thinking” formulation reduces that sensitivity, improves efficiency, and still uses rationales as a training signal after the fact. For a local assistant, that points to a hybrid rule: use explicit rationales during training where they help, but heavily prefer answer-first targets for production-facing behaviors that need terseness, deterministic formatting, or low latency.

A recent preprint on latent reasoning pushes the same idea further. Abstract Chain-of-Thought replaces natural-language CoT with short reserved-token sequences, using a warm-up loop that alternates bottlenecked SFT and self-distillation; the paper reports comparable performance with substantially fewer reasoning tokens. In a neighboring safety setting, Latent Policy Guardrails compress deliberation into ten latent tokens while keeping a compact verdict and running much faster than an explicit-thinking baseline. These are early results, but they matter because they suggest a practical path for rationale-free runtime safety: train on richer reasoning internally, but do not expose raw reasoning traces to end users unless a product requirement explicitly needs them.

That said, rationale-free runtime should be treated as a deployment-surface decision, not as a complete IP or safety defense. A 2026 paper on reasoning theft argues that hiding full CoT does not by itself prevent capability stealing, because an attacker with black-box access can still synthesize long-form traces that distill well. Meanwhile, prompt-leakage studies show that system prompts are attackable in multi-turn settings, and repetition-based attacks can extract hidden instructions or conversation context if the model is not hardened. So the right interpretation is narrower: rationale-free deployment reduces casual leakage and user-facing exposure, but it does not replace red teaming, prompt-leakage testing, or careful control of what hidden scaffolding is placed in context.

A practical synthesis for the task families in this report is shown below. The table is a recommended engineering policy, grounded in the literature above rather than copied from any single paper.

Synthetic data, schemas, and negative examples

Synthetic data generation should begin from task specifications, not from evaluation prompts. Self-Instruct demonstrated that a model can bootstrap instruction-following by generating instructions, inputs, and outputs and then filtering invalid or overly similar samples. CodecLM extends that logic by encoding seed instructions into metadata that captures the target instruction distribution, then decoding those metadata into tailored instructions while using self-rubrics and contrastive filtering. The lesson for a local assistant is that one should synthesize data from a latent task inventory—task family, input type, output contract, style constraints, difficulty level, and domain—rather than copy or paraphrase the public eval prompts directly.

Teacher ensembles are most valuable when they are used for role separation, not just majority vote. A “committee” setup with peer review has already been shown to improve data quality in reasoning distillation by filtering flawed rationales and adding mistake feedback. For local assistants, the same pattern maps cleanly to three teacher roles: a task teacher that solves the problem, a format teacher that checks contract compliance such as JSON shape or response length, and a critique teacher that explains why weaker candidates fail. This yields better synthetic supervision than a single monolithic teacher, especially for diagnostics and structured outputs.

Negative examples are not optional. Counterfactual distillation improves OOD robustness by generating minimally changed inputs with different correct answers, forcing the student to attend to causal differences. The same paper’s multi-view design includes both positive-view CoT and negative-view CoT, where the teacher explains why wrong options are wrong. Preference-style training provides a second negative channel: RS-DPO creates synthetic preference pairs by sampling multiple responses per prompt and ranking them with a reward model, while more recent reasoning work shows that simple rejection-sampling baselines such as RAFT can be surprisingly competitive when they keep positively rewarded samples and discard wholly bad ones. For a local assistant, these findings jointly support a dataset that contains both “good target outputs” and “bad near-miss outputs” for each task family.

A workable JSONL schema for training examples is shown below. This is a recommended schema designed to support contamination control, prompt-variant tracking, and separate public versus private supervision. Its structure is motivated by Self-Instruct, CodecLM, IFEval-style contracts, and recent contamination work.

For bad-output generation, each task family should include near failures rather than only catastrophic failures. For example, JSON should include “valid syntax but wrong typed value” as well as “trailing comma”; summarization should include “faithful but too long,” “short but coverage-poor,” and “hallucinated compression”; diagnostics should include “plausible but unsupported root cause” and “prompt-template disclosure.” This is consistent with structured-output benchmarking, which separates schema compliance from downstream correctness, and with prompt-leakage metrics that measure similarity to a known secret template rather than only binary leak / no-leak outcomes.

Training recipes

A robust small local assistant should be trained in stages. The first stage is broad supervised fine-tuning over the seven task families with output contracts attached to every example. This stage should be mostly answer-only for greeting, summarize, rewrite, and JSON generation, because those tasks are dominated by contract compliance, compression, and style control rather than visible deliberation. Explicit rationale targets belong mainly to short reasoning, explain-code, and diagnostics, and even there they should be mixed with concise traces and answer-first targets so the student does not overfit to verbose reasoning theater. That recommendation is directly supported by step-by-step distillation, answer-first adaptive thinking, and the learnability-gap results for small students.

The second stage should be structure-aware reasoning distillation, not blind long-CoT cloning. Use mixed-length traces, and deliberately include examples with reflection, backtracking, and self-correction, because simple final-solution-only traces are not enough to induce robust reasoning. If the student is in the 1B–4B range, weight short and medium traces more heavily than very long traces, then anneal in more difficult traces only after the student reaches stability on simpler ones. This recommendation follows from evidence that logical continuity matters more than local wording, and from the observation that small students benefit more from mixed-complexity data than from direct imitation of the strongest long-form teachers.

The third stage should be preference alignment with offline candidate generation. DPO is attractive because it replaces explicit reward-model optimization loops with a classification-style objective and avoids the operational complexity of standard RLHF. RS-DPO is especially practical for local-assistant post-training because it starts from an SFT model, generates several responses per prompt offline, forms synthetic preference pairs from reward gaps, and then applies DPO. Recent reasoning work also argues that a simple rejection-sampling baseline like RAFT deserves to be treated as a serious baseline, not a toy. In resource-constrained environments, a solid recipe is therefore: SFT first, then offline best-of-k response generation, then either DPO on chosen/rejected pairs or a RAFT-style positive-sample pass when preference labels are sparse.

The fourth stage should be student-aware self-training. STaR already showed that iterative rationale generation can bootstrap reasoning from limited seed rationales and larger unlabeled sets. More recent on-policy self-distillation work reframes the same idea as correcting what the student actually produces rather than what the teacher wishes the student would produce. The most useful operational interpretation for a local assistant is this: periodically sample the current student on held-back internal prompts, identify its highest-confidence correct answers and its most characteristic errors, and fine-tune on those student-specific failures rather than only on generic teacher demonstrations.

One concrete recipe for a 2B–4B local assistant is shown below. The hyperparameters are engineering recommendations rather than values copied from a single source, but the stage ordering follows the literature above.

Evaluation, anti-cheat design, and failure analysis

A trustworthy evaluation set must be designed to measure generalization under paraphrase and contamination control, not merely accuracy on a known prompt template. Multi-prompt evaluation work shows that single-template benchmarks are misleading because semantically equivalent prompts can change rankings. IFEval is useful because its instructions are verifiable; JSONSchemaBench and StructEval are useful because they separate shape compliance from actual structured-task performance; repeat-curse work provides concrete repetition metrics; and prompt-leakage evaluation can be scored by similarity between leaked text and the true template. A local assistant eval should combine all of those perspectives rather than reporting one number.

Holdout prompt generation should be split at the semantic seed level. The training set may include many prompts that ask the model to summarize, but the evaluation set must be generated from different latent seeds, different paraphrase operators, and different authors or generators, so that the student cannot pass by memorizing exact or near-exact wording. The literature suggests two good sources of secret holdouts: dynamic time-sensitive generation from fresh sources, as in LatestEval, and contamination-resistant algorithmic generators whose instances can be refreshed indefinitely while preserving task difficulty. For anti-cheat robustness, both should be used. Dynamic document-based tasks expose web-scale memorization; algorithmic generators expose prompt-template or answer-pattern memorization.

The core anti-cheat rule is simple: never let evaluation prompts become seeds for training synthesis. Training synthesis should start from abstract task specs and public seed instructions; evaluation prompts should be produced later, with separate generators, separate random seeds, and ideally separate paraphrase models or human editors. After that, run deduplication and contamination checks at three levels: exact string overlap, near-duplicate embedding overlap, and behavior-based contamination probes such as CDD or CoDeC. CDD uses output-distribution peakedness from samples; CoDeC uses how in-context examples shift confidence to distinguish memorized from unseen data.

A practical rubric for local-assistant evaluation is shown below. It is deliberately multi-axis so that a model cannot “win” by learning one hardcoded answer style. The axes are recommended, but each is grounded in one or more established benchmark ideas.

Failure analysis should be organized by signature, not by benchmark. A single underlying failure often appears in multiple task families: prompt-template leakage can show up as JSON preambles, summary disclaimers, or diagnostic overexplaining; repetition can appear as greeting politeness loops, reasoning loops, or schema-key duplication. The most valuable postmortem is therefore a two-level analysis: first cluster failures by signature such as repetition, schema_drift, template_echo, wrong_style, first_reasoning_error, and hallucinated_summary; then trace the error back to the training bucket, teacher source, prompt family, and decoding setup. Counterfactual data and reflection-guided self-distillation are especially useful here because they can turn a failed student sample into a new targeted training example rather than just a red mark on a dashboard.

To detect prompt-template leakage specifically, keep a secret canary suite of system templates and run automated attack vectors after every post-training revision. Measure both binary leakage and similarity-based leakage risk. In addition, include repetition-trigger attacks because recent work shows that repetition prompts can exfiltrate hidden templates and context. If leakage appears, do not respond by making the system prompt longer or more ornate; that frequently creates more attack surface. Instead, reduce reliance on in-context scaffolding, move behavior into weights or adapters where appropriate, and test again.

Mapping to TinyRustLM and .slm systems

For a TinyRustLM-style Rust and WASM deployment path, the key architectural move is to treat distillation outputs as typed local assistant assets, not as one giant chat prompt. Rust-native ML tooling such as Candle already targets performance and ease of use, while Rust/WASM inference stacks such as WasmEdge demonstrate end-to-end LLM inference with Rust-built wasm32-wasip1 binaries and GGUF models. TinyWasm, meanwhile, shows what a very small safe Rust runtime looks like: no_std, minimal dependencies, and a sandboxed execution model. Taken together, these tools suggest a clean path for small local assistants that need fast, portable, and auditable inference.

In that setting, a .slm artifact should ideally contain at least five logically separate components. First is the quantized base model. Second is an adapter bundle keyed by task family and, if needed, persona. Third is a contract bundle containing JSON schemas, output-style constraints, and decoding policies. Fourth is an evaluator bundle containing local canaries for repetition, leakage, and schema failure. Fifth is an optional memory bundle that remains external to the base assistant and is retrieved or adapted at runtime rather than baked into every response. This separation is consistent with recent on-device memory work showing that specialized adapters for extraction, update, and memory-augmented generation can make small models competitive with much larger systems, and with persona work that clusters users into parameter-efficient LoRA groups instead of maintaining a full custom model per user.

The runtime path should therefore look like this. A lightweight task classifier detects the task family; the host loads the corresponding adapter or decoder policy; the request is transformed into a canonical task frame rather than a large secret prompt; the model generates under a family-specific output contract; and a local validator either accepts the result or performs one controlled repair pass. This is compatible with WasmEdge’s model-plus-template approach, but the important difference is that the behavioral policy should live mostly in distilled weights, adapters, and typed contracts, not in a brittle hand-written template that can be leaked or mimicked.

A practical TinyRustLM mapping for the seven task families is shown below. This is an engineering design, but it is aligned with the training and evaluation findings in the literature.

The most important policy for a local Rust/WASM assistant is to keep memory, persona, and adapters modular. Use retrieval or memory adapters for personalized state; use persona adapters only when the product explicitly benefits from them; and avoid turning the system prompt into an ever-growing hidden database. That architecture reduces leakage surface, keeps inference predictable, and makes failure analysis easier because each faulty behavior can be traced to a task adapter, a contract, or a memory path instead of to one opaque megatemplate. From the perspective of this report, that is the cleanest way to distill useful assistant behavior into a small local model without falling back on hardcoded prompt responses.