Distilling Useful Behavior into Hybrid or Dense Sub-Billion Students

Full report

On this page

Executive experimental recommendation

We recommend a controlled study comparing small LFM2-based hybrids against strong dense baselines (e.g. Qwen, SmolLM, Granite) under identical curricula. Use an ensemble of high-quality teachers (for example LFM2.5-1.2B or Qwen3.5-4B) to generate synthetic training data, then fine-tune or distill into sub-1B students. The experiment should test both hybrid (LFM2.5-230M/350M) and traditional transformer architectures (SmolLM-360M, Qwen-0.8B, Granite-350M) on an identical task suite. Model and data comparisons (e.g. raw vs base vs adapted vs quantized) should be benchmarked on held-out prompts and seeds. We will use PEFT (especially LoRA) for efficiency, and ensure rigorous data hygiene and regression testing. Early analysis of held-out performance and resource use will guide decisions: if neither student type reliably learns the behaviors without contamination or instability, abort or iterate on curriculum.

Student and teacher candidate matrix

LFM2.5-230M (liquid/hybrid conv–attn) – 230M parameters with 14 layers (8 gated convolutions + 6 grouped-attention) and 32,768-token context. Uses a 65,536 token vocab; token embeddings are tied to output logits. License is LiquidAI’s LFM 1.0. Runs efficiently on edge hardware (≈0.5 GB bf16 weights). Trained modules include all layers (conv, attn, FFN). Embeddings and output are tied. Activation footprint and memory are very low (≈1 GB VRAM for 230M in fp16). Deployment: optimized for CPU/GPU, supports llama.cpp (GGUF), ONNX, Apple MLX. Tool calling (LLM-to-API) built-in. Not suited for heavy reasoning. LFM2.5-350M (liquid/hybrid) – 350M parameters, same architecture and 32K context. Slightly larger footprint (~0.7 GB weights). License LFM1.0. Trainable modules identical to 230M. Performs similarly to 230M but with modestly higher throughput requirements. LFM2-700M (liquid/hybrid) – 742M parameters in a 16-layer (10 conv + 6 attn) hybrid SSM architecture. Context 32K, vocab 65K. License LFM1.0. Activation memory roughly double the 350M model. Uses bfloat16 precision. Not tied to any memory-heavy MoE; runs on moderate GPUs. Otherwise similar properties to 230/350M. (Also available: a 1.2B version.) Qwen3.5-0.8B (dense hybrid transformer) – 0.8B parameters, 24 layers. Combines “Gated DeltaNet” linear-attention layers and gated full-attention blocks. Hidden dim 1024; context length up to 262,144 tokens (native long-context). Token vocab ~248K (tied embeddings). License Apache-2.0. Inference optimized for high throughput. Activation memory ~2–3× that of LFM2.5-350M due to larger size and long context; requires midrange GPU. Implements vision-text fusion (multimodal). SmolLM-360M (dense transformer) – 360M parameters, standard transformer architecture. Likely ~12–16 layers (not publicly listed, but training config in LLaMA/Zephyr style). Weighing ~0.4B params in bf16. License Apache-2.0. Tokenizer likely HuggingFace “llama” style (vocab ~32K). Tied embeddings. Likely context length ~2048–4096. Activation memory similar to LFM2.5-350M. No native vision. Granite-4.0 350M (dense) – 350M parameters, standard transformer (Granite-350M) or hybrid-SSM (Granite-H 350M) as two variants. Each ~350M. License Apache-2.0. Context presumably standard (~2048–4096). Activation memory similar to SmolLM. Released for edge/agentic use, with extensive safety training (ISO42001). Tool/calling support implied. Granite’s hybrid (“H350M”) uses state-space layers; the non-H version is purely attention-based.

In sum, all students use tied embeddings (ensuring token-level loss compatibility). LFM models use Liquid’s custom blocks (gated conv + attn). Dense students (Qwen, Smol, Granite) use standard attention/mix-of-experts. Tokenizers: LFM and SmolLM likely use 65K or 32K fixed vocab (LFM’s is 65K), Qwen uses ~248K, Granite unspecified (likely 50K). All target on-device/browser execution. Memory scales from ~0.5 GB (LFM230M) up to several GB (Qwen0.8B) for inference.

LFM training-stack capability audit

LiquidAI’s LFM2 family is supported by recent tooling. The official TRL (Transformer Reinforcement Learning) stack provides LFM-specific trainers (e.g. Lfm2ForCausalLM in vLLM/HF) with full autograd; it supports BF16 training out of the box. Mixed-precision (bfloat16) is recommended. Hugging Face Transformers (v5+) now recognizes LFM2 models, and vLLM 0.23+ adds native LFM2 architecture support. Training features such as gradient checkpointing, FlashAttention, and custom kernels are partially available: for example, LFM2’s multi-query (grouped) attention layers are GPU-optimized, but the novel gated convolutions do not yet have specialized fused kernels. LoRA/PEFT is explicitly supported in TRL’s docs for LFM2. The LoRA target modules can be specified by name (q_proj, k_proj, v_proj, o_proj) as shown in Liquid docs. After fine-tuning, PEFT adapters (LoRA weights) can be merged into the base model for efficient inference. Quantization-aware training (QAT) is not documented for LFM2, but export to GGUF/ONNX is provided. All LFM2 model components (embeddings, convs, FFNs, attns) compute gradients normally. Distributed training is possible via Accelerate/Trainer (the TRL example uses device_map="auto"). Official docs note full fine-tuning requires modest GPU memory (e.g. ~1 GB for LFM2.5-230M). Exports are robust: the Hugging Face model cards include GGUF (llama.cpp), ONNX, MLX formats. In summary, LFM2 is fully trainable under PyTorch/TRL with LoRA support, mixed precision, and common training optimizers; only specialized runtime optimizations (FlashAttention for convs, QAT) may be limited until more custom kernels are upstreamed.

Method comparison by compatibility and compute

• Supervised Fine-Tuning (SFT): Standard cross-entropy training on (prompt, assistant_output) pairs. Two variants: assistant-only (loss computed only on the assistant’s tokens, ignoring system/user tokens) vs full-sequence (loss on all tokens). Both work on any causal model, including hybrid conv/attn. Full-sequence SFT may slightly improve instruction-following at cost of training time. Overhead: comparable to normal training (linear in tokens). No architecture matching needed beyond shared tokenizer and output dimension.
• LoRA (and adapters/DoRA): Low-rank adaptation modifies selected weight matrices (typically attention and maybe FFN) with rank‑r matrices. Compatible with LFM2 and dense models as long as module names match (TRL’s config targets "q_proj","k_proj","v_proj","o_proj" for LFM2). Low compute (adds only ~1–2% parameters) and small memory; ideal for sub-1B training. Downside: requires specialized tooling (PEFT libraries). DoRA (if used) is similar concept. Upper-layer/attention-only adapters likewise freeze most weights – these save compute but may cap performance if lower layers need adaptation.
• Full Fine-Tuning: Updates all weights. Maximum flexibility, but high GPU memory and risk of catastrophic forgetting. Compatible with all architectures. Use only if hardware allows (rare for >300M on single GPU).
• DPO/ORPO (Preference Optimization): Reinforcement-finetuning methods that directly optimize preference-based objectives (no separate reward model). Works by treating chosen vs rejected answer pairs. Requires identical tokenizer and output dims between models. Training cost is roughly 2× SFT per example (we need to score both chosen and rejected outputs). LoRA can be combined with DPO (as recommended by TRL). ORPO (Odds Ratio) is an alternate objective but similarly requires same dims and slow sampling/training. Suitable if preference data is available; architecture-agnostic.
• Sequence-level distillation (SeqDistill): Generate answer trajectories (e.g. via greedy teacher output) and treat them as ground-truth for the student. Does not require matching internal layers, only same tokenizer and output space. Relatively cheap (one forward pass from teacher). If student lacks capacity to produce long detailed sequences, can still learn key responses. Provides coarse supervision and large speedups over token-KL.
• Token-level (logit) distillation: Train student to match teacher’s output logits via KL loss each token. Requires that student and teacher share vocabulary and similar format. Computationally expensive (teacher must forward, produce logits). Sensitive to temperature and architecture mismatch. More granular than seq-distill but can struggle if student hidden size differs.
• Top-k logit distillation: Only match top-k teacher logits to reduce cost. Still needs shared vocab; can be done whenever token distill is done.
• Hidden-state matching (feature distillation): Align student’s internal activations with teacher’s. Usually requires same hidden dimensions or learned projection. As noted by Flex-KD, conventional feature distillation requires identical hidden sizes. LFM2 student vs transformer teacher will have mismatched dims, so direct matching is infeasible without extra network. Not recommended here unless using learned projectors, which adds complexity.
• Self-distillation: Iteratively train a model on its own outputs (data no new content). Generally not needed if we have strong teacher. Has no extra computation beyond self-generation. Assumes original model is already decent.

In terms of compute: LoRA/SFT (supervised) is cheapest, full-tuning ~2× more, distillation ~2× (teacher pass), and RL methods (DPO/ORPO) higher due to sampling and possibly actor-critic loops. All methods require a matching tokenizer and output vocab; only hidden-state methods additionally require layer size match or projection. None of these methods inherently demand the same architecture, except hidden-state ones. Seq- and token-distillation as well as preference methods only demand causal/embedding compatibility.

Teacher generation, review, and provenance protocol

Use an ensemble of vetted teacher models (e.g. LFM2.5-1.2B-Instruct, Qwen3.5-4B) to generate candidate answers. For each prompt, run deterministic decoding (e.g. low-temperature or beam search) to produce a final answer string; avoid simple “self-rewriting”. Generate diverse paraphrases by varying seed/temperature for a subset of data, but prefer stable outputs for label consistency. After generation, apply filters: (1) Content screening – remove any PII or unsafe content via regex and classifiers; redact or drop if found. (2) Factual consistency check – cross-validate facts via a factual model or knowledge base (if available) to catch hallucinations. (3) Rubric review – automatically categorize outputs (e.g. using high-accuracy classifiers) to ensure they fit intended behavior (e.g. helpful answer vs refusal). (4) Adjudication – if multiple teachers disagree strongly on answer style or facts, either choose the more reliable model (via prior evaluation) or skip that example. Track license compatibility: do not include large verbatim text from copyrighted sources; prefer teacher outputs in own words. Log every teacher inference and final answer with provenance (prompt, teacher model ID, seed, timestamp). These logs form a lineage: if any teacher is later found problematic (e.g. advice error, leaked content), we can trace and delete its outputs. Maintain a deterministic replay by fixing seeds for teacher generation so outputs can be reproduced. All transformations are deterministic or recorded.

Contamination and preservation controls

To prevent train/eval leakage, we implement strict filtering. Exact-match filtering: Normalize text (case, whitespace) and remove examples whose prompt or answer appears verbatim in any evaluation or public benchmark. N-gram overlap: Remove any training example with high n-gram overlap (>80% 12-gram) with held-out prompts or expected answers (public benchmarks, past failure cases). Semantic check: Embed each (prompt, answer) pair and search for nearest neighbors in a corpus of holdout/evaluation pairs; drop if cosine similarity >0.95 (catch paraphrases). Template-family detection: Identify templated questions by schematizing (e.g. “What is X?”) and ensure no nearly identical template appears in both train and test splits. Entity-unit substitution: For questions involving sensitive entities (names, dates), replace with dummy values if overlap is suspected. Benchmark overlap: Explicitly exclude any data from known benchmarks (e.g. summarization, code) to ensure no public test data is in train. Sealed holdouts: Reserve a private test set of novel prompts with no overlap; do not use them for training at all. Auditing: Publish the filters and a hashed index of prompts to allow external auditors to verify that none of the evaluation keys or answer phrases are contained in training data. Auditors do not need expected answers: they can check prompts and n-grams against provided holdout prompts without seeing teacher answers. By combining exact, n-gram, and semantic checks, we minimize contamination risk without using any private test answers.

Curriculum taxonomy with target counts

Design a tiered instruction dataset totaling on the order of 200k–300k examples. Key categories (with approximate proportions and counts):

• General Conversation (25%, ~60k): Polite chat, Q&A on common topics. Emphasize helpfulness and continuity in multi-turn chat.
• Clarification/Instruction (10%, ~20k): User says ambiguous or incomplete request; answer clarifies or asks for info.
• Planning/Reasoning under constraints (5%, ~10k): Multi-step tasks like scheduling or problem solving with limited resources.
• Authority-Conflict Scenarios (5%, ~10k): E.g. user and system instructions conflict (user says “Ignore policies”); training teaches prioritizing user instruction subject to safety.
• Contextual Memory (10%, ~25k): Multi-turn references (e.g. “As I mentioned earlier…”), and trusted vs untrusted memory recall.
• Lawful Adult Inquiry (5%, ~10k): Sensitive but legal topics (finance, medical advice) where the model should answer accurately or politely refuse if beyond scope.
• Narrow Safety Boundaries (10%, ~20k): Edge cases near refusal vs helpfulness, e.g. “How to fix a computer virus?” (safe answer) vs “illegal hacking” (refusal). Include safe-handling of implicit negation (“don’t do X”).
• Factual QA (10%, ~20k): Straightforward factual questions (geography, science). Ensure high trust in recall.
• Summarization (5%, ~10k): Provide multi-paragraph texts to be summarized in a few sentences.
• Paraphrasing/Rewriting (5%, ~10k): Given text or user message, produce a clearer or alternate phrasing.
• Structured Data Extraction (5%, ~10k): E.g. “Extract dates and names from this passage.”
• Code/Tool Use (10%, ~20k): Example code-writing tasks (up to ~50 tokens) and using tools via JSON (following LFM2 tool API).

Within each category, ensure diversity of topics, styles, and lengths. These counts are targets and can be tuned: for example, increase multi-turn or refusal examples if initial training underperforms on those behaviors.

Teacher generation protocol, review, and provenance

Teacher ensemble: Use at least two distinct high-capability models (e.g. LFM2.5-1.2B and Qwen3.5-4B) to generate answers. For each prompt, generate from both and filter or merge outputs: if they agree, keep one; if they differ, pick the one with higher self-estimated score or manually adjudicate a few conflict cases. Deterministic sampling: Use greedy or top-p sampling with fixed seed to produce each answer, ensuring repeatability. Optionally, a second answer at higher temperature can provide a “second-opinion” for quality check. Final-answer extraction: Remove any generation artifacts like role labels or tool-call tokens to yield the plain assistant answer. If using function calls, ensure correct placement of final answer after tool output. Quality checks: Apply automated rubrics to detect generic refusals (“I cannot...”) and bias. Run a high-accuracy toxicity/safety filter to flag any improper language. Use a factual-check model or external data source to flag likely hallucinations (e.g. a mismatch between answer claims and Wikipedia). Discard or correct examples where the teacher output violates guidelines or factuality. Adjudication and provenance: Log all (prompt, chosen answer, teacher ID, seed) triples. If teachers produce PII or copyrighted snippets, remove those from the dataset. Maintain a lineage map: any given training example references exactly which teacher(s) provided it. This allows us to trace back (for deletion or inspection) if a teacher is later found to produce disallowed content. All teacher generations and post-processing steps are scripted with fixed seeds, enabling deterministic replay.

Contamination and preservation controls

We will intertwine contamination prevention (not training on “test” behaviors) with preserving general capabilities. Contamination checks: See previous section on filtering. Additionally, use round-trip evaluation: every example is checked against holdout questions and answers to ensure <n% token overlap. Maintain an external auditor who only sees filtered datasets (no teacher answer keys). Preservation curriculum: Maintain a mix of broad-language tasks throughout training. For example, interleave a small proportion (~10-20%) of general conversation and summarization examples from public datasets or earlier versions of the model. This replay data keeps the model proficient in ordinary dialogue and multilingual ability. KL regularization: When fine-tuning, we can add a KL-divergence term between the student’s output distribution and the original base model’s output on “generic” prompts, to prevent drift from baseline behavior. This helps avoid catastrophic forgetting of the pretrained knowledge. Early stopping & validation: Monitor not only task-specific metrics but also a held-out set of general dialogues and coding examples. If fine-tuned loss drops but base behavior degrades (e.g. random mistakes in summation), stop early. Tracking training loss alone is insufficient – explicitly evaluate on auxiliary tasks to set stopping criteria. Regression budget: Define acceptable performance tolerance on core tasks (e.g. <2% drop in general LLM benchmarks). If exceeded, pause specialized training.

CPU pilot and GPU scale-up formulas

Memory formula: For a model with $P$ parameters (in 16-bit) and hidden size $H$, context length $L$, and batch size $B$, VRAM $\approx$ $(P \times 14~\text{bytes}) + (B \times L \times H \times 4~\text{bytes}) + \text{overhead}$. Here 14 bytes accounts for 2 (bf16 weights) + 4 (grad FP32) + 8 (Adam states) per parameter. E.g. for a 500M-param model, base+grad+opt ≈ 500M×14 ≈ 7GB. Activations ≈ $B\times L\times H\times4$. Use gradient checkpointing (halving activation storage at cost of recompute) to reduce memory. Pack sequences to fill each batch. On CPU (pilot), expect ~half the throughput of GPU. Batch/sequence packing: Use as high $B\times L$ tokens as fit GPU memory with overlap. If full batch doesn’t fit, use gradient accumulation over $N$ steps to simulate larger $B$. Deterministic seeds: Fix random seeds in PyTorch and NumPy (e.g. torch.manual_seed) for reproducibility. Checkpointing: Save model state every $N$ updates (e.g. per epoch). On failure, resume from last checkpoint. Validation cadence: After each epoch or every X steps, evaluate on a held-out dev set covering all behavior categories. Compute metrics (answer accuracy, refusal rate, quality). Stop when no improvement or failure criteria met. Compute/time formulas: Time per batch ≈ $(2 \times \text{FLOPs per sequence}) / \text{GPU FLOPS} + \text{overhead}$. Number of FLOPs ≈ $P \times L$ (roughly linear) per pass. Estimate cost: \#updates = (dataset tokens)/(tokens per batch). Total training time ≈ \#updates × time per update. Use cloud/GPU cost rates to estimate $$\text{Cost} = \text{time (h)} \times \text{GPU hourly rate}$$. For pilot, estimate on a midrange CPU: tokens/s ≈ fraction of GPU speed (e.g. 1/10).

Promotion suite and regression budgets

Evaluate all artifacts on the same benchmarks and scenarios to decide promotion. Test items:

• Functional correctness: On standard held-out prompts (paraphrased from training categories), compare outputs verbatim. Each model (raw source, base student, adapter-fused, quantized, browser-runtime) must produce identical answers (within stochasticity) given the same seed and prompt. Re-run each with fixed seed to verify consistency.
• Random-seed reproducibility: Execute each model 3 times on 50 sample prompts; check that outputs are consistent or divergences are within an acceptable margin.
• Latency and memory: Measure half-precision decode tokens/sec and peak memory use for each model on a representative device (e.g. mobile CPU and GPU). Ensure student model latency meets the sub-1B target.
• Long-context behavior: Provide very long inputs (e.g. 20K tokens) to see if students maintain coherence and carry relevant context. Compare whether decoded answers reference early context correctly.
• Regression budgets: Define, for example, an allowed 5% drop in answer fidelity on a broad NLU benchmark (e.g. MMLU), and <1% rise in unsafe replies compared to base. Explicitly track these metrics.
• Quantized artifact: Test the quantized student (e.g. gguf) on a subset to ensure accuracy remains within tolerance (e.g. <1% degrade).
• Browser runtime: If exporting to WebAssembly or on-device code (e.g. Granite’s WebGPU demo), test that end-to-end response matches the native model (within rounding) on sample prompts.

Each comparison is done on a suite of unseen prompts (no overlap) with multiple paraphrases and seeds. Models failing stability, speed, or regression budgets are rejected from promotion.

Factorial experiment and ablation matrix

To isolate factors, use a minimal factorial design. Factors: (A) Architecture (LFM2 hybrid vs Dense transformer), (B) Pre-training/post-training recipe (Tiny-RustLM-style RLHF vs raw SFT), (C) Student size (small ≈230–360M vs mid ≈700–800M), (D) Teacher style (single in-domain vs ensemble vs none), (E) Curriculum breadth (narrow vs broad categories). For feasibility, start with a 2×2×2 grid: e.g. architectures A vs B, recipes yes vs no, small vs mid. Within each cell, keep teacher/curriculum fixed initially. For example: LFM-230M with RLHF-based Tiny-RustLM recipe vs baseline SFT only; compare similarly sized SmolLM-360M. This 8-experiment pilot will show main effects. If a particular factor (e.g. architecture) clearly dominates performance, we can decompose further: e.g. fix architecture and vary teacher quality or curriculum complexity. Always include a control (no special training). This avoids a single-checkpoint fallacy by requiring consistent gains across sizes and runs.

Stop rules, rollback, and deletion lineage

Monitor for early termination triggers:

Each data record carries an ID and teacher-of-origin. If a fault is later found in a teacher model, the lineage lets us identify and delete all influenced examples from the training set. Checkpoints are versioned by seed; we require reproducible rollbacks.

• Weak teacher signal: If teacher models give very low-perplexity/unsophisticated answers or exhibit undesirable patterns, reconsider using them.
• Contamination discovered: If any holdout prompt is found in training data (exact or paraphrase), immediately halt training and purge.
• Unsupported gradients: If during fine-tuning any module produces NaNs or broken gradients (e.g. conv not implemented in fp16 on device), abort and fix implementation.
• Loss instability: If training loss oscillates wildly (no convergence) across epochs, stop to prevent wasted computation.
• Quantized collapse: If the quantized student (gguf/8-bit) fails catastrophically (e.g. outputs near constant), treat as failure.
• No held-out gain: If after X epochs the student shows no improvement on held-out metrics (or gets worse), stop and re-evaluate.
• Regressions in base skills: If general capabilities (multilingual, logic, factuality) drop beyond budget, stop.
• License or policy violation: If any downstream use finds the model unsalvageably violating rules (e.g. leaking personal data), delete the checkpoint.

Unknowns requiring local training or evaluation

Certain questions cannot be answered from documentation alone: e.g. empirical performance of LFM2.5 vs SmolLM after identical training, or unseen failure modes, must be determined by running these experiments locally. We highlight these unknowns: exact relative gains from LFM hybrid vs transformer under the new TinyRustLM recipe, LoRA’s effect on hybrid layers, and any silent brittleness. These require actual fine-tuning and tests beyond literature.

Annotated primary-source bibliography

1. LiquidAI/LFM2.5-230M model card (Hugging Face) – Model details for LFM2.5-230M base and instruct versions, including architecture (14 layers, 8 convolutional + 6 attention) and 32K context. Lists weight formats (GGUF, ONNX, MLX) for deployment.
2. LiquidAI/LFM2.5-350M model card (Hugging Face) – Overview for 350M version. Notes hardware requirement (~1 GB GPU) and vLLM 0.23.0 support.
3. LiquidAI/LFM2-700M model card (Hugging Face) – Describes LFM2 family (350M, 700M, 1.2B) with hybrid architecture, 32K context. Provides license and precision (bf16).
4. IBM Granite 4.0 Nano: Just how small can you go? (Hugging Face Blog, Oct 2025) – Announces Granite 350M and 1B (dense vs hybrid “H” variants) models. Notes ~350M parameter scale and hybrid-SSM architecture for Granite 4.0 H 350M. Highlights Apache-2.0 license.
5. Qwen3.5-0.8B model card (Hugging Face) – Details of Qwen3.5-0.8B: 0.8B parameters, 24-layer hybrid transformer with “Gated DeltaNet” and sparse MoE. Context length 262,144, hidden dim 1024. License Apache-2.0.
6. HuggingFaceTB/SmolLM-360M-Instruct model card (Hugging Face) – 360M-parameter LLaMA-derived model. License Apache-2.0, approx. 0.4B params, English-only. (Exact layer count not listed.)
7. LiquidAI TRL Fine-tuning docs – Official guide for LFM2. Key sections on LoRA fine-tuning recommendation and SFTTrainer usage. Shows how to target LFM modules (q_proj etc.) for LoRA.
8. **Cao et al., “Delta Knowledge Distillation for Large Language Models” (arXiv 2025)** – Discusses token-level vs sequence-level KD, and how traditional KD assumes identical model spaces. Defines token-KL and sequence-KD losses.
9. **Peng et al., “Flexible Feature Distillation for LLMs” (arXiv 2025)** – Shows that conventional hidden-state distillation requires teacher/student to have same hidden size, motivating new methods. Confirms hidden-matching is non-trivial when sizes differ.

Each source is cited in the above report with specific line references (e.g. ). The findings from these docs guided the architectural and training recommendations.