Engineering Tiny Language Models: Architectural Minimization, Tokenization, and Browser-Native Inference

Full report

On this page

The trajectory of natural language processing has historically been defined by the pursuit of immense scale, often resulting in foundation models containing hundreds of billions of parameters. However, recent empirical research has demonstrated that coherent, grammatically sound, and contextually consistent language generation does not strictly require such massive scale. Advanced language capabilities can be reliably elicited from highly compact architectures—spanning 1 million to 20 million parameters—provided the training corpus is meticulously constrained and structurally simplified1. This report provides an exhaustive engineering and architectural analysis of designing, training, and deploying "TinyLMs" within the 1M–20M parameter regime. The analysis encompasses the minimization of the standard Llama architecture, precise tensor specification for safetensors compatibility, the intricate tradeoffs of tokenization algorithms, specialized training data regimes, and rigorous memory budgeting for browser-native execution via WebGPU.

Minimal Llama-Style Architecture for Sub-20M Models

The Llama architecture, originally formulated by Meta for models scaling from 7 billion to 405 billion parameters, provides a highly optimized decoder-only transformer framework4. Downscaling this architecture to the 1M–20M parameter regime requires a careful mathematical rebalancing of the embedding dimensions, layer depth, and attention mechanisms. At this micro-scale, the primary objective is to preserve the inductive biases that render the Llama architecture efficient while strictly avoiding parameter monopolization by the vocabulary embedding layers.

Core Architectural Modifications and Components

The minimal Llama architecture relies on several critical deviations from the original standard Transformer formulation. These components must be precisely implemented to ensure compatibility with ecosystem inference tools such as Hugging Face and local WebGPU runtimes. Pre-normalization using Root Mean Square Normalization (RMSNorm) is utilized in place of standard Layer Normalization. RMSNorm centers the activations purely on variance scaling without mean-centering, which substantially reduces computational overhead while preserving training stability across deep layers4. This is particularly critical in TinyLMs, where activation scaling anomalies can easily destabilize a highly constrained hidden dimension. The Feed-Forward Network (FFN) departs from standard ReLU or GELU activations, instead utilizing a Swish-Gated Linear Unit (SwiGLU)4. The SwiGLU mechanism requires three distinct projection matrices: gate\_proj, up\_proj, and down\_proj. The element-wise multiplication of the activated gate and the up projection necessitates a carefully tuned intermediate dimension. While large models conventionally set the intermediate dimension to approximately [Figure omitted from source export] of the hidden dimension, TinyLM architectures often tune this scalar manually to hit exact parameter budgets, such as 341 for a 1M model or 1024 for a 20M model8. Rotary Positional Embeddings (RoPE) are utilized to encode absolute positional information via a rotation matrix, naturally integrating relative position dependency directly into the self-attention mechanism4. For a query matrix [Figure omitted from source export], the transformation is mathematically applied as a rotation in the complex plane, where the frequency term [Figure omitted from source export]. However, implementing RoPE for Hugging Face ecosystem compatibility reveals a critical divergence in codebase logic. Meta's official reference implementation processes query and key vectors in an interleaved manner, whereas the Hugging Face transformers implementation applies a rotate\_half helper function that splits the tensor along the last dimension, negates the second half, and concatenates them in reverse order9. When engineering a TinyLM from scratch, utilizing the sequential chunking method is mandatory to avoid permutation errors when exporting weights to the Hugging Face format13. While Grouped-Query Attention (GQA) is critical for large models to reduce the Key-Value (KV) cache memory footprint during autoregressive decoding, TinyLMs feature extremely small hidden dimensions and head counts. Standard Multi-Head Attention (MHA) is usually sufficient and avoids the complexity of head-grouping for sub-10M models. However, at the 20M parameter scale, implementing GQA remains highly advantageous if the model is intended to process extended context windows relative to its size4.

Parameter Budgeting and Scaling Configurations

In sub-20M parameter models, the vocabulary size disproportionately impacts the total parameter count. If an embedding layer utilizes a standard GPT or Llama vocabulary of 50,257 tokens alongside a hidden dimension of 256, the embedding and unembedding matrices alone consume over 25.7 million parameters8. This immediately violates a 5M or 10M parameter budget before a single transformer block is instantiated. Consequently, constructing models in this class necessitates either aggressive weight tying (where the final language modeling head shares the exact memory pointer as the input embedding matrix) or drastically reduced vocabulary sizes14. The following table illustrates optimal architectural configurations for 1M, 5M, 10M, and 20M parameter targets, assuming untied embeddings to maximize representational capacity.

Architectural scaling derived from standard Llama scaling formulas: Embeddings ([Figure omitted from source export]), Self-Attention ([Figure omitted from source export]), MLP ([Figure omitted from source export]), and Norms ([Figure omitted from source export])8. For models approaching the absolute 1M parameter floor, the embedding layer routinely dictates greater than 25% of the total parameter budget8. Empirical analysis of extremely small models highlights that foundational grammar generation capabilities emerge optimally at shallower depths with wider embeddings (e.g., [Figure omitted from source export], Layers=2), while long-term reasoning, plot tracking, and narrative consistency strictly demand increased layer depth3.

Safetensors Serialization and State Dictionary Topologies

For a newly trained TinyLM to seamlessly integrate with the Hugging Face transformers library—specifically the LlamaForCausalLM architecture class—and downstream deployment environments like transformers.js, the PyTorch state dictionary must strictly adhere to specific tensor naming conventions and serialization formats.

Safetensors Architecture and Metadata Formulation

The safetensors format provides a secure, high-performance, and zero-copy serialization mechanism that avoids the arbitrary code execution vulnerabilities inherent in Python's traditional pickle-based .bin or .pt formats18. A valid .safetensors file is structured with an 8-byte unsigned little-endian 64-bit integer dictating the exact byte size of the subsequent header, followed by a UTF-8 JSON header, and concluding with a raw, contiguous byte buffer containing the uncompressed tensor data21. For TinyLMs, the JSON header operates as an index, mapping tensor names to their definitions. Each definition requires a dtype declaration (e.g., "F32", "BF16", "I8"), a shape declaration represented as an array of integers, and data\_offsets represented as a two-element array indicating the byte start and end relative to the beginning of the binary buffer19. Because the safetensors specification mandates strict memory mapping without holes or gaps in the byte buffer, ensuring mathematically accurate contiguous byte offsets is non-negotiable21. To conform to the Stability AI Model Spec (SAI) or Hugging Face standard metadata formats, the header supports a dedicated \_\_metadata\_\_ dictionary containing plain string key-value pairs19. For a TinyLM, this custom metadata should denote the intended context size, the architectural scale, and the applied quantization format to assist downstream inference engines in allocating appropriate hardware resources prior to full model loading24. The specification enforces a hard 100 MB limit on the JSON header to prevent denial-of-service vulnerabilities20.

Standard State Dictionary Keys and Shapes

The model weights generated during training must be mapped precisely to the standard Llama topology. A properly constructed state\_dict for a minimal Llama model, serialized into the safetensors payload, consists of the following exact keys and respective dimensional shapes8:

If the configuration parameter tie\_word\_embeddings is set to true, the lm\_head.weight tensor may be omitted from the file entirely, as the inference runtime will automatically map the final linear projection by reference to the embed\_tokens.weight matrix in memory15.

The Configuration Payload

To initialize the computational graph correctly via AutoModelForCausalLM.from\_pretrained(), an accompanying config.json file must reside alongside the .safetensors file. The minimal fields required to successfully instantiate a Llama architecture include the structural dimensions, architectural flags, and tokenizer compatibility identifiers15:

JSON { "architectures": \["LlamaForCausalLM"\], "model\_type": "llama", "vocab\_size": 4096, "hidden\_size": 256, "intermediate\_size": 682, "num\_hidden\_layers": 6, "num\_attention\_heads": 8, "num\_key\_value\_heads": 8, "hidden\_act": "silu", "max\_position\_embeddings": 1024, "initializer\_range": 0.02, "rms\_norm\_eps": 1e-05, "bos\_token\_id": 1, "eos\_token\_id": 2, "pad\_token\_id": 0, "rope\_theta": 10000.0, "tie\_word\_embeddings": false, "use\_cache": true }

Tokenization Tradeoffs: Byte, BPE, and SentencePiece

The tokenization regime fundamentally dictates both the computational performance and the physical parameter distribution of a TinyLM. In standard multi-billion parameter LLMs, vocabularies containing 32,000 tokens (Llama 2\) or 128,256 tokens (Llama 3.1) compress textual information highly efficiently, which reduces sequence lengths at the acceptable cost of massive embedding matrices15. For sub-20M models, this approach is mathematically prohibitive, requiring a strict re-evaluation of algorithmic tokenization tradeoffs.

Vocabulary Size vs. Model Capacity

Selecting the tokenization algorithm involves a zero-sum tradeoff between context density (how much text fits into a fixed sequence length) and transformer depth (how much reasoning capability the model possesses)3. High vocabulary tokenizers, such as a 50,257-token Byte-Pair Encoding (BPE), allow sequences to be processed in fewer autoregressive steps. This minimizes the KV cache size and substantially speeds up end-user inference14. However, it severely bloats the embedding and output layers. For a 5M parameter model with a hidden dimension of 256, a 50k vocabulary consumes 25.7 million parameters—which represents 84% of an expanded 30M parameter budget, starving the attention and MLP layers of representational power8. Conversely, low vocabulary tokenizers restrict the embedding layer to approximately 10% to 15% of the model budget, freeing millions of parameters to be allocated into deeper transformer blocks8. The resulting penalty is sequence elongation, which increases the KV cache memory footprint and slows down autoregressive decoding due to the [Figure omitted from source export] scaling of attention mechanisms.

Algorithm Selection

Byte-Level Tokenization treats every individual raw byte as a token, resulting in a strictly fixed vocabulary size of 2568. This approach completely eliminates out-of-vocabulary (OOV) errors and reduces the embedding matrix to an insignificant fraction of the model size (e.g., 65,536 parameters for a hidden dimension of 128\)8. However, the drawback is severe sequence inflation. A single English word may require four to eight distinct tokens to process, compounding the computational cost of the self-attention mechanism and drastically shrinking the effective context window. Subword Byte-Pair Encoding (BPE) balances the extremes of character-level and word-level representations. For a TinyLM, a truncated BPE vocabulary limited to the top 1,024 to 4,096 tokens heavily favored in the training corpus yields the optimal synthesis of sequence compression and parameter efficiency8. Empirical studies on models like TinyStories demonstrate that trimming BPE tokenizers to exclusively match the simplified lexical domain of the training data allows for highly efficient knowledge capture without wasting parameters on unused lexical combinations3. Modern implementations rely heavily on the Rust-based tokenizers library for parallelized batch encoding33. SentencePiece directly processes raw text—including spaces as a distinct character—which bypasses the complex, language-specific pre-tokenization regular expressions required by standard BPE. While the standard Llama models utilize a SentencePiece implementation with BPE, SentencePiece also supports a Unigram language model algorithm capable of dynamically optimizing the vocabulary29. However, for highly constrained English domains, standard BPE remains the dominant architecture due to its highly deterministic merging mechanics34. In morphologically rich languages, specialized tokenizers (such as BrahmicTokenizer-131K) demonstrate that adapting the BPE merge rules can result in up to 76% fewer tokens, though this necessitates larger vocabulary budgets that may strain a TinyLM32.

Structuring tokenizer.json

Regardless of the selected tokenization algorithm, modern deployment via Hugging Face transformers or browser-based WASM environments requires a strictly formatted tokenizer.json payload compatible with the core tokenizers schema33. This file completely encapsulates the tokenization logic, ensuring that no complex Python string manipulation scripts are required at inference time. A compliant tokenizer.json schema must explicitly define the sub-components33:

• model: The specific algorithm (BPE, Unigram), the base vocabulary mapping, and the learned merge rules.
• pre\_tokenizer: The deterministic rules defining how text is isolated before algorithmic splits (e.g., isolating whitespace and punctuation using regular expressions).
• normalizer: The unicode normalization protocol applied (e.g., NFKC or NFKD).
• decoder: The specific instructions for assembling predicted token arrays back into human-readable text sequences.
• added\_tokens: Mandatory definitions for structural identifiers, including the Beginning of Sequence (\[BOS\]), End of Sequence (\[EOS\]), Padding (\[PAD\]), and Unknown (\[UNK\]) tokens33.

Data Regimes, Prompts, and Smoke-Test Methodology

The dominant paradigm for training extremely small language models relies entirely on strict data curation over raw volume. The fundamental finding of the TinyStories methodology is that structural coherence, grammatical accuracy, and logical persistence in generative models are bottlenecked by the complexity of the training corpus, not strictly the mathematical scale of the model architecture1.

Synthetic Data Generation and Curation

To successfully elicit fluent English generation from models as small as 1 million to 5 million parameters, the training data must be ruthlessly simplified. Standard internet datasets, such as CommonCrawl or Fineweb, contain profound lexical diversity, idiomatic variance, and syntactic complexity that rapidly overwhelm the limited representational capacity of a TinyLM, resulting in incoherent text2. The TinyStories paradigm circumvents this limitation by restricting the entire corpus vocabulary to approximately 1,500 core words, mirroring the typical lexicon of a 3- to 4-year-old child2. Frontier models (such as GPT-3.5 and GPT-4) are programmatically instructed to generate millions of short narratives strictly adhering to this constrained vocabulary and simple syntactic structure3. The resulting corpus—comprising over 2 million stories—contains multi-turn dialogue, consistent character names, and basic plot structures spanning several paragraphs, providing the TinyLM with a perfectly clean signal for language modeling1. Refinements of this methodology include SimpleStories, which addresses the formulaic repetition found in the original TinyStories dataset. In the original data, up to 59% of the stories begin with the exact phrase "Once upon a time," which degrades semantic diversity. Adjusting the generation prompts to mandate varied sequence structures and fewer repetitions yields significantly higher quality output during zero-shot testing39. The paradigm has also been successfully adapted for morphologically complex languages via Regional-TinyStories, proving that constrained synthetic data can elicit coherence across distinct linguistic families using sub-50M parameter architectures17.

Evaluation Prompts and Smoke-Test Methodology

Evaluating a 5M or 10M parameter model presents a unique challenge. Standard LLM benchmarks (such as MMLU, GSM8K, or HumanEval) automatically fail for TinyLMs, as the models possess absolutely no real-world knowledge, mathematical reasoning, or code comprehension2. Instead, evaluation relies entirely on carefully constructed generative smoke tests and a comprehensive "LLM-as-a-judge" paradigm. The standard prompting smoke test is designed to verify the fundamental mechanics of the autoregressive generation, ensuring the model has learned structural syntax rather than merely memorizing bigram probabilities. A typical smoke test prompt introduces a character, an object, and an environmental trigger: "Lily was a little girl who loved to play with her blue ball. One day, the ball bounced into the dark forest. Lily felt"

The expected behavior mandates that the model complete the sentence with a contextually appropriate emotion (e.g., "scared" or "sad") and maintain the narrative thread without deteriorating into grammatical gibberish or hallucinating external characters not established in the prompt1. Because manual evaluation of these completions is unscalable, the methodology utilizes an automated GPT-4 grading rubric2. The generated outputs (typically 100 completions from distinct seeded prompts, using a temperature of 0.8 and standard repetition penalties) are fed into GPT-4, which scores them across three distinct axes3:

1. Grammar: An assessment of syntactic correctness and basic English rules. This capability emerges earliest in the training cycle, often stabilizing in models as small as 1M parameters with a hidden dimension of 643.
2. Creativity: An evaluation of the narrative novelty, heavily penalizing outputs that fall into endless repetitive loops or copy the exact phrasing of the prompt.
3. Consistency: The ability to retain character traits, environmental details, and logical flow over 200 to 500 tokens. This capability is the most demanding and requires significantly deeper transformer architectures, typically emerging reliably only closer to the 10M to 20M parameter scale3.

Browser-Native Deployment and Memory Budgets

The ultimate technical utility of 1M–20M parameter models lies in their ability to execute deterministically on edge devices, particularly directly within the client web browser using WebGPU and WebAssembly (WASM). Frameworks such as transformers.js and wllama successfully abstract the low-level execution, but the strict hardware constraints imposed by browsers necessitate rigorous, upfront memory budgeting40.

WebGPU Execution Constraints and Architecture

Unlike native runtime environments where an application can allocate system RAM freely, web browsers enforce hard limits on GPU memory allocation to prevent malicious or poorly optimized web tabs from crashing the underlying operating system display driver40. The most critical ceiling for LLM inference in WebGPU is the maxStorageBufferBindingSize, which dictates the absolute maximum size of a single data buffer passed to a compute shader during execution43. Hardware variations mandate that developers budget for the lowest common denominator:

• Mobile GPUs (e.g., ARM Mali, Adreno): Frequently cap maxStorageBufferBindingSize at a highly restrictive 128 MB or 256 MB43.
• Integrated GPUs (e.g., Intel Iris/UHD): Typically capped between 256 MB and 1 GB43.
• Discrete Desktop GPUs and Apple Silicon: Generally allow up to 2 GB or 4 GB, provided the browser permits it43.

Safari's Metal backend imposes strict limits by default (e.g., 256 MB on iPhone devices), while Chrome limits the default buffer size to 256 MB unless explicitly overridden via requiredLimits during the requestDevice() adapter initialization phase44. Furthermore, buffer allocation overhead is non-trivial; dynamically allocating buffers takes between 0.35ms and 1.7ms per operation. In an autoregressive loop, this overhead compounds rapidly, meaning that all memory for weights and activations must be statically pre-allocated in a size-bucketed buffer pool during model initialization to prevent the UI thread from freezing40.

Model Weight Memory Footprints (f32, q8, q4)

To ensure the model fits within the strictest 128 MB maxStorageBufferBindingSize limits of older mobile devices, quantization of the safetensors payloads becomes highly critical41. The following calculations demonstrate the static VRAM requirement for the model weights across different precisions:

A 20M parameter model in FP32 consumes approximately 80 MB of VRAM, sitting below the 128 MB threshold of low-end mobile WebGPU implementations43. However, if reduced to Q8 via GGUF or int8 Safetensors, the 20 MB footprint becomes negligible. This ultra-lightweight profile allows the browser's Cache API to instantaneously load the model on subsequent visits without incurring costly network requests or stalling the main thread41.

KV Cache Sizing and Activation Buffers

The static model weights only represent a portion of the runtime memory requirement. Autoregressive text generation requires caching historical Key and Value matrices at every layer to avoid redundant computation of past tokens. The total size of this KV Cache is determined by the sequence length, batch size, number of layers, number of KV heads, and the head dimension8. For a standard inference batch size of 1, the mathematical formula for the KV Cache is: [Figure omitted from source export] Assuming [Figure omitted from source export] (which is highly common for TinyLMs) and varying the context sequence length, the memory profiles map as follows8:

Analysis of combined footprints: A 20M parameter model processing a 2048-token context window with an FP32 KV Cache requires 72 MB purely for activations8. When combined with the 80 MB static weight tensor, the total 152 MB runtime footprint formally exceeds the 128 MB hard limit of mobile devices, triggering immediate out-of-memory driver crashes8. To safely deploy a 20M parameter model across universal browser targets without failure, engineers must implement one of three distinct mitigation strategies:

1. Quantize weights to FP16 or Q8: Slashing the static weight requirement to 20–40 MB opens enough buffer space to house the full FP32 KV cache8.
2. Quantize the KV Cache: Storing historical keys and values in Int8 drops the 2048-sequence cache requirement from 72 MB down to 18 MB8.
3. Implement Grouped-Query Attention (GQA): If [Figure omitted from source export] is reduced from 12 down to 4 via head grouping, the overall KV cache footprint is permanently reduced by 66% regardless of precision4.

When executing the model inside a Web Worker to avoid blocking the user interface, WebAssembly (WASM) fallback computation is frequently leveraged on devices lacking WebGPU hardware or driver support41. However, WASM multi-threading requires SharedArrayBuffer support. To utilize this memory buffer securely, the host server architecture must provide strict Cross-Origin-Opener-Policy: same-origin (COOP) and Cross-Origin-Embedder-Policy: require-corp (COEP) headers on all responses, otherwise the browser will silently block the model execution in production environments41.

Conclusion

The engineering and deployment of 1M to 20M parameter language models represents a fascinating inversion of traditional LLM scaling laws. Rather than achieving generalized capability through massive data memorization across hundreds of billions of weights, TinyLMs rely on intense architectural discipline, precise tensor mappings, and extreme data curation. A successful TinyLM pipeline must tightly constrain the vocabulary—opting for byte-level representations or highly focused 1,000-token BPE schemas—to prevent the embedding matrices from starving the transformer depth of crucial mathematical parameters. The training phase must be restricted to highly simplified, synthetically generated narratives modeled after child-like vocabularies, which enables coherent generative grammar and logical persistence to emerge from sub-5M parameter profiles. Finally, by strictly adhering to the safetensors model formats, utilizing proper Hugging Face configuration topologies, and meticulously managing the delicate balance of weight quantization and KV Cache memory footprint, these micro-models can execute instantaneously, privately, and completely offline directly within highly constrained edge browser environments.

Works cited

1. gary23w/gary-4-petite \- Hugging Face, https://huggingface.co/gary23w/gary-4-petite
2. TinyStories: How Small Can Language Models Be and Still Speak Coherent English? \- arXiv, https://arxiv.org/abs/2305.07759
3. TinyStories Dataset for Small Language Models \- Emergent Mind, https://www.emergentmind.com/topics/tinystories-dataset
4. Llama 3.1 405B — CUDA Graph Best Practice for PyTorch \- NVIDIA Documentation, https://docs.nvidia.com/dl-cuda-graph/examples/llama-31-405b.html
5. The Path to Achieve Ultra-Low Inference Latency With LLaMA 65B on PyTorch/XLA, https://pytorch.org/blog/path-achieve-low-inference-latency/
6. Deconstructing LLaMA 2: A PyTorch Deep Dive | by Philiprj \- Medium, https://medium.com/@philiprj2/deconstructing-llama-2-a-pytorch-deep-dive-84e97dd42def
7. \[P\] Built GPT-2, Llama 3, and DeepSeek from scratch in PyTorch \- open source code \+ book, https://www.reddit.com/r/LocalLLaMA/comments/1sm82ze/p\_built\_gpt2\_llama\_3\_and\_deepseek\_from\_scratch\_in/
8. unknown\_url
9. Rotary Position Embeddings for Long Context Length \- MachineLearningMastery.com, https://machinelearningmastery.com/rotary-position-embeddings-for-long-context-length/
10. transformers/src/transformers/models/llama/modeling\_llama.py at main \- GitHub, https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling\_llama.py
11. Is LLaMA rotary embedding implementation correct? \- Hugging Face Forums, https://discuss.huggingface.co/t/is-llama-rotary-embedding-implementation-correct/44509
12. \[LLaMA\] Rotary positional embedding differs with official implementation · Issue \#25199 · huggingface/transformers \- GitHub, https://github.com/huggingface/transformers/issues/25199
13. Is LLaMA rotary embedding implementation correct? \- \#9 by cc007 \- Transformers, https://discuss.huggingface.co/t/is-llama-rotary-embedding-implementation-correct/44509/9
14. abhilash88/tinystories-slm-gpt \- Hugging Face, https://huggingface.co/abhilash88/tinystories-slm-gpt
15. Llama2 \- Hugging Face, https://huggingface.co/docs/transformers/v4.44.1/model\_doc/llama2
16. config.json · unsloth/Llama-3.3-70B-Instruct at main \- Hugging Face, https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/blob/main/config.json
17. Regional Tiny Stories: Using Small Models to Compare Language Learning and Tokenizer Performance \- arXiv, https://arxiv.org/html/2504.07989v1
18. DeepSeek GGUF vs Safetensors: Which Format Should You Use? \- Chat-Deep.ai, https://chat-deep.ai/guide/deepseek-gguf-vs-safetensors/
19. SafeTensors \- Grokipedia, https://grokipedia.com/page/SafeTensors
20. SafeTensors Format: A Guide to Secure ML Model Serialization \- DataCamp, https://www.datacamp.com/blog/safetensors-format
21. GitHub \- safetensors/safetensors: Simple, safe way to store and distribute tensors, https://github.com/safetensors/safetensors
22. SafeTensors: Efficient Serialization Format for Deep Learning | by Nishtha kukreti | Medium, https://medium.com/@nishthakukreti.01/safetensors-efficient-serialization-format-for-deep-learning-57364317be43
23. Reading Safetensors Headers \- Zenn, https://zenn.dev/platina/articles/e65c73cb01a900?locale=en
24. Stability.AI Model Metadata Standard Specification \- GitHub, https://github.com/Stability-AI/ModelSpec
25. fsdp\_qlora/Converting the State Dict.ipynb at main \- GitHub, https://github.com/AnswerDotAI/fsdp\_qlora/blob/main/Converting%20the%20State%20Dict.ipynb
26. push\_to\_hub() for Llama 3.1 8B doesn't save lm\_head.weight tensor \#37303 \- GitHub, https://github.com/huggingface/transformers/issues/37303
27. config.json · unsloth/llama-3-8b at main \- Hugging Face, https://huggingface.co/unsloth/llama-3-8b/blob/main/config.json
28. Llama \- Hugging Face, https://huggingface.co/docs/transformers/model\_doc/llama
29. Meta-Llama-3.1-8B Files \- Read the Docs, https://paddlenlp.readthedocs.io/en/latest/\_static/website/meta-llama/Meta-Llama-3.1-8B/
30. Train Tiny Stories dataset from the paper \- "TinyStories: How Small Can Language Models Be and Still Speak Coherent English?" \- GitHub, https://github.com/SauravP97/tiny-stories-hf
31. evintunador/templateGPT: customizable template GPT code designed for easy novel architecture experimentation \- GitHub, https://github.com/evintunador/templateGPT
32. Regional-TinyStories: A Small Language Model Framework for Evaluating Language Learning , Tokenizers, and Datasets \- ACL Anthology, https://aclanthology.org/2025.findings-ijcnlp.142.pdf
33. Tokenizer \- Hugging Face, https://huggingface.co/docs/transformers/main\_classes/tokenizer
34. Tokenizers \- Hugging Face, https://huggingface.co/docs/transformers/fast\_tokenizers
35. LLM & VLM Stack | Deeplearning4j, https://deeplearning4j.konduit.ai/deeplearning4j/overview-4
36. theschoolofai/BrahmicTokenizer-131K \- Hugging Face, https://huggingface.co/theschoolofai/BrahmicTokenizer-131K
37. Huggingface Tokenizers \- Deep Java Library, https://docs.djl.ai/master/extensions/tokenizers/index.html
38. Using Local Tokenizers Without HuggingFace | NVIDIA AIPerf Documentation, https://docs.nvidia.com/aiperf/dev/tutorials/configuration/using-local-tokenizers-without-hugging-face
39. Parameterized Synthetic Text Generation with SimpleStories \- arXiv, https://arxiv.org/html/2504.09184v2
40. Llamas on the Web: Memory-Efficient, Performance-Portable, and Multi-Precision LLM Inference with WebGPU \- arXiv, https://arxiv.org/html/2605.20706v1
41. How to Run AI Models Directly in the Browser with Transformers.js and WebGPU, https://www.codersarts.com/post/how-to-run-ai-models-directly-in-the-browser-with-transformers-js-and-webgpu
42. Wllama, https://mikeesto.com/posts/wllama/
43. WebGPU Memory Limits: maxStorageBufferBindingSize \- Ayoob AI, https://ayoob.ai/blog/webgpu-maxstoragebufferbindingsize-limits-enterprise
44. WebGPU bugs are holding back the browser AI revolution | by Marcelo Emmerich | Medium, https://medium.com/@marcelo.emmerich/webgpu-bugs-are-holding-back-the-browser-ai-revolution-27d5f8c1dfca
45. GPUSupportedLimits \- Web APIs | MDN, https://developer.mozilla.org/en-US/docs/Web/API/GPUSupportedLimits
46. Why is webgpu on mac "max binding size" much smaller than reported "max buffer size"?, https://stackoverflow.com/questions/78034628/why-is-webgpu-on-mac-max-binding-size-much-smaller-than-reported-max-buffer-s
47. What's New in WebGPU (Chrome 133\) | Blog, https://developer.chrome.com/blog/new-in-webgpu-133
48. Run AI Models in the Browser with WebGPU & WASM \- Mad Devs, https://maddevs.io/writeups/running-ai-models-locally-in-the-browser/
49. Run Gemma 4 in Your Browser with WebGPU (No Server) | Blog, https://gemma4-ai.com/blog/webgpu-browser-guide
50. How to support new model in lmdeploy.pytorch, https://lmdeploy.readthedocs.io/en/v0.4.0/advance/pytorch\_new\_model.html