The Architecture and Evolution of Artificial Intelligence Runtimes: Typology, Mechanisms, and Future Trajectories

Full report

On this page

Introduction to Artificial Intelligence Runtimes and the Execution Paradigm

In the broader lifecycle of artificial intelligence and deep learning, computational workloads are strictly bifurcated into two foundational phases: training and inference. While the training phase operates as the computationally intensive period where a model analyzes massive datasets to adjust internal parameters and learn complex pattern recognition, the inference phase represents the deployment and operational realization of that learning1. Artificial intelligence inference is the execution phase wherein a fully trained model processes novel, unseen data to generate predictions, structured outputs, classifications, or generative media in real time1. An Artificial Intelligence Runtime is the highly optimized software infrastructure and execution engine responsible for facilitating this inference phase1. It acts as the critical intermediary between the mathematical abstractions of a neural network and the physical hardware of the host system6. The necessity for specialized, highly tuned AI runtimes arises from the unique computational profile of modern deep neural networks, particularly Large Language Models (LLMs) based on the transformer architecture. Inference, unlike training, does not require gradient calculations or backward passes; it relies entirely on a forward pass, which executes the learned weights against new inputs without altering the model's fundamental knowledge base1. However, as models scale from billions to trillions of parameters, the computational bottleneck shifts dramatically5. LLM inference is inherently split into two distinct stages: the prefill stage and the decode stage8. During the prefill stage, the runtime processes the entirety of the user's input prompt in parallel to compute the initial key-value (KV) states and generate the very first output token9. This phase is deeply compute-bound, relying heavily on General Matrix Multiply (GEMM) operations that saturate the floating-point operation (FLOPS) capacity of modern accelerators8. Conversely, the decode stage generates subsequent tokens sequentially, one at a time. Because each decode step must read the entire KV cache from memory to compute self-attention for the newly generated token, this phase is severely memory-bandwidth-bound and relies on General Matrix-Vector Multiply (GEMV) operations8. Balancing this duality—maximizing FLOPS utilization during prefill while mitigating memory bottlenecks during decode—is the central challenge of modern AI engineering. The future of artificial intelligence depends entirely on how runtimes navigate these bottlenecks to lower the cost of deployment, reduce latency, and scale to support continuous, multi-agent workflows7. This comprehensive analysis explores the architectural mechanisms of deep learning compilers, the diverse typology of AI runtimes across data centers, edge environments, and web browsers, and the future trajectories of orchestration and speculative execution that will define the next era of artificial intelligence.

Deep Learning Compilers and Execution Optimization

Before a neural network can be executed by a runtime, it must be translated from a high-level programming framework (such as PyTorch or TensorFlow) into low-level machine code optimized for specific hardware architectures. This translation is governed by deep learning compilers, which execute a multi-stage pipeline of Intermediate Representations (IR) to fuse operations and map tensors to silicon12.

The Compilation Pipeline and Intermediate Representations

The compilation pipeline initiates with the frontend ingestion of the user model, translating the Python-based model definition into a framework-specific Intermediate Representation, such as TorchScript or GraphDef12. At this stage, the compiler performs type checking, shape inference, and initial computational graph construction12. This graph is an acyclic representation of the deterministic mathematical steps required to execute a forward pass14. Following ingestion, the compiler elevates the graph to a High-Level IR (such as MLIR or XLA HLO), where hardware-agnostic optimizations occur6. The compiler reasons about the mathematical properties of the neural network to apply constant folding, dead code elimination, and algebraic simplification12. The most critical optimization performed at this level is computation graph fusion, which merges multiple operations to eliminate redundant memory traffic and kernel launch overheads12.

• Vertical Fusion (Element-wise Fusion): Element-wise operations along the same dimension are combined. For example, a compiler will fuse a sequential bias addition, ReLU activation function, and scaling multiplication. Without fusion, these three operations would require six memory transfers (three reads and three writes). Fused into a single kernel, the data is loaded once, all three computations are performed in the hardware registers, and the result is stored once, drastically reducing memory bandwidth constraints12.
• Horizontal Fusion (Producer-Consumer Fusion): Operations with strict data dependencies are merged. In convolutional neural networks, a convolution operation naturally requires a subsequent batch normalization12. Horizontal fusion feeds the output of the convolution directly into the batch normalization logic without materializing large intermediate tensors in main memory12.
• Megakernel Fusion: Advanced compilers generate dynamic megakernels that fuse complex, data-dependent subgraphs (such as entire Mixture-of-Experts routing layers or combined GEMM and communication operations) into a single persistent kernel15. This approach eliminates kernel launch gaps and exposes inter-kernel parallelism across streaming multiprocessors15.

Finally, the compiler lowers the optimized graph to a Mid-Level IR (such as LLVM IR) and then to a Target-Specific Low-Level IR6. During this phase, the compiler applies hardware-specific memory layout optimizations, loop tiling, and vectorization, ultimately generating executable Machine Code, PTX (for NVIDIA GPUs), or SPIR-V (for cross-vendor execution)12.

Dynamic vs. Ahead-of-Time (AOT) Compilation

The application of these compiler techniques bifurcates into dynamic, Just-in-Time (JIT) compilation and Ahead-of-Time (AOT) compilation, depending on the constraints of the deployment environment16. In highly flexible environments, frameworks like PyTorch 2.x utilize torch.compile to provide dynamic JIT compilation17. The architecture of torch.compile relies on three core components: TorchDynamo, TorchInductor, and Backend Toolchains18. TorchDynamo dynamically analyzes Python bytecode at runtime, safely capturing computation into an FX graph while managing Python-native features like dynamic control flows and closures18. This FX graph is passed to TorchInductor, which lowers the representation and generates optimized hardware-specific code, such as OpenAI Triton code for GPUs18. Modern iterations of torch.compile support symbolic shape analysis via Dim.AUTO, allowing the compiler to generate execution guards that validate shape ranges rather than recompiling the entire graph every time a user inputs a sequence of a different length19. Conversely, execution on resource-constrained embedded systems demands Ahead-of-Time (AOT) memory planning. In environments where dynamic memory allocation inflicts severe performance and power penalties, runtimes convert the computational graph into a static execution plan16. All functional operations are converted into "out variants," where memory buffers are pre-allocated during compilation and passed into the operators as arguments16. The runtime merely dispatches pre-compiled kernels over static memory, bypassing the need for runtime memory management and enabling complex models to run on highly restricted microcontrollers16.

Data Center AI Runtimes: High-Throughput LLM Serving

In enterprise cloud and data center environments, the primary objective of an AI runtime is to maximize aggregate throughput (measured in tokens per second) and minimize Time-to-First-Token (TTFT) while serving thousands of concurrent requests1. Modern serving stacks—most notably vLLM, SGLang, and TensorRT-LLM—share foundational techniques such as continuous in-flight batching (where new requests seamlessly join a running batch without waiting for the entire batch to finish) and paged memory management, but their architectural implementations represent radically different engineering philosophies20.

The Python Scheduler vs. The Compiled Engine

vLLM serves as the industry's default runtime owing to its deployment predictability and massive ecosystem support24. It utilizes a Python-level scheduler that maintains a priority queue of waiting, running, and swapped requests, allocating KV cache pages from a central block table20. While vLLM supports over 400 model architectures seamlessly, the Python-level scheduling introduces microsecond overheads (approximately 200–500 μs per scheduling cycle) that can compound negatively during extreme high-concurrency scenarios20. SGLang directly addresses this bottleneck by migrating the bulk of the scheduling logic into C++ and CUDA, achieving a near-zero-overhead CPU scheduler20. Furthermore, SGLang implements constrained decoding (e.g., forcing the model to output strict JSON schemas or adhere to specific regex grammars) at the scheduler level, rather than bolting it on as a post-hoc token filter20. This results in highly efficient structured generation that is up to ten times faster than alternative open-source solutions27. TensorRT-LLM abandons the Python flexibility entirely in favor of an AOT compiled engine20. By converting model checkpoints into TensorRT-LLM's specific weight format and compiling the model into a static engine, it avoids dynamic scheduling overhead completely20. This engine relies on auto-tuned, hardware-calibrated CUDA kernels to deliver the absolute highest raw throughput—often 15–25% higher than vLLM on dense models running on NVIDIA H100s20. However, the operational complexity is immense; altering the model requires a lengthy 28-minute recompilation process, making TensorRT-LLM suitable only for stable, long-term deployments on homogeneous NVIDIA infrastructure21.

The Evolution of the KV Cache: PagedAttention to RadixAttention

The most critical innovation in data center runtimes has been the virtualization of the KV cache. In naive inference, runtimes allocated contiguous memory blocks based on the maximum possible sequence length of a user request. Because actual sequence lengths vary, this resulted in massive internal and external memory fragmentation, wasting up to 40% of available VRAM and artificially capping concurrent user capacity20. vLLM resolved this by introducing PagedAttention, which manages the KV cache similarly to an operating system's virtual memory page tables20. It dynamically allocates KV cache into fixed-size physical blocks (e.g., 16 tokens per block) that do not need to be contiguous in memory, dropping VRAM fragmentation from 40% to under 4%20. While PagedAttention solved fragmentation for individual requests, it did not resolve redundancy across multiple requests. In agentic workflows, multi-turn chatbots, and Retrieval-Augmented Generation (RAG) systems, multiple concurrent requests often share massive system prompts or context documents21. SGLang advanced the paradigm by introducing RadixAttention20. RadixAttention structures the entire global KV cache as a radix tree (or compressed trie)23. When a new request arrives, the runtime traverses the tree to locate the longest matching prefix sequence that has already been computed for a previous request23. The runtime reuses the exact physical memory pages for that prefix, completely bypassing the compute-heavy prefill phase for those tokens23. For example, if a swarm of five AI agents are instructed to translate a legacy codebase, sharing a 4,000-token system prompt and a 2,000-token source document, a standard runtime allocates 30,000 tokens of redundant memory before generating a single response28. With RadixAttention, the 6,000 shared tokens form the root of the tree, and the five agents act as active leaf nodes, allocating new memory solely for their unique generated outputs28. This 80% reduction in memory footprint allows workloads that previously required multi-node clusters to execute on a single GPU28.

Universal Frameworks and Heterogeneous Computing

While data center models rely on massive GPU arrays, the democratization of artificial intelligence requires runtimes capable of universal deployment across consumer-grade heterogeneous hardware29. A heterogeneous compute platform integrates multiple types of processors—Central Processing Units (CPUs), Graphics Processing Units (GPUs), Neural Processing Units (NPUs), and Field-Programmable Gate Arrays (FPGAs)—within a single system architecture29.

Universal Execution: llama.cpp and MLC LLM

Universal frameworks are engineered to abstract away the underlying hardware, allowing a single model checkpoint to run optimally on whatever silicon is available32. The llama.cpp framework represents a foundational breakthrough in universal execution. Built entirely in pure C/C++ with no external dependencies, llama.cpp dynamically routes the computational graph across available hardware backends9. It leverages the ggml tensor library to execute custom CUDA kernels for NVIDIA hardware, Metal frameworks for Apple Silicon, HIP for AMD GPUs, and highly tuned assembly using AVX, AVX2, AVX-512, and ARM NEON SIMD (Single Instruction, Multiple Data) instructions for CPU-only execution9. Crucially, llama.cpp utilizes advanced block-wise quantization formats (such as the k-quants system, ranging from 1.5-bit to 8-bit precision) to aggressively compress the memory footprint33. By employing memory mapping, the runtime loads models directly from disk without copying them fully into RAM, allowing consumers to run massive open-source models on standard consumer laptops33. Similarly, MLC LLM (Machine Learning Compilation for Large Language Models) leverages the Apache TVM Unity compiler to provide high-performance code generation32. By treating a language model as a tvm.ir.IRModule with native dynamic shape support, MLC LLM performs extensive loop-level tensor operations (TensorIR) and fusion passes before lowering the model to the target hardware via the Relax Virtual Machine32. This results in a natively compiled library that operates seamlessly across iOS, Windows, Linux, and Android32.

Standardizing Heterogeneity: SYCL and oneAPI

As the industry shifts toward multi-vendor heterogeneous clusters to optimize the Cost Per Million Tokens (CPMT), the reliance on proprietary programming models like NVIDIA's CUDA creates severe vendor lock-in and systemic friction7. The future of heterogeneous AI runtimes relies on open standards. SYCL, an open, royalty-free standard governed by the Khronos Group, provides a cross-platform abstraction layer based on modern ISO C++39. SYCL allows developers to write single-source code that targets CPUs, GPUs, and FPGAs simultaneously39. Intel's oneAPI initiative serves as a primary implementation of SYCL, providing the DPC++/C++ Compiler that translates SYCL code for execution across multi-vendor architectures38. Frameworks like CHARM-SYCL extend this further by providing portable compiler and runtime workflows that can dynamically switch target accelerators at runtime within extreme heterogeneous systems, ensuring that no compute resource sits idle during large-scale inference42.

Edge, Mobile, and IoT Execution Environments

Moving artificial intelligence to the extreme edge—smartphones, wearables, embedded systems, and automotive platforms—imposes severe constraints regarding power consumption, thermal limits, and hardware diversity2. Edge AI runtimes process data locally, guaranteeing low-latency responses, robust data privacy, and continuous offline operation independent of cloud connectivity2.

PyTorch-Native Edge Deployment: ExecuTorch

Deploying AI to the edge has historically suffered from extreme fragmentation. Developers were forced to author models in PyTorch, then convert them through lossy pipelines into TensorFlow Lite, ONNX, or vendor-specific runtimes like Apple's CoreML or Qualcomm's SNPE, leading to numerical mismatches and degraded accuracy6. ExecuTorch resolves this by providing a unified, PyTorch-native deployment framework43. Utilizing torch.export(), the compiler translates the model into an Export Intermediate Representation (EXIR) graph composed of fewer than 300 Core ATen primitives, stripping away all Python dependencies while retaining debug symbols16. Through selective backend delegation, ExecuTorch partitions the graph, sending standard operations to the CPU via highly optimized XNNPACK libraries, graphics workloads to mobile GPUs via Vulkan or Metal, and dense neural computations to specialized NPUs43. This allows developers to validate Post-Training Quantization (PTQ) entirely within PyTorch before deploying the generated .pte binary to sub-dollar microcontrollers43.

Silicon-Specific Acceleration and Edge Orchestration

Chip manufacturers provide deeply integrated runtime APIs to maximize hardware efficiency. Qualcomm's AI Engine Direct (QNN), distributed within the Snapdragon AI Runtime (QAIRT) SDK, allows a single exported model to target the Snapdragon CPU, Adreno GPU, and the highly efficient Hexagon Tensor Processor (HTP)48. By quantizing models to 16-bit activations and INT8 weights, QNN delivers real-time, low-power inference on billions of mobile devices48. Similarly, the OpenVINO toolkit abstracts silicon complexity for Intel AI PCs, providing an execution API that orchestrates workloads across Intel Core CPUs and integrated AI NPUs47. Managing these diverse runtimes across an enterprise edge deployment requires dedicated orchestration platforms.

Platforms like Portainer and SUSE Edge allow organizations to deploy and manage containerized AI runtimes across disparate geographical locations, ensuring that factories, retail stores, and telecom nodes run synchronized, updated models regardless of underlying hardware variations49.

Web-Native AI and the Agentic Browser Landscape

The browser is rapidly becoming a premier execution environment for artificial intelligence. By executing models entirely client-side, web-native AI runtimes eliminate server costs, preserve user privacy, and deliver zero-latency interactions50. The W3C Web Machine Learning Working Group is actively standardizing this paradigm shift50.

WebAssembly, WebGPU, and the WebNN Standard

Historically, browser-based AI was restricted to WebAssembly (Wasm), which executes solely on the CPU, or WebGL, which was designed for graphics and lacks the precision controls necessary for deep learning50. The landscape is currently undergoing a structural evolution driven by WebGPU and WebNN50. WebGPU is a modern web API that exposes low-level GPU rendering and compute capabilities directly to the browser54. It supports compute shaders for massively parallel data operations and introduces half-precision (FP16) floating-point arithmetic52. By halving memory bandwidth requirements compared to single-precision formats, WebGPU allows libraries like ONNX Runtime Web and TensorFlow.js to execute sophisticated generative models, such as Stable Diffusion Turbo, directly inside Chrome or Edge within seconds52. However, WebGPU is inherently limited to graphics processing units. The Web Neural Network API (WebNN) represents the definitive future of browser-based runtimes50. WebNN is a hardware-agnostic abstraction layer that routes machine learning operations to the most efficient underlying processor on the user's device50. As integrated Neural Processing Units (NPUs) become standard in consumer hardware (such as Intel Core Ultra processors), WebNN allows the browser to bypass the GPU entirely for AI workloads, offloading inference directly to the NPU to achieve near-native execution speeds without depleting battery life50. While currently an experimental Candidate Recommendation within the W3C, WebNN's integration into Chromium and Firefox signals a paradigm shift where AI capabilities are built directly into the fabric of the web50.

The Agentic Browser

This integration of runtimes directly into the browser has birthed the "Agentic Browser"58. Rather than users manually navigating websites, agentic browsers utilize embedded, locally running LLMs to execute tasks autonomously58. Standalone AI browsers like Perplexity Comet and ChatGPT Atlas offer agentic memory, learning from user browsing history to provide highly contextual assistance without repeated prompting58. Meanwhile, mainstream browsers are integrating AI deeply into their architectures. Google Chrome’s "Auto Browse," powered by the Gemini 3 model, turns the browser into an autonomous agent capable of scrolling, clicking, filling forms, and navigating across authenticated sessions on behalf of the user58. To support these agents, website architectures must evolve, ensuring high-quality semantic HTML and avoiding overly aggressive anti-bot CAPTCHAs that disrupt legitimate AI crawlers58.

Multi-Node Orchestration: Scaling Reasoning Models

As AI models advance from simple chatbots to complex, multi-modal reasoning models, the computational demands frequently exceed the physical limits of a single server node8. Scaling inference across clusters requires advanced network orchestration and disaggregated serving8.

Disaggregated Serving and NVIDIA Dynamo

Co-locating the prefill and decode stages of LLM inference on the same GPU leads to severe inefficiencies, especially for long input sequences8. Disaggregated serving solves this by separating the cluster into distinct worker pools: prefill nodes and decode nodes8. Prefill nodes, optimized for massive compute capacity, process the user prompt and generate the initial KV cache8. This cache is then transferred asynchronously over ultra-high-bandwidth interconnects (via libraries like the NVIDIA Inference Transfer Library, or NIXL) to decode nodes8. The decode nodes, which require immense memory bandwidth but fewer compute cores, take over the autoregressive generation8. This allows infrastructure managers to scale the prefill and decode clusters independently based on traffic shapes61. Orchestrating this separation across thousands of GPUs is the primary function of NVIDIA Dynamo8. Dynamo acts as a high-throughput control plane deployed on top of engines like vLLM or SGLang8. It continuously monitors GPU capacity metrics and Application Service Level Objectives (SLOs) to dynamically allocate worker nodes8. Dynamo introduces highly sophisticated routing mechanisms. Its Smart Router tracks KV cache blocks across massive multi-node fleets8. When a request arrives, Dynamo hashes the prompt and queries its distributed Radix Tree, routing the request to the specific node that already holds the matching KV cache, thus bypassing the costly prefill computation entirely8. Furthermore, Dynamo natively understands "Agentic Hints." By passing metadata regarding latency sensitivity and expected output length to the frontend API, Dynamo can prioritize multi-turn interactive traffic over background processing and execute cache pinning for persistent conversational agents61. Dynamo also accelerates cold start times by 7x through ModelExpress, which loads a model's weights into a primary node and streams them across the cluster fabric, avoiding independent network storage downloads by every single worker61.

Future Trajectories: Speculative Execution and the AI Operating System

The fundamental trajectory of AI runtimes is transitioning away from brute-force hardware scaling toward algorithmic efficiency and deeper operating system integration. The most profound advancement in algorithmic efficiency is speculative decoding10.

Overcoming the Autoregressive Bottleneck: Speculative Decoding

Autoregressive token generation is intrinsically serial; a model cannot predict the [Figure omitted from source export] token until it has generated the [Figure omitted from source export] token, leaving the massive parallel compute capacity of modern GPUs severely underutilized during the decode phase10. Speculative decoding mitigates this bottleneck by introducing intra-request parallelism10. The architecture relies on pairing two models that share an identical tokenizer and vocabulary: a small, highly efficient "draft" model and the massive "target" model10. The draft model rapidly generates a sequence of multiple candidate tokens ([Figure omitted from source export]). The target model then ingests this entire sequence and verifies it in a single, parallel forward pass10. If the target model agrees with the draft sequence, the tokens are accepted, effectively generating multiple tokens in the time it usually takes to generate one10. If there is a divergence, the target model rejects the sequence from that point forward and corrects the output10. The efficiency of this mechanism is governed by the acceptance rate ([Figure omitted from source export]). High acceptance rates dramatically reduce the number of memory-bound decode steps required by the target model, lowering the overall Cost Per Million Tokens (CPMT)7. AWS Neuron Distributed Inference natively supports multiple speculation modes, including Vanilla speculation, Fused speculation (where draft and target models are compiled together), EAGLE (where the draft model leverages hidden-state context from the target), and Medusa (where multiple small prediction heads run in parallel)10. Advanced runtimes like SGLang employ Adaptive Speculative Decoding, which dynamically adjusts the speculative length based on real-time metrics63. SGLang monitors the Jensen-Shannon (JS) distance between the draft and target probability distributions, as well as the draft-token entropy, to determine the optimal moment to halt speculation and trigger target verification, preventing wasted compute cycles63.

Speculative Thinking for Large Reasoning Models

As the industry pivots toward Large Reasoning Models (LRMs) that generate extensive "Chains of Thought" (CoT) prior to answering, a novel variant called Speculative Thinking (or SpecReason) is emerging65. Traditional speculative decoding demands strict, token-level equivalence between the draft and target models66. However, intermediate "thinking tokens" do not require absolute semantic precision; they merely serve to guide the model's logic65. SpecReason exploits this approximation tolerance by delegating reflective and reasoning steps to a smaller model65. The smaller model speculates the reasoning path based purely on semantic alignment, deferring to the massive base model only for final assessment and factual fallback66. This paradigm significantly accelerates inference—achieving 1.4 to 3.0x speedups—while simultaneously improving the logical accuracy of the output by reducing unnecessary backtracking inherent in massive models65.

The AI Operating System (AI OS)

Ultimately, the complex layers of memory management, distributed orchestration, hardware abstraction, and speculative execution are converging into the concept of the AI Operating System (AI OS)11. Just as traditional operating systems abstracted hardware complexity to allow software to run seamlessly, the AI OS abstracts distributed computational networks to orchestrate intelligent, autonomous workflows11. The AI OS architecture spans three distinct layers:

1. The Infrastructure Layer: Platforms like Red Hat AI OS and NVIDIA Dynamo act as the foundational deployment environment, leveraging technologies like Kubernetes, vLLM, and TensorRT-LLM to handle raw model execution, GPU scaling, and memory fragmentation11.
2. The Agent Orchestration Layer: Frameworks like CosmOS (by HPIQ) and AIOS represent a fundamental shift in application design11. Instead of a user interacting with a single LLM, the orchestration layer coordinates swarms of specialized AI agents via an internal "AI Bus," managing context statefulness, resolving API tool calls, and arbitrating communication between agents working collaboratively on a task11.
3. Domain-Specific Specialized OS: Vertically integrated, closed-stack environments like Tesla's Full Self-Driving (FSD) represent hyper-specialized AI operating systems designed exclusively for real-time edge inference, completely bypassing cloud connectivity to guarantee millisecond latency11.

Conclusion

The evolution of Artificial Intelligence runtimes represents the most critical engineering frontier in modern computing. As neural network architectures stabilize and homogenize, the true competitive differentiator for global enterprises lies entirely within the runtime optimization layer. The mechanisms detailed in this analysis illustrate a rapid transition away from simplistic, single-device execution toward highly sophisticated, orchestrated ecosystems. Deep learning compilers are aggressively optimizing computational graphs through horizontal and megakernel fusion, enabling edge devices to execute complex models within severely constrained thermal envelopes via static memory planning. In the data center, the virtualization of the KV cache through PagedAttention, the radical deduplication of context states via Radix Tree architectures, and the architectural disaggregation of the prefill and decode stages represent a paradigm shift in how computational memory is managed. Looking to the future, the integration of speculative decoding and speculative thinking promises to bypass the intrinsic sequential limitations of autoregressive generation, drastically lowering the cost per token. Furthermore, the dominance of proprietary hardware ecosystems will increasingly be challenged by open standards like SYCL and WebNN, facilitating a reality where AI inference is distributed dynamically—from heterogeneous data center clusters down to the user's local web browser. Ultimately, as artificial intelligence advances toward continuous, stateful reasoning and multi-agent collaboration, the AI runtime will complete its metamorphosis, serving as the foundational operating system of the cognitive era.

Works cited

1. What is AI inference? How it works and examples | Google Cloud, https://cloud.google.com/discover/what-is-ai-inference
2. What is AI Inference \- Arm, https://www.arm.com/glossary/ai-inference
3. AI inference vs. training: What is AI inference? \- Cloudflare, https://www.cloudflare.com/learning/ai/inference-vs-training/
4. https://www.akamai.com/glossary/what-is-ai-inferencing\#:\~:text=AI%20inference%20is%20the%20process,a%20prediction%20or%20a%20decision.
5. What is AI inference? \- Red Hat, https://www.redhat.com/en/topics/ai/what-is-ai-inference
6. Compiling AI Workloads for On-Device Inference on Heterogeneous Systems using MLIR | DVCon Proceedings, https://dvcon-proceedings.org/wp-content/uploads/134-Compiling-AI-Workloads-for-On-Device-Inference-on-Heterogeneous-Systems-using-MLIR.pdf
7. Heterogeneous AI – the next AI revolution \- DriveNets, https://drivenets.com/blog/heterogeneous-ai-the-next-ai-revolution/
8. NVIDIA Dynamo, A Low-Latency Distributed Inference Framework for Scaling Reasoning AI Models | NVIDIA Technical Blog, https://developer.nvidia.com/blog/introducing-nvidia-dynamo-a-low-latency-distributed-inference-framework-for-scaling-reasoning-ai-models/
9. Explore llama.cpp architecture and the inference workflow \- Arm Learning Paths, https://learn.arm.com/learning-paths/servers-and-cloud-computing/llama\_cpp\_streamline/2\_llama.cpp\_intro/
10. Accelerating decode-heavy LLM inference with speculative decoding on AWS Trainium and vLLM | Artificial Intelligence, https://aws.amazon.com/blogs/machine-learning/accelerating-decode-heavy-llm-inference-with-speculative-decoding-on-aws-trainium-and-vllm/
11. AI Operating Systems Explained: Types, Examples, and Use Cases \- Picovoice, https://picovoice.ai/blog/ai-operating-system/
12. The Critical Role of Compilers in Machine Learning and AI. | by Santhosraj \- Medium, https://medium.com/@santhosraj14/the-critical-role-of-compilers-in-machine-learning-and-ai-2d6851b9b9c5
13. Introduction to ML Compilers \+ Roadmap (MLIR, TVM, GPU Kernels) \- DEV Community, https://dev.to/aabhinavg/introduction-to-ml-compilers-roadmap-mlir-tvm-gpu-kernels-24hb
14. Deep Learning Compiler Optimization Techniques \- Aussie AI, https://www.aussieai.com/research/compilers
15. 1 Introduction \- arXiv, https://arxiv.org/html/2604.13327v2
16. High-level Architecture and Components of ExecuTorch \- PyTorch documentation, https://docs.pytorch.org/executorch/0.4/getting-started-architecture.html
17. sd-scripts/docs/anima\_torch\_compile.md at main \- GitHub, https://github.com/kohya-ss/sd-scripts/blob/main/docs/anima\_torch\_compile.md
18. Demystifying the Silence of Correctness Bugs in PyTorch Compiler \- arXiv, https://arxiv.org/html/2604.08720v1
19. torch.compile and CUDA Graphs for LLM Inference: Production PyTorch 2.6 Guide (2026), https://www.spheron.network/blog/torch-compile-cuda-graphs-llm-inference-pytorch-2-6/
20. vLLM vs SGLang vs TensorRT-LLM \- Inference Engineering, https://inferenceengineering.tech/learn/vllm-vs-sglang-vs-tensorrt-llm/
21. vLLM, SGLang, or TensorRT-LLM? Picking an LLM Serving Stack | Jarvis Labs Blog, https://jarvislabs.ai/blog/vllm-sglang-trtllm-comparison
22. vLLM vs TensorRT-LLM vs SGLang: H100 Benchmarks (2026) | Spheron Blog, https://www.spheron.network/blog/vllm-vs-tensorrt-llm-vs-sglang-benchmarks/
23. RadixAttention \- SGLang, https://sgl-project-sglang-93.mintlify.app/concepts/radix-attention
24. I Served the Same Model on vLLM, SGLang, and TensorRT-LLM — the Default Gives Up 29% | Towards AI, https://towardsai.net/p/machine-learning/i-served-the-same-model-on-vllm-sglang-and-tensorrt-llm-the-default-gives-up-29
25. vLLM vs SGLang vs TensorRT-LLM vs Ollama: Choosing an Inference Engine in 2026, https://leetllm.com/blog/llm-inference-engine-comparison-2026
26. GitHub \- sgl-project/sglang: SGLang is a high-performance serving framework for large language models and multimodal models., https://github.com/sgl-project/sglang
27. What is the SGlang Inference Engine, and How Does it Stack Up?, https://inference.net/content/sglang/
28. RadixAttention Explained: How SGLang Beats PagedAttention at Scale \- Rajat Pandit, https://rajatpandit.com/ai-engineering/radixattention-vs-pagedattention/
29. Heterogeneous AI Technologies Enable Scalable, Efficient AI Systems \- Arm, https://www.arm.com/markets/artificial-intelligence/technologies
30. What is heterogeneous compute? \- Arm, https://www.arm.com/glossary/heterogeneous-compute
31. What is Heterogeneous Computing? \- Supermicro, https://www.supermicro.com/en/glossary/heterogeneous-computing
32. mlc-llm \- Codesandbox, https://codesandbox.io/p/github/BRILLIANT-ESYSTEMS-LIMITED/mlc-llm
33. Llama.cpp \- Run LLM Inference in C/C++, https://llama-cpp.com/
34. llama.cpp: Introduction, https://ggml-org-llama-cpp.mintlify.app/introduction
35. ggml-org/llama.cpp: LLM inference in C/C++ \- GitHub, https://github.com/ggml-org/llama.cpp
36. MLC LLM: A Deployment Engine for ML Compilation \- Cordatus Blog, https://blog.cordatus.ai/featured-articles/mlc-llm-deployment-engine/
37. MLC LLM: Universal Large-language Model Deployment with ML Compilation \- Hao AI Lab @ UCSD, https://haoailab.com/cse234-w25/assets/slides/feb6.pdf
38. oneAPI: A New Era of Heterogeneous Computing \- Intel, https://www.intel.com/content/www/us/en/developer/tools/oneapi/overview.html
39. Experiences Building an MLIR-based SYCL Compiler \- arXiv, https://arxiv.org/pdf/2312.13170
40. SYCL \- C++ Single-source Heterogeneous Programming for Acceleration Offload \- The Khronos Group, https://www.khronos.org/sycl/
41. The Rapidly Evolving Intel® Software Developer Ecosystem, https://www.intel.com/content/www/us/en/developer/articles/technical/rapidly-evolving-software-developer-ecosystem.html
42. CHARM-SYCL & IRIS: A Toolchain for Performance Portability on Extremely Heterogeneous Systems \- OSTI, https://www.osti.gov/servlets/purl/2480028
43. 1 Introduction \- arXiv, https://arxiv.org/html/2605.08195v1
44. Edge AI \- Microchip Technology, https://www.microchip.com/en-us/solutions/technologies/machine-learning
45. ExecuTorch \-- A Unified PyTorch Solution to Run AI Models On-Device \- arXiv, https://arxiv.org/pdf/2605.08195
46. Quick Start Pathway — ExecuTorch 1.3 documentation, https://docs.pytorch.org/executorch/stable/pathway-quickstart.html
47. Optimizing ExecuTorch on Intel AI PCs with OpenVINO™ Backend, https://www.intel.com/content/www/us/en/developer/articles/community/optimizing-executorch-on-ai-pcs.html
48. Qualcomm QNN Export for Ultralytics YOLO Models, https://docs.ultralytics.com/integrations/qnn
49. 5 Best Edge Computing Platforms in 2026: Full Breakdown \- Portainer, https://www.portainer.io/blog/edge-computing-platforms
50. Generative AI | 2025 | The Web Almanac by HTTP Archive, https://almanac.httparchive.org/en/2025/generative-ai
51. AI at TPAC 2025 | 2025 | Blog \- W3C, https://www.w3.org/blog/2025/ai-at-tpac-2025/
52. ONNX Runtime Web unleashes generative AI in the browser using WebGPU, https://opensource.microsoft.com/blog/2024/02/29/onnx-runtime-web-unleashes-generative-ai-in-the-browser-using-webgpu/
53. WebAssembly and WebGPU enhancements for faster Web AI, part 1 | Blog, https://developer.chrome.com/blog/io24-webassembly-webgpu-1
54. Unlock the Potential of AI and Immersive Web Applications with WebGPU \- Intel, https://www.intel.com/content/www/us/en/developer/articles/technical/unlock-potential-ai-immersive-web-apps-with-webgpu.html
55. Web Neural Network API \- W3C, https://www.w3.org/TR/webnn/
56. WebNN Overview | Microsoft Learn, https://learn.microsoft.com/en-us/windows/ai/directml/webnn-overview
57. WebNN · Web Neural Network API \- Browser Compatibility, https://webnn.io/en/api-reference/browser-compatibility/api
58. The Agentic Browser Landscape in 2026: A Complete Guide \- No Hacks, https://nohacks.co/blog/agentic-browser-landscape-2026
59. Exploring Thermal-Aware Heterogeneous Compute Orchestration Concepts \- DGX Systems (Data Center) \- NVIDIA Developer Forums, https://forums.developer.nvidia.com/t/exploring-thermal-aware-heterogeneous-compute-orchestration-concepts/370965
60. Scale and Serve Generative AI | NVIDIA Dynamo, https://www.nvidia.com/en-us/ai/dynamo/
61. NVIDIA Dynamo: The Missing Layer for Scaling Generative AI Inference \- Medium, https://medium.com/@aman.kohli1/nvidia-dynamo-the-missing-layer-for-scaling-generative-ai-inference-2a5b8f557045
62. How NVIDIA Dynamo 1.0 Powers Multi-Node Inference at Production Scale, https://developer.nvidia.com/blog/nvidia-dynamo-1-production-ready/
63. Speculative decoding | LLM Inference Handbook \- BentoML, https://bentoml.com/llm/inference-optimization/speculative-decoding
64. Efficient LLM System with Speculative Decoding by Xiaoxuan Liu A dissertation submitted in partial satisfaction of the requireme \- EECS, https://www2.eecs.berkeley.edu/Pubs/TechRpts/2025/Archive/EECS-2025-224.pdf
65. Speculative Thinking: Enhancing Small-Model Reasoning with Large Model Guidance at Inference Time \- arXiv, https://arxiv.org/html/2504.12329v2
66. SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoning \- NIPS, https://papers.neurips.cc/paper\_files/paper/2025/file/12c45a68e8433b21b91cd47731387fa4-Paper-Conference.pdf