The Definitive AI Runtime Architecture Guide: A Content Expansion Blueprint for ARuntime.com

Full report

On this page

The transition from localized experimental artificial intelligence models to planetary-scale, multi-agent enterprise deployments has exposed profound limitations in legacy software infrastructure. To position ARuntime.com as the industry's premier knowledge hub and technical resource for machine learning infrastructure, the platform must host an exhaustive, rigorously detailed repository of contemporary AI runtime architectures. The modern AI runtime is no longer merely an execution script; it is a highly specialized, disaggregated, hardware-aware orchestration stack spanning kernel compilation, distributed memory management, agentic routing, and edge deployment. The following report provides the expanded, high-density technical content required to elevate ARuntime.com into the definitive reference architecture guide for LLM inference, edge deployment, and agentic orchestration in 2025 and 2026\.

Section 1: Standardizing Inference Telemetry and MLPerf Benchmarking

For ARuntime.com to serve as an authoritative resource, it must standardize the operational vocabulary used by infrastructure engineers. Traditional metrics like floating-point operations per second (FLOPS) are insufficient for evaluating the user experience of autoregressive large language model (LLM) generation1. The industry has definitively shifted toward latency-bounded evaluation, driven heavily by standards bodies such as MLCommons1. ARuntime.com must structure its benchmarking resources around these exact latency profiles.

The Granular Latency Profile

The lifecycle of an LLM request dictates the user experience. This lifecycle is fundamentally divided into a parallel compute-bound "prefill" phase (processing the input prompt and building the Key-Value cache) and a sequential, memory-bandwidth-bound "decode" phase (generating output tokens)1. Consequently, ARuntime.com's telemetry guidelines must mandate the measurement of the following precise metrics:

The mathematical distinction between TPOT and ITL is critical for engineers analyzing server loads. For a single request, TPOT is calculated as the End-to-End Latency minus TTFT, divided by the total output tokens minus one: [Figure omitted from source export]

1. When evaluating system-wide performance, Average TPOT is request-weighted, treating all requests equally regardless of output length, while Average ITL is token-weighted, giving heavier influence to longer generations5.

The Ascendancy of Goodput and MLPerf Standards

Raw throughput—measured in requests per second (RPS) or tokens per second (TPS)—is a flawed metric because it can be artificially inflated by an engine accepting requests that subsequently experience massive tail latencies7. ARuntime.com's content must champion the concept of Goodput, defined as the percentage of requests that successfully complete while adhering to strict Service Level Objectives (SLOs) for TTFT and TPOT5. [Figure omitted from source export] 5. This metric forms the core of modern evaluation suites like the MLPerf Inference benchmark, developed by MLCommons2. As MLPerf evolved through versions 4.0, 5.0, and the current 6.x era, it shifted focus from raw compute benchmarks to latency-aware LLM benchmarking1. MLPerf now enforces strict TTFT and Time Per Output Token (TPOT) constraints for interactive server scenarios, utilizing autoregressive models like Llama 2 70B and sophisticated reasoning models like DeepSeek-R11. Furthermore, MLPerf Client tests these models on client-class hardware, including Intel Core Ultra architectures, Snapdragon X Elite platforms, and AMD Ryzen AI processors, reflecting the push toward localized edge deployment2. By anchoring its evaluations in Goodput and MLPerf standards, ARuntime.com will provide industry-standard telemetry guidance that goes beyond misleading average TPS figures.

Section 2: Disaggregated Prefill and Decode Orchestration

A core technical pillar for ARuntime.com must be the architectural paradigm of Disaggregated Prefill and Decode (PD-Disaggregation). As autoregressive LLMs became the dominant workload, engineers discovered a structural conflict: the highly parallel, compute-heavy prefill phase severely interferes with the memory-bound, sequential decode phase when both execute on the same GPU1. This phenomenon, known as prefill interference, causes massive ITL jitter because a 4,000-token prompt prefill will monopolize the GPU compute cores, stalling the decode steps of all other concurrent user requests4.

Phase Splitting Architecture

Advanced systems address this by decoupling the phases onto specialized hardware clusters4. In this model, an incoming user request is routed exclusively to a prefill worker4. This worker processes the prompt, calculates the Key-Value (KV) cache, and transmits the state directly to an available decode worker4. The decode worker, optimized for sequential memory retrieval rather than raw dense compute, then streams the response to the user4. This decoupling prevents prefill requests from blocking decode loops, enables independent hardware scaling, and routinely yields a 2x to 7x gain in usable throughput compared to monolithic architectures4.

Interconnects and Transfer Mechanisms

The vulnerability of PD-Disaggregation is the physical transfer of the KV cache, which scales linearly with sequence length and batch size11. Transferring gigabytes of KV tensors over standard networks introduces latency that negates the benefits of disaggregation4. ARuntime.com should detail cutting-edge solutions to this bottleneck, such as the Mooncake serving platform built by Moonshot AI4. Mooncake employs a KV-centric architecture featuring a Transfer Engine that relies on Remote Direct Memory Access (RDMA) for zero-copy data transfer10. This engine integrates with the vLLM and SGLang routing layers, enabling the transfer of massive multimodal embeddings and KV states across nodes virtually instantaneously10. Furthermore, NVIDIA's NIXL library and Dynamo configurations are widely utilized for intra-rack KV cache transfers over NVLink4. Alibaba Cloud's Role-Based Generative (RBG) orchestration abstracts these topologies further by defining the router, prefill node, and decode node as distinct Kubernetes orchestration units, allowing automated scaling and deep observability of the disaggregated pipeline15. ARuntime.com's architecture guides must stress that PD-Disaggregation is strictly for high-throughput production environments with shared memory architectures or high-speed interconnects (InfiniBand, NVLink, or 100GbE minimum); it is counterproductive for isolated development testing or small batch scenarios4.

Section 3: Advanced Key-Value (KV) Cache Management

Because the KV cache prevents redundant attention calculations for previously generated tokens, its management is the primary determinant of LLM memory efficiency11. The ARuntime.com platform must provide exhaustive documentation on how modern inference engines avoid memory fragmentation and enable semantic cache sharing.

PagedAttention versus RadixAttention

Naive inference engines allocate massive, contiguous blocks of High Bandwidth Memory (HBM) corresponding to the absolute maximum context length of a model12. If a user submits a minimal prompt, the unutilized memory block remains locked, resulting in severe internal fragmentation and artificially capping batch concurrency16. PagedAttention, popularized by the vLLM framework, revolutionized memory management by mapping operating system virtual memory paradigms onto the KV cache11. The cache is partitioned into small, fixed-size physical blocks (e.g., 16 tokens per block) managed via a block table12. As a sequence generates tokens, new pages are allocated dynamically on-demand, virtually eliminating internal memory waste and permitting a 2x to 4x increase in effective batch sizes12. However, PagedAttention is optimized for linear, single-shot request execution16. For complex, multi-agent workflows or deeply branched conversations, RadixAttention—introduced by the SGLang framework—represents a superior paradigm11. RadixAttention does not treat the cache as a flat list; instead, it structures the global KV cache as a radix tree (a compressed prefix tree) keyed mathematically by the exact token sequences11. When an application features a massive shared system prompt used by multiple autonomous agents, RadixAttention computes the KV cache for that prompt exactly once, anchoring it at the root of the tree14. As individual agents generate unique conversational branches, the system forks the tree, allocating new physical memory solely for the newly generated tokens while maintaining zero-cost memory sharing for the shared root12. This node-level structure, combined with a Least Recently Used (LRU) node eviction policy, routinely delivers 75% to 95% cache hit rates in multi-turn environments, reducing TTFT to near-zero for overlapping contexts12.

Tiered KV Offloading and Peer-to-Peer Sharing

To further optimize costs, enterprise infrastructures no longer treat the KV cache as ephemeral GPU data19. Frameworks like LMCache act as a caching middleware that extracts KV chunks and offloads them to a tiered storage hierarchy spanning CPU RAM, local NVMe storage, and remote object storage13. If multiple vLLM instances operate behind a load balancer, standard deployments create isolated "cache silos," where each instance must independently compute the KV cache for identical prompts, resulting in massive RAM contention and duplicated computation21. LMCache resolves this by implementing multi-node Peer-to-Peer (P2P) CPU memory sharing19. A centralized controller manages metadata detailing the exact locations of KV chunks across the cluster21. Utilizing highly refined, non-blocking lock mechanisms, active workers can seamlessly query and retrieve cached KV states from the CPU memory of sibling instances without triggering expensive OS context switches21. For immense datasets, LMCache integrates directly with enterprise storage engines. Dell's AI Storage architecture, for example, utilizes LMCache alongside NVIDIA's NIXL library to offload KV caches directly to Dell PowerScale file storage using NFS-over-RDMA, or to ObjectScale via S3-compatible interfaces13. This allows the persistent storage of AI-native context, accelerating TTFT by up to 19x for large document analysis and multi-round QA by entirely bypassing prefill recomputation13. ARuntime.com should highlight tiered KV storage as a mandatory architecture for RAG and agentic orchestration13.

Section 4: Algorithmic Acceleration via Speculative Decoding

Because autoregressive decoding requires loading the entire model weight matrix from HBM to compute the next token, the GPU's compute cores are chronically underutilized1. ARuntime.com must feature a dedicated module on Speculative Decoding, a technique that transforms sequential generation into a highly efficient parallel verification task23. In speculative decoding, a fast, computationally inexpensive "draft model" autoregressively predicts a sequence of upcoming tokens23. These candidate tokens are submitted as a single batch to the massive "target model." Because transformers calculate self-attention for all tokens in a sequence simultaneously during a forward pass, the target model can verify the probabilities of all drafted tokens in parallel23. Tokens are accepted based on a probability threshold [Figure omitted from source export], defined as: [Figure omitted from source export]

1. Upon encountering the first rejected token, the speculation chain breaks, remaining drafts are discarded, and a bonus token is sampled from a modified residual distribution: [Figure omitted from source export]23. This mathematical guarantee ensures the final output exactly mirrors the target model's original quality distribution without degradation23.

Primary Speculative Architectures in Production

ARuntime.com should categorize modern speculative deployments into the following primary architectural paradigms24:

Section 5: Hardware-Aware Execution, Kernels, and Precision

The software instructions governing AI operations must be hyper-optimized for the physical layout of the target GPU27. ARuntime.com must dissect the intricacies of kernel authoring and execution runtimes to provide value to infrastructure engineers.

FlashAttention-3 and Hopper Optimization

Standard attention mechanisms exhibit a quadratic [Figure omitted from source export] memory footprint scaling with sequence length12. FlashAttention revolutionized transformer efficiency by tiling the Key, Query, and Value matrices, fusing the operations, and retaining intermediate calculations in ultra-fast on-chip SRAM rather than writing them to slow global HBM29. FlashAttention-3 (FA3) takes this further by deeply exploiting the asynchronous capabilities of the NVIDIA Hopper (H100/H200) architecture29. FA3 utilizes the Tensor Memory Accelerator (TMA) to handle global-to-shared memory data transfers entirely asynchronously, removing the need for index calculation and freeing hardware registers for larger computational tile sizes30. Furthermore, FA3 implements warp specialization29. Thread warps are distinctly bifurcated: "producer warps" fetch data tiles utilizing Hopper's Warpgroup Matrix Multiply-Accumulate (WGMMA) instructions, while "consumer warps" concurrently execute the GEMM (General Matrix Multiply) and Softmax routines on the SRAM tiles29. Aping-pong scheduling system between two SRAM buffers enables total overlap of data movement and computation29. This incoherent processing allows the H100 to achieve up to 740 TFLOPS (75% theoretical maximum utilization) with FP16, massively reducing TTFT on contexts exceeding 8K tokens30.

Extreme Precision Quantization (FP8 and FP4)

To alleviate the HBM memory bandwidth wall, the industry is transitioning to extreme low-bit quantization28. ARuntime.com should extensively document the shift from 16-bit floats to 8-bit and 4-bit precision formats. Frameworks like Neural Magic's LLM Compressor are critical for this pipeline33. By employing data-free post-training quantization techniques like model\_free\_ptq, models can be rapidly converted to INT8, FP8, or MXFP4 without requiring extensive calibration datasets34. Advanced algorithms, such as Intel's AutoRound, introduce trainable parameters to optimize rounding decisions and clipping ranges, utilizing block-wise output reconstruction errors to drastically reduce the accuracy degradation traditionally associated with extreme quantization33. This precision shift directly impacts hardware economics. NVIDIA's Blackwell architecture features native 4-bit (NVFP4) inference support35. Deploying NVFP4 quantization provides a 1.6x throughput improvement over BF16 with a 41% reduction in energy consumption, while sacrificing only 2% to 4% in model quality35. Additionally, FA3 allows the execution of the [Figure omitted from source export] and [Figure omitted from source export] matrix multiplications directly in FP8, utilizing per-tensor scaling headers to maintain numerical stability prior to upcasting the final output back to BF1629.

The Engine War: TensorRT-LLM vs. Inductor

The runtime orchestrating these kernels is equally critical27. General-purpose serving engines like vLLM utilize a Python-based scheduler to drive a standard PyTorch forward pass, utilizing NCCL for tensor parallelism across GPUs37. This offers immense flexibility and broad model support37. In contrast, NVIDIA's TensorRT-LLM abandons Python scheduling in favor of a rigid, highly optimized C++ compiled engine path37. The TensorRT build process aggressive applies operator fusion (combining attention, layer norm, and linear layers into monolithic kernels) and performs hardware-specific auto-tuning, benchmarking numerous kernel variants on the host GPU to select the optimum implementation37. This compiled execution delivers 15% to 25% higher raw throughput on NVIDIA hardware and the most energy-efficient token generation profile available37. However, this demands severe engineering overhead; any change to batch size, sequence length, or quantization forces a multi-hour engine recompilation37. To counter this, the PyTorch ecosystem relies on torch.compile, a Just-In-Time (JIT) compilation stack39. TorchDynamo captures the Python execution graph, AOTAutograd generates the forward and backward passes, and TorchInductor lowers the graph into ATen operators, producing highly optimized Triton kernels39. The AOTInductor allows these graphs to be exported as static shared libraries, providing C++ level performance while remaining natively within the PyTorch ecosystem41. This stack is particularly crucial for non-NVIDIA hardware, serving as the bridge for AWS Trainium (via TorchNeuron) and AMD MI300X (via ROCm Composable Kernels)42.

Section 6: Expanding Runtimes to the Edge and Browser

ARuntime.com must dedicate substantial coverage to localized execution. The requirements of data privacy, zero latency, and offline availability are pushing LLMs out of the data center and onto mobile devices and browsers44.

ExecuTorch: The Universal Mobile Framework

Historically, deploying PyTorch models to varied edge environments required complex conversions into proprietary formats like Apple's CoreML or Qualcomm's SNPE, fragmenting the deployment landscape46. ExecuTorch was developed as PyTorch's native, unified edge deployment solution44. Through ahead-of-time (AOT) graph-level compilation, ExecuTorch converts full PyTorch models into a portable .pte format, executing on a C++ runtime with a base footprint of merely 50KB46. Most significantly, ExecuTorch introduces backend delegation46. The runtime can dynamically partition a model, assigning massive matrix multiplications to a dedicated NPU (via Qualcomm Hexagon or Arm Ethos-U delegates), routing image processing to the mobile GPU (via Vulkan), and executing fallback logic on the CPU using the highly optimized XNNPACK and Arm Kleidi libraries44. This allows researchers to utilize standard 4-bit group-wise weight quantization and sliding-window attention directly on-device without leaving the PyTorch ecosystem46.

WebGPU, WebNN, and In-Browser Inference

Simultaneously, the web browser is evolving into a high-performance AI execution environment45. Legacy web architecture mandated that AI requests be sent to a heavy cloud backend, incurring round-trip latency and massive token billing costs45. The introduction of WebGPU allows JavaScript to access local hardware graphics compute directly, bypassing the limitations of WebGL and providing a 10x speedup for transformer architectures45. Concurrently, WebNN (Neural Network API) interfaces directly with native hardware accelerators, tapping into DirectML on Windows or CoreML on macOS directly through the browser45. Using optimized runtime formats like WebLLM, MLC LLM, and ONNX Runtime Web, developers can now run 3-billion parameter models (such as Llama, Phi, or Gemma-3n E4B) entirely offline in the browser at 20 to 40 tokens per second45. By leveraging Web Workers and SharedArrayBuffers, these models stream tokens seamlessly without freezing the main application UI thread, unlocking fully local, privacy-compliant AI for healthcare applications, multi-player shared knowledge bases, and Google Docs-style offline productivity suites45.

Section 7: Agentic Orchestration and The Semantic Layer

The final layer of the modern AI runtime is semantic orchestration. As LLMs evolve into autonomous agents capable of tool utilization, the runtime must manage state, context, and enterprise data integrity50.

The Model Context Protocol (MCP)

To solve the "N×M integration problem"—where every AI agent must build custom connectors for every external API—Anthropic introduced the Model Context Protocol (MCP)50. ARuntime.com should present MCP as the fundamental interoperability standard for modern agentic workflows52. MCP architecture separates the decision-making LLM (the MCP Client) from the stateless, capability-providing backend (the MCP Server)50. The protocol relies on three pillars:

1. Tool-Based Discovery: The MCP Server exposes highly specific semantic tools imbued with business logic and validation, moving beyond raw REST API endpoints50.
2. Persistent Context Preservation: MCP inherently maintains conversational and tool state across interactions, allowing multi-step reasoning agents to iterate without manually rebuilding the context array for every step50.
3. Streaming Communication: Because database retrieval or vector embeddings can take several minutes, MCP utilizes Server-Sent Events (SSE) to stream intermediate progress updates back to the client without polling, preventing destructive timeouts50.

To mitigate the massive security vulnerabilities created by granting autonomous agents access to enterprise APIs, architectures must incorporate proxy layers like MCP Guardian53. These proxies sit transparently between the MCP Client and Server, monitoring real-time telemetry, enforcing access controls, and preventing arbitrary code execution attacks without requiring modifications to the LLM or the tool server54.

The Enterprise Semantic Layer

Finally, ARuntime.com must stress that agentic AI cannot function accurately atop raw databases55. If an LLM attempts to generate SQL directly against a data lake, it inevitably hallucinates logic, leading to catastrophic business errors55. Industry leaders—including Uber, Netflix, and Airbnb—mitigate this by orchestrating AI exclusively through a Semantic Layer51. A Semantic Layer, such as Uber's uMetric or Netflix's Python-based Metrics Repo, centralizes the absolute definition of business metrics, establishing a rigid "Define Once, Compute Once" architecture51. The system relies on centralized Feature Stores to ensure that the exact same data computation logic is used during model training, A/B testing, and real-time agentic serving51. By enforcing this structure, companies eliminate "training-serving skew"—the silent degradation of model performance caused by disparate feature computation51. In a mature, Level-6 semantic architecture, the AI agent (via protocols like MCP) does not query raw tables; it queries the Semantic Layer utilizing predefined metadata aliases stored in vector databases like OpenSearch55. This grounds the LLM strictly in enterprise truth, providing the ultimate control plane for safe, accurate, and scalable agentic execution55.

Strategic Conclusion

For ARuntime.com to dominate the technical discourse, its content strategy must comprehensively integrate the layers detailed in this report. From the foundational mathematics of TTFT latency and FlashAttention-3 kernels to the macro-orchestration of Disaggregated Prefill pipelines and MCP semantic routing, the modern AI runtime is an interconnected, full-stack discipline. By structuring its knowledge base around these highly specific, deeply technical realities, ARuntime.com will serve as the indispensable guide for the next generation of artificial intelligence infrastructure engineering.

Works cited

1. MLPerf and the rise of latency-aware LLM benchmarking \- EDN Magazine, https://www.edn.com/mlperf-and-the-rise-of-latency-aware-llm-benchmarking/
2. MLPerf Client Benchmark \- MLCommons, https://mlcommons.org/benchmarks/client/
3. Industry-Standard Benchmarks for ML & LLMs — what's current in 2025 \- Medium, https://medium.com/@jagatpm/industry-standard-benchmarks-for-ml-llms-whats-current-in-2025-1444c6cd72a2
4. Disaggregated Prefill-Decode: The Architecture Behind Meta's LLM Serving \- Jarvis Labs, https://jarvislabs.ai/blog/llm-optimization-disaggregated-prefill-decode
5. Understand LLM latency and throughput metrics \- Anyscale Docs, https://docs.anyscale.com/llm/serving/benchmarking/metrics
6. Key metrics for LLM inference \- BentoML, https://bentoml.com/llm/llm-inference-basics/llm-inference-metrics
7. Benchmarking LLM Inference: The Metrics That Actually Matter | roeybc \- Roey Ben Chaim, https://roeybc.com/blog/llm\_inference\_benchmark
8. Understanding LLM Performance, https://cseweb.ucsd.edu/\~yiying/cse291a-fall25/reading/llm-perf.pdf
9. Benchmark Work | Benchmarks MLCommons, https://mlcommons.org/benchmarks/
10. Welcome to Mooncake, https://kvcache-ai.github.io/Mooncake/
11. LLM Inference Optimization \- Jonas Geiping, https://jonasgeiping.github.io/teaching/llm-inference-seminar/
12. KV Cache vs Radix Attention vs Page Attention? | by rahularyansharma \- Medium, https://medium.com/@rahularyan786/kv-cache-vs-radix-attention-vs-page-attention-941ff222be2e
13. Scaling Multi-Turn LLM Inference with KV Cache Storage Offload and Dell RDMA-Accelerated Architecture, https://infohub.delltechnologies.com/p/scaling-multi-turn-llm-inference-with-kv-cache-storage-offload-and-dell-rdma-accelerated-architecture/
14. SGLang Production Deployment Guide: RadixAttention and Multi-Turn Inference on GPU Cloud (2026) | Spheron Blog, https://www.spheron.network/blog/sglang-production-deployment-guide/
15. Removing the Guesswork from Disaggregated Serving | NVIDIA Technical Blog, https://developer.nvidia.com/blog/removing-the-guesswork-from-disaggregated-serving/
16. RadixAttention Explained: How SGLang Beats PagedAttention at Scale \- Rajat Pandit, https://rajatpandit.com/ai-engineering/radixattention-vs-pagedattention/
17. PagedAttention vs Continuous Batching vs vLLM vs SGLang — A Practical Breakdown | by varun rao | Python in Plain English, https://python.plainenglish.io/pagedattention-vs-continuous-batching-vs-vllm-vs-sglang-a-practical-breakdown-4c19cc9e21c0
18. Benchmark: Radix vs. PagedAttention Scaling (SGLang / vLLM) · Issue \#37730 \- GitHub, https://github.com/vllm-project/vllm/issues/37730
19. LMCache: Supercharge Your LLM with the Fastest KV Cache Layer \- GitHub, https://github.com/lmcache/lmcache
20. Integration \- LMCache, https://docs.lmcache.ai/developer\_guide/integration.html
21. LMCache Multi-node P2P CPU Memory Sharing & Control: From Experimental Feature to Production, https://blog.lmcache.ai/en/2026/01/21/p2p-1/
22. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference \- arXiv, https://arxiv.org/html/2510.09665v2
23. Speculative Speculative Decoding \- arXiv, https://arxiv.org/html/2603.03251v3
24. PARD: Accelerating LLM Inference with Low‑Cost PARallel Draft Model Adaptation, https://openreview.net/forum?id=XbOyv7iVGL¬eId=ivFc2YHjZS
25. Speculative Decoding 2026: 2.8x Faster LLM Inference for 出海 | SyncSoft.AI, https://www.syncsoft.ai/en/blog/speculative-decoding-eagle3-medusa-deepseek-mtp-chinese-chuhai-2026
26. \#speculative-decoding | vLLM Blog, https://vllm.ai/blog/tags/speculative-decoding
27. The GPU Inference Stack: TensorRT, vLLM, Triton, and ONNX Runtime Compared \- Medium, https://medium.com/@sharatonline/the-gpu-inference-stack-tensorrt-vllm-triton-and-onnx-runtime-compared-54259e4a8dd5
28. Open-source LLM training is a mess. Here is how it all works. \- Baseten, https://www.baseten.co/blog/open-source-llm-training-is-a-mess-here-is-how-it-all-works/
29. FlashAttention 2 vs FlashAttention 3: H100 and H200 Speedups, FP8 Support, and Migration Guide (2026) | Spheron Blog, https://www.spheron.network/blog/flashattention-2-vs-flashattention-3-h100-h200-guide/
30. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision \- PyTorch, https://pytorch.org/blog/flashattention-3/
31. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision \- arXiv, https://arxiv.org/html/2407.08608v1
32. A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithms \- arXiv, https://arxiv.org/html/2409.16694v3
33. vllm-project/llm-compressor: Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM \- GitHub, https://github.com/vllm-project/llm-compressor
34. LLM Compressor 0.9.0: Attention quantization, MXFP4 support, and more, https://developers.redhat.com/articles/2026/01/16/llm-compressor-090-attention-quantization-mxfp4-support-and-more
35. Private LLM Inference on Consumer Blackwell GPUs: A Practical Guide for Cost-Effective Local Deployment in SMEs \- arXiv, https://arxiv.org/html/2601.09527v1
36. (PDF) Private LLM Inference on Consumer Blackwell GPUs: A Practical Guide for Cost-Effective Local Deployment in SMEs \- ResearchGate, https://www.researchgate.net/publication/399776363\_Private\_LLM\_Inference\_on\_Consumer\_Blackwell\_GPUs\_A\_Practical\_Guide\_for\_Cost-Effective\_Local\_Deployment\_in\_SMEs
37. vLLM vs SGLang vs TensorRT-LLM \- Inference Engineering, https://inferenceengineering.tech/learn/vllm-vs-sglang-vs-tensorrt-llm/
38. Determining Energy Efficiency Sweet Spots in Production LLM Inference \- arXiv, https://arxiv.org/html/2602.05695v1
39. Compiling Deepchem Torch Models, https://deepchem.io/tutorials/compiling-deepchem-torch-models/
40. Introduction to torch.compile — PyTorch Tutorials 2.12.0+cu130 documentation, https://docs.pytorch.org/tutorials/intermediate/torch\_compile\_tutorial.html
41. AOTInductor: Ahead-Of-Time Compilation for Torch.Export-ed Models, https://docs.pytorch.org/docs/stable/user\_guide/torch\_compiler/torch.compiler\_aot\_inductor.html
42. State of PyTorch Hardware Acceleration 2025, https://tunguz.github.io/PyTorch\_Hardware\_2025/
43. Native PyTorch for AWS Trainium — AWS Neuron Documentation, https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/pytorch-native-overview.html
44. Introducing ExecuTorch 1.0: Powering the next generation of edge AI \- PyTorch, https://pytorch.org/blog/introducing-executorch-1-0/
45. JavaScript for Real-Time AI: On-Device LLMs with WebGPU \+ WebNN | by Karuna \- Medium, https://medium.com/codetodeploy/javascript-for-real-time-ai-on-device-llms-with-webgpu-webnn-0daaaea2a2fb
46. ExecuTorch \-- A Unified PyTorch Solution to Run AI Models On-Device \- arXiv, https://arxiv.org/pdf/2605.08195
47. GitHub \- pytorch/executorch: On-device AI across mobile, embedded and edge for PyTorch, https://github.com/pytorch/executorch
48. Web Bro: Full in-browser AI agent with WebGPU, ONNX, and direct File System Access API integration : r/javascript \- Reddit, https://www.reddit.com/r/javascript/comments/1sq9ild/web\_bro\_full\_inbrowser\_ai\_agent\_with\_webgpu\_onnx/
49. LLM Inference guide for Web | Google AI Edge, https://developers.google.com/edge/mediapipe/solutions/genai/llm\_inference/web\_js
50. Model Context Protocol architecture patterns for multi-agent AI systems \- IBM Developer, https://developer.ibm.com/articles/mcp-architecture-patterns-ai-systems/
51. How Netflix, Uber, and Google Build AI Systems: Architecture Deep Dive \- DEV Community, https://dev.to/matt\_frank\_usa/how-netflix-uber-and-google-build-ai-systems-architecture-deep-dive-17g5
52. What is the Model Context Protocol (MCP)? \- Databricks, https://www.databricks.com/blog/what-is-model-context-protocol
53. MCP: Model Context Pitfalls in an Agentic World \- HiddenLayer, https://www.hiddenlayer.com/research/mcp-model-context-pitfalls-in-an-agentic-world
54. MCP Guardian: Essential Security for Agentic Tool Use \- EQTY Lab, https://www.eqtylab.io/blog/securing-model-context-protocol
55. Secrets of the Semantic Layer in Big Tech: How Uber, Netflix, and Airbnb Manage Metrics — and How You Can Apply It in Your Company | by Sergey Gromov | Towards AI, https://pub.towardsai.net/secrets-of-the-semantic-layer-in-big-tech-how-uber-netflix-and-airbnb-manage-metrics-1b9f7680ac25
56. Real-world gen AI use cases from the world's leading organizations | Google Cloud Blog, https://cloud.google.com/transform/101-real-world-generative-ai-use-cases-from-industry-leaders
57. Real-World AI System Architecture: Case Studies from Netflix, Uber, and More \- InfraSketch, https://infrasketch.net/blog/real-world-ai-system-architecture