Skip to content

Edge AI

Build useful AI within device budgets, from efficient models and inference to multimodal systems, agents and product design.

Updated View as Markdown

Edge AI brings intelligence close to where it is used: a phone, a laptop, a camera, an embedded system or a small sensor. The challenge is not simply to make a large model smaller. It is to deliver a useful capability within the device’s memory, compute, responsiveness and energy limits.

Figure 1 maps the collection: capable models, efficient execution and dependable systems, all shaped by shared device budgets.

Figure 1. The Edge AI learning landscape. AI generated illustration informed by device deployment and co-design.12

This is broader than a smaller language model. It includes speech, streaming perception and specialist models. Local execution is a design option, not a rule that every task must stay on the device.

The three territories are connected decisions, not fixed pipeline stages. An architecture or training change can alter execution cost; the product’s deadline or sensory inputs can change which model is useful. The image’s shrinking blocks represent cheaper numerical storage, not removed layers; its traces are schematic, not measured performance.

Explore the topics

Start with foundations, then follow the bottleneck or question that matters to your work. Chapters and focused companions explain mechanisms, tradeoffs and the evidence needed to choose between them.

Topics and focused companions

12 topics
Topic The question it helps answer
Foundations and bottlenecks What really limits the workload: capacity, movement, arithmetic or sustained operation?
On device computing How do CPU, GPU, NPU, memory and power differ from cloud computing, and which platforms serve each device?
Efficient model architectures Which design choices reduce stored weights, cache or work without losing the needed capability?
Quantization and compression What can be represented more cheaply, and will the backend exploit it?
Training capable small models How can data, distillation and task training make a small model useful?
Efficient inference How do prefill, decoding, caching and scheduling change the execution cost?
Runtimes, compilers and kernels How does a checkpoint become a supported, efficient device workload?
Multimodal and continuous workloads How do images, audio and video change context, evidence and resource budgets?
Edge agents and hybrid systems What stays local, what goes remote, and how are actions controlled?
Local and cloud model collaboration How can local specialists and stronger remote models cooperate, and when does the whole system improve?
Evaluation and profiling How do we measure useful quality, latency, memory and energy on the actual path?
Production and product design How does the capability survive downloads, updates, failures and diverse devices?

Scroll to compare. Focus view gives the table more room.

Choose a reading path

Fit the capability

Understand the memory budget, choose an architecture, then decide whether compression or better training addresses the limiting factor.

Foundations → Models → Compression → Training

Make it run well

Understand the device hardware, separate inference phases, inspect the backend contract, and measure the complete workload rather than a favorable kernel alone.

Hardware → Inference → Runtimes → Evaluation

Build the useful system

Add real sensory inputs, bounded actions and local or remote routing. Then consider the lifecycle that makes the feature dependable.

Multimodal → Agents → Production

The Transformer foundations supply prerequisites for the language and vision language chapters. Hardware 101 introduces circuits, sensors, memory, firmware and power for readers who want to build physical AI devices. On device computing adds a hardware companion to the landscape, with current chips and device execution paths. Model Deep Dives examine individual reports, while Agents and Document AI explore systems that may benefit from local execution.

From the task to the device

A model that fits on disk can still fail in memory. An accelerated graph can lose time crossing processor boundaries. Mobile deployment guidance makes both constraints explicit; MCUNet demonstrates designing a model and runtime together.12

Figure 2 turns the landscape into a practical feedback loop: task needs, model choices, execution and device evidence.

Start with a useful task. Close the loop on the device.
Design and test
  1. Task contractQuality, deadline, privacy and allowed actions
  2. Model and dataCapacity, architecture, training and compression
  3. Execution pathRuntime, kernels, memory and scheduling
  4. Device evidenceCompleted tasks under realistic conditions

Return the measured failure to the right decision: change the model, execution path or product scope.

Figure 2. Capability, execution and product design form one feedback loop. Independent synthesis informed by mobile deployment guidance and MCUNet.12

How to read the evidence

This is a learning map, not a device leaderboard. Chapters distinguish reported findings, mechanism explanations and our own illustrative calculations. The original collection was reviewed on October 12, 2026; focused companions give their own review dates. Mutable runtime documentation should be checked again for the versions you deploy. No device benchmark was run for this collection.

The practical goal is to find a feasible design that meets the task. When a reported gain matters, carry its model, precision, input length, backend and measurement conditions with it.

References

  1. ONNX Runtime, Deploy on mobile, model size, execution providers and performance considerations. Inspected October 12, 2026. ↩ ↩2 ↩3

  2. Ji Lin et al., MCUNet: Tiny Deep Learning on IoT Devices, official project summary of model and inference-engine co-design, NeurIPS 2020. ↩ ↩2 ↩3

Navigation

Type to search…

↑↓ navigate↵ selectEsc close