Local large language models on mobile devices

AIThis description was written by a model from public signals. The facts behind it are listed under Evidence.

Problem

Running large language models on mobile devices requires prohibitive amounts of RAM and storage. Developers and end users cannot deploy state-of-the-art models like 35B and 80B parameter LLMs locally on phones without expensive cloud inference services.

Solution concept

A runtime system that streams expert weights from device storage on-demand, allowing large mixture-of-experts models to run locally with minimal RAM footprint. This enables running 35B and 80B parameter models directly on iPhones and other Apple devices without cloud dependency.

Who has it

Mobile app developers and device users who want to run large language models locally on Apple devices, including iPhones, without relying on cloud APIs.

How they handle it today

Today, developers either use cloud-based LLM APIs (which incur per-request costs and latency) or run smaller, less capable models locally on devices. End users depend on cloud services or cannot access advanced language capabilities on their phones at all.

Why now

Large open-source mixture-of-experts models like Qwen are now available; Apple devices have improved metal compute capabilities; and developer demand for on-device inference is growing to reduce latency, cost, and privacy concerns with cloud APIs.

How it could charge

Inference as a service for developers (per-API-call or subscription), or distribution through app store as a framework/SDK that developers pay for; direct monetization through consumer apps that embed local LLM capabilities.