Magnitude
Easy local inference for agents
About Magnitude
Magnitude is an open source inference server for running local models behind a coding agent you already use. What separates it from the other ways to run a model on your own machine is that it starts by profiling the hardware. It reads your chip, memory and bandwidth, then presents a catalogue of models that will actually fit, each with an estimated tokens-per-second figure for your specific machine, and only then downloads and tunes the one you pick. The question it answers is the one that actually stops people running models locally, which is not how to start a server but which model is worth running on this laptop and how fast it will be.
Its own FAQ takes the obvious objection head on: why not just ask your agent to set up Ollama. The stated answer is that the agent would be guessing, because it does not know your hardware, which quantisation fits, or how fast the result will run - so Magnitude supplies a catalogue with recommendations computed for the machine, an onboarding flow that writes the harness configuration, and inference tuned for agent workloads rather than chat. Tuning covers speculative decoding and concurrency, and models load just in time and unload when idle or when memory gets tight, so it does not sit on your RAM between tasks. It is Apache-2.0, everything stays on the machine, and once a model is downloaded it works with no network at all. Two honest caveats: the project is young, and there is a "Backed by" section on the site, which means investor money behind a project with no stated business model yet.
There are two entry points. The agent-first path is a prompt you paste into the agent you already run, telling it to install the CLI and follow the onboarding, after which the agent walks you through model selection and writes your harness configuration for you. The manual path is installing the CLI globally with npm and running the setup command. Either way, setup profiles the machine, shows which models fit with their expected throughput, downloads what you choose, and connects it to your harness - the documented list is Pi, OpenCode, Hermes, OpenClaw, Codex, Claude Code, Oh My Pi and Cline, and there is a built-in harness if you would rather not use one of those. After that it runs in the background and needs no management: models are loaded when the agent asks for them and unloaded when idle or when memory fills. Adding or switching models later is a CLI command, and the agent can do it. Models outside the curated catalogue work too - compatible GGUF files from Hugging Face can be used directly.
- •Hardware Profiling - Reads your chip, memory and bandwidth and computes what your machine can actually run, rather than leaving the choice to guesswork
- •Recommendations With Throughput Estimates - Each suggested model comes with an estimated tokens per second for your specific hardware
- •Tuned for Agent Workloads - Speculative decoding and concurrency configured end to end, rather than general-purpose chat serving defaults
- •Just-in-Time Model Loading - Models load on request and unload when idle or when memory gets tight, so the server does not squat on RAM
- •Plugs Into Existing Harnesses - Pi, OpenCode, Hermes, OpenClaw, Codex, Claude Code, Oh My Pi and Cline, with a built-in harness as an alternative
- •Fully Offline and Apache-2.0 - Prompts, files and models stay on the machine, no network is needed once a model is downloaded, and the licence permits modification
Developers who want to run an agent on local models and have been put off by not knowing what their machine can handle, and anyone whose token spend on a hosted model has become a line item worth removing. It fits best alongside an agent you already use, since integration with an existing harness is the delivery mechanism. Teams with code that cannot be sent to a provider get the privacy argument for free. Consider the alternatives carefully if you already know exactly which model and quantisation you want and are comfortable configuring a server, because in that case Ollama or vLLM already do the job and Magnitude is mostly saving you the research. It is also the youngest and least proven tool in this batch, and the curated catalogue means the recommendations are the project's editorial judgement, even though models outside it are supported.
