Skip to content
Run Edge0-145B on One GPU and Point Your Agent at It — ContentBuffer guide

Run Edge0-145B on One GPU and Point Your Agent at It

K
Kodetra Technologies··17 min read Intermediate

Summary

Serve Edge0-145B locally, dodge the empty-reply trap, benchmark it, and run a coding agent.

Edge0 dropped Edge0-145B on November 12, and by the weekend it was the model everyone on r/LocalLLaMA was trying to cram onto a single consumer GPU. The pitch is blunt: a 145B-parameter mixture-of-experts that activates about 10B parameters per token, was trained natively at 4 bits, and, per the vendor's own numbers, decodes at roughly 80 to 98 tokens per second on one 24 GB card. Its two Hugging Face builds passed 140K downloads in under a week.

This guide skips the hype and does the work. You will pick the right build for your hardware, start the bundled edge0-infer server, hit its OpenAI-compatible endpoint from Python, dodge the empty-reply trap that catches almost everyone on day one, benchmark the model on your machine instead of trusting a table, and finally point a small test-fixing coding agent at it. Everything runs locally: no API key, no per-token bill, no code leaving your laptop.

One framing note before we start. Every benchmark number in this post comes from Edge0's blog or model card and is self-reported. Treat them as a hypothesis. The benchmark script in Step 4 exists so you can check them.

What you are actually downloading

Edge0-145B is not a requantized checkpoint. The model card says the expert layers of Qwen3.8-Flash-Next were rebuilt as a latent MoE and post-trained on Edge0's own data. That detail matters because it explains why it fits where it does:

  • 48 layers with hybrid attention: full attention every fourth layer, a sparse indexer elsewhere.
  • 512 experts per layer, 10 routed plus 1 shared active per token. Only ~10.12B of 145B parameters do work on any given token, which is why decode is fast once the hot experts sit in VRAM.
  • A 51.2B-parameter n-gram embedding table stored in fp8 and memory-mapped, so a big slice of the model can live on disk or in page cache instead of VRAM.
  • A 1.26B multi-token-prediction head used for speculative decoding (the --spec and --mtp-q4 flags).
  • 256K context, with an int8 KV cache that keeps 32K resident.

There are two builds. Edge0/Edge0-145B-q3k (about 40.8 GiB) runs on CUDA 13 under Linux or Windows and on Metal on Apple Silicon. Edge0/Edge0-145B-NVFP4 (about 47.8 GiB) is CUDA-only and wants a Blackwell-class card with FP4 tensor cores. A smaller sibling, Edge0-120B, ships alongside it.

Licensing is the part people skim and later regret. The weights use the custom edge0-community-model-1.0 license: personal use, research, internal use and commercial use of outputs are fine, but selling the model, running it as a commercial model service, or shipping it inside a commercial coding assistant needs a separate written license from Edge0. The inference framework on GitHub is Apache-2.0. If you are building a product, read the license before you build the product.

Prerequisites

  • An NVIDIA GPU with 24 GB VRAM (RTX 4090, 5090, 5090 D) plus at least 48 GB of system RAM, or an Apple Silicon Mac with 64 GB+ unified memory for the Q3_K build. 16 GB cards work with offload but expect roughly half the speed.
  • CUDA 13 drivers on Linux or Windows (x64 or ARM64) if you are on NVIDIA.
  • About 50 GB of free disk for one build, on an SSD. The memory-mapped tables page in from disk, so a spinning drive hurts.
  • Python 3.10+ with pip install openai psutil pytest for the scripts in this guide.
  • Git and a C++ toolchain to build edge0-infer from source.

Step 1: Pick the build your machine can actually run

The most common failure in the first week has been people downloading the NVFP4 build onto an Ampere or Ada card, or onto a Mac, then wondering why nothing loads. NVFP4 is CUDA-only and needs FP4 hardware. The script below reads your GPU and RAM and turns the model card's memory figures into a recommendation, including the offload flags you will need when the weights do not fit in VRAM.

"""Pick an Edge0-145B build and edge0-infer flags for this machine.

Numbers come from the Edge0-145B model card (Nov 2026):
  Q3_K  file ~40.81 GiB, runs on CUDA 13 (Linux/Windows) and Metal (Apple Silicon)
  NVFP4 file ~47.79 GiB, CUDA only (Blackwell-class FP4 tensor cores)
  RTX 5090 D 24 GB with residency: ~23 GiB VRAM + 31-38 GiB host RAM
"""
import platform
import shutil
import subprocess

import psutil


def gpu_vram_gib():
    """Total VRAM of GPU 0 in GiB, or None if no NVIDIA GPU is visible."""
    if not shutil.which("nvidia-smi"):
        return None
    out = subprocess.run(
        ["nvidia-smi", "--query-gpu=memory.total,name", "--format=csv,noheader,nounits"],
        capture_output=True, text=True,
    ).stdout.strip().splitlines()
    if not out:
        return None
    mib, name = [x.strip() for x in out[0].split(",", 1)]
    return int(mib) / 1024, name


def recommend(vram_gib, ram_gib, gpu_name="", system=platform.system()):
    is_mac = system == "Darwin" and platform.machine() == "arm64"
    if is_mac:
        # Unified memory: weights + KV share one pool. Leave ~12 GiB for the OS.
        if ram_gib < 64:
            return {"ok": False, "why": "Q3_K needs ~41 GiB of weights plus KV; 64 GB+ unified memory recommended"}
        return {"ok": True, "repo": "Edge0/Edge0-145B-q3k", "flags": []}

    if vram_gib is None:
        return {"ok": False, "why": "no CUDA GPU found; the Edge0 engine targets CUDA 13 or Metal"}

    blackwell = any(k in gpu_name for k in ("5090", "5080", "Spark", "B200", "RTX PRO 6000"))
    repo = "Edge0/Edge0-145B-NVFP4" if blackwell else "Edge0/Edge0-145B-q3k"
    file_gib = 47.79 if blackwell else 40.81

    if vram_gib >= file_gib + 8:
        return {"ok": True, "repo": repo, "flags": []}  # everything fits on the GPU
    host_needed = file_gib - (vram_gib - 4)  # experts that spill to RAM
    if ram_gib < host_needed + 8:
        return {"ok": False, "why": f"need ~{host_needed + 8:.0f} GiB RAM for spilled experts, have {ram_gib:.0f}"}
    budget = max(int(vram_gib) - 4, 8)
    return {"ok": True, "repo": repo,
            "flags": ["--resident-budget-gib", str(budget), "--mmap-experts", "--expert-cache", "auto"]}


if __name__ == "__main__":
    ram = psutil.virtual_memory().total / 2**30
    g = gpu_vram_gib()
    vram, name = g if g else (None, "")
    print(f"GPU: {name or 'none'}  VRAM: {vram or 0:.1f} GiB  RAM: {ram:.1f} GiB")
    print(recommend(vram, ram, name))

Sample results for a few machines (these come from calling recommend() directly with each hardware profile):

recommend(24, 64, 'NVIDIA GeForce RTX 5090 D')
{'ok': True, 'repo': 'Edge0/Edge0-145B-NVFP4',
 'flags': ['--resident-budget-gib', '20', '--mmap-experts', '--expert-cache', 'auto']}

recommend(24, 64, 'NVIDIA GeForce RTX 4090')
{'ok': True, 'repo': 'Edge0/Edge0-145B-q3k',
 'flags': ['--resident-budget-gib', '20', '--mmap-experts', '--expert-cache', 'auto']}

recommend(16, 32, 'NVIDIA GeForce RTX 4080')
{'ok': False, 'why': 'need ~37 GiB RAM for spilled experts, have 32'}

The thresholds are deliberately conservative. The card's own measurement for a 5090 D with residency on is about 23 GiB of VRAM plus 31 to 38 GiB of host RAM, so a 24 GB card with 32 GB of RAM is right on the edge and will swap under load. If the script says no, believe it.

Step 2: Download the weights and build the server

# 1. Weights (pick the repo Step 1 told you to)
pip install -U huggingface_hub
huggingface-cli download Edge0/Edge0-145B-q3k --local-dir ~/models/Edge0-145B-q3k

# 2. Inference framework (Apache-2.0)
git clone https://github.com/Edge0-AI/Edge0.git edge0
cd edge0
# Follow the platform-specific build section in the repo docs
# (Linux/NVIDIA, Windows/NVIDIA, or macOS/Metal), then:

# 3. Serve. Keep the model folder's name: edge0-infer reads model_name from model.json
build/cpu/bin/edge0-infer serve --model ~/models/Edge0-145B-q3k \
    --resident-budget-gib 20 --mmap-experts --expert-cache auto

On startup you should see a line like Listening: http://127.0.0.1:8000 model=edge0_145b. That model= value is the string you pass as model in every request. The server speaks both the OpenAI and Anthropic wire formats on port 8000. On Windows the binary is build\windows\bin\edge0-infer.exe, and there is also a desktop app, Edge0 Chat, in the GitHub releases if you only want to chat.

If you would rather use a runtime you already have, the model card lists llama.cpp, Ollama, LM Studio, Jan, vLLM and Docker Model Runner. Two caveats: those configs default to port 9931 rather than 8000, and the speed numbers in Edge0's tables were measured with its own engine, which is where the residency and speculative-decoding tricks live. Start with edge0-infer so you have a baseline.

A quick curl confirms the server is alive before you write any Python:

curl localhost:8000/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model":"edge0_145b","messages":[{"role":"user","content":"hi"}],"max_tokens":64}'

Step 3: Talk to it from Python, and avoid the empty-reply trap

Because the endpoint is OpenAI-compatible, the official openai SDK works unchanged; you only swap base_url. The catch is reasoning. Edge0-145B thinks before it answers by default, and with a small max_tokens the whole budget can go to the hidden reasoning pass. You get back an empty string with finish_reason="length", and it looks exactly like a broken server. The model card's fix is to send "reasoning_effort": "none" when you want the answer without a thinking pass.

"""Smoke-test a local Edge0 server and show the empty-reply trap."""
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="local")  # key is ignored locally
MODEL = "edge0_145b"  # name printed by `edge0-infer serve` on startup


def ask(prompt, max_tokens=64, effort=None):
    extra = {"reasoning_effort": effort} if effort else {}
    r = client.chat.completions.create(
        model=MODEL,
        messages=[{"role": "user", "content": prompt}],
        max_tokens=max_tokens,
        extra_body=extra,
    )
    c = r.choices[0]
    return c.message.content or "", c.finish_reason, r.usage


q = "What is 17 * 23? Answer with just the number."
for effort in (None, "none"):
    text, finish, usage = ask(q, effort=effort)
    print(f"effort={effort!s:5}  finish={finish:7}  completion_tokens={usage.completion_tokens:3}  text={text!r}")

Output:

effort=None   finish=length   completion_tokens= 64  text=''
effort=none   finish=stop     completion_tokens=  2  text='391'

Same question, same token cap, opposite outcome. Pass reasoning_effort through extra_body rather than as a typed keyword: the SDK's type hints list OpenAI's own effort values, and extra_body forwards the field verbatim to whatever server is on the other end. The rule of thumb: use "none" for classification, extraction, short answers and anything latency-bound; leave reasoning on, with a generous max_tokens (2K to 8K), for multi-step coding and debugging.

Step 4: Benchmark it on your own hardware

Vendor speed tables are measured on vendor rigs with vendor flags. The script below streams responses at three prompt sizes and reports median time-to-first-token (dominated by prefill) and decode tokens per second. It asks the server for a final usage chunk so the token counts are exact, and falls back to counting stream chunks if the server does not send one.

"""Measure time-to-first-token and decode speed on YOUR hardware.

Vendor tables are measured on their rigs. This tells you what you actually get.
"""
import statistics
import sys
import time

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="local")
MODEL = "edge0_145b"
FILLER = "def f{i}(x):\n    return x * {i} + 1\n\n"


def one_run(prompt_chars, out_tokens=256):
    code = "".join(FILLER.format(i=i) for i in range(prompt_chars // len(FILLER) + 1))[:prompt_chars]
    msgs = [{"role": "user", "content": code + "\n\nWrite a docstring for every function above."}]
    t0 = time.perf_counter()
    first = None
    pieces = 0
    usage = None
    stream = client.chat.completions.create(
        model=MODEL, messages=msgs, max_tokens=out_tokens, stream=True,
        stream_options={"include_usage": True},
        extra_body={"reasoning_effort": "none"},
    )
    for chunk in stream:
        if chunk.usage:
            usage = chunk.usage
        if chunk.choices and chunk.choices[0].delta.content:
            if first is None:
                first = time.perf_counter()
            pieces += 1
    end = time.perf_counter()
    out = usage.completion_tokens if usage else pieces  # fall back to chunk count
    prompt_toks = usage.prompt_tokens if usage else None
    ttft = (first or end) - t0
    decode = out / max(end - (first or end), 1e-6)
    return ttft, decode, prompt_toks


if __name__ == "__main__":
    sizes = [int(s) for s in sys.argv[1:]] or [2_000, 16_000, 64_000]
    print(f"{'prompt_chars':>12} {'prompt_tok':>10} {'ttft_s':>8} {'decode_tok/s':>13}")
    for n in sizes:
        runs = [one_run(n) for _ in range(3)]
        ttft = statistics.median(r[0] for r in runs)
        dec = statistics.median(r[1] for r in runs)
        print(f"{n:>12} {runs[0][2] or '?':>10} {ttft:>8.2f} {dec:>13.1f}")

Run it with python bench.py or pass your own sizes, for example python bench.py 4000 32000. The output has this shape:

prompt_chars prompt_tok   ttft_s  decode_tok/s
        2000        487     ...          ...
       16000       3961     ...          ...

Compare what you get against the card's own figures:

Hardware (model card)Build16K prefill tok/sChat decode tok/sAgent decode tok/s
RTX 5090 D 24 GBNVFP43165.779.698.3
RTX 5090 D 24 GBQ3_K3554.999.7121.2
RTX 4090 24 GBQ3_K3179.469.991.7
Apple M2 Ultra (Metal)Q3_K800.360.957.0
16 GiB VRAM + 20 GiB RAMQ3_K33536.844.9

Two things jump out. First, on the 5090 D the Q3_K build is faster than NVFP4, so do not assume the bigger file is the better choice. Second, the 16 GiB row shows what offload costs: decode roughly halves and prefill drops by an order of magnitude, which you will feel on every long prompt. Also note that the only Apple number on the card is an M2 Ultra, a desktop chip. The blog mentions MacBook Pros but publishes no laptop measurement, so if you are on a laptop your benchmark is the first real data point you will have.

Step 5: Point a coding agent at it

Edge0 is pitched as a local coding and agent model, so let's test that claim with a real loop. The agent below gets four tools (list a directory, read a file, write a file, run pytest), a repo path, and one instruction: make the failing tests pass without editing the tests.

There is one wrinkle. Neither the blog nor the model card documents the tool-call format, and behavior differs between edge0-infer, llama.cpp and Ollama. So the agent starts by probing: it sends one request with a tools array and checks whether a structured tool_calls field comes back. If yes, it uses native function calling. If the server rejects the field or the model answers in prose, it falls back to a plain JSON action protocol that every chat endpoint can handle. The tool implementations, path sandbox and step limit are shared by both paths.

"""A small test-fixing agent that runs entirely against a local Edge0 server.

It probes whether the server accepts native `tools`; if not, it falls back to a
JSON action protocol. Either way the loop, the tools and the guards are the same.
"""
import json
import pathlib
import subprocess
import sys

from openai import OpenAI, BadRequestError

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="local")
MODEL = "edge0_145b"
ROOT = pathlib.Path(sys.argv[1] if len(sys.argv) > 1 else ".").resolve()
MAX_STEPS = 12


def _safe(path):
    p = (ROOT / path).resolve()
    if ROOT not in p.parents and p != ROOT:
        raise ValueError(f"path escapes repo: {path}")
    return p


def list_dir(path="."):
    return "\n".join(sorted(x.name + ("/" if x.is_dir() else "") for x in _safe(path).iterdir()))


def read_file(path):
    return _safe(path).read_text()[:20_000]


def write_file(path, content):
    _safe(path).write_text(content)
    return f"wrote {len(content)} chars to {path}"


def run_tests():
    r = subprocess.run([sys.executable, "-m", "pytest", "-q", "-x"], cwd=ROOT,
                       capture_output=True, text=True, timeout=120)
    return (r.stdout + r.stderr)[-4_000:]


TOOLS = {"list_dir": list_dir, "read_file": read_file, "write_file": write_file, "run_tests": run_tests}
SCHEMAS = [
    {"type": "function", "function": {"name": "list_dir", "description": "List a directory in the repo",
     "parameters": {"type": "object", "properties": {"path": {"type": "string"}}}}},
    {"type": "function", "function": {"name": "read_file", "description": "Read a text file",
     "parameters": {"type": "object", "properties": {"path": {"type": "string"}}, "required": ["path"]}}},
    {"type": "function", "function": {"name": "write_file", "description": "Overwrite a file with new content",
     "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}},
                    "required": ["path", "content"]}}},
    {"type": "function", "function": {"name": "run_tests", "description": "Run pytest, return the tail of output",
     "parameters": {"type": "object", "properties": {}}}},
]
SYSTEM = ("You fix failing Python tests. Inspect files, make the smallest correct change to the "
          "source (never edit tests), re-run tests, and stop when they pass.")
JSON_PROTOCOL = ("\nReply with ONE JSON object and nothing else: "
                 '{"tool": "<list_dir|read_file|write_file|run_tests>", "args": {...}} '
                 'or {"done": "<summary>"}.')


def supports_native_tools():
    try:
        r = client.chat.completions.create(
            model=MODEL, max_tokens=64, tools=SCHEMAS[3:], extra_body={"reasoning_effort": "none"},
            messages=[{"role": "user", "content": "Call run_tests."}])
        return bool(r.choices[0].message.tool_calls)
    except BadRequestError:
        return False


def call_tool(name, args):
    if name not in TOOLS:
        return f"error: unknown tool {name}"
    try:
        return str(TOOLS[name](**args))
    except Exception as e:  # feed errors back instead of crashing the loop
        return f"error: {type(e).__name__}: {e}"


def run_native(task):
    msgs = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": task}]
    for step in range(MAX_STEPS):
        r = client.chat.completions.create(model=MODEL, messages=msgs, tools=SCHEMAS,
                                           max_tokens=4096, extra_body={"reasoning_effort": "low"})
        m = r.choices[0].message
        if not m.tool_calls:
            return m.content
        msgs.append(m.model_dump(exclude_none=True))
        for tc in m.tool_calls:
            out = call_tool(tc.function.name, json.loads(tc.function.arguments or "{}"))
            print(f"[{step}] {tc.function.name} -> {out[:80]!r}")
            msgs.append({"role": "tool", "tool_call_id": tc.id, "content": out})
    return "step limit reached"


def run_json(task):
    msgs = [{"role": "system", "content": SYSTEM + JSON_PROTOCOL}, {"role": "user", "content": task}]
    for step in range(MAX_STEPS):
        r = client.chat.completions.create(model=MODEL, messages=msgs, max_tokens=4096,
                                           extra_body={"reasoning_effort": "low"})
        raw = (r.choices[0].message.content or "").strip()
        msgs.append({"role": "assistant", "content": raw})
        try:
            act = json.loads(raw[raw.index("{"): raw.rindex("}") + 1])
        except ValueError:
            msgs.append({"role": "user", "content": "That was not valid JSON. Reply with one JSON object."})
            continue
        if "done" in act:
            return act["done"]
        out = call_tool(act.get("tool", ""), act.get("args", {}))
        print(f"[{step}] {act.get('tool')} -> {out[:80]!r}")
        msgs.append({"role": "user", "content": f"Tool result:\n{out}"})
    return "step limit reached"


if __name__ == "__main__":
    native = supports_native_tools()
    print(f"native tool calling: {native}")
    task = "The test suite is failing. Find out why and fix it."
    print("RESULT:", (run_native if native else run_json)(task))

Here is a tiny broken repo to try it on:

mkdir demo
printf 'def add(a, b):\n    return a - b\n' > demo/calc.py
printf 'from calc import add\n\ndef test_add():\n    assert add(2, 3) == 5\n' > demo/test_calc.py
python agent.py demo

A run on the JSON fallback path looks like this:

native tool calling: False
[0] run_tests -> 'F\n=================================== FAILURES ================================='
[1] read_file -> 'def add(a, b):\n    return a - b\n'
[2] write_file -> 'wrote 32 chars to calc.py'
[3] run_tests -> '.                                                                        [100%]\n'
RESULT: add() subtracted instead of adding; fixed calc.py, 1 test passes

And demo/calc.py now reads return a + b. The native path prints the same four steps with native tool calling: True. The exact sequence a real model takes will vary between runs, and on harder repos it may read several files before writing anything. That is fine; the step cap keeps it from wandering forever.

A few design choices worth copying into your own agents:

  • Errors go back to the model, not up the stack. call_tool catches exceptions and returns them as text, so a bad path or a typo in a tool name becomes something the model can correct on its next turn.
  • The sandbox is enforced in code. _safe() resolves every path and refuses anything outside the repo root. Never rely on the system prompt for this; a local model will happily write to ../../.bashrc if it decides that helps.
  • The JSON parser is forgiving on the outside, strict on the inside. It slices from the first { to the last }, which tolerates chatty preambles like "Sure!", then requires valid JSON. On a parse failure it asks again instead of crashing.
  • Reasoning effort is per call. The probe uses "none" because it only needs a yes or no; the work loop uses "low" with a 4K budget so the model can plan a fix without burning a minute on every step.

Worked example: a private pre-commit fixer

Here is where a local model earns its keep. Suppose your team cannot send source code to a hosted API, but you want something that tries to repair a failing test suite before a developer pushes. Wire the agent into a git hook:

#!/usr/bin/env bash
# .git/hooks/pre-push  (chmod +x)
if ! python -m pytest -q -x >/dev/null 2>&1; then
  echo "Tests failing; asking local Edge0 for a fix..."
  python ~/tools/agent.py "$(git rev-parse --show-toplevel)"
  git --no-pager diff --stat
  echo "Review the diff above, commit it if it's right, then push again."
  exit 1
fi

The hook never commits on its own. It proposes a change, shows the diff stat, and blocks the push until a human looks. On a 24 GB card at the card's quoted ~90 tokens per second, a four-step fix with short tool outputs is a matter of seconds of generation, and none of the code touches the network. If your repo is large, cap run_tests to the failing test file to keep each step's context small; prefill time grows with every tool result you feed back.

Common pitfalls and how to fix them

  • Empty replies with finish_reason="length". The reasoning pass ate your token budget. Send reasoning_effort: "none" or raise max_tokens. This is the single most reported day-one issue.
  • NVFP4 build on the wrong hardware. NVFP4 is CUDA-only and needs FP4 tensor cores. On an RTX 4090, any 30-series card or a Mac, download the Q3_K repo instead.
  • Renamed model folder. edge0-infer reads model_name from model.json inside the folder. Rename or flatten the directory and the server either fails to start or reports a different model name than your client sends.
  • Wrong port. edge0-infer listens on 8000. The llama.cpp-style configs on the model card use 9931. If your client gets connection refused, check which runtime you actually started.
  • RAM is the real bottleneck. With residency on, the card measured 31 to 38 GiB of host RAM on top of 23 GiB VRAM. On a 32 GB machine the OS starts swapping the memory-mapped tables and decode collapses. Close the browser, or step down to Edge0-120B.
  • Slow first requests. The n-gram table and cold experts page in from disk on first use. Run one throwaway request after startup, and benchmark only after that warm-up. The --expert-profile flag lets you persist which experts are hot for your workload.
  • Treating benchmark claims as settled. DeepSWE 64.6 and Terminal-Bench 2.1 86.5 are Edge0's internal evals. They may hold up; they have not yet been independently reproduced. Run your own task set before you swap out a hosted model you trust.
  • Tool calling that silently degrades. Some runtimes accept a tools array and then ignore it, returning prose. That is why the agent probes for a real tool_calls response instead of trusting that the request succeeded.
  • Licensing. Internal use is allowed; offering it as a paid service or bundling it into a commercial coding assistant is not, without a separate agreement. Check before you put it behind a customer-facing endpoint.

Quick reference

TaskCommand or setting
Serve (Edge0 engine)edge0-infer serve --model
Interactive chatedge0-infer chat --model
Endpointhttp://127.0.0.1:8000/v1/chat/completions (OpenAI and Anthropic formats)
Model name in requestsedge0_145b (as printed on startup)
Skip thinkingextra_body={"reasoning_effort": "none"}
Offload experts to RAM--resident-budget-gib N --mmap-experts --expert-cache auto
Reserve VRAM for other apps--vram-reserve-mib N
Speculative decoding--spec N, --mtp-q4 all|proj|head
Build for Ada/Ampere/MacEdge0/Edge0-145B-q3k (~40.8 GiB)
Build for BlackwellEdge0/Edge0-145B-NVFP4 (~47.8 GiB) or q3k
Context256K, int8 KV, 32K resident
Weights licenseedge0-community-model-1.0 (no commercial model service)

Next steps

  • Swap the toy demo for a real repo and record how many steps the agent takes per fix. That number tells you more than any leaderboard.
  • Run bench.py with and without --spec to see what speculative decoding buys on your card.
  • Try Edge0-120B if you are RAM-limited; it trades a few points of benchmark score for a smaller footprint.
  • Point an existing tool at the local endpoint: anything that accepts a custom OpenAI base URL, including the Anthropic-format clients, can use http://127.0.0.1:8000.
  • Build a small private eval: 20 real bugs from your issue tracker, run the agent on each, and compare the pass rate against the hosted model you use today.

Sources: Edge0 launch blog, Edge0-145B model card on Hugging Face, Edge0 framework on GitHub. The scripts in this guide were exercised end to end against a scripted OpenAI-compatible server to check every code path; throughput figures are from the model card, not our measurements.

Comments

Subscribe to join the conversation...

Be the first to comment

Found this useful?

Get new AI guides for builders by email. Free.

Join the builders who read ContentBuffer.

Also get