⚡Google Ships LiteRT Tensor API, Runs SAM 2 1.5x Faster Than MLX on Mac

Write your model once in C++, run it everywhere

Google just made writing on-device ML in C++ way less painful. The new LiteRT Tensor API lets you build computation graphs operation by operation, then run the same graph on CPU, GPU, and NPU without rewriting it for each target. If you've ever hand-written backend-specific code and then rewritten it for the next chip, this is for you. The API also handles direct weight loading from checkpoints and zero-copy buffer sharing across pre-processing, inference, and post-processing, which is exactly the stuff that eats weeks of engineering time. It's also built to work with AI coding agents, which can generate, modify, validate, and benchmark graphs in C++. Google tested this with a multi-object segmentation and tracking app built by a coding agent. The same SAM 2 C++ code ran on GPUs in a MacBook Pro, an iPhone, and an Android phone. On the MacBook, it came out about 1.5x faster than Apple MLX using fp16 for the image encoder plus mask decoder. The agent also rewrote operations that fell back to CPU on WebGPU using GPU-compatible arithmetic. Worth watching: Google says it's adding direct ML Drift backend support, more NPU-friendly ops, automatic full-graph quantization, and broader dynamic shapes. Those are the pieces that decide whether this replaces your current stack or stays a demo.