Bringing generative AI capabilities to edge devices

Ambarella, Inc. announced during CES that it demonstrates multi-modal large language models (LLMs) running on its new N1 SoC series at a fraction of the power-per-inference of leading GPU solutions. It aims to bring generative AI – a transformative technology that first appeared in servers due to the large processing power required – to edge endpoint devices and on-premise hardware across various applications such as video security analysis, robotics and various industrial applications.

The company will initially offer optimised generative AI processing capabilities on its mid to high-end SoCs, from the existing CV72 for on-device performance under 5W, through to the new N1 series for server-grade performance under 50W. Compared to GPUs and other AI accelerators, it provides complete SoC solutions up to 3x more power-efficient per generated token while allowing immediate and cost-effective product deployment.

"Generative AI networks are enabling new functions across our target application markets that were just not possible before," said Les Kohn, CTO and co-founder of Ambarella. "All edge devices are about to get a lot smarter, with our N1 series of SoCs enabling world-class multi-modal LLM processing in a very attractive power/price envelope."

"Virtually every edge application will get enhanced by generative AI in the next 18 months," said Alexander Harrowell, principal analyst, Advanced Computing at Omdia. "When moving genAI workloads to the edge, the game becomes all about performance per watt and integration with the rest of the edge ecosystem, not just raw throughput."

Its new Cooper Developer Platform supports all of the company's AI SoCs. Also, to reduce customers' time-to-market, it has pre-ported and optimised popular LLMs, such as Llama-2 and the Large Language and Video Assistant (LLava) model running on N1 for multi-modal vision analysis of up to 32 camera sources. These pre-trained and fine-tuned models will be available for partners to download from the Cooper Model Garden.