Microsoft Unveils Custom AI Chips: Inside Microsoft’s Labs

As companies worldwide are beginning to see the advantages presented by custom chip solutions, Microsoft has recently announced its first range of custom chips that it says will power future AI systems and its Azure cloud infrastructure. Why are custom chips becoming more popular, what specs do Microsoft’s new devices boast, and what does this mean for the future of custom silicon?

 In a Redmond, Washington Microsoft lab, a custom rack hosts the Maia 100 AI Accelerator and its cooling "sidekick" in a thermal chamber. The sidekick, akin to a car radiator, circulates liquid to cool the chips managing AI workloads. Photo by John Brecher for Microsoft.

Why are custom chips becoming more popular?

One trend that has shown great promise in the past decade is the rise of custom silicon. While custom silicon devices have been around for decades (uncommitted logic arrays were highly popular in 80s home computer systems), most of these have either been application-specific devices manufactured in small volumes for extremely niche applications or based on layouts with a reduced degree of design freedom. 

Design Flexibility and Optimisation

Furthermore, many engineering companies have been designing their own custom integrated circuits for years, with foundries such as TSMC offering fabrication services. However, what differentiates these devices from what we could consider truly custom ICs is that they are sold to other engineering companies who then incorporate those devices into their products. 

Simply put, if an IC is manufactured and sold on the open market for anyone to use, it is not a custom IC. However, devices being manufactured by companies, such as the M series from Apple, are perfect examples of custom ICs as they are only used in the products being sold by the chip designers. 

Comparing Microsoft's approach to custom silicon with industry counterparts, it's evident that Microsoft's focus is on optimising its cloud and AI workloads. Unlike Apple's M series chips, which are designed for a broad range of devices, Microsoft's Maia 100 AI Accelerator and Cobalt 100 CPU are tailored specifically for Azure's infrastructure, demonstrating a commitment to specialised, high-performance computing.

So why are these custom devices becoming increasingly popular?

Well, the reason for the popularity of custom ICs comes down to the fact that they offer designers the highest degree of freedom, being able to dictate exactly how many transistors are integrated into the chip, the packaging used, and what features it integrates. In the case of mobile devices, this allows engineers to focus on reducing energy consumption while ensuring maximum performance per watt, and in high-performance applications, it allows for the integration of numerous cores, accelerators, and peripherals.

Furthermore, ICs that are entirely custom at the transistor level can be fully exploited by software. While engineers using off-the-shelf processors can certainly make the most of integrated features, having a truly custom IC means that large portions of bloated hardware, which are not generally needed, can be entirely removed from the custom IC, thus leaving space for more processors, memory, and accelerators. 

This also means that the resulting software written for the custom IC becomes far more coupled to the hardware. Such code can be made to be ultra-efficient, ultra-fast, and reliable, something which can be difficult at times when targeting software for a wide range of different processors.

Microsoft announces new custom silicon ICs

Following the trend of other tech companies, Microsoft has recently announced that it has successfully developed two new ICs that it will use to accelerate its future AI and cloud server capabilities.

The first device, the Maia 100, is an AI accelerator that integrates around 105 billion transistors, which is around 30 per cent less than its competitor, the MI300X AI GPU from AMD. Built using the TSMC 5nm process, it is hoped by Microsoft that the Maia 100 will help to power its future generative AI systems. Furthermore, it was stated that the new device supports sub-8-bit data types, but besides this, not many other details have been released, with the exception of a number of images, and that it is currently being tested on ChatGPT 3.5 to determine its performance. 

 In a Redmond, Washington Microsoft lab, a custom rack hosts the Maia 100 AI Accelerator and its cooling "sidekick" in a thermal chamber. The sidekick, akin to a car radiator, circulates liquid to cool the chips managing AI workloads. Photo by John Brecher for Microsoft.

The technical prowess of the Maia 100 AI Accelerator is particularly noteworthy. It's designed from the ground up to cater to AI tasks, particularly those involving large language models like those used in OpenAI's services. This level of specialisation in the chip’s architecture could herald a new era of AI-optimised hardware, setting a benchmark in the industry.

The second device, the Cobolt 100, is a 64-bit ARM-based CPU incorporating 128 cores. With such a high core count, the Cobolt 100 is highly suitable for Microsoft’s Azures cloud services, especially when considering that the new device is able to provide a performance increase of around 40% compared to ARM devices currently being used by Azure services. As of now, the new device is being tested with Azure services, including SQL and Microsoft Teams. 

 Inside Microsoft's Quincy, Washington datacenter, showcasing the advanced servers powered by the innovative Azure Cobalt 100 CPU, a testament to Microsoft's cutting-edge technology in cloud computing infrastructure. Photo credit: John Brecher, Microsoft. 

Microsoft's commitment to integrating hardware and software is highlighted by Rani Borkar, corporate vice president for Azure Hardware Systems and Infrastructure, who emphasises, "Software is our core strength, but frankly, we are a systems company. At Microsoft, we are co-designing and optimising hardware and software together so that one plus one is greater than two". This statement underscores the strategic direction Microsoft is taking with its custom silicon endeavours.

The sustainability aspect of Microsoft's chip design, particularly the Cobalt 100 CPU, is noteworthy. Built on Arm architecture, known for its energy efficiency, it aligns with Microsoft's broader sustainability goals. This focus on 'performance per watt' demonstrates a commitment to eco-friendly technology solutions, setting a precedent for environmentally conscious hardware development in the tech industry. With Microsoft's emphasis on optimising power efficiency across its datacenters, the Cobalt 100 CPU represents a step forward in reducing the environmental impact of high-performance computing.

What does this mean for the future of custom silicon?

While it comes as no surprise that Microsoft has developed their own custom IC solution, it does go to show that as technology progresses, custom ICs will provide engineers with major benefits. Even though custom ICs are not suitable for all designs (as off-the-shelf components can do extremely well while providing exceedingly high-cost savings), those looking to design extremely high-performance devices that need to reduce energy consumption will certainly benefit from custom ICs.

The development of Microsoft's custom chips is not just a technical achievement but also has significant implications for users. Enhanced efficiency and performance in cloud services, powered by these chips, could lead to more robust and responsive cloud-based applications, directly benefiting end-users and businesses relying on Azure’s cloud ecosystem.

So, going forward

We are unlikely to see custom IC solutions in off-the-shelf PCs and laptops as they will continue to utilise devices manufactured by AMD and Intel, but Apple products will undoubtedly move towards totally custom solutions. 

However, if semiconductor foundries start to open up economic services for smaller production volumes of custom chips, then it is possible that everyday domestic devices may adopt such technologies. For example, IoT devices may see significant benefits to custom ICs thanks to the reduced energy consumption and the ability to only integrate hardware features that are absolutely needed.

Will custom IC services become cheap enough for individuals to use? Unlikely. However, it is possible that future fabrication services could even allow individuals to combine chip dies into custom packages to provide an excellent balance between flexibility and price.

The advent of Microsoft's custom chips marks a pivotal moment in the industry, showcasing a trend towards greater integration of hardware and software. This approach, where companies like Microsoft develop chips precisely suited for their software and services, may inspire other tech giants to follow suit, potentially reshaping the landscape of the cloud computing and AI sectors.

Looking ahead, Microsoft's venture into custom silicon is just the beginning. With plans already underway for the second-generation versions of the Azure Maia AI Accelerator and Azure Cobalt CPU series, the future seems bright for further advancements in AI and cloud computing technology. This ongoing innovation reaffirms Microsoft's commitment to optimising every layer of its technology stack, from silicon to service.