Microsoft recently announced the release of “Copilot+ PC’s” – computers specifically designed to have generative AI as a core part of the user experience. While a lot of emphasis has been put on the software side of generative AI, the hardware evolution of generative AI is just as important. It will also has major implications for business use of AI now and in the future.
Here’s a brief overview – and how it should shape your business’ AI strategy.
GPU: The Main Hardware of AI
You may be familiar with the CPU – the central processing unit, or main brain of the computer. It coordinates all of the software and hardware activities that drive your computing experience.
Modern computers also frequently have an additional component, called a GPU. GPU stands for “graphics processing unit“, and was initially designed to accelerate graphics tasks by running large quantities of calculations simultaneously. As AI researchers focused on scaling ever larger systems, they realized that GPU’s were also an ideal platform for AI systems due to the similarity between calculations in both AI and graphics. Since then, GPU’s have a been a central part of AI training and deployment.
GPU’s are both a major accelerator and bottleneck to generative AI systems today. While two companies – AMD and Nvidia – dominate the GPU market today, only Nvidia has specifically leaned into AI as an important use case for their products. Along with consumer GPU’s, they offer multiple classes of high-powered GPU targeted toward AI applications. Due to both COVID and the release of ChatGPT in November 2022, access to GPU supply has been a major challenge in recent years. The ability to obtain these GPU’s has become a proxy for an organization’s ability to train and deploy their own AI models. Unfortunately, this situation doesn’t appear to be changing any time soon. Nvidia is already teasing the next generation of Nvidia GPU’s, specifically highlighting their ability to handle ever larger generative AI models with 25x greater energy efficiency.
Apple Silicon: AI’s Close Friend
However, Nvidia GPU’s aren’t the only game in town. In 2020, Apple introduced their own custom-designed chips to power their products. These new chips contain CPU, GPU, and a component called a “Neural Engine”, designed specifically to deliver high performance with AI-type workloads.
With the growing availability of highly-capable generative AI systems that you can run on your own computer, Apple computers have been standout performers. While Windows and Linux users have gotten poor-to-decent performance on consumer-grade GPU’s, Apple users have consistently seen excellent results when running open-source models on their own machines. This has been great for individuals with top-end Apple computers, but has not yet caught on broadly in the business world.
The Future, Part 1: Microsoft and NPU’s
This AI performance gap between Apple and Windows has not gone unnoticed by Microsoft. On May 20th, Microsoft announced a new class of hardware for Windows called “Copilot+ PC’s”, which feature AI throughout the operating system. Key features include:
- Real-time transcription and translation of any video feed (Youtube, Zoom, etc.) running directly on your computer,
- Using generative AI to create images in real time based on your sketches,
- A dedicated Copilot button to quickly use Copilot AI for a range of applications,
And much, much more. (This also explains, in part, why OpenAI recently announced a desktop app for Mac but not Windows: Microsoft was already building it!)
Central to these new AI-enabled PC’s is what Microsoft calls the NPU, or “neural processing unit“. It expands on GPU’s highly parallel design and uses Microsoft’s own machine learning architecture to accelerate AI applications directly on the device. The new Copilot+ PC’s are said to contain 40 local AI models to start with, along with the ability to leverage other models, like ChatGPT, as needed.
The Future, Part 2: Custom AI Hardware
While NPU’s are an important development due to Microsoft’s reach, we’ve already seen the potential of hardware custom-designed for AI. For example, AI service Groq uses their own custom AI hardware called LPU’s (for “language processing unit”) which can deliver results 2x faster than the next best service and 10x faster than models run on Azure or AWS. The Groq team has a history with this sort of technology; the company founder designed an earlier AI-focused chip for Google before striking out on his own.
And this is just the beginning. It has only been a year and a half since the release of ChatGPT and the demand for AI services continues to grow. While systems like Groq are now only available as a service (you can’t run Groq chips on your own computer yet), the interest in running bigger models faster and locally will almost certainly drive major hardware innovation over the next several years.
What the Future Hardware of Generative AI Means For Business
While this is helpful background, what does it mean for businesses interested in AI today?
There are two key takeaways:
- There’s a saying in the AI world that “the AI you use today is the worst AI you’ll ever use.” The capabilities of these systems will improve in every dimension over time, and quickly. If there is something you use it for today, it will be cheaper and faster tomorrow. If AI can’t do something today, try again in a few years.
- AI is a fast-evolving space, and you should be careful when making contracts and commitments for AI resources – because even better options will be available soon. This means: don’t tie yourself to a specific AI service provider or buy hardware specifically for AI unless that is critical to your business today. Keep your switching costs low and your options open.
This last piece is even more important in the context of the Copilot+ PC’s. While they may offer a lot of powerful AI functionality in a Windows-friendly package, their new systems are still largely untested in the commercial market. Surprises are common in new hardware, usually for the worse. You can be an early adopter if it’s critical for your business, but I would advise you to wait for another hardware generation or two before purchasing them for your business.
Stay informed on the most important AI news in under 30 minutes a month by signing up for the Executive Summary newsletter by AI For Business.
If you liked this post, have future posts delivered straight to your inbox by subscribing to the AI For Business newsletter. Thank you!