The rise of the AI PC

Since the winter of 2022, AI has become a major topic of discussion. Generative AI, such as ChatGPT, has evolved from being a novelty for creating stories and images to being integrated into various tools and applications. Now, nearly every major writing tool, notes app, to-do list, and help desk chat is incorporating AI.

The term “AI PC” has emerged, and it raises questions about how AI will be integrated into computer hardware. Both Microsoft and hardware manufacturers are embracing this concept by introducing next-generation devices referred to as “AI PCs” or similar names. Everyone is getting on the AI PC bandwagon. But, what is an AI PC? What precisely are the requirements for such a device? I sought to answer this question.

Why is there a push toward AI PCs?

As far as I can tell, there are two factors responsible for Microsoft, and PC hardware makers’, push to optimize their devices around AI.

The first reason is one that followers of the PC industry can probably guess. It’s the emergence of OpenAI (and ChatGPT) and its various chatbot competitors. Microsoft got lucky with AI. Its Azure cloud platform has served as an OpenAIs engine and Microsoft’s early investment in their technology gave them an advantage in the market. Suddenly every Microsoft product, including Windows itself, had to incorporate AI in some fashion. There is some speculation among those familiar with Microsoft that there was an early top-down directive to incorporate AI – somehow – into each business unit, much like how Bill Gates directed each department to leverage the Internet in the late 1990s and early 2000s. Today, we have the Copilot brand signalling to users where AI has been embedded.

What’s most interesting about Copilot is that it’s not a singular product. The Copilot you use on the web is different than the Copilot helping you in Windows 11 and is different still from GitHub Copilot. In this sense, Copilot just means a AI-powered productivity tool that works in concert with users.

The second reason for the AI PCs’ emergence is largely because of Apple. When Apple began making the transition from Intel-based Macs to its homegrown Apple Silicon, it was met with both excitement and skepticism. Well, I can report that after several iterations of these Apple hit a home run. Not only are the chips considerably more powerful (even at the low end) than similar Intel-equipped Macs, but they perform at that level while using a fraction of the power.

Apple was forward-thinking when it rolled out Apple Silicon. It outfitted each chip with a neural processing unit (NPU) – future-proofing it for the AI age. Tim Bajarin from Forbes explains it best.

The M1 chips caused headaches for PC semiconductor vendors and companies that relied on Intel processors to power their computers. The key reason is that Apple invented a way to deliver powerful processors yet have low power demands for use in all MacBooks. Apple also integrated an Neural Processing Unit, in the M1 series, which, from the beginning, gave Apple an edge in AI processing capabilities.

For various reasons, Intel had not been able to match Apple’s low-power functionality until recently. It gave Qualcomm an edge with its Snapdragon multi-chip set to revive the Windows on ARM program they introduced in 2016. In the meantime, Apple’s MacBook started gaining ground in the mobile PC space. It increasingly threatened traditional PC makers who used Intel’s X86 architecture in its PCs.

Recent data from Canalys, as reported by Seeking Alpha, underscored Apple’s impact on its PC competitors: “The number of artificial intelligence-capable PCs shipped during the second quarter of 2024 hit 8.8M, with the Apple Mac accounting for 60%, while 39% had Microsoft Windows operating systems, according to the latest data by Canalys.”
QUOTE

~Tim Bajarin, Forbes

Similar to Apple, Microsoft had been trying to make Windows run on ARM-based processors for years, but their record had been spotty. The first Surface RT and later the Surface Pro X were less than ideal experiences, as the software was slow and buggy. Today, the offerings are much more compelling, with Surface devices only being offered with ARM-based Qualcomm processors. There are no more Intel options. I recommend reading Paul Thurrott’s review and Andrew Cunningham’s ArsTechnica review of these devices.

As with the Yoga, my day-to-day experience using Surface Laptop 7 has been nothing short of phenomenal: Every single time I open its lid, the display fires on immediately. As noted below, the battery life always lasted a full day. During all that uptime, Surface Laptop handled everything I threw at me, from the traditional productivity apps I use to Visual Studio 2022 Preview—which I’m using for my Modernizing .NETpad 2024 project—with effortless, silent, and consistent performance.

~ Paul Thurrott, Thurrott.com

So, the AI PC is a product of Microsoft’s almost accidental success with OpenAI, combined with the desire to have their computers be competitive with Apple’s offerings.

Defining an AI PC

Now that we know how we got here, users might be asking themselves “What qualifies as an AI PC?” Well, that depends on who you ask.

Currently, there are two competing names for AI-enabled Windows PCs which makes things doubly confusing for users. AI PC is the broader term and applies to any PC that leverages a modern CPU combined with a neural processing unit (NPU) the latter of which is for handling AI-intensive tasks. A select number of these PCs – those with Qualcomm’s ARM-based chips – are branded as “Copilot+” PCs which Microsoft’s Yusef Mehdi defined in a company blog post.

Copilot+ PCs are the fastest, most intelligent Windows PCs ever built. With powerful new silicon capable of an incredible 40+ TOPS (trillion operations per second), all-day battery life and access to the most advanced AI models, Copilot+ PCs will enable you to do things you can’t on any other PC.

~ Yusef Mehdi, Microsoft

In the future, the Copilot+ brand will be extended to Intel and AMD-based computers running their upcoming chipsets. For now, it’s just Qualcomm computers. That sounds like an exclusivity agreement to me.

Software features

From a Windows feature standpoint, Wired has provided a good summary of the Copilot+ “exclusive” features.

• Automatic Super Resolution: This tool elevates gaming visuals by automatically adjusting the screen resolution in real-time during gameplay
• Live Translate: Offering real-time audio translation, this feature captures sound from streaming sources or your microphone and translates it into any of 44 available languages, displaying the translated text directly on your screen.
• Paint Cocreator: Integrated into Microsoft Paint, this AI-assisted feature allows you to describe an image and sketch a basic version.
• Restyle Image and Image Creator in Photos: Similar to Paint Cocreator but within the Photos app, this feature lets you transform existing photos into different styles—like turning a snapshot into a cartoon or an oil painting—or create entirely new images from scratch using AI.
• Windows Studio Effects (Full Experience): Enhancing video calls, this suite offers automatic adjustments for lighting and sound, introduces new visual filters, and includes an “eye contact” feature.
• Recall: Perhaps the most controversial addition, Recall logs your computer activities to create a timeline, helping you rediscover files or content you’ve previously accessed but can’t quite remember. Due to significant privacy concerns—some critics have labelled it “spyware”—its rollout was delayed when Copilot+ PC launched. The latest updates indicate it will re-emerge in preview form with enhanced privacy protections by October 2024.

Hardware features

Based on the list of features above, it might not be clear to users why their high-end laptops can’t already do these things. Through future updates, it’s possible that older devices with high-end graphics cards can undertake these operations, but Microsoft is taking a different approach.

An AI PC is a high-performance computer designed to execute complex AI workloads locally rather than relying solely on cloud-based services. As it turns out, constantly accessing AI via the cloud is an expensive endeavour and it’s not clear that there’s a compelling business case for having large language models run exclusively from the cloud. The cost of running these services in data centres is reflected in the $20 US cost that services like OpenAI, Anthopic, and Microsoft charge to use their premium AI features.

What does running AI services locally, on-device, mean? This includes tasks like running generative AI models (e.g., Stable Diffusion), handling sophisticated data analysis, operating advanced chatbots with local language models, and performing intricate simulations. By processing these tasks on-device, AI PCs aim to reduce latency, enhance privacy, and improve energy efficiency [ZDNet].

As a result of these requirements, this is where the NPU comes in. The NPU sits at the heart of these AI PCs. NPUs are specialized processors designed specifically for AI computation. They excel and doing the heavy lifting for AI-related tasks that graphics cards have traditionally been used for. If you have a gaming PC, you have a top-notch AI machine. But, graphics cards use enormous amounts of power whereas NPUs are better from a performance-per-watt perspective. This power efficiency means battery life can be extended in portable devices. Want a thin and light Windows laptop with all-day battery life and AI features? Then an NPU is necessary. Furthermore, thanks to NPUs local processing allows AI models to learn and adapt to individual user preferences securely, enabling more proactive and customized assistance.

When considering the performance of NPUs, there are several performance metrics to keep in mind. The speed of these chips is measured in TOPS – or “trillions of operations per second.” Higher TOPS indicate the ability to process more data simultaneously, which is essential for AI tasks [PCWorld]. Below are the top scores (pun intended) for some of the current, and upcoming, CPUs on the market.

Current NPU Offerings:

• Qualcomm Snapdragon X Elite and X Plus: Equipped with the Hexagon NPU, delivering up to 45 TOPS [PCMag].
• AMD Ryzen 8040 Series (“Strix Point”): Features an NPU capable of 39 to 50 TOPS, edging out Intel’s upcoming processors [PCMag].
• Intel Core Ultra (Meteor Lake): Provides around 34 TOPS, with upcoming “Lunar Lake” processors expected to reach 48 TOPS [PCWorld].

So, after scouring the Internet for articles that provide decent summaries of AI PC hardware requirements, I’ve compiled a list of basic functions that seem to be necessary for a hardware manufacturer to claim they have an AI PC that meets Microsoft’s “Copilot+ PC” standard.

These computers must have…

1. Dedicated NPU (as mentioned above): A specialized processor designed for AI tasks, offering high TOPS performance to handle complex computations efficiently [Tom’s Guide].
2. Advanced CPU and GPU: Modern processors and graphics units that collaborate with the NPU to optimize overall performance and energy consumption [ZDNet].
3. AI-Optimized Software Integration: Support for AI assistants like Microsoft’s Copilot, potentially featuring dedicated hardware buttons (e.g., a Copilot key on the keyboard) for quick access [PCMag].
4. On-Device AI Processing Capabilities: The ability to run AI models locally, minimizing the need for cloud services and enhancing data privacy and responsiveness [PCWorld].

When you combine all these things, you have an AI PC!

The Future of AI PCs

Having nailed down a definition for the AI PC, including its software requirements and hardware requirements, I think it’s important to ask where this AI PC trend is going.

As I was reaching some conclusions, Paul Thurrott published a (premium member’s only) article where he outlined some of the things I had planned to say. He and I have both concluded that the idea of an AI PC is temporary. His piece compares Microsoft’s Copilot+ branding to previous PC innovations, arguing that the branding will eventually become unnecessary as all PCs ship with the specs to be considered AI-compliant. Microsoft’s rollout has been messy – with the delay of its extremely controversial Recall features. In contrast, Apple’s (and Google’s) approach have been more deliberate. Apple still leads in terms of at least addressing user privacy.

Microsoft’s rush to market has led to mixed results, and the promised on-device AI features haven’t provided compelling reasons for customers to invest in new hardware. Moreover, competitors like Apple and Google have taken a more deliberate approach to AI, with Apple’s methodical integration being praised for its responsibility, and Google embracing a more chaotic, feature-driven strategy.

From a user perspective, it’s not clear how useful the AI-specific features are at this point. The offshoot of AI-enabled Windows-based PCs is that their power-efficient chips allow for a Mac-like experience while retaining the choice afforded by the Windows ecosystem. If you view AI as a feature that enhances products, rather than a product itself, AI-enabled PCs could sway users. But, I don’t believe the average user understands the benefit of an NPU.

For example, if a user is already an avid user of ChatGPT Plus or Anthropic’s Claude, is it clear to that person how an AI PC would help them? I don’t believe so. The benefit of an NPU is that many of the cloud-based AI features can be offloaded onto the device. That could be beneficial not only to Microsoft (as it could cut down on the costs of running the cloud) but also be useful for users. In this scenario, AI features could work on a computer with a spotty Internet connection. Image generation queries might be produced faster and could work inside a variety of applications. A local AI could automatically tailor the energy usage of your computer on a minute-by-minute basis depending on your usage thereby further optimizing battery life. If marketed correctly, the AI PC could be a compelling package for certain power users.

Lastly, the biggest hurdle for the AI PC – even if marketed correctly – is the issue of privacy. The Recall debacle is a case study of how not to sell an AI feature that runs on the device. Many users and tech pundits viewed recall as spyware – a system that tracks your every move and silently builds a database of your actions in case you need to “recall” something later. So far, Apple Intelligence (a complete coopting of the AI acronym) is the only feature set that made clear the importance of user privacy. Without a light touch, the AI PC could be a class of devices that take two steps forward and one step back.