AI Inference Is a Data Movement Problem: Microsoft's Saurabh Dighe on Maia 300

About this episode

Microsoft's Maia 300 is a fully co-designed system, optimized to solve the memory-bound "data movement problem" of emerging agentic AI workloads.  Saurabh Dighe, Corporate VP of Azure Hardware Systems at Microsoft, sits down with Austin to detail the architecture, from its TSMC N2P chiplet design enabling 18 TB/s of HBM4 memory bandwidth to its use of AI agents for automated kernel optimization.  They discuss how this full-stack approach aims to maximize intelligence per watt and per dollar for Microsoft's planet-scale services. This episode is presented by @CrusoeAI. Crusoe's Serverless Fine-Tuning trains an open model like Llama, DeepSeek, or GLM on your own data, and you pay per million tokens, not per GPU hour. About $2 buys roughly 5M training tokens on smaller models. Get $5 in credit to start: https://semidoped.com/crusoe Key Takeaways: - Microsoft views AI inference as a data movement problem, not a compute problem. The energy cost of moving data to feed the compute engine can be two orders of magnitude higher than the compute itself. - The design of Maia 300 is driven by the shift from chat to agentic AI, which creates memory-bound workloads. The system delivers a 7x