What If Every SQL Query Could Update Incrementally? (with Lalith Suresh)

About this episode

There's a problem that's bugged the database industry since the 1980s: you run an expensive query over millions of rows, cache the result, and then a single new row arrives. Logically that's one small update, but most engines throw the cached answer away and recompute everything from scratch. Some will handle changes incrementally, but only for "simple" queries - and the rules for what counts as simple are arbitrary and brittle. So can you incrementally maintain *any* SQL query, no matter how complex? For decades the answer was no. Then an award-winning paper called DBSP proved that the answer is yes - all queries are simple enough. Joining me to explain how that works is Lalith Suresh, CEO of Feldera, the company built on top of DBSP. We start with the problem itself, then trace how a group of VMware researchers arrived at it from the unlikely direction of Kubernetes and network control planes. Lalith walks through Z-sets, the weighted data structure that turns database changes into something you can add and subtract, and the four DBSP operators - including one borrowed straight from digital signal processing - that let you compile any SQL program into an incremental version dete