Serving AI faster and more efficiently

Which request should run next, where should it run, and what resources should it receive?

I study how to allocate compute, memory, and power across AI workloads. This includes scheduling requests, balancing work across models and experts, and deciding how much reasoning effort to spend on a question. I develop methods that use predictions and feedback to adapt these decisions as execution unfolds.

Can signals from inside a model help the system make better decisions?

A recurring approach in my work is model-informed systems. I use internal model signals to predict remaining work, guide expert routing, and decide when a reasoning branch can produce an answer. These signals connect the model’s progress to the system’s execution decisions.

Projects

Scheduling and memory

  • TRAIL — Predict remaining output length from internal model representations to guide memory-aware scheduling.

  • MARS — Schedule tool-using LLM requests together with decisions about memory during tool calls.

  • Prefix- and length-aware scheduling — Account for both shared prompt prefixes and predicted output lengths when distributing requests across GPUs.

Routing and load balancing

  • LASER — Use expert-routing scores to balance work among suitable experts without retraining the model.

  • SCORE — Route requests across models using predicted quality and length, subject to cost and latency constraints.

Reasoning and communication

  • DUCHESS — Use internal model signals to decide when to extract an answer, branch out, or continue reasoning.

  • Predictive Scheduling — Allocate a shared reasoning-token budget using predictions of question difficulty and expected benefit.

  • SVR-MAD — Use evidence from agent debate to guide communication and reduce unnecessary exchanges.

Power and energy

  • PALS — Coordinate GPU power limits and batching to improve energy efficiency while meeting performance targets.

Related work on prediction costs, queueing, and robustness: SkipPredict; Queueing, predictions, and LLMs; Jumping the Line.

Co-designing AI applications and systems

How can application structure and goals inform the design of the systems that execute them?

I study how to design the AI stack as a whole, bringing together application logic, models, serving systems, storage, and databases. Treating an LLM as an API call hides opportunities to optimize across these layers. By co-designing them, we can make decisions about data access, computation, and execution together.

These layers make coupled decisions. A database or query engine determines which data reaches a model. A workflow determines which calls can run in parallel and which share context. The serving system determines how long that context occupies GPU memory. Optimizing each layer independently can save work in one place while creating data movement, recomputation, or waiting elsewhere.

I work on abstractions and execution policies that make these dependencies visible across layers. This lets us consider operator order, model choice, data access, context reuse, and scheduling together, with application quality, latency, and cost as shared constraints.

Projects