Advancing Efficient AI Workflows

Your online order arrives damaged, so you request a refund. What often follows is a artificial intelligence workflow involving multiple AI models: One model checks your request against company policy, another analyzes the image you uploaded, and yet another drafts a response.

Since many of today’s AI applications no longer rely on a single model, engineers must custom-build these increasingly complex products — deciding what model should handle each step, and how they should coordinate. Researchers at the Harvard John A. Paulson School of Engineering and Applied Sciences (SEAS) have something more streamlined in mind.  

A team led by Minlan Yu, the Gordon McKay Professor of Computer Science; and Michael Mitzenmacher, the Thomas Watson, Sr. Professor of Computer Science, with postdoctoral fellow Rana Shahout and Boston University software developer Hayder Tirmazi, are developing a new AI framework called Orla that simplifies building and running AI workflows while automatically optimizing them for cost, accuracy, and speed. Engineers simply describe the workflow they want, and Orla determines how best to execute it.

The project was recently funded by the nonprofit Laude Institute with a Slingshot award, which supports computer scientists building foundational AI infrastructure. 

For the Harvard team, Orla is part of a broader effort to bring together computer systems and theoretical ideas to tackle emerging AI challenges. Previous work by the same group focused on scheduling, load balancing, and memory management for AI inference. 

“As AI evolves from individual models to teams of collaborating agents, we’re applying those same ideas to managing entire AI workflows,” said Shahout, lead author of a recent demo paper on Orla presented at the ACM Conference on AI and Agentic Systems.

In tests, Orla reduced computing costs and response times without sacrificing the quality of the answers. As AI applications continue to grow in complexity, the researchers believe systems like Orla could make them more practical to build and operate at scale.

Learn more: 

Orla: A Library for Serving LLM-Based Multi-Agent Systems
Don’t stop me now: Embedding based scheduling for LLMs 
Queueing, predictions, and large language models: Challenges and open problems
Fast inference for augmented large language models