When someone says “computer,” the image that comes to mind could be a desktop machine, a laptop, or even a tablet or mobile phone. But whichever device you envision, they all have a lot in common—for instance, if you click a button with the mouse or tap the screen, a result occurs.
But how would you input information if your computer had no screen, no mouse, and no keyboard?
That’s one of the challenges of spatial computing, where the human and machine interact in the real world, using tools like augmented reality, cameras, and depth sensors. It’s a complicated problem that Ishan Chatterjee, S.B. ’16, is exploring on a daily basis through his work at Microsoft and the University of Washington.
Chatterjee is a hardware architect for the Microsoft HoloLens, a head-mounted holographic device that enables mixed-reality interactions, those that blend the physical and virtual world in real time.
He focuses on display and optics incubation, imagining new types of displays that could make the hardware more compact and the entire experience more comfortable, immersive, and realistic for the user.
“In this role, I’ve been functioning as one part engineer, one part designer, and one part investor, looking at the cutting edge of the nascent augmented reality industry,” said Chatterjee, an electrical engineering concentrator at the John A. Paulson School of Engineering and Applied Sciences. “The design creativity is what I enjoy the most. It is looking at what’s available, which might be a few dots in the design space, and then trying to connect them to envision the bigger picture.”
When Chatterjee arrived at Harvard, he was already thinking about the bigger picture. He knew he wanted to be an engineer, and originally considered bioengineering because of its human-centered focus.
But the rapid pace of innovation in computer science was a draw too powerful for him to pass up, so he shifted to electrical engineering in his sophomore year.
A summer internship in a research lab at Carnegie Mellon University introduced him to the fascinating challenges of using gaze and free-space gesturing to manipulate three-dimensional virtual objects.
At the same time, he was working with the Harvard chapter of Engineers Without Borders, where he learned valuable lessons about the potential impact of engineering projects.
“As we were building this water distribution system in the Dominican Republic, the social aspect of how the community engaged with this engineering project were potentially even more pivotal to the success of the project than how you laid the pipe,” he said. “These aspects together reinforced this perspective that centering on the user is truly important for the design of engineering systems.”
Chatterjee held onto that human-centered focus the following summer, when he embarked on an internship at Microsoft. When he interviewed for the position, the interviewer wasn’t able to say much about the project, but hinted that it seemed to be right up Chatterjee’s alley.
This image shows how the Microsoft HoloLens could be used in a manufacturing setting. (Photo courtesy of Microsoft)
He quickly immersed himself in the project, which turned out to be the first iteration of the HoloLens.
“This is really powerful because it has the potential to democratize computing beyond the white-collar information workers who are sitting at a desktop,” he said. “It’s not like car mechanics or surgeons or front line workers don’t need computing—having information at their hands is really important to making their tasks more productive and efficient—but there hasn’t existed a computing paradigm that has supported them in their endeavors.”
His work on the HoloLens hardware also enabled him to explore emerging technologies, like the concept of “holoportation,” where people from separate locations interact with one another in the same physical space through the use of holograms.
Chatterjee’s fascination with pushing what is possible led him into the more incubation and architecture-focused role he holds now on the HoloLens team.
“As a technology like augmented reality rolls out, there is this constant need to square between what is possible with the hardware and what provides value to the user,” he said. “We can all imagine a utopia 20 years away, but how can we provide value today with the technologies that we have? How can we sustain a business as we keep building this technology?”
Always a tinkerer, Chatterjee built this Nixie Tube Clock just for fun.
To dig deeper into some of the murky questions of where spatial computing technology is going in the future, Chatterjee began pursuing a Ph.D. in the Ubiquitous Computing Lab, led by Shwetak Patel, the Washington Research Foundation Entrepreneurship Endowed Professor in Computer Science & Engineering and Electrical Engineering, at the University of Washington in 2019.
“The future of augmented reality is more ubiquitous than it is today. We see people using these in contexts that are more social and public, so we see the need for lighter-weight hardware with more subtle interactions,” he said. “You wouldn’t necessarily be sitting in a café making gestures like you’re in ‘Minority Report.’ You wouldn’t want irrelevant virtual objects in your field of view while driving or talking to your friend, that would be intrusive or even dangerous. So what would this future of input look like? That’s one of the questions I’m exploring in my Ph.D. now.”
It’s a future he hopes to continue to dig into by someday running his own lab at company or in academia where he can imagine solutions with both a technical and user experience focus.