HPE and Department of Energy Prepare for the Second Phase of Newest Supercomputer

Kestrel uses HPE Cray EX supercomputers. The system is made of individual compute “blades,” which carry all the components that make the supercomputer go: central processing units, fabric connections, printed circuit boards, and cooling and power components. There are also blades that hold HPE’s Slingshot switching elements. Cooling is built into each blade.

Liquid cooling loops running through the compute infrastructure cool the cabinets and components. A cooling distribution unit cools the liquid itself and removes heat from the system via a heat exchanger with data center water. The incoming water can be as warm as , which means chilling isn’t necessary and less electricity is required.

HPE manufactured the boards and blades using chips from Intel, NVIDIA and AMD. The onboard communications infrastructure and cooling system are proprietary.

“A lot of that stuff is unique to HPE,” Damkroger says.

MORE FROM FEDTECH: The HPE Aruba AP22 maximizes device bandwidth in the office.

Preparing for Kestrel’s Second Phase: GPUs

Kestrel will have 2,436 compute nodes available for high-performance computing tasks.

Phase two will begin in December with the installation of 132 graphics processing unit nodes, each with four NVIDIA H100 GPUs. Originally created for video rendering in computer games and simulators, GPUs have revolutionized supercomputing.

A CPU does serial tasks at very high speed. At best, it can handle a handful of operations at once. By contrast, a GPU uses parallel processing to do multiple calculations simultaneously and can handle thousands of operations in an instant.

“There’s been a huge push toward using GPUs,” Damkroger says. “They’re easy to program, so you can go faster.”

AI applications require massive number crunching. It turns out GPUs are uniquely suited to the task, as vast quantities of data are run through neural networks to train AI applications such as driverless cars and ChatGPT.

The downside: GPUs use a huge amount of electricity. That’s one reason it’s unlikely you’ll hear your AI overlord banging on your front door anytime soon. A human brain imagining a nice day at the beach costs nothing. Meanwhile, a supercomputer can gobble up to 30 megawatts of power per year — by some estimates, the same amount used by a small city.

DIVE DEEPER: Why agencies should be part of the AI proof-of-concept process.

“Kestrel represents new capabilities so that we can do better science, faster,” says Aaron Andersen, Kestrel engineering lead at NREL, in a video. “There's an awful lot of research questions and problems that just need more capability or need the GPU capability to make considerable progress.”

Kestrel is one of a new breed of machines that allows for far deeper research than previously possible.

“You can do much more with a larger compute size,” Damkroger says. “You can get to a small enough geometry to put an individual cloud in atmospheric modeling. You can fit more into a computer to see the interactions between things.”

Brought to you by: