A technical evaluation using data from a recent scientific-user experiment demonstrated how the U.S. Department of Energy’s (DOE) Energy Sciences Network (ESnet) enables Facility for Rare Isotope Beams (FRIB) scientists to send large amounts of data across the country, analyze it in near real-time, and return results, enabling quicker data-informed experimental choices. The high-speed network is funded by the DOE Office of Science (DOE-SC).
Scientific-user experiments performed at FRIB can take a significant amount of data, which requires considerable computing resources to interpret.
FRIB is located at Michigan State University (MSU), while the National Energy Research Scientific Computing Center (NERSC), one of DOE-SC’s primary scientific computing facilities, is housed at Lawrence Berkeley National Laboratory (Berkeley Lab) in California—2,320 miles apart. Previously, if you wanted to integrate the two facilities for your work, you were out of luck.
Enter ESnet, a science research network that can move data at the speed of science. FRIB’s recent 100-gigabit-per-second connection to the ESnet backbone—4,000 times faster than typical 25-megabit-per-second (Mbps) home Internet—allowed ultra-high-speed transfers of terabytes of research data between where the experiment was performed (FRIB) and where it can be analyzed rapidly (NERSC). This instance was the first time at FRIB that scientists used an analysis model to take advantage of capabilities provided at other locations outside MSU to do data analysis faster than could be done locally, and other FRIB scientific users are enthusiastic about the network’s potential to speed their analysis of large quantities of data.
Collaborative experiment demonstrates network success
The experiment was led from Berkeley Lab by Heather Crawford, Berkeley Lab staff scientist.
“As a collaboration, we were fortunate enough to have the opportunity to take the first beam for a user experiment at FRIB in May of last year,” said Crawford. “It’s now really exciting to also have the chance to demonstrate, with that first data, the power of combining FRIB, ESnet, and NERSC for our analysis pipeline.”
Sean Liddick, associate professor of chemistry at FRIB and in MSU’s Department of Chemistry and FRIB associate director for experimental science, was co-experimenter.
“We collected the data at FRIB, and it was analyzed across the country at NERSC, then data was returned to FRIB,” said Liddick. “The importance of the network is to make that a completely transparent process for the scientific user. A lot of people had to work together in order to make this happen.”
The analysis experiment’s success involved coordination between scientific software and IT groups at FRIB, MSU IT, Berkeley Lab, NERSC, and ESnet; scientific users—led by Crawford; and those providing detection equipment for the FRIB Decay Station Initiator (FDSi). FDSi is a device supported through collaboration among multiple groups, including FRIB, the University of Tennessee Knoxville, Argonne National Laboratory, and Oak Ridge National Laboratory.
The FRIB experiment focused on the decay of exotic, short-lived isotopes produced in the lab at FRIB. Isotopes are different forms of elements found in nature. Isotopes of each element contain the same number of protons, but a different number of neutrons. These isotopes have far too many neutrons to be stable, and all of them will decay within fractions of a second (on the order of tens of milliseconds). During the decay process, multiple particles are emitted including electrons, photons, and neutrons. Scientists detect these emitted particles and reconstruct the decay path, learning about the properties of the isotope before its decay, and about the structure of the isotope(s) it decays into.
“If we can have a predictive model of the nucleus and really understand how these things fall apart,” said Liddick. “We can then apply that knowledge to other nuclei that we haven't been able to study yet—in stars, on Earth, anywhere. Terabytes of data can be taken from the experiment and it has to be processed quickly to ensure the best possible experimental outcomes.”
“It was a great collaborative prototype,” continued Liddick. “We transferred 13 terabytes (TB) of data across the country at speeds of 4 TB per hour, analyzed the data using the Perlmutter supercomputer at NERSC in approximately 90 minutes, and sent data back to FRIB. It was an impressive demonstration of what we can do in the future.”
Building the network connections
Clint Jones, Business Information Technology (IT) Department manager at FRIB, helped facilitate the connection process in 2021. In spring 2023, FRIB was connected via two 100 Gbps circuits to ESnet’s next-generation network, ESnet6.
“There was a lot of configuration involved by both FRIB IT and ESnet engineers,” said Jones. “FRIB IT worked closely with MSU network staff to get fiber from the FRIB datacenter to MSU’s network border. Then from there, fiber was connected to the statewide Merit Network. This configuration provides uninterrupted fiber from our server room in the FRIB building all the way to Chicago.”
Jason Zurawski, science engagement engineer at ESnet, led the effort to document the network requirements for FRIB. He also led the team that helped design the FRIB perimeter and how it will fully utilize the capabilities of ESnet.
"FRIB includes several high-intensity data use cases that rely heavily on network transfers,” Zurawski said. “We worked closely with FRIB to understand the workflows, access requirements, and timelines to design and implement appropriate data mobility solutions. We are confident that this groundwork will enable these approaches to scale to the current, and future needs of the FRIB facility."
Optimizing the connection to ESnet is an important step in serving scientific users globally who can use FRIB to conduct such breakthrough, data-intensive discovery research.
“The ESnet connection enables us to take advantage of other computing resources around the world at rates as fast as if they were right here in the building,” said Jones. “That's how fast this network is.”
Michigan State University operates the Facility for Rare Isotope Beams (FRIB) as a user facility for the U.S. Department of Energy Office of Science (DOE-SC), supporting the mission of the DOE-SC Office of Nuclear Physics. Hosting what is designed to be the most powerful heavy-ion accelerator, FRIB enables scientists to make discoveries about the properties of rare isotopes in order to better understand the physics of nuclei, nuclear astrophysics, fundamental interactions, and applications for society, including in medicine, homeland security, and industry.
The U.S. Department of Energy Office of Science is the single largest supporter of basic research in the physical sciences in the United States and is working to address some of today’s most pressing challenges. For more information, visit energy.gov/science.