#PandemicGoals: Building a record-breaking observatory while locked down in Chile

Mar 30, 2022

by Bryné Hadnott, KIPAC Data Curator & Storyteller

Giant Eye in the Sky

Sometime this fall, the Legacy Survey of Space and Time (LSST) camera will be delivered to Santiago on a 747 jumbo jet and trucked to the Rubin Observatory Summit Facility. Located nearly nine thousand feet above sea level in the Andean foothills—about two hours from Chile’s second-oldest city, La Serena—the observatory will house an 8.4 meter (almost 28 feet!) telescope containing the largest digital camera ever built. Each night, the SUV-sized camera will collect thousands of wide-field images of the southern sky, looking further back into the history of the universe with each exposure.

Commissioning engineers and scientists from the SLAC National Accelerator Laboratory (SLAC) have been developing novel ways to handle the many technical challenges that come with building a Guinness World Record-breaking observatory.

“In a single visit, we look at three and a half degrees across, or ten square degrees on the sky. And we repeat those visits quickly so that we look at the entire sky every week and repeat for ten years,” says Kevin Reil, SLAC staff scientist and the commissioning scientist for LSST.

The LSST camera focal plane will be able to image an area over forty times the size of the full moon.
The LSST camera focal plane will be able to image an area over forty times the size of the full moon. (Credit: Rubin Obs/NSF/AURA.)

 

In order to capture the entire southern sky in one week, the Simonyi Survey Telescope will be able to pivot ten degrees per second, while the camera reads out all 3,200 megapixels of each image in just two seconds. During the first year of its ten-year survey, the LSST camera will collect more than a petabyte of imaging data. That’s 1015 bytes or a thousand of the new one-terabyte iPhone 13 Pros.

“The amount of data that's going to be coming off of the telescope is the real challenge for building this observatory,” says SLAC staff engineer, Boyd Bowdish. “It's not really building the mirrors or the camera, which are incredibly complex and ‘drive us crazy’ kind of projects. The data collection and transfer is beyond state of the art. It is going to create new technology.”

Practice Makes Perfect

To prepare for the incoming avalanche of data, engineers have been developing a miniature version called the commissioning camera, or “ComCam.” Instead of over three billion pixels distributed across 189 sensors, ComCam will have nine sensors comprising about five percent—a mere 150 million pixels—of the camera’s total focal plane. To make up for the lower mass, the team installed a cage of black metal around the mini-camera.

Engineers prepare to install ComCam (bronze/black tube, bottom center) onto the Integrating Structure (cyan tube, top center).
Engineers ready ComCam (bronze/black-colored tube, bottom center) for installation onto the Integrating Structure (cyan-colored tube, top center), which will then be attached to the telescope. (Credit: Margaux Lopez.)

 

“For testing requirements, we have a full-size camera—full mass, all of the utilities and interfaces are the same—that we’ve been using for testing for a long time,” explains Reil. “The idea then, is if we build a commissioning version of the camera where only the size of the focal plane and optics are scaled down, we can verify a lot of requirements before the camera gets here.”

ComCam will be used to conduct pointing exercises, practice camera calibration maneuvers, and test the complicated data pipeline that will whisk images from the observatory in Chile to the data processing facility at SLAC.

“Then, once the camera actually arrives in Chile, we have about six months of putting it together and doing post shipment testing to make sure it’s still working as expected before we actually put it on the telescope,” says SLAC engineer, Margaux Lopez.

Lockdown Realities

Ideally, the commissioning engineers would have operated out of both La Serena and the IR2 clean room at SLAC, coordinating with international contractors and Chilean technicians to assemble the many subsystems that will support the telescope. But in the spring of 2020, the entire city of La Serena was placed under quarantine.

“I arrived when the country closed itself down for COVID,’ says Bowdish. “You would get one hour a week to go shopping or go to the pharmacy and that was it.”

Nevertheless, the team installed the massive telescope mount assembly (TMA), a 300-ton support structure floating on a hair-thin layer of near-frictionless, hydraulic oil. The lack of friction allowed just a handful of engineers to manually pivot the TMA, but in the future, a motorized system will be able to quickly point the telescope to any position in the night sky.

“I helped Asturfieto [a Spain-based contractor] rotate the telescope a few times when they needed an extra hand,” says Shawn Osier, a mechanical engineer at SLAC. “I was amazed that ten of us could rotate the entire TMA.”

The Telescope Mount Assembly (TMA) is 300 tons, but due to its unique design, can be manually rotated by just ten engineers.
The 300-ton Telescope Mount Assembly (TMA) has a unique, compact design that allows the entire telescope to rotate to a point on the Southern sky in seconds. (Credit: Rubin Obs/NSF/AURA.)

 

Osier shared a view from the telescope pier, a cavern of electronics and oil pumps four stories beneath the TMA. He’s working on a Gordian knot of narrow, stainless steel tubes that will deliver refrigeration fluid to the telescope above. It looks like something from the Hoth rebel base.

“The original tubing shipped from Spain had some issues,” shouts Osier over the whir of vacuum pumps and cooling fans. “We decided to replace that tubing, install new tubing, and route the new lines back to where the compressors cabinet will go.”

Contractors in Spain pre-shaped the original tubing for the telescope’s refrigeration lines, but after pandemic-related shipping delays, the tubing arrived in Chile filled with dirt and dust.

“It can be a workplace where you’re going really hard, trying to get a bunch of stuff done, but then you get held up because you're missing a specialized tool and that's the limiting factor,” says Lopez.

La Vida en La Serena

Life in La Serena amidst the global COVID-19 pandemic wasn’t as “serene” as the name would suggest, but Osier, Lopez, and Bowdish all discovered a sense of community while locked down in Chile.

“La Serena is very modern, but at the same time, you can go out of the grocery store and go through places that look like National Geographic,” remarks Bowdish.

Lopez, an avid climber and mountaineer, loves exploring the craggy granite peaks and icy mountain lakes of the Andes Mountains. She has scaled glaciers in Patagonia and rafted the Río San Pedro, but one of her most thrilling adventures happened on a soccer fútbol field in La Serena.

Margaux Lopez (red jersey) tears it up on the soccer field during a game for Club Deportes La Serena.
SLAC engineer, Margaux Lopez (red jersey), played on the professional team, Club Deportes La Serena, while working on the Rubin Observatory in Chile. (Credit: Margaux Lopez.)

 

She ended up playing for La Serena’s professional women’s soccer team, Club Deportes La Serena, for two full seasons. While her responsibilities at the observatory prevented Lopez from continuing for another season, her time as a futbólista served as a “dive off the deep end” into Chilean culture.

First Light 

With travel restrictions still in place, SLAC engineers will have to stay in Chile a bit longer, but the light, or perhaps “first light,” at the end of the tunnel (or in this case, the telescope) is within reach. Once the LSST camera has been successfully installed, commissioning engineers will be able to train the telescope on a portion of the night sky and start exploring.

“For these ‘mini-surveys,’ we go deep and we go wide,” says Reil. “We’ll stare at a little piece of the sky for a whole month to get to a twenty year depth. Then we'll go as wide as we can for a month. When we finish both of those things, that's the end of commissioning.”

 

A rosy sunrise illuminates the Rubin Observatory Summit Facility perched on a mountaintop high above a layer of clouds.
A rosy sunrise illuminates the Rubin Observatory Summit Facility perched on a mountaintop high above a layer of clouds. (Credit: Margaux Lopez.)

 

Until then, the commissioning team will be doing everything they can to prepare the Simonyi Survey Telescope’s cutting-edge subsystems for the camera’s arrival and, hopefully, enjoying some of the sunnier sides of life in La Serena.

“It's been hard, but great, to work with people from different cultures, with different languages and different work styles,” says Lopez.

"To build something this incredible, it takes a truly diverse team of individuals willing to explore new ideas and places, work hard, and sacrifice to make it a reality,” adds Reil.