2024-02-06
Mars Helicopter Flights: Caught on Video!
Contributed by Justin Maki, Mark Lemmon, and Jim Bell and UPDATED on 6 February 2024
In the spring of 2021 the Ingenuity Mars Helicopter became the first aircraft to fly on the surface of another planet. When the helicopter lifted off of the Martian surface at 7:50 UTC on April 19th, 2021 (Sol 58 of the Perseverance surface mission), it marked the beginning of a new era for space exploration. With that first flight, the aerial exploration of Mars had begun, occurring almost 45 years after the Viking landing (1976) and 24 years after the Mars Pathfinder Sojourner rover mission (1997). Humanity could now robotically explore Mars by air. And Mastcam-Z caught it all on video.
Ingenuity development started back in 2013, and a question that the team was asking at the time was “can we record a video of the helicopter flights with the rover cameras?”. When the Mastcam-Z instrument was selected by NASA in 2014, that question was answered. Mastcam-Z inherited a video capability from the Mars Science Laboratory (MSL) Curiosity Mastcam camera design. The Mastcams were developed on MSL with the MAHLI (Mars Hand Lens Imager) and MARDI (Mars Descent Imager) cameras at Malin Space Science Systems, Inc. (MSSS) in San Diego, CA. All three of the MSL camera types (“MMM”, for short) shared common base designs. MSSS also built the Mastcam-Z cameras for Arizona State University, and because of this commonality, the Mastcam-Z cameras have the same video capability that was developed for the MARDI camera. And while video has been used only sparingly on MSL Mastcam, it was just simpler, more efficient, and more cost effective to keep the video capability in the inherited software design for Mars 2020. During MARDI development back around 2005, the idea of acquiring video of a helicopter flying around on Mars wasn’t even imagined. But 16 years later, that video capability would be used to record the historic first flights of Ingenuity.
By 2017, the dimensions and form factor of the helicopter were more or less finalized, and the Mastcam-Z optical design was also nearly complete. With the basic design parameters in hand, it was possible to generate a notional, simulated view of the helicopter in an image from the MSL telephoto (100-mm focal length) Mastcam camera, to scale (see figure below). The simulated view assumed that Mastcam-Z would be at full zoom (110-mm focal length), with the helicopter approximately 120 meters away. It looked plausible that the videos would show some detail, including the landing legs and the rotors. The simulated views were obviously just notional, but as we approached launch (and landing), the idea that we might record the flights with Mastcam-Z video started to gain wide support. And then it became part of the baseline plan! The first actual videos of Ingenuity were acquired at a closer distance of roughly 70 meters away, so the actual resolution was better than the simulation.
The Mastcam-Z uplink operations team at MSSS, led by experienced planetary instrument operations engineer Elsa Jensen, has a lot of experience pointing cameras to targets on the surface of Mars as the rover drives from one place to another. We use various Cartesian and spherical coordinate frames to point the cameras to points of interest, and for the most part the process is fairly straightforward. But unlike most of the rocks and hillsides, the helicopter moves around. With this added dimension, the targeting process became a question of not just targeting the right location, but targeting it at the right time. As easy as that timing challenge sounds, it required a great deal of coordination between the Mastcam-Z operations team, the rover vehicle operations team, and the helicopter operations team (as with many things in space mission operations, it was a multi-team effort).
One of our first attempts at such coordination was intended to take a Mastcam-Z movie of what the Ingenuity team called the blade release or “wiggle test” activity, designed to confirm that the rotor blades were released from their launch-and-landing locked positions before a full-up high-speed spin test. That activity was successfully captured by Mastcam-Z on sol 48 (April 8, 2021), and represented a great initial test of the complex inter-team communications that would ultimately be needed to capture the actual helicopter flights themselves. See the left eye movie posted here, and more details posted on the NASA/Photojournal media release here.
For the initial demonstration flights, our adopted strategy to cover any residual uncertainties in timing was to start recording images well before the expected flight time, and end recording well after the expected landing time. We would then downlink and review the thumbnail videos (thumbnails are much smaller, spatially downsampled versions of the larger, original images), along with a small number of full frame images, to determine which particular frames caught the actual flight. The cameras would be later commanded to copy those particular frames of interest from internal camera storage over to the rover computer for subsequent downlink to Earth. The video frames were compressed as individual JPEG images at quality value of 50. Once on the ground, the individual frames were assembled into mpeg-compressed video files using ffmpeg. Mastcam-Z team member and atmospheric science expert Mark Lemmon also did a bunch of post-processing of the videos to bring out both subtle and not-so-subtle details of the ways that the helicopter created an artificial dust-lifting wind as it flew.
These videos are exciting to watch, even months later. The full set of 10 “mission success” videos are included here (5 initial demonstration flights x 1 video per camera x 2 cameras). Additionally, specially-processed versions of these videos have been created by Mastcam-Z team member Mark Lemmon, using the same processing methods that were described in his “Dust Devil Photobombers!” blog entry. These processed versions provide dramatic evidence for unique dust-lifting and dust-transport processes that are initiated and sustained by Ingenuity’s “blade wash”. We also include Mastcam-Z videos for two extended mission flights, designed to further study dust lifting and transport induced by the helicopter’s rotor blades: Flight #13 (September 5, 2021) and Flight #47 (March 9, 2023)!
Mastcam-Z Movies of the Demonstration Flights
Flight #1. The historic first flight, around 12:30 pm Local Mean Solar Time (LMST) on mission sol 58 (April 19, 2021), was a simple takeoff, followed by a hover at a height of 3 meters above the ground, with a rotation, and a landing. We captured the full flight at a wide angle (34 mm, 6.5 frames/second (fps)) in the right camera and at full zoom (110 mm, 6.3 fps) in the left camera. Total flight time was approximately 39 seconds.
(processing by Mark Lemmon)
Flight #2: The second flight, around 12:30 pm LMST on mission sol 61 (April 22, 2021), hovered at 5 meters above ground, and consisted of a 2 meter lateral move in the direction to and from the rover (4 meters total). We captured the full flight in the right camera (34 mm, 6.5 fps) and part of the flight in the left camera (48 mm, 6.5 fps). Total flight time was approximately 52 seconds.
Flight #3: The third flight, around 12:30 pm LMST on mission sol 64 (April 25, 2021), hovered at 5 meters above ground, and flew 50 meters out and back (100 meters total). The entire flight was captured by both cameras at the widest focal length setting (26 mm, 6.6 fps). Flight 3 was the only flight captured in stereo video thus far. The helicopter flies out of view on the right side of the image, and comes back into view from the right, near the end of the video. Total flight time was approximately 80 seconds.
Grab your red/blue glasses and see https://www.youtube.com/watch?v=Q75-HetU57A for the stereo video!
Flight #4: The fourth flight, around 12:30 pm LMST on mission sol 69 (April 30, 2021), hovered at 5 meters above ground, and flew 133 meters to the left and back (266 meters total). For this video the entire flight was captured in the left camera (26 mm, 6.7 fps) and part of the flight in the right camera (26 mm, 6.8 fps). For this flight, the SuperCam microphone also recorded audio. Total flight time was approximately 117 seconds.
(processing by Mark Lemmon)
(processing by Mark Lemmon)
(processing by Mark Lemmon)
Flight #5: The fifth and final demonstration flight, around 12:30 pm LMST on mission sol 76 (May 7, 2021), was the first time the helicopter flew off to a new landing location. The landing was 129 meters to the south, flying at a cruising altitude of 5 meters above ground. Near the end of the flight (above the landing site), the helicopter flew from 5 meters altitude up to height of 10 meters, hovered, then landed. Because the remote sensing mast (RSM) could not be pointed during any of the helicopter flights (due to potential radio interference with the heli communications link), we decided to point the cameras at the flight #5 landing location and wait for the helicopter to arrive on the scene. Both cameras captured the landing, and the right camera also caught a number of dust devils before and during the landing. The right camera video was acquired at 26 mm, 6.8 fps, and the left camera captured the landing at full zoom (110 mm) and at our highest video frame rate so far: 15.4 fps. The left camera video was subframed vertically to increase the frame rate, allowing a fun “super slo-mo” version of the springy landing to be captured. The SuperCam microphone audio was also acquired for this flight. Total flight time was approximately 108 seconds.
(processing by Mark Lemmon)
Mastcam-Z Movies of some of the Flights During Ingenuity’s “Operational Demonstration” Mission
Flight #13: Mastcam-Z recorded another movie of Flight #13 around noon LMST on sol 193 (Sept. 5, 2021), as part of a cool atmospheric science experiment that used the helicopter as an artificial dust devil maker. The helicopter was about 300 meters from the rover for this flight, so it’s challenging to spot it in the wide-angle left Mastcam-Z view at 26-mm focal length, but much easier to watch in the telephoto right Mastcam-Z view at 110-mm focal length. The total duration of the flight was about 160 seconds, but only part of that was actually caught in the videos. Lots more details about this flight can be found in the NASA/JPL press release at https://www.nasa.gov/feature/jpl/nasa-s-perseverance-captures-challenging-flight-by-mars-helicopter
As of December 2022, Ingenuity has now flown a total of 37 flights, with flight numbers 6 and onward being part of an Operational Demonstration and Science Support extended mission for the helicopter. It’s been exciting to see Ingenuity help scout out areas where the rover either hasn’t driven yet or can’t drive into, helping provide the science team with unique new perspectives on the geology of Jezero crater.We’ll try to update this blog with any other Ingenuity movies that Mastcam-Z takes, too, once they are fully downlinked and processed from the rover!
Flight #47: Mastcam-Z recorded another movie of Flight #47 around 16:00 LMST on sol 729 (March 9, 2023), as part of a continuing atmospheric science experiment that uses the helicopter as an artificial dust lifting source. The helicopter was about 120 meters from the rover for this flight, which was filmed at 34-mm focal length in both the left and right eyes of the Mastcam-Z. The left eye movie was shot at a relatively high compression level (JPEG = 50) to enable max possible frame rate (6.7 frames/sec) for a subsample of the field of view designed to capture the takeoff and early flight. The right eye movie was shot using lossless JPEG compression at a lower frame rate (2.1 frames/sec) for a wider subset of the field of view, designed to try to capture details of the dust lifting and transport processes at higher spatial resolution. The total duration of the flight was about 160 seconds and the heli traveled up to 12 meters above the surface over a total distance of 440 meters, but only part of that was actually caught in the videos. Lots more details about this flight can be found in the NASA/JPL press release at https://mars.nasa.gov/resources/27351/perseverances-mastcam-z-views-ingenuitys-47th-takeoff.
Coming soon: Right eye video! (still on the rover, awaiting downlink…)
Unsuccessful flight attempt: Sol 748 (March 29, 2023; 15:00 LMST): Mastcam-Z left eye 110-mm zoom video acquired, but heli flight did not occur due to colder than expected temperatures, so the video shows no helicopter motion or any other activity in the scene during a pretty boring 3 minute, 21 sec video…
Flight #54: Mastcam-Z recorded another movie of Flight #54 around 15:10 LMST on sol 872 (August 3, 2023). The helicopter was about 55 meters from the rover for this flight, which was filmed at 110-mm focal length in the left eye and at 48-mm focal length in the right eye of Mastcam-Z. Both movies were shot at a relatively high compression level (JPEG = 50) to enable max possible frame rate (5.9 and 6.5 frames/sec, respectively) for a 1280×720 pixel subsample of the field of view. The total duration of the flight was about 105 seconds. After performing a preflight “wiggle check” with its rotors, the helicopter takes off, hovers at an altitude of 5 meters, and rotates to the left, before touching back down. The mission conducted the short pop-up flight to check Ingenuity’s navigation system. More details about this flight can be found in the NASA/JPL press release at https://mars.nasa.gov/resources/27635/perseverance-rover-watches-ingenuity-mars-helicopters-54th-flight/
Flight #59: Mastcam-Z recorded another movie of Flight #59 around 11:00 LMST on sol 915 (September 16, 2023). The helicopter was about 55 meters from the rover for this flight, which was filmed at 110-mm focal length in just the left eye of Mastcam-Z. The movie was shot at a lossless compression level to enable max possible resolution for a 864×1008 pixel subsample of the field of view at a frame rate of ~3 fps. In this flight Ingenuity flew to its highest altitude yet — 20 meters [66 feet]. The rotorcraft was in the air for 142.59 seconds. More details about this flight can be found in the Space.com article at https://www.space.com/mars-helicopter-ingenuity-flight-59-altitude-record
[videos coming soon]
FOR MORE SCIENCE DETAILS: Check out some of the technical details about the ways that atmospheric scientists are using Ingenuity to learn about the dust on Mars in the 2022 Journal of Geophysical Research open access research paper by Mastcam-Z team member Mark Lemmon and colleagues titled “Lifting and Transport of Martian Dust by the Ingenuity Helicopter Rotor Downwash as Observed by High-Speed Imaging From the Perseverance Rover.” And you can learn even more about how Ingenuity’s sounds are processed for the merged Mastcam-Z and SuperCam microphone movies posted here in this article by SuperCam team member Ralph Lorenz and colleagues, titled “The Sounds of a Helicopter on Mars.”