2021-03-22

Bad Pixels!

by Mastcam-Z team members Stephanie Fleron, Michael Hansen, Alex Hayes, Kjartan Kinch, Marco Merusi, and Christian Tate

As the Perseverance rover has begun its mission in Jezero Crater, and the Mastcam-Z camera system is seeing its first use on Mars, a lot of Mastcam-Z raw images have already accumulated at the JPL “raw images” site at: https://mars.nasa.gov/mars2020/multimedia/raw-images/ Staring at these you may have noticed the occasional odd tiny black speck in some images. One example is shown below from the high-resolution maximum zoom panorama that was acquired by Mastcam-Z on sol 4 of the mission.

Mastcam-Z image frame ZRF_0004_0667301112_000FDR_N0010052AUT_04096_110085J01_1200 was acquired on Feb. 22 (sol 4) at the local mean solar time of 14:01:42 using the right Mastcam-Z’s filter 0. The annotated white circle and arrow points to a cluster of dead pixels as described in the text. Image Credit: NASA/JPL-Caltech/ASU/MSSS

Cropped and zoomed subframe from Mastcam-Z image frame ZRF_0004_0667301112_000FDR_N0010052AUT_04096_110085J01_1200. This image was acquired on Feb. 22 (sol 4) at the local mean solar time of 14:01:42 using the right Mastcam-Z’s filter 0. The annotated white circle and arrow points to a cluster of dead pixels as described in the text. Image Credit: NASA/JPL-Caltech/ASU/MSSS

What is that black speck? It is a tiny speck of dust on the detector at the heart of the right Mastcam-Z camera, and it leads us to a broader discussion of “bad pixels” in digital cameras in general.

This is a photo of the ON Semiconductor (formerly Kodak) KAI-2020 CCD, the sensor at the heart of both the Perseverance rover’s Mastcam-Z cameras and the Curiosity rover’s Mastcam cameras. The sensor is about 13.4×9.5 mm (0.53×0.37 inches) in size and consists of an array of 1648 x 1200 = 1,977,600 pixels that are each about 7.4×7.4 micrometers (0.00029 inches) in size. See references below for more details

The Mast Camera Zoom (Mastcam-Z or ZCAM) system is a pair of (zoomable, focusable, multi-spectral, and color) Charge-Coupled Device (CCD) cameras. The CCD is a kind of sensor that is at the core of many modern digital cameras. It is an integrated circuit etched onto a silicon surface, forming a large array of light sensitive picture elements (pixels) in a square or rectangular shape. When light hits the sensor it generates an electrical charge in each pixel, which can be measured by electronics and then turned into a digital copy of the specific pattern of charges across the array (an image).

A common and simple analogy for the operation of a CCD, based on astronomer Steve Howell’s classic book “Handbook of CCD Astronomy”, is that pixels are like little buckets on conveyor belts, like in this cartoon:

*Cartoon of astronomer Steve Howell’s “light bucket” analogy for an array of pixels collecting photons on a CCD sensor.*

As Howell writes:

“A number of buckets (pixels) are distributed across a field (focal plane of a telescope) in a square array. The buckets are placed on top of a series of parallel conveyor belts and collect rainfall (photons) across the field. The conveyor belts are initially stationary, while the rain slowly fills the buckets (during the course of the exposure). Once the rain stops (the camera shutter closes) the conveyor belts start turning and transfer the buckets of rain, one by one, to a measuring cylinder (electronic amplifier) at the corner of the field (at the corner of the CCD).”

Now, if you were to actually go through the exercise of filling a field with an array of nearly 2 million buckets (the number of pixels in each Mastcam-Z CCD) in order to measure rainfall, you might expect, despite your best efforts at quality control, that a few of those buckets would be perhaps leaky or otherwise somehow defective. The exact same thing happens with CCD’s. Despite the best efforts of CCD manufacturers and engineers assembling deep space cameras, there is always a small number of pixels that are not up to standards. These pixels are collectively known as “bad pixels”, a broad term which covers a number of different specific problems.

A lot of different terms are used to distinguish different types of bad pixels, and the terminology is not very standardized. The terms “dead”, “hot”, and “grey” pixels are widely used and a number of other terms are also used by various authors.

Dead pixels are non-responsive pixels that measure no signal. Dead pixels might be genuinely dead due to a defect in the CCD itself, so that they are receiving no electrical power, in which case they are permanently dead, or the pixels might be blocked by dust or other objects on the detector, in which case the situation might change with physical manipulation. One way to tell the difference is to look at the neighboring pixels. Electrically dead pixels have sharp borders with neighboring non-dead pixels, whereas dust-blocked pixels will show a “fuzzier” border because the debris is not exactly the same size as the pixels. In the Aerospace industry, clusters of dead pixels are often referred to as Foreign Object Debris (FOD) defects.

Mastcam-Z, like most commercial digital cameras, has red, green, and blue (RGB) color filter arrays also known as “Bayer filters” bonded onto the front of the CCD (see “The Mastcam-Z Filter Set: How Perseverance Will See the Colors of Mars” by Emily Lakdawalla and Melissa Rice). The signal from the CCD is sometimes processed using a method known as “deBayering”, which pulls out the individual red, green, and blue images from the original pixel pattern and creates a regular RGB image similar in structure to a typical color JPEG photo. If this processing step is performed (for example, it is performed in the camera electronics on Mars whenever we request that a compressed JPEG version of the images is to be downlinked), it can complicate the appearance of a dead pixel or group of dead pixels so that instead of showing up black, they can show up as either a different color than the adjacent pixels or just be slightly darker than them. This is also the reason for the extended “noise” around the dead pixel cluster shown in the above feature. Depending on the method used, the deBayering process uses the brightness of surrounding pixels to interpolate and create the final RGB image. The presence of the bad pixels can affect this process for nearby pixels, creating the effect observed in the above image. See below for a description of a process the Mastcam-Z team uses to minimize this effect.

Gray pixels is a term used for pixels that are responsive, but at significantly higher or lower sensitivity than the average. These pixels can often be corrected during the calibration process by determining how much more or less sensitive they are than the average pixel and multiplying by a corrective value during what is called “flatfield” correction. Pixels that are more than 2 times more or less sensitive than the average, however, are deemed “bad” and flagged as such during calibration. The reason for this is that they are either much noiser than surrounding pixels (if significantly less sensitive than the average) or can easily become saturated if more sensitive, since exposure times are set by the average pixel response.

Hot pixels are pixels that accumulate a lot of signal even when no light is hitting them. These can be caused by signal that accumulates slowly in a CCD even with no light on it, an effect called “dark current”. Dark current affects all pixels, and is caused by thermal energy moving electrons around inside the silicon that makes up the CCD, mimicking the effect of light falling on the CCD. Thus, dark current depends on the temperature of the CCD and electronics. At the very cold temperatures of Mars, we’ve found that it is a pretty minor effect (see the paper by Alex Hayes and colleagues in the references below for LOTS more details). One variety of hot pixels are pixels that are much more affected by dark current than average pixels. Other types of hot pixels can be caused by electrical defects in the detector. Since dark current is heavily dependent on temperature, a hot pixel can work perfectly at low temperatures but become totally saturated and max out the signal at high temperatures, such as when the cameras heated up with extended use when we tested them in room temperature conditions on Earth back in 2019 and 2020. As is the case for dead pixels, the Bayer pattern of color filters and associated deBayering can complicate the actual appearance of hot pixels in the images.

Individual pixels that appear saturated are not always due to defects of the CCD. Saturated pixels can also be caused by high-energy particles randomly striking the detector. Most of these particles, known as cosmic rays, are protons and atomic nuclei that get expelled at incredibly high speed from celestial objects like supernovae and galactic nuclei (from the Milky Way or even from other galaxies). After a long undisturbed journey, they penetrate the Martian atmosphere, and hit the sensors of our cameras. Other high-energy particles come from the Sun, as part of the so-called “solar wind,” or even from Perseverance’s own Radioisotope Thermoelectric Generator (RTG) nuclear power source, which emits some high-energy particles as the RTG’s Plutonium supply slowly decays. When one of these energetic particles hits the CCD, it enhances the production of electric signal in a small number of adjacent pixels, which will appear white or brightly colored in the image. The good news is that cosmic rays arrive randomly, which means that the bright pixels caused by them will always occur in a different pattern in every image. By comparing images, it is quite easy to distinguish between hot pixels produced by the electronics, which light up systematically, and those caused by cosmic rays, whose positions always change. Cosmic ray hits often show up a streaks across the CCD that either trace the path of the energetic particle or move down columns of the array because of charge leakage as the pixels are read out (known as blooming). In general, the effect of cosmic rays on pixels depends on different factors. The ones that are most worth mentioning are time, since long exposure times will make the CCD collect more cosmic rays, and particle energy: weaker and slower particles will generate small sharp spots of few white or colored pixels, while more energetic and faster particles, though rarer, may cause long bright streaks. Cosmic rays hits can be detected in an image by looking for pixels whose response is significantly higher than its nearest neighbors.

Cropped subframe from Mastcam-Z image ZL7_0016_0668373570_064RAD_N0030578ZCAM01500_1100LUC01.IMG. This image was acquired on March 15, 2021, (sol 16) at the local mean solar time of 15:57:33 using the left Mastcam-Z’s solar filter 7. This was the first time Mastcam-Z imaged the Sun on Mars, which requires longer exposure times because the solar filter are very opaque, like welder’s glasses. The arrows are pointing to possible cosmic ray hits, characterized by the bright pixel with a long streak following it. Image Credit: NASA/JPL-Caltech/ASU/MSSS

Before being mounted on the rover the Mastcam-Z cameras were extensively characterized and calibrated. This process is described in lots of technical detail in the paper led by Alex Hayes, with the help of the entire Mastcam-Z team, in the reference list below. During this process the Mastcam-Z CCD’s were investigated to identify bad pixels. Of the 1600 x 1200 active pixels on each Mastcam-Z detector, only 14 in the left Mastcam-Z and 17 in the right Mastcam-Z were identified as “bad”. Given that each detector is made up of nearly 2 million pixels, that is amazing! Pre-flight calibration showed that the left camera had 1 intermittent “hot” pixel, 5 “gray” pixels and 8 “dead” pixels. The right camera had 4 “gray” pixels and 13 “dead pixels”, all of which appear to be related to specks of dust on the detector. The black speck on the image at the top of the page here shows the main cluster of those right-eye dead pixels. Since landing on Mars, the intermittent pixel on the left camera has not acted up, suggesting that it behaves better at the colder temperatures of Mars as compared to the ambient room temperature on Earth during lab calibration. The vibrations from launch, cruise, and landing on Mars have not appeared to have dislodged or moved any of the dust specks either, likely because they are electrostatically “clung” to the detector (like a drier sheet!).

*Table of Mastcam-Z Bad Pixels from the research paper by Alex Hayes and colleagues listed in the references below.*

These bad pixels are therefore known to the team and can be routinely corrected for during image processing. For instance, a dead pixel can be replaced by an average of the value in surrounding pixels. The images posted on the raw images website at https://mars.nasa.gov/mars2020/multimedia/raw-images/ are however just that: raw images, before any of the routine processing that will turn them into a calibrated product has been performed, and therefore bad pixels are visible in these images. For the calibrated and processed natural color and false color images presented here on the Mastcam-Z public web site, we will typically correct these bad pixels to produce cosmetically more appealing images. We are careful, however, not use these cosmetically corrected images for our science analysis. An example of bad pixel correction is shown on the image below, where the bad pixel cluster was replaced by a weighted average of neighboring pixels of the same Bayer patter filter (red, green, or blue). In some instances, bad pixels may be replaced before deBayering to avoid the artifacts you see in the first image above, and then removed again the final RGB product.

Cropped subframe (270 pixels wide by 214 pixels high) from Mastcam-Z image frame ZR0_0012_0668004285_064IOF_N0030000ZCAM03100_0260LUC01. This image was acquired on March. 6, 2021, (sol 12) at the local mean solar time of 12:07:31 using the right Mastcam-Z’s filter 0. On the left, the bad pixel clusters have been left untouched. On the right, the bad pixel cluster has been repalced by the weighted linear average of nearby pixels (in the same Bayer filter). Image Credit: NASA/JPL-Caltech/ASU/MSSS

The Mastcam-Z cameras are continually monitored for the appearance of new bad pixels. For example, past cameras in deep space have shown that continued exposure to solar and galactic cosmic radiation can damage CCDs, causing new bad pixels to appear. And the radiation environment on Mars is more challenging that on the Earth, since Mars doesn’t have a magnetic field that helps to divert or mitigate some of those high-energy solar or galactic particles. Bad pixels can also go away if, for example, the dust blocking a particular set of pixels is dislodged. Hot or gray pixels can also be affected by “annealing,” a process where the CCD is heated up for a while. While Mastcam-Z will not be annealing its detectors during flight, other space camera systems have heaters installed near their detectors exactly for this process (see, for an example, the paper by J. Rainer Kramm and H. Uwe Keller in the references that describes the CCDs used for the OSIRIS camera on the European Space Agency’s Rosetta mission). So, we expect that more bad pixels will appear as the mission goes on, and we’ll be watching for them!

References:

Jim Bell and 48 colleagues, “The Mars 2020 Rover Mast Camera Zoom (Mastcam-Z) Multispectral, Stereoscopic Imaging Investigation,” Space Science Reviews, 217, doi:10.1007/s11214-020-00755-x, 2020.

Jim Bell and 19 colleagues, “The Mars Science Laboratory Curiosity rover Mast Camera (Mastcam) instruments: Pre-flight and in-flight calibration, validation, and data archiving,” Earth and Space Science, 4, doi:10.1002/2016EA000219, 2017.

Alex Hayes and 29 colleagues, “Pre-Flight Calibration of the Mars 2020 Rover Mastcam Zoom (Mastcam-Z) Multispectral, Stereoscopic Imager, Space Sci. Rev., 217, doi:10.1007/s11214-021-00795-x, 2020.

Steve Howell, Handbook of CCD Astronomy (2nd ed., Cambridge Observing Handbooks for Research Astronomers). Cambridge: Cambridge University Press. doi:10.1017/CBO9780511807909, 2006.

J. Rainer Kramm and H. Uwe Keller, “CCDs for the Rosetta science imaging system OSIRIS”, In: Amico, P. Beletic J. W. (eds) Optical Detecors for Astronomy II, Astrophysics and Space Science Library, vol. 252, Springer, Dordrecht, pp. 55-62, https://doi.org/10.1007/978-94-011-4361-5_8, 2000.