Volume 8, Issue 2 e2020EA001237
Research Article
Open Access

Further Investigation Into Detection Efficiency and False Alarm Rate for the Geostationary Lightning Mappers Aboard GOES-16 and GOES-17

Monte Bateman

Corresponding Author

Monte Bateman

Universities Space Research Association, Huntsville, AL, USA

Correspondence to:

M. Bateman,

[email protected]

Search for more papers by this author
Douglas Mach

Douglas Mach

Universities Space Research Association, Huntsville, AL, USA

Search for more papers by this author
Michael Stock

Michael Stock

Earth Networks, Germantown, MD, USA

Search for more papers by this author
First published: 09 December 2020
Citations: 16

Abstract

The Geostationary Lightning Mapper (GLM) is a geostationary lightning detection and location instrument, developed for the R generation of Geostationary Operational Environmental Satellites (GOES-R, S, T, and U). This paper details a new technique to assess detection efficiency (DE) and false alarm rate (FAR), which indicate how well the instrument is detecting lightning and rejecting nonlightning. In an attempt to compare GLM with the best possible ground truth data, we clustered several ground-based lightning networks into a single “virtual” network and compare it to the GLM results. A major issue with determining the GLM DE and FAR values is that over much of the instrument field of view (FOV), there are no high DE systems. To assess the GLM DE and FAR over these regions, we modified our prior coincidence criteria by increasing the time window from ±1 s to as much as ±10 min to account for the lower DE of the ground truth systems. Using the expanded time window, we compare GLM flash data from August 1, 2019 through January 31, 2020 for both instruments against the virtual network lightning flash data. We find that increasing the time window, while maintaining the distance criteria of 50 km, greatly improve the DE and FAR values. With the full ±10 min time window, over the whole GLM FOV, the GLMs on GOES-16 and GOES-17 have a DE of over 90%. For the same time window, the FAR for GLM on GOES-16 is just over 5%, while the FAR for the GLM on GOES-17 is just under 20%.

Key Points

  • We present a new technique to calculate detection efficiency (DE) and false alarm rate (FAR) for the Geostationary Lightning Mappers (GLMs) on GOES-16 and GOES-17 by expanding the time coincidence window

  • Using this new technique, the DE and FAR are actually much better than we have been previously able to show

  • With more months of data, the GLM DE and FAR should improve in regions with very little lightning

Plain Language Summary

In order to evaluate the quality of the Geostationary Lightning Mapper (GLM) data, we need to compare it to other, well-understood sources of lightning data. To account for the lack of high detection efficiency ground truth data over much of the viewing GLM area, we have increased the time window for comparisons to ±10 min. Using the larger time window, we find that the GLMs see as much as 90% of the lightning seen by other lightning detecting systems.

1 Introduction

The Geostationary Lightning Mapper (GLM) (Goodman et al., 2013; Rudlosky et al., 2019) is a lightning detection and location instrument designed for the R series of the NOAA Geostationary Operational Environmental Satellites (GOES). As of the year 2020, there are two GLMs in geostationary orbit, one on GOES-16 (East) and one on GOES-17 (West). We will refer to the two instruments as GLM-16 and GLM-17, respectively. The final two GLM instruments will be launched as part of GOES-T and GOES-U. The instruments in orbit have a combined field of view (FOV) from 57°N Latitude to 57°S Latitude, and from 198°W Longitude to 15°W Longitude (see Figure 1). As lightning has been declared an Essential Climate Variable for studying climate change due to the availability of lightning observations from space by the WMO and Global Climate Observing System (GCOS) (Aich et al., 2018), the full characterization of the GLM instrument capabilities becomes even more important.

Details are in the caption following the image

Field of view (FOV) for both the Geostationary Lightning Mappers (GLM) on GOES-East (blue) and the GLM on GOES-West (red). The combined GLM FOV covers most of North America, nearly all of South America, and much of the eastern Pacific.

In our prior work Bateman and Mach (2020), we presented preliminary detection efficiency (DE) and false alarm rate (FAR) for GLM-16. In that paper, we clustered together five different ground-based lightning detection networks in order to create the best possible source of ground truth data for GLM. Our goal for this work is to further improve the assessment of the DE and FAR for the GLM-16 and add the assessment of DE and FAR for GLM-17. Our approach will be similar to the one used in Bateman and Mach (2020) in that we will compare the GLMs to a ground truth dataset consisting of several ground-based lightning detection systems. As in that study, we have clustered the ground truth data into a single, “virtual” network source for lightning flash data. In that study, we used a time coincidence criteria of ±1 s and a distance criteria of 50 km. Another previous study (Harkema et al., 2019) used this same time and distance criteria when comparing GLM data to NLDN.

The difficulty in assessing DE and FAR for the GLMs is that over most of the FOV, ground truth data are very limited, and the farthest reaching source has fairly low DE (Burgesseur, 2017). Within the GLM FOV, the Continental United States (CONUS) and much of South America are well covered, but the oceans and the southern part of South America are less so. The reason for the lack of data is due to the ground truth sources being all land-based and most have much lower detection rates far from their sensor locations. Sparseness of sensors results in sparse ground truth data for the ocean regions and parts of South America (see Figures 2 and 3; right panel in each). The only source for corroborating data over much of the regions outside of the CONUS, including the far oceans, is the World Wide Lightning Location Network (WWLLN), which claims a DE of about 10%–20% (Burgesseur, 2017). Up to now, this has resulted in lower DE and higher FAR calculations for the GLM due to inadequate ground-truth data (Bateman & Mach, 2020).

Details are in the caption following the image

Flash density for GLM-16 (left) and the virtual network (right).

Details are in the caption following the image

Flash density for GLM-17 (left) and the virtual network (right).

A confounding factor is: Because the ground truth systems are radio frequency based, while the GLM uses optical signals to detect lightning, the two types of systems often detect different parts of the lightning flash (Zhang & Cummins, 2020). When comparing two dissimilar systems, if the flash rates are high and both systems have a reasonably high detection rate, DE values are reasonably easy to determine. Most flashes are detected by both systems. However, if one system has a particularly low DE and detects different components of the lightning flash, it is very possible that the two systems will not see the same flashes. Both systems will likely see other flashes in the same storm. One technique to counter this problem is to expand the time window used to compare the two datasets, thereby detecting some flashes from the storm by both systems. In this case, we are validating GLM flashes against WWLLN storms.

In the current study, we have taken this approach to calculating the DE and FAR for the two GLM instruments. We have allowed the time-comparison window to extend to ±10 min (on either side of the flash duration) when seeking comparisons between GLM flashes and the ground-based virtual network. We left the distance criterion at 50 km, as in the previous paper (Bateman & Mach, 2020), so as to keep the cross-storm detection chances to a minimum. We did not continue to expand the time window beyond ±10 min, as that would also increase the chances of the coincident storms moving out of the grid boxes. When extending the time window to this size, it allows us to use WWLLN as a “storm detector,” rather than an individual lightning flash detector when comparing to the GLM data. The results of this new, extended-time comparison are presented here.

2 Instrumentation

The GLM data used in this study are the flash data from both GLMs (GLM-16 and GLM-17). The GLMs sense total lightning based on their optical properties at the 777.4 nm oxygen triplet (Christian & Goodman, 1987). The optical signals are detected and then processed into lightning flashes by a series of filters and a clustering algorithm (Goodman et al., 2013; Mach, 2020). GLM-16 was declared provisional on January 19, 2018 while the GLM-17 was declared provisional on December 20, 2018. The flash data used in this analysis are from August 1, 2019 through January 31, 2020 for both instruments. These dates were chosen to only include GLM data after the updated blooming filter was installed in late July 2019 (GOES-R Series Data Book, 2019).

The first ground-based system we use in our virtual network is Earth Network's Global Lightning Network (ENGLN) (Heckman, 2014). This is a source that combines the Earth Network's Total Lightning Network (ENTLN) (Liu & Heckman, 2010) with the World Wide Lightning Location Network (WWLLN) (Dowden et al., 2002; WWLLN website, 2012). ENTLN provides high DE stroke data (what they call “portions”) that covers the CONUS and northern and eastern South America. The WWLLN adds global coverage, but at a much lower DE (10%–20%) (Burgesseur, 2017).

The next set of systems in the virtual network are several made and operated by Vaisala. The Global Lightning Dataset GLD360 (Mallick et al., 2014) provides wide area, near global coverage, high DE lightning stroke data (Said et al., 2010, 2013). The data purchased for this study are limited by contract to between −30 and −150 Longitude. These limits are apparent in the maps that will be presented. Next, we used the National Lightning Detection Network (NLDN) stroke and flash data (Cummins et al., 1998; Orville et al., 2011). These provide high DE lightning data over CONUS and to about 100 km beyond the shores of CONUS. Finally, to fill in the northern regions of the GLM FOV in North America, we use the Canadian Lightning Detection Network (CLDN) (Burrows et al., 2002). This network provides lightning flash data with sensors similar to the NLDN (Orville et al., 2011). All systems used for ground truth data detect total lightning (in-cloud and cloud-to-ground).

3 Methodology

As was done in Bateman and Mach (2020), we combine the various ground systems described above into a virtual ground truth system. A challenge in merging the different reference datasets is that some ground-based lightning detection networks report strokes, others report flashes, and some report both. To overcome this issue, we use the same technique as done in Bateman and Mach (2020). We combined data from the various networks using a clustering technique similar to that used in clustering GLM events into flashes. All ground truth datasets are first combined and time sorted, then all sources within 330 ms and 16.5 km of the first source are clustered into a flash. As a clustered flash grows, the temporal and spatial limits expand so that any subsequent ground truth system stroke, flash, or other unit of lightning is added in if it is within 330 ms and 16.5 km of any other item already in the cluster. This means that the time and space “size” of the clustered flash is allowed to grow as long as new lightning data can be found that is within 330 ms and 16.5 km of any other item already in the clustered flash.

This technique works well to produce a combined high DE ground truth system if there are multiple systems capable of detecting flashes with a reasonably high DE in a region (as in the CONUS and similar regions in the Bateman and Mach [2020] work). However, when there is only one system with a lower DE in a region, the comparison to GLM for the purpose of DE and FAR calculations, begins to fail. In such regions, we need an alternative method to fully assess the GLM system DE and FAR.

One technique to compare GLM to a low DE system is to expand the time window of the comparison. Since a large part of the GLM FOV is over ocean far from land (see Figure 1), and in those areas, much of the ground truth data is from WWLLN. WWLLN has only a 10%–20% DE, so expanding the time window can increase the chances for a coincidence between the WWLLN data and GLM. Flash rates for oceanic storms have been reported by Mach et al. (2009); they found an average flash rate of about 1 min−1. If those storms are observed with a system that has a 10% DE, then a storm needs to be monitored for about 10 min to ensure detecting at least one flash with the ground truth system. We arrived at this figure of 10 min by dividing the flash rate of 1 min−1 by the DE of 0.1. GLM is designed to have a DE of greater than 70% (Goodman et al., 2013) and GLM-16 has an average DE of 77% (Bateman & Mach, 2020), and thus may not see the same flash as the ground truth systems (Zhang & Cummins, 2020). However, we will know that both systems have seen the same storm, as long as we keep the spatial limits the same as before. Note that this essentially turns the WWLLN into a “storm detector.” So now, in parts of the GLM FOV, we are using “storm detection” to validate GLM flashes.

To document the incremental effects of increasing the time coincidence window, we started the time window of the DE and FAR analysis at the value used for the previous work of ±1 s (2s total) (Harkema et al., 2019; Bateman & Mach, 2020). We then repeated the DE and FAR analysis for the same dataset, increasing the time window for each run. We ran the comparisons for 10 different time window widths, essentially doubling the width each time. The final time window was ±10 min (20 min total). We ran the analysis for both the GLM-16 and GLM-17 datasets. DE is defined as the number of flashes detected that are coincident to GLM and the virtual ground truth system, divided by the total number of flashes detected by the virtual ground truth system. FAR is defined as the number of flashes detected by GLM that are not coincident with the virtual ground truth system, divided by the total number of GLM flashes. We understand that these are not absolute (but instead relative) DE and FAR, as neither system is 100% efficient. Details on how we calculate DE and FAR are given in Bateman and Mach (2020). Note that as before, all GLM data and virtual network ground truth data are gridded into 1 × 1° grid boxes.

4 Results

Shown in Figure 2 are the flash densities for both GLM-16 and the virtual network over the 6 months of this study. The GLM-16 data are in the left panel, while the virtual network data are in the right panel. Figure 3 shows the flash densities for GLM-17 (left) and the virtual network (right). Note that all three systems show a lack of lightning data over the ocean to the west of South America. This lack of lightning has been seen in other studies with different lightning sensors, for example, Albrecht et al. (2016), Blakeslee et al. (2020). To keep the DE and FAR statistics from having too much variation, we will not plot DE and FAR values when the number of flashes per grid point drops below 20 flashes for either the GLMs or the virtual ground truth source.

Rather than attempt to show the full plots of GLM-16 and GLM-17 DE for all 10 time windows, we have chosen to show the comparison of the two extremes (2 s and 20 min total) for GLM-16 and GLM-17 in Figure 4. Note that almost all of the dark green (DE between 0.50 and 0.70) and gray (light gray 0.25–0.50; dark gray, 0.00–0.25) areas in the right plots have changed to light green (DE > 0.70) in the left plots. Except for a few areas of yellow (indicating GLM detects something but the virtual ground truth network does not), and the large area of almost no lightning to the west of South America, nearly the whole GLM-16 and GLM-17 FOVs are light green. Of the boxes with coincident data (greens and grays), 97.7% of GLM-16 and 94.1% of GLM-17 are light green. The left plot for GLM-16 shows an average DE of 0.97 over the whole map, while the left plot for GLM-17 shows an average DE of 0.93. The values of DE were averaged over the whole FOV for both GLM-16 and GLM-17 and plotted for all time comparison windows in Figure 5. Note that both DE curves have the same shape between GLM-16 and GLM-17, but the initial and final values are different. Note that we have marked a time of 60 s, a point at which the sharp increases begin to flatten out. More on this later.

Details are in the caption following the image

Comparison of DE calculated with a time window of 20 min (left) and 2 s (right). The upper plots are for GLM-16 while the lower plots are for GLM-17.

Details are in the caption following the image

DE versus time window widths for GLM-16 (blue) and GLM-17 (red). The time t = 60 s is marked with a dashed line. Past a time window width of 60 s, the curves begin to flatten out.

Figure 6 shows the GLM-16 DE separated into day/night for the ±10 min time window. To do this, we chose 6 h in which the GLM-16 FOV was in total daytime and another 6 h when the FOV was in total nighttime. Other than where there is insufficient data, almost all coincidence pixels (greens and grays) are light green (98.6% day, 98.7% night). Figure 7 shows the same DE values for GLM-17 with the ±10 min time window, again divided into day and night. As with GLM-16, we chose a 6-h period where the whole GLM-17 FOV was either in daylight or nighttime. Again, most all coincident pixels (93.8% day, 94.3% night) of the GLM-17 FOV is light green. There are larger areas of insufficient data and more areas of GLM detections without corresponding ground truth data (shown as yellow).

Details are in the caption following the image

DE for GLM-16, using a time window width of 20 min (±10 min). Shown is daytime (left) and nighttime (right).

Details are in the caption following the image

DE for GLM-17, using a time window width of 20 min (±10 min). Shown is daytime (left) and nighttime (right).

Figure 8 is the FAR version of Figure 4. The left side plots are the FAR for the ±10 min time window while the right plots are the FAR for the ±1 s time window. The upper plots are for GLM-16 while the lower plots are for GLM-17. Note that most of the GLM-16 FOV for ±10 min has FAR of less than 5% (shown as light gray). The light gray region increases from 8.7% (right) to 73.9% (left). The number of flashes for both GLM-17 and the virtual ground truth dataset are much lower, owing to the mostly oceanic coverage of GLM-17. The amount of light gray increases for GLM-17 from 6.6% (right) to 32.5% (left). Both instruments show considerable improvements from a time window of ±1 s to ±10 min. Figure 9 is the FAR equivalent of Figure 5. Again, both GLM-16 and GLM-17 show significant improvements in the FAR as the time window increases, although GLM-17’s final values are not as good as those of GLM-16.

Details are in the caption following the image

Comparison of FAR calculated with a time window of 20 min (left) and 2 s (right). The upper plots are for GLM-16 while the lower plots are for GLM-17.

Details are in the caption following the image

FAR versus time window widths for GLM-16 (blue) and GLM-17 (red).

As above for DE, Figures 10 and 11 show the FAR values split into day/night hours. Figure 10 shows the FAR day/night data for GLM-16. Most of the FAR grid boxes are still below 5%, except for the areas with insufficient data. Figure 11 shows that the FAR day/night values for GLM-17, which suffer from a lack of ground-truth data. Many grid boxes have less than 20 flashes, either in the GLM data or the virtual ground truth data.

Details are in the caption following the image

FAR for GLM-16, using a time window width of 20 min (±10 min). Shown is daytime (left) and nighttime (right).

Details are in the caption following the image

FAR for GLM-17, using a time window width of 20 min (±10 min). Shown is daytime (left) and nighttime (right).

5 Discussion

Figures 2 and 3 illustrate the major constraint with this analysis: Over a significant fraction of the GLM-16 FOV (and a majority of the GLM-17 FOV), there are not much data over most of the Pacific Ocean. For these 6 months of data, there is also not much lightning in the Northwestern CONUS. The lower DE and higher FAR areas over this region, which is seen in both the GLM-16 and GLM-17 data, mostly vanish when a longer time coincidence window is used. Also, most of the lightning is over land, and for these 6 months less lightning occurred in North America than South America.

For the virtual network, notice the extreme lack of lightning on either side of South America. Over this 6-month period, there are fewer than 20 flashes per grid box over the region. For GLM-16 there is little data west of South America; for GLM-17 there is slightly more data. It is suspected that much of the data that are reported by the GLMs in this region are not real lightning and are possibly due to glint or noise. In Figures 6 and 7, note that the yellow pixels west of South America disappear at night. This is why we think that these yellow pixels are attributed to sun glint off the ocean or increased noise detections. Even with the longer time coincidence window, the lack of data to the west of South America simply makes it impossible to determine accurate DE and FAR measurements in that region. The lack of data in that region becomes even more apparent when the data are split into night and day. In many of the grid boxes in that region, there are not even 20 flashes over the 6-month period of our analysis. Another concern is the red pixels (ground truth detect with no GLM detection) west of Peru/Chile (Figure 4). These virtual network flashes may be caused by erroneous locations from some of Vaisala's sensors, which are being fixed (Said & Murphy, 2019).

Despite these data limitations, we are able to produce stable DE and FAR values for most of the GLM-16 and GLM-17 FOVs. The DE values improved rapidly until the time window was about 60 s wide (shown in Figure 5) for both GLMs. After that point, there was an incremental change up to the final values at ±10 min. Figure 12 shows the DE and FAR values for both GLMs at a time window width of 60 s (±30 s). At the 60 s time window width, the average DEs are: GLM-16 (0.91) and GLM-17 (0.84); the average FARs are: GLM-16 (0.18) and GLM-17 (0.38). Continued increases in the time window size slowly improve the DE and FAR values, and by ±10 min, we are approaching the asymptotic limit of the DE and FAR improvements. Any increase beyond ±10 min would greatly increase the chance of storm misidentification without greatly improving the DE or FAR values.

Details are in the caption following the image

Detection efficiency (top plots) and false alarm rate (bottom plots) for GLM-16 (right plots) and GLM-17 (left plots), using a time window width of 60 s. At this time window width, the average DEs are: GLM-16 (0.91) and GLM-17 (0.84); the average FARs are: GLM-16 (0.18) and GLM-17 (0.38).

6 Concluding Remarks

GLM-16 shows an average DE of 0.97 across the FOV and GLM-17 shows a DE of 0.93 at the coincidence time window width of ±10 min. These numbers come from data for August 1, 2019 through January 31, 2020. The target specification for GLM is DE greater than 70% (GOES-R Series Data Book, 2019; Goodman et al., 2013). So at the maximum time coincidence window, both GLMs well exceed this target value.

GLM-16 shows an average FAR of 0.06 across the whole FOV while GLM-17 shows a FAR of 0.19 averaged across the whole FOV. Again, this was using the maximum time coincidence window of ±10 min. The specification value for FAR is less than 5% (GOES-R Series Data Book, 2019; Goodman et al., 2013), so GLM-16 is slightly over that value and GLM-17 is nearly 4 times higher than that value. Note that our previous work (Bateman & Mach, 2020) showed that FAR was only performing well over CONUS and NW South America—exactly the regions where the ground-truth data are of the highest quality. Our new extended time coincidence technique increases that area to include much of the WWLLN coverage, and we see that the FAR improves greatly.

Acknowledgments

This work was supported by NASA 80MSFC17M0022, Cooperative Agreement with Universities Space Research Association, and NASA Research Opportunities in Space and Earth Science, grant NNX17AJ10G U.S. and European Geostationary Lightning Sensor Cross-Validation Study.

    Data Availability Statement

    All the data used in this study are available from the NOAA Class system (https://www.avl.class.noaa.gov/saa/products/search?datatype_family=GRGLMPROD). Request Level 2 data from GLM-16 and GLM-17. The time period used was August 1, 2019 through January 31, 2020.