Volume 19, Issue 8 e2020SW002709
Research Article
Open Access

Local Magnetic Activity Index Nowcast Based on Concurrent Measurements at Two Observatories in Close Proximity

Tobias G. W. Verhulst

Corresponding Author

Tobias G. W. Verhulst

Royal Meteorological Institute (RMI) of Belgium, Brussels, Belgium

Solar Terrestrial Centre of Excellence (STCE), Brussels, Belgium

Correspondence to:

T. G. W. Verhulst,

[email protected]

Search for more papers by this author
Stephan Bracke

Stephan Bracke

Royal Meteorological Institute (RMI) of Belgium, Brussels, Belgium

Search for more papers by this author
François Humbled

François Humbled

Royal Meteorological Institute (RMI) of Belgium, Brussels, Belgium

Search for more papers by this author
Stanimir M. Stankov

Stanimir M. Stankov

Royal Meteorological Institute (RMI) of Belgium, Brussels, Belgium

Solar Terrestrial Centre of Excellence (STCE), Brussels, Belgium

Search for more papers by this author
First published: 24 June 2021

Abstract

Permanent monitoring and real-time indexing of the geomagnetic activity is of utmost importance for space weather services. In particular, providing a continuous and reliable local magnetic activity nowcast is of increasing importance as it can be used to generate alerts for various users, for instance for GNSS-based applications or to protect electric power grids. This study presents a novel nowcast system for calculating a K-type index of the local magnetic activity in Belgium. This includes a fully automated computer procedure for digital magnetogram data acquisition, data cleaning, calculating the index, displaying the results, and possibly issuing an alert to interested users, if the storm-level activity is identified. Novel characteristics are the higher cadence of index calculation, an index reliability estimate based on data quality, and the high availability of the nowcast guaranteed by the operation of two identical instruments at observatories in close proximity. An evaluation of the index nowcast accuracy was carried out, and its results are also presented.

Key Points

  • An improved system for real-time monitoring of the local geomagnetic activity is presented

  • Combining data from two compatible magnetic observatories allows automatic data cleaning

  • Use of two geomagnetic observatories improves availability of magnetic field nowcasting

Plain Language Summary

As we increasingly rely on GNSS based technologies, not only major geomagnetic storms are worth being nowcasted, but also more frequent lower intensity events. The well-known Kp-index is a geomagnetic activity scale based on ground-based magnetic field variations. Aside this planetary index, a local K-index provides a more detailed geomagnetic activity level over a limited area. This study addresses two issues related to local K-index nowcasting: data validation and service availability. Validation of geomagnetic data is a challenging issue because it can significantly fluctuate from one second to another, unpredictably, resulting in a difficult outliers identification process. For instance, a spike due to local powergrid issue may generate an alarm if not removed. Our approach is to produce a local K-index, KBEL, based on two independent magnetometers, located in the two magnetic observatories of Belgium. The distance between the two sites is sufficient to consider local conditions are independent, but small enough that the variations recorded by the magnetometers should be similar. The real-time comparison of the data sets offers a reliable tool to eliminate any outliers. This publication also discloses the parameters, specific to the chosen sites, as a support to any other local K-index nowcasting service implementations.

1 Introduction

Geomagnetic activity monitoring is important for understanding the complex dynamic nature of the Earth's magnetism and the near-Earth space environment. A key element of this monitoring is the production of indices that are able to capture the irregular disturbances of the geomagnetic field on time scales of minutes to days due to the solar activity (Mayaud, 1980; Siebert, 1996), on a local and a global scale. One of the oldest and most reliable indices, is the K-index, a quasilogarithmic index characterizing the three-hourly range of irregular magnetic activity relative to the regular quiet-day variations at a single location. A derivative, mean standardized K-index from several globally distributed stations, called “planetary” index (Kp), provides a convenient measure of the global magnetic activity (Mayaud, 1980; Menvielle & Berthelier, 1991).

The Kp-index (Matzka et al., 2021) is widely used in ionospheric and space weather research, modeling, and services. To address the needs of the rapidly proliferating space weather services, a lot of progress has been made in recent years in nowcasting, as well as forecasting the Kp-index through various methods, which still remains an active area of research and development (Chakraborty & Morley, 2020; Liemohn et al., 2018; Luo et al., 2017; Sexton et al., 2019; Wintoft et al., 2017).

However, there are some important limitations to using the standard Kp-index. One drawback is that, being a planetary index, Kp might not be accurate enough to represent the magnetic activity background when monitoring ionospheric disturbances of a smaller scale. For this reason, more local indicators of the geomagnetic activity are preferred for some applications, based on regional or even single-station observations, see for instance Lam (2006); Forbes and St. Cyr (2008); Denardini et al. (20132015); Beggan et al. (2018) for some examples. Another drawback is the fixed, three hour time interval at which the index is calculated. With such a low cadence, the chance of accurately catching the start of a geomagnetic storm is rather small. Moreover, such a time interval is much larger than the characteristic times of many localized ionospheric phenomena that are of particular interest to many users. For many technological applications, real-time nowcasts or short-term forecasts of magnetic activity levels are more appropriate, for example, Lam (2006); Viljanen et al. (2006); Kervalishvili et al. (2019).

To be able to more efficiently utilize the local magnetic measurements, we developed a nowcast service to be used primarily for local ionosphere and space weather monitoring. The original nowcast system (K-Logic) providing a proxy, K-type magnetic index, is described in Stankov et al. (2011). As a K-type index, this is a local characterization of the magnetic activity which is based on the 1-min data from a single magnetic observatory. The nowcast is based on a fully automated computer procedure for real-time digital magnetogram data acquisition, establishing the solar regular variation of the geomagnetic field, careful data cleaning, calculating the K-index, and visualizing the output. K-Logic delivers as its output not only an estimate of the K-index value but also a quality flag, providing an estimate of the reliability of the data used in the calculation of that value (as described below in Section 5).

The nowcast service has been in operation for over a decade already, utilizing real-time measurements from the magnetic observatory in Dourbes, Belgium. Over the years, it has been used in various space weather projects and services, such as GAlileo LOcal Component for nowcasting and forecasting of Atmospheric Disturbances (Galocad) (Warnant et al., 2008), Space Weather and Navigation Systems (Swans) (Stankov et al., 2010; Warnant et al., 2010; Wautelet et al., 2010), and Advanced Forecast For Ensuring Communications Through Space (Affects) (Verbeeck, 2013). In addition, ever since its launch, it features in the regular Solar Terrestrial Center of Excellence newsletter to users (STCE, 2011). Most recently, it has been incorporated into the Pan-European network Pan-European Consortium for Aviation Space weather User Services (Pecasus, http://pecasus.eu), delivering space weather services for the International Civil Aviation Organization (Harri et al., 2019; Haukka et al., 2020).

Overall, the K-Logic nowcast (which will be called Kold here to distinguish it from the other indices discussed) proved to be a reliable tool for local geomagnetic activity monitoring. Nevertheless, three areas for possible improvement have been identified. First, technical problems at the observatory or with the data transmission infrastructure can cause interruptions in the real-time availability of the system. Second, the automatic data cleaning procedures, indispensable for a real-time service, present a major difficulty. Figure 1 shows the average quality flag provided by the old nowcasting system in relation to the obtained K value. It can be seen that the quality goes down slightly for higher index values. Especially at the onset of magnetic disturbances, the quality-flag of the index often drops down, indicating lower confidence in the obtained value. The reason for this is the inherent difficulty of distinguishing real sudden disturbances from faulty data points when subsequent data are not yet available: the data cleaning procedure often initially identifies the onset of a disturbance as faulty data. Finally, it was discovered that the K-Logic implementation does not completely remove the solar regular variation from the magnetic observations, and in fact does not precisely follow the procedure described in Stankov et al. (2011).

Details are in the caption following the image

Left panel: the averages of the data quality flag provided by the initial nowcasting system (Stankov et al., 2011), for those cases where the quality flag is at least 6. This excludes situations where long periods of the data are missing, and is therefore determined only by the number of outliers removed by the data cleaning procedure. Right panel: the distribution of occurrences of K-values in the nowcast (logarithmic scale).

In order to address the problems mentioned above, this works aims to exploit the unique opportunity presented by the close proximity of two magnetic observatories located in Belgium, in Dourbes (IAGA code: DOU) and in Manhay (MAB). The use of concurrent measurements using identical instruments at two observatories offers two important advantages. One is the possibility of promptly identifying an erroneous record by constantly comparing the corresponding records at both stations and looking for an unusually large difference. The other advantage is in the implementation of a much more resilient and reliable service because either station, on its own, can be used to provide a K-index nowcast should the data from the other station not be available due to technical problems.

The main goal of this work is to produce the local index we call KBEL. This index is ultimately calculated from the observations made in Dourbes. However, before the index is calculated the variations in the magnetic field are compared to the concurrent observations done at the Manhay observatory for the purpose of validating the data. At both observatories, real-time indices are also calculated using the data from that observatory only, called KDOU and KMAB respectively, relying on internal data integrity checks to validate the input data. We will show that the method of comparing the data from the two stations generally results in more accurate index values. Still, all three indices are available in the database in real time. This allows us to provide users with the best choice available, in terms of confidence level of the nowcast and reliability of the data, at any given time.

This study is organized as follows. First, in Section 2, we present the magnetic observatories and their instrumentation, together with the data acquisition and processing procedures. The measurements, made at both observatories in the last couple of years, are statistically analyzed in terms of possible differences between the two data-sets, as well as data availability and quality. Next, in Section 5, the nowcast procedure for data cleaning and calculating the newly introduced index is described, including the quality and reliability assessment of the calculated index value. Comparison with the data cleaning procedure used in K-Logic is also presented. In the following Section 8, the results from the thorough evaluation of the nowcast system are presented. The study concludes, in Section 12, with a summary of the improvements provided by the proposed magnetic activity index nowcast, the evaluation results, and possible further developments.

2 Magnetic Measurements

2.1 Data Acquisition and Availability

The proposed nowcast procedure relies on real-time measurements from two closely located magnetic observatories: Dourbes and Manhay, both located in the south of Belgium. The locations of these two stations are shown on Figure 2, and the geographic and geomagnetic coordinates are listed. The observatories, at a distance of about 80 km from each other, are sufficiently afar so that the noise and interference at both locations can be assumed to be independent. On the other hand, due to the close proximity of the observatories and the low level of crustal magnetic anomalies around the observatories, as illustrated on Figure 2, the local magnetic field variations detected at both observatories are highly correlated, as presented in Section 3.

Details are in the caption following the image

The locations of the Dourbes (DOU) and Manhay (MAB) INTERMAGNET geomagnetic observatories shown on a map of the crustal magnetic anomalies in Belgium. The distance between the observatories is about 80 km, with the separation being mostly in longitude. Notice that the mean total field in MAB (48,903 nT in 2019) is larger than in DOU (48,716 nT), but the reverse is true for the horizontal component (20,029 nT for DOU and 20,157 nT for MAB). This difference is due to a larger vertical component in MAB.

The two observatories collect real-time observations of the variations of the magnetic field vector. The main instruments employed in these observatories are two identical Lemi-025 fluxgate variometer (Marusenkov et al., 2019). These instruments measure the three magnetic field components, which are defined according to the Iaga convention (Jankowski & Sucksdorff, 1996) with X directed northwards, Y directed eastwards, and Z directed vertically with positive values being downwards. It measures variations of each component within a range of ±3,000 nT and with a precision of 0.001 nT. The sampling rate of the instrument is 10 Hz, and these data are stored in our data center.

The variometer at each station is connected to a small micro controller board that applies Gaussian filtering to the 10 Hz data to produce INTERMAGNET (International Real-Time Magnetic Observatory Network) compliant second and minute resolution data with a precision of 0.01 nT (St-Louis, 2012). In this work, we only make use of the data with 1-min time resolution. The micro controllers have an ARM Linux operating system with a continuous running process that listens to the serial port, and on data availability puts it on a remote Message Queuing Telemetry Transport (MQTT) topic (ISO, 2016). This process serves as a mediator of the data from the instrument toward a remote back end, as shown in the upper part of Figure 3. On the back-end side, hosted in the Royal Meteorological Institute (RMI)'s data center in Brussels, each station has its separate MQTT topic. For each station, a data collecting process is being run so that, upon arrival, the data from the MQTT topics is taken and placed in a database. There, all the data is collected and available in real time for further processing. We use PostgreSQL 12.2 for storing the magnetic data as well as the calculated nowcast values.

Details are in the caption following the image

Schematic overview of the data-flow in the operational nowcasting system. Both magnetic observatories operate entirely independent from each other. The three indices KBEL, KDOU, and KMAB are calculated in parallel. The most appropriate index or combination of indices, available at each time can be selected for dissemination based on the use case. For example, on the web page http://ionosphere.meteo.be/geomagnetism/K_BEL/ all three indices are shown (together with the corresponding quality flags) at all times. Other users may receive only one index at each time, selected based on the quality.

Both geomagnetic observatories individually have a high level of data availability. For the two years between 2018-10-24 and 2020-10-23, data availability from DOU was 99.80%, and from MAB 99.78%. Observations from both stations need to be available in order to use the combined method for nowcasting. This limits its use to periods when both stations are providing the data, which covered 99.59% of this 2-year period. There was only a period of 51 min when the data were not available from either observatory. Therefore, when taking into account the possibility to use a single-station nowcast as a fallback when one observatory does not provide the data, the service availability for the considered period is 99.995%.

2.2 Data Analysis and Comparison

The magnetic data is stored as X, Y, Z components, and the H-component is calculated from X and Y:
urn:x-wiley:15427390:media:swe21177:swe21177-math-0001(1)
where s indicates the observatory and the time t increases in 1-min steps.
The variation in the observed magnetic field should always be considered with respect to the solar regular variation SR (Bartels et al., 1939; Menvielle et al., 1995). Obtaining the true SR curve in real time is not possible, since its derivation for any given day depends not only on the data from the past days but also on data from the future days as well. Therefore, an approximate SR is obtained instead by taking the median value for each minute over the preceding 27-day period. More precise handling of the solar regular variations can be done when deriving final index values, but these methods are not suitable for real-time operations due to the need of the data from both before and after the time period under consideration (Mayaud, 1980; Menvielle et al., 1995; Takahashi et al., 2001). Taking median values over a longer period is thus used here as an approximation for selecting only a few quietest days to establish the SR curve. Because of the size of the interval of variations assigned to a single index value, as listed in Table 1 below, the difference between either methods should not significantly influence the obtained index except possibly at the lowest values. This approximate SR curve is subtracted from the real time observations in order to obtain regularized measurements (the subscript s is again used to indicate the different observatories):
urn:x-wiley:15427390:media:swe21177:swe21177-math-0002(2)
Table 1. Limit of Range Classes for the KBEL-Index in nT, for the Dourbes (DOU) Observatory
K-index δH range classes
0 0.0–4.9
1 4.9–9.7
2 9.7–19.4
3 19.4–38.9
4 38.9–68.0
5 68.0–116.6
6 116.6–194.4
7 194.4–320.8
8 320.8–483.0
9 483.0≤
  • Note. These limits are identical to the ones used for the single-station nowcasting procedure described in Stankov et al. (2011).

Figure 4 (left panel) shows the diurnal patterns of the yearly medians of the H-components, illustrating the typical SR variations. Note that there is not only a diurnal pattern at each observatory, but also a baseline difference between the data from the DOU and MAB stations. This is in agreement with the annual means indicated on Figure 2. In addition, the diurnal variations is found to be a little larger for MAB than for DOU: the range for MAB is 14.8 nT and for DOU 12.8 nT. As can be seen in the right panel of Figure 4, the difference between the medians in the two stations only varies by a few nanotesla.

Details are in the caption following the image

Left panel: Median diurnal variations of the H-component at the Dourbes (DOU) and Manhay (MAB) observatories for 2019. In order to show the diurnal pattern similarities clearly, the data have been shifted: the vertical axis for the DOU data is on the left and for the MAB data on the right. Right panel: Distribution of differences between the medians at the DOU and MAB observatories throughout the day.

In Figure 5, the differences between the observed H-component and the median taken over the trailing 27-day period are shown for both Dourbes and Manhay on 2 days, one magnetically quiet day and one magnetically disturbed day. It is evident that the measurements obtained at both stations do indeed follow each other very closely, both during quiet and disturbed conditions. It can be seen in the bottom panel of Figure 5 that the deviations during the disturbed period are slightly larger for DOU than for MAB, while the range of the medians is slightly larger for MAB than for DOU, as presented above.

Details are in the caption following the image

Residuals of the H-component of the magnetic field at the Dourbes and Manhay observatories. Top panel: data for October 14, 2020, a magnetically quiet day. Bottom panel: data for September 28, 2020, a day with significant magnetic disturbances.

Figure 6 shows the residuals of the H-component for both magnetic observatories (top panel), as well as the difference between them in the bottom panel. A moderate geomagnetic storm occurred on November 5, 2018, with the Kp-index reaching a value of 6−, but it can be seen that the disturbances started already on November 4. The differences between the residuals in this period range from −24.9 to 17.2 nT. Manual inspection of the data showed that there are no faulty measurements in this data set, so differences of this order do not indicate data errors. The largest differences between the residuals at both stations, shown in the lower panel, occur at the beginning of the major disturbances. The behavior of the magnetic field at both observatories is very similar, as can be seen from the upper panel, but both do not react equally quickly to the onset of the disturbance. This gives rise to the brief fluctuations seen in the lower panel. This phase difference between observatories in proximity to each other is due to differences in the surroundings, in particular the ground conductivity.

Details are in the caption following the image

Top panel: the regularized data from the Dourbes (DOU) and Manhay (MAB) observatories around the time of a geomagnetic storm in November 2018. Bottom panel: The differences between the residuals at both stations.

An extreme example of an outlier in the magnetic data is presented in Figure 7. On this day, a short interruption of the measurements happened at the Dourbes observatory due to technical problems. The first measurement recorded after the restart of the magnetometer is clearly erroneous. Note that data obtained at the other observatory in Manhay does not show any disturbance at this moment.

Details are in the caption following the image

Raw magnetic data for June 11, 2020. On this day, a short interruption occurred in the observations at the Dourbes (DOU) station. The first minute of the data after the interruption is clearly faulty.

Figure 8 (left) shows the histogram of the differences between the residuals at both observatories, for a period of almost two years between 2018-11-01 and 2020-10-23. Note that the vertical scale in Figure 8 is logarithmic. The right side panel of this figure shows the relations between the respective residuals. In the period under consideration, there is only one extreme outlier, not shown in this histogram, with a difference of −150 nT; this is the example shown in Figure 7.

Details are in the caption following the image

Left panel: Histogram of the differences between the regularized H-components observed at Dourbes (DOU) and Manhay (MAB) for the period from 2018-11-01 to 2020-10-23 (note that the vertical axis uses a logarithmic scale). Right panel: correlation between the residuals at both observatories (R2 = 0.98). The outlier shown in Figure 7 is excluded from both panels.

The mean difference between the residuals at both stations is −0.004 nT, and the standard deviation is 1.089 nT. However, it can be seen from the left side panel of Figure 8 that the differences are not exactly normally distributed. This is to be expected as for instance auto-correlation between magnetic field vectors over periods of several minutes, as evident from Figures 5 and 6, can introduce such non-normal behavior. From the right side panel of Figure 8, it is evident that the residuals at both stations are indeed highly linearly correlated. This provides the justification for using intercomparison as a method for the detection of outliers. It can also be seen from this panel that deviations from the running median are more often negative than positive.

3 The Nowcast Procedures

3.1 Description and Implementation

The procedure described here for obtaining the local, two-station based KBEL nowcast comprises three main steps: first, residual variations at both observatories are calculated by regularizing the data using an approximate SR curve, as described in the previous section. Next, the residuals for the two observatories are compared, in order to identify and remove spikes and outliers in the data. Finally, the data from the Dourbes observatory are used to obtain a nowcast value. Note that the order of the steps is not the same as in the earlier work described in Stankov et al. (2011). A schematic depiction of the procedure is shown in Figure 9.

Details are in the caption following the image

Flowchart of the procedures for calculating the various K-indices. The data-cleaning step is different for the KBEL index compared to the one for KDOU and KMAB.

Based on the analyzed data from both observatories, a threshold value Dmax of 25 nT is selected as the largest allowed difference between the residuals urn:x-wiley:15427390:media:swe21177:swe21177-math-0003 and urn:x-wiley:15427390:media:swe21177:swe21177-math-0004 at the two stations. If the difference is larger than this threshold value, the point is considered an outlier and is removed from the time-series. This value is chosen in such a way as to not exclude any of the data that were verified manually to be correct. On the other hand, it can be seen from Table 1 that variations of this order will rarely affect the KBEL-index. Since the range classes for higher K-values get progressively wider, the same outlier becomes less likely to influence the index value. Since for the most space weather user applications, it is not important to distinguish between precise values of KBEL below storm level, this threshold is acceptable.

After residuals deviating more than Dmax between both stations are removed from the data set, an assessment is made of the quality of the data, based on the number of removed or unavailable data points in the 3-h interval under consideration. This assessment results in a quality control flag, an integer code indicating how reliable the cleaned data set is, based on the number of data points available in the past 1 and 3-h intervals. This takes into account both data points missing altogether due to technical problems at the observatories or interruptions of the data transmission and points removed as outliers. The criteria for determining the value of this quality flag Q are taken from Stankov et al. (2011) and are listed in Table 2.

Table 2. Criteria for Determining the Quality Flag Q
P1h P3h Q Description
60 (100%) 180 (100%) 10 Nominal
60 (100%) >171 (95%) 9 Very good
>57 (95%) >171 (95%) 8 Very good
>57 (95%) >135 (75%) 7 Good
>45 (75%) >135 (75%) 6 Good
>45 (75%) >90 (67%) 5 Fair
40 (67%) >90 (67%) 4 Fair
>20 (33%) >60 (33%) 3 Poor
P1h or P3h < 33% 2 Poor
Insufficient data 1 NA
Error value, no nowcast 0 NA
  • Note. P1h and P3h are the number of available data points in the past 1 and 3-h periods, after removal of outliers. The nominal number of points are 60 for 1 h and 180 for 3-h periods. The quality flag indicates that both the criteria for P1h and P3h at the indicated level are met.

In the data cleaning procedure described in Stankov et al. (2011), instead the standard deviation σs of the H-components is calculated for station s. The data are then excluded when Hs(t) deviates from its median value by more than 2.5σs (up to a maximum of 10% of the data). A similar procedure is employed here, for both observatories and in parallel with the one described above, for the purpose of obtaining KDOU and KMAB: the standard deviations of the residuals urn:x-wiley:15427390:media:swe21177:swe21177-math-0005 are calculated and values outside the interval ±2.5σs are discarded. Thus, three separate data frames with cleaned data are produced, one each for the calculation of KBEL, KDOU, and KMAB.

After the data have been regularized and outliers removed the three indices are calculated. Every hour, the range of urn:x-wiley:15427390:media:swe21177:swe21177-math-0006 between the maximal and minimal urn:x-wiley:15427390:media:swe21177:swe21177-math-0007-values during the past 3-h is determined:
urn:x-wiley:15427390:media:swe21177:swe21177-math-0008(3)

This range is expressed in nanotesla and is converted to the appropriate index value by comparing to the class limits in Table 1 (Stankov et al., 2011).

As illustrated in Figure 3, the KBEL-values, together with the accompanying quality flags QBEL, are stored back into the database. These values can then either be disseminated in real time to interested users or can be accessed retrospectively.

3.2 Comparison of Data Processing Methods

All indices described in this work make use of the same method of removing the solar regular variation from the magnetic data, as discussed above. However, in order to understand the source of discrepancies between our indices and Kold, we processed the horizontal components measured in Dourbes between 2018-11-01 and 2020-10-31 through the old K-Logic software used to calculate Kold. The results of this comparison are shown in Figure 10. It can be seen from the right side panel of this figure that differences in δSR between the solar regular curves exceed 10 nT in about 10% of the cases. A total of 67% of the cases have δSR of less than 10 nT in either direction. The left side panel of this figure presents the diurnal pattern of δSR, showing the largest peak around the time of sunrise. As can be seen from the class limits listed in Table 1, these differences are sufficiently large to influence the index value obtained, especially at lower index values. Such differences occur because of the different procedures for determination of the solar regular curves. The here applied method of SR estimation by using 27-day trailing medians is considered to be more appropriate for real-time applications.

Details are in the caption following the image

Left: the diurnal variations in the difference δSR between the solar regular curve we calculated and the one obtained using the old K-Logic code, averaged over the period from 2018-11-01 to 2020-10-31. The largest (negative) differences happen around the time of sunrise. Right: the distribution of δSR from the same data set. It can be seen from this histogram that variations of more than 10 nT are not uncommon. This is less obvious from the diurnal variations on the left because seasonal variations in the time of sunrise cause the largest variations to be averaged out.

Another source of discrepancies between Kold or KDOU and KBEL is in the data cleaning methods. The differences in the number of data points removed from the same set of observations from the Dourbes observatory by both the two-station and single-station data-cleaning methods is shown in Figure 11. Positive values in this plot indicate the single-station data cleaning method removed additional points, which are not really outliers. Negative values indicate the data cleaning procedure based on comparison with observations from MAB removed more measurements. It is evident from this figure that the first case is far more common. A few instances can be seen where the comparison method removed 10 or more points that were not identified as outliers by the single station procedure. Manual inspection of these cases confirmed that they are caused by problems with the data from the Manhay observatory.

Details are in the caption following the image

Differences in the number of points removed by the new and old procedures from the 3-h of data used in the calculation of an hourly index value (logarithmic scale). Data taken from the Dourbes (DOU) observatory between 2018-11-01 and 2020-10-31. Positive values mean the single-station method removed more points than the two-station comparison method.

4 Evaluation of Performance

As shown in Section 2, data availability at both stations is above 99%. Since a K-index value can still be calculated when some data is missing, that is, when less than 180 observations are available over the past 3-h interval, this high level of observational coverage translates into a better than 99.99% availability of the K-index as well.

Nevertheless, it must be considered that sometimes data are available only with some delay, due to possible technical issues with the network connection between the observatories and the central database hosted in Brussels. However, since both observatories operate entirely independent of each other, it is unlikely that interruptions would delay the transmission of the data from both stations at the same time (see Section 5). It is for this reason that the single station nowcasts are continuously run in parallel to the two-station procedure, providing a fallback system during periods when KBEL is not available.

4.1 Comparison Between the Different Nowcasting Procedures

The top row Figure 12 shows the relations between the ranges δH of residuals, calculated according to Equation 3, for the three different nowcasting procedures: the two-station nowcast KBEL and the two single-station systems KDOU and KMAB, for the period from 2018-11-01 to 2020-10-31. The bottom row of this figure shows the differences between the ranges. These ranges directly relate to the value of the index nowcast according to Table 1, but comparing the ranges gives some more detail. From the relation between δDOU and δBEL (shown in red in the middle panels), it is clear that they coincide exactly in many cases, and that in those cases where they do not coincide the range δBEL used for the two-station procedure is usually larger. For the period under consideration, δBEL and δDOU are equal in 64% of the cases. This is the expected behavior, as the two-station procedure will usually cause more data points to be retained, which are falsely identified as outliers by the single station method. Note that retaining more data points can only ever result in a larger δH, never a smaller one. The high degree of asymmetry in the differences between δBEL and δDOU is expected from the distribution shown in Figure 11. As discussed in Section 6, the rare cases where δDOU is larger than δBEL are caused by instances of the missing data from the MAB station. This leads to otherwise the acceptable data from DOU being rejected for the calculation of KBEL. These cases are therefore always associated with low values of the quality indicator QBEL.

Details are in the caption following the image

Top row: Correlation between the ranges δH as defined in Equation 3 for the three different nowcasts, after regularization and cleaning of the data. Bottom row: histogram of the differences between the various ranges of the regularized data (using logarithmic scales). All the data are taken for the 2-year period from 2018-11-01 to 2020-10-31.

The two-station nowcasting procedure ultimately derives a K-value from the data obtained from the Dourbes observatory, the data from Manhay is only used for data validation. Thus, it can be expected that the differences between δBEL and δMAB, shown in blue in the right side panels of Figure 12, are larger than those between δBEL and δDOU. Nevertheless, it can be seen from the figure that even these differences rarely exceed ±10 nT, in most cases they are below ±5 nT. Therefore, the resulting K-index nowcast values are almost always identical. Because the boundary values listed in Table 1 are closer together for lower index values, the most differences between nowcast values from the three different procedures occur for these cases. This is not an issue in practice, as most user applications are not affected by the exact K-value at low values, as already stated.

Figure 13 displays the distribution of KBEL-values and the corresponding QBEL, similar to Figure 1 for the old nowcasting system. It can be seen here that the distribution of K-values for the new system is similar to that of the old system. However, there is about an order of magnitude less data available, and in particular, no cases of KBEL > 6 have occurred in the period used for this study. Nevertheless, the quality of the new nowcasting system is much better and is approximately constant for all index values, lacking the negative correlation between K and Q seen for the old index in Figure 1.

Details are in the caption following the image

Left panel: the averages of the data quality flag QBEL for the new, two-station nowcast, for those cases where the quality flag is at least 6 (leaving out again the cases of low data availability, as in Figure 1). Comparison to Figure 1 shows that the averages of the data quality flag have been significantly increased. Here, the vertical axis starts at 9.75, while it starts at 7 in Figure 1. Right panel: distribution of occurrences of KBEL-values in the nowcast (logarithmic scale).

4.2 Comparison With the Old Nowcasting System

Table 3 shows the frequency distribution of the index values provided by the old and new nowcasting systems. The period for which the new system has been tested falls entirely during a period of a low solar activity while the old system is available since 2002, covering more than a full solar cycle. Therefore, the higher values of K appear less often, while the most common index value of 1 occurred significantly more frequently. When comparing the old and new procedures for only the period from 2018-11-01 to 2020-10-31, during which both are available, the distributions are much closer to each other. The main difference is that the new system gives a value of 1, slightly more often (40.1% compared to 35.5%) while giving somewhat fewer instances of K = 0 (9.1% instead of 12.2%). This can be understood, as the old data cleaning procedure is more likely to falsely identify the data as outliers during very quiet conditions, thereby artificially lowering the nowcast value further. The outlier identification in the old system, as in the current single station procedures, relies on the standard deviation σ of the residuals. σ is smaller during quiet conditions, leading to more data being removed as outliers. Most user applications are not concerned with the precise value of the index at low activity conditions, rendering this difference less important in practice. In principle, however, the new two-station system is more accurate since the limits for outlier identification do not vary with the level of the geomagnetic activity.

Table 3. Frequency of Occurrence for K-Index Values for the Old and New Nowcasting Systems
Value Old nowcast (All) Old nowcast (2 years) New nowcast KBEL
0 9.1% 12.2% 9.1%
1 27.0% 35.5% 40.1%
2 35.3% 35.8% 35.8%
3 21.3% 13.9% 12.4%
4 5.7% 2.2% 2.3%
5 1.3% 0.4% 0.4%
6 0.2% <0.1% <0.1%
7 <0.1%
8 <0.1%
9 <0.1%
  • Note. Only values with a quality flag of at least 6 are counted. The data for the new system span the period from 2018-11-01 to 2020-10-31; the data for the old system ranges from 2002-01-01 to 2020-10-31, with the third column showing the statistics of only the data coinciding with the new system's operation. No index values of 7 or higher have yet occurred during the operation of the new nowcasting system.

Over the 2-year period during which both the old and new nowcasting systems are available, there is a total of 17,166 hourly nowcasts, where both systems have a quality flag of six or higher. For these cases, we calculated the differences in K-values. This distribution is shown in Table 4

Table 4. Distribution of Differences Between the New and Old Nowcasting Systems (Data Taken Between 2018-11-01 and 2020-10-31, for Those Times When Both Indices are Available)
KBEL − Kold Occurrences
Qold ≥ 6 6 ≤ Qold  ≤ 9
−3 3 0
−2 159 10
−1 2,577 502
0 11,578 3,846
1 2,759 1,147
2 89 45
3 1 0

The middle panel of the top row of Figure 12 shows that δBEL − δDOU is almost always zero or positive, while the distribution shown in Table 4 shows that the K value from the old system is often larger than that provided by the new system. The reason for this is are the differences in the removal of the SR variation, as discussed in Section 6. This is further demonstrated in Figure 14, where the average value of the nowcast is plotted as a function of the time of day. It is evident from this figure that the cases where the old system provides a larger K value are concentrated primarily around the period of sunrise, and to a lesser extend in the evening around the time of sunset. These discrepancies are due to the different methods for removing the SR curves, as illustrated in Figure 10. We can conclude that for these cases the value from the new nowcast system should be accepted as more reliable, since the differences are caused by faulty removal of the correct data in deriving Kold.

Details are in the caption following the image

Diurnal pattern of the K values obtained from the old nowcasting system, using the data from 2018-11-01 to 2020-10-31. The vertical bars show the number of cases where KBEL − Kold ≤ −2.

Conversely, the cases where the new KBEL method provides a higher value are associated with a lower quality of the old nowcast. KBEL − Kold ≥ 1 for 14.3% of the cases where Qold = 10, but in 21.5% of the cases when Qold ≤ 9. KBEL − Kold ≥ 2 occurs with a probability of 0.4% among cases where Qold = 10 but is twice is likely, with probability 0.8%, in cases where Qold ≤ 9. Therefore, in these cases as well the new system can be considered to be the most accurate one.

Regardless of whether KBEL is larger or smaller than Kold, the new system provides a more accurate nowcasts. The reasons for the discrepancies are, however, different: larger values of Kold are primarily the result of residual influences of the SR curve (see the comparison in Section 6), while smaller values are due to the incorrect removal of real disturbances, as discussed above. Note that even the current implementation of the nowcast system does not completely remove all quiet time diurnal variation from the data because the SR curve is only approximated with a trailing average. The top panel of Figure 5, for instance, exhibits some small variations in the residuals throughout the day.

4.3 Comparison With the Planetary Kp-Index

Here, we compare the proposed local nowcast system with the definitive Kp-values for the period from 2018-11-01 to 2020-10-31, provided by GFZ Potsdam through the International Service of Geomagnetic Indices. A direct comparison between the planetary Kp-index and our local nowcasts is not so straightforward, as the nowcasting system is not intended to correlate very closely to the planetary index but rather to indicate local levels of geomagnetic disturbances. Nevertheless, there should be a correlation between the global activity levels indicated by Kp and our local index KBEL. The Kp-index can be converted back into a range value ap, in units of nanotesla. However, this range does not correspond directly to observations at any single observatory, as the Kp-index itself is based on the data from multiple stations. Therefore, we do the comparison here with the values of the index. For this purpose, the Kp-values were binned in integer intervals, with Kp equal to 0o, 0+, 1− binned under 0, 1o, 1+, 2− binned under 1, and so on. Since the Kp-index is only provided every 3-h, we consider here only the values of our KBEL index at these same times.

The left panel of Figure 15 shows the distribution of the values of KBEL and Kp, again considering the same 2-year period as before. The KBEL distribution is of course nearly identical to the right side panel of Figure 13. The distribution of the Kp-index can be seen to differ, especially giving more often a value equal to 0 and fewer instances of values 1 and 2. In part, this may be due to the way the Kp indices were converted to integer values: if 1− was converted to 1, instead of 0, 2− to 2 instead of 1, and so on, the distributions may match each other more closely. However, in part this is also a physically reasonable finding because the KBEL index is sensitive to small, regional disturbances that may not be significant enough on a global scale to affect Kp. Remember that the index scale is quasilogarithmic, so at small index values even a minor local disturbance can change the value. The right side panel of Figure 15 shows the relation between both indices. It is again evident that at low values KBEL tends to be somewhat larger than Kp at the same moment, while at larger values both indices usually coincide.

Details are in the caption following the image

Left panel: the distribution of Kp (blue squares) and of KBEL (red diamonds) taken at the same times, for the 2-year period identical to the one used in Figure 13. Right panel: the correlation between Kp and KBEL, using the same data.

Due to the current very low solar activity levels and the associated rarity of major geomagnetic storms, few instances of large index values are available for comparison at this time. Follow-up evaluations will therefore be required when a longer period of the data, including more severe disturbances, is available.

5 Conclusions

An improved nowcast service for estimating the local magnetic activity was presented. A novel approach is the use of concurrent magnetic measurements carried out by two identical instruments at two observatories in close proximity. In this way, the data cleaning is more reliable. Primarily, it allows for discerning a real disturbance from a spike in the magnetogram, which is always difficult to achieve with single-station methods. At the same time, the method for removing the quiet time SR is improved compared to the previous nowcasting method, further increasing the reliability of the system. Single-station nowcasts are still calculated in parallel for both magnetic observatories, resulting in three concurrently and independently operating nowcasting systems. This approach all but guarantees the availability of the nowcast at all times, which is a prerequisite for inclusion into space weather services related to crucial safety-of-life applications. The new system, including the single station indices for both observatories, can be found online at http://ionosphere.meteo.be/geomagnetism/K_BEL/.

The nowcasting system, as it is described here, provides an index value every hour. However, the method used is independent of the cadence of the nowcasting, and can readily be configured to provide values at, for instance, 15-min intervals, or even shorter. Over a period of two years, the combined method for nowcasting could be used to cover 99.59% of the time. When including the use of the single station fallback procedure, availability is better than 99.99%. Therefore, even a nowcasting cadence of one minute is achievable, if future user requirements call for it. Such high time resolution of providing the index can decrease substantially the usual delay in identifying an oncoming geomagnetic storm. The nowcast service has been evaluated using two years of magnetic records and results show that it is much improved compared to the one that was previously in operation.

Comparison of the local KBEL index to the planetary Kp index shows that the local index gives on average a somewhat higher value, especially at low index values. Particularly, the local index gives most often values of 1 or 2, while the Kp index often has values below 1 (including 0+ and 1−). At higher values, our local nowcast more often coincides with the value of Kp. This can be due to a combination of factors: the local index, especially since it is calculated hourly, is more sensitive to short and small scale disturbances, and there is some ambiguity in converting the Kp scale to simple integers as we use for our index.

Further evaluation of the nowcasting system will continue, especially with regards to its performance during periods of the higher geomagnetic activity, and improvements to the system made. In particular, the threshold value Dmax defined in Section 5 could be modified in the future to fine tune performance.

The method described here can be implemented without much difficulty to other locations where two geomagnetic observatories are located sufficiently close to each other. One requirement is of course that the limits for the K-classes, such as those in Table 1, are known for the primary observatory. An open question at this time is how far the two observatories considered can be separated from each other while still allowing the use of the procedure we describe here. In general, this maximal possible distance will vary across the globe, and one has to compare the variations at the two observatories involved to verify their compatibility in order to implement a similar nowcasting system in other locations.

Acknowledgments

This study is supported by the Belgian Solar-Terrestrial Center of Excellence (STCE). F. Humbled and S. Bracke acknowledge the support of BELSPO via the Magnetic Valley project. The authors thank the involved national institutes, the INTERMAGNET network and ISGI (isgi.unistra.fr). They thank A. Gonsette for providing the map of crustal anomalies for Belgium shown in Figure 2.

    Data Availability Statement

    The provisional as well as definitive data from the two Belgian observatories in Dourbes and Manhay are available for download at INTERMAGNET (www.intermagnet.org). The results presented in this paper rely on the Kp-index calculated and made available by GFZ Potsdam from data collected at magnetic observatories (Matzka et al., 2021).