Level-3.3: USTAR filtering

Level-3.3: USTAR filtering#

Info

Level-3.3 creates additional quality flags to flag time periods of low turbulence. No data are removed in this step, only the flags are created.

Info

Fluxes filtered with USTAR threshold: NEE, FN2O, FCH4

Fluxes not filtered with USTAR threshold: LE (and therefore ET), H

  • see reasoning in Pastorello et al. (2020):

The USTAR filtering is not applied to H and LE, because it has not been proved that when there are CO2 advective fluxes, these also impact energy fluxes, specifically due to the fact that when advection is in general large (nighttime), energy fluxes are small.

Summary#

The Level-3.3 USTAR filtering process flags low-turbulence periods to quantify uncertainty in \(\text{NEE}\) (Net Ecosystem Exchange) estimates, leaving energy fluxes (H, LE) unfiltered. This is achieved by calculating three Constant USTAR Thresholds (CUT_50, CUT_16, CUT_84) based on the median and uncertainty bounds (16th and 84th percentiles) of the USTAR distribution. These thresholds are derived from a REddyProc bootstrapping analysis run separately for different IRGAs, with the highest threshold chosen for the final combined application across all years.

Description#

Turbulence filtering at Level-3.3 involved flagging low turbulence periods using constant USTAR thresholds (CUT) across all years. These thresholds were detected using the function usControlUstarEst from the R library REddyProc. The method is based on the approach described in Pastorello et al. (2020). Detected USTAR thresholds were used to filter NEE fluxes for low turbulence conditions, while energy fluxes LE and H were not filtered because the impact of advection on energy fluxes differs (Pastorello et al., 2020).

Since in this dataset we have two time periods that use different IRGAs (open-path IRGA LI-7500 between 2005 and 2016, and for a short period in 2019; enclosed-path LI-7200 from 2016 onwards), the USTAR thresholds were detereminded separately for each of these two periods. Following Pastorello et al. (2020), three USTAR scenarios were considered to calculate three NEE versions to provide a range of best-estimate and uncertainty-bounding flux versions: CUT_50, CUT_16, and CUT_84, corresponding to the 50th, 16th, and 84th percentiles from the REddyProc bootstrapping result distribution (from 100 bootstrapping runs), respectively.

Details#

A constant USTAR threshold (CUT) is used for all years (same threshold for all years). Threshold values are based on USTAR detection results from the REddyProc bootstrapping analysis. The threshold detection was done using data between 2005 and 2016 (2019) for the LI-7500, and from 2016 onwards for the LI-7200.

Before the USTAR detection algorithm was run, Level-1 fluxes were quality-checked and storage-corrected fluxes (NEE) were calculated. Low quality records were rejected and the retained records were used in the USTAR analysis. The settings for the USTAR detection algorithms were slightly different for the two IRGAs, see next sections.

Results for LI-7500 (2005-2016, 2019)#

Compared to the default settings of the function usControlUstarEst in REddyProc, the following adjustments were made: ustPlateauFwd = 8 (default: 10), ustPlateauBack = 4 (default: 6), UstarClasses = 30 (default: 20). The input NEE data used for the analysis was less strictly filter compared to LI-7200: only high-quality nighttime fluxes (QCF=0) were used.

  • CUT_50: 0.303628125

  • CUT_16: 0.268383876

  • CUT_84: 0.339684084

Results for LI-7200 (2016 onwards)#

Compared to the default settings of the function usControlUstarEst in REddyProc, the following adjustments were made: ustPlateauFwd = 9 (default: 10), ustPlateauBack = 5 (default: 6), UstarClasses = 30 (default: 20). The input NEE data used for the analysis was more strictly filter compared to LI-7500: medium-quality and high-quality nighttime fluxes (QCF=0 and QCF=1) were used. This way the detection of the USTAR plateau (nighttime NEE vs USTAR) worked better.

  • CUT_50: 0.30181125

  • CUT_16: 0.271619922

  • CUT_84: 0.333477687

Constant thresholds used for all years#

Since the thresholds for the two IRGA models are very similar, they were combined by selecting the higher threshold detected for the respective USTAR scenario. The three resulting thresholds used for all years were:

  • CUT_50: 0.303628125 This thresholds is used to create the the main NEE version (best estimate), corresponding to the 50th percentile (median) from the REddyProc detection results. This value represents the central tendency of the u∗ distribution. It’s the best statistical estimate of the point where NEE becomes independent of u∗. The NEE calculated using this threshold is considered the best estimate of the true ecosystem flux.

  • CUT_16: 0.271619922 Lower bound scenario, corresponding to the 16th percentile from the REddyProc detection results. The 16th percentile is mathematically equivalent to the mean minus one standard deviation (μ−1σ), assuming the distribution of u∗ values is roughly normal (Gaussian). This lower threshold means less data are filtered out. It results in an NEE version that is typically less positive (or even negative) at night because it retains more low-turbulence measurements, which are often biased toward zero or net uptake.

  • CUT_84: 0.339684084 Upper bound scenario, corresponding to the 84th percentile from the REddyProc detection results. The 84th percentile is equivalent to the mean plus one standard deviation (μ+1σ). This higher threshold means more data are filtered out.

TODO#

TODO#

TODO QCF: overall quality control flag after Level-3.3 tests#

See here for a description of how QCF is calculated.

  • For subsequent steps (Level-4.1+), fully quality-filtered fluxes are needed.

  • Therefore, after running the individual quality tests in previous steps, the single flags are combined into the overall QCF.

  • This creates flux variables that have been filtered by all quality flags created in Level-2, Level-3.2 and Level-3.3.

  • The overall quality flag after Level-3.3 is named FLAG_L3.3_<ustar_scenario>_<flux>_QCF.

  • Example for NEE:

    • FLAG_L3.3_CUT_50_NEE_L3.1_QCF is the QCF for NEE_L3.1 after Level-3.3.

    • This flag is used to filter the variable NEE_L3.1 and creates the quality-filtered variable NEE_L3.1_L3.3_CUT_50_QCF, that is then used in the following steps.

    • Note that NEE_L3.1 is the unfiltered, but storage-corrected variable FC from the flux calculations.

  • Examples for <ustar_scenario>:

    • CUT_16, CUT_50, CUT_84

  • Examples for <flux>:

    • FC_L3.1, LE_L3.1, H_L3.1, FN2O_L3.1, FCH4_L3.1

    • These are fluxes that are storage-corrected (Level-3.1), but are not filtered yet.

  • Example output from diive for NEE_L3.1 (storage-corrected FC from Level-3.1) for the USTAR scenario CUT_50, detailing the sequential application of all individual quality flags from Level-2, Level-3.2 and Level-3.3:

========================================
QCF FLAG EVOLUTION
========================================
This output shows the evolution of the QCF overall quality flag
when test flags are applied sequentially to the variable NEE_L3.1.

Number of NEE_L3.1 records before QC: 295350
+++ FLAG_L2_FC_MISSING_TEST rejected 0 values (+0.00%)      TOTALS: flag 0: 295350 (100.00%) / flag 1: 0 (0.00%) / flag 2: 0 (0.00%)
+++ FLAG_L2_FC_SSITC_TEST rejected 133899 values (+45.34%)      TOTALS: flag 0: 115080 (38.96%) / flag 1: 46371 (15.70%) / flag 2: 133899 (45.34%)
+++ FLAG_L2_FC_COMPLETENESS_TEST rejected 690 values (+0.23%)      TOTALS: flag 0: 114423 (38.74%) / flag 1: 46338 (15.69%) / flag 2: 134589 (45.57%)
+++ FLAG_L2_FC_SCF_TEST rejected 194 values (+0.07%)      TOTALS: flag 0: 114201 (38.67%) / flag 1: 46366 (15.70%) / flag 2: 134783 (45.64%)
+++ FLAG_L2_FC_SIGNAL_STRENGTH_TEST rejected 9808 values (+3.32%)      TOTALS: flag 0: 110029 (37.25%) / flag 1: 40730 (13.79%) / flag 2: 144591 (48.96%)
+++ FLAG_L2_FC_CO2_VM97_SPIKE_HF_TEST rejected 942 values (+0.32%)      TOTALS: flag 0: 109335 (37.02%) / flag 1: 40482 (13.71%) / flag 2: 145533 (49.27%)
+++ FLAG_L2_FC_CO2_VM97_AMPLITUDE_RESOLUTION_HF_TEST rejected 3826 values (+1.30%)      TOTALS: flag 0: 106807 (36.16%) / flag 1: 39184 (13.27%) / flag 2: 149359 (50.57%)
+++ FLAG_L2_FC_CO2_VM97_DROPOUT_TEST rejected 0 values (+0.00%)      TOTALS: flag 0: 106807 (36.16%) / flag 1: 39184 (13.27%) / flag 2: 149359 (50.57%)
+++ FLAG_L2_FC_VM97_AOA_HF_TEST rejected 2607 values (+0.88%)      TOTALS: flag 0: 105860 (35.84%) / flag 1: 37524 (12.70%) / flag 2: 151966 (51.45%)
+++ FLAG_L3.2_NEE_L3.1_QCF_OUTLIER_ABSLIM_TEST rejected 2038 values (+0.69%)      TOTALS: flag 0: 105590 (35.75%) / flag 1: 35756 (12.11%) / flag 2: 154004 (52.14%)
+++ FLAG_L3.2_NEE_L3.1_QCF_OUTLIER_MANUAL_TEST rejected 731 values (+0.25%)      TOTALS: flag 0: 105303 (35.65%) / flag 1: 35312 (11.96%) / flag 2: 154735 (52.39%)
+++ FLAG_L3.2_NEE_L3.1_QCF_OUTLIER_HAMPELDTNT_TEST rejected 4302 values (+1.46%)      TOTALS: flag 0: 103478 (35.04%) / flag 1: 32835 (11.12%) / flag 2: 159037 (53.85%)
+++ FLAG_L3.2_NEE_L3.1_QCF_OUTLIER_LOCALSD_TEST rejected 282 values (+0.10%)      TOTALS: flag 0: 103432 (35.02%) / flag 1: 32599 (11.04%) / flag 2: 159319 (53.94%)
+++ FLAG_L3.3_CUT_50_NEE_L3.1_USTAR_TEST rejected 16564 values (+5.61%)      TOTALS: flag 0: 92913 (31.46%) / flag 1: 26554 (8.99%) / flag 2: 175883 (59.55%)

In total, 175883 (59.55%) of the available records were rejected in this step.
INFO Rejected DAYTIME records where QCF flag >= 2
INFO Rejected NIGHTTIME records where QCF flag >= 1

========================================
SUMMARY: FLAG_L3.3_CUT_50_NEE_L3.1_QCF, QCF FLAG FOR NEE_L3.1
========================================
Between 2005-01-01 00:15 and 2024-12-31 23:45 ...
    Total flux records BEFORE quality checks: 295350 (84.23% of potential)
    Available flux records AFTER quality checks: 119467 (40.45% of total)
    Rejected flux records: 175883 (59.55% of total)
    Potential flux records: 350640
    Potential flux records missed: 55290 (15.77% of potential)
Contents