View source for Differences of Calculated Results

[[de:Differenzen der Berechnungsergebnisse]]
__NOAUTOLINKS__
__NOAUTOLINKTARGET__ 

==Introduction==

For data generated by
* [[Mathematical Models for Coastal Areas and Estuaries|mathematical models]] ([[model]] results), or
* [[Analysis of Calculated Results|analysis of calculated results]] (characteristic numbers), or
* measured data (observational data)
Various differences can be computed. Input data can be typically categorized as follows:
* '''Category K0''': <math>f(x,y,z)</math>, time-independent quantities;
* '''Category K1''': <math>f(x,y,z,t_1)</math>, time-dependent quantities, one time step;
* '''Category KC''': <math>f(x,y,z,t_i)</math>, time-dependent quantities, several discrete time steps, constant time step <math>\Delta_t</math>;
* '''Category KN''': <math>f(x,y,z,t_i)</math>, time-dependent quantities, several discrete time steps, varying time step <math>\Delta_t(i)</math>.
For geophysical data categories K1, KC and KN are of significance. Examples:
* Category K1: topography/bathymetry <math>h(x,y,z,t_1)</math> for a specific instant in time;
* Category KC: [[water level]] <math>\eta(x,y,z,t_i)</math> at discrete times <math>t_i</math> with constant time step, e. g. computed by a mathematical model;
* Category KN: tidal high water <math>\eta^{\rm{HW}}(x,y,z,t_i)</math> for times <math>t_i</math> at ''non-equidistant'' time intervals, e.g. derived from a water level time serie. 

==Definitions==

* '''reference data <math>r</math>''': with respect to <math>r</math> various ''deviations'' for <math>f</math> can be evaluated. Typical data are either observational data or computational as well as analysis results for a specific (reference) state (situation); 
* '''variant data <math>f</math>''': can be also either observational data or computational as well as analysis results, for which deviations shall be computed with respect to the reference state. Typically variant data are given for a different period in time (natural variation) or a different state of the system under study.
* '''valid operator 1''': <math>V(r_i)</math> returns .T. or .F., in dependence whether <math>r_i</math> is valid or invalid. Can be also applied to <math>f_i</math>.
* '''valid operator 2''': <math>V(r_I,f_i)</math> returns .T. or .F., in dependence whether <math>V(r_i)\land V(f_i)</math> is valid or invalid.
* '''integer operator 1''': <math>P(r_i)</math> returns 1 if <math>V(r_i)</math> else 0. Similar for <math>f_i</math>.
* '''integer operator 2''': <math>P(r_i,f_i)</math> returns 1 if <math>V(r_i)\land V(f_i)</math> .T. else 0.

==Requirements for the computation of differences==

The following requirements must be fulfilled by <math>r</math> and <math>f</math>:
# <math>r</math> and <math>f</math> must belong to the same category (see above);
# the number of times <math>t_i</math> must be identical for <math>r</math> and <math>f</math>;
# for data belonging to category KC constant time steps must coincide <math>\Delta t</math> for <math>r</math> and <math>f</math>;
# (physical) dimension as well as meaning must be equivalent for <math>r</math> and <math>f</math>;
# <math>r_i</math> (short for <math>r(x,y,z,t_i)</math>) as well as <math>f_i</math> (short for <math>r(x,y,z,t_i)</math>) must be valid data for the same instant <math>i</math> in time; otherwise the dervied results will become invalid. 

==Computational results==

Program [[NCDELTA]] can be used to compute all subsequent results. Locations of <math>r</math> are not required to coincide with those of <math>f</math>. Values <math>r</math> are interpolated to locations of <math>f</math>, as long as the geographical distance between the different locations does not exceed <math>R^\max</math>. In case the distance ''exceeds'' that limit, no results will be computed. In such a situation an  ''invalid'' result value will be generated. The follwing results can be computed using [[NCDELTA]]. 

===Ordinary differences===

====Difference====

A result is computed for all times (one value for time-independent data) at all locations <math>(x,y,z)</math>:
# The difference between <math>f_i</math> and <math>r_i</math> is calculated in case <math>V(r_i,f_i)</math> returns .T.:
#:<math>d_i = f_i - r_i</math>, if <math>V(r_i,f_i)</math>; 
# Result will be ''invalid'', if <math>V(r_i,f_i)</math> returns .F.:
#:<math>d_i = \rm{invalid}</math> if <math>\lnot V(r_i,f_i)</math>.
Results are computed for data belonging to categories '''K0''', '''K1''', '''KC''' und '''KN''', which means for all types of data.

====Maximum difference====

Maximum difference is determined using ''absolute'' value (modulus) in combination with sign preservation:
# At first all differences <math>d_i</math> will be computed as indicated above;
# Out of all valid data [[index]] <math>i^\max</math> is determined in such a way that <math>\left|d_i\right|</math> is maximal
#:<math>d^\max = d_{i^\max}</math>
#:is equal to the ''maximum difference'' according to this definition; this value can be negative, positive or zero;
# In case all values <math>d_i</math> are invalid <math>d^\max = \rm{invalid}</math> will be set.
Computation is performed for categories '''KC''' and '''KN'''. A valid <math>d^\max</math> is obtained as long there exists at least one valid difference <math>d_i</math>. During visualization [[NCPLOT]] enables filtering using ancillary variable ''Number of valid differences''.

====Minimum difference====

Minimum difference is determined using ''absolute'' value (modulus) in combination with sign preservation:
# At first all differences <math>d_i</math> will be computed as indicated above;
# Out of all valid data index <math>i^\min</math> is determined in such a way that <math>\left|d_i\right|</math> is minimal
#:<math>d^\min = d_{i^\min}</math>
#:is equal to the ''minimum difference'' according to this definition; this value can be negative, positive or zero;
# In case all values <math>d_i</math> are invalid <math>d^\min = \rm{invalid}</math> will be set.
Computation is performed for categories '''KC''' and '''KN'''. A valid <math>d^\min</math> is obtained as long there exists at least one valid difference <math>d_i</math>. During visualization [[NCPLOT]] enables filtering using ancillary variable ''Number of valid differences''.

====Mean difference====

Mean value is computed for all valid differences:
# At first all differences <math>d_i</math> will be computed as indicated above;
# From all valid differences the mean value is computed as
#:<math>d^{\rm{mit}}=\frac{\sum_{i\in I}P(d_i)d_i}{\sum_{i\in I}P(d_i)}</math>;
# In case all <math>d_i</math> are invalid <math>d^{\rm{mit}} = \rm{invalid}</math> will be set.
Computation is performed for categories '''KC''' and '''KN'''. A valid <math>d^{\rm{mit}}</math> is obtained as long there exists at least one valid difference <math>d_i</math>. During visualization [[NCPLOT]] enables filtering using ancillary variable ''Number of valid differences''.

====Mean deviation====

Mean deviation is computed for all valid differences:
# At first all differences <math>d_i</math> will be computed as indicated above;
# From all valid differences the mean deviation is computed as
#:<math>d^{\rm{abw}}=\frac{\sum_{i\in I}P(d_i)\left|d_i\right|}{\sum_{i\in I}P(d_i)}</math>;
# In case all <math>d_i</math> are invalid <math>d^{\rm{abw}} = \rm{invalid}</math> will be set.
Computation is performed for categories '''KC''' and '''KN'''. A valid <math>d^{\rm{abw}}</math> is obtained as long there exists at least one valid difference <math>d_i</math>. During visualization [[NCPLOT]] enables filtering using ancillary variable ''Number of valid differences''.

====Root mean square error (RMSE)====

RMSE is computed for all valid differences:
# At first all differences <math>d_i</math> will be computed as indicated above;
# From all valid differences the mean deviation is computed from the well known definition for RMSE;
# In case all <math>d_i</math> are invalid RMSE = '''invalid''' will be set.
Computation is performed for categories '''KC''' and '''KN'''. A valid RMSE is obtained as long there exists at least one valid difference <math>d_i</math>. During visualization [[NCPLOT]] enables filtering using ancillary variable ''Number of valid differences''.

====Number of valid differences====

The number of valid differences may vary between different locations. As a consequence the above mentioned quantities may be computed from data sets of different size: 
# At first all differences <math>d_i</math> will be computed as indicated above;
# The number of valid differences is computed as
#:<math>N_{\rm{ord}}=\sum_{i\in I}P(d_i)</math>;
Computation is performed for categories '''KC''' and '''KN'''. In program [[NCPLOT]] quantities like ''maximum difference'', ''minimum difference'', ''mean value'' as well as ''mean deviation'' can be visualised using this variable as a [[filter]]. 
In that way visualizations can be created where the results are shown for points only where e. g. either the maximum number of events or a specific number of events occurred.

===Data for a Taylor diagram===

Taylor diagrams provide "''a concise statistical summary of how well patterns match each other in terms of their correlation, their root-mean-square difference and the ratio of their variances''."  Additional information such as [[bias]] can be added to the conventional Taylor diagram.  The Taylor diagram provides a graphical framework that allows a suite of variables from a variety of (say) one or more models or reanalyses to be compared to reference data. The reference data can be observationally based (eg, [[reanalysis]]) or to another model or a control run. 

Literature and further informations:
# Taylor, K. E. (2001), ''Summarizing multiple aspects of model performance in a single diagram'', Journal of Geophysical Research, 106 (D7), 7183–7192, doi: [http://dx.doi.org/10.1029/2000JD900719 http://dx.doi.org/10.1029/2000JD900719]; 
# [http://www-pcmdi.llnl.gov/about/staff/Taylor/CV/Taylor_diagram_primer.htm http://www-pcmdi.llnl.gov/about/staff/Taylor/CV/Taylor_diagram_primer.htm] with a short introduction by K. E. Taylor as well as links to example applications.

====Standard deviation for reference data====

Standard deviation is going to be computed for all valid reference data:
# Mean value for <math>r_i</math> is computed according to
#:<math>\bar{r}=\frac{\sum_{i\in I}P(r_i,f_i)r_i}{\sum_{i\in I}P(r_i,f_i)}</math>;
# If a valid <math>\bar{r}</math> was computed standard deviation is obtained from
#:<math>\sigma_r = \sqrt{\frac{\sum_{i\in I}P(r_i,f_i)\left(r_i-\bar{r}\right)^2}{\sum_{i\in I}P(r_i,f_i)}}</math>;
# In case all <math>P(r_i,f_i)</math> are 0 <math>\sigma_r = \rm{invalid}</math> is set.
Computation is performed for categories '''KC''' and '''KN'''. A valid <math>\sigma_r</math> is obtained as long as there exists at least one valid difference <math>V(r_i,f_i)</math>. Standard deviations for different locations may be computed from data sets of different size. During visualization [[NCPLOT]] enables filtering using ancillary variable ''Number of valid Taylor data''.

Remark: We apply <math>P(r_i,f_i)</math> instead of <math>P(r_i)</math>. This guarantees that all Taylor data are computed for the same data set size at a specific location.

====Standard deviation for variant data====

Standard deviation is going to be computed for all valid variant data:
# Mean value for <math>f_i</math> is computed according to
#:<math>\bar{f}=\frac{\sum_{i\in I}P(r_i,f_i)f_i}{\sum_{i\in I}P(r_i,f_i)}</math>;
# If a valid <math>\bar{f}</math> was computed standard deviation is obtained from
#:<math>\sigma_f = \sqrt{\frac{\sum_{i\in I}P(r_i,f_i)\left(f_i-\bar{f}\right)^2}{\sum_{i\in I}P(r_i,f_i)}}</math>;
# In case all <math>P(r_i,f_i)</math> are 0 <math>\sigma_f = \rm{invalid}</math> is set.
Computation is performed for categories '''KC''' and '''KN'''. A valid <math>\sigma_f</math> is obtained as long as there exists at least one valid difference <math>V(r_i,f_i)</math>. Standard deviations for different locations may be computed from data sets of different size. During visualization [[NCPLOT]] enables filtering using ancillary variable ''Number of valid Taylor data''.

Remark: We apply <math>P(r_i,f_i)</math> instead of <math>P(r_i)</math>. This guarantees that all Taylor data are computed for the same data set size at a specific location.

====Correlation====

Correlation is computed for <math>r</math> and <math>f</math>:
# Mean value <math>\bar{r}</math> for reference data is computed as indicated above;
# Mean value <math>\bar{f}</math> for variant data is computed as indicated above;
# Standard deviation <math>\sigma_r</math> for reference data is computed as indicated above;
# Standard deviation <math>\sigma_f</math> for variant data is computed as indicated above;
# Correlation <math>R</math> is given as follows
#:<math>R=\frac{\sum_{i\in I}P(r_i,f_i)\left(r_i-\bar{r}\right)\left(f_i-\bar{f}\right)}{\sigma_r\sigma_f\sum_{i\in I}P(r_i,f_i)}</math>;
# In case all <math>V(r_i,f_i)</math> are invalid <math>R = \rm{invalid}</math> is set.
Computation is performed for categories '''KC''' und '''KN'''. A valid <math>R</math> is obtained as long as there exists at least one valid difference <math>V(r_i,f_i)</math>. Correlations for different locations may be computed from data sets of different size. During visualization [[NCPLOT]] enables filtering using ancillary variable ''Number of valid Taylor data''.

====Pattern RMS====

Pattern RMS is computed for <math>r</math> and <math>f</math>:
# Mean value <math>\bar{r}</math> for reference data is computed as indicated above;
# Mean value <math>\bar{f}</math> for variant data is computed as indicated above;
# Pattern RMS <math>E'</math> is given as follows
#:<math>E'=\sqrt{\frac{\sum_{i\in I}P(r_i,f_i)\left[\left(r_i-\bar{r}\right)\left(f_i-\bar{f}\right)\right]^2}{\sum_{i\in I}P(r_i,f_i)}}</math>;
# In case all <math>V(r_i,f_i)</math> are invalid <math>E' = \rm{invalid}</math> is set.
Computation is performed for categories '''KC''' und '''KN'''. A valid <math>E'</math> is obtained as long as there exists at least one valid difference <math>V(r_i,f_i)</math>. Correlations for different locations may be computed from data sets of different size. During visualization [[NCPLOT]] enables filtering using ancillary variable ''Number of valid Taylor data''.

====Deviation of means (''Bias'')====

Deviation of means is computed for <math>\bar{r}</math> and <math>\bar{f}</math>; this quantity is also called ''bias'':
# Mean value <math>\bar{r}</math> for reference data is computed as indicated above;
# Mean value <math>\bar{f}</math> for variant data is computed as indicated above;
# Deviation of means <math>\bar{E}</math> is given as follows
#:<math>\bar{E}=\bar{f}-\bar{r}</math>;
# In case all <math>V(r_i,f_i)</math> are invalid <math>\bar{E} = \rm{invalid}</math> is set;
# Overall RMS <math>E</math> can be computed from <math>\bar{E}</math> and <math>E'</math> according to
#: <math>E = \sqrt{\bar{E}^2+E'^2}</math>;
Computation is performed for categories '''KC''' und '''KN'''. A valid <math>\bar{E}</math> is obtained as long as there exists at least one valid difference <math>V(r_i,f_i)</math>. Biases for different locations may be computed from data sets of different size. During visualization [[NCPLOT]] enables filtering using ancillary variable ''Number of valid Taylor data''.

====Root mean square error (RMSE) according to Taylor====

RMSE is computed according to Taylor (2001, equation 3) from ''Pattern RMS'' and ''Bias''.  Computation is performed for categories '''KC''' and '''KN'''. A valid RMSE is obtained as long as there exists at least one valid difference <math>d_i</math>. During visualization [[NCPLOT]] enables filtering using ancillary variable ''Number of valid differences''.

Definition of RMSE according to Taylor (2001) is mathematically identical to the standard equation for RMSE but computational results may differ slightly due to numerical roundoff.

====Taylor Skill 4====

Skill S4 is computed according to Taylor (2001, equation 4) from correlation and normalized variance. 
Computation is performed for categories '''KC''' and '''KN'''. A valid S4 is obtained as long as there exists at least one valid difference <math>d_i</math>. S4 can be characterised as follows:
* S4 = 1.0 indicates perfect fit;
* S4 = 0.0, if correlation R = -1.0 or variance of the variant tends towards 0.0 or infinity;
* S4 is linear with respect to R (at constant variance);
* For zero variance S4 is not defined;
* Bias between variant and reference has no influence on S4.
S4 ''punishes'' deviations in ''Pattern RMS'', and is more ''tolerant'' with respect to deviations in R. See Taylor (2001, figure 10).
 
====Taylor Skill 5====

Skill S5 according to Taylor (2001, euqation 5) is computed from correlation and normalized variance.
Computation is performed for categories '''KC''' and '''KN'''.

In contrast to S4 deviations with respect to ''Pattern RMS'' and correlation are treated (''punished'') similarly. See Taylor (2001, figure 11).

====Number of valid reference data====

Number of valid reference data <math>r_i</math> may differ between locations:
# The number of valid reference data is given by
#:<math>N_r=\sum_{i\in I}P(r_i)</math>;
Computation is performed for categories '''KC''' und '''KN'''. This quantity is of purely informative character and is not truely required for any of the Taylor diagram data.

====Number of valid variant data====

Number of valid variant data <math>f_i</math> may differ between locations:
# The number of valid variant data is given by
#:<math>N_f=\sum_{i\in I}P(f_i)</math>;
Computation is performed for categories '''KC''' and '''KN'''. This quantity is of purely informative character and is not truely required for any of the Taylor diagram data.

====Number of valid Taylor data====

The number of valid Taylor data <math>\bar{r},\bar{f},\sigma_r,\sigma_f,R,E'</math> and <math>\bar{E}</math> may be different from location to location:
# The number of valid Taylor data is given by
#:<math>N_T=\sum_{i\in I}P(r_i,f_i)</math>;
Computation is performed for categories  '''KC''' and '''KN'''. In programs like [[NCPLOT]] the above mentioned Taylor data can be filtered using this quantity. This offers the opportunity to present results for e. g. locations where all events have occurred, or for locations where only a specific number of events has occurred.

===Median and quantiles===

A prerequisite for the computation of all subsequent quantities is that all valid differences <math>d_i</math> have to be sorted in ascending order: all <math>N_{\rm{ord}}</math> valid differences <math>d_i</math> are sorted in ascending order into <math>s_j</math>, with <math>j \in [1:N_{\rm{ord}}]</math>. Subsequently <math>n:=N_{\rm{ord}}</math> will be used for sake of simplicity.

Computation is carried through if <math>n \ge 32</math> holds.

====Median====

Median is computed for all valid differences <math>d_i</math>:
# if <math>n</math> odd: <math>d_{\rm{Med}} = s_\frac{n+1}{2}</math>;
# if <math>n</math> even: <math>d_{\rm{Med}} = 0.5\left( s_{\frac{n}{2}}+s_{\frac{n}{2}+1}\right)</math>;
# in case that there are less than 32 valid data <math>d_{\rm{Med}} = \rm{invalid}</math> is set.
Computation is performed for categories '''KC''' and '''KN'''. During visualization [[NCPLOT]] enables filtering using ancillary variable ''Number of valid differences''.

====Quantile Q01====

Quantile <math>p=0.01</math> is computed for all valid differences <math>d_i</math>. In other words: we obtain a specific value 
for <math>d_i</math>, which is deceeded by just 1 % of all events but exceeded by 99 % of all events:
# if <math>n \cdot p</math> integer: <math>d_{\rm{Q01}} = 0.5\left( s_{n \cdot p}+s_{n \cdot p+1}\right)</math>;
# if <math>n \cdot p</math> real: <math>d_{\rm{Q01}} = s_{\lceil n \cdot p \rceil}</math>;
# in case that there are less than 32 valid data <math>d_{\rm{Q01}} = \rm{invalid}</math> is set.
Computation is performed for categories '''KC''' and '''KN'''. <math>d_{\rm{Q01}}</math> may be computed for different data set sizes at different locations. During visualization [[NCPLOT]] enables filtering using ancillary variable ''Number of valid differences''.

====Quantile Q05====

Quantile <math>p=0.05</math> is computed for all valid differences <math>d_i</math>. In other words: we obtain a specific value 
for <math>d_i</math>, which is deceeded by 5 % of all events but exceeded by 95 % of all events:
# if <math>n \cdot p</math> integer: <math>d_{\rm{Q05}} = 0.5\left( s_{n \cdot p}+s_{n \cdot p+1}\right)</math>;
# if <math>n \cdot p</math> real: <math>d_{\rm{Q05}} = s_{\lceil n \cdot p \rceil}</math>;
# in case that there are less than 32 valid data <math>d_{\rm{Q05}} = \rm{invalid}</math> is set.
Computation is performed for categories '''KC''' and '''KN'''. <math>d_{\rm{Q05}}</math> may be computed for different data set sizes at different locations. During visualization [[NCPLOT]] enables filtering using ancillary variable ''Number of valid differences''.

====Quantile Q95====

Quantile <math>p=0.95</math> is computed for all valid differences <math>d_i</math>. In other words: we obtain a specific value 
for <math>d_i</math>, which is deceeded by 95 % of all events but exceeded by 5 % of all events:
# if <math>n \cdot p</math> integer: <math>d_{\rm{Q95}} = 0.5\left( s_{n \cdot p}+s_{n \cdot p+1}\right)</math>;
# if <math>n \cdot p</math> real: <math>d_{\rm{Q95}} = s_{\lceil n \cdot p \rceil}</math>;
# in case that there are less than 32 valid data <math>d_{\rm{Q95}} = \rm{invalid}</math> is set.
Computation is performed for categories '''KC''' and '''KN'''. <math>d_{\rm{Q95}}</math> may be computed for different data set sizes at different locations. During visualization [[NCPLOT]] enables filtering using ancillary variable ''Number of valid differences''.

====Quantile Q99====

Quantile <math>p=0.99</math> is computed for all valid differences <math>d_i</math>. In other words: we obtain a specific value 
for <math>d_i</math>, which is deceeded by 99 % of all events but exceeded by just 1 % of all events:
# if <math>n \cdot p</math> integer: <math>d_{\rm{Q99}} = 0.5\left( s_{n \cdot p}+s_{n \cdot p+1}\right)</math>;
# if <math>n \cdot p</math> real: <math>d_{\rm{Q99}} = s_{\lceil n \cdot p \rceil}</math>;
# in case that there are less than 32 valid data <math>d_{\rm{Q99}} = \rm{invalid}</math> is set.
Computation is performed for categories '''KC''' and '''KN'''. <math>d_{\rm{Q99}}</math> may be computed for different data set sizes at different locations. During visualization [[NCPLOT]] enables filtering using ancillary variable ''Number of valid differences''.

===Other skill definitions===

====Murphy Skill 4====

Literature:
# Murphy, Allan H. (1988) "Skill Scores Based on the Mean Square Error and Their Relationship to the Correlation Coefficient". Monthly Weather Review, Dec. 1988, pages 2417 - 2424. 

Computation is performed for categories '''KC''' and '''KN'''. Skill S4 according to Murphy (1988, equation 4) can be characterised as follows:
* 1.0 indicates perfect fit;
* 0.0 indicates that the mean value of the reference data models the (reference) data as good as the variant data, because both "models" show the same mean square error (MSE);
* negative skill indicates, that the mean value of the reference data models the (reference) data better than the variant data do;
* ''bias'' is taken into account.

====Willmott Skill (1981)====

Literature:
# Willmott, Cort J. (1981) "On the validation of models". Physical Geography, pages 184–194.

Computation is performed for categories '''KC''' and '''KN'''. Skill according to Willmott (1981, equation 9) can be characterised as follows:
* 1.0 indicates perfect fit;
* 0.0 is the lower limit, for example resulting from phase shift by 180°;
* ''bias'' is taken into account.

----
back to [[Pre- and Postprocessing]]
----
[[Overview]]