This study aims at providing improved closed-form refraction estimates for observations near the horizon. In this first part, over 1800 previously published direct measurements of the horizon’s depression (dip) over the sea are reanalyzed using a nonconventional robust procedure for coping with numerous real, large, and asymmetric outliers from abnormal dips. The derived 1-parameter function agrees with those proposed in modern almanacs and for land surveying. It is found that the dips of warmer and colder sea surfaces vs. air are best described with two different functions. The two proposed 3-parameter functions, also using temperature difference between air and sea and wind speed, reduce the estimated error of the 1-parameter function by $\sim \u2153$ and the number of outliers by $\sim \u2154$.

Martin Hieronymi Opt. Express 24(14) A1045-A1068 (2016)

References

You do not have subscription access to this journal. Citation lists with outbound citation links are available to subscribers only. You may subscribe either as an OSA member, or as an authorized user of your institution.

You do not have subscription access to this journal. Cited by links are available to subscribers only. You may subscribe either as an OSA member, or as an authorized user of your institution.

You do not have subscription access to this journal. Figure files are available to subscribers only. You may subscribe either as an OSA member, or as an authorized user of your institution.

You do not have subscription access to this journal. Article tables are available to subscribers only. You may subscribe either as an OSA member, or as an authorized user of your institution.

You do not have subscription access to this journal. Equations are available to subscribers only. You may subscribe either as an OSA member, or as an authorized user of your institution.

You do not have subscription access to this journal. Article level metrics are available to subscribers only. You may subscribe either as an OSA member, or as an authorized user of your institution.

Numerical example on how in the KMPR dataset the height-weights ${w}_{Hj}$ allow each height dataset, $j$, to contribute, independent of its number of data, ${n}_{j}$, with the same weight to the total of the squared differences $\mathrm{\Sigma}{w}_{Hj}\mathrm{\Sigma}{\mathrm{\Delta}}_{ji}^{2}$. $\u3008H\u3009$ corresponds to the mean of the wave height corrected HoE. This example is from a full-sized dataset; the height-weights differ for each of the 12 jack-knifed datasets.

Table 2.

Results from ${\mathsf{R}}_{\mathsf{T}}$ Estimate Functions When Applied on Full-Sized KMPR Dataset and Its Six Randomly Split 12 Jack-Knifed $\sim \mathsf{\xbd}$ Sized Subsets^{a}

Resulting parameters and correlation indicators for ${R}_{T}$ estimate functions Eqs. (5), (9), (10), and (11) together with Eqs. (12) and (13) derived from the analysis of the KMPR dip measurements. Each function’s four estimates result from the two analysis types (FIT and MEAN) with each of them calculated with the two different least-square fits ($\mathrm{\Delta}$-LIMITED and WEIGHTED). The results correspond to mean/fitted values of the full-sized KMPR dataset and its 12 randomly jack-knifed half-sized subsets, counting the result from the full-sized dataset twice. For Eq. (11) the results relate only to the corresponding $\mathrm{\Delta}\theta $ part of the KMPR dataset. The percent values correspond to the estimated errors as derived from the 12 plus one different results. The column headed “Perf.” contains the combined correlation performance with ${\u3008r\u3009}_{w}$ being the weighted mean ratio. For each of the four functions the estimate with the lowest combined correlation performance is selected as the relevant one and shown in bold. The column under “Outliers” shows the mean value of the number of outliers ($|\mathrm{\Delta}|>1.5\text{\hspace{0.17em}\hspace{0.17em}}\mathrm{arcmin}$) from the 1 plus 12 results, counting the number of outliers in the full-sized dataset only half but twice. The dip factor, ${F}_{\mathrm{Da}}$, as defined in Eq. (5b), is only shown for comparison and applies only to Eqs. (5) and (9).

Table 3.

Comparison of Various Published Terrestrial Refraction Estimates with Those of This Study^{a}

Freiesl., Eq. (3) in [11], $\mathrm{\Delta}{T}_{\text{air-sea}}$

9.40

6.312

3.765

2.807

1073

(1.69)

0.235

7.50

Max =

6.312

6.204

2.826

1073

Min =

0.519

0.413

0.406

49

Comparison showing the combined correlation performances $\u3008r\u3009$ resulting for the full-sized KMPR dataset from the different terrestrial refraction estimates. The average ratio $\u3008r\u3009$ is calculated here unweighted. The number of outliers ${N}_{|\mathrm{\Delta}|>{1.5}^{\prime}}$ gives an indication on the function’s capability for coping with abnormal terrestrial refractions.

Numerical example on how in the KMPR dataset the height-weights ${w}_{Hj}$ allow each height dataset, $j$, to contribute, independent of its number of data, ${n}_{j}$, with the same weight to the total of the squared differences $\mathrm{\Sigma}{w}_{Hj}\mathrm{\Sigma}{\mathrm{\Delta}}_{ji}^{2}$. $\u3008H\u3009$ corresponds to the mean of the wave height corrected HoE. This example is from a full-sized dataset; the height-weights differ for each of the 12 jack-knifed datasets.

Table 2.

Results from ${\mathsf{R}}_{\mathsf{T}}$ Estimate Functions When Applied on Full-Sized KMPR Dataset and Its Six Randomly Split 12 Jack-Knifed $\sim \mathsf{\xbd}$ Sized Subsets^{a}

Resulting parameters and correlation indicators for ${R}_{T}$ estimate functions Eqs. (5), (9), (10), and (11) together with Eqs. (12) and (13) derived from the analysis of the KMPR dip measurements. Each function’s four estimates result from the two analysis types (FIT and MEAN) with each of them calculated with the two different least-square fits ($\mathrm{\Delta}$-LIMITED and WEIGHTED). The results correspond to mean/fitted values of the full-sized KMPR dataset and its 12 randomly jack-knifed half-sized subsets, counting the result from the full-sized dataset twice. For Eq. (11) the results relate only to the corresponding $\mathrm{\Delta}\theta $ part of the KMPR dataset. The percent values correspond to the estimated errors as derived from the 12 plus one different results. The column headed “Perf.” contains the combined correlation performance with ${\u3008r\u3009}_{w}$ being the weighted mean ratio. For each of the four functions the estimate with the lowest combined correlation performance is selected as the relevant one and shown in bold. The column under “Outliers” shows the mean value of the number of outliers ($|\mathrm{\Delta}|>1.5\text{\hspace{0.17em}\hspace{0.17em}}\mathrm{arcmin}$) from the 1 plus 12 results, counting the number of outliers in the full-sized dataset only half but twice. The dip factor, ${F}_{\mathrm{Da}}$, as defined in Eq. (5b), is only shown for comparison and applies only to Eqs. (5) and (9).

Table 3.

Comparison of Various Published Terrestrial Refraction Estimates with Those of This Study^{a}

Freiesl., Eq. (3) in [11], $\mathrm{\Delta}{T}_{\text{air-sea}}$

9.40

6.312

3.765

2.807

1073

(1.69)

0.235

7.50

Max =

6.312

6.204

2.826

1073

Min =

0.519

0.413

0.406

49

Comparison showing the combined correlation performances $\u3008r\u3009$ resulting for the full-sized KMPR dataset from the different terrestrial refraction estimates. The average ratio $\u3008r\u3009$ is calculated here unweighted. The number of outliers ${N}_{|\mathrm{\Delta}|>{1.5}^{\prime}}$ gives an indication on the function’s capability for coping with abnormal terrestrial refractions.