On the accuracy of covariance matrix

Hessian versus Gauss-Newton methods in atmospheric remote sensing with infrared spectroscopy

Research output: Contribution to journalArticle

4 Citations (Scopus)

Abstract

Most retrieval schemes use a linear approximation of the radiative transfer function within each iteration as well as for error analysis. Like most standard methods, the improved Hessian method relies on a quadratic form of the cost function and linear approximation in the error analysis. Often, there is no robust criterion in determining step size that can be used to calculate covariance matrix by discrete perturbation of the cost function in the Hessian approach. The Hessian method improved recently, however, overcomes this problem by employing adaptive algorithm which uses small step sizes in steep directions and large step sizes in flat directions of the cost function. The results of retrievals of atmospheric trace gases from simulated limb emission spectra show that Gauss-Newton algorithm and the improved Hessian generally give nearly identical volume mixing ratios and error covariance matrices in the original state vector space. Due to interlevel correlations, however, the agreement in the uncertainities in the original state vector coordinate system is partly lost in a space in which the elements of state vector are independent after orthogonal coordinate transformation. The significant discrepancies between the estimated uncertainities by the two methods are found to be related with elements of state vector that are dominantly controlled by flattest eigenvector directions of the inverse covariance matrix. The improved Hessian method determines the uncertainities in those shallowest directions with better accuracy than Gauss-Newton approach. The performance of the Hessian method is also found to be better in resolving structures related to the shallowest eigenvector directions as revealed by better vertical resolutions in the retrieved profiles of the trace species.

Original languageEnglish
Pages (from-to)103-121
Number of pages19
JournalJournal of Quantitative Spectroscopy and Radiative Transfer
Volume96
Issue number1
DOIs
Publication statusPublished - Nov 15 2005

Fingerprint

Hessian matrices
Newton methods
Newton-Raphson method
Covariance matrix
state vectors
remote sensing
Infrared spectroscopy
Remote sensing
infrared spectroscopy
Cost functions
error analysis
costs
Eigenvalues and eigenfunctions
Error analysis
newton
retrieval
eigenvectors
vector spaces
coordinate transformations
Radiative transfer

All Science Journal Classification (ASJC) codes

  • Radiation
  • Atomic and Molecular Physics, and Optics
  • Spectroscopy

Cite this

@article{4149987f78374e8fb07f10b389d141b7,
title = "On the accuracy of covariance matrix: Hessian versus Gauss-Newton methods in atmospheric remote sensing with infrared spectroscopy",
abstract = "Most retrieval schemes use a linear approximation of the radiative transfer function within each iteration as well as for error analysis. Like most standard methods, the improved Hessian method relies on a quadratic form of the cost function and linear approximation in the error analysis. Often, there is no robust criterion in determining step size that can be used to calculate covariance matrix by discrete perturbation of the cost function in the Hessian approach. The Hessian method improved recently, however, overcomes this problem by employing adaptive algorithm which uses small step sizes in steep directions and large step sizes in flat directions of the cost function. The results of retrievals of atmospheric trace gases from simulated limb emission spectra show that Gauss-Newton algorithm and the improved Hessian generally give nearly identical volume mixing ratios and error covariance matrices in the original state vector space. Due to interlevel correlations, however, the agreement in the uncertainities in the original state vector coordinate system is partly lost in a space in which the elements of state vector are independent after orthogonal coordinate transformation. The significant discrepancies between the estimated uncertainities by the two methods are found to be related with elements of state vector that are dominantly controlled by flattest eigenvector directions of the inverse covariance matrix. The improved Hessian method determines the uncertainities in those shallowest directions with better accuracy than Gauss-Newton approach. The performance of the Hessian method is also found to be better in resolving structures related to the shallowest eigenvector directions as revealed by better vertical resolutions in the retrieved profiles of the trace species.",
author = "{Mengistu Tsidu}, Gizaw",
year = "2005",
month = "11",
day = "15",
doi = "10.1016/j.jqsrt.2004.11.014",
language = "English",
volume = "96",
pages = "103--121",
journal = "Journal of Quantitative Spectroscopy and Radiative Transfer",
issn = "0022-4073",
publisher = "Elsevier Limited",
number = "1",

}

TY - JOUR

T1 - On the accuracy of covariance matrix

T2 - Hessian versus Gauss-Newton methods in atmospheric remote sensing with infrared spectroscopy

AU - Mengistu Tsidu, Gizaw

PY - 2005/11/15

Y1 - 2005/11/15

N2 - Most retrieval schemes use a linear approximation of the radiative transfer function within each iteration as well as for error analysis. Like most standard methods, the improved Hessian method relies on a quadratic form of the cost function and linear approximation in the error analysis. Often, there is no robust criterion in determining step size that can be used to calculate covariance matrix by discrete perturbation of the cost function in the Hessian approach. The Hessian method improved recently, however, overcomes this problem by employing adaptive algorithm which uses small step sizes in steep directions and large step sizes in flat directions of the cost function. The results of retrievals of atmospheric trace gases from simulated limb emission spectra show that Gauss-Newton algorithm and the improved Hessian generally give nearly identical volume mixing ratios and error covariance matrices in the original state vector space. Due to interlevel correlations, however, the agreement in the uncertainities in the original state vector coordinate system is partly lost in a space in which the elements of state vector are independent after orthogonal coordinate transformation. The significant discrepancies between the estimated uncertainities by the two methods are found to be related with elements of state vector that are dominantly controlled by flattest eigenvector directions of the inverse covariance matrix. The improved Hessian method determines the uncertainities in those shallowest directions with better accuracy than Gauss-Newton approach. The performance of the Hessian method is also found to be better in resolving structures related to the shallowest eigenvector directions as revealed by better vertical resolutions in the retrieved profiles of the trace species.

AB - Most retrieval schemes use a linear approximation of the radiative transfer function within each iteration as well as for error analysis. Like most standard methods, the improved Hessian method relies on a quadratic form of the cost function and linear approximation in the error analysis. Often, there is no robust criterion in determining step size that can be used to calculate covariance matrix by discrete perturbation of the cost function in the Hessian approach. The Hessian method improved recently, however, overcomes this problem by employing adaptive algorithm which uses small step sizes in steep directions and large step sizes in flat directions of the cost function. The results of retrievals of atmospheric trace gases from simulated limb emission spectra show that Gauss-Newton algorithm and the improved Hessian generally give nearly identical volume mixing ratios and error covariance matrices in the original state vector space. Due to interlevel correlations, however, the agreement in the uncertainities in the original state vector coordinate system is partly lost in a space in which the elements of state vector are independent after orthogonal coordinate transformation. The significant discrepancies between the estimated uncertainities by the two methods are found to be related with elements of state vector that are dominantly controlled by flattest eigenvector directions of the inverse covariance matrix. The improved Hessian method determines the uncertainities in those shallowest directions with better accuracy than Gauss-Newton approach. The performance of the Hessian method is also found to be better in resolving structures related to the shallowest eigenvector directions as revealed by better vertical resolutions in the retrieved profiles of the trace species.

UR - http://www.scopus.com/inward/record.url?scp=18844408106&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=18844408106&partnerID=8YFLogxK

U2 - 10.1016/j.jqsrt.2004.11.014

DO - 10.1016/j.jqsrt.2004.11.014

M3 - Article

VL - 96

SP - 103

EP - 121

JO - Journal of Quantitative Spectroscopy and Radiative Transfer

JF - Journal of Quantitative Spectroscopy and Radiative Transfer

SN - 0022-4073

IS - 1

ER -