Validating Gaussian Process Emulators
Transcrição
Validating Gaussian Process Emulators
Validating Gaussian Process Emulators Leo Bastos University of Sheffield Joint work: Jeremy Oakley and Tony O’Hagan Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 1 / 29 Outline 1 Computer model Definition 2 Emulation Gaussian Process Emulator Toy Example 3 Diagnostics and Validation Numerical diagnostics Graphical diagnostics Examples 4 Conclusions Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 2 / 29 What is a computer model? Computer model is a mathematical representation η(·) of a complex physical system implemented in a computer. We need Computer models when real experiments are very expensive or even implossible to be “done” (e.g. Nuclear experiments) Computer models have an important role in almost all fields of science and technology System Biology models (Rotavirus outbreaks) Cosmological models (Galaxy formation) Climate models* (Global warming) Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 3 / 29 What is a computer model? Computer model is a mathematical representation η(·) of a complex physical system implemented in a computer. We need Computer models when real experiments are very expensive or even implossible to be “done” (e.g. Nuclear experiments) Computer models have an important role in almost all fields of science and technology System Biology models (Rotavirus outbreaks) Cosmological models (Galaxy formation) Climate models* (Global warming) Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 3 / 29 What is a computer model? Computer model is a mathematical representation η(·) of a complex physical system implemented in a computer. We need Computer models when real experiments are very expensive or even implossible to be “done” (e.g. Nuclear experiments) Computer models have an important role in almost all fields of science and technology System Biology models (Rotavirus outbreaks) Cosmological models (Galaxy formation) Climate models* (Global warming) Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 3 / 29 What is a computer model? Computer model is a mathematical representation η(·) of a complex physical system implemented in a computer. We need Computer models when real experiments are very expensive or even implossible to be “done” (e.g. Nuclear experiments) Computer models have an important role in almost all fields of science and technology System Biology models (Rotavirus outbreaks) Cosmological models (Galaxy formation) Climate models* (Global warming) Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 3 / 29 What is a computer model? Computer model is a mathematical representation η(·) of a complex physical system implemented in a computer. We need Computer models when real experiments are very expensive or even implossible to be “done” (e.g. Nuclear experiments) Computer models have an important role in almost all fields of science and technology System Biology models (Rotavirus outbreaks) Cosmological models (Galaxy formation) Climate models* (Global warming) Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 3 / 29 What is a computer model? Computer model is a mathematical representation η(·) of a complex physical system implemented in a computer. We need Computer models when real experiments are very expensive or even implossible to be “done” (e.g. Nuclear experiments) Computer models have an important role in almost all fields of science and technology System Biology models (Rotavirus outbreaks) Cosmological models (Galaxy formation) Climate models* (Global warming) Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 3 / 29 C-GOLDSTEIN Model C-GOLDSTEIN is a simplified* climate model Sea surface temperature ocean salinity and ocean temp at different depths in the ocean area of sea ice thickness of sea ice atmospheric CO2 concentrations ... Large number of outputs (Both time series and field data) Several inputs (e.g. model resolution, initial conditions) Each run takes about an hour on the Linux Boxes at NOC Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 4 / 29 C-GOLDSTEIN Model C-GOLDSTEIN is a simplified* climate model Sea surface temperature ocean salinity and ocean temp at different depths in the ocean area of sea ice thickness of sea ice atmospheric CO2 concentrations ... Large number of outputs (Both time series and field data) Several inputs (e.g. model resolution, initial conditions) Each run takes about an hour on the Linux Boxes at NOC Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 4 / 29 C-GOLDSTEIN Model C-GOLDSTEIN is a simplified* climate model Sea surface temperature ocean salinity and ocean temp at different depths in the ocean area of sea ice thickness of sea ice atmospheric CO2 concentrations ... Large number of outputs (Both time series and field data) Several inputs (e.g. model resolution, initial conditions) Each run takes about an hour on the Linux Boxes at NOC Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 4 / 29 C-GOLDSTEIN Model C-GOLDSTEIN is a simplified* climate model Sea surface temperature ocean salinity and ocean temp at different depths in the ocean area of sea ice thickness of sea ice atmospheric CO2 concentrations ... Large number of outputs (Both time series and field data) Several inputs (e.g. model resolution, initial conditions) Each run takes about an hour on the Linux Boxes at NOC Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 4 / 29 C-GOLDSTEIN Model C-GOLDSTEIN is a simplified* climate model Sea surface temperature ocean salinity and ocean temp at different depths in the ocean area of sea ice thickness of sea ice atmospheric CO2 concentrations ... Large number of outputs (Both time series and field data) Several inputs (e.g. model resolution, initial conditions) Each run takes about an hour on the Linux Boxes at NOC Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 4 / 29 C-GOLDSTEIN Model C-GOLDSTEIN is a simplified* climate model Sea surface temperature ocean salinity and ocean temp at different depths in the ocean area of sea ice thickness of sea ice atmospheric CO2 concentrations ... Large number of outputs (Both time series and field data) Several inputs (e.g. model resolution, initial conditions) Each run takes about an hour on the Linux Boxes at NOC Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 4 / 29 C-GOLDSTEIN Model C-GOLDSTEIN is a simplified* climate model Sea surface temperature ocean salinity and ocean temp at different depths in the ocean area of sea ice thickness of sea ice atmospheric CO2 concentrations ... Large number of outputs (Both time series and field data) Several inputs (e.g. model resolution, initial conditions) Each run takes about an hour on the Linux Boxes at NOC Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 4 / 29 C-GOLDSTEIN Model C-GOLDSTEIN is a simplified* climate model Sea surface temperature ocean salinity and ocean temp at different depths in the ocean area of sea ice thickness of sea ice atmospheric CO2 concentrations ... Large number of outputs (Both time series and field data) Several inputs (e.g. model resolution, initial conditions) Each run takes about an hour on the Linux Boxes at NOC Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 4 / 29 C-GOLDSTEIN Model C-GOLDSTEIN is a simplified* climate model Sea surface temperature ocean salinity and ocean temp at different depths in the ocean area of sea ice thickness of sea ice atmospheric CO2 concentrations ... Large number of outputs (Both time series and field data) Several inputs (e.g. model resolution, initial conditions) Each run takes about an hour on the Linux Boxes at NOC Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 4 / 29 C-GOLDSTEIN Model C-GOLDSTEIN is a simplified* climate model Sea surface temperature ocean salinity and ocean temp at different depths in the ocean area of sea ice thickness of sea ice atmospheric CO2 concentrations ... Large number of outputs (Both time series and field data) Several inputs (e.g. model resolution, initial conditions) Each run takes about an hour on the Linux Boxes at NOC Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 4 / 29 Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 5 / 29 Computer models can be very expensive IBM supercomputers used for climate and weather forecasts One single run of the computer model can take a lot of time Most of analyses need several runs Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 5 / 29 Computer models can be very expensive IBM supercomputers used for climate and weather forecasts One single run of the computer model can take a lot of time Most of analyses need several runs Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 5 / 29 Emulating a computer model η(·) is considered an unknown function Emulator is a predictive function for the computer model outputs Assumptions for the computer model: Deterministic single-output model η(·) Relatively ”Smooth” function η : X ∈ <p → < Statistical Emulator is an stochastic representation of our judgements about the computer model η(·). Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 6 / 29 Emulating a computer model η(·) is considered an unknown function Emulator is a predictive function for the computer model outputs Assumptions for the computer model: Deterministic single-output model η(·) Relatively ”Smooth” function η : X ∈ <p → < Statistical Emulator is an stochastic representation of our judgements about the computer model η(·). Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 6 / 29 Emulating a computer model η(·) is considered an unknown function Emulator is a predictive function for the computer model outputs Assumptions for the computer model: Deterministic single-output model η(·) Relatively ”Smooth” function η : X ∈ <p → < Statistical Emulator is an stochastic representation of our judgements about the computer model η(·). Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 6 / 29 Emulating a computer model η(·) is considered an unknown function Emulator is a predictive function for the computer model outputs Assumptions for the computer model: Deterministic single-output model η(·) Relatively ”Smooth” function η : X ∈ <p → < Statistical Emulator is an stochastic representation of our judgements about the computer model η(·). Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 6 / 29 Emulating a computer model η(·) is considered an unknown function Emulator is a predictive function for the computer model outputs Assumptions for the computer model: Deterministic single-output model η(·) Relatively ”Smooth” function η : X ∈ <p → < Statistical Emulator is an stochastic representation of our judgements about the computer model η(·). Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 6 / 29 Emulating a computer model η(·) is considered an unknown function Emulator is a predictive function for the computer model outputs Assumptions for the computer model: Deterministic single-output model η(·) Relatively ”Smooth” function η : X ∈ <p → < Statistical Emulator is an stochastic representation of our judgements about the computer model η(·). Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 6 / 29 Gaussian Process Emulator Gaussian process emulator: η(·)|β, σ 2 , ψ ∼ GP (m0 (·), V0 (·, ·)) , where m0 (x) = h(x)T β V0 (x, x0 ) = σ 2 C(x, x0 ; ψ) Prior distribution for (β, σ 2 , ψ) Conditioning on some training data yk = η(xk ), Leo Bastos (University of Sheffield) k = 1, . . . , n Diagnostics UFRJ, December 2008 7 / 29 Gaussian Process Emulator Gaussian process emulator: η(·)|β, σ 2 , ψ ∼ GP (m0 (·), V0 (·, ·)) , where m0 (x) = h(x)T β V0 (x, x0 ) = σ 2 C(x, x0 ; ψ) Prior distribution for (β, σ 2 , ψ) Conditioning on some training data yk = η(xk ), Leo Bastos (University of Sheffield) k = 1, . . . , n Diagnostics UFRJ, December 2008 7 / 29 Gaussian Process Emulator Gaussian process emulator: η(·)|β, σ 2 , ψ ∼ GP (m0 (·), V0 (·, ·)) , where m0 (x) = h(x)T β V0 (x, x0 ) = σ 2 C(x, x0 ; ψ) Prior distribution for (β, σ 2 , ψ) Conditioning on some training data yk = η(xk ), Leo Bastos (University of Sheffield) k = 1, . . . , n Diagnostics UFRJ, December 2008 7 / 29 Gaussian Process Emulator Predictive Gaussian Process Emulator η(·)|y, X, ψ ∼ Student-Process (n − q, m1 (·), V1 (·, ·)) , where m1 (x) = V1 (x, x 0 ) = × b h(x)T βb + t(x)T A−1 (y − H β), h σ̂ 2 C(x, x 0 ; ψ) − t(x)T A−1 t(x 0 ) + h(x) − t(x)T A−1 H T (H T A−1 H)−1 h(x 0 ) − t(x 0 )T A−1 H . Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 8 / 29 Toy Example η(·) is a two-dimensional known function GP emulator: η(·)|β, σ 2 , ψ ∼ GP h(·)T β, σ 2 C(x, x0 ; ψ) , h(x) = (1, x)T " X xk − x0 2 0 k C(x, x ) = exp − ψk # k p(β, σ 2 , ψ) ∝ σ −2 Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 9 / 29 Toy Example η(·) is a two-dimensional known function GP emulator: η(·)|β, σ 2 , ψ ∼ GP h(·)T β, σ 2 C(x, x0 ; ψ) , h(x) = (1, x)T " X xk − x0 2 0 k C(x, x ) = exp − ψk # k p(β, σ 2 , ψ) ∝ σ −2 Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 9 / 29 Toy Example η(·) is a two-dimensional known function GP emulator: η(·)|β, σ 2 , ψ ∼ GP h(·)T β, σ 2 C(x, x0 ; ψ) , h(x) = (1, x)T " X xk − x0 2 0 k C(x, x ) = exp − ψk # k p(β, σ 2 , ψ) ∝ σ −2 Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 9 / 29 Toy Example η(·) is a two-dimensional known function GP emulator: η(·)|β, σ 2 , ψ ∼ GP h(·)T β, σ 2 C(x, x0 ; ψ) , h(x) = (1, x)T " X xk − x0 2 0 k C(x, x ) = exp − ψk # k p(β, σ 2 , ψ) ∝ σ −2 Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 9 / 29 Toy Example η(·) is a two-dimensional known function GP emulator: η(·)|β, σ 2 , ψ ∼ GP h(·)T β, σ 2 C(x, x0 ; ψ) , h(x) = (1, x)T " X xk − x0 2 0 k C(x, x ) = exp − ψk # k p(β, σ 2 , ψ) ∝ σ −2 Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 9 / 29 0.8 1.0 Toy example 0.0 Leo Bastos (University of Sheffield) 12 9 0.0 3 13 10 4 11 5 0.2 6 7 8 0.4 2 0.6 5 0.2 0.4 Diagnostics 0.6 0.8 1.0 UFRJ, December 2008 10 / 29 1.0 Toy example ● ● ● 0.8 ● ● 5 ● ● 0.6 ● ● ● 2 ● 0.4 ● 8 ● 7 ● 5 0.2 6 ● ● 0.0 Leo Bastos (University of Sheffield) ● ● 12 9 0.0 3 13 10 4 11 ● ● 0.2 0.4 Diagnostics 0.6 0.8 1.0 UFRJ, December 2008 10 / 29 1.0 Toy example ● 4 ● ● 0.8 ● 2 ● 5 ● ● 4 0.6 ● ● 6 ● 2 ● 0.4 ● 8 8 ● 7 ● 10 5 11 ● ● 0.0 Leo Bastos (University of Sheffield) 12 ● ● 14 12 9 0.0 3 13 10 4 0.2 6 ● ● 0.2 0.4 Diagnostics 0.6 0.8 1.0 UFRJ, December 2008 10 / 29 1.0 Toy example ● 4 ● ● 0.8 ● 2 ● 5 ● ● 4 0.6 ● ● 6 ● 2 ● 0.4 ● 8 8 ● 7 ● 10 5 ● ● 0.0 Leo Bastos (University of Sheffield) 12 ● ● 14 12 9 0.0 3 13 10 4 11 0.2 6 ● ● 0.2 0.4 Diagnostics 0.6 0.8 1.0 UFRJ, December 2008 10 / 29 1.0 Toy example ● ● ● ● ● ● ● ● 0.8 ● 2 ● ● 55 ● ● ● ● ● 0.6 ● ● ●● ● ● ● 23 ● ● 0.4 ● ● ● 8 4 ● ● 7 ● ● 5 ● 6 5 4 11 10 13 13 8 9 Leo Bastos (University of Sheffield) ● ● ● ● ● 12 3 0.0 9 11 0.0 ● ● ● ● 10 12 7 0.2 6 ● ● ● 0.2 0.4 Diagnostics 0.6 0.8 1.0 UFRJ, December 2008 10 / 29 1.0 Toy example ● ● ● ● ● ● ● ● 0.8 ● ● ● 5 5 ● ● ● ● 2 ● 0.6 ● ● ●● ● ● ● 2 ● 0.4 8 7 9 8 13 7 13 ● ● ● ● ● ● 12 9 ● ● ● 12 Leo Bastos (University of Sheffield) ● ● 11 10 11 4 5 6 3 0.2 ● ● ● ● ● ● 0.0 ● ● 3 6 4 ● ● 10 5 ● ● ● 0.0 ● ● 0.2 0.4 Diagnostics 0.6 0.8 1.0 UFRJ, December 2008 10 / 29 MUCM Topics Design for Computer models Emulation (Multiple output emulation, Dynamic emulation) UA/SA - Uncertainty and Sensitivity Analyses Calibration (Bayes Linear and Full Bayesian approaches) Diagnostics and Validation Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 11 / 29 MUCM Topics Design for Computer models Emulation (Multiple output emulation, Dynamic emulation) UA/SA - Uncertainty and Sensitivity Analyses Calibration (Bayes Linear and Full Bayesian approaches) Diagnostics and Validation Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 11 / 29 MUCM Topics Design for Computer models Emulation (Multiple output emulation, Dynamic emulation) UA/SA - Uncertainty and Sensitivity Analyses Calibration (Bayes Linear and Full Bayesian approaches) Diagnostics and Validation Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 11 / 29 MUCM Topics Design for Computer models Emulation (Multiple output emulation, Dynamic emulation) UA/SA - Uncertainty and Sensitivity Analyses Calibration (Bayes Linear and Full Bayesian approaches) Diagnostics and Validation Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 11 / 29 MUCM Topics Design for Computer models Emulation (Multiple output emulation, Dynamic emulation) UA/SA - Uncertainty and Sensitivity Analyses Calibration (Bayes Linear and Full Bayesian approaches) Diagnostics and Validation Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 11 / 29 Diagnostics and Validation Every emulator should be validated Non-valid emulators can induce wrong conclusions There is little research into validating emulators Validation generally means: “the emulator predictions are close enough to the simulator outputs”. We want to take account all the uncertainty associated with the emulator. “Do the choices that I have made, based on my knowledge of this simulator, appear to be consistent with the observations?” Choices for the Gaussian process emulator: Normality Stationarity Correlation parameters Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 12 / 29 Diagnostics and Validation Every emulator should be validated Non-valid emulators can induce wrong conclusions There is little research into validating emulators Validation generally means: “the emulator predictions are close enough to the simulator outputs”. We want to take account all the uncertainty associated with the emulator. “Do the choices that I have made, based on my knowledge of this simulator, appear to be consistent with the observations?” Choices for the Gaussian process emulator: Normality Stationarity Correlation parameters Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 12 / 29 Diagnostics and Validation Every emulator should be validated Non-valid emulators can induce wrong conclusions There is little research into validating emulators Validation generally means: “the emulator predictions are close enough to the simulator outputs”. We want to take account all the uncertainty associated with the emulator. “Do the choices that I have made, based on my knowledge of this simulator, appear to be consistent with the observations?” Choices for the Gaussian process emulator: Normality Stationarity Correlation parameters Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 12 / 29 Diagnostics and Validation Every emulator should be validated Non-valid emulators can induce wrong conclusions There is little research into validating emulators Validation generally means: “the emulator predictions are close enough to the simulator outputs”. We want to take account all the uncertainty associated with the emulator. “Do the choices that I have made, based on my knowledge of this simulator, appear to be consistent with the observations?” Choices for the Gaussian process emulator: Normality Stationarity Correlation parameters Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 12 / 29 Diagnostics and Validation Every emulator should be validated Non-valid emulators can induce wrong conclusions There is little research into validating emulators Validation generally means: “the emulator predictions are close enough to the simulator outputs”. We want to take account all the uncertainty associated with the emulator. “Do the choices that I have made, based on my knowledge of this simulator, appear to be consistent with the observations?” Choices for the Gaussian process emulator: Normality Stationarity Correlation parameters Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 12 / 29 Diagnostics and Validation Every emulator should be validated Non-valid emulators can induce wrong conclusions There is little research into validating emulators Validation generally means: “the emulator predictions are close enough to the simulator outputs”. We want to take account all the uncertainty associated with the emulator. “Do the choices that I have made, based on my knowledge of this simulator, appear to be consistent with the observations?” Choices for the Gaussian process emulator: Normality Stationarity Correlation parameters Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 12 / 29 Diagnostics and Validation Every emulator should be validated Non-valid emulators can induce wrong conclusions There is little research into validating emulators Validation generally means: “the emulator predictions are close enough to the simulator outputs”. We want to take account all the uncertainty associated with the emulator. “Do the choices that I have made, based on my knowledge of this simulator, appear to be consistent with the observations?” Choices for the Gaussian process emulator: Normality Stationarity Correlation parameters Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 12 / 29 Diagnostics and Validation Every emulator should be validated Non-valid emulators can induce wrong conclusions There is little research into validating emulators Validation generally means: “the emulator predictions are close enough to the simulator outputs”. We want to take account all the uncertainty associated with the emulator. “Do the choices that I have made, based on my knowledge of this simulator, appear to be consistent with the observations?” Choices for the Gaussian process emulator: Normality Stationarity Correlation parameters Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 12 / 29 Diagnostics and Validation Every emulator should be validated Non-valid emulators can induce wrong conclusions There is little research into validating emulators Validation generally means: “the emulator predictions are close enough to the simulator outputs”. We want to take account all the uncertainty associated with the emulator. “Do the choices that I have made, based on my knowledge of this simulator, appear to be consistent with the observations?” Choices for the Gaussian process emulator: Normality Stationarity Correlation parameters Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 12 / 29 Diagnostics and Validation Every emulator should be validated Non-valid emulators can induce wrong conclusions There is little research into validating emulators Validation generally means: “the emulator predictions are close enough to the simulator outputs”. We want to take account all the uncertainty associated with the emulator. “Do the choices that I have made, based on my knowledge of this simulator, appear to be consistent with the observations?” Choices for the Gaussian process emulator: Normality Stationarity Correlation parameters Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 12 / 29 Validating a GP Emulator Our diagnostics should be based on a set of new runs of the simulator Why? Because predictions at observed input points are perfect. Validation data (y∗ , X∗ ) : yk∗ = η(xk ∗ ), k = 1, . . . , m Simulator and the predictive emulator outputs are compared Numerical diagnostics Graphical diagnostics Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 13 / 29 Validating a GP Emulator Our diagnostics should be based on a set of new runs of the simulator Why? Because predictions at observed input points are perfect. Validation data (y∗ , X∗ ) : yk∗ = η(xk ∗ ), k = 1, . . . , m Simulator and the predictive emulator outputs are compared Numerical diagnostics Graphical diagnostics Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 13 / 29 Validating a GP Emulator Our diagnostics should be based on a set of new runs of the simulator Why? Because predictions at observed input points are perfect. Validation data (y∗ , X∗ ) : yk∗ = η(xk ∗ ), k = 1, . . . , m Simulator and the predictive emulator outputs are compared Numerical diagnostics Graphical diagnostics Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 13 / 29 Validating a GP Emulator Our diagnostics should be based on a set of new runs of the simulator Why? Because predictions at observed input points are perfect. Validation data (y∗ , X∗ ) : yk∗ = η(xk ∗ ), k = 1, . . . , m Simulator and the predictive emulator outputs are compared Numerical diagnostics Graphical diagnostics Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 13 / 29 Validating a GP Emulator Our diagnostics should be based on a set of new runs of the simulator Why? Because predictions at observed input points are perfect. Validation data (y∗ , X∗ ) : yk∗ = η(xk ∗ ), k = 1, . . . , m Simulator and the predictive emulator outputs are compared Numerical diagnostics Graphical diagnostics Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 13 / 29 Validating a GP Emulator Our diagnostics should be based on a set of new runs of the simulator Why? Because predictions at observed input points are perfect. Validation data (y∗ , X∗ ) : yk∗ = η(xk ∗ ), k = 1, . . . , m Simulator and the predictive emulator outputs are compared Numerical diagnostics Graphical diagnostics Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 13 / 29 Numerical diagnostics Individual predictive errors DiI (y∗ ) = (yi∗ − m1 (x∗i )) p V1 (x∗i , x∗i ) However, the D I (y∗ )s are correlated: D I (η(X∗ )) ∼ Student-tm (n − q, 0, C1 (X∗ )) Mahalanobis distance DMD (y∗ ) = (y∗ − m1 (X∗ ))T V1 (X∗ )−1 (y∗ − m1 (X∗ )) (n − q) DMD (η(X∗ )) ∼ Fm,n−q m(n − q − 2) Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 14 / 29 Numerical diagnostics Individual predictive errors DiI (y∗ ) = (yi∗ − m1 (x∗i )) p V1 (x∗i , x∗i ) However, the D I (y∗ )s are correlated: D I (η(X∗ )) ∼ Student-tm (n − q, 0, C1 (X∗ )) Mahalanobis distance DMD (y∗ ) = (y∗ − m1 (X∗ ))T V1 (X∗ )−1 (y∗ − m1 (X∗ )) (n − q) DMD (η(X∗ )) ∼ Fm,n−q m(n − q − 2) Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 14 / 29 Numerical diagnostics Individual predictive errors DiI (y∗ ) = (yi∗ − m1 (x∗i )) p V1 (x∗i , x∗i ) However, the D I (y∗ )s are correlated: D I (η(X∗ )) ∼ Student-tm (n − q, 0, C1 (X∗ )) Mahalanobis distance DMD (y∗ ) = (y∗ − m1 (X∗ ))T V1 (X∗ )−1 (y∗ − m1 (X∗ )) (n − q) DMD (η(X∗ )) ∼ Fm,n−q m(n − q − 2) Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 14 / 29 Numerical diagnostics Individual predictive errors DiI (y∗ ) = (yi∗ − m1 (x∗i )) p V1 (x∗i , x∗i ) However, the D I (y∗ )s are correlated: D I (η(X∗ )) ∼ Student-tm (n − q, 0, C1 (X∗ )) Mahalanobis distance DMD (y∗ ) = (y∗ − m1 (X∗ ))T V1 (X∗ )−1 (y∗ − m1 (X∗ )) (n − q) DMD (η(X∗ )) ∼ Fm,n−q m(n − q − 2) Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 14 / 29 Numerical diagnostics Pivoted Cholesky errors D PC (y∗ ) = (G−1 )T (y∗ − m1 (X∗ )) where V1 (X∗ ) = GT G, and G = PRT . Properties: D PC (y∗ )T D PC (y∗ ) = DMD (y∗ ) Var (D PC (η(X)) = I Invariant to the data order Pivoting order given by P has an intuitive explanation Each D PC (y∗ ) associated with a validation element Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 15 / 29 Numerical diagnostics Pivoted Cholesky errors D PC (y∗ ) = (G−1 )T (y∗ − m1 (X∗ )) where V1 (X∗ ) = GT G, and G = PRT . Properties: D PC (y∗ )T D PC (y∗ ) = DMD (y∗ ) Var (D PC (η(X)) = I Invariant to the data order Pivoting order given by P has an intuitive explanation Each D PC (y∗ ) associated with a validation element Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 15 / 29 Numerical diagnostics Pivoted Cholesky errors D PC (y∗ ) = (G−1 )T (y∗ − m1 (X∗ )) where V1 (X∗ ) = GT G, and G = PRT . Properties: D PC (y∗ )T D PC (y∗ ) = DMD (y∗ ) Var (D PC (η(X)) = I Invariant to the data order Pivoting order given by P has an intuitive explanation Each D PC (y∗ ) associated with a validation element Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 15 / 29 Numerical diagnostics Pivoted Cholesky errors D PC (y∗ ) = (G−1 )T (y∗ − m1 (X∗ )) where V1 (X∗ ) = GT G, and G = PRT . Properties: D PC (y∗ )T D PC (y∗ ) = DMD (y∗ ) Var (D PC (η(X)) = I Invariant to the data order Pivoting order given by P has an intuitive explanation Each D PC (y∗ ) associated with a validation element Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 15 / 29 Numerical diagnostics Pivoted Cholesky errors D PC (y∗ ) = (G−1 )T (y∗ − m1 (X∗ )) where V1 (X∗ ) = GT G, and G = PRT . Properties: D PC (y∗ )T D PC (y∗ ) = DMD (y∗ ) Var (D PC (η(X)) = I Invariant to the data order Pivoting order given by P has an intuitive explanation Each D PC (y∗ ) associated with a validation element Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 15 / 29 Numerical diagnostics Pivoted Cholesky errors D PC (y∗ ) = (G−1 )T (y∗ − m1 (X∗ )) where V1 (X∗ ) = GT G, and G = PRT . Properties: D PC (y∗ )T D PC (y∗ ) = DMD (y∗ ) Var (D PC (η(X)) = I Invariant to the data order Pivoting order given by P has an intuitive explanation Each D PC (y∗ ) associated with a validation element Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 15 / 29 Numerical diagnostics Pivoted Cholesky errors D PC (y∗ ) = (G−1 )T (y∗ − m1 (X∗ )) where V1 (X∗ ) = GT G, and G = PRT . Properties: D PC (y∗ )T D PC (y∗ ) = DMD (y∗ ) Var (D PC (η(X)) = I Invariant to the data order Pivoting order given by P has an intuitive explanation Each D PC (y∗ ) associated with a validation element Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 15 / 29 Graphical diagnostics Some possible Graphical diagnostics: Individual errors against emulator’s predictions Problems on mean function, non-stationarity Errors againts the pivoting order Poor estimation of the variance, correlation parameters QQ-plots of the uncorrelated standardized errors Non-normality, Local fitting problems or non-stationarity Individual or (pivoted) Cholesky errors against inputs Non-stationarity, pattern not included in the mean function Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 16 / 29 Graphical diagnostics Some possible Graphical diagnostics: Individual errors against emulator’s predictions Problems on mean function, non-stationarity Errors againts the pivoting order Poor estimation of the variance, correlation parameters QQ-plots of the uncorrelated standardized errors Non-normality, Local fitting problems or non-stationarity Individual or (pivoted) Cholesky errors against inputs Non-stationarity, pattern not included in the mean function Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 16 / 29 Graphical diagnostics Some possible Graphical diagnostics: Individual errors against emulator’s predictions Problems on mean function, non-stationarity Errors againts the pivoting order Poor estimation of the variance, correlation parameters QQ-plots of the uncorrelated standardized errors Non-normality, Local fitting problems or non-stationarity Individual or (pivoted) Cholesky errors against inputs Non-stationarity, pattern not included in the mean function Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 16 / 29 Graphical diagnostics Some possible Graphical diagnostics: Individual errors against emulator’s predictions Problems on mean function, non-stationarity Errors againts the pivoting order Poor estimation of the variance, correlation parameters QQ-plots of the uncorrelated standardized errors Non-normality, Local fitting problems or non-stationarity Individual or (pivoted) Cholesky errors against inputs Non-stationarity, pattern not included in the mean function Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 16 / 29 Graphical diagnostics Some possible Graphical diagnostics: Individual errors against emulator’s predictions Problems on mean function, non-stationarity Errors againts the pivoting order Poor estimation of the variance, correlation parameters QQ-plots of the uncorrelated standardized errors Non-normality, Local fitting problems or non-stationarity Individual or (pivoted) Cholesky errors against inputs Non-stationarity, pattern not included in the mean function Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 16 / 29 Example: Nuclear Waste Repository Source: http://web.ead.anl.gov/resrad/ RESRAD is a computer model designed to estimate radiation doses and risks from RESidual RADioactive materials. Output: 10,000 year time series of the release of contamination in drinking water (in millirems) Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 17 / 29 Example: Nuclear Waste Repository Source: http://web.ead.anl.gov/resrad/ RESRAD is a computer model designed to estimate radiation doses and risks from RESidual RADioactive materials. Output: 10,000 year time series of the release of contamination in drinking water (in millirems) Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 17 / 29 5 Output - Log of maximal dose of radiation in drinking water 0 27 inputs Training data: n = 190∗ (900) −5 Emulator 10 15 Example: Nuclear Waste Repository −10 Validation data: m = 69∗ (300) −5 0 5 10 Simulator Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 18 / 29 ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● −2 ● ● ● 20 ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● 10 ● ● ● ● ● ● ● ● ● ● −1 ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● 30 40 50 60 70 ● ● ● −5 0 5 10 E(η(x*i )|y) Data order Leo Bastos (University of Sheffield) ● −2 ● ● ● ● ● ● ● ● ●● ● ●● ● ● ● 1 1 ● ● ● ● ● ● ● 0 Di(y*) ● ● ● ● 0 ● ● −1 ● ● ● 0 ● 2 ● Di(y*) 2 Graphical Diagnostics: Individual errors Diagnostics UFRJ, December 2008 19 / 29 Graphical Diagnostics: Individual errors ● ● 0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● −0.5 0.0 ● 0.5 1.0 ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● −1 ● ● ● ● −0.5 0.0 −1.0 ● 0.5 DU234.S Leo Bastos (University of Sheffield) −0.5 ● ● 0.0 0.5 DSU238.S ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● −1.0 ● ●● ●● ● ● −2 ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0 ● ● ● ● ●● ● ● ● 1 ● ● ● ● ● Di(y*) ● ● ● ● 0.5 ● ● −1 ● ● ● ● ● ● ● ● ● ● ● ● −2 0 ● ● Di(y*) 2 2 1 ● ● 1 0.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● −0.5 ● ● ● ● ● DUU238.S ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● −1.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● DCU238.S ● ● ● ● ● ● ● ● −1.0 ● ● ● ● ● −2 ● ● ● ● ● ● ● ●● ● −1 ● ● ● ● ● ● ● ● ● ● −2 −1 ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ●● ● ● 1 ● ●● ● ● ● ● ● Di(y*) ● ● ● −1 ●● ● ● ● ● ● ● ● ● ● ● ● 0 ● Di(y*) 1 ● ● ● ● ● 0 ● ● ● ● ● 2 ● ● ● Di(y*) ● ● ● −2 2 2 ● ● ● ● ● ● ● ● −1.0 −0.5 0.0 0.5 DUU234.S Diagnostics UFRJ, December 2008 20 / 29 Graphical Diagnostics: Correlated errors ● 2 2 ● ● 1 * DPC i (y ) ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● −1 ● ● ●● ● ●● ● ● ● ● −2 −2 ● ● ● ● ●● ● ●● ●●● ● ●● ● ● ● ● ●● ●● ● ● ● ●● ● 0 1 0 ● ● ● ● ● ●●● ●●●● ●● ●● ●●● ●● ● ●● ●●● ● ●●●●●● ● ●● ●●●● ● ● −1 Sample Quantiles ●● ● ● ● ●●● ● ●●● ● ● ● ● ● −2 −1 0 1 2 0 Theoretical Quantiles 10 20 30 40 50 60 70 Pivoting order DMD (y∗ ) = 58.96 and the 95% CI is (47.13; 104.70) Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 21 / 29 Graphical Diagnostics: Correlated errors ● 2 2 ● ● 1 * DPC i (y ) ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● −1 ● ● ●● ● ●● ● ● ● ● −2 −2 ● ● ● ● ●● ● ●● ●●● ● ●● ● ● ● ● ●● ●● ● ● ● ●● ● 0 1 0 ● ● ● ● ● ●●● ●●●● ●● ●● ●●● ●● ● ●● ●●● ● ●●●●●● ● ●● ●●●● ● ● −1 Sample Quantiles ●● ● ● ● ●●● ● ●●● ● ● ● ● ● −2 −1 0 1 2 0 Theoretical Quantiles 10 20 30 40 50 60 70 Pivoting order DMD (y∗ ) = 58.96 and the 95% CI is (47.13; 104.70) Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 21 / 29 Example: Nilson-Kuusk model Nilson-Kuusk model is a plant canopy reflectance model. For interpretation of remote sensoring data For determination of agronomical and phytometric parameters The Nilson-Kuusk model is a single output model with 5 inputs The training data contains 150 points The validation data contains 100 points Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 22 / 29 Example: Nilson-Kuusk model Nilson-Kuusk model is a plant canopy reflectance model. For interpretation of remote sensoring data For determination of agronomical and phytometric parameters The Nilson-Kuusk model is a single output model with 5 inputs The training data contains 150 points The validation data contains 100 points Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 22 / 29 Example: Nilson-Kuusk model Nilson-Kuusk model is a plant canopy reflectance model. For interpretation of remote sensoring data For determination of agronomical and phytometric parameters The Nilson-Kuusk model is a single output model with 5 inputs The training data contains 150 points The validation data contains 100 points Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 22 / 29 Example: Nilson-Kuusk model Nilson-Kuusk model is a plant canopy reflectance model. For interpretation of remote sensoring data For determination of agronomical and phytometric parameters The Nilson-Kuusk model is a single output model with 5 inputs The training data contains 150 points The validation data contains 100 points Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 22 / 29 Example: Nilson-Kuusk model Nilson-Kuusk model is a plant canopy reflectance model. For interpretation of remote sensoring data For determination of agronomical and phytometric parameters The Nilson-Kuusk model is a single output model with 5 inputs The training data contains 150 points The validation data contains 100 points Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 22 / 29 Example: Nilson-Kuusk model Nilson-Kuusk model is a plant canopy reflectance model. For interpretation of remote sensoring data For determination of agronomical and phytometric parameters The Nilson-Kuusk model is a single output model with 5 inputs The training data contains 150 points The validation data contains 100 points Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 22 / 29 0.6 10 Graphical Diagnostics - Individual Errors ● ● ● ● ● ● ● ● 5 ● ●● ● ● ● ● 0 ● ● ● DI(y*) 0.4 0.3 0.1 ● ● ● ●● ● ●● ●● ● ●● ● ●● ●● ● ● ● ● ● ● ● ●●●● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ●● ● ● ● ●● ●● ●● ● ●● ● ●● ● ● ●●● ●● ●● ● ● ● ●●●● ● ● ●●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● −5 0.0 ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ●● ●● ● ● ● ● ●● ●● 0.2 Emulator 0.5 ● ●● ● ● 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.0 0.2 0.3 0.4 0.5 0.6 E(η(x*i )|y) Simulator Leo Bastos (University of Sheffield) 0.1 Diagnostics UFRJ, December 2008 23 / 29 Graphical Diagnostics - Uncorrelated Errors 1 10 10 15 ● 15 ● ● 45 ● 24 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●●● ● ● ● ●● ● ● ● ● ● ● ●●●●● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ●6 ● 100 ● 17 ● ●3 ● 40 10 ● 12 ● 39 20 40 60 80 5 0 0 ● ● ●● ● ●● ●● ●● ●●●●● ●●● ●●●●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●●●●●● ●●●● ●●●●● ● ●●● ● ● 0 DPC(y*) 70 ● ● −5 ● ● −5 80 ● ● 712 5 DPC(y*) ● 98 ● ● ●● ● ● 100 −2 Pivoting order −1 0 1 2 Theoretical Quantiles DMD (y∗ ) = 750.237 and the 95% CI is (69.0, 142.6) Indicating a conflict between emulator and simulator. Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 24 / 29 0.7 10 Graphical Diagnostics - Input 5 ● ● ● ● 0.6 ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● 0.5 5 ●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ●●● ● ● ● ● ● ● ● ●● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● 0.4 ●● ● ● ● ● ● ●●● ● ● ● ● ● ● ● 650 700 750 0.0 −5 ● 600 800 850 ● ● ● ● ● ● ● ●● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●●●●● ● ● ●● ● ● ● ●● ● ●●●●●● ●● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ●●● ● ●● ● ● ● ● ● ●● ●● ●●● ● ●● ● ●● ●● ● ●●● ●● ●● ●●● ●●●●●●●●● ● ● ●●● ●● ● ● 550 Input 5 Leo Bastos (University of Sheffield) ● ● ●● ● ● ● ● ● ● ● ● ● ●● ●● ● ●● ● ●●● ● ● ● ● ● ●● ● 550 ● ●●● ● ● ● 0.1 ● 0.3 ● 0.2 DI(y*) ● ● ● Model Output ● ● 0 ● 600 650 700 750 800 850 Input 5 Diagnostics UFRJ, December 2008 25 / 29 Actions for the Kuusk emulator The mean function h(·) = (1, x, x52 , x53 , x54 ) Log transformation on outputs “new” dataset for validation Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 26 / 29 Actions for the Kuusk emulator The mean function h(·) = (1, x, x52 , x53 , x54 ) Log transformation on outputs “new” dataset for validation Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 26 / 29 Actions for the Kuusk emulator The mean function h(·) = (1, x, x52 , x53 , x54 ) Log transformation on outputs “new” dataset for validation Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 26 / 29 Individual errors ● 2 ● ● ● ● ● ● ● ● ● ● ● DI(y*) ● ● ● ● ● ● ● ● ● ● ● ● ● ● −2 ● ● ● −3 −4 −3.0 ● ● ●● ● ● ● ● ●● ● ● ● ●● ● ●●● ● ● ● ● ● −1 ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● −3.5 ● ● ● ● 0 −2 ●● −3 Emulator ● ● ●● ● ● ● ● ● ● 1 −1 ● ● ● ● ●●● ● ● −2.5 −2.0 −1.5 −1.0 −0.5 −3.0 −2.5 −2.0 −1.5 −1.0 −0.5 E(η(x*i )|y) Simulator Leo Bastos (University of Sheffield) ● −3.5 Diagnostics UFRJ, December 2008 27 / 29 Uncorrelated Errors 2 ● ● 2 ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● −1 ● ● ●● ●●● ● ●● ●● ●● ●● ●● ● ●● −1 ● ●● DPC(y*) ● ● ● ● ●●● ● ●● ● ●● ● ● ● ● 0 DPC(y*) 1 ● ● ● ● ●● ●●●● 0 1 ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● −2 −2 ● ● 19 −3 ● 0 10 20 30 31 ● −3 ● 40 50 −2 Pivoting order −1 0 1 2 Theoretical Quantiles DMD (y∗ ) = 63.873 and the 95% CI is ( 32.582, 79.508) Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 28 / 29 Conclusions Emulation is important when the computer model is expensive. Validating the emulator is necessary before using it for analyses using tyhe emulator as a surrogate of the computer model. Our diagnostics are useful tools inside the validation process. Thank you! Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 29 / 29 Conclusions Emulation is important when the computer model is expensive. Validating the emulator is necessary before using it for analyses using tyhe emulator as a surrogate of the computer model. Our diagnostics are useful tools inside the validation process. Thank you! Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 29 / 29 Conclusions Emulation is important when the computer model is expensive. Validating the emulator is necessary before using it for analyses using tyhe emulator as a surrogate of the computer model. Our diagnostics are useful tools inside the validation process. Thank you! Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 29 / 29 Conclusions Emulation is important when the computer model is expensive. Validating the emulator is necessary before using it for analyses using tyhe emulator as a surrogate of the computer model. Our diagnostics are useful tools inside the validation process. Thank you! Leo Bastos (University of Sheffield) Diagnostics UFRJ, December 2008 29 / 29
Documentos relacionados
Validating Gaussian Process Emulators
physical system implemented in a computer. We need simulators when real experiments are very expensive or even implossible to be “done” (e.g. Nuclear experiments) Simulators have an important role ...
Leia mais