Validating Gaussian Process Emulators

Transcrição

Validating Gaussian Process Emulators
Leo Bastos
University of Sheffield
Joint work: Jeremy Oakley and Tony O’Hagan
Leo Bastos (University of Sheffield)
Diagnostics
UFRJ, December 2008
1 / 29
Outline
1
Computer model
Definition
2
Emulation
Gaussian Process Emulator
Toy Example
3
Diagnostics and Validation
Numerical diagnostics
Graphical diagnostics
Examples
4
Conclusions
Diagnostics
UFRJ, December 2008
2 / 29
What is a computer model?
Computer model is a mathematical representation η(·) of a
complex physical system implemented in a computer.
We need Computer models when real experiments are very
expensive or even implossible to be “done” (e.g. Nuclear
experiments)
Computer models have an important role in almost all fields of
science and technology
System Biology models (Rotavirus outbreaks)
Cosmological models (Galaxy formation)
Climate models* (Global warming)
Diagnostics
UFRJ, December 2008
3 / 29
experiments)
Diagnostics
UFRJ, December 2008
3 / 29
experiments)
Diagnostics
UFRJ, December 2008
3 / 29
experiments)
Diagnostics
UFRJ, December 2008
3 / 29
experiments)
Diagnostics
UFRJ, December 2008
3 / 29
experiments)
Diagnostics
UFRJ, December 2008
3 / 29
C-GOLDSTEIN Model
C-GOLDSTEIN is a simplified* climate model
Sea surface temperature
ocean salinity and ocean temp at different depths in the ocean
area of sea ice
thickness of sea ice
atmospheric CO2 concentrations
...
Large number of outputs (Both time series and field data)
Several inputs (e.g. model resolution, initial conditions)
Each run takes about an hour on the Linux Boxes at NOC
Diagnostics
UFRJ, December 2008
4 / 29
C-GOLDSTEIN Model
area of sea ice
...
Diagnostics
UFRJ, December 2008
4 / 29
C-GOLDSTEIN Model
area of sea ice
...
Diagnostics
UFRJ, December 2008
4 / 29
C-GOLDSTEIN Model
area of sea ice
...
Diagnostics
UFRJ, December 2008
4 / 29
C-GOLDSTEIN Model
area of sea ice
...
Diagnostics
UFRJ, December 2008
4 / 29
C-GOLDSTEIN Model
area of sea ice
...
Diagnostics
UFRJ, December 2008
4 / 29
C-GOLDSTEIN Model
area of sea ice
...
Diagnostics
UFRJ, December 2008
4 / 29
C-GOLDSTEIN Model
area of sea ice
...
Diagnostics
UFRJ, December 2008
4 / 29
C-GOLDSTEIN Model
area of sea ice
...
Diagnostics
UFRJ, December 2008
4 / 29
C-GOLDSTEIN Model
area of sea ice
...
Diagnostics
UFRJ, December 2008
4 / 29
Diagnostics
UFRJ, December 2008
5 / 29
Computer models can be very expensive
IBM supercomputers used for climate and weather forecasts
One single run of the computer model can take a lot of time
Most of analyses need several runs
Diagnostics
UFRJ, December 2008
5 / 29
Computer models can be very expensive
IBM supercomputers used for climate and weather forecasts
One single run of the computer model can take a lot of time
Most of analyses need several runs
Diagnostics
UFRJ, December 2008
5 / 29
Emulating a computer model
η(·) is considered an unknown function
Emulator is a predictive function for the computer model outputs
Assumptions for the computer model:
Deterministic single-output model η(·)
Relatively ”Smooth” function
η : X ∈ <p → <
Statistical Emulator is an stochastic representation of our
judgements about the computer model η(·).
Diagnostics
UFRJ, December 2008
6 / 29
η : X ∈ <p → <
Diagnostics
UFRJ, December 2008
6 / 29
η : X ∈ <p → <
Diagnostics
UFRJ, December 2008
6 / 29
η : X ∈ <p → <
Diagnostics
UFRJ, December 2008
6 / 29
η : X ∈ <p → <
Diagnostics
UFRJ, December 2008
6 / 29
η : X ∈ <p → <
Diagnostics
UFRJ, December 2008
6 / 29
Gaussian process emulator:
η(·)|β, σ 2 , ψ ∼ GP (m0 (·), V0 (·, ·)) ,
where
m0 (x) = h(x)T β
V0 (x, x0 ) = σ 2 C(x, x0 ; ψ)
Prior distribution for (β, σ 2 , ψ)
Conditioning on some training data
yk = η(xk ),
k = 1, . . . , n
Diagnostics
UFRJ, December 2008
7 / 29
η(·)|β, σ 2 , ψ ∼ GP (m0 (·), V0 (·, ·)) ,
where
m0 (x) = h(x)T β
V0 (x, x0 ) = σ 2 C(x, x0 ; ψ)
yk = η(xk ),
k = 1, . . . , n
Diagnostics
UFRJ, December 2008
7 / 29
η(·)|β, σ 2 , ψ ∼ GP (m0 (·), V0 (·, ·)) ,
where
m0 (x) = h(x)T β
V0 (x, x0 ) = σ 2 C(x, x0 ; ψ)
yk = η(xk ),
k = 1, . . . , n
Diagnostics
UFRJ, December 2008
7 / 29
Predictive Gaussian Process Emulator
η(·)|y, X, ψ ∼ Student-Process (n − q, m1 (·), V1 (·, ·)) ,
where
m1 (x) =
V1 (x, x 0 ) =
×
b
h(x)T βb + t(x)T A−1 (y − H β),
h
σ̂ 2 C(x, x 0 ; ψ) − t(x)T A−1 t(x 0 ) + h(x) − t(x)T A−1 H
T (H T A−1 H)−1 h(x 0 ) − t(x 0 )T A−1 H
.
Diagnostics
UFRJ, December 2008
8 / 29
Toy Example
η(·) is a two-dimensional known function
GP emulator:
η(·)|β, σ 2 , ψ ∼ GP h(·)T β, σ 2 C(x, x0 ; ψ) ,
h(x) = (1, x)T "
X xk − x0 2
0
k
C(x, x ) = exp −
ψk
#
k
p(β, σ 2 , ψ) ∝ σ −2
Diagnostics
UFRJ, December 2008
9 / 29
Toy Example
GP emulator:
η(·)|β, σ 2 , ψ ∼ GP h(·)T β, σ 2 C(x, x0 ; ψ) ,
h(x) = (1, x)T "
X xk − x0 2
0
k
C(x, x ) = exp −
ψk
#
k
p(β, σ 2 , ψ) ∝ σ −2
Diagnostics
UFRJ, December 2008
9 / 29
Toy Example
GP emulator:
η(·)|β, σ 2 , ψ ∼ GP h(·)T β, σ 2 C(x, x0 ; ψ) ,
h(x) = (1, x)T "
X xk − x0 2
0
k
C(x, x ) = exp −
ψk
#
k
p(β, σ 2 , ψ) ∝ σ −2
Diagnostics
UFRJ, December 2008
9 / 29
Toy Example
GP emulator:
η(·)|β, σ 2 , ψ ∼ GP h(·)T β, σ 2 C(x, x0 ; ψ) ,
h(x) = (1, x)T "
X xk − x0 2
0
k
C(x, x ) = exp −
ψk
#
k
p(β, σ 2 , ψ) ∝ σ −2
Diagnostics
UFRJ, December 2008
9 / 29
Toy Example
GP emulator:
η(·)|β, σ 2 , ψ ∼ GP h(·)T β, σ 2 C(x, x0 ; ψ) ,
h(x) = (1, x)T "
X xk − x0 2
0
k
C(x, x ) = exp −
ψk
#
k
p(β, σ 2 , ψ) ∝ σ −2
Diagnostics
UFRJ, December 2008
9 / 29
0.8
1.0
Toy example
0.0
12
9
0.0
3
13
10
4
11
5
0.2
6
7
8
0.4
2
0.6
5
0.2
0.4
Diagnostics
0.6
0.8
1.0
UFRJ, December 2008
10 / 29
1.0
Toy example
●
●
●
0.8
●
●
5
●
●
0.6
●
●
●
2
●
0.4
●
8
●
7
●
5
0.2
6
●
●
0.0
●
●
12
9
0.0
3
13
10
4
11
●
●
0.2
0.4
Diagnostics
0.6
0.8
1.0
UFRJ, December 2008
10 / 29
1.0
Toy example
●
4
●
●
0.8
●
2
●
5
●
●
4
0.6
●
●
6
●
2
●
0.4
●
8
8
●
7
●
10
5
11
●
●
0.0
12
●
●
14
12
9
0.0
3
13
10
4
0.2
6
●
●
0.2
0.4
Diagnostics
0.6
0.8
1.0
UFRJ, December 2008
10 / 29
1.0
Toy example
●
4
●
●
0.8
●
2
●
5
●
●
4
0.6
●
●
6
●
2
●
0.4
●
8
8
●
7
●
10
5
●
●
0.0
12
●
●
14
12
9
0.0
3
13
10
4
11
0.2
6
●
●
0.2
0.4
Diagnostics
0.6
0.8
1.0
UFRJ, December 2008
10 / 29
1.0
Toy example
●
●
●
●
●
●
●
●
0.8
●
2
●
●
55
●
●
●
●
●
0.6
●
●
●●
●
●
●
23
●
●
0.4
●
●
●
8
4
●
●
7
●
●
5
●
6
5
4
11
10
13
13
8
9
●
●
●
●
●
12
3
0.0
9
11
0.0
●
●
●
●
10
12
7
0.2
6
●
●
●
0.2
0.4
Diagnostics
0.6
0.8
1.0
UFRJ, December 2008
10 / 29
1.0
Toy example
●
●
●
●
●
●
●
●
0.8
●
●
●
5
5
●
●
●
●
2
●
0.6
●
●
●●
●
●
●
2
●
0.4
8
7
9
8
13
7
13
●
●
●
●
●
●
12
9
●
●
●
12
●
●
11
10
11
4
5
6
3
0.2
●
●
●
●
●
●
0.0
●
●
3
6
4
●
●
10
5
●
●
●
0.0
●
●
0.2
0.4
Diagnostics
0.6
0.8
1.0
UFRJ, December 2008
10 / 29
MUCM Topics
Design for Computer models
Emulation (Multiple output emulation, Dynamic emulation)
UA/SA - Uncertainty and Sensitivity Analyses
Calibration (Bayes Linear and Full Bayesian approaches)
Diagnostics
UFRJ, December 2008
11 / 29
MUCM Topics
Diagnostics
UFRJ, December 2008
11 / 29
MUCM Topics
Diagnostics
UFRJ, December 2008
11 / 29
MUCM Topics
Diagnostics
UFRJ, December 2008
11 / 29
MUCM Topics
Diagnostics
UFRJ, December 2008
11 / 29
Every emulator should be validated
Non-valid emulators can induce wrong conclusions
There is little research into validating emulators
Validation generally means: “the emulator predictions are close
enough to the simulator outputs”.
We want to take account all the uncertainty associated with the
emulator.
“Do the choices that I have made, based on my knowledge of this
simulator, appear to be consistent with the observations?”
Choices for the Gaussian process emulator:
Normality
Stationarity
Correlation parameters
Diagnostics
UFRJ, December 2008
12 / 29
emulator.
Normality
Stationarity
Diagnostics
UFRJ, December 2008
12 / 29
emulator.
Normality
Stationarity
Diagnostics
UFRJ, December 2008
12 / 29
emulator.
Normality
Stationarity
Diagnostics
UFRJ, December 2008
12 / 29
emulator.
Normality
Stationarity
Diagnostics
UFRJ, December 2008
12 / 29
emulator.
Normality
Stationarity
Diagnostics
UFRJ, December 2008
12 / 29
emulator.
Normality
Stationarity
Diagnostics
UFRJ, December 2008
12 / 29
emulator.
Normality
Stationarity
Diagnostics
UFRJ, December 2008
12 / 29
emulator.
Normality
Stationarity
Diagnostics
UFRJ, December 2008
12 / 29
emulator.
Normality
Stationarity
Diagnostics
UFRJ, December 2008
12 / 29
Validating a GP Emulator
Our diagnostics should be based on a set of new runs of the
simulator
Why? Because predictions at observed input points are perfect.
Validation data (y∗ , X∗ ) : yk∗ = η(xk ∗ ), k = 1, . . . , m
Simulator and the predictive emulator outputs are compared
Diagnostics
UFRJ, December 2008
13 / 29
simulator
Diagnostics
UFRJ, December 2008
13 / 29
simulator
Diagnostics
UFRJ, December 2008
13 / 29
simulator
Diagnostics
UFRJ, December 2008
13 / 29
simulator
Diagnostics
UFRJ, December 2008
13 / 29
simulator
Diagnostics
UFRJ, December 2008
13 / 29
Individual predictive errors
DiI (y∗ ) =
(yi∗ − m1 (x∗i ))
p
V1 (x∗i , x∗i )
However, the D I (y∗ )s are correlated:
D I (η(X∗ )) ∼ Student-tm (n − q, 0, C1 (X∗ ))
Mahalanobis distance
DMD (y∗ ) = (y∗ − m1 (X∗ ))T V1 (X∗ )−1 (y∗ − m1 (X∗ ))
(n − q)
DMD (η(X∗ )) ∼ Fm,n−q
m(n − q − 2)
Diagnostics
UFRJ, December 2008
14 / 29
DiI (y∗ ) =
(yi∗ − m1 (x∗i ))
p
V1 (x∗i , x∗i )
DMD (y∗ ) = (y∗ − m1 (X∗ ))T V1 (X∗ )−1 (y∗ − m1 (X∗ ))
(n − q)
m(n − q − 2)
Diagnostics
UFRJ, December 2008
14 / 29
DiI (y∗ ) =
(yi∗ − m1 (x∗i ))
p
V1 (x∗i , x∗i )
DMD (y∗ ) = (y∗ − m1 (X∗ ))T V1 (X∗ )−1 (y∗ − m1 (X∗ ))
(n − q)
m(n − q − 2)
Diagnostics
UFRJ, December 2008
14 / 29
DiI (y∗ ) =
(yi∗ − m1 (x∗i ))
p
V1 (x∗i , x∗i )
DMD (y∗ ) = (y∗ − m1 (X∗ ))T V1 (X∗ )−1 (y∗ − m1 (X∗ ))
(n − q)
m(n − q − 2)
Diagnostics
UFRJ, December 2008
14 / 29
Pivoted Cholesky errors
D PC (y∗ ) = (G−1 )T (y∗ − m1 (X∗ ))
where V1 (X∗ ) = GT G, and G = PRT .
Properties:
D PC (y∗ )T D PC (y∗ ) = DMD (y∗ )
Var (D PC (η(X)) = I
Invariant to the data order
Pivoting order given by P has an intuitive explanation
Each D PC (y∗ ) associated with a validation element
Diagnostics
UFRJ, December 2008
15 / 29
D PC (y∗ ) = (G−1 )T (y∗ − m1 (X∗ ))
Properties:
D PC (y∗ )T D PC (y∗ ) = DMD (y∗ )
Diagnostics
UFRJ, December 2008
15 / 29
D PC (y∗ ) = (G−1 )T (y∗ − m1 (X∗ ))
Properties:
D PC (y∗ )T D PC (y∗ ) = DMD (y∗ )
Diagnostics
UFRJ, December 2008
15 / 29
D PC (y∗ ) = (G−1 )T (y∗ − m1 (X∗ ))
Properties:
D PC (y∗ )T D PC (y∗ ) = DMD (y∗ )
Diagnostics
UFRJ, December 2008
15 / 29
D PC (y∗ ) = (G−1 )T (y∗ − m1 (X∗ ))
Properties:
D PC (y∗ )T D PC (y∗ ) = DMD (y∗ )
Diagnostics
UFRJ, December 2008
15 / 29
D PC (y∗ ) = (G−1 )T (y∗ − m1 (X∗ ))
Properties:
D PC (y∗ )T D PC (y∗ ) = DMD (y∗ )
Diagnostics
UFRJ, December 2008
15 / 29
D PC (y∗ ) = (G−1 )T (y∗ − m1 (X∗ ))
Properties:
D PC (y∗ )T D PC (y∗ ) = DMD (y∗ )
Diagnostics
UFRJ, December 2008
15 / 29
Some possible Graphical diagnostics:
Individual errors against emulator’s predictions
Problems on mean function, non-stationarity
Errors againts the pivoting order
Poor estimation of the variance, correlation parameters
QQ-plots of the uncorrelated standardized errors
Non-normality, Local fitting problems or non-stationarity
Individual or (pivoted) Cholesky errors against inputs
Non-stationarity, pattern not included in the mean function
Diagnostics
UFRJ, December 2008
16 / 29
Diagnostics
UFRJ, December 2008
16 / 29
Diagnostics
UFRJ, December 2008
16 / 29
Diagnostics
UFRJ, December 2008
16 / 29
Diagnostics
UFRJ, December 2008
16 / 29
Example: Nuclear Waste Repository
Source: http://web.ead.anl.gov/resrad/
RESRAD is a computer model designed to estimate radiation
doses and risks from RESidual RADioactive materials.
Output: 10,000 year time series of the release of contamination in
drinking water (in millirems)
Diagnostics
UFRJ, December 2008
17 / 29
Source: http://web.ead.anl.gov/resrad/
RESRAD is a computer model designed to estimate radiation
doses and risks from RESidual RADioactive materials.
Output: 10,000 year time series of the release of contamination in
drinking water (in millirems)
Diagnostics
UFRJ, December 2008
17 / 29
5
Output - Log of maximal dose
of radiation in drinking water
0
27 inputs
Training data: n = 190∗ (900)
−5
Emulator
10
15
−10
Validation data: m = 69∗ (300)
−5
0
5
10
Simulator
Diagnostics
UFRJ, December 2008
18 / 29
●
●
●
●
●
●
●
●●
●
● ●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
−2
●
●
●
20
●
●
●
●
●
●
●
●
●
●
● ●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
10
●
●
●
●
●
●
●
●
●
●
−1
●
●
●
●
●
●
●
●
●
● ●
●
●●
●
●
●
●●
●
●
●
●
30
40
50
60
70
●
●
●
−5
0
5
10
E(η(x*i )|y)
Data order
●
−2
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
1
1
●
●
●
●
●
●
●
0
Di(y*)
●
●
●
●
0
●
●
−1
●
●
●
0
●
2
●
Di(y*)
2
Graphical Diagnostics: Individual errors
Diagnostics
UFRJ, December 2008
19 / 29
Graphical Diagnostics: Individual errors
●
●
0
●
●
●
●
●
●
● ● ●
●
●
●
●
●
●
●
●
●
●
●
●
−0.5
0.0
●
0.5
1.0
●
●
●
●
●●
●
●
●●
●
● ●
● ● ●
−1
●
●
●
●
−0.5
0.0
−1.0
●
0.5
DU234.S
−0.5
●
●
0.0
0.5
DSU238.S
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
−1.0
●
●●
●●
●
●
−2
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
● ●
●
●
●
●
●
0
●
●
●
●
●●
●
● ●
1
●
● ●
● ●
Di(y*)
●
●
●
●
0.5
●
●
−1
● ●
●
●
●
●
● ●
●
●
●
●
−2
0
● ●
Di(y*)
2
2
1
●
●
1
0.0
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
−0.5
●
●
●
●
●
DUU238.S
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
−1.0
●
●
●
●
●
●
●
●
●
●
●
●
●
●
DCU238.S
●
●
●
●
●
●
●
●
−1.0
●
●
●
●
●
−2
●
● ●
●
●
●
●
●●
●
−1
●
●
●
●
●
●
●
●
●
●
−2
−1
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
1
●
●●
●
●
●
●
●
Di(y*)
●
●
●
−1
●●
●
●
●
●
●
●
●
●
●
●
●
0
●
Di(y*)
1
●
●
●
●
●
0
●
●
●
●
●
2
●
●
●
Di(y*)
●
●
●
−2
2
2
●
●
●
●
●
●
●
●
−1.0
−0.5
0.0
0.5
DUU234.S
Diagnostics
UFRJ, December 2008
20 / 29
Graphical Diagnostics: Correlated errors
●
2
2
●
●
1
*
DPC
i (y )
●
●
●
●● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
−1
●
●
●●
●
●●
●
●
●
●
−2
−2
●
●
●
●
●●
●
●●
●●●
● ●●
●
●
●
●
●●
●●
●
●
●
●● ●
0
1
0
●
●
●
●
●
●●●
●●●●
●●
●●
●●●
●●
●
●●
●●●
●
●●●●●●
●
●●
●●●●
●
●
−1
Sample Quantiles
●● ● ●
●
●●●
●
●●●
●
●
●
●
●
−2
−1
0
1
2
0
Theoretical Quantiles
10
20
30
40
50
60
70
Pivoting order
DMD (y∗ ) = 58.96 and the 95% CI is (47.13; 104.70)
Diagnostics
UFRJ, December 2008
21 / 29
Graphical Diagnostics: Correlated errors
●
2
2
●
●
1
*
DPC
i (y )
●
●
●
●● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
−1
●
●
●●
●
●●
●
●
●
●
−2
−2
●
●
●
●
●●
●
●●
●●●
● ●●
●
●
●
●
●●
●●
●
●
●
●● ●
0
1
0
●
●
●
●
●
●●●
●●●●
●●
●●
●●●
●●
●
●●
●●●
●
●●●●●●
●
●●
●●●●
●
●
−1
Sample Quantiles
●● ● ●
●
●●●
●
●●●
●
●
●
●
●
−2
−1
0
1
2
0
10
20
30
40
50
60
70
Pivoting order
DMD (y∗ ) = 58.96 and the 95% CI is (47.13; 104.70)
Diagnostics
UFRJ, December 2008
21 / 29
Example: Nilson-Kuusk model
Nilson-Kuusk model is a plant canopy reflectance model.
For interpretation of remote sensoring data
For determination of agronomical and phytometric parameters
The Nilson-Kuusk model is a single output model with 5 inputs
The training data contains 150 points
The validation data contains 100 points
Diagnostics
UFRJ, December 2008
22 / 29
Diagnostics
UFRJ, December 2008
22 / 29
Diagnostics
UFRJ, December 2008
22 / 29
Diagnostics
UFRJ, December 2008
22 / 29
Diagnostics
UFRJ, December 2008
22 / 29
Diagnostics
UFRJ, December 2008
22 / 29
0.6
10
Graphical Diagnostics - Individual Errors
●
● ●
●
●
●
●
●
5
●
●●
●
●
●
●
0
●
●
●
DI(y*)
0.4
0.3
0.1
●
●
● ●●
●
●●
●●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●●
●
●
●
●●
● ●
●
●
●●
●
●
●
●●
●●
●● ●
●●
●
●●
●
● ●●●
●●
●●
●
●
●
●●●● ●
●
●●●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
−5
0.0
●
● ●
● ●
● ●●●
●
● ●
●
●● ●
●●
●●
●
● ●
●
●●
●●
0.2
Emulator
0.5
●
●●
●
●
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.0
0.2
0.3
0.4
0.5
0.6
E(η(x*i )|y)
Simulator
0.1
Diagnostics
UFRJ, December 2008
23 / 29
Graphical Diagnostics - Uncorrelated Errors
1
10
10
15
●
15
●
● 45
● 24
●
● ● ● ●
●
●
●
●
●
●
● ● ●
● ●
● ● ●
● ● ●
●
●
●●
● ●
●
● ●
●
●●● ●
●
●
●●
●
●
●
●
●
● ●●●●●
●
● ●●
●
●
● ● ● ●●●
● ●
●
●
●
●
●
●● ● ● ●
●
●
●●
●
●
●
●6
●
100
● 17 ●
●3
● 40 10
● 12
● 39
20
40
60
80
5
0
0
● ●
●●
●
●●
●●
●●
●●●●●
●●●
●●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●
●●●●
●●●●●
●
●●●
●
●
0
DPC(y*)
70
●
●
−5
●
●
−5
80
● ●
712
5
DPC(y*)
●
98
●
●
●●
●
●
100
−2
Pivoting order
−1
0
1
2
DMD (y∗ ) = 750.237 and the 95% CI is (69.0, 142.6)
Indicating a conflict between emulator and simulator.
Diagnostics
UFRJ, December 2008
24 / 29
0.7
10
Graphical Diagnostics - Input 5
●
●
●
●
0.6
●
●
●
●
● ●●
●
●
●
● ●
● ●
●
● ●
●
●
●
● ●●
●
●●
●
●
●●
● ●
● ●
●●
●
●●
●
●●
●
● ●●
●
●
●
●
● ● ●
●
●
●
●●
●
●
● ●
0.5
5
●●
●
●
●● ●
●
●
●
●
● ● ● ● ●
●●
●●●
●
●
●
●
●
●
●
●● ●
●●● ●
●
●
●
●
●
●
●
● ●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
0.4
●● ●
●
●
●
●
●●●
●
●
●
●
●
●
●
650
700
750
0.0
−5
●
600
800
850
● ●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
● ●
●
●
●
● ●
● ●
● ● ●●
●
● ●●●●● ● ●
●● ● ● ● ●● ●
●●●●●●
●●
●
●
● ●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●● ● ●
●● ● ●
●●●
● ●●
●
●
●
● ●
●● ●●
●●●
●
●●
●
●●
●●
● ●●●
●●
●●
●●●
●●●●●●●●● ●
● ●●● ●● ●
●
550
Input 5
●
●
●● ●
● ●
●
● ●
● ●
●
●● ●●
●
●●
●
●●●
● ●
●
●
●
●●
●
550
●
●●●
●
●
●
0.1
●
0.3
●
0.2
DI(y*)
●
●
●
Model Output
●
●
0
●
600
650
700
750
800
850
Input 5
Diagnostics
UFRJ, December 2008
25 / 29
Actions for the Kuusk emulator
The mean function h(·) = (1, x, x52 , x53 , x54 )
Log transformation on outputs
“new” dataset for validation
Diagnostics
UFRJ, December 2008
26 / 29
Diagnostics
UFRJ, December 2008
26 / 29
Diagnostics
UFRJ, December 2008
26 / 29
Individual errors
●
2
●
●
●
●
●
●
●
●
●
●
●
DI(y*)
●
●
●
●
● ●
●
●
●
●
●
●
●
●
−2
●
●
●
−3
−4
−3.0
●
●
●●
●
● ●
●
●●
●
● ● ●●
●
●●●
●
●
●
●
●
−1
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
−3.5
●
●
●
●
0
−2
●●
−3
Emulator
●
● ●●
●
●
●
●
●
●
1
−1
●
● ●
●
●●●
●
●
−2.5
−2.0
−1.5
−1.0
−0.5
−3.0
−2.5
−2.0
−1.5
−1.0
−0.5
E(η(x*i )|y)
Simulator
●
−3.5
Diagnostics
UFRJ, December 2008
27 / 29
Uncorrelated Errors
2
●
●
2
●
●●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
−1
●
●
●●
●●●
●
●●
●●
●●
●●
●●
●
●●
−1
●
●●
DPC(y*)
●
●
●
●
●●●
●
●●
●
●●
●
● ●
●
0
DPC(y*)
1
●
●
●
●
●●
●●●●
0
1
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
−2
−2
●
●
19
−3
●
0
10
20
30
31
●
−3
●
40
50
−2
Pivoting order
−1
0
1
2
DMD (y∗ ) = 63.873 and the 95% CI is ( 32.582, 79.508)
Diagnostics
UFRJ, December 2008
28 / 29
Conclusions
Emulation is important when the computer model is expensive.
Validating the emulator is necessary before using it for analyses
using tyhe emulator as a surrogate of the computer model.
Our diagnostics are useful tools inside the validation process.
Thank you!
Diagnostics
UFRJ, December 2008
29 / 29
Conclusions
Thank you!
Diagnostics
UFRJ, December 2008
29 / 29
Conclusions
Thank you!
Diagnostics
UFRJ, December 2008
29 / 29
Conclusions
Thank you!
Diagnostics
UFRJ, December 2008
29 / 29

Validating Gaussian Process Emulators

Transcrição

Documentos relacionados

Validating Gaussian Process Emulators

Sys. Info. Read Dtc Clear Dtc Data stream Actuation Special

Author`s comment

ICMT 2_First Announcement - University of Southampton Blogs

What you need to know to revalidate a foreign

Investment Grade Leverages Power Sector - Nuca

Phylogenetic position and re-description of the endangered cichlid

Instruções aos Autores de Contribuições para o SIBGRAPI

Metroid prime emulator dolwin

letters of appreciation

Riley Russell and Sutton Bailey Host Intro For many young adults

The caudal skeleton of extant and fossil