Presentation

Mutual Occlusions
on Table-top
p Displays
p y
in Mixed Reality Applications
Daniel Kurz
Bauhaus-Universität Weimar, Germany
Kiyoshi
y
Kiyokawa,
y
Haruo Takemura
Osaka University, Japan
Mutual Occlusions on Table-top Displays in Mixed Reality Applications
Daniel Kurz, Kiyoshi Kiyokawa and Haruo Takemura
29.10.2008
Introduction
Microsoft Surface
(courtesy of Microsoft Corporation)
Mutual Occlusions on Table-top Displays in Mixed Reality Applications
Daniel Kurz, Kiyoshi Kiyokawa and Haruo Takemura
29.10.2008
Introduction (cont
(cont‘d)
d)
L. Cutler, B. Fröhlich, P. Hanrahan.
1997 Symposium
y p
on Interactive 3D Graphics
p
Mutual Occlusions on Table-top Displays in Mixed Reality Applications
Daniel Kurz, Kiyoshi Kiyokawa and Haruo Takemura
29.10.2008
Introduction (cont
(cont‘d)
d)
L. Cutler, B. Fröhlich, P. Hanrahan.
1997 Symposium
y p
on Interactive 3D Graphics
p
Mutual Occlusions on Table-top Displays in Mixed Reality Applications
Daniel Kurz, Kiyoshi Kiyokawa and Haruo Takemura
29.10.2008
Motivation
Mutual Occlusions on Table-top Displays in Mixed Reality Applications
Daniel Kurz, Kiyoshi Kiyokawa and Haruo Takemura
29.10.2008
Related work
• Mutual occlusions for
f
head-mounted displays
– Kiyokawa et al.,
ISMAR 2003
• ELMO
– Enhanced see-through
display using an
– LCD panel for
– Mutual
– Occlusion
Mutual Occlusions on Table-top Displays in Mixed Reality Applications
Daniel Kurz, Kiyoshi Kiyokawa and Haruo Takemura
29.10.2008
Related work (cont
(cont‘d)
d)
• M
Mutual
t l occlusions
l i
for
f
the Virtual Showcase
– Bimber and Fröhlich
Fröhlich,
ISMAR 2002
– Spatial optical seethrough
h
h display
d l
• Occlusion Shadows
– P
Projector-based
j t b d
illumination
– Illuminating
g those parts
p
that are not occluded by
virtual content only
Mutual Occlusions on Table-top Displays in Mixed Reality Applications
Daniel Kurz, Kiyoshi Kiyokawa and Haruo Takemura
29.10.2008
Motivation
Mutual Occlusions on Table-top Displays in Mixed Reality Applications
Daniel Kurz, Kiyoshi Kiyokawa and Haruo Takemura
29.10.2008
Approach
Bimber and Raskar, Spatial
p
Augmented
g
Reality
y
Mutual Occlusions on Table-top Displays in Mixed Reality Applications
Daniel Kurz, Kiyoshi Kiyokawa and Haruo Takemura
29.10.2008
Setup
• Top-projector
– Stereo at 120Hz
• Shutter glasses
– Attached tracking
g
receiver
• Tracking
g system
y
• Rear-projectors
– LC shutters
– Mirror
Mutual Occlusions on Table-top Displays in Mixed Reality Applications
Daniel Kurz, Kiyoshi Kiyokawa and Haruo Takemura
29.10.2008
Rendering
• Rear projection
– The green part
– Virtual objects only
– Totally independent of
the real objects
– Off-axis stereo
– Display result
Mutual Occlusions on Table-top Displays in Mixed Reality Applications
Daniel Kurz, Kiyoshi Kiyokawa and Haruo Takemura
29.10.2008
Rendering (cont
(cont‘d)
d)
• Top projection
– Parts of virtual objects
that
h would
ld occlude
l d reall
ones (red)
– Multiple
M lti l render
d passes
– Projective texture
mapping
– Display result
Mutual Occlusions on Table-top Displays in Mixed Reality Applications
Daniel Kurz, Kiyoshi Kiyokawa and Haruo Takemura
29.10.2008
Rendering (cont
(cont‘d)
d)
• Top projection
– Parts of virtual objects
that
h would
ld occlude
l d reall
ones (red)
– Multiple
M lti l render
d passes
– Projective texture
mapping
– Display result
Mutual Occlusions on Table-top Displays in Mixed Reality Applications
Daniel Kurz, Kiyoshi Kiyokawa and Haruo Takemura
29.10.2008
Rendering (cont
(cont‘d)
d)
• Top projection
– Parts of virtual objects
that
h would
ld occlude
l d reall
ones (red)
– Multiple
M lti l render
d passes
– Projective texture
mapping
– Display result
Mutual Occlusions on Table-top Displays in Mixed Reality Applications
Daniel Kurz, Kiyoshi Kiyokawa and Haruo Takemura
29.10.2008
Applications
• Simple Occlusion Testing
– Y-shaped tool in
user‘s hand
h d
– Occlusions with virtual
object
bj t (e.g.
(
d
danseuse))
Mutual Occlusions on Table-top Displays in Mixed Reality Applications
Daniel Kurz, Kiyoshi Kiyokawa and Haruo Takemura
29.10.2008
Applications (cont
(cont‘d)
d)
GrImage INRIA Rhône
GrImage,
Rhône-Alpes
Alpes
• Bare-hand Interaction
– „Killer application“
– Geometry of the hand
must be captured
• Non-static illumination
and environment
– Imperfect reconstruction
– Errors and artifacts
• Current tracking
quality
q
y is not sufficient!
Mutual Occlusions on Table-top Displays in Mixed Reality Applications
Daniel Kurz, Kiyoshi Kiyokawa and Haruo Takemura
29.10.2008
Applications (cont
(cont‘d)
d)
• A
Augmented
t d
Archaeological Finds
– EE.g. shard
h d off a pott
– What did the pot look
like?
– Modelling in clay is
very static
• Approach
– Scan 3d model
– Augment a choice of
possible contexts
Mutual Occlusions on Table-top Displays in Mixed Reality Applications
Daniel Kurz, Kiyoshi Kiyokawa and Haruo Takemura
29.10.2008
Applications (cont
(cont‘d)
d)
Mutual Occlusions on Table-top Displays in Mixed Reality Applications
Daniel Kurz, Kiyoshi Kiyokawa and Haruo Takemura
29.10.2008
Conclusions
• Approach
A
h
• Issues
I
– Projector-based
– Mutual occlusions on
Table-top displays
– Mixed reality scenarios
• Proof-of-concept
prototype
– Approach works!
– Limitations, problems and
challenges
– Misregistration
• Poor calibration
• System latency
– Surface reflectance
properties
ti
– Shadow casts
• Future work
– How can consistent
occlusions support
pp
3D
user interfaces on tabletop displays in MR?
Mutual Occlusions on Table-top Displays in Mixed Reality Applications
Daniel Kurz, Kiyoshi Kiyokawa and Haruo Takemura
29.10.2008
Thanks for your attention!
Further information:
http://lab.ime.cmc.osaka-u.ac.jp
E-mail:
daniel [email protected] uni weimar de
[email protected]
[email protected]
[email protected]
Mutual Occlusions on Table-top Displays in Mixed Reality Applications
Daniel Kurz, Kiyoshi Kiyokawa and Haruo Takemura
29.10.2008