M3L Multimodale Markup Language im SmartKom

M3L Multimodale Markup Language im
SmartKom-Projekt
Ziel ist die Datenbeschreibung, die
• unabhängig von der Programmiersprache und
• unabhängig vom Betriebssystem funktioniert,
• einfach zu benutzen ist,
• offene Standards verwendet,
• die Verzahnung multipler Modalitäten unterstützt sowohl
• für die Maschine schnell parsebar, wie
• für den Menschen gut lesbar ist.
167
M3L Representation of an Intention Lattice Fragment
<intentionLattice>
<intentionLattice>
[…]
[…]
<hypothesisSequences>
<hypothesisSequences>
<hypothesisSequence>
<hypothesisSequence>
<score>
<score>
<source>
<source> acoustic
acoustic </source>
</source>
<value>
0.96448
<value>
0.96448 </value>
</value>
</score>
</score>
<score>
<score>
<source>
gesture
<source>
gesture </source>
</source>
<value>
0.99791
</value>
<value>
0.99791
</value>
</score>
</score>
<score>
<score>
<source>
understanding
<source>
understanding </source>
</source>
<value>
0.91667
</value>
<value>
0.91667
</value>
</score>
</score>
<hypothesis>
<hypothesis>
<discourseStatus>
<discourseStatus>
<discourseAction>
<discourseAction> set
set </discourseAction>
</discourseAction>
<discourseTopic><goal>
<discourseTopic><goal> epg_info
epg_info </goal></discourseTopic>
</goal></discourseTopic>
[…]
[…]
<event
id="dim868">
<event id="dim868">
<informationSearch
<informationSearch id="dim869">
id="dim869">
<pieceOfInformation>
<pieceOfInformation>
<broadcast
<broadcast id="dim863">
id="dim863">
<avMedium>
<avMedium>
<avMedium
<avMedium id="dim866">
id="dim866">
<avType>
<avType> featureFilm
featureFilm </avType>
</avType>
<title>
<title> Enemy
Enemy of
of the
the State
State </title>
</title>
[…]
[…]
</hypothesisSequence>
</hypothesisSequence>
[…]
[…]
</hypothesisSequences>
</hypothesisSequences>
</intentionLattice>
</intentionLattice>
I would like to know
more about this
Confidence in the Speech
Recognition Result
Confidence in the Gesture
Recognition Result
Confidence in the Speech
Understanding Result
Planning Act
Object Reference
168
The Markup Language Layer Model of
SmartKom
M3L
MultiModal Markup
Language
OIL
Ontology Inference
Layer
eXtended Markup
Language Schema
XMLS
RDFS
Resource Description
Framework Schema
eXtended Markup
Language
XML
RDF
Resource Description
Framework
HTML
Hypertext Markup
Language
169
The Role of the Semantic Web Language M3L
M3L (Multimodal Markup Language) defines the data exchange formats used for
communication between all modules of SmartKom
M3L is partioned into 40 XML schema definitions covering SmartKom‘s discourse
domains
The XML schema event.xsd
captures the semantic
representation of concepts
and processes in SmartKom‘s
multimodal dialogs
170
OIL2XSD: Using XSLT Stylesheets to Convert an
OIL Ontology to an XML Schema
171
Using Ontologies to Extract Information from the Web
Film.de-Movie
MyOnto-Movie
:o-title
:title
:description
:title
:description
:actors
MyOnto-Person
:director
Kinopolis.de-Movie
:name
:critics
:main actor
:name
:birthday
Mapping of Metadata
172
OIL in OIL
class-def oil-or
documentation "A list of two or more class expressions that is to be treated as a disjunction"
subclass-of class-expression
slot-constraint has-operand
value-type class-expression
cardinality 2 class-expression
class-def oil-slot-def
documentation "a slot definition associates a slot name with a slot description"
slot-constraint oil-name
value-type string
slot-constraint oil-documentation
value-type string
min-cardinality 0 string
slot-constraint oil-subslot-of
value-type slot-expression
min-cardinality 0 slot-expression
slot-constraint oil-inverse
value-type slot-expression
min-cardinality 0 slot-expression
slot-constraint oil-properties
value-type (((equal "transitive") or (equal "symmetric")) or (equal "functional"))
173
M3L as a Meaning Representation Language
for the User‘s Input
I would like to send
an email to Dr.Reuse
<domainObject>
<sendTelecommunicationProcess>
<sender>....................</sender>
<receiver>..............</receiver>
<document>..........</document>
<email>...........</email>
</sendTelecommunicationProcess>
</domainObject>
174
Exploiting Ontological Knowledge to
Understand and Answer the User‘s Queries
Which movies with Schwarzenegger are shown on the Pro7 channel?
<domainObject>
<epg>
<broadcastDefault>
<avMedium>
<actors>
<beginTime>
<time>
<function>
<at>
2002-05-10T10:25:46
</at>
</function>
</beginTime>
<actor><name>Schwarzenegger/name></actor>
</actors>
</avMedium>
<channel><name>Pro7</name></channel>
</broadcastDefault>
</epg>
</domainObject>
175
A Fragment of a Presentation Goal, as specified in
M3L
<presentationTask>
<presentationGoal>
<inform> <informFocus> <RealizationType>list </RealizationType> </informFocus> </inform>
<abstractPresentationContent>
<discourseTopic> <goal>epg_browse</goal> </discourseTopic>
<informationSearch id="dim24"><tvProgram id="dim23">
<broadcast><timeDeictic id="dim16">now</timeDeictic>
<between>2003-03-20T19:42:32 2003-03-20T22:00:00</between>
<channel><channel id="dim13"/> </channel>
</broadcast></tvProgram>
</informationSearch>
<result> <event>
<pieceOfInformation>
<tvProgram id="ap_3">
<broadcast> <beginTime>2003-03-20T19:50:00</beginTime>
<endTime>2003-03-20T19:55:00</endTime>
<avMedium> <title>Today’s Stock News</title></avMedium>
<channel>ARD</channel>
</broadcast>……..</event>
</result>
</presentationGoal>
</presentationTask>
176
A Dynamically Generated Multimodal Presentation based on a
Presentation Goal
Here is a
listing of
tonight's TV
broadcasts.
Today's Stock News
Yes, Dear
Evening News
Down to Earth
The King of Queens
Everybody Loves Raymond
Crossing Jordan
Still Standing
Bonanza
Mr. Personality
Passions
Weather Forecast
Today
177
M3L als Schnittstellenspezifikation
<complexType
name="WordLattice"
content="elementOnly">
<annotation>
<documentation>
<description>
Wörtergitter bestehend aus Wortkanten
</description>
</documentation>
</annotation>
<element
name="word"
type="sk:WordEdge"
minOccurs="0"
maxOccurs="unbounded"/>
</complexType>
...
<complexType
name="WordEdge"
content="elementOnly">
...
<element
name="frameFrom"
type="sk:TimeDuration">
...
<attribute
ref="sk:id">
...
<attribute
name="meta"
type="sk:Boolean"
use="default"
value="false">
...
___________________________________
<simpleType
name="Boolean"
base="boolean">
...
<enumeration
value="true"/>
<enumeration
value="false"/>
</simpleType>
...
<attribute
name="id"
type="sk:Id">
...
<simpleType
name="Id"
base="ID">
...
178
M3L als Schnittstellenspezifikation
• Semantik einer Schnittstelle kann mittels Syntax kodiert werden
• Schnittstellen müssen vor ihrer Verwendung spezifiziert
• Zentrales Repository schafft Verbindlichkeit
• Struktur von Schnittstellen ist einheitlich visualisierbar
• Strukturelle Fehler in übertragenen Informationsstrukturem sind
automatisch zu entdecken
• Lokalisierung von Fehlern ist modulextern und uniform möglich
• Einheitliche Anbindung an alle unterstützten Program-miersprachen
(C++, C, Java, Sicstus, Prolog)
179
M3L als Schnittstellenspezifikation
<word
id="UNIQUE....57926473.1-1"
meta="true">
<begin>
1
</begin>
<end>
2
</end>
<text>
&lt;UNKNOWN&gt;
</text>
<score>
8.92294
</score>
<frameFrom>
PT0.01S
</frameFrom>
<frameTo>
PT0.11S
</frameTo>
</word>
<word
id="UNIQUE.....57926473.1-3"
meta="false">
<begin>
3
</begin>
<end>
4
</end>
<text>
Spielfilme
</text>
<score>
28.1613
</score>
<frameFrom>
PT0.45S
</frameFrom>
<frameTo>
PT1.10S
</frameTo>
</word>
180