Kerstin Fischer

The Role of Users’ Preconceptions
about the Human-Computer Situation
Kerstin Fischer
Initial Observations
● problems in the structural description of
CT
● literature: different studies, different results
● categories often too coarse-grained
○ e.g. discourse particles
● gender differences
● adaptation, shaping, attitudinal changes
The Approach
● ethnomethodological, i.e. focus on
● speakers’ common sense reasoning
underlying their linguistic behaviour
● displays of their understanding of the
affordances of the situation
Case Study 1:
Human-Computer Interaction
● task: design a corpus for the investigation
of emotion in human-computer interaction
● appointment scheduling domain
The method developed
● Compare speakers’ linguistic behaviour in
a situation in which nothing changes but
the speakers’ attitude
● Human-computer interaction is particularly
suited for eliciting such data
Method
● Elicitation of spoken HCI, repeating sequences
of computer output so that speakers are
confronted with the same computer behaviour
over and over again
● Speakers (just) get the impression that the
system is not functioning well
● Repeated use of system malfunction makes the
users reformulate their utterances and thus
reveal their hypotheses about their artificial
communication partner
Corpus
● Human-computer appointment scheduling
●
●
●
●
64 German & 8 English dialogues (18-33 min each)
fixed schema of computer utterances
27 females and 37 males, 17 – 61 years old
annotated for prosodic, lexical, and conversational
properties
● cooperative ‘test’ phase 0 for comparison
● the remaining 100 turns were divided into 5 phases of
20 turns each
Appointment Scheduling Corpus
Id
Dialogue Act
Utterance
2101
2102
2103
2201
2202
2203
2301
2302
2303
2304
2305
2306
2307
2308
3101
3102
3103
Nonsense
Nonsense
Request proposal
Reject proposal
Misunderstanding
Request proposal
Misunderstanding
Failed understanding
Misunderstanding
Misunderstanding
Reject proposal
Misunderstanding
Misunderstanding
Accept proposal
Nonsense
Nonsense
Request proposal
What for date whatthehell bla
Bla rabartibla blurb.
Please propose a date.
This time is already occupied.
Vacation time is June 15 – July 20
Please propose a date.
Feb. 7th is a Sunday.
I did not understand.
The weekend is already occupied.
It is impossible to meet at 4am.
This time is already occupied.
Friday suits me well.
1rst of March is already taken.
I have noted the appointment.
What for date whatthehell bla
Bla rabartibla blurb.
Please propose a date.
Advantages of the Method
● control for inter- and intrapersonal
variability
● prosodic peculiarities, lexical means, and
conversational strategies can be
correlated with particular phases in the
dialogues and thus with changes in
speaker attitude as an objective measure
● reformulations as clues to speakers’
concepts
Example from the
Appointment Scheduling Corpus
e0582201: FÜNFter erster. (5th of January)
s0582202: die Urlaubszeit ist fünfzehnten Juni bis
zwanzigsten Juli. (vacation time is from 15th of June to
20th of Juli)
e0582202: ja, das hat ja auch nicht viel damit zu tun, da wir
uns im Januar befinden, ne? (well this has not much to do
with the fact that we are in January, right?)
s0582203: bitte machen Sie einen Vorschlag. (please make a
proposal)
e0582203: fünfter Januar. (5th of January)
Example from the
Appointment Scheduling Corpus
e0584101: ACHTZEHN bis zweiundzwanzig Uhr. (6 to 10pm)
s0584102: die Urlaubszeit ist fünfzehnten Juni bis zwanzigsten
Juli. (vacation time is from 15th of June to 20th of Juli)
e0584102: <B> ja, klasse. <P> Dienstag, zwölfter erster,
ACHTzehn bis zweiundzwanzig Uhr. (yes, great. Tuesday,
12th of January, 6 to 10pm)
s0584103: bitte machen Sie einen Vorschlag. (please make a
proposal)
e0584103: Dienstag, zwölfter erster, achtzehn bis
zweiundzwanzig Uhr. (Tuesday, 12th of January, 6 to 10pm)
Example from the
Appointment Scheduling Corpus
e0586206: sechster <P> Januar, <P> zwanzig bis
zweiundzwanzig Uhr. (6th of January, 8 to 10pm)
s0587102: die Urlaubszeit ist fünfzehnten Juni bis zwanzigsten
Juli. (vacation time is from 15th of June to 20th of Juli)
e0587102: dich sollte man feuern. <B> sechster <P> Januar ,
<P> zwanzig bis zweiundzwanzig Uhr. (you should be fired.
6th of January, 8 to 10pm)
s0587103: bitte machen Sie einen Vorschlag. (please make a
proposal)
e0587103: se<L>chster Ja<L>nua<L>r, <P> <;<zwa<L>nzig
bi<L>s zweiundzwa<L>nzig Uhr> ;with very low voice>. (6th
of January, 8 to 10pm)
Conversational Peculiarities
● reformulations,
● meta-linguistic statements,
● new proposals without any relevant relationship
to the previous utterances,
● thematic breaks,
● rejections,
● repetitions,
● evaluations.
Prosodic Peculiarities
e4077101: Monday <P> the eLEVenth <P> of JANuary <P> at
twelve pm.
s4077102: vacation time is from the tenth of June till the
fifteenth of July.
e4077102: no<L>, no <P> <<;slow> JAN<L>uary.> <P>
<Swallow> <B> JANuary the <:<B> elEVenth:> <P> <B> at
TWELve pm.
s4077103: will you please make a suggestion for an
appointment?
e4077103: <Swallow> okay. <Swallow> let's try JANuARy <B>
the <:<B> e<L>LEVenth:> <P> <B> at <P> TWELve pm.
Prosodic Peculiarities
● hyper-articulation
● syllable lengthening (e.g. Mon<L>day)
● pauses (between words and syllables, e.g.
on <P> Thurs<P>day)
● stress variation
● variation of loudness
● variation of intonation contours
● laughter or sighing
Intrapersonal Variation
● prosody
● significant variation
of prosodic
peculiarities
through time
Intrapersonal Variation
● conversational strategies
● cooperative vs. uncooperative strategies
Interpersonal Variation
● Not all speakers are alike – why not?
● Gender is an accepted socio-linguistic
variable
● Age is also likely to influence users’ ways of
interacting with artificial agents
Gender
in Human-Computer Interaction
● Metalanguage
e4022306a: Tuesday,
January fifth, from eight
o'clock until one o'clock.
s4022307: the first week of
March is already
occupied.
e4022307: I mean January
fifth.
Gender
in Human-Computer Interaction
● Reformulations
e4032301: the fifth of
January, Tuesday <P> an
appointment for five
hours.
s4032302: I did not
understand.
e4032302: an appointment
on Tuesday January fifth
<P> for five hours.
Gender
in Human-Computer Interaction
● Repetitions
e4024101: January
fourteenth, <P> from six
until ten at night?
s4024102: vacation time is
from the tenth of June till
the fifteenth of July.
e4024102: January
fourteenth, from six until
ten at night?
Gender
in Human-Computer Interaction
● New proposals
e4022201: then the<L>
twenty-second? <P> at
<P>eight in the morning?
<P> until two in the
afternoon?
s4022202: vacation time is
from the tenth of June till
the fifteenth of July.
e4022202: <B> <P> um
<P> on January fifth, at
eight o'clock?
Gender: Conclusions
● Women tend to be a little more patient
than men
● they reformulate and use metalanguage a
little longer
● they use new proposals and repetitions a
little later in the dialogues
● The differences are for the most part
statistically insignificant
Attitude
in Human-Computer Interaction
● Users seem to have very different concepts of
their artificial communication partner and the
situation, e.g.
e0045206: können Sie denn Ihre Mittagspause auch erst
um vierzehn Uhr machen? [could you take your lunch break
as late as 2pm?]
e0387103: Sprachsysteme sind dumm. [language systems
are stupid]
● Clark (1999): HCI always involves a level of joint
pretense
Attitude
in Human-Computer Interaction
● Two user groups can be distinguished:
● Non-players:
S: hello, how do you do?
U: fine. <laugh>
S: hello, how do you do?
U: I want an appointment on January third.
● Players:
S: hello, how do you do?
U: fine, thanks. <laugh> <P> and how do you
do?
Attitude
in Human-Computer Interaction
● Metalanguage
● Players versus nonplayers
Attitude
in Human-Computer Interaction
● New proposals
without relevance to
the previous
utterance
● Players versus nonplayers
Attitude in Human-Computer
Interaction
● prosody
Shaping the Users` Language
● Directives have short-lived effects or are
even counter-productive, e.g. please
speak more clearly
● In contrast, having the system apologise is
more effective and holds over many turns
● Thus, in the same way as attitude is most
relevant for designing one`s language, it is
also the most suitable starting point for
shaping the users` language
Case Study I: Conclusions
● Speakers` attitude toward their
communication partner determines their
linguistic behaviour more than external
sociolinguistic variables
Case Study 2:
Human-Robot Interaction
● the I1-Ontospace Project:
● Elicit human-robot interaction data in
scenarios that differ with respect to single
parameters
● determine the influence of each parameter on
the users’ linguistic choices
● identify correlations in users’ choices in order
to predict behaviour on the basis of nonintrusive user models
Scenarios: Overview
●
●
●
●
home-tour scenario
distance measurement
route instructions
landmark-based movement descriptions
Parameters Varied
● robot
● aibo, scorpion, box, pioneer
● rolland
● WOz-scenario vs. dialogue system
● typed vs. spoken
● robot output
● male/female voice
● behaviour/language
● type of linguistic output
● prompts
Factors Influencing Choice
concept of
partner
concept of
language
linguistic
behaviour
concept of
world
interaction
Concepts of Language
● Example: relative clauses
S: go to the block which is on your right
R: error
S: go to the block on your right
● Example: complexity reduction
S021 zum rechten Karton [to the right cardboard box]
ROBOT Ich verstehe nicht. [I don't understand.]
S021 rechter karton [right cardboard box]
ROBOT Ich verstehe nicht. [I don't understand.]
S021 rechts [right]
● Example: understandability
Concepts of Language
● Example: up-down metaphor
usr13-30:miss die entfernung zwischen dem objekt links
oben und dem objekt rechts oben [measure the distance
between the object up left and the object up right]
sys:Die Instruktion konnte nicht erkannt werden. Bitte
formulieren Sie neu. [The instruction could not be
processed. Please reformulate.]
usr13-31:miss die entfernung zwischen dem objekt links
hinten und dem objekt rechts hinten [measure the
distance between the object back left and the object
back right]
Concepts of the Partner
● Example: robots prefer absolute reference
systems
S: go left
R: error
S: go East
● Example: robots are formal thinkers
S006 gehe los [start walking]
ROBOT Ich verstehe nicht. [I don't understand.]
S006 12 cm vorwärts [12 cm straight ahead]
● Example: Robots have no self-awareness
usr10-3: erstes Objekt: das am nächsten vor dem Roboter [first
object: that closest in front of robot]
Clarification Questions
topic
example
perception
siehst du die tassen? [do you see the
cups?]
readiness
Bist du für eine weitere Aufgabe bereit?
[Are you ready for another task?]
linguistic
capabilities
cognitive
capabilities
Oder kennst du nur becher? [Or do you
only know mugs?]
weist du wo rechts und links von dir ist? [do
you know where is left and right of you?]
Concepts of the World
● Example: implicational scale
S012 fahre zum linken Objekt. [drive to the left
object.]
ROBOT ich verstehe nicht. [I don't understand.]
S012 zum linken Kasten. [to the left box.]
ROBOT ich verstehe nicht. [I don't understand.]
S012 drehe nach links. [turn to the left.]
ROBOT ich verstehe nicht. [I don't understand.]
S012 rot. [red.]
Factors Determining Linguistic
Variation
● appearance of robot
● major role
○ in foreigner talk
○ in avatar studies
● robot is an uncommon communication
partner
¾ users are expected to rely on cues from the
‚source domain‘
Experiment:
● 3 different robots
● aibo
● pioneer
● scorpion
● identical behaviour
(independent of user utterances)
● 30 English dialogues
Hypotheses
● different degrees of emotional expression
(Aibo > Pioneer > Scorpion)
● interjections, e.g. oh, ah
● contact signals, e.g. hello
● displays of relationship, e.g. my friend
Hypotheses
● different amounts of feedback
(Aibo > Pioneer > Scorpion)
● feedback signals, e.g. okay
● explicit evaluation, e.g. good job
● character traits, e.g. good boy
Hypotheses
● different assumptions of competence
(Pioneer > Aibo > Scorpion)
● structuring cues, e.g. first task
● intensification, e.g. slightly vs. 30 degrees
● abstractness, e.g. object vs. bowl
Hypotheses
● different displays of relationship
(Aibo > Scorpion > Pioneer)
● dialogue opening, e.g. go right vs. hello
(players vs. non-players)
● politeness, e.g. please, thank you
● sentence modality, e.g. imperative vs.
declarative mood
● zoomorphisation, e.g. dog, pee, sting
● reference to robot, e.g. he vs. it
Results
● no significant interaction between robot
and any of the features investigated can
be found
Results
● there is however a significant gender
effect for:
● zoomorphisation
● reference to the robot
● there is a very significant effect between
dialogue opening and:
●
●
●
●
emotional expression
sentence modality
structuring cues
reference to the robot
Conclusions: Experiment
● contrary to expectations, the appearance
of the robot has no significant influence on
the occurrence of particular linguistic
properties
● however, again the concept of the artificial
communication partner itself, as revealed
by the different dialogue openings, seems
to be a prominent factor
Summing up
● in the emotional HCI-corpus, the users`
concept of the communication partner as
tool or as a conversation partner was
significantly related to
● the prosodic properties of their utterances
● the conversational strategies taken
● there was only a statistical tendency
towards a gender effect
Summing up
● the number of clarification questions
asked can be significantly related to
different dialogue openings as well
(Fischer & Bateman 2006)
Summing up
● in the HRI-dialogues,
● gender was found to be a relevant factor
● more important however was the effect of the
conceptualisation of the robot as a tool or as
a conversation partner (as apparent from the
dialogue opening)
How stable
are these preconceptions?
● to what degree do speakers align with the
robot`s output?
● can we manipulate/ shape the
users`concepts directly?
Alignment in HRI
● of situation models, e.g. after off-topic
utterances
● of linguistic structures
● syntactic structures
● the lexical items chosen
● of the reference system
● of the instructional strategy taken
Lexical Alignment
S011 bewege dich zur linken box [move to the left box]
ROBOT: Action
S011 bewege dich zur hinteren box [move to the back box]
ROBOT: Action
ROBOT Ich sehe drei Kartons. Wohin soll ich fahren? [I see
three cardboard boxes. Where should I drive to?]
S011 bewege dich zum rechten karton [move to the right
cardboard box]
Constructional Alignment
ROBOT Ich sehe drei Kartons. Wohin soll ich
fahren? [I see three cardboard boxes. Where
should I drive to?]
S007 zu dem linken. [to the left one.]
ROBOT Ich verstehe nicht. [I don't understand.]
S007 Du sollst zu dem linken Karton fahren [you
should drive to the left cardboard box]
Alignment of Reference System
A003: (2) zu der Tasse, nord-östlich. (2) (to the
cup, north-east)
R: Soll ich mich zu dem Objekt begeben, das
vorne liegt? (do you want me to go to the
object that is in front?)
A003: (4) nord-west. (laughter) mein Fehler.
(laughter) (north-west. my mistake.)
Alignment
of Instructional Strategies
User: gehe nach rechts. (go to the right.)
Robot: Soll ich mich zu dem Objekt
begeben, das vorne liegt? (do you want
me to go to the object that is in front?)
User: nein, geh zu dem Objekt rechts von
Dir. (no, go to the object to your right.)
Shaping the Concepts
● comparison between human-aibo
interaction with and without linguistic
output
● after the robot`s initial utterance hello, how
do you do there are significantly more
higher level instructions than in the no
output condition
● after 3 turns, almost consistent use of
goal-based instructions
Limits to Alignment
S013 10 GRAD RECHTS [10 DEGREES RIGHT]
ROBOT Ich verstehe nicht. [I don't understand.]
S013 GO
ROBOT Ich verstehe nicht. [I don't understand.]
S013 speed++
ROBOT Ich verstehe nicht. [I don't understand.]
S013 VELOCITY, PLEASE
ROBOT Ich verstehe nicht. [I don't understand.]
ROBOT Ich sehe drei Kartons. Wohin soll ich fahren?
[I see three cardboard boxes. Where should I drive
to?]
S013 links [left]
Limits to Alignment
R: yeah, hello, how do you do?
A008: (2) go straight.
R: what can I do for you?
A008: go straight. (7)
R: do you want me to go to the object that is
in front?
A008: (1) no, go straight.
Conclusions: Alignment
● alignment is a natural mechanism in HRI
● alignment depends on the users` concepts of
their communication partner
General Conclusions
● the users` concepts of their communication
partner turned out to be a powerful factor in
the explanation of
● inter- and intrapersonal variation
● occurrence of, and limits to, alignment
Thank you!