Evaluating the Quality of Web Based Sustainability Reports: A Multi

2012 45th Hawaii International Conference on System Sciences
Evaluating the Quality of Web Based Sustainability Reports:
A Multi-Method Framework
Frank Teuteberg
Research Group in Accounting and Information Systems, University of Osnabrueck, Germany
[email protected]
Michael Freundlieb
Research Group in Accounting and Information Systems, University of Osnabrueck, Germany
[email protected]
development and decision making of companies and
stakeholders alike. Since many companies publish their
SR in form of a website [21], web based sustainability
reporting poses an interesting research topic for the IS
community.
SRs need to fulfill the requirements of internal and
external stakeholders such as employees, supply chain
partners, regulatory bodies, consumers and the general
public [9,28]. Therefore, designing and implementing a
SR that fits the needs of these different stakeholder
groups is a difficult task. Aid is provided by a variety
of standards and guidelines on sustainability reporting.
Section 2 gives an overview of existing standards
and guidelines and outlines the main goals and quality
criteria mentioned within these guidelines. Our review
of the standards and guidelines shows that existing
quality criteria mainly refer to the contents, but not to
the ease of use or the visual appeal of sustainability
reports, although these have proven to be decisive
factors for the acceptance of related IT artifacts such as
websites [4,6,19,24,25]. In view of the stakeholder
focus of SRs [14], we propose a multi-method framework that involves the different internal and external
stakeholder groups directly into the definition of quality criteria and the quality evaluation process. In Section 3 the developed framework is described in more
detail. In order to evaluate our framework, we conducted a preliminary study with 8 participants. In Section 4 we present our key findings regarding the
framework itself, the design and implementation of
SRs as well as implications for future research. Section
5 sums up our results, outlines the limitations of our
research approach and provides a number of starting
points for future research.
Abstract
Sustainability reports bear the potential to improve
a company’s image and to create competitive advantages, for they influence consumers’ buying decisions
and the investment decisions of potential shareholders.
There are multiple standards and guidelines on sustainability reporting, which, however, mainly relate to the
contents of the reports. We argue that due to the stakeholder-focused nature of sustainability reports, additional quality criteria exceeding the mere content need
to be considered to improve the stakeholders’ acceptance. Since these criteria are stakeholder-specific,
they need to be defined and evaluated with direct involvement of the stakeholders. Thus, we propose a
multi-method framework for the quality evaluation of
web based sustainability reports. In order to evaluate
our framework and to gain first insights into stakeholder-focused quality and acceptance criteria, we
applied our framework within a preliminary study.
First experiences and results are presented and implications for theory and practice are derived.
1. Introduction
Sustainability reports (SRs) provide stakeholders
with information on a company’s efforts to balance its
economic, ecological and social goals, the so-called
triple bottom line [10]. In this context sustainability
can be defined as “meeting the needs of the present
without compromising the ability of future generations
to meet their own needs” [2]. For the majority of the
world’s 250 largest companies sustainability reporting
has become a standard procedure [21]. Studies show
that SRs bear the potential to improve a company’s
image and to influence consumers’ buying decisions as
well as the investment decisions of potential shareholders [1,23]. Hence, there are potential competitive
advantages inherent in sustainability reporting which
make SRs an integral part of environmental online
communication and a key factor in the sustainable
978-0-7695-4525-7/12 $26.00 © 2012 IEEE
DOI 10.1109/HICSS.2012.255
2. Existing standards and guidelines for
sustainability reports
In order to identify scientific literature dealing with
quality criteria for and/or the evaluation of sustainability reports, we conducted a systematic literature review
1177
[29]. We searched the Top 20 journals from the AIS
MIS journal ranking as well as the high quality conferences AMCIS, ECIS, HICSS ICIS and Wirtschaftsinformatik for the search terms “quality” or “evaluation”
in combination with the terms “sustainability reporting” or “corporate social responsibility reporting”. Our
search did not lead to relevant results, which is in line
with other researches who have concluded that the
sustainable development and environmental sustainability have not yet been adequately explored by the IS
community [16,28].
Hence, we focused on more practically oriented
standards and guidelines for SRs, which are often referred to when rankings or scorings of SRs are performed [5]. Table 1 provides a short overview of the
sustainability reporting standards and guidelines we
considered most relevant to our work.
gives an overview of the central reporting goals stated
in the above listed guidelines and norms. The goals
were summarized from the standards and guidelines by
the authors and categorized into internal and external
goals. Internal goals represent goals within the reporting company towards its internal stakeholders whereas
external goals represent goals towards external stakeholders.
Table 2. Internal and external goals of sustainability reporting
Organization
GRI [11]
Table 1. Sustainability reporting standards
and guidelines
Organization
Global Reporting Initiative
(GRI) [11]
World Business
Council for
Sustainable
Development
(WBCSD) [26]
EcoManagement
and Audit
Scheme
(EMAS) [8]
International
Organization
for Standardization (ISO)
[12]
Background
The GRI is a guideline on the content and
design of sustainability reports which is based on
the “consensus seeking consultation” of different
stakeholder groups such as business, labor, nongovernmental organizations, investors and accountancy.
13 quality criteria for sustainability reports
are developed and according test procedures are
proposed. Also, a reference outline including
concrete key performance indicators for sustainability reports is suggested.
The approach of the WBCSD is similar to
that of the GRI, but it is of a more general and
recommendatory character. The recommendations primarily concern the information needs of
the stakeholders as well as the contents of reports.
In order to motivate non-reporters to publish a
sustainability report, especially successful companies are referred to and citations from their top
management are given.
EMAS is a regulation regarding the environmental management and environmental audits
of organizations which goes beyond the requirements of the international ISO 14001 norm. It
applies to European companies only. Annex IV of
EMAS includes regulations on environmental
reporting, especially regarding environmental
statements and key performance indicators, as
well as availability and accountability.
The ISO standard 14063 suggests 5 quality
principles for environmental management and
environmental communication and provides a
survey of different channels for environmental
communication as, for example, sustainability
reports. A reference procedure for environmental
communication is described, but there are no
concrete suggestions regarding content or key
performance indicators.
WBCSD
[26]
EMAS
[8]
ISO [12]
We believe that in order to develop a framework
for the quality evaluation of SRs, the goals pursued by
these reports need to be sufficiently clear. Table 2
1178
Internal Goals
External Goals
- provide a balanced and
reasonable representation of
the sustainability performance
of a reporting organization including both positive and
negative contributions
- demonstrate how the organization influences and is influenced by expectations about
sustainable development
- compare performance between different organizations
over time
- showing a balanced and reasonable presentation of an
organization's economic, environmental and social
performance ('true and fair view', 'presented fairly')
- promote continuous im- the active inprovements in the environvolvement of emmental performance of organiployees in organizazations by the establishment
tions and appropriand implementation of enviate training
ronmental management systems by organizations
- the systematic, objective and
periodic evaluation of the
performance of such systems
- the provision of information
on environmental performance
- an open dialogue with the
public and other interested
parties
- promoting an organization's
- providing inenvironmental credentials,
puts/suggestions for
achievements and performance
improving the
- increasing business support
environmental
and shareholder confidence
performance of an
- enhancing interested parties'
organization's
perceptions of the organization
activities, products
- addressing interested parties'
and services, and
concerns and complaints about
progress toward
operational and emergency
sustainability
environmental hazards
- raising the impor- improving understanding of
tance and level of
interested parties' needs and
environmental
awareness to support concerns to foster trust and
dialogue
an environmentally
- assisting interested parties in
responsible culture
understanding an organizaand values within
tion's environmental committhe organization
ments, policies and performance
- benchmarking and
assessing sustainability performance
with respect to laws,
norms, codes,
performance standards, and voluntary
initiatives
- comparing performance within an
organization over
time
In order to constitute an initial set of quality criteria
for sustainability reports, Table 3 gives an overview of
the quality criteria included in the standards and guidelines. Within the table, references to stakeholders are
underlined in order to highlight the stakeholder focus
inherent in the quality criteria.
credibility: Conduct environmental communication in an honest
and fair manner, and provide information that is truthful, accurate,
substantive and not misleading to interested parties. Develop information and data using recognized and reproducible methods and
indicators.
responsiveness: Ensure that environmental communication is open
to the needs of interested parties. Respond to the queries and concerns of interested parties in a fully and timely manner. Make
interested parties aware of how their queries and concerns have
been addressed.
clarity: Ensure that environmental communication approaches and
language are understandable to interested parties to minimize
ambiguity.
EMAS [8]
relevance: Indicators should be relevant to an organization’s significant direct environmental aspects.
understandability: Indicators shall be understandable and unambiguous.
comparability:
- indicators shall allow for a year on year comparison to assess the
development of the environmental performance of the organization
- indicators shall allow for comparison with sector, national or
regional benchmarks as appropriate
- indicators shall allow for comparison with regulatory requirements
as appropriate
accuracy: indicators shall give an accurate appraisal of the organization’s environmental performance
clarity: Environmental information shall be presented in a clear and
coherent manner.
availability: Anybody interested in the organization’s environmental performance can easily and freely be given access to the information.
WBCSD [26]
relevance: Information is relevant when it helps to evaluate a
company's activities and to confirm or correct past evaluations.
materiality: A company should assess the materiality, i. e. the
importance of the information it discloses. Information is to be
considered material if its omission or misstatement could influence
users when making decisions about their involvement/relations with
the company.
reliability: Information is reliable when it is free from material
error and bias, and faithfully reflects activities and processes.
comparability: Users should be able to compare the sustainable
development reports of a company over time in order to identify
trends in its sustainable development performance and position.
cost effectiveness: The benefits derived from producing a report
should justify the cost.
Table 3. Quality criteria for sustainability
reports
GRI [11]
transparency: complete disclosure of information on the topics and
indicators required to reflect impacts and enable stakeholders to
make decisions, and the processes, procedures and assumptions
used to prepare those disclosures
balanced and reasonable presentation of the organization's
performance considering the organization's purpose and experience
and the reasonable expectations and interests of the organization's
stakeholders
materiality: The information in a report should cover topics and
indicators that reflect the organization's significant economic,
environmental, and social impacts, or that would substantively
influence the assessments and decisions of stakeholders.
stakeholder inclusiveness: The reporting organization should
identify its stakeholders and explain in the report how it has responded to their reasonable expectations and interests.
sustainability context: The report should present the organization's
performance in the wider context of sustainability.
completeness: Coverage of the material topics and indicators and
definition of the report boundary should be sufficient to reflect
significant economic, environmental and social impacts and enable
stakeholders to assess the reporting organization's performance in
the reporting period.
comparability: Issues and information should be selected, compiled, and reported consistently. Reported information should be
presented in a manner that enables stakeholders to analyze changes
in the organization's performance over time, and could support
analysis relative to other organizations.
balance: The report should reflect positive and negative aspects of
the organization's performance to enable a reasoned assessment of
overall performance.
accuracy: The reported information should be sufficiently accurate
and detailed for stakeholders to assess the reporting organization's
performance.
timeliness: Reporting occurs on a regular schedule and information
is available in time for stakeholders to make informed decisions.
clarity: Information should be made available in a manner that is
understandable and accessible to stakeholders using the report.
reliability: Information and processes used in the preparation of a
report should be gathered, recorded, compiled, analyzed, and disclosed in a way that could be subject to examination and that establishes the quality and materiality of the information.
report boundary: The sustainability report boundary should
include the entities over which the reporting organization exercises
control or significant influence both in and through its relationships
with various entities upstream (e.g. supply chain) and downstream
(e.g. distribution and customers).
ISO [12]
transparency: Make the processes, procedures, methods, data
sources and assumptions used in environmental communication
available to all interested parties, taking account of the confidentiality of information as required. Inform interested parties of their role
in environmental communication.
appropriateness: Make information provided in environmental
communication relevant to interested parties, using formats, language and media that meet their interests and needs, enabling them
to participate fully.
Despite the different background of the standards
and guidelines, the contained quality criteria are quite
similar. However, the criteria mostly relate to the contents of sustainability reports – many quality criteria
and their definitions are analogous to common data
quality dimensions [27]. The multitude of references to
stakeholders within the quality criteria stresses the
stakeholder focus of sustainability reporting. In view of
this focus, we found that four important aspects were
missing in existing guidelines and standards which we
intend to address with our framework:
• Companies are left in doubt about the specific
needs of the different stakeholder groups since the
existing standards and guidelines do not differentiate between different stakeholder groups.
1179
• Although ease of use and usefulness are common
factors in many established acceptance models (e.g.
[6,24,25]), existing standards and guidelines do not
give concrete advice on how to improve the usability.
• Research on websites indicates that the visual appeal has a significant impact the level of acceptance
[4]. As many companies publish their sustainability
reports in the form of a website [21], visual appeal
is also likely to be a decisive acceptance factor for
web based SRs.
• The existing standards and guidelines do not disclose detailed information on how the included
quality criteria were determined. Since many references to stakeholders are included in the definitions
of the quality criteria, further assistance on how to
involve the stakeholders into the evaluation of these
quality criteria needs to be provided.
include availability, reliability, adaptability and response time of the system as well as content that is
personalized, complete, relevant, easy to understand
and secure [6]. Other researchers also confirm the
importance of usability and add site design and media
richness to the list of important influence factors on the
success of websites [19].
In order to fulfill the collected quality criteria, a company needs to know its stakeholders’ needs, expectations and capabilities. Thus, concrete quality criteria
can only be determined and evaluated by directly involving the stakeholders in the evaluation process. Our
framework, which is presented in the following section, is designed to enable such an interactive process.
Nevertheless, existing standards and guidelines on
sustainability reporting or the usability of websites can
provide a useful set of basic quality criteria, which can
be complemented with the help of the proposed
framework.
Due to the mentioned shortcomings in existing
standards and guidelines for sustainability reporting,
we have extended our search to standards and guidelines in the field of software usability. Table 4 provides
a survey of the quality criteria for usability stated in the
ISO standard 9241-110, which deals with the ergonomics of human-system interaction [7]. Again, we have
underlined references to stakeholders within the definitions of the quality criteria.
3. Framework for the quality evaluation of
web based sustainability reports
Multi-method approaches have been successfully
used by other researches in order to evaluate websites
and other IT artifacts [3,4,15,17]. Following these
researchers, we have designed a framework including a
multi-method approach for the quality evaluation of
sustainability reports in order to overcome the shortcomings of existing standards and guidelines we
pointed out in Section 2. With respect to the identified
stakeholder focus, our framework directly involves the
stakeholder groups into the evaluation process.
Table 4. Quality criteria for user dialogues
ISO 9241-110 [7]
Suitability for the task: “An interactive system is suitable for the
task when it supports the user in the completion of the task, i.e.
when the functionality and the dialogue are based on the task characteristics (rather than the technology chosen to perform the task).”
Self-descriptiveness: “A dialogue is self-descriptive to the extent
that, at any time, it is obvious to the users which dialogue they are
in, where they are within the dialogue, which actions can be taken
and how they can be performed.”
Conformity with user expectations: “A dialogue conforms with
user expectations if it corresponds to predictable contextual needs
of the user and to commonly accepted conventions.”
Suitability for learning: “A dialogue is suitable for learning when
it supports and guides the user in learning to use the system.”
Controllability: “A dialogue is controllable when the user is able
to initiate and control the direction and pace of the interaction until
the point at which the goal has been met.”
Error tolerance: “A dialogue is error-tolerant if, despite evident
errors in input, the intended result may be achieved with either no,
or minimal, corrective action by the user. Error tolerance is
achieved by means of error control (damage control), error correction, or error management, to cope with the errors that occur. ”
Suitability for individualization: “A dialogue is capable of individualization when users can modify interaction and presentation of
information to suit their individual capabilities and needs.”
Literature in the field of website quality also confirms
that quality criteria exceeding the mere content need to
be taken into account. Besides usability, these criteria
Figure 1. Framework for the quality evaluation
1180
The starting point of the framework is the iterative
development of a catalogue of quality criteria. In order
to assemble this catalogue, different research methods
such as literature reviews, empirical surveys or expert
interviews can be applied. For the preliminary study of
our framework, we used the quality criteria outlined in
Section 2.
As a next step, the technological basis for the implementation of the reports as well as the evaluation is
constituted. For the preliminary study, we implemented
a set of alternative reports using a data warehouse
implemented in an Oracle database and choosing Cognos as a reporting engine.
The constitution of the technological basis is followed by the design and implementation of alternative
reports. During this phase, different color schemes,
menu structures and layouts should be applied in order
to evaluate different alternatives in the following multimethod approach [4].
During the multi-method approach, different research and analyses methods are applied in order to
evaluate the reports [3,4,15,17]. Representatives of the
different stakeholder groups such as customers, investors, employees and suppliers should be directly involved in the evaluation process [18]. Research on the
acceptance of IT artifacts often relies on questionnaires
in order to measure the acceptance factors [24,25]. We
argue that questionnaires are a good method to capture
the users’ (self-)perception, but in order to achieve a
complete view of the acceptance factors, more objective methods also need to be applied [24]. Thus, we
suggest adding the following research methods to the
multi method approach:
• observation from multiple perspectives,
• eye-tracking,
• a software-based evaluation.
It should be noted that the framework can be easily
modified by adding or disregarding one or more of the
proposed research methods. One of the strengths of the
proposed framework is that it may also be applied in
order to evaluate other IT artifacts such as websites or
software user interfaces.
In the preliminary study of our framework, we used
an eye-tracking device to be able to monitor the users’
attention behavior. The participants were given tasks to
retrieve information on a company’s environmental
performance from the reports and, in addition to the
eye-tracking, filmed and observed from multiple perspectives during the usage of the reports. This allowed
identifying perceived shortcomings of the report alternatives by closely observing the users’ behavior. The
users could later on be confronted with the recordings
in order to further investigate the causes for the perceived shortcomings. In order to back up the eyetracking results, we also performed a software-
supported attention analysis using the software EyeQuant [30], which emulates the attention behavior of
users by means of a neural network. The software was
originally designed and trained for the evaluation of
websites. However, we tested whether the software
could also aid in evaluating sustainability reports. In
addition to these rather innovative research methods,
we also used a questionnaire in order to evaluate the
reports.
As soon as the design, conduction and analysis of
the chosen research methods are completed, the findings are consolidated. In the course of this consolidation process, the results provided by the different research methods are checked for consistency and contradictions are identified and interpreted.
Subsequently, the catalogue of quality criteria is reviewed in the light of the research results: Which report alternatives met the requirements, which did not?
Were any additional requirements raised by the involved stakeholders or did some of the requirements
turn out to be obsolete?
If the quality criteria are not sufficiently satisfied, a
cause analysis is conducted. The cause analysis may
result in changes made to the design and content of the
reports, or a different technological basis may be
created that better suits the quality criteria. Alternatively, the catalogue of quality criteria itself may be modified.
If the quality criteria are finally satisfied, the
process model should be cycled after some time to
account for changes in stakeholder requirements over
time and to enable a continuous improvement process.
4 Evaluation and discussion of preliminary
results
In order to evaluate our proposed framework and
gain first insights into the stakeholder-focused requirements, we used the quality criteria collected in
Section 2 to design and implement a number of sustainability report alternatives and later on applied our
multi-method framework to them. Our initial test involved 8 participants (master students and employees
from the University of Osnabrueck) who acted as
stakeholders. We intentionally violated some of the
quality criteria in order to check whether our framework was suitable to discover these violations. Our
findings can be subdivided into findings on the framework itself, findings regarding the design and implementation of sustainability reports and implications for
future research.
1181
4.1 Findings on the framework
retrospectively and thus without distracting the participant during the completion of a task.
In addition to eye-tracking, we conducted a software-supported analysis using the software EyeQuant
which is based on a neural network that emulates human attention behavior [30]. It allows generating
heatmaps and other visual analyses of the presumed
user’s attention. Figure 3 shows a heatmap that was
generated by applying the software to one of our report
designs.
We will focus on the eye-tracking and softwaresupported evaluation results in this section since most
researchers are probably familiar with questionnaires
and observations. Additionally, we will point out the
main benefits and problems we encountered using a
multi method approach.
Although conducting eye-tracking is both timeconsuming and expensive, we found that it was well
worth the effort. By using eye-tracking, we gained
valuable information on what parts of a report the
participants actually used to solve certain tasks. Figure
2 shows the eye-tracking results of one participant for
one report alternative. The blue lines indicate saccades
(movements of the eyes) whereas the red circles indicate fixations (areas the participant focused on for a
longer period of time) [20].
Figure 3. EyeQuant results
Our experiences with EyeQuant were mixed since
the software was originally designed to evaluate websites and not sustainability reports: On the one hand,
compared to eye-tracking the analysis was a lot easier,
faster, and cheaper to conduct. The software produced
quite good results for reports that contained many
different elements such as tables, charts, legends etc.
However, for reports that included one large table only,
the software was not able to produce comprehensive
results. This is probably due to the fact that, in contrast
to real participants, the software is not able to focus on
a certain task. Generally, the software-supported analysis could not replace, but complement the eye-tracking
data.
Figure 2. Eye-tracking results
The participant was given the task to state the costs
for the cleaning of exhaust air in the year 2010 for the
product group washing machines. The eye-tracking
allowed us to find out that instead of using the table on
the left hand side, which could have easily provided
the exact number within a few seconds, this participant
decided to use the bar chart on the right hand side
instead, which did not allow him to complete the task
successfully. Obviously, the bar chart violated the
quality criterion suitability for the task which was
successfully discovered by our framework.
Some of our participants were interested in the
technology of eye-tracking and asked to see the visualized eye-tracking data afterwards. During the presentation of the eye-tracking results, we coincidently made
the experience that when viewing the recorded eyetracking data, participants were suddenly able to remember small details and decisions they made that
were not included in the results of our other research
methods. We thus believe that a retrospective confrontation of the participants with their eye-tracking data
can lead to interesting insights. In contrast to other
methods investigating the users’ cognitive processes,
such as thinking aloud [22], this method can be applied
Overall, the combination of different research and
analyses methods proved to be a good approach, which
corresponds to findings from existing literature
[3,4,15,17]. The results of each research method provided valuable insights into the stakeholders’ view of
quality criteria for web based sustainability reports:
• The eye-tracking data allowed us to evaluate
which components of a report were used to
complete a certain task and which components
were ignored. Furthermore, the retrospective
eye-tracking provided valuable insights into
the users’ cognitive processes.
1182
•
•
•
The questionnaire allowed us to evaluate the
stakeholder’s perception of the reports.
The observation allowed us to see whether the
stakeholder’s perception was biased.
The software-supported analysis extended our
findings from the eye-tracking in a more timeand cost-effective way.
2.
Naturally, using different research and analyses
methods may also lead to contradicting results: For
example, some participants perceived a report alternative to be useful and easy to use, although they were
not able to solve a given task using the report. We
found that these contradictions provide good starting
points for future research that we point out in Section
4.3 and 5.
ber for the cleaning costs of exhaust air for the
product group washing machines. For this task, the
report shown in Figure 2 was used.
For the results on the right hand side of the table,
the participants were asked to state whether the
revenue of a certain company location was going
up or down within a period of several weeks. The
layout of the report used to solve this task was
similar to Figure 2, offering both, a table as well as
a bar chart.
The two tasks were presented to the participants in
different orders to exclude bias due to the order in
which the tasks were presented or due to learning effects. After the completion of the tasks, the participants
were asked to fill out a questionnaire using a Likert
scale ranging from 1 (strong disagree) to 5 (strong
agree). The participants were asked whether the table
was helpful for the task, whether the chart was helpful
for the task and whether the visual appeal of the chart
was greater than that of the table.
We extended the questionnaire answers of the participants in Table 5 by adding whether the chart was
perceived to be more helpful than the table. Our observation contributed whether the given task was solved
correctly by the participant.
Obviously, we expected the table to be considered
more helpful for the first task, whereas we expected the
chart to be more helpful for the second task. Although
this expectation was generally fulfilled by our participants, as the higher mean values of the table for task 1
and of the chart for task 2 indicate, the results were not
as clear as we had imaged: 3 out of 8 participants
found the chart to be more helpful for the first task and
one participant was of the opinion that the table was
more helpful to solve task 2. Accordingly, out of these
4 participants highlighted in bold print in Table 5, only
one was able to correctly solve the given task.
This underlines our recommendations to
• provide coaching to internal stakeholders,
• offer a help function, wizard, instructional video or avatar to external stakeholders and
• preselect the visualization that is suitable for
the task where possible.
4.2 Findings on the design and implementation
of web based sustainability reports
Since the stakeholders did not always choose the
visualization alternative of report contents that was
suitable for the given task (cf. Figure 2), we conclude
that coaching of the users is desirable and should be
considered when designing and implementing SRs.
While coaching can be easily provided to internal
stakeholders such as employees, coaching obviously
cannot be easily realized for external stakeholders such
as customers. For web based reports coaching of the
users can be accomplished by adding a help function, a
wizard or an instructional video or avatar which demonstrates how to switch between different types of
visualization and which kind of visualization is suitable
for which task. In accordance with the quality criteria
suitability for the task (cf. Table 4) the appropriate
visualization should be pre-selected where possible.
In view of the fact that we discovered contradictions between the perceived and actual usefulness and
ease of use (cf. Section 4.1), we argue that sustainability reports need to address both aspects, the users’ perception as well as the factual properties, since both
may contribute to the overall acceptance of a report.
Table 5 shows an excerpt from our questionnaire
and observation results. The 8 participants were given
two tasks:
1. For the results on the left hand side of the table,
the participants were asked to state an exact num-
Surprisingly, the results for the second task were
more significant than for the first task although the
report designs were quite similar.
1183
Table 5.Questionnaire and observation results (excerpt)
task
participant
A
B
C
D
E
F
G
H
mean
table
helpful for
task
3
5
1
5
5
3
4
5
3.875
stating a concrete number
(cf. report Figure 2)
chart
helpful
helpful for
chart >
task
task
table?
solved?
4
no
no
3
yes
yes
4
no
yes
3
yes
yes
1
yes
yes
5
no
no
2
yes
no
1
yes
yes
2.875
visual appeal
chart > table?
5
3
4
4
4
4
1
1
3.25
table
helpful for
task
2
2
3
1
1
3
2
2
2
identifying a trend
(report similar to Figure 2)
chart
helpful
helpful for chart >
task
task
table?
solved?
5
no
yes
5
no
yes
2
yes
no
5
no
yes
5
no
yes
5
no
yes
3
no
yes
5
no
yes
4.375
visual appeal
chart > table?
5
3
1
4
4
4
4
5
3.75
hand side were too glaring which might be the reason
why the results were not as significant as we expected.
Regarding the color scheme, we conclude the following recommendations:
• color schemes should be intuitive,
• a legend of the color scheme should be included in order to provide orientation,
• colors should not be too glaring and friendly to
the eye.
Although the chart was by and large considered to
be less helpful for the first task, the paarticipants considered the chart to be visually more appealing in both
cases. One could conclude that sustaainability reports
should therefore generally include charts rather than
tables, but as our results also indicated charts may not
always be suitable for the task at hand. Moreover, as
our eye-tracking and observation results indicated, the
participants were not always able to select the appropriate visualization of data for the given task.
4.3 Implications for future research
Figure 4 shows two alternative reports that were
designed using different color schemes.
Although we were overall satisfied with the combination of research methods we chose for our framework and the preliminary study, further research on the
combination of different research methods is needed:
Which research methods complement each other? How
can contradictions between results from different research methods be resolved during the consolidation of
findings?
We believe that due to the fact that the research and
analyses methods included in the multi-method approach can be replaced by others, our framework can
be easily adapted to suit other problem domains.
We have already made the experience, that some
research methods, e.g. eye-tracking, are more timeconsuming and costly than others, e.g. the softwaresupported analysis. While we only considered a rather
small group of participants, an economical evaluation
of different combinations of research and analyses
methods could provide interesting insights for researchers and practitioners alike: Which combination
of research methods in a multi-method approach offers
the best price performance ratio?
Figure 4. Color Schemes
Although the two alternatives use the same layout
and include the same information, the stakeholders’
experiences differed strongly. While the report on the
left uses established traffic light colors (green for low
costs/high profits, yellow for medium costs and profits,
red for high costs/low profits) the report on the right
uses colors that have no common connotation. Although the tasks provided to the participants were
equally difficult, not surprisingly, thee participants on
average needed 72% more time in order to complete
the task using the report on the right. The eye-tracking
results backed up this finding by shoowing confused
participants without orientation. Surprisingly, the
stakeholders’ questionnaire answers found the two
reports to be almost equally appealing (2.9 for the
report on the left compared to 2.75 for tthe report on the
right). The results were more significant regarding the
helpfulness for the task (3.3 vs. 2.63) and the intuitiveness of the color scheme (3.5 vs. 2.5). Some participants remarked that the colors in the report on the left
In view of many scientific contributions on the acceptance of technologies that rely on questionnaires in
order to determine acceptance factors such as ease of
use and usefulness, our discovered discrepancies between questionnaire and observation results show that
future research might have to distinguish between the
1184
perceived (questionnaire based) and actual (observed,
measured, eye-tracked) ease of use as well as perceived
and actual usefulness. Some researchers have already
started to make a distinction between perceived and
objective acceptance criteria [24]. However, we believe that further research into the cause for the differences between perceived and actual acceptance criteria
and their impact on the overall acceptance is needed.
Our preliminary results indicate that the visual appeal
of a report may have a strong influence on the user’s
perceived ease of use and usefulness. The empirical
investigation of this hypothesis could be another starting point for future research.
tation of sustainability reports as well as implications
and possible starting points for future research.
Naturally, our research approach suffers from some
limitations: The preliminary study of our framework
was set in an artificial environment (laboratory experiment) that possibly biased the results. However, we
believe that the applied multi-method approach reduces
this problem to a large extend. Nevertheless, future
research can focus on settings that are more ‘natural’ to
the participants.
Furthermore, our preliminary study only involved a
small number of participants and we only conducted
one iteration of the framework. Ideally, a larger number of users and a more representative user population
could improve the findings. Thus, our proposed
framework needs to be further evaluated in the future.
So far, our framework is a theoretical construct that
still needs to prove its value in practice. Due to the
interchangeability of the included research and analyses methods, however, it can be easily modified by
researches and practitioners alike to meet their needs.
We invite other researchers to apply and to adopt our
framework to other problem domains than sustainability reporting (e.g. the design of websites or software
user interfaces) in order to further evaluate it.
For the authors of standards and guidelines on sustainability reporting, we conclude that the quality criteria should be extended by common acceptance factors
from the IS disciple [6,24,25]. Since many companies
publish their SRs in the form of a website [21], it is no
longer sufficient to focus on quality criteria regarding
the content of static PDF documents [13,14]. Also, the
guidelines could include more advice on how to evaluate the stated quality criteria and information on how
the criteria were constituted in the first place.
Other researchers have already started to explore
the additional possibilities that web based sustainability reports offer in comparison to traditional, paper
based reports. Web based reports can, for example,
facilitate two-way communication between the reporting company and its stakeholders and provide the possibility to generate custom-tailored reports for different
stakeholder groups [13,14]. The application of the
presented framework might help to derive additional
quality criteria and further evaluate these approaches.
Acknowledgement
This work is part of the project IT-for-Green (Next
Generation CEMIS for Environmental, Energy and
Resource Management). The IT-for-Green project is
funded by the European regional development fund
(grant number W/A III 80119242). The authors are
pleased to acknowledge the support by all involved
project partners. Furthermore, we would like to thank
the anonymous reviewers for their insightful and constructive comments.
5 Conclusion
This paper contributes to the research on sustainable development in general and on web based sustainability reports in particular by giving an overview of the
goals and quality criteria in existing reporting standards and guidelines. Our overview shows that existing
standards and guidelines are too focused on the content
of the reports and, in view of the stakeholder-focused
nature of sustainability reporting, neglect common IS
acceptance criteria which have proven to be valid in
many other problem domains. In response to these
shortcomings, we presented a multi-method framework
that directly involves the different stakeholder groups
into the definition of quality criteria and the corresponding evaluation. This paper presented initial findings on the framework and the included multi-method
approach itself, findings on the design and implemen-
6 References
[1] W. Bartels, J. Iansen-Rogers, and J. Kuszewski, Count
Me In - The Readers’ Take on Sustainability Reporting,
KPMG, 2008.
[2] G.H. Brundtland, editor., Our Common Future, Oxford:
Oxford University Press, 1987.
[3] J. Cao, M. Lin, J. Crews, A. Deokar, and J. Burgoon,
“The Interaction Of Research Methods For System Evaluation And Theory Testing: A New Vision Of The Benefits Of
Multi-Methodological Information Systems Research,” Proceedings Of The International Conference On Information
Systems (ICIS), 2004.
1185
[4] D. Cyr, M. Head, H. Larios, and B. Pan, “Exploring
Human Images In Website Design: A Multi-Method Approach,” MIS Quarterly, vol. 33, 2009, pp. 1-32.
[17] J. Mingers, “Combining IS Research Methods: Towards
A Pluralist Methodology,” Information Systems Research,
vol. 12, Sep. 2001, pp. 240-259.
[5] C.-H. Daub, “Assessing The Quality Of Sustainability
Reporting: An Alternative Methodological Approach,” Journal of Cleaner Production, vol. 15, 2007, pp. 75-85.
[18] H. Oesterle and B. Otto, “Consortium Research,” Business & Information Systems Engineering, vol. 2, Aug. 2010,
pp. 283-293.
[6] W.H. DeLone and E.R. McLean, “The DeLone And
McLean Model Of Information Systems Success : A TenYear Update,” Journal of Management Information Systems,
vol. 19, 2003, pp. 9-30.
[19] J.W. Palmer, “Web Site Usability, Design, and Performance Metrics,” Information Systems Research, vol. 13, Jun.
2002, pp. 151-167.
[7] Deutsches Institut für Normung e.V., DIN EN ISO 9241110, Deutsches Institut für Normung e.V., 2008.
[20] D.D. Salvucci and J.H. Goldberg, “Identifying Fixations
And Saccades In Eye-Tracking Protocols,” Proceedings Of
The 2000 Symposium On Eye Tracking Research And Applications, 2000, pp. 71-78.
[8] EMAS Eco-Management and Audit Scheme, “Regulation
(EC) No 1221/2009 Of The European Parliament And Of
The Council,” Official Journal of the European Union L
342/1, 2009.
[21] A. Slater, International Survey Of Corporate Responsibility Reporting 2008, KPMG, 2008.
[22] M.W. van Someren, Y.F. Barnard, and J.A.C. Sandberg,
The Think Aloud Method - A Practical Guide To Modelling
Cognitive Processes, London: Academic Press, 1994.
[9] O. El-Gayar and B.D. Fritz, “Environmental Management
Information Systems (EMIS) for Sustainable Development A Conceptual Overview,” Communications of the Association for Information Systems, vol. 17, 2006, pp. 756 - 784.
[23] S. Townsend, W. Bartels, and J.-P. Renaut, Reporting
Change: Readers & Reporters Survey 2010, Futerra Sustainability Communications, 2010.
[10] J. Elkington, “Partnerships from cannibals with forks –
The triple bottom line of 21st-century business,” Environmental Quality Management, vol. 8, 1998, pp. 37-51.
[24] V. Venkatesh and H. Bala, “Technology Acceptance
Model 3 And A Research Agenda On Interventions,” Decision Sciences, vol. 39, May. 2008, pp. 273-315.
[11] GRI Global Reporting Initiative, Sustainability Reporting Guidelines 3.1, Global Reporting Initiative, 2011.
[25] V. Venkatesh, M.G. Morris, G.B. Davis, and F.D. Davis,
“User Acceptance Of Information Technology - Toward A
Unified View,” MIS Quarterly, vol. 27, 2003, pp. 425-478.
[12] ISO International Organization for Standardization,
ISO/FDIS 14063: Environmental Management - Environmental Communication - Guidelines and Examples, 2006.
[26] WBCSD World Business Council for Sustainable Development, Sustainable Development Reporting - Striking
The Balance, World Business Council for Sustainable Development, 2003.
[13] R. Isenmann, C. Bey, and M. Welter, “Online Reporting
For Sustainability Issues,” Business Strategy and the Environment, vol. 16, 2007, pp. 487-501.
[27] R. Wang and D.M. Strong, “Beyond Accuracy: What
Data Quality Means to Data Consumers,” Journal of Management Information Systems, vol. 12, 1996, pp. 5-34.
[14] R. Isenmann, J.M. Gómez, and D. Süpke, “Making
Stakeholder Dialogue For Sustainability Issues Happen –
Benefits , Reference Architecture And Pilot Implementation
For Automated Sustainability Reporting à la Carte,” Proceedings of the 44th Hawaii International Conference on
System Sciences, IEEE Computer Society, 2011.
[28] R.T. Watson, M.-C. Boudreau, and A.J. Chen, “Information Systems and Environmentally Sustainable Development:
Energy Informatics and New Directions for the IS Community,” MIS Quarterly, vol. 34, 2010, pp. 23-38.
[15] J. Maib, R. Hall, H. Collier, and M. Thomas, “A MultiMethod Evaluation Of The Implementation Of A Student
Response System,” Proceedings Of the Americas Conference
On Information Systems (AMCIS), 2006.
[29] J. Webster and R.T. Watson, “Analyzing The Past To
Prepare For The Future - Writing A Literature Review,” MIS
Quarterly, vol. 26, 2002, p. xiii-xxiii.
[16] N.P. Melville and S.M. Ross, “Information Systems
Innovation for Environmental Sustainability,” MIS Quarterly, vol. 34, 2010, pp. 1-21.
[30] WhiteMatter Labs, “EyeQuant Attention Analytics,”
URL: http://eyequant.com/ last accessed: 15.06.2011.
1186