The publication of rankings of schools

The publication of school rankings: a step toward increased accountability?
Ana Balcão Reis*,t
Carmo Seabra*,t
Luis C. Nunes*,t
*Nova School of Business and Economics,
Universidade Nova de Lisboa,
Campus de Campolide 1099-032 LISBOA
Portugal
October 2013
Abstract
This paper contributes to the discussion of the effects of the publication of school rankings. We
study the effectiveness of this (low-stakes) accountability mechanism. Our results suggest that
the publication of rankings has strong effects upon families and schools. After the rankings
publication, students move out of badly classified schools and the probability of closing for
these schools increases. These effects are stronger for private schools.
Keywords: school accountability, school rankings
JEL: I21, I28, H42
* Email: [email protected] , [email protected] , [email protected]
We would like to thank João Miguel Silva and Adriana Ferro for excellent research assistance. Financial
support from Fundação para a Ciência e Tecnologia is gratefully acknowledged.
t
0
The publication of school rankings: a step toward increased accountability?
October 2013
Abstract
This paper contributes to the discussion of the effects of the publication of school rankings. We
study the effectiveness of this (low-stakes) accountability mechanism. Our results suggest that
the publication of rankings has strong effects upon families and schools. After the rankings
publication, students move out of badly classified schools and the probability of closing for
these schools increases. These effects are stronger for private schools.
Keywords: school accountability, school rankings
JEL: I21, I28, H42
1
1. Introduction
During the last decade, many countries and/or states have introduced some form of
accountability in schools in an attempt to increase their performance. This movement was at
least partly generated by the frustration felt by many countries with the results of their heavily
financed public education systems. The weaknesses of these results became clearer as the
realization of internationally comparable exams, like PISA and TIMMS, was generalized.
Accountability policies vary considerably, from the so-called low-stakes policies (disclosing
information regarding the performance of schools) to high-stakes or consequential systems,
whereby the financing conditions of the schools and/or the payments to teachers are adjusted
according to the performance of students. The effectiveness of these accountability systems on
the learning conditions of students has been studied extensively in the literature.
Using different data sets and methodological tools, several authors have shown that
accountability policies, even low-stakes ones, such as the one considered in this paper, have
significant and positive effects on students’ behavior. Figlio and Lucas (2004) showed that lowstakes accountability policies have strong effects on parents’ choices and are reflected in
changes in housing prices. Using data from Florida, where a hard-stakes policy was launched,
Figlio and Rouse (2006) concluded that the improvements of poorly performing schools were,
in fact, very large; Chakrabarti (2008) examined the impact of different incentive schemes and
showed that the 1999 Florida accountability program unambiguously improved schools’
performance. Chiang (2009) also found persistent gains in performance due to Florida’s highstakes accountability policy; Hanushek and Raymond (2004) reported increased scores for all
students, following the introduction of accountability policies in the US, although they found a
much weaker impact of low-stakes policies relative to the effects of sequential or high-stakes
measures. Using data from PISA 2003 for 27 OECD countries, Schutz et al. (2007) also found a
positive relationship between accountability and academic outcomes. 1
In this paper we analyze the impacts of the public disclosure of school rankings, a low-stakes
accountability policy. We consider the case of Portugal, where every year several newspapers
publish rankings of every high school (public and private) based on the average scores obtained
by students on national exams.
Secondary school rankings were first published in 2001, were not published in 2002, and then
published every year thereafter. We study the effect of the publication of the rankings in terms
of the ability of schools to attract students and of the increased probability of closing of badly
classified ones. To identify the effect of the public disclosure of school results, we compare the
effects of the rankings before and after their publication. We distinguish between private and
public schools, as we expect that there is more flexibility and freedom of choice in the subset of
private schools, allowing for a stronger effect of the rankings on these schools.
1 Positive effects of several accountability systems have been found for several other geographical and
institutional settings (see, for instance, Koning and van der Wiel (2012) for the Netherlands, Anghel et al.
(2012) for Spain, Rockoff and Turner (2010) for New York City).
2
The organization of the paper is the following: Section 2 briefly describes the institutional
setting of the education system in Portugal for the period under analysis. Section 3 describes
the data. In Section 4 we present our methodology and our results for the effect of rankings
publication on students’ relocations. In this section we also study the impact of the published
ranking on the probability that schools close. These analyses help to understand the impact and
the vehicle through which the publication of rankings affects schools. Section 5 discusses the
results and concludes.
2. The Institutional setting in the period 1998 to 2005
The Portuguese educational system did not undergo significant changes from 1998 to 2005.
Mandatory schooling starts at the age of six and lasts 9 years.2 These 9 years are divided into
three cycles with durations of 4, 2, and 3 years, respectively. Tracking in general starts at the
end of mandatory schooling 3, with the choice between 3 additional years of academic studies
aimed at the pursuit of studies at the university level, or professionally oriented studies.
Vocational study tracks can consist of either 2 or 3 years, and transition to academic tracks or
to the university is possible. The net rate of enrollment in secondary education was around 60%
in the period analyzed. Below we describe only academic studies 4, as these are the subject of
this paper.
To finish academic secondary school, students have to take final exams, carried out at the
national level. Final scores per subject are computed as an average of the score attributed by
the student’s school (internal score) and the score obtained at the final exam.
These national exams perform two roles in the educational system. Besides being required to
graduate from secondary school, they also determine the conditions for admission to the
university. Portuguese Public Universities have fixed slots set for each course by the Ministry,
that fall short of demand for places in the most prestigious degrees. Candidates for each
degree are selected (centrally, by the government) according to their order number in the
candidacy grade. This grade is a weighted average of the score obtained at the end of
secondary studies, calculated as described above, and the scores obtained in one or two of the
mandatory national exams 5. These specific exams, thus, enter twice in the definition of the
candidacy grade. These rules place a strong weight on the scores obtained in secondary final
exams.
Public and Private schools co-exist in the Portuguese educational system. The level of
autonomy of Portuguese schools is very limited, with teaching contents and learning methods
being decided by the government. Private schools can choose teachers and students, and set
2
After our period of analysis, some changes occurred, namely the extension of mandatory schooling by 3
years. The first cohort of students affected by this measure entered secondary schools only in 2010.
3
There are some professionally oriented paths initiated at the age of 14, but these tracks remained
exceptional, accounting for only 4 to 5% of the schooling population.
4
The net rate of enrollment in secondary education ran from 59.1% in 1998 to 59.8% in 2005; out of
these, the percentage of students in academic studies decreased during the period, from 65% in 1998 to
55% in 2005.
5
The choice of the specific subjects that are considered twice on the candidacy grade for each course is
left to the course authorities. This choice has been kept stable since 1998.
3
tuition fees. 6 They are allowed to allocate teaching time to the different subjects, but only
according to strict requirements set by the government. They are also subject to a set of rules
concerning the premises, the number of students allowed per class, and so forth. Public schools
are strictly restricted in their educational supply, and neither choose nor influence the choice of
teachers.
The allocation of students to public schools and to publicly financed private school is based
largely on geographical criteria. For the years analyzed, there were legal restrictions on the
specific school where students could enroll, according to the geographical proximity, first, to
the home of the family, and second, to the work place of the mother or father. Nevertheless,
the large movements of students in and out of schools in our data reveal that those legal
restrictions were not being strictly enforced.
3. Data Description
The publication of school rankings in Portugal occurred for the first time in 2001. The first
ranking was published by the newspaper Público, following a court decision on a law suit by this
newspaper, claiming its right to disclose the data that were being denied by the Government.
In the following year, 2002, the Ministry of Education decided to publish its own ranking, taking
into account some socio-economic characteristics of the school region. This ranking was
strongly criticized, and from 2003 on the Ministry started disclosing the raw data to the media
that treat them and publish the rankings.
The published ranking is the simple ordering that results from calculating the average score on
national exams for each school. This average is obtained taking into account a selection of
exams that correspond to the courses with more students at the national level (Mathematics,
Portuguese, etc.). The choice of which exams to include in this ranking is a decision of the
newspaper. 7
Our dataset consists of these same data disclosed to the media and includes scores for all
student-exam pairs for all of the 12th grade national exams since 1998. The exam scores have
been published online since 2003 by the Ministry of Education, and may be freely
downloaded. 8 The data for the period before the beginning of publication, covering years 1998
to 2001, became publicly available only at the end of 2012.
Our main variable of interest is the ranking of each school, RANK. We replicated the school
rankings published by the newspaper Público, given that our purpose was to analyze the impact
of published rankings. For the period before publication we built equivalent rankings based on
the students’ scores in each school, following the criteria adopted by Público in its publication.
The variable RANK takes the value 1 for the school with higher average and increases as the
school average decreases.
6
Some private schools are publicly financed and neither choose their students nor set tuition fees.
After this several newspapers started publishing their own rankings, but Público remained the principal
source on this issue.
8
http://www.dgidc.min-edu.pt/jurinacionalexames/
7
4
To determine whether the publication of school rakings affects students' choices we first
analyze whether the percent change in the number of students in each school depends on
previous rankings. As we have no access to the number of students enrolled in each school, we
extracted from the dataset described above the number of exams taken at each school in each
year. This is our measure of the size of the school, DIMS. The percent change in the number of
exams taken at each school in the period under analysis is our dependent variable, ΔN.
We then look at the closing of schools. We define a dummy variable, CLOSE, that takes the
value 1 if the school has students taking exams in the initial year of the period under analysis
and no students taking exams in the final and subsequent years, and 0 otherwise.
The dataset also includes information about school characteristics that we use as additional
controls: public, PUB, versus private, PRIV, status and geographical localization. We use this last
information to build several variables concerning the availability of schools in each municipality,
and thus, the effective competitive pressure and freedom of choice of school by the students.
We considered three alternatives: (i) CHOICE, a dummy that equals 1 if there is more than one
school in the municipality, and thus, the possibility of choice, and 0 otherwise; (ii) NSMUN, the
number of schools in the municipality, and (iii) HH, the Herfindahl-Hirschman Index of
concentration of schools at the municipal level.
To take into account that it is probably easier to move to a school in the nearby region, we
have also calculated regional rankings. These would be the most relevant rankings if each
school competes mainly with the schools in the same region. We considered that for each
school the relevant region would be the one resulting from considering only the neighboring
municipalities.
The dataset does not include any information on the socio-economic characteristics of the
students, and for the period under analysis this information is not available at the school level,
either.
4. The impact of published rankings
Our goal is to study the effect of the publication of the rankings. The gold standard for
evaluating the impact of a treatment (in our case the publication of the rankings) is the
randomized experiment in which units (in our case the schools) are assigned into two groups at
random. One of the groups receives the treatment while the other, the control group, does not.
In such a setting, one can simply take the difference between the outcomes of the two groups
to measure the impact of the treatment. Randomized experiments are relatively rare in
practice. For instance, assignment into the two groups may not be random, leading to
differences in characteristics in the two groups which confound the impact of the treatment. If
data before and after the treatment are available, the impact of the differences in
characteristics can be eliminated by analyzing the difference-in-differences of the outcome
variable. For this approach to be valid it requires that in the absence of the treatment, the two
groups would have followed parallel trends. When the probability of receiving the treatment is
a discontinuous function of one or more variables, an alternative approach consists in a
regression discontinuity design. The idea is that by using only observations lying close to either
5
side of the threshold it is possible to obtain two groups with similar characteristics, except that
one received the treatment and the other did not.
In our study, the rankings were published for all schools. If the form of publication consisted of
assigning specific labels to each school based on student performance on the exams, it would
be possible to divide the schools into different groups and to follow a difference-in-differences
approach (since we have data before and after the publication) or a regression discontinuity
design (if the assignment of the labels was a discontinuous function of student performance).
However, the form of publication consisted in disclosing only an ordered list of schools.
Basically, each school is associated with a particular order so that there are no explicit control
and treatment groups. As a result, we have to follow a different strategy.
To capture the impact of the publication of the ranking, and because we cannot divide the
schools into treatment and control groups, we estimate linear regression and probit models for
the outcome variables with the ranking variable entering as an explanatory (continuous)
variable. We include additional controls to capture other differences in the characteristics of
the schools that may affect the outcome variables. Since we have data before and after the
publication of the rankings, we estimate the models for the two periods and check for
differences in how the ranking variable affects the outcome variable. If the impact of the
ranking is the same in the two periods, we conclude that the publication had no impact. For
this interpretation to be valid, the main assumption required is that, apart from the control
variables included in the estimated models, the only difference between the two periods is the
publication. In fact, as discussed above, between 1998 and 2005 there were no major changes
in the educational system.
To define the two estimation periods, before and after the publication, it is important to
understand the timing of rankings’ publication and of students’ enrollments. Rankings are
published at the end of summer, after the enrollment of students for the following academic
year, which happens in June and July. So, the 2001 rankings are expected to have affected
enrollments for the academic years of 2002/2003 and following. Moreover, although some
students change school in the transition to the 12th grade, most of the movements between
schools occur in the transition to the 10th grade. The exams are taken at the end of each
academic year, in June. Thus, the impact of the 2001 rankings is expected to appear in the
number of students taking exams in 2003 and, more strongly, in 2005, when secondary school
exams are taken by those students who enrolled in the 10th grade in June 2002. We define our
periods of analysis taking these timings into account.
For the impact of publication, we look at the percent variation in the number of students taking
exams in each school between 2001 and 2005. We regress this variation on the rank in 2001
and school characteristics at the beginning of the period. We consider the number of exams in
2001 to be an appropriate point of departure for the analysis as we may expect it to be
exogenous relative to the ranking published in 2001, which was the first published ranking.
For the period before publication we look at the percent variation in the number of students
between 1998 and 2002. As before, we regress this variation on the rank in 1998 and school
characteristics at the beginning of the period.
6
Table 1: Descriptive statistics
Period before publication (schools with more than 50 exams in 1998): 569 schools
Variable
Description
Mean
St deviation
Max
Min
∆N
percent change in the number of exams between
1998 and 2002
-11.19
36.98
352.59
-100.00
RANK
Rank in 1998
299.43
171.59
607
2
PUB
Dummy=1 for public schools
0.83
0.38
1
0
DIMS
Number of exams in the school in 1998
375.88
236.48
1531
53
NSMUN
Number of schools in the municipality in 1998
11.28
18.12
62
1
0.73
0.44
1
0
0.46
0.37
1
0.02
0.01
0.11
1
0
CHOICE
HH
CLOSE
Dummy=1 if there is another school in the
municipality in1998
Herfindahl-Hirschman Index of Concentration for
1998
Dummy=1 if the school has no exams from 2002
onwards
Period after publication (schools with more than 50 exams in 2001): 583 schools
Variable
Description
Mean
St deviation
Max
Min
∆N
percent change in the number of exams between
2001 and 2005
1.70
44.11
307.35
-100.00
RANK
Rank in 2001
309.93
175.37
620
2
PUB
Dummy=1 for public schools
0.82
0.38
1
0
DIMS
Number of exams in the school in 2001
339.52
221.01
1648
51
NSMUN
Number of schools in the municipality in 2001
10.52
16.86
60
1
0.72
0.45
1
0
0.47
0.37
1
0.02
0.04
0.19
1
0
CHOICE
HH
CLOSE
Dummy=1 if there is another school in the
municipality in 2001
Herfindahl-Hirschman Index of Concentration for
2001
Dummy=1 if the school has no exams from 2005
onwards
We restricted our analysis to schools that had in the initial year of analysis more than 50
exams. 9 This eliminates extreme cases with very large percent variations due to the low initial
number of exams. Table 1 lists the main variables and presents some descriptive statistics for
the two periods of analysis, 1998-2002 and 2001-2005. Except for ∆N and CLOSE, there are no
noticeable differences in the distribution of the variables between the two periods.
In 1998 there were 569 secondary schools with more than 50 students applying to 12th grade
exams, 99 private and 470 public. Of these, 7 closed before 2002. In 2001, there were 583
schools, 104 private and 479 public. Of these, 22 closed before 2005.
9
The variable RANK corresponds to the ranking published by Público which includes all the schools, 609
in 1998 and 622 schools in 2001.
7
4.1. The impact of published rankings on the Percent Change in the Number of Students
In order to evaluate whether the publication of the rankings affects students’ decisions about
the school that they choose to enroll in, we regressed the percent change in the number of
exams performed in each high school on the rank of the school for the periods before and after
the first publication, and to test whether there are systematic differences between those two
periods. For the period before publication the explanatory variable is the rank implied by
school scores in 1998 (which was not public knowledge at that time), and for the period after
publication the independent variable is the rank published in 2001.
We controlled for the private versus public nature of the school. In order to test whether the
impact of rankings was weaker for public schools, we also considered the cross-effect between
the rank and the public character of the school, by including the interaction variable PUB×RANK
and PRIV×RANK, where the variable PRIV is a dummy that identifies private schools. We also
controlled for the size of the school and for the degree of competition among schools in each
region, as stated above.
To test if the impact of published rankings was greater for schools subject to stronger
competition, we also considered the cross-effect between the rank and the competition
measure.
Our benchmark specification is the following:
∆N i = α + β1 PUBi + β 2 PUBi × RANK i + β 3 PRIVi × RANK i
+ β 4 DIMS i + β 5CHOICEi + β 6 CHOICEi × RANK i + ε i
Models were estimated by ordinary least squares. We also included in all the regressions
dummy variables for the administrative regions (18 plus the islands) to control for differences
in the demographic evolution across regions and other regional characteristics. Estimation
results are shown in Tables 2.a and 2.b.
8
Table 2.a: Estimation Results of the Linear Regression Model
Dependent variable: ∆N, 1998-2002
1
-5.94
CONST
2
1.71
5
28.52***
6
21.00**
7
43.36***
8
41.74***
6.57
8.5
7.6
9.95
9.07
9.98
11.72
13.67
0.37
0.84
0.61
0.52
0.00
0.04
0.00
0.00
-0.02**
-0.04**
-0.07***
-0.05**
s.e.
0.01
0.02
0.01
0.02
Prob
0.03
0.03
0.00
0.01
-1.63
-1.52
-5.30
-7.03
-5.04
-5.07
-28.95***
-28.64**
s.e.
5.27
5.27
7.3
7.42
6.23
6.25
11.68
11.89
Prob
0.76
0.77
0.47
0.34
0.42
0.42
0.01
0.02
-0.03***
-0.03***
-0.03***
-0.03***
-0.02**
-0.02**
-0.01*
-0.01
s.e.
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
Prob
0.00
0.00
0.00
0.00
0.05
0.05
0.1
0.1
-0.47
-9.33
-0.07
-10.58
-11.76***
-2.76
-10.45**
-8.77
s.e.
4.11
7.59
4.13
7.64
4.05
-8.11
4.03
8.14
Prob
0.91
0.22
0.99
0.17
0.00
0.73
0.01
0.28
DIMS
CHOICE
0.02
0.03
-0.03
-0.005
s.e.
0.02
0.02
0.02
0.02
Prob
0.29
0.21
0.25
0.83
CHOICE×RANK
RANK×PUB
s.e.
Prob
-0.02*
-0.04**
-0.05***
-0.05***
0.01
0.02
0.01
0.02
0.08
0.04
0.00
0.01
-0.03
-0.06*
-0.13***
-0.12***
s.e.
0.02
0.03
0.03
0.03
Prob
0.19
0.07
0.00
0.00
RANK×PRIV
District Dummies
2
2








0.09
0.10
0.09
0.10
0.15
0.15
0.17
0.17
0.06
0.06
0.05
0.05
0.12
0.12
0.13
0.13
-0.02
-0.08***
s.e.
0.01
0.01
Prob
0.14
0.00
β RANK +β CHOICE×RANK
0.01
0.02
0.08***
0.08***
s.e.
0.03
0.03
0.03
0.03
Prob
0.63
0.48
0.01
0.01
569
569
583
583
β RANK×PUB -β RANK×PRIV
# Observations
4
6.36
s.e.
PUB
ADJUSTED R
3
-3.93
Prob
RANK
R
Dependent variable: ∆N, 2001-2005
569
569
583
583
Notes: s.e stands for robust standard-errors
*** significant at the 1 percent level, ** significant at the 5 percent level, * significant at the 10 percent level.
9
Table 2.b: Estimation Results of the Linear Regression Model
Dependent variable: ∆N , 1998-2002
2
3
4
5
6
7
8
10.32
8.88
11.56
9.16
39.61***
34.62***
55.69***
50.93***
s.e.
7.08
7.13
8.32
9
9.66
9.56
12.18
12.92
Prob
0.15
0.21
0.17
0.31
0.00
0.00
0.00
0.00
CONST
RANK
-0.03***
-0.02**
-0.08***
-0.06***
s.e.
0.01
0.01
0.01
0.01
Prob
0.01
0.04
0.00
0.00
-3.75
-3.94
-5.81
-4.31
-5.31
-5.41
-29.99***
-27.27**
s.e.
5.31
5.27
7.41
8.08
6.16
6.16
11.72
12.24
Prob
0.48
0.46
0.43
0.59
0.39
0.38
0.01
0.03
PUB
DIMS
-0.03***
-0.03***
-0.03***
-0.03***
-0.03***
-0.03***
-0.02***
-0.02***
s.e.
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.01
Prob
0.00
0.00
0.00
0.00
0.00
0.00
0.01
0.00
0.38***
-0.30**
-0.38***
-0.30**
-0.60***
-0.31
-0.58***
-0.42**
s.e.
0.1
0.13
0.1
0.13
0.17
0.21
0.17
0.22
Prob
0.00
0.02
0.00
0.02
0.00
0.14
0.00
0.05
NSMUN
NSMUN×RANK
-0.00
-0.00
-0.00**
-0.00
s.e.
0.00
0.00
0.00
0.00
Prob
0.42
0.48
0.04
0.28
RANK×PUB
-0.03**
-0.02**
-0.05***
-0.04***
s.e.
0.01
0.01
0.01
0.01
Prob
0.01
0.04
0.00
0.00
-0.03
-0.02
-0.13***
-0.11***
0.02
0.03
0.03
0.03
RANK×PRIV
s.e.
Prob
District Dummies
2
R
ADJUSTED R
β RANK*PUB β RANK×PRIV
Dependent variable: ∆N , 2001-2005
1
2
0.18
0.42
0.00
0.00








0.11
0.11
0.11
0.11
0.17
0.18
0.19
0.19
0.07
0.07
0.07
0.07
0.13
0.14
0.15
0.15
0.01
0.00
0.08***
0.07**
s.e.
0.03
0.03
0.03
0.03
Prob
0.79
0.96
0.01
0.03
569
569
583
583
# Observations
569
569
583
583
Notes: s.e stands for robust standard-errors
*** significant at the 1 percent level, ** significant at the 5 percent level, * significant at the 10 percent level.
We discuss in detail the results shown in Table 2.a. where the control variable CHOICE is
considered. Similar conclusions are derived from Table 2.b. considering instead the control
variable NSMUN.
10
For both periods and all models, the size of the school, DIMS, has a negative impact on the
percent variation in the number of exams. The parameters are in general significant at the 1
percent level. The number of schools available in the region also has a negative and significant
impact on the percent variation in the number of exams. 10
For the period before publication, according to model 1 (column 1 in Table 2.a) the effect of the
ranking is negative and significant. In model 2 we include the interaction between choice and
ranking to capture the possibility that the impact of the ranking is amplified when there are
alternative schools in the municipality. However, this interaction is not significant. In model 3
we check whether the impact of the rank is different for public and private schools. We find no
significant differences, as can be checked in the last row of Table 2.a. Finally, in model 4 we
allow for both the interaction between choice and the rank and the difference between the
impact of the rank in public and private schools and obtain that none is significant. Thus, model
1 appears as the one that best fits the data for the period before publication.
For the period after publication we estimated the same models, and obtain a negative and
significant effect of the rank in model 5. We obtain again, in models 6 and 8, that the
interaction between rank and choice is never significant. In model 7 we obtain that now the
impact of the rank is significantly different for private and public schools. Hence, for this period
model 7 is the preferred specification.
The above findings suggest that the impact of the ranking changed due to the publication of the
ranking. This is also confirmed by the results of the tests for differences in the relevant
parameters comparing models 1 versus 5 and 3 versus 7 shown in Table 3.
Table 3: Tests for Equality of the Parameters before and after publication (t-statistics)
RANK
RANK×PUB
RANK×PRIV
Models
1 versus 5
3 versus 7
-3.21***
-1.84*
-2.65***
*** significant at the 1 percent level, * significant at the 10 percent level
Our main finding is that the direct impact of the ranking on the number of exams has the
expected negative sign and is significantly stronger for the period after publication. An increase
in the rank of the school (corresponding to a deterioration of quality) leads to a decrease in the
number of exams carried out in that school and this effect is much stronger when the
information about rankings is published, and thus, available to everyone, as shown in Table 3.
This Table also shows that the impact of the rank in both public and private schools changed
10
We do not show the results with the Herfindahl-Hirschman Index of concentration, as they are
qualitatively identical to those shown.
11
significantly with the publication of the rankings. Furthermore, private schools were more
strongly affected by publication.
4.2. The impact of published rankings on the probability of closing
As an important part of the effect of published rankings may be the closing of badly classified
schools, we now look at the event of closing. In our analysis a closed school is one that presents
no students to the national exams in the final year of the period analyzed.
We estimated the probability that a school i closes as a function of published rankings,
considering the same controls as in the previous regressions. We assumed a probit function for
the probability to close. 11
Prob(CLOSEi = 1) = Φ (α + γ 1 PUBi + γ 2 PUBi × RANK i + γ 3 PRIVi × RANK i
+ γ 4 DIMSi + γ 5 NSMUN i + γ 6 NSMUN i × RANK i )
Again we look separately at the periods before and after publication and compare the results.
The estimation results are shown in Table 4. The table shows marginal effects. 12 For instance,
model 6 obtains that an increase of 100 places in the ranking increases the probability that a
school will close by 3 percentage points.
Table 4: Estimation results of the probit model for the probability of closing - Marginal Effects
Dependent variable: CLOSE, 1998-2002
1
RANK
2
3
4
Prob
-0.00003
0.18
0.0004
0.14
Prob
-0.003
0.69
-0.009
0.34
0.05
0.24
Prob
-0.0002**
0.04
-0.0001**
0.04
Prob
0.0005***
0.00
0.004
0.13
PUB
DIMS
NSMUN
NSMUN×RANK
Prob
5
6
7
8
0.0001***
0.004
0.0003***
0.01
0.10
0.21
0.01
0.60
0.01
0.52
0.09
0.14
0.11*
0.07
-0.0001**
0.04
-0.0001**
0.04
-0.0002**
0.01
-0.0002***
0.01
-0.0002***
0.01
-0.0002***
0.004
0.0004**
0.01
0.004*
0.10
0.001***
0.00
-0.000***
0.001
0.001***
0.00
-0.000***
0.001
-0.0000
0.15
RANK×PUB
Dependent variable: CLOSE, 2001-2005
-0.0000
0.10
-0.0000*
0.06
-0.0000**
0.04
Prob
-0.0000
0.74
0.0004
0.14
0.00007
0.22
0.0002**
0.04
Prob
0.0001
0.15
0.0005*
0.08
0.0003**
0.03
0.0004***
0.002
RANK×PRIV
McFadden R2
0.39
0.46
0.43
0.52
0.33
0.35
0.35
0.38
# Observations
569
569
569
569
583
583
583
583
Notes: the p-values correspond to the estimated coefficients of the Probit regression.
*** significant at the 1 percent level, ** significant at the 5 percent level, * significant at the 10 percent level.
11
The variable CHOICE does not have enough variability in the sample to allow its inclusion in this
regression. Thus, in this analysis we consider only the variable NSMUN as a measure of competition.
12
These are marginal effects averaged across schools.
12
For both periods and all models, the size of the school has a significant negative impact on the
probability of closing. The number of schools available in the region also has a significant
impact in almost all the models: the larger the number of alternatives, the greater the
probability of closing.
For the period before publication, the effect of the ranking is not significant in models 1 and 2.
If we distinguish between private and public schools, in models 3 and 4, the effect of the rank is
not significantly different between them. In models 2 and 4 we include the interaction between
NSMUN and RANK to capture the possibility that the impact of the ranking is amplified when
there are alternative schools in the municipality. However, this interaction is not significant.
For the period after publication we estimated the same models. We obtain a positive and
significant effect of the rank in models 5 and 6. In models 7 and 8 we allow for the impact of
the rank to be different for private and public schools and find, in model 8, that the impact is
significantly stronger for private schools. Moreover, the effect of the rank is amplified by the
existence of alternatives, as shown in models 6 and 8. As in the previous analysis, our results
show that the publication of the rank had a strong impact.
We estimated both the linear regressions and the probit models considering the regional
rankings described at the end of Section 3 and obtained results similar to those obtained with
the national rankings. 13 This corroborates our previous findings.
5. Discussion and Conclusions
Our findings suggest that the publication of rankings has strong effects upon families and
schools. Students react by moving in and out of the schools, according to their published rank.
This result is in line with the empirical literature about the effects of low-stakes accountability
policies on the performance of schools measured by scores on national exams. This result is
especially interesting given that, at least for public schools, there are typically legal restrictions
on enrollment decisions according to the geographic location of the student’s home. Even so,
students exert some choice over which school to enroll in.
We find that after the rankings publication, badly classified schools lose students and their
probability of closing increases. We also find that the effect for private schools is stronger as
would be expected, given that the freedom of choice between private schools is greater than
between public ones.
As several authors have already stressed, school quality measures such as the ones studied in
this paper may be very imprecise measures of school quality. There are several reasons for
these weaknesses: (i) rankings of schools are too volatile and imprecise (Kane and Staiger
(2002), Chay et al. (2005)); (ii) rankings reflect factors that are outside the control of schools,
especially the socioeconomic background of students’ families, and not the better quality of
well performing schools (Mizala et al. (2007)); (iii) public disclosure of schools‘ results biases
13
This may be justified by the small size of Portugal.
13
schools’ and teachers’ efforts toward an excessive investment on test-specific skills, easily
observable, at the cost of other school outputs, not so easily measurable (Jacob (2005), Reback
(2009), Neal and Schanzenbach, (2010)); (iv) accountability systems can be “cheated on” in
several ways, so that any reported beneficial effects of accountability may be merely the result
of schools gaming or cheating on the system, with no real content as far as effective quality
evaluation is concerned (Cullen and Reback (2006), Figlio and Getzler (2002)).
Moreover, the public disclosure of bad outcomes might have perverse consequences over
equity due to cream-skimming of both students and teachers, or to motivational effects
(Clotfelter et al. (2004), Figlio and Lucas (2004), Schutz et al.(2007)). The position emphasizing
the possible negative effects of the publication of school rankings is shared by politicians in
many countries, and calls for the prohibition of public disclosure of school rankings that is seen
in Spain and some other European countries (see Anghel et al., 2012).
Our point here is not to state that the rankings are a good instrument of policy, but rather to
analyze whether their publication has consequences over the education system. Our conclusion
is that it has, and that the effects are strong. Kane and Staiger (2002) regret that, for the US,
“school accountability systems have led to little reallocation of students across schools” as they
believe this would be an important mechanism to improve school quality. We show that in
Portugal people have reacted to school rankings by moving to better schools.
Our results reinforce the need to verify the type and quality of the information implicit in those
rankings. Also, as mentioned above, the disclosure of this information is independent from any
government intervention. Given that, as illustrated by Mizala et al. (2007), it may be very hard
to produce a ranking of schools that is not too volatile and simultaneously does not just reflect
students’ socioeconomic status, the issue of the existence of potential benefits stemming from
the public disclosure of complementary information gains relevance.
References
Anghel, Brindusa, Antonio Cabrales, Jorge Sainz, and Ismael Sanz, 2012, “Publicizing the results
of standardized external tests: Does it have an effect on school outcomes?”, Working-Paper 1234, Economic Series, Universidad Carlos III de Madrid.
Chakrabarti Rajashri, 2008,” Impact of Voucher Design on Public School Performance: Evidence
from Florida and Milwaukee Voucher Programs”, Federal Reserve Bank of New York Staff
Report no. 315.
Chay, Kenneth, Patrick MacEwan, and Miguel Urquiola, 2005, ''The Central Role of Noise in
Evaluating Interventions that Use Test Scores to Rank Schools'', The American Economic Review,
95 (4): 1237-1258.
Chiang, Hanley, 2009, “How accountability pressure on failing schools affects student
achievement”, Journal of Public Economics, 93, 1045-1057.
14
Clotfelter, Charles T. Helen F. Ladd , Cob L. Vigdor, and Aliaga Diaz, 2004, “Do School
Accountability systems Make It More Difficult for Low-Performing Schools to Attract and Retain
High-Quality Teachers?” Journal of Policy Analysis and Management, Vol. 23, ( 2), 251–271.
Cullen, Julie Berry and Randall Reback, 2006, “Tinkering toward accolades: school gaming under
a performance accountability system” in T. Gronberg and D. Jansen, eds., Improving School
Accountability: Check-Ups or Choice, Advances in Applied Microeconomics 14 (Amsterdam:
Elsevier Science): 1-34.
Elstad, Eyvind, 2009, "Schools that are Named, Shamed and Blamed by the Media: School
Accountability in Norway", Educational Assessment, Evaluation and Accountability, 21(2): 173189.
Figlio, David N. and Lawrence S. Getzler, 2002, “Accountability, ability and disability: gaming the
system”,
National
Bureau
of
Economic
Research
Working
Paper
9307,
at http://www.nber.org/papers/w9307 .
Figlio, David N. and M. Lucas, 2004, “What’s in a grade? School Report Cards and the Housing
Market,” American Economic Review, 94 (3): 591-604.
Figlio, David N. and C. Rouse, 2006, “Do accountability and voucher threats improve lowperforming schools?”, Journal of Public Economics, 90.
Hanushek, Eric and M. Raymond, 2005, “Does school accountability lead to improved student
performance”, Journal of Policy Analysis and Management, 24, (2): 297-325.
Jacob, Brian, 2005, “Accountability, Incentives and behavior: the impact of high-stakes testing
in the Chicago Public Schools”, Journal of Public Economics, 89: 761-896.
Kane, J. Thomas and Douglas O. Staiger, 2002, “The Promise and Pitfalls of Using Imprecise
School Accountability”, Journal of Economic Perspectives, 16 (4), pages 91-114.
Koning, Pierre and Karen van der Wiel, 2012, “School Responsiveness to Quality Rankings: An
Empirical Analysis of Secondary Education in the Netherlands”, De Economist, vol 160 (4), pages
339-355.
Mizala, Alejandra, Pilar Romaguera, and Miguel Urquiola, 2007, “Socioeconomic status or noise?
Tradeoffs in the generation of school quality information”, Journal of Development Economics,
84, pages 61-75.
Neal, D. and D. Schanzenbach, 2010, “Left behind by design: proficiency counts and test-based
accountability”, The Review of Economics and Statistics, 92 (2): 263-283.
Reback, R., 2008, Teaching to the rating: School accountability and the distribution of student
achievement, Journal of Public Economics, 92: 1394-1415.
Schutz, Gabriela, Martin R. West, and Ludger Wobmann, 2007, “School Accountability,
Autonomy, Choice, and the Equity of Student Achievement: International Evidence from PISA
2003”, OECD Education Working Papers, No. 14OECD.
West, Martin R. and Paul E. Peterson, 2006, “The Efficacy of Choice Threats
Within School Accountability Systems”, The Economic Journal, 116, ( 510): C46-C62.
15