My Career Plan

Abstract. This paper presents an update of the ranking of economics journals by the invariant method, as introduced by Palacio-Huerta and Volij, with a broader sample of journals. By comparison with the two other most prominent rankings, it also proposes a list of ‘target journals’, ranked according to their quality, as a standard for the field of economics. JEL classification: A12, A14. Keywords: Journal ranking; economics journals; business administration journals; finance journals, citations.

1. INTRODUCTION
The ranking of professional journals in economics has attracted growing ´ interest during the past decade (see Kalaitzidakis et al., 2003; Koczy and Strobel, 2007; Kodrzycki and Yu, 2006; Laband and Piette, 1994; Liebowitz and Palmer, 1984; Liner and Amin, 2006; Palacio-Huerta and Volij, 2004). Journal rankings have been used to evaluate the research performance of economics departments (e.g. Bommer and Ursprung, 1998; Combes and Linnemer, 2003; Lubrano et al., 2003) and of individual economists (e.g. ´ Coupe, 2003). They provide ‘objective’ information about the quality of publications in a world where academic publications have reached an overwhelming extent and variety.

While half a century ago a well-trained economist may have comprehended all key developments in economics at large, today it is difficult to follow even the pace of subfields. Thus, the judgment by an individual academic is accurate only in so far as it concerns her or his own field of specialization. Still, hiring, tenure, promotion and funding decisions should ideally be based on judgments of scientific quality, even when expertise about the specializations of all candidates is unavailable. For that reason economists have turned to journal rankings as a substitute for a direct judgment of scientific quality of individuals and institutions. r 2008 The Author ¨ Journal Compilation r Verein fur Socialpolitik and Blackwell Publishing Ltd. 2008, 9600 Garsington Road, Oxford OX4 2DQ, UK and 350 Main Street, Malden, MA 02148, USA.

Ranking of Journals This comes with virtues and vices. An advantage is certainly that something as elusive as ‘scientific quality’ is not left any more to hearsay and rumors. Rankings also constrain the verdicts by influential scientists, who are sometimes subject to perverse incentives. By this token they foster the development of a scientific standard and provide a rough indicator of scientific quality for politics, administration and the general public. As for the scientific community, rankings can correct misperceptions, both with respect to journal quality and the importance of fields. Narrow field definitions are sometimes used to insulate against judgments of scientific quality. By defining my field as ‘papers written by myself ’ I can ensure to be on top of my field. But rankings reveal how important my field is compared with others, provided the sample is large enough. An advantage of the ranking presented in this paper is that it covers a broad range of journals and, thereby, sheds light on how important the different fields are.

Most importantly, rankings provide objective information on journal quality. This puts into perspective judgments of journal quality that are governed by the abilities, preferences and publications of incumbents. Examples of such ratings abound. Take, for instance, the Journal Rating that the Vienna University of Economics and Business Administration1 had in place until the end of 2007. It puts Econometrica – the number 1 journal in the current ranking – into the same basket (called Aþ) as the Journal of Marketing Research (number 43 in the current ranking), Administrative Science Quarterly (number 73 in the current ranking), Regional Science and Urban Economics (number 84 in the current ranking) or Regional Studies (number 151 in the current ranking). And it puts the Journal of Economic Theory – a top-ten journal in all of the three objective rankings used here – into the same basket (called A) as the local Austrian periodical Empirica. According to hearsay this list was put together by asking incumbent personnel for opinions.

Of course, there are better lists than this one. But not even the list published by the Kiel Institute for the World Economy2 is free of obscurities: They list the Journal of Economic Theory in rank C together with journals like the Energy Journal (number 97 in the present ranking), the Journal of Regulatory Economics (number 108 in the current ranking) or the Southern Economic Journal (number 173 in the present ranking). This is probably an effect of double-counting, as this list was put together by averaging across the rankings by Kodrzycki and Yu (2006) for which the policy ranking is a subset of the social science ranking. The Tinbergen list3 makes more sense. But it still puts the International Economic Review (number 5 in the present ranking) and the Journal of Monetary Economics (number 7 in the current ranking) into the same bracket (A) as the Journal of Environmental Economics and Management (number 54 in the present ranking) and the Journal of Urban Economics (number 64 in 1. 2. 3. See http://bach.wu-wien.ac.at/bachapp/cgi-bin/fides/fides.aspx?journal=true See www.ifw-kiel.de/research/internal-journal-ranking/?searchterm=Journal See http://www.tinbergen.nl/research/ranking2.html

K. Ritzberger the current ranking); and it puts Economic Theory (number 23 in the present ranking), the Journal of Financial and Quantitative Analysis (number 31) and the Journal of Economic Dynamics and Control (number 35) into the same basket (B) as the International Journal of Industrial Organization (number 90), the Journal of Evolutionary Economics (number 114) and the Economics of Education Review (number 120).4 Such judgments may reflect subjective opinions or policy goals. But those should be made explicit and contrasted with objective data. On the other hand, ‘objective’ rankings are no substitute for reading the papers. Given the high variance of quality within any given journal, where a paper gets published is a very imperfect proxy for its quality. Moreover, many of the decisions that are aided by rankings need to take into account other dimensions than where an author has published. How a candidate’s specialization fits into a department and the effect on the age structure are at least as important considerations for hiring and promotion decisions. Furthermore, it may often be preferable to hire a candidate who is willing and able to take on hard challenges instead of one who rides on a fashion wave, even though the latter may have a better publication record.

Likewise, funding decisions should be guided by a vision about scientific development, rather than by past successes. On none of these considerations do rankings provide a clue. Rankings are based on the idea that one paper quotes another, because the former uses a result obtained in the latter. Therefore, citation analysis should provide an ‘objective’ image of quality. This is not always the case, however, for the following (at least) ten reasons. First, the most important contributions are often not quoted, but used without reference: few papers that use Nash equilibrium cite Nash (1950), among the many papers on continuum economies, a minority quotes Aumann (1964), and almost nobody acknowledges Hurwicz (1973) when working on mechanism design. Second, and related, the papers that get quoted most frequently are often not the ones that contain the deepest results.

Deep results are often hard to understand and, therefore, do not attract a large readership. Hence, even though they may ultimately be most important for scientific progress, they do not collect many citations. Third, new developments in sciences often appear in new journals. But for a new journal to be included in the citation index takes ages and is subject to political manipulation. A prime example is the Journal of the European Economic Association, which is still not included in the SSCI, even though it has certainly published high-quality papers ever since its inception. Fourth, some of the journals that fare very well in the rankings only do so because a small handful of articles from these journals get quoted excessively and the others not at all. The average paper from such a journal may in fact be quite 4. How this list was compiled is not quite clear. On the webpage it is claimed that ‘important inputs’ were Kalaitzidakis et al. (2003) and Kodrzycki and Yu (2006). The webpage remains silent on how these inputs were combined. r 2008 The Author ¨ Journal Compilation r Verein fur Socialpolitik and Blackwell Publishing Ltd. 2008

Ranking of Journals bad quality, but the few seminal contributions secure a high ranking. That is, journal rankings give no information about the variance among individual contributions. Furthermore, there are several aspects of insider–outsider problems that affect rankings. Fifth, there clearly exist citation and refereeing cartels (see Pieters and Baumgartner, 2002), often supported by editorial board representation, that are upheld intentionally to defend the research agenda of insiders and restrain outsiders.5 Sixth, the peer review system of journals is biased against authors that are not affiliated with top universities or are employed at non-academic institutions (see Blank, 1991). Seventh, editors and influential scientists actively place publications of their students in top journals, often irrespective of quality, to improve the job market prospects of their teaching output. Hence, quotations sometimes reflect placement policies more than quality, and contain more references to the advisors than to seminal contributions.

Eighth, and related, many of the papers in good journals are minor variations of known results. This is due to the peer review system, where manuscripts are often refereed by the authors of preceding work. The latter, of course, have a vested interest in follow-ups that appear in good journals, because this increases their visibility. Innovative ideas, on the other hand, are often met with reluctance, because the referees have a hard time to digest the ideas. Ninth, successful journals sometimes get ‘highjacked’ by special interest groups that make them their realm through representation on editorial boards and reject any contribution from outside. Tenth, most journal rankings are manipulable by editors and publishers. This may distort editorial policy against pure quality and bias the rankings. For instance, if a society runs a regular submission journal, it can improve its ranking by also running a few other journals that only publish solicited papers, and make sure that solicited papers quote predominantly papers from the regular submission journal

All they have to do is to ensure that journals belonging to the cartel get quoted more often than outsiders, or that outside journals do not get quoted too often (but instead working paper versions get quoted, for instance). Having listed all these shortcomings, there remains the lack of an alternative. The field of economics has grown too large and diverse for any committee to judge scientific quality of individuals or institutions. Thus, rankings are there to stay, their numerous problems notwithstanding. The task, therefore, becomes to improve their quality. The remainder of the paper is organized as follows. Section 2 describes the ranking methods that have been proposed in the literature, inclusive of the one used here. Section 3 presents the data. Section 4 discusses the results of the present ranking and their robustness. Section 5 puts together the current with two of the most prominent prior rankings to obtain a qualitative list of recommended journals. Section 6 concludes. 5. ¨ Colin Camerer’s rejection of the critique of neuroeconomics by Faruk Gul and Wolfgang Pesendorfer for the Journal of Economic Literature has become a famous example.

2. RANKING METHODS
Many diverse ranking methods have been proposed, but no single method is considered authoritative. The most popular one is the impact factor (Garfield, 1955), the ratio of the number of citations of a given journal to the number of articles published in this journal (for a fixed period). This indicator depends on field size, citation intensity and turnover rate ( Jemec, 2001). It is thus biased in favor of certain journals and fields and does not take into account that citations from a more important journal count more than citations from a less important one. Most of this criticism also holds for various modifications of the impact factor (see Hirst, 1978; Lindsey, 1978; Sombatsompop et al., 2004). The share of uncited papers (Koenig, 1982) is likely to be close to zero for most journals and allows little differentiation at the top. The H-index (Hirsch, 2004) was developed to rank individual scientists, but has been adapted to rank journals (Braun et al., 2005). It is the largest integer n such that the journal has n papers with n citations each (exclusive of self-citations).

This indicator is vulnerable to size. The BT-method (Bradley and Terry, 1952), as applied by Stiegler et al. (1995), is a logit-type model that is used to estimate the odds ratio that one journal will cite another. It suffers from a lack of fit and becomes quickly uninformative (see Liner and Amin, 2006). The LP-method (Liebowitz and Palmer, 1984), in contrast to the aforementioned, takes into account that journals ought to be weighted differently according to their importance. Thus, less established journals will carry a lower weight, so that it makes little difference whether or not they are included. This makes the LP-method robust to field size.

If the entries cij of the J Â J matrix C 5 [cij] represent the number of citations to journal i by journal j (for i, j 5 1, . . ., J ), and the diagonal entries ai of the J Â J diagonal matrix A 5 [ai] record the number of articles published by journal i (in the relevant period), the LP-method computes the weights vector v 5 [vi] of journals as the solution to the equation system where e 5 [1 . . . 1] denotes the summation (row) vector. This method has also been used by Kalaitzidakis et al. (2003, henceforth KMS), Kodrzycki and Yu (2006) and Laband and Piette (1994). But this assignment of weights is vulnerable to citation intensity, i.e. to the number of citations per article. ( Journals that, say, publish only surveys, without contributing to scientific progress, will have a high citation intensity.) ´ The tournament method (Koczy and Strobel, 2007) ranks journals according to their score ti given by Ranking of Journals This method is invariant to journal size, journal or article splitting, and it is not manipulable: the rank of a journal cannot be increased by making additional cites.

It does not take into account, however, that beating an important journal in pairwise comparison ought to be worth more than winning against an unimportant journal. Palacio-Huerta and Volij (2004, henceforth PV) have proposed a method that is characterized by five plausible axioms. A1. Anonymity: The ranking does not depend on the names of the journals. A2. Invariance to citation intensity: Ceteris paribus the ranking is not affected by the length of the reference section of the papers published in a journal. A3. Weak homogeneity: The relative ranking of any two journals is a function of their mutual citations. A4. Weak consistency: The ranking method is ‘consistent’ when applied to problems involving different numbers of journals. A5. Invariance to splitting of journals: If a journal is subdivided into two identical subjournals in terms of their citations, each of the two receives half the original weight of the mother journal, while the valuations of the other journals are unaffected. This invariant method results in the valuation vector that is the unique solution6 to the system of equations where diag w denotes the operation of writing a vector w as a diagonal matrix.

Note that Av is the right-hand eigenvector of the stochastic matrix C (diag eC)À1 that belongs to the Frobenius root (which equals 1). Therefore, one is free to choose a normalization. Here, the normalization is to assign 100% to the top journal. Thus, the ‘value’ of a journal is to be interpreted as the ratio of the number of impact-weighted citations received by that journal to those obtained by the best journal in the sample. The invariant method is also used by Kodrzycki and Yu (2006) for their perarticle valuation within the economics discipline and the social sciences at large. The algorithm used by Google to rank search hits on the internet (Brin and Page, 1998) is also a variant of this method. The invariant method works well for closely knit fields, but is problematic when the matrix C becomes reducible (see Serrano, 2004), that is, when C can be put into block upper-triangular form by permutations of rows and columns. In such a case there are subfields between which the citation flows are unidirectional; then the solution to (3) ceases to be unique and numerical results may be quite obscure. The simplest instance of that would occur if selfcitations were included and a journal only quotes itself and is never quoted by any other journal; in that case this journal can be assigned an arbitrary 6. More precisely, the solution is unique if the problem is irreducible.

K. Ritzberger value without affecting the values of other journals. For the present computations self-citations are excluded, but picking a sample that is too large can still lead to a reducible matrix. For that reason some minor journals had to be excluded from the current ranking. This also represents a general caveat to rankings for large samples. Sensible results can only be expected if the citation flows between the journals in the sample are sufficiently strong. That is, numerical results on fields that are connected too loosely will be quite arbitrary.

3. DATA
The current paper applies the invariant method to a larger sample and a more recent time period than PV or KMS. PV rank 37 journals based on the period 1993–99, and KMS rank 159 journals for the period 1994–98. The present paper considers 261 journals for the three years 2003–05.7 On the other hand, this study excludes some journals. Some minor journals are excluded because of a lack of citations and/or missing data on the number of articles. Journals that have only self-citations are also excluded, because the invariant method is vulnerable to reducibility. Some of the more important journals are excluded, because they either state on their webpages that they solicit papers rather than taking submissions ( Journal of Economic Literature and Journal of Economic Perspectives), because they are volumes rather than journals (NBER Macroeconomic Annuals) or because they are pure conference volumes (Brookings Papers on Economic Activity). This is done to enable a fair application of the ranking, because otherwise researchers, who do not have access to those publications’ authorship pool, would be at a disadvantage.