University of Wollongong Economics Working Paper Series 2006

Transcription

1 Unversty of Wollongong Economcs Workng Paper Seres Modellng the Research Output of Australan Unverstes by Dscplne Abbas Valadkhan and Smon Vlle WP November 2006

2 Modellng the Research Output of Australan Unverstes by Dscplne Abbas Valadkhan * and Smon Vlle School of Economcs and Informaton Systems, Unversty of Wollongong, NSW 2522, Australa Abstract: Ths paper develops and estmates a cross-sectonal model for forecastng research output across the Australan unversty system. It bulds upon an exstng lterature that focuses ether on nsttutonal comparsons or studes of specfc subjects, by provdng dscplne-specfc results across all of the ten major dscplnary areas as defned by Australa s Department of Educaton, Scence and Tranng (DEST). The model draws upon four dscplne-specfc explanatory varables; staff sze, research expendture, PhD completons, and student-staff ratos to predct output of refereed artcles. When compared wth actual averaged output for , the results are hghly statstcally sgnfcant. I. Introducton There s a growng focus n the Australan unversty system on quanttatve research performance assessment. However, to date ths has manly concerned performance assessment n aggregate and ths s nconsstent wth the most recent polcy emphass on unversty dversty (Abbott & Doucoulagos 2004; Valadkhan & Worthngton, 2006). Put bluntly, focusng on research performance at the nsttutonal level gnores the vared performance that occurs at the dscplnary level, and the applcaton of fundng on ths bass serves to stfle nnovaton n key research areas and mantan underperformng and outdated research areas. Ths approach serves as a dsncentve to focused, responsve, nnovatve and dverse research n Australan unverstes. Where specfc dscplnes, such as economcs, have been analysed, ths has tended to be on an ndvdual rather than comparatve bass (Men 2002; Pomfret & Wang 2003; Ner & Rodgers 2006; Johnes 1995). It s nterestng to note that data on the number of publshed refereed artcles by academc staff members afflated to Australan unverstes have been reported and analysed only at the nsttutonal (aggregate) level. We contrbute to the debate on * Correspondng author. E-mal: abbas@uow.edu.au 1

3 research performance by provdng the dscplne-specfc estmates of the annual average number of refereed journal artcles (referred to as research output hereafter) durng the perod The major objectve of ths study s to specfy and estmate a cross-sectonal model for the Australan unversty research output usng all avalable dscplne-specfc data durng the perod The rest of the paper s organsed as follows. In Secton II we specfy a model to explan the number of publshed refereed artcles by unversty and dscplne. Secton III dscusses the source, descrpton and type of data employed n ths forecastng exercse. Secton IV presents and analyses the emprcal results of the study, and Secton V offers some concludng remarks. II. Theoretcal Framework The research output (Y) n ths paper has been proxed by the number of artcles publshed n natonal and nternatonal refereed journals by academc staff members afflated to Australan unverstes. The research-output determnaton model s specfed as follows: Ln(Y )= β Ln( S ) + β Ln( RE ) + β Ln( SSR ) + β Ln( PhD ) + ε (1) where Y =research output proxed by the annual average number of refereed journal artcles publshed by unversty, S =the annual average number of research only and teachng and research academc staff members (full-tme equvalent) n unversty, RE = the annual average expendture on research and expermental development n unversty, SSR = the annual average full-year student-staff rato n unversty, PhD = the annual average number of PhD completons n unversty, ε =homoscedastc resdual term, =1,2, n=37, and n denotes the number of Australan unverstes. It s hypotheszed that as the number of academc-staff members (S), whose job descrpton requres undertakng research, ncreases, the magntude of research output rses due to the sze factor. Ths means that the expected sgn for the sze factor s postve β 1 > 0. It s also assumed that, ceters parbus, the avalablty of more research fundng and PhD students can boost the research output, suggestng that both β2 and β 4 > 0. However, an ncrease n teachng and admnstraton workload n a partcular unversty (proxed by rsng student-staff ratos) can curtal the research output. It s thus expected that β 3 < 0. It s postulated that f all the four 2

4 explanatory varables n equaton (1) are equal to zero (partcularly the number of staff members), the research output wll be equal to zero. Based on ths argument, we have adopted a regressonthrough-the-orgn model n ths paper and as a result the ntercept has been removed from equaton (1). In order to dentfy any possble outlers or abnormal observatons we wll compute ˆ σ = ( Y Yˆ ) Y. If ˆ σ < 0.30, we keep the th observaton n the estmaton procedure, otherwse t wll be excluded. Fnally one can substtute the dscplne-specfc values of the four explanatory varables (rather than the aggregate fgures as dscussed above) nto the fnal estmated equaton (1) to obtan the dscplne-specfc values of research output n the followng manner: ˆ ˆ ˆ ˆ ˆ β1 β2 β3 β4 j j j j j Y =S RE SSR PhD (2) where: Y ˆj s the estmated value of research output produced by the j th dscplne n the th unversty usng the estmated coeffcents of equaton (1). In order to ensure that the research outputs by varous dscplnes n a partcular unversty add up to the actual aggregate research output for the concerned unversty, n equaton (3) t s assumed that ( m= 10 ˆ ε ˆ ) Y Y j= 1 j = or the dfference between the actual total research output and the estmated sum of the research output by varous dscplnes n unversty, wll be proportonally dstrbuted (adjusted) across varous dscplnes wthn each unversty. That s: ˆ ˆ Y Y Y ˆ Yˆ (3) j j = j + ε m= 10 j= 1 j Where Y j s the forecasted (and adjusted) value of research output produced by the j th dscplne n the th unversty, m s equal to the number of dscplnes. III. The Database Before embarkng on our emprcal quest, t s mportant to look at the sources and defntons of the data employed n ths analyss. Thrty-seven Australan unverstes have been ncluded n the analyss, all of whch are publcly funded and members of the Australan Vce-Chancellor s Commttee (AVCC). The total number of refereed artcles publshed wthn each of these 3

5 unverstes has been obtaned from a report enttled Hgher Educaton Research Data Collecton Tme Seres Data publshed by The Australan Vce-Chancellors Commttee (AVCC, 2006). An unpublshed database used n ths study was purchased from the Department of Educaton, Scence and Tranng (DEST) n December 2005 (see below for more detals). The data ncludes the number of PhD completons (the DEST source reference number OZUP ) as well as the number of academc staff members (the DEST source reference number: Staf2001.dat - Staf2004.dat) by nsttuton and across 10 consstently defned broad felds of educaton, all of whch we have averaged usng avalable annual observatons wthn the perod These 10 broad felds of educaton (whch are also referred to as dscplnes nterchangeably) are shown n Tables 2 and 3. In order to mnmse bas n our results, we consder only those academc staff members who are classfed as undertakng research-only and teachng-and-research actvtes. In other words, the varable that s referred to as academc staff (S) does not nclude teachng only staff. The next varable RE j or the annual average expendture on research and expermental development, also avalable by unversty and the same dscplnes, was averaged n the same way usng all avalable data durng the perod ($A'000). Ths varables ncludes: (1) Natonal Compettve Research Grants (.e. Commonwealth Schemes and Non-Commonwealth Schemes); (2) State and Local Government; (3) Other Commonwealth Government; (4) Other Australan Sources (.e. Busness Enterprses; General Unversty Funds; and Other); and (5) Overseas sources. The last varable employed n ths paper s SSR j or the average full-year student-staff rato (all students) whch s also avalable by nsttuton and the same 10 consstently defned broad felds of educaton for the perod Smlarly we averaged all avalable observatons durng ths perod to avod any possble abnormal observaton for a partcular dscplne wthn any unversty. Both RE j and SSR j are also avalable from the DEST webste. The full database employed n ths paper has been ncluded n Appendx. IV. Emprcal Results and Polcy Implcatons The estmaton procedure nvolved the followng three steps. Frst, the OLS method was used to estmate equaton (1) usng all 37 cross-sectonal (unversty level) data. We looked at the resultng resduals and f ˆ σ < 0.30, we ncluded the th unversty n our sample. Based on ths crteron, we excluded the followng 5 unverstes: Charles Sturt, RMIT, Southern Queensland, 4

6 Sunshne Coast and Swnburne. The relatonshp between the research output and ts four major determnants (as specfed n equaton 1) for these 5 unverstes was very dfferent from the other 32 unverstes ( 0.30 < ˆ < 0.58 ). Thrd, we then used the aggregate unversty-level data (32 σ unverstes sorted n alphabetcal order) to estmate equaton (1) and the results are presented n Table 1. Table 1. The estmated equaton for Australa unversty research output, Varable or test statstcs Coeffcent t-rato P-value Ln( S ) Ln( RE ) Ln( SSR ) Ln( PhD ) R R Jarque-Bera statstcs χ 2 = Whte Heteroskedastcty Test: wthout cross terms F(8,23)= wth cross terms F(14,17)= Ramsey RESET Test F(1,27)= Chow Breakpont Test: 16 th observaton F(4,24)= Chow Forecast Test: Forecast from 23 to 32 Out of sample forecast perod usng the last 10 observatons F(10,18)= Thel Inequalty Coeffcent=0.065 Bas Proporton=0.046 Varance Proporton=0.44 Covarance Proporton=0.51 Ln(Y ) 5

7 Fg.1. Graphcal tests for parameter constancy CUSUM 5% Sgnfcance CUSUM of Squares 5% Sgnfcance One-Step Probablty Recursve Resduals N-Step Probablty Recursve Resduals Recursve C(1) Estmates ± 2 S.E. Recursve C(2) Estmates ± 2 S.E Out of sample forecast Recursve C(3) Estmates ± 2 S.E. Recursve C(4) Estmates ± 2 S.E. Y YF As can be seen, the estmated equaton performs very well n terms of goodness-of- 2 ft R = 0.981, each and every coeffcent beng statstcally sgnfcant (at the 1 per cent level or better), and havng the expected theoretcal sgns. In other words, ceters parbus, f the number of academc staff members ncreased, say by 10 per cent, on average ths would led to a rse of 5.4 per cent n the number of refereed journal artcles n these 32 unverstes. On the other hand, a smlar 10 per cent rse n the expendture on research and expermental development and the number of PhD completons would have resulted n a 1.8 and 3.1 per cent rse n the research output, respectvely. Consstent wth theoretcal expectatons, t s also found that an ncrease n the student-staff rato (by say 10 per cent) leads to a fall of 3.9 per cent n the research output. 6

8 The equaton passes successfully all of the reported dagnostc tests: the Jarque-Bera normalty test, the Whte heteroskedastcty test (wth or wthout cross terms), the Ramsey RESET specfcaton test, the Chow breakpont test (splttng the sample n the mddle,.e. the 12 th observaton), and the Chow forecast test (usng the last 10 observatons for out of sample forecasts). The Thel nequalty coeffcent for the out-of-sample forecast s also as low as These results clearly show the ablty of our model to forecast beyond ts estmaton sample. One problem assocated wth analyss of ths knd s non-constancy of estmated coeffcents whch can create economc and econometrc complcatons n dervng any nference from the emprcal model. Gven dfferences among the 32 unverstes n terms of ther sze, portfolos and research actvtes, parameter constancy s pvotal n modellng the determnants of research output. Therefore, the estmated model n Table 1 has been evaluated by a number of recursve dagnostc tests, whch are dsplayed n Fgure 1 n the followng order: a b c d e f g h where panel (a) dsplays the CUSUM test; panel (b) shows the CUSUM of squares; panels (c) and (d) depct the recursve resduals and the assocated one-step and n-step probabltes, respectvely; panels (e) to (h) show the recursvely estmated 4 coeffcents usng observatons 9-32 n the same order that these coeffcents appear n Table 1 (from top to bottom); and fnally panel () presents the actual research output as well as the forecasted values, Y ˆ, of both n-sample (the frst 22 unverstes) and out-of-sample (the last 10 unverstes). These evaluatve tests are useful n assessng the parameter constancy of the model, as recursve algorthms avod arbtrary splttng of the sample. Overall, the graphcal tests reported n Fgure 1 pont to the n-sample and out-of-sample constancy of the estmated coeffcents. It should be noted that all observatons (unverstes) n our samples are sorted n alphabetcal order. Gven the fact that dscplne-specfc average values of the four explanatory varables n equaton (2) can be obtaned durng the perod , we can now predct the number of refereed artcles by dscplne ory ˆj. The forecasts are presented n Table 2. As explaned n Secton II, we also calculated the dfference between the total actual research output and the total predcted research output for each unversty usng( ) m= 10 Y ˆ Y. In order to mantan the j= 1 j equalty between the total research output and the sum of the dscplne-specfc research outputs 7

9 wthn each unversty, equaton (3) s used to proportonally dstrbute the dfference across the 10 dscplnes to obtany j. The results are then presented n Table 3 n whchy m= 10 = Y j= 1 j. To ease the comparsons and valdate our results, the values ofy, Y ˆ and m = 10 j = 1 Yˆ j and the resultng devatons among them are also presented n Table 4. As can be seen, all the reported devatons, usng both aggregate-unversty data and dscplne-level data, are less than 0.30 for each and every unversty. In fact, the average absolute values of the above devatons,.e. Y, Y ˆ, are ˆ σ = 0.11and ˆ σ = 0.09 d1, respectvely. Both Y ˆ and that f we had presentedy, Y ˆ and m = 10 j = 1 j Yˆ m= 10 j= 1 j track the actual data ory so well Yˆ n one-sngle graph, they would have become almost ndstngushable, the correspondng R 2 between Y Yˆ andy ˆ m = 10 Y beng and 0.982, j = 1 j respectvely. Both m = 10 j = 1 Yˆ j andy ˆ are also very hghly correlated R 2 =0.998 (see also ˆ σ = 0.08 n d 2 Table 4). We also argue that the dfferences among unverstes can be as substantal as the dfferences among varous dscplnes wthn the same unverstes. Therefore, f the relatonshp between research output and ts four major determnants works so well among so many dfferent unverstes, the same average relaton can probably be applcable n the predcton of research output across dfference dscplnes wthn each unversty. Fnally, the fnal dscplne-specfc number of refereed artcles n Table 3 can be used to dentfy the key research areas across Australan unverstes. V. Concluson Our paper provdes a vtal brdge n the exstng lterature between those studes comparng nsttutons n the broad and those focused upon specfc subject areas. Our results, whch are hghly robust, provde evdence of recent dscplne level output across most Australan unverstes, enablng nsttutons to analyse the performance of each subject area both across that unversty and wth cognate dscplnes n other unverstes. Besdes ts dagnostc value, t also provdes an accurate forecastng tool, ndcatng the relatve mportance of dfferent explanatory varables n plannng future output, of whch staff numbers and student-staff ratos are the most potent. 8

10 References Australan Vce-Chancellors Commttee, AVCC (2006) Hgher Educaton Research Data Collecton Tme Seres Data , (Accessed 3 November 2006). Abbott, M. and Doucoulagos, C. (2004) Research output of Australan unverstes, Educaton Economcs, 12, Johnes, G. (1995) Scale and techncal effcency n the producton of economc research: data envelopment analyss of the 1992 research assessment exercse of Brtsh unversty departments of economcs, Appled Economcs Letters, 2, Men, D. G. (2002) Research productvty and career status, Appled Economc Letters, 9, Pomfret, R. and Wang, L.C. (2003) Evaluatng the research output of Australan unverstes, Australan Economc Papers, 42, Ner, F. and Rodgers, J.R. (2006) Rankng Australan economcs departments by research productvty, Economc Record, 82, Valadkhan, A. and Worthngton, A. (2006) Rankng and clusterng Australan unversty research performance, , Journal of Hgher Educaton Polcy and Management, 28,

11 Table 2. The forecasted-annual number of refereed artcles publshed by nsttuton and dscplne (averaged ) Unversty Natural and Physcal Scences Informaton Technology Engneerng & Related Technologes Archtecture & Buldng Agrculture, Envronment al & Related Studes Health Educaton Manag. & Commerce Socety & Culture Creatve Arts m= 10 Yˆ = Y j = 1 Adelade ANU Australan Catholc Ballarat Canberra Central Qld Charles Darwn Curtn Deakn Edth Cowan Flnders Grffth James Cook La Trobe Macquare Melbourne Monash Murdoch New England Newcastle Queensland QUT South Australa Southern Cross Sydney Tasmana UNSW UTS Vctora Western Australa Western Sydney Wollongong Source: The Authors calculatons. Note: If the correspondng S j and/or SSR j are zero, Y j are also assumed to be zero. In almost all cases when S j was zero, SSR j was zero too, mplyng no staff, no students and consequently no publcaton. total Y total ˆ σ 10

12 Table 3. The forecasted and adjusted annual number of refereed artcles publshed by nsttuton and dscplne (averaged ) Unversty Natural and Physcal Scences Informaton Technology Engneerng & Related Technologes Archtectur e & Buldng Agrculture, Envronmenta l & Related Studes Health Educaton Manag. & Commerce Adelade ANU Australan Catholc Ballarat Canberra Central Qld Charles Darwn Curtn Deakn Edth Cowan Flnders Grffth James Cook La Trobe Macquare Melbourne Monash Murdoch New England Newcastle Queensland QUT South Australa Southern Cross Sydney Tasmana UNSW UTS Vctora Western Australa Western Sydney Wollongong Source: The Authors calculatons. Socety & Culture Creatve Arts Y total 11

13 Table 4. Actual research output and estmated research output (total and the sum of dscplnes) Unversty Y Actual Y ˆ Estmated usng aggregate data m= 10 Yˆ j = 1 j Estmated usng dscplne data ˆ σ = ( Y Yˆ ) Y m = 10 m = 10 ( Y ˆ Y ) Y ( ) j = 1 j j = 1 j ˆ σ d1 = ˆ σ d2 = Yˆ Yˆ Yˆ Adelade ANU Australan Catholc Ballarat Canberra Central Qld Charles Darwn Curtn Deakn Edth Cowan Flnders Grffth James Cook La Trobe Macquare Melbourne Monash Murdoch New England Newcastle Queensland QUT South Australa Southern Cross Sydney Tasmana UNSW UTS Vctora Western Australa Western Sydney Wollongong Mean ˆ σ = 0.10 ˆ σ = 0.10 ˆ σ d1 d 2 =

14 Table A1 : The average number of academc staff members (full-tme equvalent) by nsttuton and dscplne durng the perod (persons) Unversty Natural and Physcal Scences Informaton Technology Engneerng & Related Technologes Appendces: Database Archtecture & Buldng Agrculture, Envronmental & Related Studes Health Educaton Management & Commerce Adelade ANU Australan Catholc Ballarat Canberra Central Qld Charles Darwn Charles Sturt Curtn Deakn Edth Cowan Flnders Grffth James Cook La Trobe Macquare Melbourne Monash Murdoch New England Newcastle Queensland QUT RMIT South Australa Southern Cross Southern Qld Sunshne Coast Swnburne Sydney Tasmana UNSW UTS Vctora Western Australa Western Sydney Wollongong Source: The authors calculaton usng a database purchased from DEST n December 2005 (the DEST source reference number: Staf2001.dat - Staf2004.dat). Socety & Culture Creatve Arts Total 13

15 Table A2: Annual average Expendture on Research and Expermental Development durng the perod ($A'000) Unversty Natural and Physcal Scences Informaton Technology Engneerng & Related Technologes Archtecture & Buldng Agrculture, Envronmental & Related Studes Health Educaton Management & Commerce Adelade ANU Australan Catholc Ballarat Canberra Central Qld Charles Darwn Charles Sturt Curtn Deakn Edth Cowan Flnders Grffth James Cook La Trobe Macquare Melbourne Monash Murdoch New England Newcastle Queensland QUT RMIT South Australa Southern Cross Southern Qld Sunshne Coast Swnburne Sydney Tasmana UNSW UTS Vctora Western Australa Western Sydney Wollongong Source: The DEST webste. Socety & Culture Creatve Arts Total 14

16 Table A3: Average full-year student-teacher rato (all students) by nsttuton and dscplnes durng the perod Unversty Natural and Physcal Scences Informaton Technology Engneerng & Related Technologes Archtecture & Buldng Agrculture, Envronmental & Related Studes Health Educaton Management & Commerce Adelade ANU Australan Catholc Ballarat Canberra Central Qld Charles Darwn Charles Sturt Curtn Deakn Edth Cowan Flnders Grffth James Cook La Trobe Macquare Melbourne Monash Murdoch New England Newcastle Queensland QUT RMIT South Australa Southern Cross Southern Qld Sunshne Coast Swnburne Sydney Tasmana UNSW UTS Vctora Western Australa Western Sydney Wollongong Source: The DEST webste. Socety & Culture Creatve Arts Total 15

17 Table A4: Annual Average number of PhD Course Completons by Insttuton and dscplne durng the perod (persons) Unversty Natural and Physcal Scences Informaton Technology Engneerng & Related Technologes Archtecture & Buldng Agrculture, Envronmental & Related Studes Health Educaton Management & Commerce Adelade ANU Australan Catholc Ballarat Canberra Central Qld Charles Darwn Charles Sturt Curtn Deakn Edth Cowan Flnders Grffth James Cook La Trobe Macquare Melbourne Monash Murdoch New England Newcastle Queensland QUT RMIT South Australa Southern Cross Southern Qld Sunshne Coast Swnburne Sydney Tasmana UNSW UTS Vctora Western Australa Western Sydney Wollongong Source: The authors calculaton usng a database purchased from DEST n December 2005 (the DEST source reference number OZUP ). Socety & Culture Creatve Arts Total 16