REF Sub-panel 17: Meeting 2
10 - 12 February 2014
Ettington Chase, Stratford-upon-Avon
Minutes
Day 1
Present:
Neil Adger (Output assessor)
John Baines
Graeme Barker
Richard Black
Chris Clark
Kevin Edwards
Giles Foody
Ian Freestone
Lynne Frostick
Clive Gamble (Deputy sub-panel chair)
Roberta Gilchrist
Chris Gosden
Jeremy Hill
Andrew Jones
Glynis Jones
Jo Lakey (Sub-panel secretary)
Wendy Larner
Nina Laurie
Barbara Maher
Terry Marsden
David Mattingly
Joe Painter (Output assessor)
Chris Philo
Philip Rees
Keith Richards (Sub-panel chair)
Charlotte Roberts
Jennifer Robinson (Output assessor)
Jon Sadler (Output assessor)
David Thomas
Paul Valdes (Output assessor)
Charles Withers
Michael Wykes (Sub-panel adviser)
Page 1 of 7
1. Introduction and competence to do business
1.1 The chair welcomed the sub-panel and thanked them for taking part.
1.2 In the light of the attendance, the sub-panel confirmed its competency to do
business.
2. Minutes of previous meeting on 6th February 2013
2.1 No formal minutes had been produced from the previous meeting, which had
been concerned with the implications of the reported submission intentions.
3. Main Panel C Meeting 2, 23 January
3.1 The chair gave an update on the main panel meeting held in January.
3.2 Minutes from the meeting would be circulated to sub-panel members once
confirmed.
3.3 The sub-panel noted paper 2.2014.01 which contained a minute from the main
panel and was intended to be helpful for sub-panel members when considering criteria of
originality, significance and rigour.
3.4 The sub-panel was reminded that cross-referred outputs needed to be
considered under the criteria of the sub-panel to which they had been submitted.
3.5 The sub-panel was reminded of the need to have agreed scores for 50% of
outputs by the June meeting to meet main panel targets.
4. Conflicts of interest
4.1 The sub-panel reviewed the register of their declared major conflicts of interest
and confirmed they were correct. Individuals were asked to inform the executive group of
any updates to their conflicts of interest during the year.
5. An introduction to the REF IT systems
5.1 The sub-panel adviser presented a set of summary statistics on the sub-panel
submission and gave a brief presentation on the IT systems. The REF admin team were
available on-site to give individual assistance to sub-panel members.
6. Requirements for audit
6.1 The sub-panel noted paper 2.2014.03 which described audit processes.
6.2 The sub-panel were reminded to raise all audit queries through the secretariat.
7. Timetable of future meetings
Page 2 of 7
7.1 The sub-panel were informed of the agenda items which would be included at the
next two meetings and of the work that would need to be completed before each one.
8. Review of output allocations
8.1 The sub-panel discussed paper 2.2014.03.
8.2 The sub-panel agreed that two assessors would be allocated to each output, with
a view to enabling its examination in sufficient detail to reach a grade. Where necessary,
a third assessor could be identified if there was disagreement and moderation was
required.
8.3 The sub-panel split into three groups to re-allocate outputs where members felt
unable to assess an output, or where they had conflicts of interest. These two sessions
were also used to identify items for cross-referral and to allocate second readers.
8.4 During the group discussions, any member with a conflict of interest with the
institution being discussed left the room.
9. Review of Double Weighting cases
9.1 The sub-panel discussed paper 02.2014.04.
9.2 The sub-panel were reminded that the judgement on double-weighting was
separate from the judgement about an output’s quality.
9.3 The panel split into three groups, each looking at a third of the double-weighting
requests which had been submitted.
9.4 During the group discussions, any member with a conflict of interest with the
institution being discussed left the room.
Page 3 of 7
Day 2
Present:
Neil Adger (Output assessor)
John Baines
Graeme Barker
Bob Bewley (Impact assessor)
Richard Black
Chris Clark
Kevin Edwards
Giles Foody
Ian Freestone
Lynne Frostick
Clive Gamble (Deputy sub-panel chair)
Roberta Gilchrist
Chris Gosden
Jeremy Hill
Andrew Jones
Glynis Jones
Jo Lakey (Sub-panel secretary)
Wendy Larner
Nina Laurie
Barbara Maher
Terry Marsden
David Mattingly
Joe Painter (Output assessor)
Chris Philo
Philip Rees
Keith Richards (Sub-panel chair)
Charlotte Roberts
Jennifer Robinson (Output assessor)
Jon Sadler (Output assessor)
David Thomas
Paul Valdes (Output assessor)
Charles Withers
Michael Wykes (Sub-panel adviser)
9. Review of Double Weighting cases (continued)
9.5 The sub-panel reconvened to discuss individual double-weighting requests,
where groups had found it difficult to come to a decision.
9.6 The sub-panel agreed that it would be helpful to have more explicit criteria and
guidance for double-weighting in a future research assessment process.
9.7 Four members of the sub-panel left the room when conflicts arose.
Page 4 of 7
9.8 It was agreed that the chair would record the decisions on double-weighting and
would review and decide on any cases where a decision had not been reached.
10. Output calibration issues
10.1 The sub-panel discussed paper 2.2014.05.
10.2 The sub-panel split into three groups, focusing on the criteria of originality,
significance and rigour in relation to six calibration outputs.
10.3 The sub-panel reconvened to discuss the calibration outputs in a plenary session.
10.4 Six members left the room when conflicts arose.
10.5 The sub-panel then discussed unusual types of outputs and where possible
decided on a range of criteria for use when assessing them.
11. Process for assessing environment and impact templates
11.1 The sub-panel adviser explained how the environment profiles would be
calculated using half-grades and explained the definition of an unclassified grade.
11.2 The sub-panel agreed the process for discussing the environment and impact
profiles at the March meeting.
Page 5 of 7
Day 3
Present:
Tim Allen (Impact assessor)
John Baines
Graeme Barker
Bob Bewley (Impact assessor)
Richard Black
Colin Church (Impact assessor)
Chris Clark
Kevin Edwards
Giles Foody
Ian Freestone
Lynne Frostick
Clive Gamble (Deputy sub-panel chair)
Simon Gardner (Impact assessor)
Roberta Gilchrist
Chris Gosden
Jeremy Hill
Valerie Johnson (Impact assessor)
Andrew Jones
Glynis Jones
Gary Kass (Impact assessor)
Jo Lakey (Sub-panel secretary)
Wendy Larner
Nina Laurie
Barbara Maher
Terry Marsden
David Mattingly
Chris Philo
Philip Rees
Keith Richards (Sub-panel chair)
Charlotte Roberts
Louise Shaxson (Impact assessor)
David Thomas
Charles Withers
Michael Wykes (Sub-panel adviser)
12. Impact and environment template calibration group meetings
12.1 The sub-panel used paper 2.2014.06 as the basis for their discussion in groups.
They were reminded about the impact criteria of significance and reach, sustainability
and vitality for environment.
12.2 The sub-panel then met in plenary to discuss the generic points raised in the
group discussions.
13. Any Other Business
13.1 The chair and deputy chair met the impact assessors to review their involvement
in the assessment process.
Page 6 of 7
13.2 Separate sub-groups of archaeologists and geographers met to review progress
made during the meeting.
13.3 There being no further business the chair thanked members for their
contributions and declared the meeting closed.
Page 7 of 7
REF Sub-Panel C17: Meeting 3
24-26 March
Ettington Chase, Banbury Road, Ettington, Stratford-upon-Avon, CV37 7NZ
Minutes
Day 1
Present:
Tim Allen (Impact assessor)
John Baines
Graeme Barker
Trevor Barnes (Main panel C international member)
Richard Black
Colin Church (Impact assessor)
Chris Clark
Kevin Edwards
Giles Foody
Ian Freestone
Lynne Frostick
Clive Gamble (Deputy sub-panel chair)
Simon Gardner (Impact assessor)
Roberta Gilchrist
Chris Gosden
Jeremy Hill
Valerie Johnson (Impact assessor)
Glynis Jones
Andrew Jones
Gary Kass (Impact assessor)
Jo Lakey (Sub-panel secretary)
Wendy Larner
Nina Laurie
Barbara Maher
Terry Marsden
David Mattingly
Chris Philo
Philip Rees
Keith Richards (Sub-panel chair)
Charlotte Roberts
David Thomas
Diana Wilkinson (Impact assessor)
Charles Withers
Michael Wykes (Sub-panel adviser)
Page 1 of 6
Apologies:
Bob Bewley (Impact assessor)
1. Introduction and competence to do business
1.1 The chair welcomed the sub-panel and thanked them for taking part.
1.2 Apologies were noted.
1.3 In the light of the attendance, the sub-panel confirmed its competency to do
business.
2. Minutes of SP17 Meeting 2 (10th-12th February)
2.1 The minutes from the previous meeting, held on 10 – 12 February 2014, were
approved.
2.2 The sub-panel chair informed members that decisions had been made on most of
the double-weighting requests. The remaining requests would be discussed during the
course of the meeting and decisions would be made shortly.
3. Main Panel C Meeting 2, 6 March
3.1 The sub-panel noted paper 3.2014.02 which contained a digest of the main panel
minutes, recording observations from the main panel’s calibration exercises.
3.2 The sub-panel were informed that the main panel had not yet undertaken a
calibration exercise for the REF5 (environment template) and therefore the sub-panel’s
REF5 scores would be provisional until after that exercise had taken place.
4. Conflicts of interest
4.1 The sub-panel reviewed the register of their declared major conflicts of interest
and confirmed they were correct. Individuals were reminded to inform the executive
group of any updates to their conflicts of interest during the year.
5. Requirements for audit
5.1 The sub-panel noted the procedures and timescales for raising audit queries
arising from impact case studies.
6. Assessment of outputs (progress reports)
6.1 The sub-panel chair updated members on progress with output scoring to date.
Scores for approximately 11% of outputs had been uploaded to the panel members’
website.
7. Date and purpose of next two meetings
7.1 The sub-panel were reminded that part one of the next meeting would involve
compiling the draft impact profiles for each institution. Members were asked to ensure
that 50% of impact case study scores were uploaded to the panel members’ website by
Page 2 of 6
16 April to feed in to the main panel meeting at the end of April. Members were asked to
ensure that 100% of impact case study scores were uploaded to the panel members’
website by 26 May.
7.2 Part two of the next meeting would involve reviewing output and combined
profiles based on the scores for 50% of outputs. Members were reminded to upload
scores for 50% of outputs by 26 May.
8. Sub-groups review and grade environment and impact templates
8.1 The sub-panel split into three groups to discuss grades for environment and
impact templates. Each group discussed twelve institutions and used paper 3.2014.03 to
facilitate their discussion.
8.2 The sub-panel reconvened in two disciplinary groups during the last session of
the day to discuss the emerging institutional profiles from the discussions during the day.
8.3 During the group discussions, any member with a conflict of interest with the
institution being discussed left the room.
Day 2
Present:
Tim Allen (Impact assessor)
John Baines
Graeme Barker
Trevor Barnes (Main panel C international member)
Richard Black
Colin Church (Impact assessor)
Chris Clark
Kevin Edwards
Giles Foody
Ian Freestone
Lynne Frostick
Clive Gamble (Deputy sub-panel chair)
Simon Gardner (Impact assessor)
Roberta Gilchrist
Chris Gosden
Jeremy Hill
Valerie Johnson (Impact assessor)
Glynis Jones
Andrew Jones
Gary Kass (Impact assessor)
Jo Lakey (Sub-panel secretary)
Wendy Larner
Nina Laurie
Barbara Maher
Terry Marsden
Page 3 of 6
David Mattingly
Chris Philo
Philip Rees
Keith Richards (Sub-panel chair)
Mark Robson (Main panel C user member)
Charlotte Roberts
Graeme Rosenberg (REF manager)
Louise Shaxson (Impact assessor)
David Thomas
Diana Wilkinson (Impact assessor)
Charles Withers
Michael Wykes (Sub-panel adviser)
Apologies:
Bob Bewley (Impact assessor)
8. Sub-groups review and grade environment and impact templates
8.4 The sub-panel split into three groups to discuss grades for environment and
impact templates for the remaining institutions.
8.5 The sub-panel reconvened in two disciplinary groups in the final session of the
day to discuss the emerging institutional environment profiles arising from the
discussions held both during the day, and over both days.
8.6 During the group discussions, any member with a conflict of interest with the
institution being discussed left the room.
Day 3
Present:
Tim Allen (Impact assessor)
John Baines
Graeme Barker
Trevor Barnes (Main panel C international member)
Richard Black
Colin Church (Impact assessor)
Chris Clark
Kevin Edwards
Giles Foody
Ian Freestone
Lynne Frostick
Clive Gamble (Deputy sub-panel chair)
Simon Gardner (Impact assessor)
Roberta Gilchrist
Chris Gosden
Jeremy Hill
Valerie Johnson (Impact assessor)
Page 4 of 6
Glynis Jones
Andrew Jones
Gary Kass (Impact assessor)
Jo Lakey (Sub-panel secretary)
Wendy Larner
Nina Laurie
Barbara Maher
Terry Marsden
David Mattingly
Chris Philo
Philip Rees
Keith Richards (Sub-panel chair)
Mark Robson (Main panel C user member)
Charlotte Roberts
Louise Shaxson (Impact assessor)
David Thomas
Martin Walsh (Main panel C user member)
Diana Wilkinson (Impact assessor)
Charles Withers
Michael Wykes (Sub-panel adviser)
Apologies:
Bob Bewley (Impact assessor)
8. Sub-groups review and grade environment and impact templates
8.7 The sub-panel reviewed some summary data on the emerging environment
grade profiles and approved the scores which would remain provisional until after the
main panel’s calibration exercise.
9. Impact case study allocation and audit issues
9.1 The sub-panel adviser gave a short presentation about threshold decision making
for impact case studies. He answered queries about threshold judgements and advised
sub-panel members to raise audit queries if a case study could be awarded an
unclassified grade.
9.2 The sub-panel were reminded to raise any audit queries on impact case studies
by 10 April.
10. Impact case study calibration exercise
10.1 The sub-panel had been sent a set of impact case studies and templates for
calibration.
10.2 Paper 3.2014.04 was used to facilitate the discussion.
10.3 The sub-panel split into three groups to discuss the calibration case studies and
how they had demonstrated the
criteria of significance and reach.
Page 5 of 6
10.4 The sub-panel reconvened in a plenary session to discuss issues relating to the
calibration case studies. There had been a high degree of consistency both in grading
the sample case studies and in identifying the key issues associated with them, amongst
the three groups.
10.5 It was agreed that the calibration exercise had been valuable.
11 Any other business
11.1 There being no further business, the sub-panel chair thanked members for
attending and closed the meeting.
Page 6 of 6
REF Sub-Panel C17: Meeting 4
02 – 04 June
Ettington Chase, Banbury Road, Ettington, Stratford-upon-Avon, CV37 7NZ
Minutes
Day 1
Present:
Tim Allen (Impact assessor)
John Baines
Graeme Barker
Trevor Barnes (Main panel C international member)
Bob Bewley (Impact assessor)
Richard Black
Colin Church (Impact assessor)
Chris Clark
Kevin Edwards
Giles Foody
Ian Freestone
Lynne Frostick
Clive Gamble (Deputy sub-panel chair)
Simon Gardner (Impact assessor)
Roberta Gilchrist
Chris Gosden
Jeremy Hill
Valerie Johnson (Impact assessor)
Andrew Jones
Glynis Jones
Gary Kass (Impact assessor)
Jo Lakey (Sub-panel secretary)
Wendy Larner
Nina Laurie
Barbara Maher
Terry Marsden
David Mattingly
Chris Philo
Philip Rees
Keith Richards (Sub-panel chair)
Charlotte Roberts
Mark Robson (Main panel C user member)
Louise Shaxson (Impact assessor)
Page 1 of 7
David Thomas
Martin Walsh (Main panel C user member)
Diana Wilkinson (Impact assessor)
Charles Withers
Michael Wykes (Sub-panel adviser)
1. Introduction and competence to do business
1.1 The chair welcomed the sub-panel and thanked them for taking part.
1.2 In the light of the attendance, the sub-panel confirmed its competency to do
business.
2. Minutes of SP17 Meeting 3 (24th-26th March)
2.1 The minutes from the previous meeting, held on 24 – 26 March 2014, were
approved without change.
3. Draft Minutes of Main Panel C Meeting 4 (24 April 2014)
3.1 The sub-panel noted Paper 4.2014.02 which contained the draft minutes from the
main panel meeting held on 24 April 2014.
3.2 The sub-panel chair drew members’ attention to the minute covering the
production of feedback reports to HEIs and subject overview reports. The sub-panel
would discuss these reports later in the meeting.
3.3 The sub-panel chair reported that SP17’s emerging impact profile appeared to be
largely consistent with the overall impact profile for main panel C, although it was noted
that the data was incomplete and in some cases was based on unreconciled scores
3.4 Main panel C had discussed two REF5s from SP17 in their environment
calibration and had arrived at grades consistent with those given by SP17.
4. Conflicts of interest
4.1 The sub-panel reviewed the register of their declared major conflicts of interest
and confirmed they were correct. Individuals were reminded to inform the executive
group of any updates to their conflicts of interest during the year.
4.2 The sub-panel were reminded that during discussions of impact case studies,
members who were conflicted with the HEI should leave the room. Additionally, any
member who had a conflict of interest with an individual case study should leave the
room during the discussion of that case study.
5. Report on audits
5.1 The sub-panel secretary gave a summary of the audit queries which had been
raised on impact case studies.
Page 2 of 7
5.2 The sub-panel noted that a range of submissions had been audited. The majority
of audit queries had been resolved before the meeting.
6. Dates and purpose of next two meetings
6.1 The sub-panel noted paper 4.2014.03 which contained guidance on the feedback
statements which would be sent to HEIs.
6.2 At the next meeting in July, in additional to reporting 75% of output scores, the
sub-panel would start to consider the feedback reports for individual submissions and the
subject overview reports.
6.3 The sub-panel noted that at the following meeting, in September, the feedback
reports would be agreed.
7. Assessment of Impact Case Studies
7.1 The sub-panel then split into three different groups to discuss grades for impact
case studies, using paper 4.2014.04 to facilitate the discussion. Each group discussed
between 74 and 86 case studies.
7.2 The sub-panel reported agreed scores to the sub-panel secretary for recording
and uploading to the panel members’ website.
7.3 During the discussions, any member with a conflict of interest with the institution
being discussed left the room.
Page 3 of 7
Day 2
Present:
Tim Allen (Impact assessor)
John Baines
Graeme Barker
Trevor Barnes (Main panel C international member)
Bob Bewley (Impact assessor)
Richard Black
Colin Church (Impact assessor)
Chris Clark
Kevin Edwards
Janet Finch (Main panel C chair)
Giles Foody
Ian Freestone
Lynne Frostick
Clive Gamble (Deputy sub-panel chair)
Simon Gardner (Impact assessor)
Roberta Gilchrist
Chris Gosden
Jeremy Hill
Valerie Johnson (Impact assessor)
Andrew Jones
Glynis Jones
Gary Kass (Impact assessor)
Jo Lakey (Sub-panel secretary)
Wendy Larner
Nina Laurie
Barbara Maher
Terry Marsden
David Mattingly
Chris Philo
Philip Rees
Keith Richards (Sub-panel chair)
Charlotte Roberts
Mark Robson (Main panel C user member)
Louise Shaxson (Impact assessor)
David Thomas
Martin Walsh (Main panel C user member)
Diana Wilkinson (Impact assessor)
Charles Withers
Michael Wykes (Sub-panel adviser)
7. Assessment of Impact Case Studies
7.1 The sub-panel resumed their group discussions of impact case studies.
Page 4 of 7
8. Feedback from impact assessors
8.1 The sub-panel reconvened in plenary to discuss the impact assessment process.
8.2 The impact assessors were invited to feed back on their experiences of the
impact assessment.
8.3 Impact assessors agreed that the process had been successful and that a wide
range of high quality impacts had been identified within the sub-panel. Additionally, it was
agreed that it was important to feed back to the community about impact to ensure that
good practice was shared and that strong impact submissions could be made to the next
REF.
8.4 It was agreed that assessing the impact and environment templates together was
a valuable exercise, allowing the impact assessors to have additional contextual
information.
8.5 The sub-panel members recorded their thanks to the impact assessors, who had
made an extremely valuable contribution to the process of assessing impact, and to
wider aspects of the panel's work.
9. Discuss and approve draft impact sub-profiles
9.1 The sub-panel discussed the draft impact sub-profiles, comprising aggregated
scores from the impact template and impact case studies for each HEI.
9.2 The sub-panel discussed those HEIs where there were no conflicts of interest
first.
9.3 Members left the room when they had a conflict of interest with the HEI under
discussion.
9.4 An audit query had arisen for one case study during the sub-panel’s discussions.
It was agreed that the audit query result would be sent to the two readers, and they
would inform the executive group of the agreed score for the case study.
9.5 The sub-panel agreed to recommend the draft impact sub-profiles to the main
panel.
Page 5 of 7
Day 3
Present:
Neil Adger (Output assessor)
John Baines
Graeme Barker
Trevor Barnes (Main panel C international member)
Richard Black
Chris Clark
Kevin Edwards
Giles Foody
Ian Freestone
Lynne Frostick
Clive Gamble (Deputy sub-panel chair)
Jeremy Hill
Andrew Jones
Glynis Jones
Jo Lakey (Sub-panel secretary)
Wendy Larner
Nina Laurie
Barbara Maher
Terry Marsden
David Mattingly
Joe Painter (Output assessor)
Chris Philo
Philip Rees
Keith Richards (Sub-panel chair)
Charlotte Roberts
Jennifer Robinson (Output assessor)
Jon Sadler (Output assessor)
David Thomas
Paul Valdes (Output assessor)
Charles Withers
Michael Wykes (Sub-panel adviser)
Apologies:
Roberta Gilchrist
Chris Gosden
10. Output scoring
10.1 The sub-panel chair welcomed the output assessors.
10.2 The sub-panel agreed the procedure for transferring scores to the ‘panel agreed’
column.
10.3 The sub-panel noted that 63% of outputs had panel agreed scores and agreed to
report these scores to the main panel.
Page 6 of 7
10.4 The sub-panel were reminded that there was a target for 75% of output scores to
be reported to the main panel following the July meeting and were asked to continue
reading outputs and uploading scores.
10.5 The sub-panel adviser presented some data on output scoring to date, showing
individual members’ scoring averages and standard deviation.
10.6 The sub-panel agreed to continue the process of second reading/moderation. It
was agreed that it was the responsibility of the first reader to upload the agreed score to
the panel members’ website.
11. Sub-group discussions of outputs
11.1 The sub-panel split into three disciplinary groups to discuss outputs, agree scores
between readers and to identify areas of note which could be included in HEI feedback or
subject overview reports.
12. Any other business
12.1 There being no further business, the sub-panel chair thanked members for
attending and closed the meeting.
Page 7 of 7
REF Sub-Panel C17: Meeting 5
14 – 15 July
Wotton House, Guildford Road, Dorking, Surrey RH5 6HS
Day 1
Present:
John Baines
Graeme Barker
Trevor Barnes (Main panel C international member)
Richard Black
Chris Clark
Kevin Edwards
Janet Finch (Main panel C chair)
Giles Foody
Ian Freestone
Lynne Frostick
Clive Gamble (Deputy sub-panel chair)
Roberta Gilchrist
Chris Gosden
Jeremy Hill
Andrew Jones
Glynis Jones
Jo Lakey (Sub-panel secretary)
Wendy Larner
Nina Laurie
Barbara Maher
Terry Marsden
David Mattingly
Joe Painter (Output assessor)
Chris Philo
Philip Rees
Keith Richards (Sub-panel chair)
Charlotte Roberts
David Thomas
Charles Withers
Michael Wykes (Sub-panel adviser)
1. Introduction and competence to do business
1.1 The chair welcomed the sub-panel and thanked them for taking part.
Page 1 of 5
1.2 In the light of the attendance, the sub-panel confirmed its competency to do
business.
2. Minutes of SP17 Meeting 4 (2 – 4 June)
2.1 The minutes from the previous meeting, held on 2 – 4 June 2014, were approved
without change.
3. Draft Minutes of Main Panel C Meeting 5 (19 June 2014)
3.1 The sub-panel noted Paper 5.2014.02 which contained the draft minutes from the
main panel meeting held on 19 June 2014.
3.2 The main panel had discussed the emerging impact profiles from the sub-panels
and were undertaking an audit exercise to ensure that they were confident that all sub-
panels had applied the criteria consistently.
3.3 The sub-panel adviser presented a set of anonymised weighted impact profiles
for the main panel.
3.3 The main panel also discussed the emerging output profiles and were
comfortable that sub-panels were working with a good degree of consistency.
3.4 The sub-panel adviser presented a set of anonymised weighted output profiles
for the main panel.
3.5 The main panel had also discussed double-weighting of outputs.
4. Conflicts of interest
4.1 The sub-panel reviewed the register of their declared major conflicts of interest
and confirmed they were correct. Individuals were reminded to inform the executive
group of any updates to their conflicts of interest during the year.
4.2 The sub-panel were reminded that during discussions of environment templates,
members who were conflicted with the HEI should leave the room.
5. Equality and Diversity Report
6.1 The chair presented paper 5.2014.03 which outlined recommendations for
individual staff circumstances.
6.2 A sub-panel member who was also a member of the Equality and Diversity
Advisory Panel (EDAP) commented on the very thorough process followed by EDAP for
considering staff with complex circumstances.
6.3 The sub-panel approved the recommendations within the paper.
6. Dates and purpose of next two meetings
Page 2 of 5
6.1 The sub-panel noted that emerging profiles for all HEIs would be reviewed at the
next meeting in September. It was therefore important that 100% of outputs had been
scored by that time. The sub-panel would also review the feedback reports for each HEI.
6.2 The sub-panel noted that the October meeting would involve agreeing the subject
overview reports and signing everything off.
7. Draft feedback report
7.1 The sub-panel then split into four different groups to discuss feedback reports for
HEIs, using paper 5.2014.04 to facilitate the discussion.
7.2 The sub-panel reconvened in a plenary session, where examples of feedback
reports were presented and discussed.
7.3 It was agreed that feedback for impact and environment should be completed by
the rapporteur for each HEI and sent to the executive group by 26 August 2014.
7.4 The sub-panel secretary would produce data on output scoring for each
rapporteur which would be sent out at the end of August to enable feedback on outputs
to be completed before the September meeting.
7.5 The sub-panel continued their group discussions on impact and environment
feedback.
Page 3 of 5
Day 2
Present:
Neil Adger (Output assessor)
John Baines
Graeme Barker
Trevor Barnes (Main panel C international member)
Richard Black
Chris Clark
Kevin Edwards
Janet Finch (Main panel C chair)
Giles Foody
Ian Freestone
Lynne Frostick
Clive Gamble (Deputy sub-panel chair)
Roberta Gilchrist
Chris Gosden
Jeremy Hill
Andrew Jones
Glynis Jones
Jo Lakey (Sub-panel secretary)
Wendy Larner
Nina Laurie
Barbara Maher
Terry Marsden
David Mattingly
Joe Painter (Output assessor)
Chris Philo
Philip Rees
Keith Richards (Sub-panel chair)
Charlotte Roberts
Jennifer Robinson (Output assessor)
Jon Sadler (Output assessor)
David Thomas
Paul Valdes (Output assessor)
Charles Withers
Michael Wykes (Sub-panel adviser)
8. Output scoring
8.1 The sub-panel chair welcomed the output assessors.
8.2 The sub-panel adviser presented some data on output scoring to date, showing
individual members’ scoring averages and standard deviation.
8.3 The sub-panel noted that more than 75% of outputs had panel agreed scores and
agreed to report these scores to the main panel.
Page 4 of 5
8.4 The sub-panel adviser presented emerging profiles for each institution.
8.5 Members noted that the profile data was not complete, as some HEIs had fewer
output scores.
8.6 Twenty eight Members left the room when they had a conflict of interest with the
HEI under discussion.
8.7 The sub-panel were confident that the emerging profiles had been reached as a
result of a rigorous assessment process, and that criteria had been consistently applied.
9. Sub-group discussions of outputs
9.1 The sub-panel split into three disciplinary groups to discuss outputs, agree scores
between readers and to identify areas of note which could be included in HEI feedback or
subject overview reports.
12. Any other business
12.1 There being no further business, the sub-panel chair thanked members for
attending and closed the meeting.
Page 5 of 5
REF Sub-Panel C17: Meeting 6
23 – 24 September 2014
Ettington Chase, Banbury Road, Ettington, Stratford-upon-Avon, CV37 7NZ
Minutes
Day 1 - 23 September 2014
Present:
John Baines
Graeme Barker
Richard Black
Chris Clark
Kevin Edwards
Giles Foody
Ian Freestone
Lynne Frostick
Clive Gamble (Deputy sub-panel chair)
Roberta Gilchrist
Chris Gosden
Jeremy Hill
Andrew Jones
Glynis Jones
Jo Lakey (Sub-panel secretary)
Wendy Larner
Nina Laurie
Barbara Maher
Terry Marsden
David Mattingly
Chris Philo
Philip Rees
Keith Richards (Sub-panel chair)
Charlotte Roberts
David Thomas
Charles Withers
Michael Wykes (Sub-panel adviser)
1. Introduction and competence to do business
1.1 The chair welcomed the sub-panel and thanked them for taking part.
1.2 In the light of the attendance, the sub-panel confirmed its competency to do
business.
Page 1 of 4
2. Minutes of SP17 Meeting 5 (14 – 15 July)
2.1 The minutes from the previous meeting, held on 14 -15 July 2014, were approved
without change.
3. Draft Minutes of Main Panel C Meeting (24 July)
3.1 The sub-panel noted paper 6.2014.02 which contained the draft minutes from the
main panel meeting held on 19 June 2014.
4. Conflicts of interest
4.1 The sub-panel reviewed the register of their declared major conflicts of interest
and confirmed they were correct.
4.2 The sub-panel were reminded that during discussions of quality profiles,
members who were conflicted with the HEI should leave the room.
5. Audit
5.1 Members were informed that there were no outstanding audits at the time of the
meeting.
6. Dates and purpose of next two meetings
6.1 The sub-panel noted that feedback for all HEIs would be signed off at the next
meeting in October.
6.2 The sub-panel noted that the October meeting would also involve agreeing the
subject overview reports.
6.3 Members were asked to bring their USB pens to the next meeting for collection
by the sub-panel secretary.
7. Review quality profiles and feedback reports
7.1 The sub-panel discussed the draft quality profiles and feedback paragraphs for
each HEI, referring to paper 6.2014.03.
7.2 The sub-panel discussed those HEIs where there were no conflicts of interest
first.
7.3 Eight members left the room when they had a conflict of interest with the HEI
under discussion.
Page 2 of 4
Day 2 – 24 September 2014
Present:
John Baines
Graeme Barker
Richard Black
Chris Clark
Kevin Edwards
Giles Foody
Ian Freestone
Lynne Frostick
Clive Gamble (Deputy sub-panel chair)
Roberta Gilchrist
Chris Gosden
Jeremy Hill
Andrew Jones
Glynis Jones
Jo Lakey (Sub-panel secretary)
Wendy Larner
Nina Laurie
Barbara Maher
Terry Marsden
David Mattingly
Chris Philo
Philip Rees
Keith Richards (Sub-panel chair)
Charlotte Roberts
David Thomas
Charles Withers
Michael Wykes (Sub-panel adviser)
7. Review quality profiles and feedback reports
7.4 The sub-panel continued their discussions of the draft quality profiles and
feedback paragraphs for each HEI.
7.5 Sixteen members left the room when they had a conflict of interest with the HEI
under discussion.
7.6 Members were asked to send revisions to feedback paragraphs to the executive
group no later than 5 October 2014.
8. Sign off 100% of output scores
8.1 The sub-panel resolved to recommend 99.63% of output scores to the main
panel.
Page 3 of 4
8.2 Members waiting for cross-referral advice were asked to let the sub-panel
secretary have the details as soon as possible.
8.3 Members were asked to upload all remaining scores by 10 October 2014. These
would be approved by chair’s action.
9. Sub-group discussions of subject overview reports
9.1 The sub-panel split into three disciplinary groups to discuss and to identify areas
of note which could be included in HEI feedback or subject overview reports.
10. Any other business
10.1 There being no further business, the sub-panel chair thanked members for
attending and closed the meeting.
Page 4 of 4
REF Sub-Panel C17: Meeting 7
21 October 2014
Selsdon Park Hotel, Addington Road, Sanderstead, South
Croydon, CR2 8YA
Minutes
Present:
John Baines
Graeme Barker
Richard Black
Chris Clark
Hazel Crabb-Wyke (REF team)
Kevin Edwards
Giles Foody
Ian Freestone
Lynne Frostick
Clive Gamble (Deputy sub-panel chair)
Roberta Gilchrist
Chris Gosden
Jeremy Hill
Glynis Jones
Jo Lakey (Sub-panel secretary)
Wendy Larner
Nina Laurie
Barbara Maher
Terry Marsden
David Mattingly
Chris Philo
Philip Rees
Keith Richards (Sub-panel chair)
Charlotte Roberts
David Thomas
Charles Withers
Michael Wykes (Sub-panel adviser)
Apologies:
Andrew Jones
Page 1 of 5
1. Introduction and competence to do business
1.1 The chair welcomed the sub-panel to the final meeting and thanked them for
taking part.
1.2 In the light of the attendance, the sub-panel confirmed its competency to do
business.
2. Minutes of SP17 Meeting 6 (23 – 24 September 2014)
2.1 The minutes from the previous meeting, held on 23 – 24 September 2014, were
approved without change.
3. Draft Minutes of Main Panel C Meeting (1 October 2014)
3.1 The sub-panel noted the draft minutes from the main panel meeting held on 19
October 2014.
4. Matters arising
4.1 The chair informed the panel that since the last meeting, a discrepancy between
scores given to an output submitted twice had been discussed and resolved.
4.2 Following discussions at the main panel about the level to which different sub-
panels had used 100% 4* in the environment profile, all sub-panels had been invited to
revisit the issue to ensure that the 4* boundary had been appropriately applied and the
grades awarded reflected the quality of the material submitted.
4.3 The chair informed the sub-panel that he had requested a review of the
environment profiles to ensure that SP17’s scoring was in line with the main panel
calibration. This review was undertaken by an international assessor for the main panel,
who recommended one change in scoring.
4.4 The sub-panel approved this change, and recorded its thanks to the international
assessor for their help with this matter.
5. Conflicts of interest
5.1 The sub-panel reviewed the register of their declared major conflicts of interest
and confirmed they were correct.
6. Audit
6.1 Members were informed that following the last meeting, audits were raised on
several outputs to ensure that the unclassified grade had been fairly and consistently
applied. Changes had been made to one output grade as a result.
6.2 The sub-panel approved this change.
7. Housekeeping
7.1 The sub-panel adviser presented a set of slides which outlined the timetable for
the release of results, and procedures to follow at the end of the assessment.
Page 2 of 5
7.2 Members were reminded of the confidentiality of the assessment process, and
that they should only comment on information which would be publicly available.
7.3 Members were reminded to destroy all hand-written notes pertaining to the
assessment before the publication of the results on 18 December.
7.4 The sub-panel secretary collected members’ USB pens for returning to the REF
team.
8. Feedback on submissions
8.1 The sub-panel confirmed recommended output, impact and environment sub-
profiles and an overall quality profile for each of the following submissions to SP17,
based in each case on its full and final assessment of the complete submission, made in
accordance with the published criteria and working methods:
Aberdeen (University of)
Aberystwyth University
Anglia Ruskin University
Birmingham (University of)
Bournemouth University
Bradford (University of)
Bristol (University of) - Archaeology and Anthropology
Bristol (University of) - Geography
Cambridge (University of) - Archaeology
Cambridge (University of) - Geography
Cardiff University
Central Lancashire (University of)
Chester (University of) - Archaeology
Chester (University of) - Geography and Development Studies
Cumbria (University of)
Dundee (University of)
Durham (University of) - Archaeology
Durham (University of) - Geography
Edge Hill University
Edinburgh (University of)
Exeter (University of) - Archaeology
Exeter (University of) - Geography and Environmental Studies
Glasgow (University of) - Archaeology
Glasgow (University of) - Geography
Gloucestershire (University of)
Highlands and Islands (University of the)
Hull (University of)
King's College London
Kingston University
Leeds (University of)
Page 3 of 5
Leicester (University of) - Archaeology
Leicester (University of) - Geography
Liverpool (University of) - Archaeology
Liverpool (University of) - Geography and Environmental studies
Liverpool Hope University
London School of Economics and Political Science
Loughborough University
Manchester (University of) - Archaeology
Manchester (University of) - Geography
Middlesex University
Newcastle upon Tyne (University of) - Archaeology
Newcastle upon Tyne (University of) - Geography
Northampton (University of)
Northumbria at Newcastle (University of)
Nottingham (University of) - Archaeology
Nottingham (University of) - Geography
Nottingham Trent University
Open University
Oxford (University of) - Archaeology
Oxford (University of) - Geography and Environmental studies
Oxford Brookes University
Plymouth (University of)
Portsmouth (University of)
Queen Mary, University of London
Queen's University Belfast
Reading (University of) - Archaeology
Reading (University of) - Geography
Royal Holloway, University of London
Salford (University of)
Sheffield (University of) - Archaeology
Sheffield (University of) - Geography
Southampton (University of) - Archaeology
Southampton (University of) - Geography
St Andrews (University of)
Stirling (University of)
Sussex (University of)
Swansea University
Trinity Saint David (University of Wales)
University College London - Archaeology
University College London - Geography
West of England, Bristol (University of the)
Winchester (University of)
Worcester (University of)
York (University of)
Page 4 of 5
8.2 The sub-panel resolved to recommend the quality profiles for each of the
submissions listed above, as set out in the panel spreadsheet, to the main panel for
agreement.
8.3 HEI feedback had been circulated to members in advance of the meeting
(respecting conflicts of interest).
8.4 The sub-panel resolved to recommend the feedback on submissions to the main
panel, subject to further edits which may be made by the executive group.
9. Subject overview report
9.1 The draft subject overview report had been circulated to members in advance of
the meeting.
9.2 In plenary, the sub-panel discussed issues and made suggestions for items to
include in the report.
9.3 The sub-panel split into three disciplinary groups to discuss specific sections of
the subject overview report in more detail.
9.3 The sub-panel resolved to recommend the draft subject overview report to the
main panel, and agreed that chair’s action would be taken to make any necessary edits
or amendments.
10. Any other business
10.1 There being no further business, the sub-panel chair thanked members for their
hard work throughout the assessment phase and closed the meeting.
Page 5 of 5
