Responsible Research Assessment: Evaluating Contributions Beyond Publication Counts

Define what the assessment is for

Research assessment may inform recruitment, promotion, funding or the review of a research unit. Different decisions need different evidence. Begin by identifying the role, expectations and period under consideration. A teaching-focused appointment and a specialist research grant should not automatically use the same criteria or weights.

DORA recommends assessing research on its own merits rather than using journal-based metrics as a substitute for quality.^[1] For an institution, this creates a practical design question: what evidence would allow reviewers to understand the importance, rigour and contribution of the work being assessed?

Recognise a range of relevant contributions

Depending on the role and discipline, evidence may include publications, datasets, software, methods, scholarly editions, mentoring, collaboration or engagement with users of research. Define which contributions matter and why. A broader range does not mean that every applicant must demonstrate excellence in every category.

Ask for a small, selected portfolio with explanations of contribution. The individual should identify their role, the problem addressed, the evidence of quality and relevant limitations. For collaborative outputs, distinguish the team's achievement from the person's contribution. Avoid rewarding claims of impact that cannot be supported by appropriate evidence.

Use indicators as contextual information

Indicators can help describe reach, activity or aspects of influence, but their meaning depends on field, career stage, time and data coverage. Check the source, calculation and limitations. A citation count may reflect attention without explaining whether the attention is supportive, critical or appropriate to the decision.

Do not replace one simplistic metric with another. Download counts, social media attention and grant income also require interpretation. Use quantitative information to inform a reasoned review of the work. Make clear when an indicator is unavailable or unsuitable, and avoid penalising contributions that are valuable but poorly represented in common databases.

Design a fair review process

Publish criteria, evidence expectations and the assessment procedure before submissions are prepared. Train reviewers to apply the criteria and manage conflicts of interest. Allow appropriate consideration of career context and opportunity, using a process that protects sensitive information. Do not ask applicants to disclose more personal detail than is necessary.

CoARA's agreement sets a shared direction for research assessment reform while allowing contextual implementation.^[2] An institution adopting relevant ideas should explain its own process and resources. Membership or endorsement claims should be made only where they are actually true; thoughtful practice does not require presenting an unverified affiliation.

Illustrative example: two research contributions

Imagine a promotion panel considering two applicants. One has many short publications with limited explanation of individual contribution. Another presents a smaller portfolio including a well-documented dataset, a substantial paper and evidence of mentoring that enabled a collaborative project. The panel examines the quality and relevance of each contribution against the published role criteria.

The outcome is not predetermined by either volume or diversity. Reviewers read selected work, seek clarification where needed and explain how the evidence meets the standard. The example shows why a count cannot complete the judgement, while a narrative also needs scrutiny rather than being accepted at face value.

Calibrate judgement and record reasons

Use anonymised sample portfolios or fictional cases to discuss how criteria should be applied. Identify differences in disciplinary expectations and agree how the panel will handle them. Give reviewers sufficient time and access to expertise. A process that asks for richer evidence but allocates only a few minutes per case may reproduce shortcuts.

Record the material reasons for decisions and provide feedback appropriate to the process. Establish a review or appeal route under the institution's rules. Monitor whether the workload is proportionate for applicants and reviewers and whether particular forms of contribution are repeatedly overlooked.

Implement reform in manageable stages

Begin with a defined assessment process, consult affected researchers and test the revised criteria. Compare the evidence produced and the reasoning used with the previous approach. Review unintended effects, such as excessive narrative-writing demands or advantages for applicants with more coaching.

Responsible assessment remains a matter of informed judgement. It does not promise perfect objectivity or eliminate difficult choices. Its credibility comes from relevant criteria, transparent evidence, competent review and an honest account of why a decision was reached.