91ÑÇÉ«´«Ã½

The 91ÑÇÉ«´«Ã½ on Monday sent a letter to the National Institutes of Health Center for Scientific Review in which it advocated for changes aimed at reducing bias during the evaluation of research proposals and changes that would level the playing field for training grant applicants.

The letter was submitted in response to two matters discussed March 28 at a meeting of the CSR : recent bias training initiatives and ways to improve the review of fellowship applications.

In short, the 91ÑÇÉ«´«Ã½ told the CSR that (A) it strongly supports the implementation of bias training at every level of peer review and that (B) it approves of the NRSA Working Group’s thoughtful recommendations— with a few tweaks. 

Bias training piloted

The CSR rolled out its “Bias Awareness in Review” training program in the fall.

Director Noni Byrnes said at the advisory council meeting that the program had a lot of successes as well as other “places to adjust” for the next round. Byrnes also emphasized that bias training “has to become mandatory” in the future.

In addition, Hope Cummings, a senior social science analyst at CSR, shared data collected over the past year about reviewers’ impressions of bias during review.

Strikingly, she reported that “the great majority of reviewers believe the NIH has a problem with bias in review” and “most reviewers (53%) never or rarely intervened when they thought bias was present.”

The advisory council recommended that CSR:

1. Establish bias training as a requirement to participate in the review process.

2. Create additional programming on implicit bias and other biases.

3. Empower NIH scientific review officers and chairs of review panels to intervene when biased comments or assessments are made.

4. Ensure that the framework for intervening in and reporting bias circumvents the power differentials present between review panel members. 

In its letter this week to CSR, the 91ÑÇÉ«´«Ã½ emphasized that bias training should be mandatory for everyone involved in peer review — “all reviewers, chairs and scientific review officers” — and that the training should be “offered recurrently to facilitate sustained and enhanced awareness of bias.”

The society also recommended “a thoughtful review of feedback from the voluntary participants and incorporating it into new material that can enhance the curriculum, such as providing more examples of implicit and other types of bias and intervention tools for reviewers to use to counteract bias without fear of retaliation.”

The 91ÑÇÉ«´«Ã½ has been vocal about instilling diversity, equity, and inclusion during scientific review, as exemplified by its recent recommendations to the CSR strategic plan.

Adjusting fellowship application criteria

At the advisory council meeting, Deputy Director Bruce Reed explained that CSR had heard “persistent concerns that fellowship review may disadvantage some highly qualified applicants,” leading CSR to to “make recommendations to make the process as fair and effective as possible for all.”

According to CSR, should “emphasize the candidate’s potential for a productive career, the candidate’s need for the proposed training, and the degree to which the research project and training plan, the sponsor(s), and the environment will satisfy those needs.”

Under this definition, the applicant’s scientific environment had become a significant factor in the evaluation criteria, leading to research-intensive universities having a leg up in securing fellowship applications under the current review criteria. Furthermore, that fellowship awardees do not always outperform their peers without fellowship awards.  

The working group recommended the following:

1. Eliminate grades as indicators of qualifications.

2. Eliminate the “Sponsor/Collaborator” and “Institutional Environment/Commitment to Training” criteria.

3. Allow an optional statement of special circumstances (with an option to have the school submit a separate letter).

4. Explicitly allow a wider range of career paths in fellowship training.

5. Encourage a statement of qualifications that extends beyond grades and publications.

6. Have review criteria that measure the total impact that the fellowship can bring to the individual and reduce the importance of external factors, such as mentor, institution, etc.

The working group also presented a “delta” concept that aims to provide a more holistic approach to review, reducing the emphasis on the sponsor and institution and focusing more on the applicant’s potential to succeed under the fellowship.

Elizabeth Villa, chair of the working group, explained that they were aiming to capture “the idea of casting a wider net to try to find the diamonds in the rough — incredible scientists with incredible potential that might have less strong conventional backgrounds.”

To accomplish this, she stated that they would “ask reviewers to try to not focus only on their accomplishments so far but also how likely it is that they will substantially benefit from the proposed fellowship — (to) pay a lot of attention to what they talk about their determination persistence creativity and otherwise potential to do incredibly well.”

The 91ÑÇÉ«´«Ã½ already had made many of the above recommendations to improve fellowship review on Jan. 24. The society noted in its letter to CSR on Monday that it was pleased the working group had incorporated many of its suggestions, such as eliminating grades as an evaluation criterion since “grades are not an accurate measurement of success.”

However the society expressed concern about removing the sponsor/collaborator section altogether because “mentorship does play an important role in trainee success.” It recommended that the section be replaced with “a detailed mentorship plan, similar to the mentorship plan required by NSF fellowship applications.”

The 91ÑÇÉ«´«Ã½ also had suggestions for how to implement an optional statement of special circumstances, which would give applicants the opportunity to explain setbacks from COVID-19, gaps in their training or transitions between labs. The society asked CSR to provide this option on future applications in a very intentional and clear format so that applicants “1) understand what qualifies as a special circumstance and 2) feel reassured that explaining certain special circumstances, such as harassment, will not penalize them in any way.”

Overall, the society was encouraged by the progress shared at the March meeting and asked that the CSR begin implementing the recommendations to the best of its ability.

Read the 91ÑÇÉ«´«Ã½’s full comments here.