Proposal Reviews at the National Science Foundation
Recently I had the opportunity to participate on a review panel for a
set of proposals designed to improve one or more aspects of undergraduate
science, technology, engineering, and mathematics education. It all took
place over four days, including travel to and from Arlington, VA. As it
was my first such experience, I was very curious to see how the review
process worked. I had been involved in a couple of preliminary grant
proposals over the past six years, including one to the NSF, so it was
great to get a first-hand look at what qualities separate those proposals
that get funded from those that do not.
For the review session in which I participated, there were about 300
total reviewers for proposals from across all the natural sciences,
including biology, chemistry, engineering and mathematics. In the
mathematical sciences, reviewers were put into five different panels. One
focused on proposals for teacher education, one for developmental and
precalculus mathematics, one for lower division courses (calculus, linear
algebra, differential equations), one for upper-division courses, and one
that handled large, national dissemination proposals. The panel of eight
reviewers on which I served was responsible for 13 proposals.
After flying to DC, the first evening featured several orientation
meetings. The first of those was a group meeting of all reviewers with
NSF staff to get us acquainted with what to focus on in the review
process, and what to include in our individual reviews to be put into
NSF's web-based system for later review by NSF program directors. The
second was a meeting of reviewers from individual disciplines with
NSF program directors in that discipline. After that, individual panels
got together in their assigned rooms to set up meeting times for the next
day and to decide which proposals reviewers should be prepared to discuss
in detail. Once the preliminaries were determined, it was time to get
down to the business of reading each proposal and rate them individually.
We did this for two full days, and then on the morning of the fourth day
we double-checked our entries on the NSF system. We left all of our
copies of proposals and notes behind for later destruction.
Across the sciences, 580 proposals were received during the NSF call
for proposals, 35 of which were for "national dissemination" funding and
the rest of which were two-year "proof-of-concept" proposals designed to
do preliminary studies. The former tended to be three years in length,
and requests for funds were on the order of $500,000. The latter
proposals sought funds that (at least for the proposals I reviewed)
averaged between $25,000 and $50,000 per year. Of the 35 national
dissemination proposals, were we told that only five would be funded. Out
of a total of about $108 million requested, only about $20 million in
funding was to be awarded. So, it's an extremely competitive process, and
prospective principal investigators (PIs) should know this up front.
The rating system works as follows:
-
Excellent - should receive top
priority for funding;
-
Very Good - strong proposal in most respects,
deserving of funding if at all possible;
-
Good - worthy of support;
-
Fair - proposal is lacking in one or more areas;
-
Poor - seriously deficient.
Our
panel was set up for us to give individual ratings to each proposal, but
not a panel rating. As we read each proposal, we noted the project's
"intellectual merits" and "broader impacts" and then wrote an individual
summary to share at our panel meeting. Those meetings were very important
for me as a first-time reviewer, since those who were more experienced
picked up on key subtleties that affected their individual ratings. Each
proposal also had one panelist assigned to serve as "scribe." The scribe
for a proposal took complete notes on the panel's discussion and then
entered the panel's summary into the NSF system for later review, along
with their own individual summary and rating.
Each proposal contained some initial pages with the name(s) of PI(s), a
one-page proposal summary, 15 pages of project description, a budget along
with justification for funds sought, PI(s) qualifications for pursuing the
study, and references to related work. For me, the one-page summary was
of vital importance - the overall essence and scope of the project needed
to be clear up front. With a high volume of proposals to read and limited
time to digest each one, it is incumbent on the proposal's writers to get
the critical details across coherently and succinctly. Then, the 15-page
description had to be detailed enough to understand the project's goals
and objectives for me to develop a feel for how strong an impact the
project would have on the PI's students. Concrete examples of learning
modules, abbreviated sample text materials, etc., were a big help, as were
any included web links to computer-based learning materials to be
incorporated in the study.
According to the panel discussions we had for each proposal, the
15-page project description was the make-or-break for most panelists. The
budget was important, but only caused concern if costs (such as those for
release time) appeared high. There were also proposals where one or more
of the PIs' backgrounds did not appear to be a good fit for the project,
in terms of any previous grant work, publications, and related activities.
In such a case it seems a good idea for PIs to explain that their
proposed project will take them in a new direction, rather than leaving
the panel to wonder why it is exactly they are applying for funding in the
first place. In one instance, it did look like a PI was planning a
project at someone else's behest (perhaps administration at their
institution), and that caught the eyes of several panelists.
Several years ago when I was in my active year of Fellowship in Project
NExT, I attended a grant-writing workshop hosted by one of the NSF program
directors. He discussed a number of problems that inexperienced grant
writers often encounter. For those seriously thinking about committing to
doing a grant proposal, it is important to spend time researching what
others have done that is similar to your project. Laying out specifics of
how your work will forge a new path, together with statements showing how
your work will be distinct when compared with similar projects will not
only help the reviewing panel to understand the nature and scope of your
proposal, but higher ratings will also result. Two proposals that we read
contained lengthy details about prior successful grant work; several
panelists viewed parts of that as superfluous. Succinctness is important
here, so as not to eat up too much of the allotted 15 pages discussing
past work instead of carefully detailing the new project. Since obtaining
funding is so difficult, first-time applicants should be prepared to be
turned down and to go over reviewers comments to help them fix and
resubmit their proposals in a future call.
The entire experience was very worthwhile for me. Although there is no
current plan to obtain external funding in my department for a project,
knowing how the review process works will be invaluable to us if and when
the need arises. The NSF is always looking for individuals interested in
reviewing proposals. If you are interested in reviewing proposals for DUE
please visit:
http://www.nsf.gov/pubsys/ods/getpub.cfm?form428a
Another helpful link is:
http://www.ehr.nsf.gov/ehr/due/programs/general/advice.asp
Respectfully submitted,
Kevin Charlwood (Washburn University)
E-mail: zzcharlw@washburn.edu
