This was written in 2005 and some things have changed (scoring now goes from 10-90) for instance but a lot of it is still the same especially conceptually/sociologically.
The grant application review process is very arbitrary at times, but yet the level of rigour is extremely high. I am going to focus here on what happens here in terms of NIH study sections and NSF review panels for investigator-initiated research grant applications and attempt to explain why this contradictory dynamic exists. Understanding this is key to being a successful grantee, in my view.
See also my grant application writing tips. These are just my reviews based on the NIH study sections and NSF review panels I've been on (see my CV for more details). I am not sure how general it is.
For NIH, initially the Scientific Review Administrators (SRAs) put together a panel of scientists similar to you to help review the grant application they have on their hands. For each application, the SRA usually picks three reviewers: primary, secondary, and discussant. The quality of the review by the people in these roles has a strong impact on the success of your grant, in the order listed, as you'll see.
The primary reviewer typically provides the most exhaustive written critique of your application. The secondary reviewer also provides a rigourous written critique but the expectation is that the primary reviewer would cover most of the bases and the second reviewer acts as a "backup" of sorts. The discussant provides a third opinion and is usually not required to provide a detailed written critique.
NIH critiques are based on predetermined review criteria which may vary due to a variety of factors. But a general overview is available and I highly encourage you read it since I don't wish to repeat that information here. In summary, there are five criteria that go into the assignment of a priority score: significance, approach, innovation, investigators, and environment.
All three initial reviewers assign a priority score that ranges from 1.0 (outstanding) to 5.0 (terrible) using this review criteria. Reviewers are usually given a number of applications (say 8-12) and will usually play all three roles (primary, secondary, and discussant) over the spectrum of applications that they are assigned. Reviewers are asked to utilise the entire scoring range (i.e., the best application they review should be toward 1.0 and the worst should be toward 5.0 in the context of all the applications that they have reviewed). This typically never happens in practice.
The reviewers are given a deadline to enter their scores and written critiques into the online NIH review system (in the past, it was sent to the SRA). When they do so, they are allowed to see the other reviewers' scores and critiques. At this stage, they are no longer allowed to make any changes to their scores and critiques.
The first steps that determine the success or failure of your application are in place. All reviewers are not equal and in general weigh different criteria differently, albeit in a consistent manner across study sections. For example, I typically tend to place the highest value on the investigator and the approach. Some may value innovation the most. My view is that the SRA, who has worked with the reviewers before, typically has a good idea of what reviewers' preferences are. This is all fuzzy of course, and just as there is consistency to human behaviour, there is also a certain stochastic element to it. Thus the initial assignment of reviewers itself, an action that is largely controlled by the SRA, has an impact on how well your application will score.
It should also be obvious by now that the SRA has a huge responsibility to make sure your application is reviewed fairly, which is most likely possible when the right match exists between the scope of your application and the reviewer's interest. In my experience, the SRAs do an excellent (and thankless) job in this regard. Investigators can try to control a bit of this process by asking to exclude certain reviewers (which may or may not be honoured), but it also possible to shoot yourself in the foot in this regard (since even though you may think someone may not like you personally, they may be the ones most qualified to review your application).
The bottom line is that the primary, secondary and discussant reviewers are the ones who mostly determine the outcome of your application at this stage.
At the meeting, the SRA would've compiled a list of all the applications to be discussed and the scores of all the reviewers and sorted it by the average score for different types of applications (R01, R21, etc.). The reviewers who looked at the applications will typically be present in person (though some may only be available by a conference call and there can also be mail-in reviewers). There will be a chairperson who will conduct the study section discussion. The chair will start with the list of the grants and a cutoff average score assigned by the SRA (typically one that would include the top 50% of the grants, i.e., the median score for a particular type of application). The chair then looks at each application at or below that score asks the entire study section if there are any objections to "streamlining" applications below that cutoff.
Streamlining (or triaging or unscoring) means that the application will be given a status of "unscored" and will not be discussed further (which typically means the application will not be funded). This is done since the study section time is limited and NIH will only fund a certain percentage of applications, and that percentage is well below 50. However, an application that is discussed will usually have a summary of the discussion provided by the SRA, which will help the applicant understand the score better.
Sometimes when an application particularly stands out (diverse scores, represents an interesting topic to a study section member), any member of the study section can ask that an application be discussed. Conversely, even if an application has a higher score than the cutoff, a study section member can submit that it be streamlined (if there are no objections, it will be). Sometimes further discussion may be warranted just to determine whether it should be streamlined or not.
Again, at this stage knowing people on the study section pays off. In my experience, applications by people who have established reputations, or applications by people whose work is familiar to a member of the study section, are slightly more likely to not be streamlined even if they score below the median. Even though discussing an application that is supposed to streamlined may not improve its score very much, the summary of the discussion may help an applicant revise their application better than if it had not been discussed.
Once approximately 50% of the applications are taken out of the picture, then the real work of the study section begins. Each application is considered turn by turn. First the chair asks each of the reviewers to list their scores (which now could be different from what they submitted on the online system, due to a reread or due to consideration of the other reviewer's written critiques). If the score now falls below the median score, it may be a candidate for streamlining. Usually the scores remain about the same and almost all applications that don't get streamlined do get discussed.
The chair then asks the primary reviewer to provide a brief summary of the application and their critique, which typically takes five minutes. In my view, if your application has made it this far, it is the most important portion of the study section process. The rest of the study section has probably not read your application. The very first impressions they will form of your application wholly depends on the words that come out of the primary reviewer's mouth. Thus the reputation, delivery, and presentation of the primary reviewer has a tremendous impact on how your application is perceived. Even if the written critique is extremely meticulous and fair and results in a high score for your application, if the primary reviewer is not able to convey the essence of your grant (for whatever reason), it will not be well received. Conversely, an outstanding presentation by the primary reviewer for an excellent but not outstanding application may result in a higher score for the application after the discussion.
The secondary reviewer now has heard the primary reviewer's oral summary. Since there is emphasis on moving things forward, the secondary reviewer focuses on points that the primary reviewer may have missed and provides additional information. The discussant then adds their point of view, particularly if different from the one or both of the previous reviewers.
Depending on the study section and the personalities of the people, there may be strong disagreements. The discussion between the initial reviewers is what the members of the study section use to assign their scores. Thus it behooves you to present your grant SO clearly that all reviewers will agree on what is being said. The reviewers do not have the time to discerning your intent (though some kinder reviewers may try).
At some point the chair will stop the discussion and redo the scoring. Again, the three initial reviewers are polled for their scores and they may or may not change the scores based on the discussion (for example, one reviewer may have convinced the other, or reviewers may disagree). The entire study section then scores usually within the ranges set by the initial reviewers. There's a strong motivation to do this since scoring outside the range requires the outside scorer to write a critique explaining why they are doing so. However, this does happen once in a while.
The chair then moves on to the next application. The average of all the scores given by the members of the study section is your overall priority score. This, along with the summary statement, and the written critiques of the reviewers, are made available to you. The priority score may or may not be converted into a percentile which is then used by the Program Officer (PO) responsible for your application as one factor in deciding whether to fund your application. The score however is usually the primary factor. The NIH website goes into a lot about how funding decisions are actually made.
So again, the ability of a reviewer to argue on behalf of or against your application plays a huge role in the final priority score that you obtain. Fortunately, most of the reviewers are very bright people in academic professions and are used to doing this. This is why novice reviewers are usually secondary reviewers or discussants since it takes a while learn the system. So even though some reviewers may have an off-hour (or day!), the system ends up working really well. In fact, typically there is no high-scoring application that doesn't deserve it. What happens typically is that excellent (but not outstanding) applications/projects may fall through the cracks due to uncertainties of the review process that arises from the subjective choices I've listed above. This is particularly relevant to keep in mind during difficult funding climates.
Not thoroughly completed.
The NSF investigator-initiated application review system has some parallels to the NIH system. In NSF, the scientific review is conducted by the Program Officer (PO). Again, when you submit a grant, it is assigned to a PO. The PO then solicits reviewers who are asked to submit their critiques online and rate the grant in terms of its strengths and weakness. See NSF's Grant Proposal Guide, particularly the chapter on NSF Proposal Processing and Review.
There is no distinction made between the assigned reviewers (i.e., no primary or secondary reviewers) who typically rate the grant as some combination of Excellent/Very Good/Good/Fair/Poor. The reviews are usually submitted before the panel meeting.
The panel then convenes and the PO controls the review session going through the applications. The reviewers of each application usually present their views on the application followed by a general discussion. Once the discussion is finished, the application is generally put in one of three categories: "highly competitive", "competitive", and "not competitive". One of the assigned reviewers writes a panel summary, summarising the discussion that occurred. The rating by the panel is used by the PO to make funding decisions.