Going on the market can be one of the most dispiriting things that a recent Ph.D. can do. While nasty rejection letters from academic journals are par for the course, usually even the most snide letters from the meanest of journal editors contain some kernel of insight into why the paper was considered weak and what you need to do to improve it.
On the other hand, job search rejection letters are almost uniformly unhelpful. Most simply thank you for the application, note the special nature of the applicant pool this year, regret that you were not selected, and wish you well. Usually they are only two or three sentences long. They never tell you why you were not selected.* Don't give up hope!
One of the common mistakes is to assume that if you were rejected at a Research I university or an elite private liberal arts college, then you should set your sights on lower targets. This results in a lowering of both your hopes and the jobs you apply to each cycle. Pretty soon, you find yourself applying for jobs at the local MacDonalds....
Don't fall into this negative feedback loop. Job searches are entirely random processes. Unlike college or grad school, where you at least felt some control over the process -- i.e., that excellence was rewarded, job searches are not based on meritocratic standards.
Graduates from competitive Ph.D. programs are often punted from searches at smaller colleges because there is an underlying assumption that they will not stay around, they are merely using the program as a stepping stone in their career (this is a prejudice that you will need to actively address in your cover letter or during your on-campus interview). No one wants to hire someone who will only stay for 2-3 years before moving on.
You can be overqualified or underqualified or nonqualified. Your application might be rejected because one of your courses overlaps a favorite of one of the faculty. Or in one of your papers, you did not cite the right people. Or one of the search committee members saw you getting a bit too sauced at the open bar at the last national meeting. Or you put too much emphasis in research in your cover letter for a liberal arts college, or too much on teaching at a research university. Or your application was read at 3:50 on a Friday afternoon and people were tired and grouchy.
You cannot take rejection letters from job searches personally nor should you read too much into them. If you ask, a very few search chairs might tell you afterwards why you were rejected, but the great majority won't because of the legal issues involved. I.e., if they tell you that you were rejected because of X characteristic and it wasn't the case that you were X, they are afraid you might sue. That explains why rejection letters are so blandly written and generic.
If you are dying to know, you need to contact one of the faculty by phone or in person at a conference and ask. It will be extremely awkward. They will not want to leave any sort of a paper trail so e-mail inquiries will be ignored. And ultimately, the information you get will most likely not be helpful. Faculty positions are all about qualitative 'fit' and rarely about quantitative comparisons. Be the best scholar you can be and the right college position for you will come along. Trust me.
p.s. That all being said, one thing you need to be very careful about is one of your references deliberately torpedoing you. This happens much much much more often that we like to think. For this reason, it's helpful to apply early on to a school where you know someone on the search committee who might alert you to negative letters in your file.
* Usually this is done for legal reasons. As they say, anything you say can and will be used against you in a court of law. Any reason that a job letter gives for denial of a position could conceivably be used in a lawsuit against the college for hiring discrimination. I.e., if the letter said: "We didn't hire you because your job talk was terrible" and you had a speech disability, you could conceivably sue the university under the ADA. Not likely, but this is what keeps university lawyers justifying their keep.
I'm finding these posts very helpful, please continue. =)
Mac -
Thanks for the kind words. Right now, I'm working through a list of questions that I had when I was in grad school (and wish someone had told me). If you are curious about anything, feel free to pose the question here on the forums or by e-mailing me.
Karen