Suppose you want to hire an assistant to alleviate the mundane tasks of your job. Every day that you have the job search open, an assistant comes for an interview. Immediately after the interview you have to choose whether to hire or not hire the interviewee. The hire/no hire decision is irreversible — i.e. if you reject someone, you can’t later decide to hire them. Under these conditions, how do you determine which candidate to hire? The trade-offs are intuitive: if you don’t hire, you incur the overhead of interviewing and don’t reap the benefits of having an assistant; if you hire too early, you may be missing out on a better assistant that’s potentially coming up as a candidate.
The so-called secretary problem (also known as the marriage problem, sultan’s dowry problem, and best choice problem) is a member of a class of mathematical problems called Optimal Stopping Problems, and has been studied in the fields of applied probability, statistics, and decision theory. There is a closed-form solution: optimal stopping theory prescribes always rejecting the first ~37% of the applicants that are interviewed and then stopping at the first applicant who is better than every applicant interviewed so far (or continuing to the last applicant if this never occurs).
Although there are some stylized conditions in this problem, it is not too dissimilar to the decision process that we face when dating. For an example, take the constraint that you have to give an immediate decision to every candidate. While it is possible (and common) to date multiple people at the same time, except for the truly [un]committed, this number peaks at a handful of people — so the process of getting to know someone enough to really get an idea of where they fall in your personal dating distribution can be thought of as a serial selection like in the secretary problem. A more significant complication in applying optimal stopping point theory to dating is that dating is a double-opt-in process. In the secretary problem, there is the assumption that every applicant wants to work for you; in dating, just because you choose someone, doesn’t mean that they’ll choose you back.
Despite these limitations, we can indeed learn something about dating from optimal stopping theory. The fundamental idea behind the mathematical solution to the secretary problem is to interview enough candidates to get a reasonably confident idea of what the distribution of applicants looks like, and once you have an idea of that distribution, make a (now more informed) choice about each subsequent candidate. The initial 37% trial phase essentially serves to calibrate our expectations.
When it comes to dating, we instinctively know this — most people do not marry their high-school sweetheart (although it’s been known to happen, and successfully!). And dating people is not the only way to get an idea of the distribution. When we interact with humans in any context, we can ostensibly gather data that helps us create this distribution of what humanity is like.
However, some of us don’t do enough ‘primary research’ before we reach the point of commitment. We commit too early, and perhaps later meet people that we think might be better for us, and are always left wondering “what if?”. And some of us likely stick with ‘bad candidates’ too long. We know, sometimes through cognition and sometimes through instinct, that somebody doesn’t fall in the right part of the distribution, that we’ve seen others that would have been better matches for us, that there’s no long term potential. And yet we persist, perhaps because it’s fun, or for companionship (because someone is better than no-one), or because breaking up is hard and painful, or the nagging feeling that perhaps we’re being too picky, that we need to settle down sooner or later.
For both of these types of people, keeping an idea of optimal stopping theory can give some structure to the process, a cognitive foundation to back up our intuition, a language to describe our experience, and perhaps even an impetus to make difficult choices. There are often no easy answers, but there is a magic number: about 37%.