Going with the border checkpoint analogy, I would say:
Sync border employees
You have 4 incoming lanes and 2 officers. The officer handling the incoming monstertruck, Jack, does not know the exact rules for monstertrucks, so he calls the office.
Now, if he works sync, he stays on the phone waiting for a response. So he is not able to handle anything else - his colleague Mary will have to do that.
Fortunately Mary works parallel, so she can process the 3 unblocked lanes in the meantime. However, because she also works sync, she only one processes one vehicle at a time.
So when she has to check the rules for a motorcycle with a cheetah in the sidecar, she has to call the main office and also stays on the phone for the answer.
Now we have two lanes blocked by pending jobs, and two lanes blocked because there are no employees.
Async border employees
Now if Jack works async - he will not hold the line waiting for a response. Instead he hangs up, and goes to the second lane, and processes another car, while waiting for the main office to return his call.
Because the second lane is a very nervous lady with a stutter, this is taking him quite some time.
But fortunately Mary now also works async, and when she finished processing her second vehicle (or paused it, because she had to check on the cheetah), she can take the return call from the office on the monstertruck. So she can finish processing the monstertruck.
But of course the monstertruck is not gone right after she finished - the driver has log his time spent at the checkpoint. Fortunately Mary still works parallel, so she can start processing a car in another lane.
Costs
Then one day, the Burning Man festival is starting, and a lot of unusual vehicles arrive at the border. These all take a lot of calls to the office, therefore blocking all 4 lanes. So Jack and Mary can only sit around waiting for return calls, while the line of vehicles is growing.
Fortunately, land is cheap in this area, so their boss decides to add 4 more lanes. While some of these extra lanes are also blocked waiting for return calls from the office, at least Jack and Mary keep busy, and don't have to sit and wait for calls. Of course their boss can consider hiring some extra employees to reduce the traffic jam, but he knows they will need housing and training, and the festival will be over soon, so he leaves it as it is...
Recap
Leaning towards ASP.Net:
- lanes (e.g. connections) are cheap, and can be scaled easily (and you should probably increase the connection limit and queue size in IIS for async, e.g. see introduction-reference below)
- employees (threads) are more expensive (e.g. see introduction-reference below))
- 'slow' parts of a job (e.g. I/O, or remote http-requests), should not block your expensive employees (i.e. one should use async)
- NB: jobs may change employees (pun intended), so don't keep data with your employees. E.g. Jack should not put monstertruck-papers in his pocket, because Mary may process it further - he should rather return it, or keep it in a job-dependent storage (don't store data within the thread, but rather in the HttpContext)
Literature
FWIW: I like this technical introduction from 2014 by Stephen Cleary.