The association line on the diagram shows one association. There could be many associations, or none of them, that doesn't matter. We are speaking on one concrete association:
a doctor can have many patients.
So, your second interpretation is also correct, but it speaks on ONE exemplar of associations only.
If we'll recall, that the number of associations is arbitrary, for the whole system we are having the first sentence.
An association can be realized so, that every patient will have a reference (possibly null one) to a doctor. We could realize it as a list of patients for every instance of doctor. Or both - for easier search, for example. But all this information speaks only about ONE relationship - one doctor to many patients.
Notice, that there could be 2(or more) such relationships. For example, a doctor had visited today some patients. It is the relationship that we'll name DoctorIsActive. And the other one - doctor was visited by some patients. It will be another relationship, PatientIsActive. So, we'll have two lines connecting these blocks, with different names on them, both one-to many. But we don't have two doctors because of it. It could be even more complicated - for example, a doctor can make on operation a day.

Don't be ashamed - I had sometimes the same problem with understanding this fact, too. And as I know, it is very widespread misreading.
Look here for more info on associations.