I believe the easiest approach to determine all families that have been instantiated is to retrieve all family instances.
From the instances, you can determine the family symbol and the family itself, and be certain that it has been instantiated.
If there is no instance, you will get no family or family symbol entry.
Oh, on re-reading, I see that you want the opposite, a list of all families regardless of whether they have been instantiated or not. Oh no, a list of all family symbols of a specific category, regardless of whether they have been instantiated or not.
That is in fact already demonstrated by one of the numerous filtered element collector snippets in The Building Coder samples CmdCollectorPerformance
module:
https://github.com/jeremytammik/the_building_coder_samples/blob/master/BuildingCoder/BuildingCoder/CmdCollectorPerformance.cs#L294-L332
To be precise, the method GetFamiliesOfCategory
implemented there retrieves all families of a given category. You can easily adapt that to retrieve the family symbols instead.