You should follow @Slauma's advice from comment and change the code because this is wrong behavior for persisted properties. Even if following description works it will be highly error prone because you will always have to query data in specific way.
Your problem can be divided into multiple parts based on the way how you load entities.
Lazy loading:
If you load Parent and
Child` are lazy loaded you cannot achieve reverse loading.
// Now parent is loaded
var parent = context.Parent.First();
// Even with lazy loading enabled and your setter accessing nav. property it
// should not load child collection because lazy loading should be temporarily
// turned off during entity materialization to avoid unexpected lazy loads
(untested) You can try manually load all child first and request parent after that:
// Now all child for given parent are loaded
var child = context.Child.Where(c => c.Parent.Id == ...).ToList();
// Now parent is loaded and if you are lucky it configures navigation property
// to loaded child prior to evaluating your setter - I guess setter will be
// evaluated prior to fixing navigation properties so it will not work
var parent = child[0].Parent;
The explicit loading will suffer the same issue.
Eager loading:
The problem is the same and it is based on the way how Include works.
So if you include child to parent the parent will be materialized first.
var parent = context.Parent.Include("Child").First();
Reverse operation in this case most probably doesn't help because calling
var childs = context.Child.Include("Parent").Where(...).ToList();
will evaluate records one by one and each record will contain single child and parent so I think the first time you access the parent you will have only single child and again you will be dependent on the order of operation EF does (the same problem as with lazy loading).