Multiple zero or one to one relationships EF 6 Code First

Question

I have a contact table

public class Contact : EntityBase
{
    public int TrivialContactProperty { get; set; }
    ...
    public virtual FiContact FiContact { get; set; }
    public virtual PuContact PuContact { get; set; }
    public virtual TrContact TrContact { get; set; }
}

and then a FiContact, PuContact, and TrContact table.

public class FiContact : EntityBase
{
    public int TrivialFiProperty { get; set; }
    ...
    public virtual Contact Contact { get; set; }
}

public class PuContact : EntityBase
{
    public int TrivialPuProperty { get; set; }
    ...
    public virtual Contact Contact { get; set; }
}

public class TrContact : EntityBase
{
    public int TrivialTRProperty { get; set; }
    ...
    public virtual Contact Contact { get; set; }
}

The contact table should have a zero or one relationship with all three other tables. So a contact can exist without any of the other three, or it can be related to one or two or all three of them.

Using fluent API I tried to configure this, after doing some research, and I came up with:

modelBuilder.Entity<FiContact>()
    .HasRequired(r => r.Contact)
    .WithOptional(o => o.FiContact);

modelBuilder.Entity<PuContact>()
    .HasRequired(r => r.Contact)
    .WithOptional(o => o.PuContact);

modelBuilder.Entity<TrContact>()
    .HasRequired(r => r.Contact)
    .WithOptional(o => o.TrContact);

But I am still getting the following error when I try to add a migration for this change:

FiContact_Contact_Source: : Multiplicity is not valid in Role 'FiContact_Contact_Source' in relationship 'FiContact_Contact'. Because the Dependent Role properties are not the key properties, the upper bound of the multiplicity of the Dependent Role must be '(asterisk symbol here)'.

PuContact_Contact_Source: : Multiplicity is not valid in Role 'PuContact_Contact_Source' in relationship 'PuContact_Contact'. Because the Dependent Role properties are not the key properties, the upper bound of the multiplicity of the Dependent Role must be '(asterisk symbol here)'.

TrContact_Contact_Source: : Multiplicity is not valid in Role 'TrContact_Contact_Source' in relationship 'TrContact_Contact'. Because the Dependent Role properties are not the key properties, the upper bound of the multiplicity of the Dependent Role must be '(asterisk symbol here)'.

From more research I saw that the primary key on the dependent entity is supposed to also be the foreign key? The only problem is that all of my entities inherit from a class "EntityBase" which defines common fields in all entities, including the primary key:

public abstract class EntityBase : IEntity
{
    [Key]
    public int Id { get; set;}
    public bool IsActive { get; set; }
    public DateTime CreatedOnDate { get; set; }

    public int? CreatedByUserId { get; set; }

    [ForeignKey("CreatedByUserId")]
    public User CreatedByUser { get; set; }

    public DateTime LastUpdatedOnDate { get; set; }
    public int? LastUpdatedByUserId { get; set; }

    [ForeignKey("LastUpdatedByUserId")]
    public User LastUpdatedByUser { get; set; }
}

Is there a way to make this kind of table/entity relationship work with EF 6 Code First? Any help getting this type of relationship to work would be much appreciated.