C# Dictionary large performance hit when key is struct

Question

I am writing a class that allows for two different indexes with the goal of O(1) lookup for each. Both indexes are currently using Dictionary as the underlying structure.

internal class RayCollection
{

  private readonly Dictionary<ulong, Ray> idLookup_;
  private readonly Dictionary<ProjectionKey, Ray> projectionLookup_;

  public RayCollection()
  {
    idLookup_ = new Dictionary<ulong, Ray>();
    projectionLookup_ = new Dictionary<ProjectionKey, Ray>();
  }

  public void Add(Ray ray)
  {
    idLookup_.Add(ray.Id, ray);
    projectionLookup_.Add(new ProjectionKey(ray.From, ray.To), ray);
  }

  public Ray this[ulong id]
  {
    get => idLookup_[id];
  }

  public Ray this[ulong from, ulong to]
  {
    get => projectionLookup_[new ProjectionKey(from, to) ];
  }

  internal struct ProjectionKey
  {
    ulong From;
    ulong To;

    public ProjectionKey(ulong from, ulong to)
    {
      From = from;
      To = to;
    }
  }

}

This works as it should, but the performance hit from using a struct is enormous! Indexing with ProjectionKey is almost 130x slower.

           Method |       Mean |      Error |     StdDev |
----------------- |-----------:|-----------:|-----------:|
         GetViaId |   5.823 ns |  0.1801 ns |  0.1769 ns |
 GetViaProjection | 646.951 ns | 12.4307 ns | 12.2086 ns |

A fair amount of this overhead is probably caused by having to create a new instance of ProjectionKey every time the index operator is called, and I presume more overhead is caused by Dictionary having to hash the struct to perform the lookup. In either case, I would like to decrease this overhead as much as possible.

So then, my questions are:

• For this[ulong from, ulong to], is there a way I can implicitly treat from and to as a struct without having to allocate it? It seems silly to instantiate a new struct when the variables are adjacent to each other on the stack.
• Is there a better way to approach this, considering a fair amount of overhead is likely being caused by Dictionary hashing the struct

And one last thing: Before any "premature optimization" or "is this actually affecting your performance" posts, yes, I do realize this. I use opportunities like these to better understand C# and .NET from a low level perspective and believe extreme optimization cases like these provide good knowledge.