This sparks a memory from back in the day i was all into tcp servers and clients. I thought the OS took time to clean up ports from crashed or aggressively killed applications.
If this is not the case, it might be because your application started another process, i.e a child process.
Anyway, i think you should start from first principles here, if you are in control of the process you are trying to kill, its better to exit this application gracefully, and not trying to just kill it. Also if this application has a main window you could try calling
p.CloseMainWindow();
p.WaitForExit();
Also make sure it hasn't started any child processes, as you "may" have to close them as well.
If you are in control of this application, try binding with ReuseAddress
however this is just trying to make up for bad design anyway
Allows the socket to be bound to an address that is already in use.
also you may want to take a look at assicoated socket options
Socket.SetSocketOption Method (SocketOptionLevel, SocketOptionName, Int32)
SocketOptionName Enumeration