I did some research and found my own answers, but now I want to share them here with others.
This post is VERY LONG, so look at the headings to figure out what you want to read first. I've updated this post over multiple days. And added new things, some things may be repeated for each main heading.
But at this point in time, I recommend reading "Answer #2" down below. And also read "Some important info going forward".
Some important info going forward:
The reason why you can't just have 2 UDP (or TCP) clients sending packets to each other over WAN (using public IP addresses) is because the devices that receive the packets, like your router, don't know which computer on the LAN (connected to the router) to send it to; it doesn't have enough information on where to send it.
Your router needs to be told, in some way, where to route packets. Achieving peer-to-peer connections requires you to use some method or technique to tell YOUR router where to route things.
"Hay router, I'm gonna send a packet to [THIS IP AND PORT]. I am expecting a response, if you get anything from that IP and port, send it to me"
"Hay router, if you get any packets with [THIS PORT], send them to me"
The following are the ways I've discovered for doing this. Personally, I prefer NAT Protocols like "NAT-PMP" and "UPnP" (Answer #2)
Answer #1 (Hole Punching) (The Last Resort)
I did find this video which honestly helps explains things a lot on the topic.
https://www.youtube.com/watch?v=TiMeoQt3K4g
And a sequel where he made a python implementation
https://www.youtube.com/watch?v=IbzGL_tjmv4
From this, I managed to make a C# implementation of NAT Hole Punching for UDP. I assume this same technique could be applied to TCP as well? If anyone could comment on that, or if the technique has to be modified to work for tcp.
// This code could be in Main, or another static function
Console.WriteLine("enter the other persons IP:");
string destinationIP = Console.ReadLine();
Console.WriteLine("type a message you want to send:");
string inputMessage = Console.ReadLine();
int sourcePort = 25000;
int destinationPort = 25001;
// Send a "dummy packet" which "punches a hole" in your router, so each router knows where to route the packets.
UdpClient sender = new UdpClient();
sender.Client.Bind( new IPEndPoint(IPAddress.Parse("0.0.0.0"), sourcePort) );
sender.Client.SendTo(new byte[] { 2 , 4 , 6 , 8 }, new IPEndPoint(IPAddress.Parse(destinationIP), destinationPort) );
sender.Close();
Console.WriteLine("Punched Hole...");
// Start a udp client to listen for incoming packets
Thread listenerThread = new Thread(() => {
UdpClient listener = new UdpClient();
listener.Client.Bind(new IPEndPoint(IPAddress.Parse("0.0.0.0"), sourcePort));
byte[] buffer = new byte[1024];
Console.WriteLine("Server Listening:");
while(true) {
int bufferSize = listener.Client.Receive(buffer);
Console.WriteLine("Recieved Message:");
string message = Encoding.UTF8.GetString(buffer, 0, bufferSize);
Console.WriteLine(message);
Console.WriteLine();
}
});
listenerThread.Start();
// waiting a bit of time just in case
Thread.Sleep(5000);
// Start a different client that will be sending actual data between the clients, through the punched holes
UdpClient sender2 = new UdpClient();
sender2.Client.Bind(new IPEndPoint(IPAddress.Parse("0.0.0.0"), destinationPort));
byte[] sendData = Encoding.UTF8.GetBytes(inputMessage);
Console.WriteLine("Preparing to send data...");
for(int i = 0; i < 10; i++) {
Thread.Sleep(1000);
sender2.Client.SendTo(sendData, new IPEndPoint(IPAddress.Parse(destinationIP), sourcePort));
}
I have a feeling there is a more elegant way to do this, but this does indeed work. I was able to test this code with a friend (because I don't want to spend money to do a simple test on a cloud server).
Things to note for other newcomers doing this:
This code has to run on both computers (yours and the other peer) at relatively the same time. Not at exactly the same time though, but within a span of seconds. Because "hole punching" has a very short "expiration date". So they have to happen close to each other in time. You can coordinate this over Discord or another chat program.
You could also make the program send out "dummy packets" once every few seconds until a connection is made. So you wouldn't have to time things necessarily
On that note, even after a hole punch has succeeded, if no data is being sent back and forth for many seconds (it probably varies between routers), then the "hole" that was punched will be "patched up", and you would have to restart the process again. So you may need to also send dummy packets once every few seconds to keep the connection alive. Or you can be smart about it and only send a dummy packet if no other 'actual packets' were sent within the last few seconds.
Answer #2 (NAT Protocols) (The Better Way)
"NAT" is a thing that most routers (I would imagine) are capable of today. NAT is responsible for figuring out which PC to send an incoming packet to. But sometimes your router needs a little bit of input from its connected PCs to know what to do.
This fact also applies to hole punching, but ideally, you might not want to send "dummy packets" just to "punch a hole" and then send "keep-alive packets" every 10 seconds to keep the connection open. It's a waste of network resources.
Ideally, you'd want a PC to talk directly to the router to set up a "temporary port forward" that might last for a couple of hours, and you can "renew the port mapping" once an hour by sending a message to the router; so no outbound WAN packets are sent.
This is what "NAT Protocols" allow you to do; Portforwarding directly from your application.
So how can I do this in C#?
I'm glad you asked!
https://en.wikipedia.org/wiki/Port_Control_Protocol#PCP_as_a_solution
This Wikipedia page lists quite a few libraries for different programming languages, at the bottom. But for C# I have chosen to use a NuGet package called "Mono.Nat" which supports "NAT-PMP" and "UPnP". From what I've heard, the "NAT-PMP" protocol is the preferred one, but either is probably good.
Here is their Github which has links to their NuGet as well. But you can just search up "Mono.Nat" in the NuGet Package Manager of your IDE (VS or Rider).
https://github.com/alanmcgovern/Mono.Nat
Also, here is some example code I used for testing NAT-PMP, and actually performing the software port-forward. This code essentially sends a message (packet) to your router saying: "Hay, can you forward incoming packets with [THIS PORT] to me?". It may not always succeed depending on your scenario. And this code does not properly handle those errors completely. But it will tell you if there was some sort of error. So this code should be improved for production uses.
public static bool SetupPortForward() {
int portToForward = 25000;
bool portMapSet = false;
bool fail = false;
NatUtility.DeviceFound += async (object sender, DeviceEventArgs args) => {
try {
INatDevice device = args.Device;
// Only interact with one device at a time. Some devices support both
// upnp and nat-pmp.
Console.ForegroundColor = ConsoleColor.Red;
Console.WriteLine("Device found: {0}", device.NatProtocol);
Console.ResetColor();
Console.WriteLine("Type: {0}", device.GetType().Name);
Console.WriteLine("IP: {0}", await device.GetExternalIPAsync());
Console.WriteLine("---");
await device.CreatePortMapAsync(new Mapping(Protocol.Udp, portToForward, portToForward));
//await device.DeletePortMapAsync(new Mono.Nat.Mapping(Mono.Nat.Protocol.Udp, 25000, 25000));
//Mono.Nat.Mapping[] mappings = await device.GetAllMappingsAsync();
//foreach(var mapping in mappings) {
// Console.WriteLine(mapping);
//}
portMapSet = true;
} catch(Exception e) {
Console.WriteLine(e.Message);
fail = true;
}
};
// Replace "192.168.0.1" with your default gateway, or use a function which finds your default gateway automatically
string defaultGateway = "192.168.0.1";
NatUtility.Search(IPAddress.Parse(defaultGateway), NatProtocol.Pmp);
while(true) {
if(portMapSet || fail) break;
Thread.Sleep(1);
}
return !fail;
}
Note: This code notably needs namespaces "Mono.Nat" and "System.Net". So you will also need the NuGet package "System.Net.Sockets" so you can use the "IPAddress" class.
But if you are using a different C# library for networking (like SFML maybe?) You could alternatively use the NuGet package "System.Net.Primitives" which just provides the IPAddress class.
I should also note that my example code is based off of code inside the Mono.Nat Github repository; At "Mono.Nat.Console/Main.cs". So there may be some more useful info in there that I missed.
https://github.com/alanmcgovern/Mono.Nat/blob/master/Mono.Nat.Console/Main.cs
But the example code I show above SHOULD automatically setup a port forward for port 25000, but you can change that. The code also assumes everyone's default gateway is the same "192.168.0.1". But it may be different for you, so make sure to check that using "ipconfig" in a command prompt. That ip is not my gateway, I got it from this youtube video.
https://www.youtube.com/watch?v=pCcJFdYNamc
Lastly, the port forward that is setup is TEMPORARY, so it will EXPIRE after a period of time. You can change this period of time in code, but the default (I think) is 2 hours, which is decent. Also, you can rerun this code to "renew" the expiration date, and keep it going effectively for the life of the program. The program doesn't need to make sure to "close" the port forward on exit. So if the program crashes, the port forward will "expire" and close itself on its own.
But how do I check to see if it actually worked???
#1 You can check with the Mono.Nat library.
The commented out code "device.GetAllMappingsAsync()" can list off any PMP or UPnP mappings that were made, but for some reason, it only works when the "NatUtility.Search" function is using the UPnP option.
#2 Check your router's logs with your web browser
Alternatively, you can see if your router's HTML settings menu has a section for displaying logs and if you can view all existing Port Forwards.
To do this, you need to get your router's default gateway (using ipconfig) and enter the default gateway into your browser. It will take you to a page generated by your router. This page will look different for each router brand and model.
For my ASUS router, I clicked on "System Log", and then I clicked on a tab called "Port Forwarding", which lists info about existing port forwards. It shows manually made ones, and ones that were made in software (PMP). Note, it may be called "port mapping" instead.
You can keep that page open, run the example code I showed above ^, and if it succeeds. You will see a new entry in the port forward logs. Make sure to refresh if there is a refresh button on the screen. And it will show your PC's private IP. That's how the router knows where to forward the packets. And it will forward packets no matter what the external IP was.
#3 Run a program on an external PC that sends packets to your IP
Without any existing port forwards, if you receive packets from another public IP, you're not going to get them, obviously.
So if you have a program that is actively listening for UDP packets, but no port forwarding for the port your using, nothing is going to happen. I know, I sound like a broken record!!!
But my point is, the best way you could probably test it, is to verify that you can receive packets from an external (public) IP in the first place; from another PC out there on the interwebs.
If you're using System.Net.Sockets, this would involve creating a new UdpClient, binding it to the port you forwarded (bind ip would be 0.0.0.0), and then using one of the "receive" functions to listen for an incoming packet.
The tricky part is finding a computer with a different public IP, and running a program that sends a UDP packet. And see if the packet gets received on your PC. If it does, congratulations, it worked!
For me, the simplest thing would be to find a friend (Discord maybe?) who is willing to help. You'd have to send them a program that would send the UDP packet to your PC. You could use another existing program like Netcat or NCat (Windows version) to send the test packet.
https://sectools.org/tool/netcat/
https://nmap.org/ncat/
You can then run your program (which listens for packets), but don't run the port forwarding code yet. Then, your friend will run the program which sends the packet to your pc. Nothing should happen (obviously).
But then, you re-run your program, but include the port forwarding part before listening for packets. When your program is ready, tell your friend to re-run their program, and you should receive their packet. IT'S MAGIC!!!
But if nothing happens, then the port forward probably didn't work. And you may need to make some tweaks or change something to make it work.
But ultimately, for some people in the world. This might not at all be possible. Some people live in apartments, or in motels, or are on vacation in a hotel, using the hotel's WiFi. So you may be unfortunately restricted from making temporary port forwards. For people who have decent "access" to their home's router. This should be possible though. But this method does help a lot more people to be able to host [game] servers on their PC without needing to have direct access to their router's settings. Unfortunately, this won't work for absolutely everyone. And hole punching will become the one fallback method for some of you.
Lastly, I want to show some example code that you can use to test the port forward.
This code is for the server (the packet receiver, aka you).
int sourcePort = 25000; // should be the same as your port forward
UdpClient udpServer = new UdpClient();
IPEndPoint localEndPoint = new IPEndPoint(IPAddress.Parse("0.0.0.0", sourcePort);
udpServer.Client.Bind(localEndPoint);
byte[] buffer = new byte[1024];
EndPoint incomingEndPoint = localEndPoint;
int bufferSize = udpServer.Client.ReceiveFrom(buffer, ref incomingEndPoint); // incomingEndPoint will contain the public ip of the sender
Console.WriteLine($"Incoming Packet From: {incomingEndPoint}");
string message = Encoding.UTF8.GetString(buffer, 0, bufferSize);
Console.WriteLine($"Message: {message}");
Console.ReadLine();
And the following is the code that would run on your friend's PC (or another remote PC).
// You probably don't need to use this port for the sender
// The sender can use any port they want, and their router will use NAT hole punching to figure things out.
// But both the client and server could forward this port if they want. But it's not needed. Only the server needs to forward its ports.
int sourcePort = 25000;
UdpClient udpClient = new UdpClient();
IPEndPoint localEndPoint = new IPEndPoint(IPAddress.Parse("0.0.0.0", sourcePort);
udpServer.Client.Bind(localEndPoint);
byte[] sendData = Encoding.UTF8.GetBytes("Hello! This is the UDP client on the other end! If you see this, it worked!!!");
Console.WriteLine("Please enter the Public IP you wish to send the packet to, and hit enter:");
string destinationIP = Console.ReadLine();
// Here, sourcePort is refering to the server's port. This does need to be the same as what's forwarded.
// So this port could be different than the port that was used in the "Bind" function, and "localEndPoint".
IPEndPoint destinationEndPoint = new IPEndPoint(IPAddress.Parse(destinationIP), sourcePort);
udpClient.Client.SendTo(sendData, destinationEndPoint); // send it off!
Console.WriteLine("sent!");
Console.ReadLine();
You can copy/paste these lines into your editor but make it so only one or the other will run. Depending on if you press "S" or "R". For either "Send" or "Receive". So you can just have 1 program instead of 2. Your friend will have to enter your public IP. You can find your public IP by googling "my ip" or something like that. When you receive their packet, it will also print their ip. As a confirmation of where the packet came from (it could come from any ip). And this is normal, your computer also needs to know the sender's ip so it can send a return message if it wanted to.
But that is it.
This example code should give you a solid idea of how to test and make sure your "software-based port forward" actually worked. And it gives you an idea of what you may need to set up in order for the code to even work in the first place. Because sometimes the issue lies in misunderstanding the API and not doing the Bind correctly, or using the wrong ports in the wrong places. So this way is guaranteed to work. But there are other ways to use the UDPClient class.
Also, I have not tested any of this for TCP, but I suspect (and hope) it will work. You just have to change the port forward code to be for TCP.
Conclusion:
I know this is a lot to read, but it details all the stuff I've learned about this subject. And I want to put it all in one place and organize it as best as possible. So this can serve as a good resource for other [game] programmers who want this to be an option in their application.
I have done many searches trying to find A SINGLE article/post that explains THE THING that is needed. Many of them talk about stuff that leads you in the right direction. But I feel like they never said enough.
I think many game developers would want a very simple multiplayer option in their game, that requires no pre-configuration, so people can play and enjoy their games with their friends. And I feel like this info isn't super well-known or shared that much, considering how many games just resort to using a "dedicated server" approach. Which requires manual port forward and direct access to your router's settings; not very accessible... But rants aside!
This post I think will help people who are working in C# at least. And if you're coding in a different language, then take all the stuff I talked about and find the C++, C, Java, Python, etc equivalent of it. Because they do exist. The concepts that I've talked about are "language agnostic", meaning they can apply to any language. You just need a library that allows you to interface with these "systems", like NAT-PMP. (You can now google "NAT PMP C++", and find exactly what you need)
Lastly, I would also want to extend this list with other options/methods that may be preferred in some cases. There may be more modern technologies than NAT-PMP, but for now, NAT-PMP works perfectly for me.
