That's a large question especially regarding X.509 security. Usually with self-signed certificate, you have to import the initial CA used to sign this certificate (usually the self-signed CA used by the application) into your keyring (to be sure that you are connecting to the right server). It's usually required because your client application will check at the connection if the certificate is signed by a known and trusted CA or matching an existing certificate in your keyring.
In C#, you can check the System.Net.Security namespace and especially the SSlStream class for more details.
If you need to import certificate in your keyring, you can use various interfaces to access the keyring from the GUI, using the command line "Certutil.exe" or via the various APIs.