The anonymous cipher suites don't perform any authentication. Even with encryption, you're talking over an encrypted channel to a remote party, but you can't be sure who that is, so it could be a man in the middle (MITM). The TLS 1.1 specification says the following about anonymous cipher suites:
The following cipher suites are used for completely anonymous
Diffie-Hellman communications in which neither party is authenticated.
Note that this mode is vulnerable to man-in-the-middle attacks and is
therefore deprecated.
In short, don't use them (except for testing perhaps), or if you're really sure that there won't be an active MITM. (The TLS 1.2 specification is even more strongly worded against their usage.)
If you want to use TLS securely, you'll need a way for the client to verify the identity's remote party. In the vast majority of cases, this is done using an X.509 certificate. In particular, for the TLS_DHE_RSA_WITH_AES_128_CBC_SHA
cipher suite, you'll need requires a certificate with an RSA public key (which explains the error message you get).
You should generate or request a certificate (self-signed or via a CA, perhaps your own, depending on how the clients can be configured) that identifies correctly the server. Not only the certificate will need to be trusted by the client (either explicitly or via an established CA), but its identity information will need to match the server's, under the name the client is trying to connect to it.
From your example, it's not clear whether you're implementing HTTPS (for example) or your own protocol over SSL/TLS. RFC 6125 is the latest specification in this area, which unifies practices around multiple protocols (e.g. HTTPS, LDAPS, IMAPS, ...). Since it's quite a recent RFC, few libraries explicitly implement it (although they may do in practice, since it consolidates best practices). In doubt, it's usually sensible to adhere to the RFC 2818, section 3.1 (HTTP over TLS) guidelines on this subject:
If a subjectAltName extension of type dNSName is present, that MUST
be used as the identity. Otherwise, the (most specific) Common Name
field in the Subject field of the certificate MUST be used. Although
the use of the Common Name is existing practice, it is deprecated and
Certification Authorities are encouraged to use the dNSName instead.
Matching is performed using the matching rules specified by
[RFC2459]. If more than one identity of a given type is present in
the certificate (e.g., more than one dNSName name, a match in any one
of the set is considered acceptable.) Names may contain the wildcard
character * which is considered to match any single domain name
component or component fragment. E.g., .a.com matches foo.a.com but
not bar.foo.a.com. f.com matches foo.com but not bar.com.
In some cases, the URI is specified as an IP address rather than a
hostname. In this case, the iPAddress subjectAltName must be present
in the certificate and must exactly match the IP in the URI.
(RFC 6125 explicitly discourages the use of wildcard certificates.)
For practical reasons, IP addresses in certificates are not ideal (I don't think many commercial CAs would generate such certificates anyway). If you can, use the Subject Alternative Name extension, failing that, putting the host name in the CN=
RDN of your Subject DN should be sufficient. There are notes on how to generate a CSR/certificate with a SAN in this answer. In particular, it's possible using the -ext
option of Java 7's keytool
(e.g. -ext san=dns:www.example.com
); note that the keystore generated by Java 7's keytool should be usable on previous versions of the JRE too.
In addition, if you don't have any other code than SSLContext.getDefault()
(you could also use the default SSLServerSocketFactory
in this case), you'll need to specify the keystore: the JRE doesn't have a default value for this (as opposed to the trust store). This can be done using the javax.net.ssl.keyStore
(and related system properties). There is more on this topic in this answer (for example): https://stackoverflow.com/a/6341566/372643