Create HTTPS test server for any client

Question

The server created by NewTLSServer can validate calls for a client that is explicitly created from it:

ts := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintln(w, "Hello, client")
}))
defer ts.Close()

client := ts.Client()
res, err := client.Get(ts.URL)

in the line client := ts.Client().

However, I have a production program that I want to set to use ts.URL as its host. I am getting

x509: certificate signed by unknown authority

errors when I call it.

How can I set ts up to authenticate with the client like a normal HTTPS server?

Answer 1

As of Go 1.11, this simply isn't possible to accomplish fully within a _test.go program, due to the mechanics of HTTPS.

However, you can do a single certificate signing and generation of server.crt and server.key files, then reference them in your _test.go programs from a local directory indefinitely.

One-time .crt and .key generation

This is an abridged, slightly streamlined version of the steps specified in Daksh Shah's Medium article, How to get HTTPS working on your local development environment in 5 minutes, which will work on a Mac.

In the directory where you want your server.crt and server.key files, create the two configuration files

server.csr.cnf

[req]
default_bits = 2048
prompt = no
default_md = sha256
distinguished_name = dn

[dn]
C=US
ST=RandomState
L=RandomCity
O=RandomOrganization
OU=RandomOrganizationUnit
emailAddress=hello@example.com
CN = localhost

and

v3.ext

authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = localhost
IP.1 = 127.0.0.1

Then enter the following commands in that directory

openssl genrsa -des3 -out rootCA.key 2048 
# create a passphrase

openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 1024 -out rootCA.pem -config server.csr.cnf
# enter passphrase

openssl req -new -sha256 -nodes -out server.csr -newkey rsa:2048 -keyout server.key -config server.csr.cnf

openssl x509 -req -in server.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out server.crt -days 500 -sha256 -extfile v3.ext
# enter passphrase

Finally, make your system trust the certificate you used to sign the files by running

open rootCA.pem

This should open the certificate in the Keychain Acces app, where it will be found in the section Certificates and named localhost. Then to Always Trust it

• Press enter to open its window
• Press space to twirl down Trust
• Change "When using this certificate:" to Always Trust
• Close the window and authenticate your decision

Note: I have tried many permutations of security add-trusted-cert from the command line and, despite the fact that it adds the cert to the keychain and marks it as "Always Trust", my Go programs just won't trust it. Only the GUI method puts the system in a state that my Go programs will trust the cert.

Any Go programs you run locally using HTTPS will now trust servers you run using server.crt and server.key.

Running the server

You can create *httptest.Server instances that use these credentials with

func NewLocalHTTPSTestServer(handler http.Handler) (*httptest.Server, error) {
    ts := httptest.NewUnstartedServer(handler)
    cert, err := tls.LoadX509KeyPair("server.crt", "server.key")
    if err != nil {
        return nil, err
    }
    ts.TLS = &tls.Config{Certificates: []tls.Certificate{cert}}
    ts.StartTLS()
    return ts, nil
}

Here is an example usage:

func TestLocalHTTPSserver(t *testing.T) {
    ts, err := NewLocalHTTPSTestServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprint(w, "Hello, client")
    }))
    assert.Nil(t, err)
    defer ts.Close()

    res, err := http.Get(ts.URL)
    assert.Nil(t, err)

    greeting, err := ioutil.ReadAll(res.Body)
    res.Body.Close()
    assert.Nil(t, err)

    assert.Equal(t, "Hello, client", string(greeting))
}

Answer 2

The httptest.Server creates a self-signed certificate that can be given to a http.Client to be able to properly verify the SSL certificate.

Since the certificate is typically bound to the hostname of the server, this is probably the better way to use SSL in testing.

Example here:

1. Setup Test Server:

    // create a handler
    handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintln(w, "Hello World")
    })

    // create a Test Server with SSL
    ts := httptest.NewTLSServer(handler)

2a. Quick client, but less flexible:

    // get the client directly from the test server ts
    cl = ts.Client()

2b. Generic client

    // create a CertPool and add the certificate from ts
    certpool := x509.NewCertPool()
    certpool.AddCert(ts.Certificate())

    // create a tls config with the certPool
    tlsconf := &tls.Config{RootCAs: certpool}
    cl := &http.Client{Transport: &http.Transport{TLSClientConfig: tlsconf}}

3. Use the client:

    // use the client normally
    resp, err := cl.Get(ts.URL)
    if err != nil {
        log.Fatal(err)
    }
    log.Println(resp.StatusCode)
    ...

If you really must use your own certificate, you should be able to create a test server without starting it, set the tls.Config and start the server…

ts := httptest.NewUnstartedServer(h)
ts.TLS = &tls.Config{ /* custom settings here */ }
ts.StartTLS()