In my setup the following the following worked as well. It's just important that IP/Name used for creating certificate matches IP/Name used for registering the runner.
gitlab-runner register --tls-ca-file /my/path/gitlab/gitlab.myserver.com.pem
Furthermore, it could be necessary to add a line for hostname lookup to the runners config.toml file also (section [runners.docker]):
extra_hosts = ["git.domain.com:192.168.99.100"]
see also https://gitlab.com/gitlab-org/gitlab-runner/issues/2209
In addition, there could be some network-trouble if for gitlab/gitlab-runner network-mode host is used, it has to be added to the config.toml as well, as it starts additional containers, which otherwise could have a problem to connect to the gitlab-host ((section [runners.docker]):
network_mode="host"
Finally, there might be an issue with the self-signed SSL-Cert (https://gitlab.com/gitlab-org/gitlab-runner/issues/2659).
A dirty workaround is to add
environment = ["GIT_SSL_NO_VERIFY=true"]
to the [[runners]] section.