BMC have a FAQ for this, note the Agent settings will need tweaking (see answer 1). Support for this is better in v9.0.19 and 9.0.20. Also check out the link to github -
1. If we provision agent on Kubernetes pod as containers what should be Agent host name? . By default it is taking kubernetes pod name as host name which is no pingable from outside.
You can use a StatefulSet so the name will be set.
If you want to Control-M/Server (from outside k8s) to connect to a Control-M/Agent inside k8s you need to change the connection type to a persistent connection (see utilities agent: ctmagcfg, ctm: ctm_menu) that will be initiate from the Control-M/Agent side.
Additional Information: Best Practices for using Control-M to run a pod to completion in a Kubernetes-based cluster
https://github.com/controlm/automation-api-community-solutions/tree/master/3-infrastructure-as-code-examples/kubernetes-statefulset-agent-using-pvc
2. Can we connect the Agent provisioned in kubernetes via load balancer?
Yes. LoadBalancer will expose a static name/ip and allow the Control-M/Server to connect the Control-M/Agent but it is not needed (see the persistent connection) and it cost money in most clouds (for example in AWS it's actually defining an elastic IP that you are paying for)
3. Since we see couple of documents from the bmc communities for installing Agent on kubernetes via docker image then there should be a way to discover it from the on-prem Control-M/Server.
The Control-M/Agent discover is done from the Control-M/Agent side using CLI (or rest call) "ctm provision setup" once the pod (container) starts.
This API configures the Control-M/Agent (for example: to use persistent connection that was mentioned) and define/register it in Control-M/Server.
4. When setting agents in a kubernetes environment, does an agent need to be installed on each node in a cluster?
The Control-M/Agent only needs to be installed once. It does not have to be installed on every node.
5. Can the agent be installed on the cluster through a DeamonSet and shared by all containers?
The agent can be installed through a DeamonSet but this will install an agent on each node in the cluster. Each agent will be considered a separate installation and each agent would individually be added in the CCM. Alternatively an agent can be installed in a StatefulSet where only one agent is installed but has access to the Kubernetes cluster