I'm looking for some guidance on exposing UDP services on GKE using the ingress-nginx controller. After following the instructions on https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/ I was able to access it when deploying to a local minikube VM using the ConfigMap method. However, when I deploy to GKE the services are unreachable over the IP of the ingress controller service.
I see the ports (1053 and 15353) on the controller are mapped correctly:
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx ingress-nginx-controller LoadBalancer 10.51.252.115 <redacted> 80:32307/TCP,443:30514/TCP,1053:32764/UDP,15353:31385/UDP 54d
The cluster itself was created using the google_container_cluster Terraform module with default settings and the controller works well handling HTTPS traffic. One thing I did notice is that the auto-generated firewall rules omit UDP for the specified ports, and use TCP instead. Manually adding a UDP firewall rule for those ports didn't work.
NAME NETWORK DIRECTION PRIORITY ALLOW DENY DISABLED
k8s-fw-a62a982b26a034e0e97258af6717b8b0 cluster-network-labs-us-west1 INGRESS 1000 tcp:80,tcp:443,tcp:1053,tcp:15353 False
I've deployed a simple UDP ping-pong server which works both locally on bare metal and on minikube as a kubernetes service using the ingress-nginx controller. That same controller with an identical configuration causes client requests to time out.
Server
package server
import (
"fmt"
"net"
"os"
"time"
"github.com/spf13/cobra"
)
func response(udpServer net.PacketConn, addr net.Addr, buf []byte) {
fmt.Println("msg", string(buf))
time := time.Now().Format(time.ANSIC)
responseStr := fmt.Sprintf("%v. msg: %v", time, string(buf))
udpServer.WriteTo([]byte(responseStr), addr)
}
var Command = &cobra.Command{
Use: "server",
Short: "Debug UDP server.",
Long: `Provides a UDP server endpoint which responds to pings.`,
RunE: func(cmd *cobra.Command, args []string) error {
udpServer, err := net.ListenPacket("udp", fmt.Sprintf(":%d", serverPort))
if err != nil {
return err
}
defer udpServer.Close()
fmt.Fprintf(os.Stdout, "listening :%d\n", serverPort)
for {
buf := make([]byte, 1024)
_, addr, err := udpServer.ReadFrom(buf)
if err != nil {
continue
}
go response(udpServer, addr, buf)
}
},
}
var serverPort int
func init() {
Command.PersistentFlags().IntVar(&serverPort, "port", 1053, "Port to open an listen for UDP packets")
}
Client
package client
import (
"fmt"
"net"
"os"
"github.com/spf13/cobra"
)
var Command = &cobra.Command{
Use: "client",
Short: "Debug UDP client.",
Long: `Provides a UDP client endpoint which responds to pings.`,
RunE: func(cmd *cobra.Command, args []string) error {
udpServer, err := net.ResolveUDPAddr("udp", fmt.Sprintf("%s:%d", queryHost, queryPort))
if err != nil {
return err
}
conn, err := net.DialUDP("udp", nil, udpServer)
if err != nil {
return err
}
defer conn.Close()
_, err = conn.Write([]byte(msg))
if err != nil {
return err
}
received := make([]byte, 1024)
_, err = conn.Read(received)
if err != nil {
println("Read data failed:", err.Error())
os.Exit(1)
}
println(string(received))
return nil
},
}
var msg string
var queryHost string
var queryPort int
func init() {
Command.PersistentFlags().StringVar(&msg, "msg", "echo", "Message used to send ping/pong requests over UDP")
Command.PersistentFlags().StringVar(&queryHost, "host", "127.0.0.1", "Host used to send ping/pong requests over UDP")
Command.PersistentFlags().IntVar(&queryPort, "port", 1053, "Port used to send ping/pong requests over UDP")
}
Has anyone seem something similar to this or have any ideas on where I can dig in further? Thanks
Versions:
- ingress-nginx helm chart -
4.4.0 - ingress-nginx -
1.5.1 - Kubernetes -
v1.24.5-gke.600 - registry.terraform.io/hashicorp/google -
v4.43.0
