I have an application deployed in a microk8s cluster. It has 3 components, frontend in angular, backend1 (port 5000) in flask and another set of backend2 (port 5001) services in another flask app. Angular invokes services in backend1. For few cases, backend1 will invoke services in backend2.
Initially there were three service load balancers with corresponding 3 services followed by deployments and pods. In order to make the application https, I switched off the 3 service LBs, and added the ingress add on in microk8s. I then added ingress objects in my application namespace.
One for each component. Backend1 is on a gunicorn/nginx and Backend2 is on a gunicorn. All of these are in docker containers that are deployed in the pod.
SSL works now without problem when I hit the application. I'm able to login and do a bunch of actions, which means the communication between frontend and backend1 is all set. Whereas in a particular flow where an action will invoke Backend1 will then invoke Backend2. This scenario does not work. It throws a 502 Bad Gateway Interface.
What did I try:
- I added a path in Backend1's nginx.conf corresponding to port 5001 (a separate location section under server with proxy_pass)
- I tried different combinations in my Backend2 ingress object
- I removed port 5001 from Backend1's invoking code, but unsure as to where else to specify the port
- I tried to see similar questions in SO, but none suits my scenario
Here are few links that I went through:
https://github.com/kubernetes/ingress-nginx/issues/1120
K8s Ingress rule for multiple paths in same backend service
How to configure ingress controller with multiple paths for the same service?
https://www.reddit.com/r/kubernetes/comments/rwlptd/creating_ingress_for_web_server_multiple_path/
Kubernetes access Service in other namespace via http request
How to make request to Kubernetes service?
All manifests and configuration files below.
ingress object for backend1:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-routes-backend1
namespace: app-namespace
annotations:
cert-manager.io/cluster-issuer: "letsencrypt-prod"
nginx.ingress.kubernetes.io/use-regex: "true"
spec:
tls:
- hosts:
- domain-name
secretName: tls-secret
rules:
- host: domain-name
http:
paths:
- path: /api/v1/(.+)
pathType: Prefix
backend:
service:
name: backend1
port:
number: 5000
ingress object for backend2:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-routes-backend2
namespace: app-namespace
annotations:
cert-manager.io/cluster-issuer: "letsencrypt-prod"
nginx.ingress.kubernetes.io/use-regex: "true"
spec:
tls:
- hosts:
- domain-name
secretName: tls-secret
rules:
- host: domain-name
http:
paths:
- path: /api/v1/data/(.*)
pathType: Prefix
backend:
service:
name: backend2
port:
number: 5001
Service file for backend1:
apiVersion: v1
kind: Service
metadata:
labels:
app: app-name
name: backend1
name: backend1
namespace: app-namespace
clusterIP: 10.xxx.yyy.128
clusterIPs:
- 10.xxx.yyy.128
externalTrafficPolicy: Cluster
ports:
- name: http
nodePort: 31261
port: 5000
protocol: TCP
targetPort: 5000
selector:
app: app-name
name: backend1
type: NodePort
status:
loadBalancer: {}
Service file for backend2:
apiVersion: v1
kind: Service
metadata:
labels:
app: app-name
name: backend2
name: backend2
namespace: app-namespace
clusterIP: 10.xxx.yyy.128
clusterIPs:
- 10.xxx.yyy.128
externalTrafficPolicy: Cluster
ports:
- name: http
nodePort: 31261
port: 5001
protocol: TCP
targetPort: 5001
selector:
app: app-name
name: backend1
type: NodePort
status:
loadBalancer: {}
A small example to show how the backend2 is being called from backend1:
@app.route('/api/v1/process-order/<customer_id>/customer/<order_id>/order/<bill_date>/billdate', methods=['GET'])
@jwt_required
def process_order(customer_id, order_id, bill_date):
data = {
"customer_id": customer_id,
"order_id": order_id,
"bill_date": bill_date,
}
response1 = requests.post(
BASE_URL + ':5001/api/v1/data/load-order', headers=header1, data=json.dumps(data))
result1 = response1.json()
BASE_URL is nothing but the domain name.
Below is the called method in backend2:
@app.route('/api/v1/data/load-order', methods=['POST'])
def load_order():
pass
Below is my nginx.conf file:
user www-data;
worker_processes auto;
pid /run/nginx.pid;
events {
worker_connections 1024;
use epoll;
multi_accept on;
}
http {
client_max_body_size 100M;
access_log /dev/stdout;
error_log /dev/stdout;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
index index.html index.htm;
server {
listen 5000;
listen [::]:5000;
root /var/www/html;
server_name localhost default_server;
location / {
proxy_read_timeout 360s;
proxy_send_timeout 360s;
proxy_connect_timeout 360s;
uwsgi_read_timeout 360s;
uwsgi_send_timeout 360s;
uwsgi_connect_timeout 360s;
include uwsgi_params;
uwsgi_pass unix:/tmp/uwsgi.socket;
}
}
}
Below is backend1's docker file:
FROM python:3.6
COPY code/ app/
WORKDIR /app
RUN apt-get clean \
&& apt-get -y update \
&& apt-get -y install nginx \
&& apt-get -y install python3-dev \
&& apt-get -y install build-essential \
&& pip install -r requirements.txt
# Start nginx flask
COPY code/nginx.conf /etc/nginx
RUN chmod +x ./start.sh
CMD ["./start.sh"]
Below is the backend2's docker file:
FROM python:3.6
COPY ./ app/
WORKDIR /app
RUN pip install -r requirements.txt
CMD ["gunicorn", "-w", "3", "-b", ":5001", "-t", "360", "--access-logfile", "-", "--error-logfile", "-", "--reload", "app:app"]
I'm literally out of ideas. I spent quite sometime on making SSL work, now this. Any help/directions will be really helpful.
