Kubernetes | Restrict your API access with Serview Mesh(Linkerd)

Introduction

Sometimes, we want to restrict some API only be accessable from certain IP/services in our kubernetes cluster. By implementing service mesh, we can easily do that. In this article, I want to introduce how we can achieve that with Linkerd.

First, Linkerd is a service mesh for Kubernetes. It makes running services easier and safer by giving you runtime debugging, observability, reliability, and security — all without requiring any changes to your code.

And Linkerd’s authorization policy allows you to control which types of traffic are allowed to meshed pods. See the Authorization Policy feature description for more information on what this means.

Linkerd’s policy is configured using two mechanisms:

• A set of default policies, which can be set at the cluster, namespace, and workload level through Kubernetes annotations.
• A set of CRDs that specify fine-grained policy for specific ports, routes, workloads, etc.

By using Authorization Policy, we could manage the API access in a granular basis.

How To

In this session, I will demo how to use Authorization Policy to restrict different API.

In assumption, we have below APIs:

• /metrics #only allow internal access from cluster
• /test/* #only allow internal access from cluster
• /debug #allow certain external IP

In this demo, we will use few Linkerd resources including HTTPRoute, Server, AuthorizationPolicy. You could look into the concept in the offical website.

Prerequisites

• Linkerd
• EKS
• ALB

Steps

1. deploy our test deployment, this is a demo flask service including below resources.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: pyweb
  name: pyweb
  namespace: test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pyweb
  template:
    metadata:
      annotations:
        linkerd.io/inject: enabled
      labels:
        app: pyweb
    spec:
      containers:
      - image: tarrantro/pyweb:0.0.1
        imagePullPolicy: Always
        name: pyweb
        ports:
        - containerPort: 5000
          name: http
          protocol: TCP
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 50m
            memory: 100Mi
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: pyweb
  name: pyweb
  namespace: test
spec:
  ports:
  - name: http
    port: 5000
    protocol: TCP
    targetPort: 5000
  selector:
    app: pyweb
  type: NodePort
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/group.name: test-ingress #use your own alb group
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]'
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    kubernetes.io/ingress.class: alb
  finalizers:
  - group.ingress.k8s.aws/ingress-controller-internal
  name: pyweb
  namespace: test
spec:
  rules:
  - host: <domain-name>
    http:
      paths:
      - backend:
          service:
            name: pyweb
            port:
              number: 5000
        path: /
        pathType: Prefix

2. create a Server resource for the pods you want to control. A Server selects a port on a set of pods in the same namespace as the server.

By using alb ingress, we should use Server instead of Service because the target of alb is map to the pod, not service.

apiVersion: policy.linkerd.io/v1beta1
kind: Server
metadata:
  namespace: test
  name: pyweb
spec:
  podSelector:
    matchLabels:
      app: pyweb
  port: 5000

3. creating internal HTTP Routes to clarify the API groups you want to restrict and using Authorization Policy for those API. Here, we have a route group includes the /metrics API and group /test/ .

apiVersion: policy.linkerd.io/v1alpha1
kind: NetworkAuthentication
metadata:
  name: allow-internal-vpc
  namespace: test
spec:
  networks:
  - cidr: 10.0.0.0/16 # cluster pod ip cidr
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata:
  name: pyweb-internal-policy
  namespace: test
spec:
  targetRef:
    group: policy.linkerd.io
    kind: HTTPRoute
    name: pyweb-internal-router
  requiredAuthenticationRefs:
    - name: allow-internal-vpc
      kind: NetworkAuthentication
      group: policy.linkerd.io
---
apiVersion: policy.linkerd.io/v1beta2
kind: HTTPRoute
metadata:
  name: pyweb-internal-router
  namespace: test
spec:
  parentRefs:
    - name: pyweb
      kind: Server
      group: policy.linkerd.io
  rules:
    - matches:
      - path:
          value: "/metrics"
        method: GET
      - path:
          value: "/test/"
          type: "PathPrefix"
        method: GET

The HTTPRoute resource is used to configure policy for individual HTTP routes, by defining how to match a request for a given route. We will now create HTTPRoute resources for the authors service.

4. creating another HTTP Routes for external users.

apiVersion: policy.linkerd.io/v1alpha1
kind: NetworkAuthentication
metadata:
  name: allow-alb-and-vpc
  namespace: test
spec:
  networks:
  - cidr: 10.0.0.0/16 # cluster pod ip cidr
  - cidr: 10.128.104.0/22  # alb ip cidr
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata:
  name: pyweb-debug-policy
  namespace: test
spec:
  targetRef:
    group: policy.linkerd.io
    kind: HTTPRoute
    name: pyweb-debug-router
  requiredAuthenticationRefs:
    - name: allow-alb-and-vpc
      kind: NetworkAuthentication
      group: policy.linkerd.io
---
apiVersion: policy.linkerd.io/v1beta2
kind: HTTPRoute
metadata:
  name: pyweb-debug-router
  namespace: test
spec:
  parentRefs:
    - name: pyweb
      kind: Server
      group: policy.linkerd.io
  rules:
    - matches:
      - path:
          value: "/debug"
        method: GET
        headers:
        - name: "X-Forwarded-For"
          value: "<your external IP>" # you can use regex to match here, or custom header

Please remember to create internet HTTP Routes if your service need to expose to the internet.

Now, we can apply above see how it goes by accessing the API.

Internal pods will use kubernetes service domain for communication, external domain will still fail only if you add the forward header and use regex to match the subnet.

Limitation

After some tests, I found some limitations when using this approach.

First, you could not set multiple condition for an API. For example, if you restrict an API in one Authorization Policy, then you could not put that API into another Authorization Policy(it will not be effective)

Second, we must allow the ALB subnet to access the server/service, because all the external traffic will access to ALB for redirect, this could be the ingress controller IP if we use another ingress.