Network Policy Recommendations

Network Policy Within a Mesh

In the context of a service mesh, Network Policies are set of rules and configurations that defines how various microservices can communicate with one another over the network. These policies play a pivotal role in controlling traffic flows, enforcing security measures and maintaining compliance within the service mesh architecture. Network Policies enable organizations to control communication between services by specifying permitted services, protocols, ports, and the allowable data types for exchange at the L3/L4 layer. This precise control stops unauthorized access, reduces areas vulnerable to attacks, and guarantees communication solely among trusted sources.

Network Policy & Zero Trust Architecture

Adding to the ideas of a service mesh, the Zero Trust Architecture (ZTA) approach to security considers that no user or device, whether from inside or outside the organization's network, should be automatically trusted. Instead of relying on old-fashioned perimeter defenses, ZTA emphasizes confirming identity and allowing access through rigorous authentication and authorization steps. You can learn more about it here.

What are the Challenges?

While TSB currently offers many security advantages via intelligent user controls and customized hierarchical access policies, the challenge arises when organizations aim to integrate Cilium or Calico network policies alongside existing identity-based access policies within a service mesh infrastructure. This situation necessitates the management of two distinct sets of policies, overseen by separate roles – Security Admins and Platform Owners, contributing to the complexities encountered by many organizations today.

How organizations can keep both identity based mesh access control policies and L3/L4 based network policies in sync. This is what TSB addresses by introducing Network Policy Recommendations.

What is Network Policy Recommendation?

Starting from TSB 1.7.0 version, TSB has the ability to suggest Kubernetes Network Policies. These recommendations are derived from the Hierarchical Access Control Policies set up by the Platform Owner or Application Owner, who handles these policies within TSB.

Any flows that would be allowed/denied by the recommended network policies are already allowed/denied by the TSB configuration. The policies are provided as a convenience, and are provided in a form that can be easily managed and understood by security teams and Kubernetes administrators. You can inspect the network policies to verify that TSB is applying appropriate access control policies to your mesh.

This feature can be enabled on each control plane cluster by setting ENABLE_NETWORK_POLICY_TRANSLATION to true in XCP component in ControlPlane CR. Once enabled, the recommended network policies are stored as namespace scoped configmap in the dedicated control plane clusters. Platform owner/Security owner who manages network specific access controls, can retrieve these policies from the configmap and verify & apply the policies in their desired namespaces.

apiVersion: install.tetrate.io/v1alpha1
kind: ControlPlane
metadata:
  name: controlplane
  namespace: istio-system
spec:
  components:
    xcp:
      ...
        kubeSpec:
        overlays:
        - apiVersion: install.xcp.tetrate.io/v1alpha1
          kind: EdgeXcp
          name: edge-xcp
          patches:
          - path: spec.components.edgeServer.kubeSpec.deployment.env
            value:
            - name: ENABLE_NETWORK_POLICY_TRANSLATION
              value: "true"
      ...
   ...

What are the Use-cases?

TSB mostly focuses on the following use-cases -

• Recommendations to Secure North-South Traffic - external traffic ingressing via Edge Gateway or Ingress Gateway into applications
• Recommendations to Secure East-West Traffic - service to service access restrictions within mesh based on the identity of the service, who owns it and who all can access it

Use-case 1: Recommendations to Secure North-South Traffic

When the user configures a Gateway object as Edge Gateway or an Ingress Gateway, TSB can make sure basic sanity by recommending the network policies that only allows the external traffic received on exposed ports like 80 or 443, for a particular host to only route to the TSB Gateway workloads ports i.e 8080, 8443 & 15443.

Pre-requisite

Enabling network policy enforcement on kubernetes clusters is not under the scope of TSB. Users need to make sure Container Network Interface (CNI) plugins are enabled in their respective kubernetes clusters before applying TSB recommended policies for it to have effect.

TSB Configurations

Configure an Edge Gateway to expose multiple hosts and route to other Tier2 clusters where an IngressGateway is deployed and configured to expose the same host names.

# gateway-config.yaml
apiVersion: v1
kind: List
items:
  - apiVersion: install.tetrate.io/v1alpha1
    kind: Tier1Gateway
    metadata:
      name: edge-gateway
      namespace: edge-gw
    spec:
      kubeSpec:
        service:
          type: LoadBalancer
  - apiVersion: gateway.tsb.tetrate.io/v2
    kind: Gateway
    metadata:
      name: edge-gateway
      namespace: edge-gw
      annotations:
        tsb.tetrate.io/organization: tetrate
        tsb.tetrate.io/tenant: tetrate
        tsb.tetrate.io/workspace: edge-gw-ws
        tsb.tetrate.io/gatewayGroup: edge-gw-gp
    spec:
      workloadSelector:
        namespace: edge-gw
        labels:
          app: edge-gateway
      http:
      - name: bookinfo
        hostname: bookinfo.tetrate.io
        port: 80
        routing:
          rules:
            - match:
                - uri:
                    prefix: "/productpage"
              route:
                clusterDestination:
                  clusters:
                    - name: gke-tetrate-us-west1-1
                      weight: 100
            - match:
                - uri:
                    prefix: "/api/v1/products"
              route:
                clusterDestination:
                  clusters:
                    - name: gke-tetrate-us-east1-2
                      weight: 100

When you apply the above configuration, based on the default Istio AuthZ policies that TSB creates and the Tier1Gateway install API configuration for creating the k8s service object for the exposed ports, TSB will start translating both in to a configmap with the recommended network policy configuration.

kubectl get configmap -l xcp.tetrate.io/recommended-network-policy=true -n edge-gw

Output

NAME                     DATA   AGE
np-authz-edge-gateway    1      31s
np-edge-gateway          1      31s

Retrieve the configmap and verify the recommended network policies before applying them on the clusters

kubectl get configmap -n edge-gw np-edge-gateway -o jsonpath='{.data.policy}' > edge-gw-policy.yaml

Recommended network policy restrict the ingress traffic only on the exposed ports configured in the service.

# edge-gw-policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  creationTimestamp: null
  labels:
    xcp.tetrate.io/recommended-network-policy: "true"
    xcp.tetrate.io/service: edge-gateway
  name: np-edge-gateway
  namespace: edge-gw
spec:
  ingress:
  - ports:
    - port: 15443
    - port: 8080
    - port: 8443
  podSelector:
    matchLabels:
      app: edge-gateway
      istio: ingressgateway
  policyTypes:
  - Ingress
status: {}

Apply Network Policies on the cluster

kubectl apply -f edge-gw-policy.yaml

kubectl describe networkpolicy np-edge-gateway -n edge-gw

Output

Name:         np-edge-gateway
Namespace:    edge-gw
Created on:   2023-08-16 21:40:49 +0530 IST
Labels:       xcp.tetrate.io/recommended-network-policy=true
              xcp.tetrate.io/service=edge-gateway
Annotations:  <none>
Spec:
  PodSelector:     app=edge-gateway,istio=ingressgateway
  Allowing ingress traffic:
    To Port: 15443/TCP
    To Port: 8080/TCP
    To Port: 8443/TCP
    From: <any> (traffic not restricted by source)
  Not affecting egress traffic
  Policy Types: Ingress

Use-case 2: Recommendations to Secure East-West Traffic

In a typical Tenant/Team/BU based access restrictions where when a user configures 2 Tenants i.e Tenant A and Tenant B and has configured deny_all as default in OrganizationSetting to deny all service to service communication. When the user configures an allow policy for Tenant A to talk to workspace-frontend under Tenant B but not workspace-backend under Tenant B. When TSB creates Istio AuthZ based policies to enforce this behaviour, TSB will also create recommended network policies as configmaps for the users to enforce the same behaviour at L3/L4 layer.

TSB Configurations

Creates the following configurations.

• Create a new Tenant Marketing & Create 2 workspaces under Marketing Tenant
  • Workspace marketing-frontend
    • maps to marketing-frontend namespace in cluster-1
    • deploy productpage in marketing-frontend namespace
  • Workspace marketing-backend
    • maps to marketing-backend namespace in cluster-1
    • deploy details, reviews & ratings in marketing-backend namespace
• Create a new Tenant Payment & Create a workspace under Payment Tenant
  • Workspace payment-chanel
    • maps to payment-channel namespace under cluster-1
    • deploy sleep service in payment-channel namespace
• Allow payment-channel to talk to only marketing-frontend using TenantSettings

apiVersion: tsb.tetrate.io/v2
kind: TenantSetting
metadata:
  name: default-setting
  annotations: 
    tsb.tetrate.io/organization: tetrate
    tsb.tetrate.io/tenant: marketing
spec:
  defaultSecuritySetting:
    authenticationSettings:
      trafficMode: REQUIRED
    authorization:
      mode: RULES
      rules:
        allow:
        - from:
            fqn: organizations/tetrate/tenants/payment
          to:
            fqn: organizations/tetrate/tenants/marketing/workspaces/marketing-frontend
        - from:
            fqn: organizations/tetrate/tenants/marketing
          to:
            fqn: organizations/tetrate/tenants/marketing
    displayName: default-setting
  displayName: default-setting

When you apply the above configuration, TSB will create the following AuthZ policies.

kubectl get authorizationpolicy -A

Output

NAMESPACE            NAME                                     AGE
marketing-backend    allow                                    8m26s
marketing-frontend   allow                                    8m26s
payment-channel      allow                                    8m26s

As we configured, since marketing-frontend and marketing-backend namespaces belong to same tenant Marketing, workloads belong to marketing-frontend namespace are allowed to talk to workloads belong to marketing-backend but not any other namespace i.e payment-channel

kubectl get authorizationpolicy allow -n marketing-backend -o yaml

Output

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  annotations:
    tsb.tetrate.io/config-mode: bridged
    tsb.tetrate.io/etag: '"VeslxrLeGvQ="'
    tsb.tetrate.io/fqn: organizations/tetrate/tenants/marketing/workspaces/marketing-backend
    tsb.tetrate.io/runtime-etag: '"enVCG/QQ2fc="'
    xcp.tetrate.io/contentHash: a654b8f44747c21a28d0b44dcd193748
  creationTimestamp: "2023-07-11T10:18:25Z"
  generation: 1
  name: allow
  namespace: marketing-backend
  resourceVersion: "4382832"
  uid: 22c3265f-0be7-42da-9584-6913c2676f16
spec:
  rules:
  - from:
    - source:
        principals:
        - gke-sreehari-us-west1-1.tsb.local/ns/marketing-backend/*
        - gke-sreehari-us-west1-1.tsb.local/ns/marketing-frontend/*

But marketing-fontend allows requests from workloads belong to all workspaces under payment tenant and marketing tenant.

kubectl get authorizationpolicy allow -n marketing-frontend -o yaml

Output

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  annotations:
    tsb.tetrate.io/config-mode: bridged
    tsb.tetrate.io/etag: '"hjW1MOChsJU="'
    tsb.tetrate.io/fqn: organizations/tetrate/tenants/marketing/workspaces/marketing-frontend
    tsb.tetrate.io/runtime-etag: '"ZWzxphzAJEQ="'
    xcp.tetrate.io/contentHash: f6098fa68d84259a71b1c42b3152402a
  creationTimestamp: "2023-07-11T10:18:25Z"
  generation: 1
  name: allow
  namespace: marketing-frontend
  resourceVersion: "4382827"
  uid: d4beae2c-1f66-4dd5-9662-7dd397519155
spec:
  rules:
  - from:
    - source:
        principals:
        - gke-sreehari-us-east1-2.tsb.local/ns/payment-channel/*
        - gke-sreehari-us-west1-1.tsb.local/ns/marketing-backend/*
        - gke-sreehari-us-west1-1.tsb.local/ns/marketing-frontend/*
        - gke-sreehari-us-west1-1.tsb.local/ns/payment-channel/*

With the above AuthZ based policies, sleep service in payment-channel namespace is allowed to call productpage in marketing-frontend but not allowed to call details, reviews, ratings in marketing-backend namespace.

kubectl exec "$(kubectl get pod -n payment-channel -l app=sleep -o jsonpath='{.items[0].metadata.name}')" -n payment-channel -c sleep -- curl -s http://productpage.marketing-frontend.svc:9080/api/v1/products -v

Output

< HTTP/1.1 200 OK

kubectl exec "$(kubectl get pod -n payment-channel -l app=sleep -o jsonpath='{.items[0].metadata.name}')" -n payment-channel -c sleep -- curl -s http://details.marketing-backend.svc:9080/details/1 -v

Output

RBAC: access denied

Verify the recommended AuthZ based Network Policies

kubectl get configmap -l xcp.tetrate.io/recommended-network-policy=true,xcp.tetrate.io/authz-policy=allow -A

Output

NAMESPACE            NAME             DATA   AGE
marketing-backend    np-authz-allow   1      1m2s
marketing-frontend   np-authz-allow   1      1m2s
payment-channel      np-authz-allow   1      1m2s

Apply authz specific network policy in marketing-backend namespace to allow ingress only from pods in marketing-frontend & marketing-backend namespace

kubectl get configmap -n marketing-backend np-authz-allow -o jsonpath='{.data.policy}' > backend-policy.yaml

Verify the recommended policy

# backend-policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  annotations:
    tsb.tetrate.io/config-mode: bridged
    tsb.tetrate.io/etag: '"VeslxrLeGvQ="'
    tsb.tetrate.io/fqn: organizations/tetrate/tenants/marketing/workspaces/marketing-backend
    tsb.tetrate.io/runtime-etag: '"enVCG/QQ2fc="'
    xcp.tetrate.io/contentHash: f53ddeb9e43de1ca56fd3ea20c21b919
  creationTimestamp: null
  labels:
    xcp.tetrate.io/authz-policy: allow
    xcp.tetrate.io/recommended-network-policy: "true"
  name: np-authz-allow
  namespace: marketing-backend
spec:
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: marketing-frontend
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: marketing-backend
  podSelector: {}
  policyTypes:
  - Ingress
status: {}

Verify the same request after applying the above policy. you would notice that the request is denied while trying to call review,details, ratings from any other namespace other than marketing-frontend and marketing-backend.

kubectl exec "$(kubectl get pod -n payment-channel -l app=sleep -o jsonpath='{.items[0].metadata.name}')" -n payment-channel -c sleep -- curl -s http://details.marketing-backend.svc:9080/details/1 -v

Output

upstream connect error or disconnect/reset before headers

Troubleshooting

Once the network policy recommendation feature is enabled in a control plane cluster, XCP would consider all the existing namespaces which are mapped to TSB workspaces as the candidates for network policy translation. However users can choose to disable the network policy configmap generation on selected clusters by setting the ENABLE_NETWORK_POLICY_TRANSLATION to false on XCP component in ControlPlane CR.

apiVersion: install.tetrate.io/v1alpha1
kind: ControlPlane
metadata:
  name: controlplane
  namespace: istio-system
spec:
  components:
    xcp:
      ...
        kubeSpec:
        overlays:
        - apiVersion: install.xcp.tetrate.io/v1alpha1
          kind: EdgeXcp
          name: edge-xcp
          patches:
          - path: spec.components.edgeServer.kubeSpec.deployment.env
            value:
            - name: ENABLE_NETWORK_POLICY_TRANSLATION
              value: "false"
      ...
   ...