Kubernetes ClusterRole objects and TSB APIs

Overview

Some organizations prefer TSB application specific configuration (e.g. traffic settings at the workspace level) to be deployed alongside their applications through Kubernetes focussed GitOps tooling (e.g. Helm, Flux, ArgoCD). The majority of this tooling expects usage of Kubernetes and its CRDs for deployment of configuration. TSB API documents can be wrapped to support deployment as Kubernetes CRDs in these environments.

From a defense in depth perspective, it is recommended to minimize the privileges GitOps tooling has to deploy TSB configuration into Kubernetes. While configuration merged into Git repositories should already be checked for correctness, security clearance, and acceptance by the people involved in the change validation and approval workflow, it is wise to add additional access scoping measures at the deployment stage of configuration changes. One way to manage this is to create Kubernetes ClusterRole resources to limit permissions on CRD wrapped TSB API objects, per cluster or namespace, in combination with RoleBindings.

This document describes how to generate Kubernetes ClusterRole objects from existing TSB roles.

Please note that it is important to understand that TSB is not a Kubernetes application, nor are its APIs native Kubernetes CRDs. While at the service they look very similar, there are significant differences. Understanding the differences will help you decide how to manage TSB objects best in your environment, when choosing to use Kubernetes as a GitOps deployment target for TSB API objects.

TSB's API and RBAC model are hierarchical in nature

Kubernetes' RBAC model differs from TSB's model. In TSB it is possible to provide access to different levels in the organizational hierarchy segmented through tenancy. This means that users with write/create access at a particular level, automatically have access to create and edit descendant resources lower in the hierarchy. Kubernetes, on the other hand, has a flat structure where each CR belongs to the same universal level. Access to Kubernetes documents can simply be given and potentially limited per namespace in which these documents can reside.

TSB also provides certain resources, like configuration profiles, to be available at various levels in the TSB hierarchy. This concept simply can't be fully translated to Kubernetes RBAC.

When converting TSB roles to Kubernetes ClusterRoles, please note that the more generic roles like writer and creator have very wide permissions as they could be used to grant access at the organization level and all resources down in the TSB hierarchy. This is a side effect of Kubernetes not having hierarchical levels. To get the best results, create dedicated TSB roles for GitOps purposes that are very explicit about the resources and the level in the hierarchy they operate at.

TBS's world view is heterogeneous & multi-cluster

TSB manages application traffic through Envoy sidecars and gateways, while these typically live in Kubernetes, they don't necessarily need to. TSB's organizational hierarchy and access control system also decouples segmentation of access to be vertical (cluster -> namespace) and allows it to be horizontal (an application workspace spanning all namespaces matching teama-dev-* in all onboarded clusters matching tenant1-dev-*).

Besides the impact of a multi-cluster world view on permissions, there is another important behavioral difference between common GitOps tooling and TSB around configuration lifecycle management. Since GitOps and lifecycle management tools only deal with single cluster scope, all actions regarding installation, changes/upgrades, and removal are expected to impact the single cluster in which the tool is operating. If using GitOps to distribute TSB configuration to the TSB management plane, this only works well if the configuration impacts behavior in a single cluster. In case of TSB configuration scoped to impact multiple clusters (e.g. a workspace managing a multi-cluster deployed application) this will enable conflicts and race conditions to occur as the TSB configuration is deployed to all applicable clusters, while the lifecycle management tool is only reconciling from a single cluster viewpoint. In the case of multi-cluster enabled applications it is strongly recommended to decouple application deployment lifecycle management (e.g. application helm chart usage) from TSB multi-cluster application configuration management. Simply put, use a dedicated deployment workflow for it.

Kubernetes service accounts can't be mapped to TSB service accounts

While the usage of ClusterRoles allows Kubernetes service accounts to be limited in which TSB CRDs they are allowed to manage in certain namespaces, it does not put limits on the contents of these CRDs. When a TSB CRD resource modification (application or deletion) is successful, the TSB GitOps webhook will pick it up and distribute to the TSB management plane. The TSB GitOps webhook is unable to infer the correct TSB user or service account for such objects to choose when sending to the management plane. This means that if an application team with access to Workspace A but not Workspace B triggers their GitOps pipeline and deploys a TSB CRD destined for Workspace B in a Kubernetes namespace they are allowed to touch, the CRD will happily be modified at the Kubernetes end and picked up by the TSB GitOps webhook. If you want to guarantee that only certain Tenants, Workspaces, or Groups can be referenced inside TSB CRDs when deployed to specific Kubernetes namespaces, you will need to use a tool like OPA Gatekeeper to enforce the correctly scoped annotations.

Generate a ClusterRole object

# list all roles
$ tctl get roles
NAME                  DISPLAY NAME           DESCRIPTION
workspace-owner       Workspace Owner        Owner for workspaces of a particular tenant
workspace-operator    Workspace Operator     Management permissions on a particular workspace
traffic-operator      Traffic Operator       Configure Ingress Gateway, Application, APIs and the TrafficSettings around them under a particular workspace
editor                Editor                 Edit permissions (Read, Write, Create) across all resources
orgsync               OrgSync                Create and Write permissions on tenants, users and teams
registryadmin         RegistryAdmin          Create and Write permissions on registry services
tenant-reader         Tenant Reader          Read access to a tenant and its settings
tenant-readonly       Tenant Readonly
creator               Creator                Provides Read access to target resources and allows Creating child ones
registryreader        RegistryReader         Read permissions to registered services
tenant-owner          Tenant Owner           Platform owner for a specific tenant
security-owner        Security Owner         Organization-wide security settings owner
user-readonly         User Readonly
traffic-viewer        Traffic Viewer         View Ingress Gateway, Application, APIs and the TrafficSettings around them under a particular workspace
admin                 Admin                  Admin (all) permissions across all resources
writer                Writer                 Read and Write permissions across all TSB Resources
reader                Reader                 Read permissions across all TSB Resources
traffic-owner         Traffic Owner          API Platform/API Gateway owner to configure Tier1 and shared gateways, and the TrafficSettings around them
cluster-readonly      Cluster Readonly
org-reader            Organization Reader    Read permissions to the organization and organization settings
tenant-operator       Tenant Operator        Management permissions for a concrete tenant
security-operator     Security Operator      Manage security policies for a particular workspace
security-viewer       Security Viewer        View security policies for a set of workspaces/tenants

To see which RBAC roles are available in TSB, you can visit the UI and select Roles under the Admin group. Another option is to use the tctl tool and run:
Let's look at the details of the default Creator role from a TSB perspective:

# list the details of a role
$ tctl get role creator -o yaml
apiVersion: rbac.tsb.tetrate.io/v2
kind: Role
metadata:
  description: Provides Read access to target resources and allows Creating child ones
  displayName: Creator
  name: creator
spec:
  description: Provides Read access to target resources and allows Creating child ones
  displayName: Creator
  fqn: rbac/creator
  rules:
  - permissions:
    - READ
    - CREATE

Now let's retrieve a Kubernetes ClusterRole resource which would allow one to access the same but wrapped resources from a Kubernetes CRD perspective. As previously mentioned, the output from the creator role is very verbose as it spans the entire TSB hierarchy.

# generate a ClusterRole object for the TSB creator role
$ tctl x role get-cluster-role rbac/creator
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: null
  name: tsb-rbac-creator
rules:
- apiGroups:
  - application.tsb.tetrate.io
  resources:
  - apis
  - applications
  verbs:
  - get
  - list
  - watch
  - create
- apiGroups:
  - extension.tsb.tetrate.io
  resources:
  - wasmextensions
  verbs:
  - get
  - list
  - watch
  - create
- apiGroups:
  - gateway.tsb.tetrate.io
  resources:
  - egressgateways
  - gateways
  - groups
  - ingressgateways
  - tier1gateways
  verbs:
  - get
  - list
  - watch
  - create
- apiGroups:
  - istiointernal.tsb.tetrate.io
  resources:
  - groups
  verbs:
  - get
  - list
  - watch
  - create
- apiGroups:
  - profile.tsb.tetrate.io
  resources:
  - profiles
  verbs:
  - get
  - list
  - watch
  - create
- apiGroups:
  - rbac.tsb.tetrate.io
  resources:
  - roles
  verbs:
  - get
  - list
  - watch
  - create
- apiGroups:
  - security.tsb.tetrate.io
  resources:
  - groups
  - securitysettings
  - servicesecuritysettings
  verbs:
  - get
  - list
  - watch
  - create
- apiGroups:
  - segmentation.tsb.tetrate.io
  resources:
  - segmentationmemberships
  - segmentationpolicys
  - segmentationruless
  verbs:
  - get
  - list
  - watch
  - create
- apiGroups:
  - traffic.tsb.tetrate.io
  resources:
  - groups
  - serviceroutes
  - servicetrafficsettings
  - trafficsettings
  verbs:
  - get
  - list
  - watch
  - create
- apiGroups:
  - tsb.tetrate.io
  resources:
  - apis
  - clusteronboardingconfigs
  - clusters
  - organizations
  - organizationsettings
  - serviceaccounts
  - teams
  - tenants
  - tenantsettings
  - users
  - workspaces
  - workspacesettings
  verbs:
  - get
  - list
  - watch
  - create

The generation of ClusterRole objects can also be handled through the TSB APIs directly. For more information see our reference documentation:

• REST API
• gRPC API

Note: when a ClusterRole is generated, the name will be the combination of tsb-rbac- and the name of the TSB role. So the TSB creator role will allow the generation of a ClusterRole named tsb-rbac-creator.

Persona based roles

For your convenience we have created a couple of higher level persona based default TSB roles, that align with the scope that common GitOps tooling requires to successfully manage TSB workload related settings at the workspace level (typically used for an application or application team).

Please inspect the following roles if they make sense for your situation, if not, they can be copied and adjusted to your requirements.

NAME                  DISPLAY NAME           DESCRIPTION
workspace-owner       Workspace Owner        Owner for workspaces of a particular tenant
workspace-operator    Workspace Operator     Management permissions on a particular workspace
traffic-operator      Traffic Operator       Configure Ingress Gateway, Application, APIs and the TrafficSettings around them under a particular workspace
registryadmin         RegistryAdmin          Create and Write permissions on registry services
tenant-reader         Tenant Reader          Read access to a tenant and its settings
tenant-readonly       Tenant Readonly
registryreader        RegistryReader         Read permissions to registered services
tenant-owner          Tenant Owner           Platform owner for a specific tenant
security-owner        Security Owner         Organization-wide security settings owner
traffic-viewer        Traffic Viewer         View Ingress Gateway, Application, APIs and the TrafficSettings around them under a particular workspace
traffic-owner         Traffic Owner          API Platform/API Gateway owner to configure Tier1 and shared gateways, and the TrafficSettings around them
tenant-operator       Tenant Operator        Management permissions for a concrete tenant
security-operator     Security Operator      Manage security policies for a particular workspace
security-viewer       Security Viewer        View security policies for a set of workspaces/tenants

For application team GitOps access, we recommend you to enable TSB GitOps at the control plane clusters where application teams (or their CD tooling) have deployment access.

If also using GitOps for higher level TSB configuration (organization and tenant level), it is recommended to enable TSB GitOps at the management plane for these configurations and generate a ClusterRole object based on the TSB Admin role.

Using the ClusterRole object

To deploy a TSB generated ClusterRole, please apply it to the needed cluster:

# assume we want all permissions as provided by the TSB workspace-operator role.
tctl x role get-cluster-role rbac/workspace-operator > tsb-rbac-workspace-operator.yaml

# assume we have kubectl access to the correct cluster.
kubectl apply -f tsb-rbac-workspace-operator.yaml

If you have a new GitOps tool or kubectl pipeline to onboard, please create a dedicated ServiceAccount for it located at the namespace where you wish to deploy TSB CRD resources:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-serviceaccount
  namespace: ns1

To bind the ClusterRole object named tsb-rbac-workspace-operator to this service account for namespace ns1, you need to create RoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: manage-tsb-appteam-resources
  namespace: ns1
subjects:
- kind: ServiceAccount
  name: my-serviceaccount
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: tsb-rbac-workspace-operator
  apiGroup: rbac.authorization.k8s.io