Understanding Tetrate Istio Configuration

Tetrate Service Express (TSE) uses Istio to implement dataplane configuration within Kubernetes clusters and for onboarded virtual machines. With TSE, users can configure security, traffic management, high-availability and observability.

Summary

Tetrate Service Express (TSE) installs and manages a private Istio distribution in Kubernetes clusters, so that users do not need to manage the configuration and lifecycle of Istio.

Users configure their traffic management policies using Tetrate's Bridged-mode APIs. These provide a richer abstraction on top of the low-level Istio APIs, and are multi-cluster, error-resistant, simpler to use and generate efficient, optimized Istio and Envoy configurations.

With care, expert users can additionally partition the generated configuration and inject specific Istio configuration using Tetrate's Direct-mode APIs, giving precise, fine-grained configuration where necessary.

TSE maintains configuration for each cluster dynamically, in response to user configuration (Bridged and Direct APIs) and to the state of each cluster under control of TSE. This allows TSE to manage very complex, dynamic scenarios such as cross-cluster traffic, high availability, and multi-cloud service discovery.

The provenance of the Istio distribution

The Istio distribution used by TSE is provided by Tetrate, and is installed and upgraded by the TSE controlplane tooling. Tetrate have tested and validated the Istio distributions for the TSE use cases, and provide timely updates for security and important bug fixes. Tetrate supports the Istio distributions for longer than the community project does, thus reducing the need to regularly upgrade.

TSE does not function with a customer-provided Istio installation, and customers cannot directly upgrade or manage the Istio distributions installed by TSE, other than by the means described in the TSE documentation. An example of an allowed user configuration is the process to expose Istio (envoy) metrics directly to external telemetry tools.

These restrictions ensure that users receive a tested, validated Istio installation that is fully-managed from a security, lifecycle and configuration perspective.

Configuring the Istio distribution

Tetrate provides a set of Tetrate-specific APIs for users to specify their desired cluster and traffic management configuration. TSE translates the API configuration into optimized Istio configuration and deploys this to the Kubernetes clusters.

Why the Tetrate APIs rather than the Istio APIs?

Istio APIs are relatively low-level and are difficult for humans to work with. It is possible to create contradictory, conflicting or inefficient configuration, impacting the operation of the K8s cluster. Istio APIs are better used as a machine-generated output from a higher-level configuration and policy system.

Specifically:

• Istio APIs are not multi-cluster aware. Each kubernetes cluster is configured directly and independently. This creates challenges when (a) operating multiple clusters, and (b) configuring cross-cluster traffic, HA and DR use cases.
• Istio APIs are not fully safe. It is possible to create breaking configurations, where a local change can impact the availability of an entire cluster. Without additional safeguards, clusters are vulnerable to unintentional misconfigurations and errors.
• Istio APIs are too low-level, operating at the level of namespaces and services. Configuring Istio accurately in a large cluster becomes a challenge of scale, and machine-generated Istio configuration is often required.
• Istio APIs are configured through each K8s cluster's API server, relying on K8s Cluster Roles and related RBAC methods. These methods are per-cluster (configuring multiple clusters is a challenge), and are both too coarse (no rich user hierarchy) and too granular (with respect to configuration controls).

The Tetrate APIs are designed to address these shortcomings:

• Higher-level, Multi-Tenant Abstractions: Tetrate APIs are built around the concept of a Workspace (a group of namespaces across clusters that host an application or service) and a Tenant (a group of services owned by a single team). These abstractions model Enterprise architectures and organizational structures more accurately than the Kubernetes abstractions of namespace, cluster and user.
• Multi-Cluster: The TSE platform provides a single API endpoint, and configuration that affects multiple clusters need only be specified once. The TSE platform performs the task of resolving the Istio configuration for each cluster and deploying it reliably. Cluster state changes that affect the configuration of other clusters (for example, a failure that provokes a failover event) are automatically detected and configuration is updated across clusters
• Higher-Level Configuration: Whereas the Istio APIs provide very detailed, low-level settings for security or traffic management, the Tetrate platform simplifies these to higher-level intents. Users can perform common operations such as requiring mTLS traffic globally with a single Tetrate API setting.
• Built-In Safeguards: The process of accepting Tetrate API updates and generating the corresponding Istio configuration has multiple checks and safeguards. Conflicting Tetrate changes (e.g. duplicate names) are rejected, real-time state information informs the Istio configuration creation process, and the TSE engine generates Istio configuration that results in optimal Envoy configuration, based on Tetrate's deep expertise with Istio.

Tetrate Bridged and Direct-mode APIs

The abstracted, value-add Tetrate APIs are known as 'Bridged mode' APIs. They operate at a Tetrate-centric view of the desired state, and then convert this (bridging) to an Istio-centric configuration for each cluster.

In some situations, users may want to manage the Istio configuration directly:

• Familiarity: Users are more familiar with individual Istio APIs than the higher-level, multi-cluster/tenant Tetrate bridge-mode abstractions
• Fine-tuning: Users may find that the Tetrate abstractions are not sufficient to address very detailed security or other configuration
• Feature-gaps: Some Istio capabilities (mostly Traffic Management), such as fault injection, are not exposed through the Tetrate bridged APIs

Tetrate's Direct-mode APIs can be used to inject specific Istio configuration. This is most common when fine-tuning Traffic Management use cases.

How Direct-Mode APIs work

TSE constructs Istio configuration along two axes:

• Per-cluster, per-namespace: configuration is applied within a namespace on a cluster
• Per-type: configuration settings reside in one of three non-overlapping groups - Traffic, Gateway or Security (plus an additional 'Istio-Internal' group)

Tetrate-Generated Configuration. For the most part, configuration is generated from Tetrate Bridged configuration (blue 'cells'). Configuration for individual 'cells' (red, yellow, green) can be specified instead using Direct Istio configuration.

In the simple case, the entire configuration set (across clusters-and-namespaces and groups) is created and managed from the Tetrate Bridged-mode settings.

You can define Traffic Groups, Gateway Groups and Security Groups which identify subsets of the configuration (using namespaces), and which can then specify that the subset is configured using Direct mode (not Bridged):

Example of a Security Group, claiming control of cluster1/ns1, */ns2 and cluster3/*

apiVersion: security.tsb.tetrate.io/v2
kind: Group
metadata:
  name: mysecgroup
  workspace: myworkspace
  tenant: myteam
  organization: myorg
spec:
  namespaceSelector:
    names:
    - "cluster1/ns1"
    - "*/ns2"
    - "cluster3/*"
  configMode: DIRECT

Each namespace can be claimed by at most one Traffic Group, one Security Group and one Gateway Group

Note that a namespace cannot be a member of more than one group of the same type. For example, the same namespace cannot belong to more than one security group, so you cannot apply potentially-conflicting security configs for that namespace using different APIs.

You can then push Istio configuration into these subsets using the Tetrate APIs, annotating the resources to reference the Tetrate Group:

Example Istio AuthorizationPolicy

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: helloworld-gateway
  namespace: helloworld
  annotations:
    tsb.tetrate.io/securityGroup: mysecgroup
    tsb.tetrate.io/workspace: myworkspace
    tsb.tetrate.io/tenant: myteam
    tsb.tetrate.io/organization: myorg
spec:
 action: DENY
  rules:
  - from:
    - source:
        namespaces: ["dev"]
    to:
    - operation:
        methods: ["POST"]

TSE will assemble the Istio configuration for each cluster from the Bridged configuration, overridden as needed by individual Direct configurations in the Traffic, Gateway and Security groups.

The Tetrate API also provides an Istio Internal Group, which can be used to provide additional Istio configuration directly. The Istio internal group is used for Istio configuration that does not reside in Traffic, Gateway or Security groups, and which is not generated by the Tetrate bridged APIs. Examples of this include Envoy filters and Istio Service Entries.

Istio Configuration Generation

The process of generating Istio configuration is complex and dynamic. Sources include the desired state, as expressed by the Tetrate Bridged-mode API settings, specific Istio configuration to be injected using Direct mode, and the current state of the clusters. TSE generates configuration dynamically, so that, for example, when a remote service is added, all clusters are updated to use it as a failover target.

It's not possible to make a simple mapping between the high-level, multi-cluster Tetrate bridge-mode state and the low-level per-cluster Istio configuration; if this were possible, a simple templating engine would be sufficient and there would be no need for the sophistication of TSE. One of the design principles of TSE is that users should very rarely need to fall-back to direct Istio configuration.

The upside of this sophistication is that TSE can manage highly-dynamic, multi-cluster environments at great scale, with in-production deployments of many thousands of kubernetes clusters under the control of a single TSE management plane. Scale does not only come from the size of the infrastructure or the number of services; it also arises from large numbers of large platform and devops teams, and TSE ensures that they can meet their traffic management goals reliably and safely, within the security guardrails defined by the host organization.