Tetrate Service Bridge Premium Enterprise Support Best Practices

Tetrate Customer Support is on standby 24/7 to help you resolve your issues. However, we need your cooperation to achieve the best outcome.

This guide is a fast 10-minute read and is an essential tutorial on how to make the most of your Tetrate Istio Subscription (TIS) Premium Enterprise Support.

We recommend that you:

1. Acquire the concepts and tools in the "Pre-requisite" section before you deploy TIS in production or take over operational responsibilities for TIS.
2. Bookmark this page and be prepared to go through the steps in "Self-service troubleshooting", so you are ready to self-diagnose and provide essential troubleshooting information to Tetrate's support team to reduce your resolution time.

Pre-requisites

Knowledge you need

We recommend that you read through the Introducing Tetrate Istio Subscription

Additional recommended training

Additionally, we recommend that you take the self-paced Istio course (free) offered by Tetrate so you are familiar with Istio concepts.

Self-diagnostic procedure

The fastest way to resolve your problem is to go through this simple checklist as soon as an error occurs and gather the essential information required for effective further troubleshooting by the Tetrate Customer Support team. Your issue resolution will be faster if you perform these procedures prior to engaging Tetrate support and including the information in your support ticket, so Tetrate can focus on issue resolution.

1. Restore

   1. The first step must always be to restore the environment to a working state.
   2. If you are deploying a change, roll it back.
   3. If you have high availability, fall-back to the working system.

2. Identify the area and troubleshoot, for example, it could be one of the following:

   1. Sidecar Injection problems:
      • If your pods are failing to start, look into the MutatingAdmissionWebhook istio-sidecar-injector. When a pod is created, the Kubernetes api-server will call the sidecar injector service (Istiod). Errors during injection, or failure to connect to the service, can result in pods not being created. These errors may look something like failed calling webhook "sidecar-injector.istio.io": Post https://istiod.istio-system.svc:443/inject?timeout=30s: context deadline exceeded.
   2. Istiod problems:
      • To capture logs: kubectl logs -n istio-system -l app=istiod --tail=100000000 -c discovery > istiod.log.
      • To capture mesh config: kubectl get configmap -n istio-system -o jsonpath={.data.mesh} istio > meshconfig.yaml
      • To capture a proxy config dump from Istiod perspective: kubectl exec ISTIOD_POD -- curl 'localhost:8080/debug/config_dump?proxyID=POD_NAME.POD_NAMESPACE'
      • If you are experiencing performance issues with Istiod, such as excessive CPU or memory usage, memory leaks, etc, it is helpful to capture profiles. Please see (this page)[https://github.com/istio/istio/wiki/Analyzing-Istio-Performance] for help.
   3. Common Issues:
      • gRPC config stream closed: 13 or gRPC config stream closed: 0 in proxy logs, every 30 minutes. This error message is expected, as the connection to Pilot is intentionally closed every 30 minutes.
      • gRPC config stream closed: 14 in proxy logs. If this occurs repeatedly it may indicate problems connecting to Pilot. However, a single occurance of this is typical when Envoy is starting or restarting.

If you can't find the cause of the issue, create a new ticket using the Tetrate Customer Support portal with the following information:

• istioctl bug-report
• Any relevant configuration related to the issue

How to file a ticket for the fastest resolution

Filing a support ticket requires information collected from self-diagnostic. If you do not have this information, please go through the steps outlined earlier first before filing a ticket. Your issue resolution will be faster if you perform these procedures prior to engaging with Tetrate support and including the information in your ticket, so Tetrate can focus on the resolution.

File a new support ticket

All tickets must be officially initiated via Tetrate's support portal (JIRA), even if you are in touch with a Tetrate employee via other means of communication (e.g., phone or Slack). Officially filing a ticket will ensure the visibility and response of the entire Tetrate support and engineering team, resulting in a faster and better response.

Additionally, even though a Tetrate Customer Engineer (CE) might be your day-to-day contact for a specific professional services engagement, break/fix support will be done by Tetrate's Customer Support team, so you should always file a ticket via Tetrate's JIRA portal even if your Customer Engineer is aware of and helping you with the issue.

Step 1: Write a detailed description with proper context

The description must provide as much context as possible. The more you can tell us on the description, the fewer follow-up questions we may need.

Please, be sure to answer the following questions in your description:

1. If the issue is caused by a new or an existing application or configuration.
2. If you made any changes to the faulty application.
3. If you made any recent changes on the platform that could have caused the problem. Examples: certificate renewal, networking changes, infrastructure upgrades...

Step 2: Include the key information from self-troubleshooting

Attach all logs and configurations from the self-troubleshooting steps.

Step 3: Choose a priority

Choose one of the following:

• Severity 1: a Production System is severely impacted and completely shut down, or the system operations or mission-critical applications are down, due to a Tetrate software failure.
• Severity 2: a Production System performance is degraded or restricted, but still operational.
• Severity 3: a Non-production System is non-operational or completely shut down.
• Severity 4: a Non-production System performance is degraded or restricted, but still operational.

Step 4: Respond to the Tetrate support team promptly in JIRA

Depending on the severity, a member of the Tetrate Customer Support team will respond to you within Tetrate's SLA.

Please respond to Tetrate's support team promptly to ensure continued support at your designated SLA level, even if your issue has been resolved outside Tetrate support engagement. We can achieve your SLA only with your timely response on tickets.