Skip to main content
logoTetrate Istio SubscriptionVersion: Next

Production Recommendation

Overview​

This section provides instructions on establishing a production environment for Prometheus and Grafana to monitor Istio within your cluster. Alternatively, you may opt for managed Grafana and Prometheus services, such as Grafana Cloud or AWS-managed Prometheus and Grafana services.

Istio Best Practices​

For a comprehensive guide on setting up Prometheus and Grafana for production, refer to Using Prometheus for production-scale monitoring.

Performance and Scalability​

Consult the Performance and Scalability guide for detailed information on optimizing Istio for production.

Production Considerations​

The following points should be taken into consideration when setting up a production environment:

Resource Allocation​

  • Prometheus and Grafana can be resource-intensive, particularly with a high volume of service mesh traffic.
  • Allocate dedicated resources (CPU, memory, and storage) for both to ensure consistent performance.
  • Continuously monitor and adjust resource quotas as necessary.

Storage and Retention​

  • Plan for data growth. Prometheus stores time-series data, which can accumulate quickly. Consider implementing retention policies or integrating with remote storage solutions (such as Thanos or Cortex) for long-term metric storage.
  • Regularly back up Grafana's configuration and dashboard data.

High Availability​

  • To ensure high availability, contemplate running multiple instances of Prometheus and Grafana across different zones or clusters.
  • Consider employing Thanos to establish a highly available Prometheus configuration.

Scalability​

  • Leverage Prometheus's sharding capability to distribute the load across multiple Prometheus instances.
  • Consider employing a horizontally scalable solution like Thanos) to handle a substantial amount of data.

Service Discovery​

  • Confirm that Prometheus is appropriately configured to automatically discover all services and workloads within the Istio service mesh.

    Refer to this demo from the Istio project.

  • Keep configurations up-to-date as your infrastructure evolves.

Access Control​

  • Secure access to both Prometheus and Grafana with robust authentication and authorization mechanisms.
  • Restrict access, modification, and deletion of data to authorized personnel only.
  • Utilize network policies to control communication with these tools.