Kubernetes v1.36 Enhances Cloud Controller Monitoring with New Route Sync Metric

Kubernetes v1.36 introduces a new alpha-level counter metric, route_controller_route_sync_total, in the Cloud Controller Manager (CCM) route controller implementation at k8s.io/cloud-provider. This metric increments each time routes are synchronized with the cloud provider, providing operators with a valuable tool to monitor and optimize route reconciliation behavior.

Purpose of the New Metric

The metric was specifically added to help operators validate the behavior of the CloudControllerManagerWatchBasedRoutesReconciliation feature gate, which was introduced in Kubernetes v1.35. By comparing sync counts with and without this feature gate enabled, operators can assess the efficiency gains from the new watch-based reconciliation approach.

A/B Testing Watch-Based Reconciliation

Previously, the route controller operated on a fixed-interval loop, synchronizing routes with the cloud provider at regular intervals regardless of whether any node changes had occurred. The new feature gate switches to a watch-based approach that only reconciles routes when nodes are actually added, removed, or updated. This reduces unnecessary API calls to the infrastructure provider, lowering pressure on rate-limited endpoints and allowing operators to utilize their available quota more efficiently.

To perform A/B testing, operators can compare the route_controller_route_sync_total counter with the feature gate disabled (default) versus enabled. In clusters where node changes are infrequent, the difference should be significant: the fixed-interval loop continues incrementing the counter steadily, while the watch-based approach only increments on actual node events.

Expected Behavior Examples

With the feature gate disabled (default fixed-interval loop), the counter increments consistently even when no node changes occur:

# After 10 minutes with no node changes
route_controller_route_sync_total 60

# After 20 minutes, still no node changes
route_controller_route_sync_total 120

With the feature gate enabled (watch-based reconciliation), the counter only increases when nodes are actually added, removed, or updated:

# After 10 minutes with no node changes
route_controller_route_sync_total 1

# After 20 minutes, still no node changes — counter unchanged
route_controller_route_sync_total 1

# A new node joins the cluster — counter increments
route_controller_route_sync_total 2

The impact is most noticeable in stable clusters where node changes are rare.

Providing Feedback

If you have feedback on this new metric or the feature gate, you can reach out through the following channels:

• The #sig-cloud-provider channel on Kubernetes Slack
• The KEP-5237 issue on GitHub
• The SIG Cloud Provider community page for other communication channels

Further Reading

For more detailed information, refer to KEP-5237.