Limits and recommendations to promote efficient resource use.

For best performance and cost reduction, VMware Aria Operations for Applications (formerly known as Tanzu Observability by Wavefront) supports limits. Some limits are recommendations–if your environment exceeds the limits, you’ll see significant performance issues. Other limits result in an error if you exceed the limit.

Concurrent Query Limits

Our service enforces the following concurrent query limits. These limits are subject to change without notice.

Per Customer Concurrent Query Limit

Our service enforces a limit on concurrent queries for each customer cluster. The default is 1000. If you are getting repeated errors that your cluster is exceeding this limit, contact us.

The following error results if your environment exceeds this limit:

HTTP 429 TOO_MANY_REQUESTS
Customer concurrent query limit exceeded. Please try again later. Contact http://support.broadcom.com for help.

Per User Concurrent Query Limit

Our service enforces a limit on per-user concurrent queries. The default limit is 100. Contact us if you believe that the setting doesn’t make sense for one of your users (for example, one of your service accounts) and we’ll discuss options with you.

The following error results if one of the users exceeds this limit:

HTTP 429 TOO_MANY_REQUESTS
“User concurrent query limit exceeded. Please try again later. Contact http://support.broadcom.com for help.”

Timeout Limits

CategoryTimeout LimitExplanation and Best Practice
Query 300s (5min) If a query does not complete in 300s (5 minutes), the chart/API times out.

Best practice: Use specific sources and/or point tags in the queries to drill down into specific data that is required.
Alert 60s (1min) When you create an alert, you define a query to get the data you want to monitor. This query must complete in less than 60 seconds (1 minute). If the query does not get the results in less than 60 seconds, you won't be able to save the alert.

Best practice: Use specific sources, point tags, or both in the queries to drill down into specific data that you need.
Dynamic Dashboard Variable 60s (1min) Dynamic dashboard variables dropdown menus allow users to make a selection that is based on metadata of a query, for example, a set of sources. If the dynamic variable query does not complete in 60s, it is cancelled by the server to avoid a bottleneck in loading a dashboard.

Best practice:Do not use wildcard characters in dynamic variables. Use specific sources, point tags, or both in the queries to drill down into specific data that is required.
Derived Metric 300s (5min) Derived metrics can synthetically create metrics based on existing metrics. The query engine can reingest those metrics. The query that runs for a derived metric, like a regular query, has a 300s or 5 minute timeout.

Best practice: Use specific sources and/or point tags in the queries to drill down into specific data that is required.

Default Customer-Specific Limits

A set of out-of-the-box limits applies to your customer account. You can contact our customer success team to request changes. In some cases, a change might involve additional costs.

Metric length limit Maximum number of characters for a metric name. 256
Histogram length limit Maximum number of characters for a histogram name. 256
Span length limit Maximum number of characters for a span name. 128
Host length limit Maximum number of characters for a source name. 128
Annotations count limit Maximum number of point tags associated with a metric. 20
Annotations key length limit Maximum number of characters in a point tag key. 64
Annotations value length limit Maximum number of characters in a point tag value. 255
Counter length limit Maximum number of characters in a counter metric. 256
Span annotations count limit Maximum number of point tags associated with a span. 50
Span annotations key length limit Maximum number of characters associated with a span point tag key. 128
Span annotations value length limit Maximum number of characters associated with a span point tag value. 128
Span topology processing Ttl 10
Span topology dimensions Dimensions associated with a span. Defaults to "application" "cluster" "shard" 128
Span logs size limit Maximum size of a span log. 32768
Log annotations count limit Maximum number of tags associated with a log. 100
Log annotations key length limit Maximum number of characters associated with a log tag key. 128
Log annotations value length limit Maximum number of characters associated with a log tag value. 128
Log length limit Maximum number of character on a log message. 20000
Search time window for logs The default search time window for the Log Browser 15 minutes
Metrics to logs tag mapping Maps the tags of the metrics to the the tags of logs. source tag
Traces to logs tag mapping Maps the tags of the traces to the the tags of logs. traceId, source, application, and service tags

Data Format Best Practices

Follow best practices to avoid hitting query limits and for improved query execution speed and meaningful results.

  • Make the metrics the most stable part of your data:
    • Do not include source names in the metric name. Our service captures sources separately.
    • Do not include data or timestamps in the metric name. Each point has an associated timestamp.
  • Aim for a metric hierarchy:
    • Partition the top level of the metric hierarchy by including at least one dot.
    • Organize metric names in a meaningful hierarchy from most general to most specific (i.e. system.cpu0.loadavg.1m instead of 1m.loadavg.cpu0.system)
  • For best performance, keep the number of distinct time series per metric and host to under 1000.

See Operations for Applications Data Naming for more best practices.

More Info

You can examine what’s going on with your cluster in several ways:

administration