Tanzu Observability by Wavefront supports tags for adding application-specific metadata to objects such alerts, dashboards, events, sources, and metrics.
Watch these videos to get started:
|Tagging Your Data with Wavefront
Understand how tagging can help you get the results that you need. By default, data include the metric name and the source. Cloud integrations include additional dimensions such as the AWS region, as point tags. You can add point tags to any data source from the UI or the API. Source tags are different. They allow you to group machines, for example, into db machines and web machines. You can then customize your queries to pull out just the data you need. You can also watch the video here
|Organizing with Tags
Want to learn how to use tags? Jason shows an example of assigning a tag (e.g. training) to several dashboards and how the tag can be used for filtering. He then uses additional tags in a hierarchy of tags (e.g. training.attendee and training.trainer) to support more fine-grained filtering using tag paths. Tag paths are available for dashboards, alerts, and events. Finally, for sources, Jason shows you can add source tags and filter directly from a query.
You can also watch the video here .
You use tags in several ways:
- Point tags – Add dimensions to your query with point tags. For example, examine only metrics from a certain region.
- Source tags – Group your sources. For example, examine only production hosts but not development hosts.
Note: Information about the source is part of each metric, but you add source tags explicitly from the UI, CLI, or API.
- Alert tags – Find alerts or exclude tagged alerts from a maintenance window.
- Event Tags – Add event tags from the Events browser or when you create a user event to make it easier to filter events.
- Object tags – Limit the number of objects (e.g. dashboards) and metrics. For example, you might display only dashboards with a certain tag.
- Span tags – Tags that are associated with a span. Certain span tags are required for a span to be valid. In addition, you can instrument an application with custom span tags.
You can use tags to filter alerts, dashboards, events, and sources from the Wavefront UI or with the REST API.
How to Use Different Tag Types
|Tag Type||Usage Example||Add and Update||Query|
|point tag||Filter metrics.||Often part of data source, or you can add using proxy preprocessor rules.||Filter and perform series matching|
|source tag||Filter sources.||Add source tags using API, CLI, or UI (discussed below).||Select only tagged sources in your query.|
Organize related alerts.
Suppress tagged alerts during a maintenance window.
|Create or add alert tags.||Reference alert metrics by tag.|
|event tags||Filter and find events.||Create or add event tags.||Use filters in events() Queries.|
Tag keys can contain alphanumeric (a-z, A-Z, 0-9), dash (-), underscore (_), dot (.), and forward slash (/) characters. The space character is not supported.
When you construct a Wavefront query, you can use a colon (:) in a tag key, only if the tag key is in quotes. For example:
Tags in the UI
In the Wavefront UI:
- Tags display when you hover your mouse over a line, point, etc. in a chart.
- Tags display as gray labeled icons in the filter bar and below each object in the browser for that object (e.g. dashboard or alerts browse).
- Tags on the left of the object browser allow you to filter your display.
Tags in Queries
In ts() and events() queries, you can filter:
- Metrics with source tags and point tags
- Events with:
- alert tags and event tags
- system tags added by alerts (severity, subtype, and type)
Select and Search Using Tag Paths
All tag types support the ability to organize tags in a hierarchy. The hierarchy is defined by separating tag components with a dot (
.), for example,
In dashboards, internal tags start with a tilde, for example,
~system.. To improve readability, tags retain case for display but they are treated as case-insensitive for searching, sorting, etc.
In the UI you operate on tag paths by selecting a component at a specific node in the hierarchy. For example:
- Select all Wavefront dashboards by clicking wavefront
- Select only tutorial dashboards by expanding the wavefront node and then selecting tutorial.
In queries, you achieve the same effect by specifying a tag path with trailing wildcards
".*". For example, enter
alertTagPath.* to match alerts named
alertTagPath.tpc1.tpc11, and so on.
When you create maintenance windows, you can use tag paths and wildcards to put a group of alerts in maintenance.
Point tags offer a powerful way of labeling data so that you can slice and dice it in almost any way you can imagine. For example, you can use point tags, to label a point’s datacenter, version, etc. and can then group by datacenter or version.
You use point tags to add extra dimensions to your data, and can then focus your exploration just on that dimension.
- Fine Tune Queries with Point Tags explains how to use point tags.
- Pairing Up Matching Series discusses how implicit series matching lets you operate on pairs of time series that have corresponding sources and point tags.
Object Tags: Tags on Alerts, Dashboards, Events, and Sources
Object tags include alert tags, dashboard tags, event tags, and source tags.
Manage Object Tags
To add tags to one or more objects:
- Open an object browser.
- For some objects, you select Browse > <object>, where <object> is Sources or Events.
- For other objects, such as Alerts, you click the object.
- Choose which objects to tag:
- Select the check boxes next to the object and click the + Tag button.
Click + at the bottom and select a tag or click Create New Tag.
- In the Add Tag dialog, click the Create Tag button at the bottom.
- Type a tag name. Tag names can contain alphanumeric (a-z, A-Z, 0-9), dash (-), underscore (_), and colon (:) characters. Tag names are case sensitive. For example, the tags
myappare stored as distinct tags. However, mixed case tag paths are collapsed into one path;
myservice.myappare collapsed into
- Click Add.
Search for Object Tags
When you have many tags in your environment, you can search for tags by typing tag names in the Search box below the Tags heading in the filter bar on the left. As you type, the list of tags is filtered by the search string.
Filter by Object Tags
To filter by a tag, click a tag icon. You can click the icon in the filter bar on the left or below an object in an object browser.
A metric in Wavefront Data Format includes a source name. If source names change frequently or if you want to filter sources, a source tag can help. Source tags are just strings–in contrast, point tags are key-value pairs.
You can add source tags from the UI or API, or you can inject source tags and source descriptions directly at the proxy.
Why Source Tags?
Source tags let you to group sources. You can specify a source tag in a query to refer to an entire group of sources in a simple expression. For example, if you have two sources,
appServer16 you can add the source tag
app to both sources to specify that both are app servers. You can then query
ts(cpu.load.metric, tag=app) instead of
ts(cpu.load.metric, source=appServer15 or source=appServer16)
Your use case determines how to use source tags:
- Use a source tag hierarchy, that is, have source tags dot-delimited, for example
env.cluster.role.role1. In that case, your query might include
... and tag=env.cluster.role.*
- Use source tags as intersection sets, that is, use multiple tags (e.g.
role, etc). In that case, your query might include
... and tag=env and tag=cluster.
Add Source Tags from the UI
To add a source tag from the UI:
- Click Browse>Sources.
- Select one or more sources and click +Tag or click the + icon below the source. You can add an existing source tag or create a new source tag.
Add Source Tags from the API
You can add source tags using the Wavefront REST API. The API supports getting and setting source tag values.
For details about the APIs, click the gear icon in your Wavefront instance and select API Documentation.
Group by Source Tag in Queries
Aggreggation functions include a group by parameter that allows you to group the results. The syntax is the following (here we use
sum as an example):
sum(<tsExpression> [,metrics|sources|sourceTags|pointTags|<pointTagKey> ])
For example the following query aggregates all time series but groups the result by point tags. That means you see 1 line if there are no point tags defined for the metric, and multiple lines, one for each point tag, if they are defined for the metric:
sum(ts(dataingester.report-points AND source="dev-2b-*"), pointTags)
When you want to group by source tag, however, you have to include the source tag name in the query, as shown in this example:
sum(ts(dataingester.report-points, source="dev-2b-*" and tag=mySourceTag), sourcetags)
Manage SourceTag and SourceDescription Properties at the Proxy
You can send metrics directly to the Wavefront proxy, and you can add source tags and source descriptions using the
SourceDescription properties. Proxy 4.24 and later supports these properties. Starting with proxy 5.0, each property works with
You send these properties to the same listening port as regular metrics. The port defaults to 2878, and you can change it with the
pushListenerPorts setting in the proxy configuration file.
Here are some simple examples:
Add source tags
red to all metrics coming from the source
@SourceTag action=add source=app-1 highPriority red
Replace all existing source tags for source
@SourceTag action=save source=app-2 qa-42
Delete the source description
app-3 reserved for qa from metrics coming from the source
@SourceDescription action=delete source=app-3 "app-3 reserved for qa"
The syntax is the same for both the SourceTag and the SourceDescription property.
@SourceTag|@SourceDescription <action> <source> <value>
|action||Can be add, save, or delete. When you specify save, you replace any existing source tags or source descriptions with the new source tag or source description.|
|source||Source to which you want to add a source tag or source description.|
|value||Name of the tag or the description. Use quotes for values that include spaces or special characters.|
See Data Naming for examples of tags and tag naming.
Read the blog post Skyline Resolves Production Incidents Faster with Alert-Based Health Dashboards for a discussion of a real-world example that uses alert tags.