Monitor usage with ingestion policies, usage dashboards, and alerts.

In addition to the dashboard for monitoring your overall usage of VMware Aria Operations for Applications (formerly known as Tanzu Observability by Wavefront), you can use ingestion policies for monitoring usage by specific accounts, groups, sources, metric namespaces, or point tags. For example, it might be valuable to understand the ingestion rates of the different teams in your organization or by the different sources and manage their consumption, cost, overage, etc.

By creating an ingestion policy, you group a set of accounts, groups, sources, metric namespaces, or point tags. Optionally, you can set a PPS limit associated with an alert. After you create an ingestion policy, you can start monitoring the policy PPS usage per billing period from the ingestion policy dashboard, which shows:

  • The P95 PPS usage out of the limit, if configured
  • The month-over-month percentage change in the PPS usage
  • The hourly PPS usage
  • The accounts that ingest most data
  • The usage by ingestion mechanism (proxy and direct ingestions)
  • The usage by ingestion type (time series, histograms, and delta counters)

For performance monitoring of your Operations for Applications instance, you can use wftop and spy, the Slow Query dashboard, and the Usage integration.

Ingestion Policy Basics

Ingestion policies allow you to combine user and service accounts, groups, sources, metric namespaces, or point tags, so that you can monitor their usage of the Operations for Applications service. For example, you can create a policy for a group of new hires. You can also create a policy for one or more source virtual machines. Also, you can set a Points per Second (PPS) limit for the policy and create an alert, so that you can receive notifications if the PPS usage exceeds certain thresholds of the limit.

Tracking the PPS usage by ingestion policy can help you understand how the overall usage is distributed and whether a particular team will need more PPS in the future or will need to reduce their overhead ingestions.

The policy scope can be accounts, groups, sources, namespaces, or point tags.

  • Once you set the scope, you cannot change it.
  • You can change only the accounts or objects assigned to the policy in accordance with the scope.
  • Each account or object can belong to more than one policy.

Permissions

  • Only Super Admin users can create and edit ingestion policies.
  • Only Super Admin users can edit or delete alerts associated with ingestions policies. Even if you have the Alerts permission, you cannot edit or delete ingestion policy alerts unless you are a Super Admin user.
  • Only Super Admin users can view ingestion policy versions.
  • All users can view the ingestion policies, the ingestion policy dashboards, and the ingestion policy alerts.

Create an Ingestion Policy

Step 0: Start the Ingestion Policy Creation

  1. Log in to your service instance (https://<your_instance>.wavefront.com) as a Super Admin user and enable Super Admin mode.
  2. From the gear icon on the toolbar, select Usage and Subscriptions.
  3. Click the Ingestion Policies tab and click New Ingestion Policy.

Step 1: Specify the Scope and PPS Limit

In the Data panel, specify the scope and, optionally, a PPS limit and click Next.

  1. Select the Scope of the policy and enter the objects from that scope that you want to assign to the policy.

    ScopeDescription
    Accounts Individual user and service accounts.
    Groups Groups of user and service accounts.
    Sources Sources that emit metrics. You can assign exact source names and names with wildcards, for example, appServer1 and appServer*.
    Namespaces Namespaces that group metrics in a hierarchy defined by a name prefix. You can assign exact metric names and namespaces, for example, request. and requests. You can also assign names with wildcards, for example, cpuloadavg* and cpu.*.
    Point Tags Point tags that are optional key-value pairs associated with a metric. You must assign exact tag keys with exact tag values or wildcards, for example, env="dev" or env="*".

    If you assign more than one point tag, you must select the match criterion - can be either Has tags (individual point tags) or Has all these tags (a combination of point tags).

  2. Choose whether you want to set a PPS limit for the policy. If you select Set PPS Limit, you must enter the PPS limit number in the PPS per billing period text box.

    The limit becomes visible in the ingestion policy dashboard charts. If you set a PPS limit, you must create the ingestion policy alert in the next steps.

Step 2: Configure the Ingestion Policy Alert

If you set a PPS limit for the ingestion policy, Operations for Applications creates an ingestion policy alert that queries the PPS usage by the policy as a percentage of the PPS limit.

  1. In the Conditions panel, configure the thresholds and severities.
    1. Select the comparison operator for the alert condition. In most cases, you alert when the usage is greater than a specified threshold percentage of the PPS limit.

    2. For at least one severity, specify a threshold percentage of the PPS limit and click Next.

      The threshold percentage becomes visible in the ingestion policy alert chart.

      You can skip the other settings in the Conditions panel.

  2. Optionally, in the Recipients panel, specify who will receive the alert notifications and click Next.
  3. In the Alert Name and Tags panel, enter a name for the alert and, optionally, tags and click Next.

After you create the ingestion policy, the associated alert will be available on the ingestion policy dashboard and on the Alerts Browser page.

Step 3: Name and Activate the Ingestion Policy

In the Create panel, enter a name for the policy and, optionally, a description and click Create.

Edit an Ingestion Policy

After you create an ingestion policy, if you need, for example, to increase the PPS limit, or add more accounts or point tags, you can edit the policy.

  1. Log in to your service instance (https://<your_instance>.wavefront.com) as a Super Admin user and enable Super Admin mode.
  2. From the gear icon on the toolbar, select Usage and Subscriptions.
  3. On the Ingestion Policies tab, click the ellipsis icon next to the policy that you want to edit and select Edit.
  4. In each panel, apply the necessary changes and click Next.
  5. In the Policy Name and Description panel, click Save.

When you edit an ingestion policy, you create a new version of that policy.

View Ingestion Policy History

To access the version history of an ingestion policy, on the Ingestion Policies page, click the ellipsis icon next to the policy and select Versions.


Ingestion policy version history shows:
  • The changes that have been made to an ingestion policy over time.
  • The user who made the changes.
  • The date and time the changes were made.
  • A description of the changes.
 alert history selected in menu

Delete an Ingestion Policy

If you no longer need an ingestion policy, for example, after a reorganization in your company, you might want to delete an ingestion policy.

  1. Log in to your service instance (https://<your_instance>.wavefront.com) as a Super Admin user and enable Super Admin mode.
  2. From the gear icon on the toolbar, select Usage and Subscriptions.
  3. On the Ingestion Policies tab, click the ellipsis icon next to the policy that you want to delete, select Delete and confirm.

Examine Ingestion Policy Usage

All users can examine the individual ingestion policy dashboards to understand their usage over time.

  1. Log in to your service instance (https://<your_instance>.wavefront.com).

  2. From the gear icon on the toolbar, select Usage Portal.

  3. Click the Ingestion Policies tab and view all existing policies. Ingestion policies page On the Ingestion Policies tab, for each policy you can see:

    • The state of the policy, i.e. whether the limit is exceeded, or the limit is not reached or not set.
    • The name of the policy. If you click the name, you can examine the respective ingestion policy dashboard.
    • The current usage vs limit for the current billing period.
    • The usage trend for the current billing period.
    • The PPS limit, if any.
    • Whether the ingestion policy has an alert associated with it.
    • Last updated information.

  4. Click the name of the policy in which you are interested and examine the policy dashboard.

    The ingestion policy dashboard consists of two main and one optional section.

    • In the Ingestion Summary section of the dashboard, you can see the following list of charts:

      ChartDescription
      Usage Limit Shows the 95th percentile PPS usage by the policy out of the PPS limit for the selected billing period.
      Previous Month to Selected Month Shows a comparison of the selected month's usage with the previous month's usage.
      Hourly Usage Shows the hourly PPS. The red line represents the PPS limit for the policy.

    • In the Optimize Usage section of the dashboard, you can see a list of charts that lets you investigate usage further by accounts, ingestion mechanisms, and data types.

      ChartDescription
      Top Accounts Contributing to Ingestion Shows which accounts contribute the most.
      Usage by Ingestion Source Shows the usage by ingestion mechanism - proxy and direct ingestions.
      Usage by Ingestion Type Shows the usage by data type - time series, histograms, and delta counters.

    • If the ingestion policy has a PPS limit with an alert, the dashboard also includes the Alert section that provides details about the associated alert.

    Usage summary per ingestion policy

In addition, the Operations for Applications Usage integration includes the Operations for Applications Ingestion Policy Explorer Dashboard, which provides a granular breakdown of the ingestion across your organization by ingestion policies, accounts, sources, and types.

Example: Monitor Which Teams Are Responsible for How Much Ingested Data

Consider the following example. You are administering an Operations for Applications service instance for two big teams, IT Team1 and IT Team2, and you want to monitor how much data each of the team uses. IT Team1 usually needs more PPS data, and you have only 10,000 of committed PPS on a monthly basis.

You can create an ingestion policy for each team to monitor how much data each team uses per month. You can also set a PPS limit for each ingestion policy and if a team consumes more than expected, you will receive an alert notification, so that you can provide additional training on how to use ingested data wisely.

Learn More!

Improve PPS and Prevent Overage explains how to use tools and dashboards to learn how much data is coming in, who is sending the data, how to get alerted if ingested data get close to monthly contracted usage, and how to optimize your ingestion rate.

administration dashboards