---
aliases:
  - /docs/tempo/latest/server_side_metrics/span_metrics/
  - /docs/tempo/latest/metrics-generator/span_metrics/
title: Generate metrics from spans
weight: 400
---

# Generate metrics from spans

The span metrics processor generates metrics from ingested tracing data, including request, error, and duration (RED) metrics.

Span metrics generate two metrics:

- A counter that computes requests
- A histogram that tracks the distribution of durations of all requests

Span metrics are of particular interest if your system is not monitored with metrics,
but it has distributed tracing implemented.
You get out-of-the-box metrics from your tracing pipeline.

Even if you already have metrics, span metrics can provide in-depth monitoring of your system.
The generated metrics will show application level insight into your monitoring,
as far as tracing gets propagated through your applications.

Last but not least, span metrics lower the entry barrier for using [exemplars](https://grafana.com/docs/grafana/latest/basics/exemplars/).
An exemplar is a specific trace representative of measurement taken in a given time interval.
Since traces and metrics co-exist in the metrics-generator,
exemplars can be automatically added, providing additional value to these metrics.

## How to run

To enable service graphs in Tempo/GET, enable the metrics generator and add an overrides section which enables the `span-metrics` generator. See [here for configuration details]({{< relref "../configuration/#metrics-generator" >}}).

## How it works

The span metrics processor works by inspecting every received span and computing the total count and the duration of spans for every unique combination of dimensions.
Dimensions can be the service name, the operation, the span kind, the status code and any attribute present in the span.

This processor is designed with the goal to mirror the implementation from the OpenTelemetry Collector of the [processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/spanmetricsprocessor) with the same name.

### Metrics

The following metrics are exported:

| Metric                         | Type      | Labels     | Description                  |
| ------------------------------ | --------- | ---------- | ---------------------------- |
| traces_spanmetrics_latency     | Histogram | Dimensions | Duration of the span         |
| traces_spanmetrics_calls_total | Counter   | Dimensions | Total count of the span      |
| traces_spanmetrics_size_total  | Counter   | Dimensions | Total size of spans ingested |

{{% admonition type="note" %}}
In Tempo 1.4 and 1.4.1, the histogram metric was called `traces_spanmetrics_duration_seconds`. This was changed later to be consistent with the metrics generated by the Grafana Agent and the OpenTelemetry Collector.
{{% /admonition %}}

By default, the metrics processor adds the following labels to each metric: `service`, `span_name`, `span_kind`, `status_code`, `status_message`, `job`, `instance`.
Additional user defined labels can be created using the [`dimensions` configuration option]({{< relref "../configuration/#metrics-generator" >}}).
When a configured dimension collides with one of the default labels (e.g. `status_code`), the label for the respective dimension is prefixed with double underscore (i.e. `__status_code`).

Custom labeling of dimensions is also supported using the [`dimension_mapping` configuration option]({{< relref "../configuration/#metrics-generator" >}}).

An optional metric called `traces_target_info` using all resource level attributes as dimensions can be enabled in the [`enable_target_info` configuration option]({{< relref "../configuration/#metrics-generator" >}}).

If you use ratio based sampler you can use custom sampler below to not lose metric information, you also need to set `metrics_generator.processor.span_metrics.span_multiplier_key` to `"X-SampleRatio"`

```go
package tracer
import (
	"go.opentelemetry.io/otel/attribute"
	tracesdk "go.opentelemetry.io/otel/sdk/trace"
)

type RatioBasedSampler struct {
	innerSampler        tracesdk.Sampler
	sampleRateAttribute attribute.KeyValue
}

func NewRatioBasedSampler(fraction float64) RatioBasedSampler {
	innerSampler := tracesdk.TraceIDRatioBased(fraction)
	return RatioBasedSampler{
		innerSampler:        innerSampler,
		sampleRateAttribute: attribute.Float64("X-SampleRatio", fraction),
	}
}

func (ds RatioBasedSampler) ShouldSample(parameters tracesdk.SamplingParameters) tracesdk.SamplingResult {
	sampler := ds.innerSampler
	result := sampler.ShouldSample(parameters)
	if result.Decision == tracesdk.RecordAndSample {
		result.Attributes = append(result.Attributes, ds.sampleRateAttribute)
	}
	return result
}

func (ds RatioBasedSampler) Description() string {
	return "Ratio Based Sampler which gives information about sampling ratio"
}
```

### Filtering

In some cases, you may want to reduce the number of metrics produced by the `spanmetrics` processor. You can configure the processor to use an `include` filter to match criteria that must be present in the span in order to be included. Following the include filter, an `exclude` filter may be used to reject portions of what was previously included by the filter policy.

Currently, only filtering by resource and span attributes with the following value types is supported.

- `bool`
- `double`
- `int`
- `string`

Additionally, these intrinsic span attributes may be filtered upon:

- `name`
- `status` (code)
- `kind`

The following intrinsic kinds are available for filtering.

- `SPAN_KIND_SERVER`
- `SPAN_KIND_INTERNAL`
- `SPAN_KIND_CLIENT`
- `SPAN_KIND_PRODUCER`
- `SPAN_KIND_CONSUMER`

Intrinsic keys can be acted on directly when implementing a filter policy. For example:

```yaml
---
metrics_generator:
  processor:
    span_metrics:
      filter_policies:
        - include:
            match_type: strict
            attributes:
              - key: kind
                value: SPAN_KIND_SERVER
```

In this example, spans which are of `kind` "server" are included for metrics export.

When selecting spans based on non-intrinsic attributes, it is required to specify the scope of the attribute, similar to how it is specified in TraceQL. For example, if the `resource` contains a `location` attribute which is to be used in a filter policy, then the reference needs to be specified as `resource.location`. This requires users to know and specify which scope an attribute is to be found and avoids the ambiguity of conflicting values at differing scopes. The following may help illustrate.

```yaml
---
metrics_generator:
  processor:
    span_metrics:
      filter_policies:
        - include:
            match_type: strict
            attributes:
              - key: resource.location
                value: earth
```

In the above examples, we are using `match_type` of `strict`, which is a direct comparison of values. An additional option for `match_type` is `regex`. This allows users to build a regular expression to match against.

```yaml
---
metrics_generator:
  processor:
    span_metrics:
      filter_policies:
        - include:
            match_type: regex
            attributes:
              - key: resource.location
                value: eu-.*
        - exclude:
            match_type: regex
            attributes:
              - key: resource.tier
                value: dev-.*
```

In the above, we first include all spans which have a `resource.location` that begins with `eu-` with the `include` statement, and then exclude those with begin with `dev-`. In this way, a flexible approach to filtering can be achieved to ensure that only metrics which are important are generated.

## Example

<p align="center"><img src="../span-metrics-example.png" alt="Span metrics overview"></p>
