---
title: V2.1
weight: 55
---

# Version 2.1 release notes

The Tempo team is pleased to announce the release of Tempo 2.1.

With this release, Tempo enhances TraceQL so you can select traces using features such as span kind, aribtrary arithmetic, and new aggregates min/max/sum. In addition, Tempo 2.1 introduces vParquet2, a new experimental Parquet version designed to be more compatible with other Parquet implementations.

>**Note**: For a complete list of changes and enhancements, refer to the [Tempo 2.1 changelog](https://github.com/grafana/tempo/releases).

## Features and enhancements

Some of the most important features and enhancements in Tempo 2.1 are highlighted below.

### Enhancements to TraceQL

Inspired by PromQL and LogQL, TraceQL is a query language designed for selecting traces in Tempo. TraceQL makes it possible to find traces that otherwise would have been difficult or even impossible to identify with Tempo’s 1.x search.

With this release, we’ve expanded the TraceQL language. In Tempo 2.0, TraceQL allowed users to select traces based upon resource attributes, time, duration, and basic aggregates. In Tempo 2.1, users can also select traces based on the following:

* Enable querying by `kind`, an intrinsic attribute  (PR [2217](https://github.com/grafana/tempo/pull/2217))
    * `{ kind = server }`, `{ kind = client }`, `{ kind = consumer }`
    * `{ kind = server && duration > 1s }`
* Support arithmetic expressions (PR [2146](https://github.com/grafana/tempo/pull/2146))
    * `{ span.bytesProcessed > 10 * 1024 * 1024 }`
    * `{ span.bytesProcessed / 1024 / 1024 > 10 }`
    * `{ span.jobs / span.bytes > 3 }`
* Enable min, max, and sum aggregate filters
    * `{} | min(duration) > 1s`
    * `{} | max(span.bytesProcessed) &lt; 1024 * 1024`
    * `{} | sum(duration) > 1h`
* Support for more numeric types: Compare floating point numbers and integers, allow floating point numbers for durations, and support negative numbers when doing aggregate filters. (PR [2139](https://github.com/grafana/tempo/pull/2139), [2304](https://github.com/grafana/tempo/pull/2304), [2289](https://github.com/grafana/tempo/pull/2289))
    * `{ 1 = 1.0 }`
    * `{ duration > 1.5s }`, `{duration > 1s500ms }`
    * `{} | count() > -1`

This update also includes major performance improvements to TraceQL query performance. [[PR 2119](https://github.com/grafana/tempo/pull/2119)]

To learn more about the TraceQL syntax, see the [TraceQL documentation](http://localhost:3002/docs/tempo/latest/traceql/). For information on planned future extensions to the TraceQL language, see [future work](http://localhost:3002/docs/tempo/latest/traceql/architecture/).

### TraceQL query editor improvements (experimental)

Starting in Grafana 9.5, users have access to a Tempo Search tab powered by TraceQL. To enable this feature, use Grafana’s `traceqlSearch` feature flag.  The editor can autocomplete queries and has a streamlined interface.

Grafana will soon include a new Search screen that can build a TraceQL query based upon options selected in drop-downs. Contact Support to enable this feature in Grafana Cloud.

![A TraceQL search constructed by selecting options](/media/docs/tempo/traceql/screenshot-explore-traceql-search.png "The new Search tab in Grafana Explore")

For more information, refer to the [TraceQL query editor documentation](http://localhost:3002/docs/tempo/latest/traceql/query-editor/).
### vParquet2 (experimental)

Tempo 2.1 offers an experimental update to the Parquet backend: `vParquet2`. Due to the dedicated span duration column, there is a performance gain for TraceQL queries containing duration. vParquet2 will also be a prerequisite for TraceQL's structural operators, which will be introduced in a future release. [[PR 2244](https://github.com/grafana/tempo/pull/2244)]

You can select the block format Tempo uses by changing the block format version option. To use the new format, set the block format to `vParquet2`. For more information, read the [Parquet configuration documentation](https://grafana.com/docs/tempo/next/configuration/parquet/#choose-a-different-block-format).

The new encoding implements Parquet schema changes that are not compatible with `vParquet`:

* Renamed columns to make the schema more consistent and closer to names and terminologies used by OTEL
* Add a dedicated span column for duration
* Additional columns that will enable new TraceQL features, including hierarchical queries

### Other enhancements and features

For a complete list, refer to the [Tempo 2.1 changelog](https://github.com/grafana/tempo/releases).

* PR [2105](https://github.com/grafana/tempo/pull/2105) Log when a trace is too large to compact. Traces are dropped when they exceed a certain limit set in the ingesters. Any dropped traces are now logged with the number of dropped spans and the trace ID.
* PR [2195](https://github.com/grafana/tempo/pull/2195) Add support for Azure Workload Identity authentication
* PR [2130](https://github.com/grafana/tempo/pull/2130) Add command to  tempo-cli to migrate a tenant
* PR [2093](https://github.com/grafana/tempo/pull/2093) Add support for s3 session token in static configuration

## Upgrade considerations

When [upgrading](http://localhost:3002/docs/tempo/latest/setup/upgrade/) to Tempo 2.1, be aware of these breaking changes.

### Breaking changes to metric names exposed by Tempo

All Prometheus metrics exposed by Tempo on its `/metrics` endpoint that were previously prefixed  with `cortex_` have now been renamed to be prefixed with `tempo_` instead. (PR [2204](https://github.com/grafana/tempo/pull/2204))

Tempo now includes SLO metrics to count where queries are returned within a configurable time range. (PR [2008](https://github.com/grafana/tempo/pull/2008))

The `query_frontend_result_metrics_inspected_bytes` metric was removed in favor of `query_frontend_bytes_processed_per_second`.

### Remove support for Search on v2 blocks

Users can no longer search blocks in v2 format. Only vParquet and vParquet2 formats support search. The following search configuration options were removed from the overrides section:

```
overrides:
  max_search_bytes_per_trace:
  search_tags_allow_list:
  search_tags_deny_list:
```

The following metrics configuration was also removed:

```
tempo_ingester_trace_search_bytes_discarded_total
```

This change means that if you upgrade directly from 1.9 to 2.1, you will not be able to search your v2 blocks.
One way to avoid this is to upgrade to 2.0 first, since 2.0 supports searching both v2 and vParquet blocks.
You can let your old v2 blocks gradually age out while Tempo creates new vParquet blocks from incoming traces.
Once all of your v2 blocks have been deleted and you only have vParquet format-blocks, you can upgrade to Tempo 2.1.
All of your blocks will be searchable.

Finally, parquet files are no longer cached when carrying out searches ([PR 2164](https://github.com/grafana/tempo/pull/2164)).

## Bug fixes

For a complete list, refer to the [Tempo 2.1 changelog](https://github.com/grafana/tempo/releases).

### 2.1.1

* [PR 2400](https://github.com/grafana/tempo/issues/2400) Fixed issue where Tempo sometimes flips booleans from false->true at storage time.

### 2.1

* [PR 2220](https://github.com/grafana/tempo/pull/2220) Correctly connect context during compaction
* [PR 2081](https://github.com/grafana/tempo/pull/2081)  Apply `rate()` to the bytes/s panel in the tenant's dashboard.
* [PR 2111](https://github.com/grafana/tempo/pull/2111) Retry copy operations during compaction in GCS backend.
* [PR 2164](https://github.com/grafana/tempo/pull/2164) Improve locking and search head block in SearchTagValuesV2
* [PR 2152](https://github.com/grafana/tempo/pull/2152)  Fix not closing WAL block file before attempting to delete the folder.
* [PR 2219](https://github.com/grafana/tempo/pull/2219) Stop searching for virtual tags if there are any hits. This prevents invalid values from showing up for intrinsics like `status`.
* [PR 2254](https://github.com/grafana/tempo/pull/2254) Correctly return unique spans when &&ing and ||ing spansets.
