Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/sources/tempo/introduction/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
aliases:
- /docs/tempo/getting-started/traces
- /docs/tempo/traces
- /docs/grafana-cloud/send-data/traces/what-are-traces
description: "What are traces? Learn about traces and how you can use them."
keywords:
- Grafana
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,17 @@ refs:
destination: https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-generator/
- pattern: /docs/enterprise-traces/
destination: https://grafana.com/docs/enterprise-traces/<ENTERPRISE_TRACES_VERSION>/metrics-generator/
- pattern: /docs/grafana-cloud/
destination: https://grafana.com/docs/grafana-cloud/send-data/traces/metrics-generator/
span-metrics:
- pattern: /docs/tempo/
destination: https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-generator/span_metrics/
- pattern: /docs/enterprise-traces/
destination: https://grafana.com/docs/enterprise-traces/<ENTERPRISE_TRACES_VERSION>/metrics-generator/span_metrics/
- pattern: /docs/grafana/
destination: https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-generator/span_metrics/
- pattern: /docs/grafana-cloud/
destination: https://grafana.com/docs/grafana-cloud/send-data/traces/metrics-generator/
---

# Identify bottlenecks and establish SLOs
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,10 @@ refs:
destination: https://grafana.com/docs/tempo/<TEMPO_VERSION>/traceql/
- pattern: /docs/enterprise-traces/
destination: https://grafana.com/docs/enterprise-traces/<ENTERPRISE_TRACES_VERSION>/traceql/
- pattern: /docs/grafana/
destination: https://grafana.com/docs/tempo/<TEMPO_VERSION>/traceql/
- pattern: /docs/grafana-cloud/
destination: https://grafana.com/docs/tempo/<TEMPO_VERSION>/traceql/
---

# Diagnose errors with traces
Expand Down
54 changes: 54 additions & 0 deletions docs/sources/tempo/shared/traceql-query-structure.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
---
headless: true
description: Shared file for TraceQL query structure concept.
labels:
products:
- enterprise
- oss
---

[//]: # 'This file explains TraceQL query structure.'
[//]: # 'This shared file is included in these locations:'
[//]: # '/grafana/docs/sources/datasources/tempo/traceql/trace-structure.md'
[//]: # '/grafana/docs/sources/datasources/tempo/introduction/trace-structure.md'
[//]: # '/explore-profiles/docs/concepts/trace-structure.md'
[//]: # '/website/docs/grafana-cloud/send-data/traces/trace-structure.md'
[//]: #
[//]: # 'If you make changes to this file, verify that the meaning and content are not changed in any place where the file is included.'
[//]: # 'Any links should be fully qualified and not relative.'

<!-- TraceQL query structure -->

The purpose of TraceQL is to search or query for spans.
The query returns a set of spans, also called a spanset.

A TraceQL query can select traces based on:

- span attributes, timing and duration
- structural relationships between spans
- aggregated data from the spans in a trace

As with PromQL and LogQL, the query is structured as a pipeline of operations (filters and aggregators).
The query expression is evaluated on one trace at a time, selecting or discarding spans from the result.
At each stage of the query pipeline, the selected spans for a trace are grouped in a spanset (set of spans).
The associated trace is also returned. The result of the query is the spansets (and their associated traces) for all the traces evaluated.

The simplest query is this one:

```
{ }
```

The curly braces encompass the select/filter conditions.
In theory, each span (and the trace it belongs to) matching those conditions is returned by the query.
In the previous example, since there are no filter conditions, all spans are matching and thus returned with their associated traces.

In practice, the query is performed against a defined time interval, relative (for example, the last 3 hours) or absolute (for example, from X date-time to Y date-time).
The query response is also limited by the number of traces (**Limit**) and spans per spanset (**Span Limit**).

![TraceQL in Grafana](/media/docs/tempo/traceql/TraceQL-in-Grafana.png)

1. TraceQL query editor
2. Query options: **Limit**, **Span Limit** and **Table Format** (Traces or Spans).
3. Trace (by Trace ID). The **Name** and **Service** columns are displaying the trace root span name and associated service.
4. Spans associated to the Trace
45 changes: 27 additions & 18 deletions docs/sources/tempo/traceql/_index.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Query with TraceQL
menuTitle: Query with TraceQL
title: TraceQL
menuTitle: TraceQL
description: Learn about TraceQL, Tempo's query language for traces
weight: 600
aliases:
Expand All @@ -11,7 +11,7 @@ keywords:
- TraceQL
---

# Query with TraceQL
# TraceQL

Inspired by PromQL and LogQL, TraceQL is a query language designed for selecting traces in Tempo. Currently, TraceQL query can select traces based on the following:

Expand All @@ -22,37 +22,41 @@ Read the blog post, [Get to know TraceQL](/blog/2023/02/07/get-to-know-traceql-a

{{< vimeo 796408188 >}}

For information on where the language is headed, see [future work]({{< relref "./architecture#future-work" >}}).
The TraceQL language uses similar syntax and semantics as [PromQL](/blog/2020/02/04/introduction-to-promql-the-prometheus-query-language/) and [LogQL](/docs/loki/latest/logql/), where possible.

Check the [release notes]({{< relref "../release-notes" >}}) for the latest updates to TraceQL.
Check the [Tempo release notes](https://grafana.com/docs/tempo/<TEMPO_VERSION>/release-notes/) for the latest updates to TraceQL.

## Requirements

TraceQL requires Tempo’s Parquet columnar format, which is enabled by default. For information on Parquet, refer to the [Apache Parquet backend]({{< relref "..//configuration/parquet" >}}) Tempo documentation.
TraceQL requires the Parquet columnar format, which is enabled by default.
For information on Parquet, refer to the [Apache Parquet backend](http://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/parquet) documentation.

## TraceQL queries in Grafana Explore
## Query using TraceQL

You can use the TraceQL query editor and query builder in the Tempo data source to build queries and drill-down into result sets.
The editor and builder are available in Grafana’s Explore interface.
For more information, refer to [Write TraceQL queries in Grafana]({{< relref "./query-editor" >}}).
You can use TraceQL query editor and query builder in the Tempo data source to build queries and drill-down into result sets.
The editor and builder are available in the [Tempo data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/) for Grafana Explore.

<p align="center"><img src="assets/query-editor-http-method.png" alt="Query editor showing request for http.method" /></p>

In addition, you can use Explore Traces to investigate your tracing data without writing TraceQL queries.
For more information, refer to the [Explore Traces](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/simplified-exploration/traces/) documentation.

### Stream query results

By streaming results to the client, you can start to look at traces matching your query before the entire query completes.

The [GRPC streaming API endpoint]({{< relref "../api_docs#tempo-grpc-api" >}}) in Tempo’s query frontend allows a client to stream search results from Tempo.
The GRPC streaming API endpoint in the query frontend allows a client to stream search results from Tempo.
The `tempo-cli` also uses this streaming endpoint.
For more information, refer to the [Tempo CLI documentation]({{< relref "../operations/tempo_cli#query-api-command" >}}).
For more information, refer to the [Tempo CLI documentation](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/tempo_cli/#query-api-command).

To use streaming in Grafana, you must have `stream_over_http_enabled: true` enabled in Tempo.
For information, refer to [Tempo GRPC API](https://grafana.com/docs/tempo/latest/api_docs/#tempo-grpc-api).
For information, refer to [Tempo GRPC API](https://grafana.com/docs/tempo/<TEMPO_VERSION>/api_docs/#tempo-grpc-api).

## Construct a TraceQL query

In TraceQL, a query is an expression that is evaluated on one trace at a time. The query is structured as a set of chained expressions (a pipeline). Each expression in the pipeline selects or discards spansets from being included in the results set. For example:
In TraceQL, a query is an expression that's evaluated on one trace at a time.
The query is structured as a set of chained expressions called a pipeline.
Each expression in the pipeline selects or discards spansets from being included in the results set. For example:

```
{ span.http.status_code >= 200 && span.http.status_code < 300 } | count() > 2
Expand All @@ -66,7 +70,7 @@ In this example, the search reduces traces to those spans where:
Queries select sets of spans and filter them through a pipeline of aggregators and conditions.
If, for a given trace, this pipeline produces a spanset then it's included in the results of the query.

Refer to [TraceQL metrics queries](https://grafana.com/docs/tempo/<TEMPO_VERSION>/traceql/metrics-queries/) for examples of TraceQL metrics queries.
Refer to [TraceQL metrics queries](https://grafana.com/docs/tempo/<TEMPO_VERSION>/traceql/metrics-queries/) for examples of TraceQL metrics queries.

### Find traces of a specific operation

Expand Down Expand Up @@ -133,7 +137,7 @@ It's a convenient request to identify misconfigurations and leaks across product
TraceQL automatically queries data contained in arrays.
Support for arrays is available in vParquet4 and on.

If `span.foo` is an array and contains the value `bar`, then it will be found by this query.
If `span.foo` is an array and contains the value `bar`, then this query will locate it.

```
{ span.foo = "bar" }
Expand Down Expand Up @@ -545,7 +549,8 @@ So far, all of the example queries expressions have been about individual spans.
- `min` - The min value of a given numeric attribute or intrinsic for a spanset.
- `sum` - The sum value of a given numeric attribute or intrinsic for a spanset.

Aggregate functions allow you to carry out operations on matching results to further refine the traces returned. For more information on planned future work, refer to [How TraceQL works]({{< relref "./architecture" >}}).
Aggregate functions allow you to carry out operations on matching results to further refine the traces returned.
For more information on planned future work, refer to [How TraceQL works](./architecture).

For example, to find traces where the total number of spans is greater than `10`:

Expand Down Expand Up @@ -601,4 +606,8 @@ TraceQL can select arbitrary fields from spans. This is particularly performant

## Experimental TraceQL metrics

TraceQL metrics are experimental, but easy to get started with. Refer to [the TraceQL metrics]({{< relref "../operations/traceql-metrics.md" >}}) documentation for more information.
TraceQL metrics are experimental, but easy to get started with.
Refer to [the TraceQL metrics](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/traceql-metrics/) documentation for more information.

You can also use TraceQL metrics queries.
For details, refer to [TraceQL metrics queries](https://grafana.com/docs/tempo/<TEMPO_VERSION>/traceql/metrics-queries/).
6 changes: 3 additions & 3 deletions docs/sources/tempo/traceql/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,11 @@ The TraceQL engine connects the Tempo API handler with the storage layer. The Tr
- Pulls spansets from the storage layer and revalidates that the query matches each span
- Returns the search response

The default Tempo search reviews the whole trace. TraceQL provides a method for formulating precise queries so you can zoom in to the data you need. Query results are returned faster because the queries limit what is searched.
The default Tempo search reviews the whole trace. TraceQL provides a method for formulating precise queries so you can zoom in to the data you need. Query results are returned faster because the queries limit what's searched.

For an indepth look at TraceQL, read the [TraceQL: A first-of-its-kind query language to accelerate trace analysis in Tempo 2.0"](/blog/2022/11/30/traceql-a-first-of-its-kind-query-language-to-accelerate-trace-analysis-in-tempo-2.0/) blog post by Trevor Jones.
For a deeper look at TraceQL, read the [TraceQL: A first-of-its-kind query language to accelerate trace analysis in Tempo 2.0](/blog/2022/11/30/traceql-a-first-of-its-kind-query-language-to-accelerate-trace-analysis-in-tempo-2.0/) blog post.

For examples of query syntax, refer to [Construct a TraceQL query]({{< relref "../traceql#construct-a-traceql-query" >}}).
For examples of query syntax, refer to [Construct a TraceQL query](../traceql#construct-a-traceql-query).

{{< vimeo 773194063 >}}

Expand Down
4 changes: 2 additions & 2 deletions docs/sources/tempo/traceql/metrics-queries/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,8 @@ Traces are a unique observability signal that contain causal relationships betwe
TraceQL metrics can help answer questions like this:

* How many database calls across all systems are downstream of your application?
* What services beneath a given endpoint are currently failing?
* What services beneath an endpoint are currently slow?
* What services beneath a given endpoint are failing?
* What services beneath an endpoint are slow?

TraceQL metrics can help you answer these questions by parsing your traces in aggregate.

Expand Down
35 changes: 3 additions & 32 deletions docs/sources/tempo/traceql/trace-structure.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,36 +22,7 @@ The differences being inherent to the very nature of searching spans and traces.

## TraceQL queries

The purpose of TraceQL is to search or query for spans.
The query returns a set of spans, also called a spanset.

A TraceQL query can select traces based on:

- span attributes, timing and duration
- structural relationships between spans
- aggregated data from the spans in a trace

As with PromQL and LogQL, the query is structured as a pipeline of operations (filters and aggregators).
The query expression is evaluated on one trace at a time, selecting or discarding spans from the result.
At each stage of the query pipeline, the selected spans for a trace are grouped in a spanset (set of spans).
The associated trace is also returned. The result of the query is the spansets (and their associated traces) for all the traces evaluated.

The simplest query is this one:

```
{ }
```

The curly braces encompass the select/filter conditions.
In theory, each span (and the trace it belongs to) matching those conditions is returned by the query.
In the previous example, since there are no filter conditions, all spans are matching and thus returned with their associated traces.

In practice, the query is performed against a defined time interval, relative (for example, the last 3 hours) or absolute (for example, from X date-time to Y date-time).
The query response is also limited by the number of traces (**Limit**) and spans per spanset (**Span Limit**).

![TraceQL in Grafana](/media/docs/tempo/traceql/TraceQL-in-Grafana.png)
[//]: # 'Shared content for best practices for traces'
[//]: # 'This content is located in /tempo/docs/sources/shared/trace-structure.md'

1. TraceQL query editor
2. Query options: **Limit**, **Span Limit** and **Table Format** (Traces or Spans).
3. Trace (by Trace ID). The **Name** and **Service** columns are displaying the trace root span name and associated service.
4. Spans associated to the Trace
{{< docs/shared source="tempo" lookup="traceql-query-structure.md" version="<TEMPO_VERSION>" >}}
Loading