Can I create a TraceQL query that filters based on referenced linked spans?

[siegenthalerroger]

We run a event-based microservice system in which we'd want to be able to query traces based on the attributes of a trace from another system. According to the OTEL semantic conventions for distributed traces the traces from each microservice are linked to each other with a referenced linked span. In Grafana this is visually displayed quite nicely.

Now it would be nice to be able to query Tempo to find all traces that are referenced from a trace with a certain name/attributes. I thought that this would be possible with the structural operators but I can't see any operator for referenced spans. Currently finding all traces that were initiated from one action requires how the systems interact, for each possible action. This doesn't really scale with an increasing number of microservices and async interactions.

Following examples are things that I'd like to be able to do:

Finding the source trace of a trace. <- would mean something like "left spanset contains any span that is referenced by any span in the right spanset"

{} <- {name = "lorem.ipsum process"}

Finding the absolute origin of a chain of traces. <* would mean "left spanset contains all spans that are referenced by any span in the right spanset recursively"

{} <* {name = "lorem.ipsum process"}

Finding all spans that are "spawned" from a trace. -> would mean "right spanset contains any spansets that reference the left spanset"

{name = "lorem.ipsum process" && enduser.id = "123456789"}  -> {resource.service.name = "dolor-est"}

This last one is probably impossible to achieve in a performance-optimal way, considering that the spans only reference eachother in the opposite direction. However for our usecase the time-performance isn't that relevant tbh.

[joe-elliott]

We do have some plans to support links detailed here, but they do not include structural operators "through" the link.

https://github.com/grafana/tempo/blob/main/docs/design-proposals/2023-11%20TraceQL%20Extensions.md#link

It would be quite difficult to achieve this b/c a linked span would require doing a full search of the database for the trace id. I'm not sure how to do it efficiently.

[siegenthalerroger]

Very interesting read. Just to verify, the link syntax hasn't been implemented it yet correct? So no way to experiment with it currently.

I do agree that I don't particularly see an efficient way of doing it, however I'm not sure that that's really required in this sense. In our usecase, we'd know exactly what we're looking for and be willing to wait (minutes not hours) for the full chain of events through our distributed system. However, while I do see a "trivial" option of following the spans in the direction they are actually referenced/linked, this is only one of our use-cases. The opposite, where we only know the initial span that will be referenced by the next trace is a much less "trivially" solveable case from my understanding.

[joe-elliott]

For an update here: the proposed link syntax is currently in development. Also, TraceQL now supports trace:id and span:id intrinsics.

The discussed analysis could be done using a script that used the links span/trace ids to create a new query like:
{ trace:id = "foo" && span:id = "bar" } >> { <whatever conditions i care about> }

this is close to what tempo would have to do internally anyway.

[siegenthalerroger]

Looking forward to being able to test the upcoming link syntax and checking out what the performance is like for our use-case.

For the discussed analysis we'd have to see how we could utilise an existing dashboarding tool (Grafana or Superset depending on the exact use-case) while performing such a recursive request.