The "use tail sampling for your traces" advice should probably also come with a strong "you must be this tall to ride this ride" caveat Simple if you have a monolith, challenging if you have a cluster of microservices, jesus take the wheel if you have geo-distributed traces crossing regions - ThreadSky

jeremymorrell.dev • 68 days ago

The "use tail sampling for your traces" advice should probably also come with a strong "you must be this tall to ride this ride" caveat

Simple if you have a monolith, challenging if you have a cluster of microservices, jesus take the wheel if you have geo-distributed traces crossing regions

Comments

hashimcolombowala.com•68 days ago

100%. And if you also do head sampling because of performance or cost reasons, then you need to be even taller to ride it.

boten.ca•68 days ago

How much RAM have you got? 😅

olane.bsky.social•67 days ago

Not even simple if you have a monolith, necessarily! We have many long running (multi hour) jobs and an often forgotten con of tail sampling is that you must wait for the end before getting _any_ of the data. Really limits how well you can ask “what is my system doing *right now*?”

jeremymorrell.dev•67 days ago

I was going to throw in another point about transactions operating on different time scales, but ran out of space 😅 It’s just way harder of a problem than it initially sounds like it will be!

olane.bsky.social•67 days ago

It’s a problem even apart from sampling tbh - we’ve had to add metrics that somewhat duplicate some of the top level spans just to be able to live-answer “how many jobs are we running”, even though the trace data can do that much better for retrospective analysis. But yeah much hidden complexity.

jeremymorrell.dev•67 days ago

Thanks for the insight! I’m adding this to the “situations where metrics can be better than traces” list that I’m collecting for a future blog post

hazelweakly.me•64 days ago

Do you want some of my mathematical ramblings that break down the concept of information density in distributed systems and uses that as a way to understand how to blend traces, metrics, and other things together effectively? 👀

jeremymorrell.dev•63 days ago

Ummm... yes please 👀

bethcodes.bsky.social•63 days ago

I would love to hear about your methods, and then maybe borrow them to describe the information content of currency in economic systems

dbsmasher.com•64 days ago

Sir where can I subscribe to this newsletter?

jeremymorrell.dev•63 days ago

No newsletter, but blog is here with an RSS feed! https://jeremymorrell.dev/blog/

(Should I add an email subscription? 🤔)

olane.bsky.social•67 days ago

Nice. For us I think it’s basically this liveness thing, shared resources that don’t have a trace they belong to, and things where the volume of spans would be too large to be practical so we’re forced to pre-aggregate. We tell folks it’s a last resort, as a rule of thumb.

lambrospetrou.com•68 days ago

In a service I was working last year, I wanted 100% of *my traces*, and sampled downstream spans.

The all or nothing sampling is not ideal if some downstream creates thousands of spans and you don't want them.

I ended up not propagating the sampling decision to cut theirs.

Sampling is hard, yes.

isburmistrov.bsky.social•68 days ago

IMO sampling is one of the most undercovered topics in the whole o11y area. There are either claims "sampling is not needed" (confusing, untrue), or this one you mentioned. There are no proper guidances, examples of how to do it properly.

jeremymorrell.dev•68 days ago

I’m not actually sure who is solving this at scale, nor how they go about it! I’d love to find some papers on this topic

It sounds like an easy problem! But once you get into the details you realize it’s a lot more challenging than it appears

marty.co.nz•68 days ago

I bet Ben Segelman has some great thoughts about this but not sure they are on bsky yet

isburmistrov.bsky.social•68 days ago

Yep exactly, would also be happy to read something about it. Especially curious about how folks are going with accuracy problems when sampling can lead to inaccurate picture - do they have some more data for specific sensitive places or what. Interesting!

neilgilbert.bsky.social•68 days ago

I just assumed that you would sample within a domain. If you want a trace say end to end of a customer journey then, well, yeah, good luck I guess!

phillipcarter.dev•65 days ago

A way we're hoping to tackle this at honeycomb is better hydration of missing data. So not just "whoopsie, select time range, wait a few hrs, and you'll have it" but support a level of granularity like "oh we don't have this trace? cool we'll grab only that for you".

jeremymorrell.dev•65 days ago

I’m not sure I follow. Does this assume that you’re delivering 100% of traces to archival storage of some kind?

phillipcarter.dev•65 days ago

Yep. A surprising amount of orgs will do that, or at least 100% of the "fed into the tail sampler" traces. There's the truly big orgs with no requirements for audits who gladly toss away stuff via head sampling first though, because the cost for archiving 100% would be too high

phillipcarter.dev•65 days ago

You could extend this to some known-decent MoE and take a random sampling of traces from the archival storage until you get to a good MoE as well, so you can at least ensure you're representative.

phillipcarter.dev•65 days ago

.....but pricing sensitivity will always throw some kind of wrench into those gears.

Sampling is just really hard for orgs who understand it, and extremely hard for those who have an uneven understanding of it

surrealjones.bsky.social•66 days ago

We're in the middle (prod freeze due to holidays) of rolling out some fun stuff to oversample for rare RPCs or specific types of errors, which should really help improve our effective visibility. I should poke the folks doing it re writing / speaking about it publicly after it's done.

sloughter.dev•67 days ago

Not exactly sampling, but curious of your and Jeremy's thoughts on "levels". I don't know that doing the same as log levels makes sense (debug->error) but maybe an integer(0-5), then a piece of TraceState can be added at any point to keep later spans > some level.

sloughter.dev•67 days ago

I need to go re-read the past proposals for levels. I figure it could always be added without changing any protos through attributes and a custom sampler to prove out the usefulness or not. Feels like more controlled sampling. Not for sampling out whole Traces obv, that'd still be necessary too.

jeremymorrell.dev•66 days ago

I built “verbose” tracing (special spans emitted x% of the time, no-ops otherwise), but no one ended up adopting it, so I don’t have any experience with it. I’ve never felt the need and fear adding more complexity.

sloughter.dev•68 days ago

There is the Sampling SIG, though I don't know what they are actually working on at the moment. And the DevEx SIG has discussed where we can help in sampling but have nothing concrete yet :(

aparker.io•68 days ago

there’s been slow but somewhat consistent progress in the sampling sig aiui

aparker.io•68 days ago

the problem imo is that “tail sampling” isn’t actually just one thing, it’s a whole constellation of things that involve trading off between data availability and resolution and cost and the answer is kinda different for everybody

Comments

Posting Rules

Reply