If you're running Macrobenchmark tests to compute interaction latencies, you should check whether the distribution of results is a normal distribution. If it's not normal, compare outlier traces with traces that are close to the median and you will find systemic perf issues! Thread 🧵 #AndroidDev - ThreadSky

p-y.wtf • 1 day ago

If you're running Macrobenchmark tests to compute interaction latencies, you should check whether the distribution of results is a normal distribution.

If it's not normal, compare outlier traces with traces that are close to the median and you will find systemic perf issues!

Thread 🧵

#AndroidDev

Comments

rahulrav.com•1 day ago

I agree. We wrote plot-benchmarks (https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:development/plot-benchmarks/README.md) to make it simpler to read these distributions. This also makes it easy to compare benchmarks before / after normalization between runs; and also provides a very valuable metric of statistical significance.

rahulrav.com•1 day ago

I should really make this tool easier to discover, and use.

Being able to compare perfetto traces is fantastic. I have been working on writing something to make this part simpler as well. Hopefully will have something to share soon.

p-y.wtf•1 day ago

Two years ago I wrote about how to compare 2 benchmark results (i.e. don't just compare two averages or 2 medians)

Read more here: https://blog.p-y.wtf/statistically-rigorous-android-macrobenchmarks

p-y.wtf•1 day ago

Since then, we've had folks at Square write and run many Macrobenchmark tests, evaluating the impact of their changes with the script I provided here:

https://blog.p-y.wtf/a-script-to-compare-two-macrobenchmarks-runs

p-y.wtf•1 day ago

One of the checks in that script is that each distribution of latencies passes the Shapiro-Wilk normality test.

Over time, I found out that engineers didn't really know what to do when a normality test fails, or even what that really means.

One eng asked if they could "see" the distribution

p-y.wtf•1 day ago

... and all I heard was "opportunity to have fun with the Android Canvas". I wrote code to assign data points into buckets, remove outliers to compute normal parameters, then draw a histogram + the distribution line & its normal version using bezier curves.

This screenshot is 100% Android Canvas

madonawambua.bsky.social•1 day ago

Huh, this is cool

p-y.wtf•1 day ago

So now you can actually *see* how the distribution isn't normal. Clearly there's a really odd data point here.

Which brings me back to the start of this thread: If it's not normal, compare outlier traces with traces that are close to the median and you will find systemic perf issues!

p-y.wtf•1 day ago

so I wrote a Kotlin script to compare perfetto traces. It parses a trace looking for known blocks and turns that into a trace. Two traces = 2 trees, and then we print out a node for node diff.

Here's an example of what that looks like.

This quickly tells me what's changed, e.g. HeapTaskDaemon = GC

Comments

Posting Rules

Reply