Benchmarking is hard! This is a great example, let's dive into the diff to highlight what I'd typically surface if someone shared this benchmark at Square. The diff: github.com/android/nowi... (it's great that Arnaud shared the diff btw, too often benchmark authors don't) #AndroidDev - ThreadSky

p-y.wtf • 97 days ago

Benchmarking is hard!

This is a great example, let's dive into the diff to highlight what I'd typically surface if someone shared this benchmark at Square.

The diff: https://github.com/android/nowinandroid/compare/main...InsertKoinIO:nowinandroid:perfs_hilt

(it's great that Arnaud shared the diff btw, too often benchmark authors don't)

#AndroidDev

Reposted from Arnaud Giuliani

I am trying the delicate experiment to benchmark #Koin #Kotlin framework with the Now in Android app, having real-world usage measurements: proandroiddev.com/benchmarking...
Sorry for flaming the internet again 😅 Feedback is welcome to help make this experiment process better 🙏

Comments

kaeawc.bsky.social•96 days ago

I'm about to do some build benchmarking myself on the various DI frameworks in both my public tiny build repo as well as much larger ones. Most likely going to check things out in Dagger Anvil (kapt and ksp), Kotlin Inject, and hand rolled. Anything you'd advise besides what you've already written?

p-y.wtf•96 days ago

I'll just copy what @swank.ca wrote: "A good "benchmark" for a DI framework would be a table of when different operations happen: at build time, at build time and incremental, at startup, upon frst use, etc"

p-y.wtf•97 days ago

First: making the app debuggable. A debuggable app performs in significantly slower and different way from a non debuggable app. This can make results unrealistic & unreliable.

p-y.wtf•97 days ago

The benchmark is run via a custom script, not microbenchmark nor macrobenchmark. These tools were written by experts and provide a lot of utility & checks that aren't in this script.

p-y.wtf•97 days ago

"Baseline Profiles Macrobenchmark" is for profiles, unrelated to this. Macrobenchmark proper actually does offer the ability to easily extract benchmark values for custom use, with TraceMetric: https://developer.android.com/reference/androidx/benchmark/macro/TraceMetric

p-y.wtf•97 days ago

A few things missing from this script:

- speed compiling the app
- locking CPU clocks
- checking thermals
- checking battery level

p-y.wtf•97 days ago

For such a benchmark, which is very focused ("How long does it take to create a view model via Koin vs Hilt?"), I would probably use Microbenchmark rather than Macrobenchmark. The former will keep iterating until code is hot & the perf stable.

p-y.wtf•97 days ago

Measuring code time with `System.currentTimeMillis()` can give unreliable results, as time can move backward. I talked about that here: https://dev.to/pyricau/android-vitals-what-time-is-it-2oih

gakisstylianos.bsky.social•97 days ago

Excellent thread PY, thanks for making it!
Are you aware of any good "this is how you get started with macro/micro benchmarking" tutorials/codelabs etc? It's a topic I've never actually looked into at all. Mostly because I feel like even if I do, I'll not make trustworthy conclusions anyway.

Comments

Posting Rules

Reply