#NeurIPS2024 paper 3, Assemblage - the dataset of source-to-binary projects compiled from GitHub that you've dreamed of bet never had before! Collab with @krismicinski.bsky.social and a multi-year effort to get to @NeurIPSConf @BoozAllen arxiv.org/abs/2405.03991 - ThreadSky

edwardraffml.bsky.social • 86 days ago

#NeurIPS2024 paper 3, Assemblage - the dataset of source-to-binary projects compiled from GitHub that you've dreamed of bet never had before! Collab with @krismicinski.bsky.social and a multi-year effort to get to @NeurIPSConf @BoozAllen https://arxiv.org/abs/2405.03991

Comments

edwardraffml.bsky.social•86 days ago

This dataset is big, and we put in a ton of effort to have a bunch of old versions of MSVC and random compilation flags so that you get a lot more diversity. Too much research in this space is "whatever compiles easily on Linux", and the results frankly just can't be trusted.

edwardraffml.bsky.social•86 days ago

We've got a whole infrastructure for dealing with multiple platforms and compilers and keeping track of metadata so that others can build, extend, modify, and customize as they need in the future. We even did an analysis of cost to reproduce and replicate our dataset!

edwardraffml.bsky.social•86 days ago

Download data from https://assemblage-dataset.net/ and you can see how so much changes once you get some more data. e.g., Compiler Provenance seems "solved" if you look at current literature, but just adding our data shows how hard it still is w/ massive perf drop going from Linux to MS

edwardraffml.bsky.social•86 days ago

Looking at function boundary identification and similarity eval, we again see a big drop just changing the data. And we don't claim Assemblage is the *best* or *in-the-wild-matching*, just that it is better and more diverse than what is used today.

edwardraffml.bsky.social•86 days ago

Read the labor of love at https://arxiv.org/abs/2405.03991, and explore it yourself! We've got a ton of information ready-to-go for all sorts of experiments to be supported via Assemblage. We knew we needed this kind of data to do a lot of the R&D we had planned, and now it is here!

Comments

Posting Rules

Reply