I’m currently working with #Rstats on a dataset that contains 2.7 millions rows and my entry-level MacBook Air M3 with 16 Gb of RAM doesn’t even blink Elon Musk is an incompetent idiot surrounded by an army of equally incompetent idiots - ThreadSky

o.simardcasanova.net • 10 days ago

I’m currently working with #Rstats on a dataset that contains 2.7 millions rows and my entry-level MacBook Air M3 with 16 Gb of RAM doesn’t even blink

Elon Musk is an incompetent idiot surrounded by an army of equally incompetent idiots

Reposted from Molly White

Elon Musk's favorite supposed data expert, who he's retweeted at least a dozen times, claims she can only process 60,000 rows of data before her "hard drive overheats"

Comments

denismongin.bsky.social•10 days ago

Same here.
Below 10 million rows, using data.table in #Rstats is just easy. HP notebook, 16G ram

dlc.bsky.social•10 days ago

That’s just hilarious.

o.simardcasanova.net•10 days ago

o.simardcasanova.net•10 days ago

For the non-specialists reading this post: from a technical point of view, this dataset is nothing special

It’s a mundane operation to work on datasets that contains a significantly larger number of rows than 60k

o.simardcasanova.net•10 days ago

I worked on it earlier today and I had Mail, Outlook, Safari with 30+ tabs, Microsoft Teams, Slack, Music, News Explorer, Fantastical, and Reminders all open at the same time and everything worked fine

On a Mac with only 16Gb of RAM

As I said, it’s just mundane

mackaszechno.bsky.social•10 days ago

I concur. I have some geocoding I do in #rstats on 118k pupils against 800k residential addresses. Both sit in memory. The only differential is how I chose to interact with them. Deal with each pupil against all addresses is a 12 hour job. Filtering both on postcode and looping is a 26 minute job.

sixshots.bsky.social•10 days ago

Gotta 3 years old HP Z book portable WS with 32GB & usual for me to process 3M+ lines data sets.
This post is nonsense.

econmaett.bsky.social•10 days ago

Any special precautions?

o.simardcasanova.net•10 days ago

I avoid using for loops, as vectorized code is much faster, but that’s it

econmaett.bsky.social•10 days ago

Damn you‘re good!

o.simardcasanova.net•10 days ago

Thanks but honestly I’m not, in this case "vectorized code" is a fancier way to say that I pipe Tidyverse functions – which is the most mundane R code one can write

o.simardcasanova.net•10 days ago

Plus I hate for loops anyway

shalashaskaman.bsky.social•10 days ago

I used to process NCES data for the entire US on an M1 Air with 8gb and it took maybe ~5 seconds

custosfr.bsky.social•10 days ago

This happen a lot if you're using a "data tool" such as Excel, which is also limited to 65k row... Imagine how far their skill on statistics go.

mackaszechno.bsky.social•10 days ago

.xls not .xlsx

Comments

Posting Rules

Reply