Data people: what are some signs that your data analysis project is actually a piece of software (with all the associated accoutrements -- requirements gathering, design, testing, packaging, maintenance, etc)? #rstats #python #datascience - ThreadSky

About ThreadSky

ataustin.bsky.social • 507 days ago

Data people: what are some signs that your data analysis project is actually a piece of software (with all the associated accoutrements -- requirements gathering, design, testing, packaging, maintenance, etc)?
#rstats #python #datascience

Comments

tanho.ca•162 days ago

if it or its outputs are meant to be used either more than once or by someone else

ataustin.bsky.social•161 days ago

Would you say that any analytical workflow that regularly ingests new data is software? Is there a distinction between building repeatable data analysis pipelines and software development practices?

tanho.ca•161 days ago

Yup, automated ingest is software to me! I don’t have a huge difference in how I think about repeated/automated analysis pipelines vs software development - only true “I will never run this thing again” stuff falls out of that category for me

olihawkins.com•506 days ago

1. When you keep reusing and adapting it (or parts of it) for new tasks. 2. When people keep asking you to re-run it on new data.

ataustin.bsky.social•506 days ago

I like the idea of "new data" as the bright line that pushes you into the realm of software.

Would this imply building a package? Or can "software" just be a collection of scripts? Whose job would it be to ensure the integrity of the new data?

olihawkins.com•506 days ago

Well I can only speak for myself, but in our team we try to turn anything that keeps getting reused into a package, because it pushes you into explicit versioning, documentation etc. so expected behaviour is more explicitly defined. The package can then be consumed by more loosely defined projects.

Posting Rules

Be respectful to others
No spam or self-promotion
Stay on topic
Follow Bluesky's terms of service

Comments

Posting Rules

Reply