Since GPT-4 came out, open source LLMs have scaled datasets much more than compute, at least from the few data points we have. Data from EpochAI - ThreadSky

Since GPT-4 came out, open source LLMs have scaled datasets much more than compute, at least from the few data points we have.

Data from EpochAI