This is an important and interesting preprint: https://doi.org/10.1101/2024.12.18.628606

Any thoughts from people with neural network expertise? Papers describing these models often show plausibly that unsupervised training has learnt something about the data. Why is this no better than random for downstream tasks?