The arXiv is where we keep papers on math, physics, etc. It's run by Cornell University. It seems unlikely that the US government would demand that Cornell destroy the arXiv. But the many copies of the arXiv around the world were shut down on September 15, 2024, and it seems that now all the […] - ThreadSky

johncarlosbaez.mathstodon.xyz.ap.brid.gy • 12 days ago

The arXiv is where we keep papers on math, physics, etc. It's run by Cornell University.

It seems unlikely that the US government would demand that Cornell destroy the arXiv. But the many copies of the arXiv around the world were shut down on September 15, 2024, and it seems that now all the […]

Comments

timohillmann.bsky.social•11 days ago

@arrr.de is working on something related.

arrr.de•11 days ago

Indeed I was in contact with the German National Library of Science and Technology in Hannover (TIB, https://www.tib.eu), who confirmed that they will download and archive the entire https://arxiv.org as a safety measure.

johntimaeus.infosec.exchange.ap.brid.gy•12 days ago

@johncarlosbaez

I'm currently doing a separate archiving project. Anybody got the time, spoons, and space?

mwguy.infosec.exchange.ap.brid.gy•12 days ago

@johntimaeus @johncarlosbaez Of this site or something else entirely.

johntimaeus.infosec.exchange.ap.brid.gy•12 days ago

@mwguy @johncarlosbaez

Other stuff. FB is deleting all old live videos. There's a lot of history and art, particularly early covid time, that's about to be lost.
yt-dlp is a really freaking useful tool.

SafeguardingResearch.fedihum.org.ap.brid.gy•11 days ago

@johntimaeus 👋 Do you want to chat about this?
(also in private, if you like)

@mwguy @johncarlosbaez

steinly0.bsky.social•11 days ago

https://info.arxiv.org/help/bulk_data/index.html

dimpase.mathstodon.xyz.ap.brid.gy•12 days ago

@johncarlosbaez 100 Gb a month? This probably counts generated data, no? I cannot imagine they get 100 Gb of TeX files a month. One medium length paper is about 50 Kb, rougly.

johncarlosbaez.mathstodon.xyz.ap.brid.gy•12 days ago

@dimpase - The arXiv keeps the PDFs, not just the LaTeX. My original post was wrong, too: I said that in March 2023 the arXiv was 2.7 terabytes, but it was actually 5.6. They say "The PDF files are about 2.7 TB, with the source files making up the remaining 2.9 TB of the estimate." I don't know […]

moritz-negwer.mstdn.science.ap.brid.gy•12 days ago

@johncarlosbaez @dimpase So by this estimate, by now the size would be roughly 8 TB.
As per Amazon pricing, traffic from an S3 bucket to the internet is 0.09$ / GB, which would mean the entire download would cost 720$. Including storage on the downloader's end this could come in well under 1k$ […]

johncarlosbaez.mathstodon.xyz.ap.brid.gy•12 days ago

@moritz_negwer - sounds right! Truly reliable storage requires multiple backups, etc., and the right people to do that are people who are already doing that sort of thing. But they exist and I'm just raising the possibility that maybe one of them would like to do this.

moritz-negwer.mstdn.science.ap.brid.gy•11 days ago

@johncarlosbaez Yes, I agree - proper live-service mirroring requires a lot more infrastructure and professionals to manage it.
What surprised me was that a bare-bones backup a) fits on a single consumer HDD, and b) can be downloaded much cheaper than I thought. This should lead to a relative […]

dimpase.mathstodon.xyz.ap.brid.gy•12 days ago

@moritz_negwer @johncarlosbaez one would typically use compression for such downloads; that will make all the TeX files only 25% of their original size.

Probably the bulk of data is experimental data (in experimental sciences), this might be harder to compress.

AuntyRed.aus.social.ap.brid.gy•12 days ago

@dimpase @moritz_negwer @johncarlosbaez unless there is something weird going on the CS side of ArXiv, we don't normally store datasets on there.

Lots of images though, I imagine might take up a lot of room.

Comments

Posting Rules

Reply