I think the fact that "cold enough to freeze the [x] off a brass monkey" is recorded where x is not balls, many years before the balls reference, shows that the naval origin is, as so often, balls.
I find that unusual that you'd not store them on wood frames rather than brass ones. Yes wood could warp in salt water air but brass corrodes, has said issues, and is much more costly.
"The shot was stored on the gun or spar decks, in shot racks—wooden planks with holes bored into them, known as shot garlands in the Royal Navy, into which round shot was inserted for ready use by the gun crew."
Ah that reminds me of my favourite sign in British Sign Language for biscuit. It is one hand tapping on the elbow of the other. It comes from tapping the weevils out of Ships Biscuit before you ate them.
Comments
https://en.wikipedia.org/wiki/Brass_monkey_(colloquialism)
Everydays a school day 👍👍😁
https://www.theguardian.com/notesandqueries/query/0,5753,-16960,00.html#:~:text=NOOKS%20AND%20CRANNIES-,The%20expression:%20%22It%20is%20cold%20enough%20to%20freeze%20the%20balls,iron%20balls%20to%20fall%20out.