second order optimization methods are having a bit of a moment/resurgence right now. suppose we solve those, what's next then? what is optimal optimization?
I wrote a new small note on my thoughts on what infinite order optimization methods could look like (link in next post)