Something is shutting down the bot at exactly 11pm PT. I’m getting alerts that the threads are deadlocked for 10s, despite being fully lockless.
Also all processes that can iterate will sleep.
I'm losing my mind. No errors, no exceptions in the log. I don't know where else to look.
Also all processes that can iterate will sleep.
I'm losing my mind. No errors, no exceptions in the log. I don't know where else to look.
Comments
It’s actually stopping usually around 10:20pm, I misread the logs. Backups look like they’re still generating.
Only thing I can think of is that the Unix socket protocol is bugged in python? But I honestly don’t know. I can’t recreate the bug in anything but prod on the cloud.
I think it might be the machine the bot runs on.
It could be a shared library causing it…