That looks to me like it's using its knowledge - does that tool provide any kind of UI indication when it's actually executing code?

It should do: the ChatGPT and Claude UIs both have clear code execution visual hints, which is definitely the right design decision