If a speaker is forcing the moderator to be "rude" (i.e., interrupt, or declare time over) then the speaker has already been rude themselves.

It is a pact on a good panel that everyone allows everyone their time. People who violate that are not deserving of special treatment.