Why would you not believe that one-fifth of people are fascist or racist? The percentages have been much higher than that at various times in the past. You don't cite any evidence for your view. The claim you're making seems to be "because this view is widely held, it cannot be racist."
Comments
I think it's a bit of a privileged position to say that migration isn't a major issue for Germany. There are, clearly, a lot of people who do think it's an important issue and that the mainstream parties haven't done enough to address it...
We quite reasonably think citizens voting for a party means they generally support its program in broad lines. Why an exception for today's far-right parties?
E.g. would you doubt Harris voters to be (broadly) liberal or NSDAP voters in 1933 to be fascist?