Currently there is no direct way to remove followers from your channel yourself. The only way a follower can be removed is if Twitch removes a follower from your channel themselves.

Many followers and users are removed on a regular basis due to Twitch bots and each of those is reviewed on a case by case basis.