Home Affairs said 'small number' affected by lifetime visa ban. It's actually quite a few