'We have to fight this': Downer residents fear high-density takeover