Downer won't be swamped with high-density housing, minister says