Oaks Estate residents fear loss of heritage protection