The Invasive Species Council is pushing for deer to be classified as pests in the states - which are the only Australian jurisdictions where deer are legally protected as game animals for hunting purposes.

There are workarounds, so landowners can kill deer on their own properties and authorities can go on to public land and cull them.

However, not classifying deer as a pest creates a conflicting message about whether they can be culled, the council says, Theoretically, the animals can run wild on neighbouring properties without restriction.

The National Feral Deer Action Plan, being developed by a working group and is being backed by the federal government, estimates there are up to two million wild deer in Australia - up from around 50,000 in 1980.

In Victoria, there could be as many as one million, the Invasive Species Council's state deer project officer Peter Jacobs says.

"That protected status of deer is very much a relic of the past when there were small numbers of deer in Victoria that were basically released into the region right back to the mid-1800s for hunting," he told AAP.

"One landowner sees them as being a serious pest, and the next landowner thinks they're cute and likes the idea of them running around ... and they don't have to do anything about it."

If feral deer populations are left uncontrolled in good conditions, they could rise by up to 50 per cent a year, which means a herd of 30 deer could grow to 500 in a decade, according to the action plan.

Jenny O'Sullivan, a beef and sheep farmer in Victoria's South Gippsland region, spoke to 15 other farmers near Cape Liptrap as part of a Landcare weed and pest control program.

Every farmer was concerned about the "enormous increase" in deer damaging trees, prompting them to form a community action group to control the animals.

"We're really keen to respect the deer - we don't like killing them ... they're very cute. But we can't stand the damage that they're doing to the bush," Ms O'Sullivan said.

"There's always going to be deer for the hunters to hunt, but we can't continue to do what we're currently doing because it's not adequate."

She emphasised deer control should be done by experienced shooters.

Victoria has its own plan - the Victorian Deer Control Strategy - which sets out a clear plan to manage deer populations.

The animal's game status does not prevent their control when they are damaging the environment or property, a spokeswoman for the Victorian Department of Energy, Environment and Climate Action said.

In Tasmania, up to 100,000 deer are estimated to cover about one-quarter of the state, encroaching on the Tasmanian Wilderness World Heritage Area.

Projections suggest more than 500,000 deer will be roaming Tasmania by 2050.

Culling permits differ for landowners in Tasmania, Invasive Species Council conservation officer Tiana Pirtle said.

"There's still a lot of bureaucratic red tape for landowners to wade through," Dr Pirtle said.

"There's lots of landowners ...in the Midlands where deer are the biggest problem and also the most difficult to get permits to control, and they're just really fed up with all these restrictions and unwillingness to recognise that this is a problem."

The Tasmanian government says its Wild Fallow Deer Management Plan takes a balanced approach to managing the impact of deer on agriculture, along with conservation areas and forestry zone, while maintaining deer as a "traditional hunting resource".

The federal government continues to work with the states and territories to address the "significant threat" of feral deer, a Department of Climate Change, Energy, the Environment and Water spokeswoman said.

The government is pouring $224.5 million into the Saving Native Species Program and has given Tasmania $400,000 to put toward eradicating deer in the Walls of Jerusalem National Park.

Some $18 million has been funnelled directly into feral deer management programs since 2015, according to the federal Department of Agriculture, Fisheries and Forestry.

The National Feral Deer Action Plan is pushing for national authorities to create a containment zone and eradicate small deer populations beyond the zone.

A draft action plan is open for public comment until March 20, after which it will be considered by the federal agriculture department's Environment and Invasives and National Biosecurity committees.

Australian Associated Press