Tony Powell, commissioner of the National Capital Development Commission between 1974 and 1985, told the audience at a National Trust function on Sunday that the building lacked historic value, and parts of it should be torn down to improve access from Parliament House to Lake Burley Griffin.

Former Liberal prime minister Malcolm Fraser, who spent 28 years serving as an MP in the building, said Mr Powell was “wrong on every historical basis”.

“How can anyone be so foolish as to put a bulldozer through a building that has been very much a part of a large part of Australia’s history. The only thing that is diminished by Powell’s remarks is Powell himself,” Mr Fraser said.

“We don’t have so much history that we can afford to tear up our own past. I think that’s ridiculous."

Mr Fraser rejected Mr Powell’s claim that no events of great historical importance happened within Old Parliament House’s halls, from politicians steering Australia through the Great Depression in the building’s earliest years, to the declaration of war for World War II, and a great number of significant party and policy developments.

“I had a lot of fond memories in that Old Parliament House. There were tough times, there were fun times, there were interesting party meetings, there were party meetings that changed the course of party policy,” Mr Fraser said.

“What other building carries the importance in Canberra of the Old Parliament House? … None of them have contributed over such a period of time to Australia’s history as the Old Parliament House.”

Heritage specialist Amy Guthrie, who works at the Australian National University, said Old Parliament House represented one of the most important works of Australia’s first chief architect, John Smith Murdoch, and it was one of the most significant buildings in Canberra.

Ms Guthrie pointed towards the building’s listing on the National Heritage register, which in itself signified its importance.

“To be listed on the National Heritage List a place has to be of outstanding value to Australia, not just to Canberra. So the fact that it’s been listed on there for several years demonstrates that it is highly regarded by Australia as a whole,” she said.

She conceded the extensions to the building, which Mr Powell said should be removed, were not of the highest quality, but Ms Guthrie said to take them away would be “purist”.

“In some ways some extensions may not have been done the best, but really the extensions as a whole are part of the evolution of the building. They show the continued stories and the growth and use of the building by its different occupants, so they become part of the building’s history,” she said.

Mr Powell’s claims that, with the exception of Whitlam, the building saw no great historical events, were also contested by political science expert John Warhurst, who said Old Parliament House was home to more than half of Australia’s federal history.

The ANU emeritus professor said that during its time as the home of Australian politics, the building saw “the whole of the career of Sir Robert Menzies, Curtin as wartime prime minister, debates about Vietnam, Labor splits, [the] Aboriginal Tent Embassy” and more.

But, years earlier, Mr Powell may have found some support from an unlikely candidate – JS Murdoch, the building’s architect.

Described in the Australian Dictionary of Biography as a “dour Scot”, Mr Murdoch did not exhibit much pride in one of his largest Canberra projects.

The building was designed to be relatively inexpensive, with a view to possibly be demolished after 50 to 100 years, and Mr Murdoch himself referred to its “plainness” and described it as a “rush job”.

“The temperament of the average Australian required nothing in the way of ‘elaborate structures’,” Mr Murdoch said, according to an extract from a chapter on Canberra in the book Glorious Days: Australia 1913.

The building now houses the Museum of Australian Democracy, and its director, Daryl Karp, said she could not understand why anyone would want to get rid of such a fabulous building.

“I think that if you remove this part of our national story, you are losing something very significant in a very young nation. There aren’t that many living examples that take you through the 60-70 years of living history, that taps into the recollection of everyday Australians as well as triggering ideas for future generations,” she told ABC radio on Monday morning.

“I think it is the most extraordinary building that is a really significant icon for all of Australia and I can’t imagine that anyone could even suggest that something with such historical, social significance should be knocked down.”