Brittany Higgins alleges she was sexually assaulted by a male colleague in a ministerial office in 2019.

Mr Morrison says his office only found out about the allegations last week and he was not aware until Monday.

Former prime minister Malcolm Turnbull said he found the timeline very hard to believe.

"I find it inconceivable that wasn't well known to at least key members of the prime minister's staff," he told the ABC on Wednesday.

"If it wasn't, there was clearly an absolutely baffling breakdown in communications."

The prime minister's account is at odds with Ms Higgins', who says at least two of his staff were involved in handling the complaint, including one senior aide who checked in with her several months ago.

Opposition Leader Anthony Albanese said he believed Ms Higgins.

"The idea that a sexual assault occurs 50 metres from the prime minister's office, in the office of the defence minister of our country two years ago, and no one in the prime minister's office knows?" he said.

Defence Minister Linda Reynolds, who employed Ms Higgins at the time, has known about the alleged rape for nearly two years but failed to tell the prime minister.

Mr Albanese said the minister would have been sacked if he was prime minister.

Mr Morrison has rebuked Senator Reynolds for failing to tell him about the allegations but denies his government has a "don't ask, don't tell" policy towards sexual assault.

Labor frontbencher Katy Gallagher told parliament the minister was "wilfully negligent" as an employer and needed to take responsibility.

The assault allegation has sparked calls for an independent complaints body to be established in Parliament House.

Crossbench MPs Helen Haines, Zali Steggall and Rebekha Sharkie say the complaints body needs to exist outside the Department of Finance, which handles such matters.

"I don't think sitting within Department of Finance with a minister still of the government of the day is really going to provide that level of confidence," Ms Sharkie said.

Mr Albanese has backed their calls.

"We need an arm's length, independent body that is able to investigate and provide support to anyone in this building who has an issue with their safety," he told reporters.

He said an internal government or Liberal Party review would not be enough.

The prime minister has established two inquiries and also agreed to Labor's call for an independent review into workplace culture inside Parliament House.

Mr Turnbull said driving cultural change would require strong leadership.

"There should be an absolutely rigorous, independent review," he said.

"The ministers and prime ministers have to lead by example. People who disrespect their colleagues and people who disrespect women, let alone assault them, have to be dealt with the full rigour of the law."

Australian Associated Press