Labor frontbencher Anthony Albanese has accused the coalition of peddling vicious rumours about Workplace Relations Minister Bill Shorten and his wife.

Mr Shorten and his wife Chloe, the daughter of Governor-General Quentin Bryce, spoke to News Ltd papers on Sunday about a slur about the couple doing the rounds of parliament house.

Mrs Shorten said the personal lives and families of politicians should be off limits.

"I understand why people are interested in the personal lives of people in public life ... I understand the curiosity, but I think the sorts of things people say can be appalling," said Mrs Shorten.

Prime Minister Julia Gillard said she did not have the time or the inclination to listen to the gossip around federal parliament.

"Like many other workplaces, there's gossip in parliament house. It can be hurtful for people and I think Mr Shorten and his wife are making that point today," she told reporters in Queensland on Sunday.

Senior Labor minister Anthony Albanese said he was concerned about the road parliament was currently going down and that the coalition and Opposition Leader Tony Abbott were behind rumours circulating at the moment.

"We've seen Bill and Chloe Shorten today have to go out and go to extraordinary lengths to bat back some of the extraordinary things that have been flowing around the parliament and on the web promoted by the conservatives," he said.

"It is indeed dangerous ground that the conservatives are going on."

Shadow Treasurer Joe Hockey condemned personal attacks against politicians and their families, saying he wouldn't be surprised if the rumours stemmed from Mr Shorten's union rivals.

"You hear about it (rumours) all the time. But, frankly, our families are not the ones in parliament," Mr Hockey said.

"They should not be subject to this sort of scuttlebutt."

Allegations involving former Labor MP Craig Thomson and sexual harassment claims against Speaker Peter Slipper reflected on all politicians and were painful for everyone to see, Mr Hockey said.

"It is a very honourable profession, politics.

"Whether you're Liberal, Labor, Green, whatever, there are a lot of honourable people in politics.

"Frankly, we should be talking about the Australian people and their aspirations and their challenges and not spending so much time on ourselves."

Even so, Mr Hockey also continued the coalition attack against Mr Thomson, who is at the centre of damning claims and findings by Fair Work Australia in relation to expenses incurred when he headed the Health Services Union.