The Sibby Ilzhofer-owned Dare Devil pulled out of the race at 5.05pm Monday afternoon, just four hours after the fleet of 88 yachts had started the race.

Freyja was already back at the dock by that stage, the 71-year-old boat having been forced to pull out with a shredded headsail.

A third entrant, Patrice, was also forced out of the race late on Monday with rudder damage.

Freyja co-owners Richard Lees and his son-in-law Andrew Carr opted to return to dock rather than raise the back-up headsail, and will sail Freyja back to Newcastle on Tuesday morning before turning their attention towards February's biennial Australian Wooden Boat Festival in Tasmania.

The withdrawal of Freyja and Dare Devil followed Manly yacht Enigma which didn't even make the start, pulling out with engine trouble early on race morning.

"We were just through the heads and we shredded the headsail," Lees said.

"We probably had too big a sail up. We were facing perfect conditions as we got out of the heads, we had two spinnakers, one on each mast.

"My son-in-law and I who are joint owners decided we'd come back rather than carry on with inadequate sails. We do have another headsail but it's not really suitable.

"I'm fairly fatalistic about it, some of the others are more disappointed than me. I haven't told the grandkids."

Meanwhile, French-built yacht Charlotte lodged an official protest against Triton shortly after the Sydney to Hobart start.