Visiting Thailand's Prime Minister Yingluck Shinawatra and her Malaysian counterpart Najib Razak.

An uneasy alliiance ... Prime Minister Yingluck Shinawatra and her Malaysian counterpart Najib Razak shake hands. Photo: Reuters

MALAYSIA will help arrange peace talks with Muslim rebel groups in Thailand's south, signalling a possible breakthrough in a conflict that has claimed more than 5300 lives.

Despite deep reservations by Thailand's powerful military and other security agencies, the government in Bangkok has agreed that Malaysia can ''facilitate'' talks with representatives of insurgent groups in troubled Malay-speaking Thai provinces.

On Thursday Thailand's National Security Council chief, Paradorn Pattanathabutr, signed an agreement to start talks with representatives of one insurgent group, the Revolution National Front (BRN).

Visiting Thailand's Prime Minister Yingluck Shinawatra and her Malaysian counterpart Najib Razak shake hands outside Razak's office in Putrajaya, outside Kuala Lumpur, February 28, 2013. REUTERS/Bazuki Muhammad (MALAYSIA - Tags: POLITICS)

The deal is done ... Shinawatra and Razak outside Razak's office. Photo: Reuters

So far, Thailand has strongly resisted the idea of any foreign mediation of the conflict, insisting it was solely a domestic affair and that the ''unrest'' was caused by criminals.

Successive Thai governments and the military have held secret informal talks with some insurgent leaders but there have never been any formal talks.

Previous attempts to broker peace in the south have failed partly because the identities of the leaders of the insurgent groups were unclear.

Malaysia's new role will be modelled on negotiations Kuala Lumpur brokered last year under which the Philippines signed a peace agreement with the Moro Islamic Liberation Front, the country's largest Muslim rebel group.

Malaysia is believed to want Thailand to label its southern provinces ''autonomous'' or ''special administrative areas'' but Thailand has so far refused the request. Thai security officials have warned publicly in recent days against its government formalising peace talks with shadowy insurgent groups about which little is known.

But Lieutenant-General Pattanathabutr told journalists negotiations were needed to end the violence in the predominantly Muslim provinces where resistance to Buddhist rule has endured for decades.

''We need Malaysia's help because some insurgents are not based in Thailand, so Malaysia will facilitate by finding out who is involved and who is ready to talk,'' he said.

''The talks will let us know what they think and want so we can design solutions, but everything will be based on the [Thai] rule of law and the constitution.''

Malaysian Prime Minister Najib Razak discussed final details of the agreement with Thai Prime Minister Yingluck Shinawatra in Kuala Lumpur on Thursday. Within days, Mr Najib is expected to call national elections that could threaten the 55-year domination of his ruling coalition.

Mr Najib will be looking to reap political benefit from the announcement of Malaysia's role, which follows weeks of escalating violence in Thailand's south, including bombing attacks on civilians.

Two weeks ago, 16 insurgents were killed in an attack on a Thai marine base, a humiliating blow for the rebels. Rebels claim pro-government death squads are carrying out targeted killings and that Thai security forces act with impunity.

The conflict - mainly in the Thai provinces of Pattani, Yala and Narathiwat - has raged since 2004, but goes largely unnoticed outside Thailand.

Few tourists venture into the provinces bordering Malaysia, even though they are only a short flight from the Phuket and Koh Samui resorts.

Thailand deploys 60,000 security forces in the troubled provinces.