While Donald Trump threatens North Korea with fire, fury and more, and as North Korea warns it has the US base on Guam in its sights, Japan and South Korea grapple with how to respond.

South Korea – all too aware that only a thin strip of demilitarised land and a deployment of about 30,000 US troops is the tripwire keeping the North at bay – has begun the deployment of the Terminal High Altitude Area Defense (THAAD) missile defence system.

For policymakers in Japan – a nation with a constitution that places severe limits on its ability to wage war, and which therefore has land, sea and air "self-defence" forces – the question of how to respond to the growing military threat posed by North Korea is even tougher.

Japan is a key ally of Australia. Its military alliance with the United States has effectively seen the island nation outsource much of its defence policy to a nation now led by the unpredictable Donald Trump.

But that alliance, and its close proximity to North Korea, make it a major target for the increasingly erratic regime of Kim Jong-un in the event of a shooting war. And, while North Korea's most recent ICBM tests have likely brought Australia into range of a strike of that handful of missiles, the hermit kingdom has almost 50 medium-range Nodong missile launchers that can reach the Japanese mainland, according to a Pentagon estimate. That has prompted debate in Japan about upgrading its own capabilities to shoot down those weapons.

China is now making a show of pressuring North Korea. But its increasing number of incursions into the East China Sea, its militarisation of the South China Sea and its ongoing support for the North Korean regime have caused alarm at the highest levels of Japan's defence and foreign affairs establishment.

From China's point of view, its rise is an inevitability and it rights an historic injustice. It views itself as a nation that had territory stolen by western colonialist powers, that now has the world's largest or second largest economy, and the largest population, and it is now re-taking its rightful place at the top international table.

Japan might not like the evolving international order, with China the pre-eminent nation in the Asia-Pacific region, but it may have to accept it.

But over a week in Tokyo and Okinawa, as Fairfax Media was granted rare and wide-ranging access to leading figures in the departments of Foreign Affairs and Defence and the national security establishment, as well as to a range of academics and think tanks, a uniformly pessimistic view of North Korea activities emerged, as did grave concerns about China's geo-strategic intentions.

Japan's political and military elite are grappling with how to calibrate their response to the existential threat posed by North Korea, just a short 1200 kilometres away across the Sea of Japan.

Much of the discussion was not for attribution so as to allow these policy makers within government ministries to speak freely or, as one official put it, to "give you more than the party line".

Yoichi Kato, a senior fellow at Tokyo think tank the Asia Pacific Initiative and also an adjunct fellow at Washington's Centre for Strategic and International Studies, is deeply enmeshed in both the Japanese and US national security establishments and a regular participant in so-called track 1.5 dialogues – but as he is not a government official, he can also say bluntly that at which officials only hint.

Japan, he says, faces two inconvenient truths.

"The first is thatballistic missile defence is not good enough, you cannot intercept and shoot down all the missiles North Korea may shoot. That's beyond BMD capability. The second is that US extended deterrence may break. We may come under nuclear attack from North Korea," he tells Fairfax Media.

"So because of those intricacies, which really jeopardise Japanese security, the argument in Japan right now is [whether] to have our own deterrence capabilities, or even our own offensive capabilities. We perhaps need to redefine the Japan-US relationship, which has traditionally been Japan being the shield and the US the spear."

His hawkish views on North Korean leader Kim Jong-un (whatever agreement he makes, he cheats), Trump (you can't trust him) and China (it wants North Korea and the South China Sea as strategic buffers) are blunt outliers in so far as how they are expressed, but not in so far as how they assess the current security environment for Japan.

He sums up the situation like this: "Japan's ultimate concern on Trump is that our national security interests will be traded away for US economic interests."

Therefore, he says, Japan has three options: building up its internal power; strengthening its network of alliances with countries like Australia, India and ASEAN nations such as Vietnam; or hugging China closer, as the Philippines appears to be doing.

Noriyuki Shikata, a senior diplomat in Japan's Ministry of Foreign Affairs, is one of the few government officials who agreed to speak on the record to Fairfax Media during a week of policy discussions.

He is far more careful in how he describes the geo-strategic challenges confronting Japan. Shikata returns again and again to two ideas: the need for a "free and open Indo-Pacific; and the importance of a rules-based international order respected by all players, including China and North Korea.

"We want to gain partners across the region, including the US, Australia, New Zealand, India and ASEAN countries. As far as the security landscape, we are extremely concerned about North Korean missile development.We think that it is extremely important to be united among like-minded countries to prevent North Korea pursuing its nuclear missile program," he says.

Asked what specifically Japan wants from Australia, Shikata demurs, and says only that his government was "very much encouraged by the message from Prime Minister Malcolm Turnbull in the context of the Shangri-La dialogue [and that] Foreign Minister Bishop has sent a strong message from Australia" on the need to respect the rules-based order.

In that Shangri-La dialogue speech, in June, Turnbull said the Asia-Pacific region had to "find new sources of leadership to help the United States shape our common good" – a message that was well received in Japan because of a perception that smaller nations in the region, confronted by the growth in the military strength of China and at the same time increasingly dependent on China for trade, appear reluctant to stand up to China over its militarisation of the South China Sea or its support for North Korea.

A case in point: an ASEAN meeting this week could only agree to a "negotiating framework for a code of conduct" in the South China Sea, despite Vietnam leading the charge for a stronger stand against Chinese expansionism.

Shikata says Japan has switched its diplomatic focus from cultivating individual countries in the region to a more regional strategy – the free and open Indo-Pacific – and that it wants to have "even closer discussions and dialogue with our allies and security partners".

Could that Indo-Pacific strategy be interpreted as a bulwark against China's "One Belt, One Road" policy, and its Asian Infrastructure Investment Bank? Shikata again demurs, and switches to highlighting the importance of the Trans-Pacific Partnership. Japan remains hopeful that a TPP11 (minus the US) will be signed off on, and that at some point in the future – presumably post-Trump – the US will sign up too.

Broadly speaking, Japanese security officials divide the world into "revisionist" states such as China and Russia – those that want to upend the current international order – and supporters of the status quo, such as Japan, the United States and Australia.

What they worry about, primarily, is North Korea's missiles and nuclear ambition; China's expansion into the South and East China seas; and Russia's desire to re-draw the map of Europe.

As one official puts it, Japan had hoped that "as they [China] became rich, so they would become good" but that hasn't happened.

Now, a sharply realpolitik consensus view is emerging. "We welcome China's rise, but if they try to take our territory, we will defend it," one official says.

Yuichi Hosoya, a politics professor at Keio University and some-time informal adviser and dinner companion to Japanese Prime Minster Shinzo Abe, says – like Kato – there is grave concern in Japan that Trump could agree to negotiate with North Korea over its nuclear program and, in so doing, agree to a "de-coupling" of the decades-old US-Japan defence alliance.

Trump's rhetoric on trade has spooked Japanese policy makers.

"One of the biggest nightmares is that the Unites States' current government will link trade issues with security issues, because of the large trade surplus with Japan. President Trump will possibly say that without reducing unbalanced trade the United States cannot defend Japan," Hosoya says.

The solution?

"In January, in Canberra, I said that Japanese, Australia and Indian co-operation will become more important to maintain the rules-based international order in the Asia-Pacific … because the United States might lose interest in playing its previous role, in maintaining the international order," Professor Hosoya says.

"So Japan, Australia, India must play a role in filling the vacuum created by the dysfunction of the White House."

While North Korea represents a clear and present danger for the Asia-Pacific region and for the rules-based international order, it is the rise of China – and the ascent of Trump to the presidency – that is the causing the most concern in Japan.

North Korea is just the most obvious manifestation of a changing international order.

But fairly or unfairly (no doubt China would say unfairly) Japan views China's rise as the greater medium-term challenge.

As Kato says, there is a near-consensus view on North Korea that the "best we can hope for" is to involve them in negotiations, buy time, and slow down development of nuclear weapons.

Can the issue be solved?

"That's the question no one can answer," he says, with a rueful grin.

James Massola travelled to Japan as a guest of the Japanese government.