A rattling cattle grid sounded our entry into Upper Diabaig; home to a reported population of one. The road then descended into the subsequently named Lower Diabaig, winding past a scattering of whitewashed cottages. A few cars were parked at the end of the pier, while the road trailed off beside more white cottages.

This hamlet, 10 miles from Torridon in the Northwest Highlands, was the end of the road.

We had detoured to Lower Diabaig from the North Coast 500 at the word of a friend. It followed three and a half days behind the wheel along the 800-kilometre (500 miles, hence the name) driving route.

The North Coast 500 was intended to promote the North Highlands and entice people to one of the more remote and less visited parts of Scotland. Officially the route starts at Inverness Castle, travels the coast to Wick and Britain's northernmost mainland settlement, John O'Groats, across the top to Thurso and Durness, and down the western coast through Lochinver, Ullapool, Gairloch, Torridon and Applecross and back to Inverness.

In a nutshell the journey serves up castles, mythical creatures, sparkling turquoise coastlines and misty rugged mountains. A lot of the route is single-track roads, so you're either pulling over when a car comes along, or they pull over for you.

Our stops had been frequent, but fleeting. There was so much to see that we always felt dragged back to the road. But there, in a remote fishing village protected from the rest of the world by mountains and water, we stopped.

I'd expected Lower Diabaig to be another brief break; we'd walk around, snap a few photos, and then jump back in the car to see what was around the next bend, over the next mountain. There didn't seem much to keep us there any longer.

The row of weathered buildings at the end of the pier looked abandoned. A few buoys and cages were scattered about, but there was no sign of the fishing boats that apparently pull in there. There were a couple of small boats anchored in the bay, and a third was being manoeuvred out of the water onto the back to a trailer. The restaurant, however, was unexpected.

Dutchman Aart and his Scottish partner Amanda opened Gille Brighde in 2012, after spotting the old schoolhouse on a Highlands Council website listing unwanted buildings. A dog called Star slept under a table. Two others sat under another, while their owners had some beers. On a wet and windy Sunday afternoon there was only one table free.

The menu was extensive despite the obstacles that must come with running a business in a place like this. By the time we'd poured our second cup of tea, we'd decided to return for a roast dinner a couple of hours later.

Enigmatic and charming, Lower Diabaig seemed like a place full of stories - except there weren't many clues. We poked around the remains of a 68-foot trawler exposed on the rocky shore. There's no information about the wreck at the site, but, from what I later read, the Dayspring broke its moorings during a hurricane in 2005 and ended up in Lower Diabaig. Now it's an eerie prop for photographers.

Aart suggested we spend the night in a bothy a couple of miles away in the hills. We'd stayed ahead of the bad weather for the first few days of the journey, but it had caught us, so we ruled out a night in a remote hut, but strolled along the start of the track. These days, there is only one road in and out, but tracks like this across the mountains once connected Lower Diabaig to other settlements.

Perhaps it didn't always feel this isolated.

Over dinner, Aart told us he reckons about 25 to 30 people call Lower Diabaig home. Many of the houses are second homes; some operated as holiday rentals.

He reminds us Inverness is only two hours away. While we sat in that old schoolhouse, tucking into a dessert of apple and ginger cake, it felt much further.