Sayers died at his Melbourne home on Sunday night, aged 58, after battling pancreatic cancer since May 2014.

He became the inaugural director of the National Portrait Gallery from 1998 to 2010, after a stint as curator and assistant director of collections at the National Gallery of Australia from 1985 to 1998.

In July 2013 Sayers left his position as National Museum director, just three years into a five-year appointment, and moved to Melbourne to be with wife Perry and successfully return to his first love of painting.

Current portrait gallery director Angus Trumble said Sayers had defined the gallery and was one of very few art museum directors who had created an institution from scratch with founding patron Marilyn Darling.

"When he started he took the trouble to come and see me and spend the day to tell me the history of the institution and give advice as he has done ever since," he said.

"His death is twice is as painful because it's so premature."

"[The gallery] would look very different today had it been someone else's job to set up the institution out of nothing.

"There's a lot of Andrew's personality in the structures and ambitions of the gallery and in its flair, adventurousness and generosity."

Canberra Times art critic Sasha Grishin said the "mercurial, loveable and personable" Sayers was one of the most remarkable figures in the Australian art scene who left a major impact on Australian culture and transformed our visual landscape.

"Andrew was always a person in a hurry, but he seemed to find time for his friends and was devoted to his family," he said.

"His death is a major loss for Australian art and to many of us who knew and admired him."

Grishin said during the gallery's construction Sayers became a teetotaller and vegetarian to remain focused on the goal.

In 2010 he became a Member of the Order of Australia for services to arts administration.

The National Museum's current director Mathew Trinca said his predecessor's career was "distinguished by his ability to marry great scholarship, a precise curatorial eye and keen administrative skills".

"It is rare that a single individual makes such a difference to the life and work of so many of Australia's national cultural institutions," he said.

Former colleague and the NGA's assistant director of curatorial and educational services Simon Elliott said Sayers' artistic practice informed his different approach to curating.

"He was just as comfortable in the gallery setting as the museum setting and the fact that he did both is unusual in Australian art terms," he said.

Trumble said Sayers' successful return as an artist was a "third feather in his cap".

Earlier this year, Sayers was a finalist in the Archibald Prize for his portrait of environmental lawyer Tim Bonyhady and had exhibitions in Melbourne and Bermagui on the NSW south coast.

In 2014 he entered a nude self-portrait in the Doug Moran National Portrait Prize.

It is understood Sayers' last exhibition of paintings and works on paper at Canberra's Beaver Galleries will open on November 5 as planned.

Elliott said Sayers was a curator/director who loved ideas and objects and wanted to make the public enjoy and be surprised by museums and galleries.

NGA director Gerard Vaughan said Sayers made an enormous contribution to the arts and culture in Australia and would be missed by many.

British-born Sayers arrived in Sydney with his family in 1964, aged seven.

He was raised on Sydney's northern fringe and after university landed a job at the Art Gallery of NSW when he decided he'd rather curate than paint.

Elliott said Sayers became a "great Canberran" who loved the city's sky, and enjoyed bushwalking and running in the capital.

"He fought cancer very bravely and was someone who went well past the treatment regime because of his fitness and his will to live and contribute," he said.

"He was painting right up to the end."

Earlier this year Sayers said his cancer was "the background static of my life", but the diagnosis had made him "fearless" about showing his paintings.

Sayers' family will gather for a small private funeral.

Trumble said he expected the intuitions to come together for an event to remember Sayers and a permanent tribute was also possible.

- with Emma Kelly