I always think of him as the best people manager we've had in 50 years.

He was a great character; every bit a larrikin. Just read his 1995 Garran oration, with such references as to a fight between public servants and ministerial advisers, "each side bad-mouthing the other as a bunch of Richard-heads".

Former Liberal minister Fred Chaney reported he was quite offended by Ayers when they first met and, as a result, initially opposed his appointment as secretary of the Department of Aboriginal Affairs. I gather Chaney thought Ayers was uncouth, disrespectful, overconfident and too blunt by half. But Chaney decided he shouldn't rush to judgment on the basis of one meeting, so arranged to have another. There, he told Ayers about his low opinion of him, which led to vigorous argument. At the end, Chaney decided: "He's just the bloke I need, really." Blunt, and as honest as the day is old.

Ayers began in the Victorian public service in the 1950s as a teacher. He was education officer in Pentridge Prison (where he met his wife Heather – outside, not inside). He joined the Commonwealth Public Service in 1967 as welfare director in the Department of the Interior (i.e. as the ACT's welfare director). In 1969, he moved to the Department of the Prime Minister as senior adviser on social welfare and was then promoted to division head in the Office of the Environment.

After a stint as chairman of Gough Whitlam's short-lived Children's Commission in 1974-75, Ayers returned to the Department of the Prime Minister and Cabinet as head of the welfare division in 1976, and was promoted to deputy secretary and undersecretary under the Fraser government.

His first appointment as secretary was in 1979 to Aboriginal Affairs, where Chaney and later Peter Baume were ministers. Ayers moved to the Department of Social Security in 1981, working with Chaney again and then Don Grimes and Brian Howe under the Hawke government. He moved to the Department of Community Services in 1987, after a brief secondment to help run the efficiency scrutiny unit with David Block. He headed the mega Department of Community Services and Health after the 1987 restructure, working with Neil Blewett.

Ayers joined the Defence Department in 1988 when Kim Beasley was minister and then worked very closely with Robert Ray and later Ian McLachlan under the Howard government. Among his junior defence ministers were John Faulkner, Gary Punch and Bronwyn Bishop (Ayers would have enjoyed Gourley's description of these three as "the good, the bad and the ugly"!).

He retired in 1998. McLachlan praised him at the time as a "consummate career administrator and adviser. Bluff good humour, determined professionalism and all-round judgment are the hallmarks of this longest-serving departmental head."

I was fortunate to work with Ayers in three departments: PM&C, Social Security and Defence. It was not all clear sailing; we were different personalities with different strengths and weaknesses. But he never held a grudge and I owe him a great deal.

For public servants, Ayers' nurturing of the public service stands as his greatest contribution. He was never head of PM&C, the Public Service Board or the Public Service Commission, yet all his peers called him "the doyen of the service" and he retained that standing until his retirement. Indeed, secretaries themselves turned to Ayers first for advice when they had difficulties, and he enjoyed being known as their "shop steward" when it came to arguments with government about pay, conditions and tenure.

His style was unique. He famously intervened when two secretaries were having a bunfight, inviting them to lunch with him. As they sat, he plonked a bottle of wine on the table, and told them to drink it and sort themselves out. Then he just walked off.

He was a great exponent of succession management. "Imagine the result," he would say, "if we were all responsible for developing two or three people each of whom was better at doing our jobs than we are." His firm view was that every secretary should do this, with an eye to having deputies who could also do other secretaries' jobs.

I left Ayers and Social Security in 1982. Eight years later, he rang me out of the blue: "Podge, I want to talk to you about your future career. Come and have lunch at my club." (His club was, of course, the Emperor's Court Chinese restaurant in Yarralumla; he would never, ever have joined the Commonwealth Club.) He told me if I stayed in the Finance Department I would eventually be promoted to deputy and forever cast as a Finance person. He went on to tell me that my supposed management prowess was based entirely on my knowing all my staff and every issue they and their line departments dealt with. "What you need to learn is how to manage when you cannot know all the people and you cannot know all the issues," he said. "Come and join me in Defence as deputy secretary acquisition and logistics so you can be better positioned for later." Acquisition and logistics had far more responsibility than the whole of the Defence Materiel Organisation today, and I knew nothing about it. I learned a hell of a lot about management in that job; in particular, the key is the appointment, development and oversight of people.

"Good people recruit good people; bums recruit bums" – a favourite Ayers line, often followed with: "Who did you last recruit and who last recruited you?"

Ayers was far more than a good recruiter. He knew how to get the most out of people. He delegated authority way down the chain. Not only did we deputies have a lot of room to get on with our jobs, we were expected to give similar room to our staff. People learn from their mistakes, and good people rise to the opportunities they are given.

I was an EL2 equivalent in PM&C with Ayers as my division head. I wrote all the briefing notes to prime minister Malcolm Fraser, and signed them, on social security and related policy issues. Noel Tanzer added his initials but I can't recall Ayers ever changing a minute I wrote. He would sometimes see or ring the prime minister to add to my advice, usually about the practical and political implications involved.

Of course, Ayers also insisted that those to whom he delegated authority got on with the job. Otherwise they would "get his size-12 Julius Marlowes up the clackers".

He also taught me that culture, trust and teamwork start at the top. He made the Defence diarchy work. His respect for Defence Force chiefs Peter Gration, Alan Beaumont and John Baker was clear to all of us. He was totally open and honest with them, and expected the same in our relations with the military.

Visitors to his office quickly learned of his disdain for any trappings. Admirals, generals and air marshalls would be offered a cup of tea or coffee on arrival, then invited to join Ayers in his kitchen out the back to make it themselves. At the end of the meeting, he would have them traipse back out to the kitchen to wash and dry the cups and saucers and put them away.

Financial pain was always to be shared for the common good. Everyone was pressured to "tap the mat" so that priority could be given to the sharp end – the capability of the Defence Force or, in other departments, on-the-ground community services.

Structures also mattered but more important was how to make them work. He streamlined Defence committees, clarifying which ones should be chaired by the military or civilians and ensuring they made the decisions that mattered and did not waste time: action had to be taken before any meeting to agree facts, settle minor differences and clarify the substantial issues for consideration.

Formal committees were complemented by two much less formal gatherings: Monday morning "prayers", where key issues for the week were identified, and Monday afternoon executive meetings over drinks, where we would canvass some senior staffing issues but mostly chew the fat, reinforcing camaraderie and shared commitment (and noting another loss by Collingwood and win by the Raiders).

Public servants may remember Ayers mainly for his contribution to the service, but ministers equally appreciated how he served them – and we learned from that as well. As Chaney found, Ayers was blunt. Other ministers also valued this highly. They discovered he had enormous respect for them and would deliver what they wanted (subject, of course, to the law) even if he had advised strongly against their wishes.

Let no one underestimate Ayers' contribution to the public. Whitlam chose him to head his Children's Commission to shift priority towards childcare as well as preschool education. Fraser got him to manage the famous prices and wages freeze, where he became known as "Dr No on the telephone".

Fraser also appointed him to head Aboriginal Affairs, where he showed how to work with Indigenous communities and the Indigenous leadership. How many other public servants had the courage and skill to defend and support Charlie Perkins, recognising his leadership and contribution to his people? His association with Indigenous leaders continued throughout his career and beyond.

In Social Security, Ayers made clear that the priority of all staff was to improve service delivery and outcomes for clients. He required all senior executives to work at the counter in a DSS office, invested in regional and state managers, and kept them informed with weekly, personally written circulars.

Bob Hawke appointed him to the efficiency scrutiny unit, which delivered significant savings and improvements to government administration.

In Defence, Ayers made a remarkable contribution to microeconomic reform, reducing civilian numbers by more than 30 per cent over and above the savings achieved from commercialising ADI and the aircraft factories. He also helped to reduce non-operational ADF staffing through the commercial support program, market-testing all support activities. All this allowed resources to be redirected to the sharp end, and was achieved without staff industrial action.

Finally, a few more "Ayersisms".

He was one helluva public servant.

Andrew Podger is a professor of public policy at the Australian National University, a former public service commissioner and a former departmental secretary. This is an edited version of his eulogy at Tony Ayers' funeral in Canberra last month. andrew.podger@anu.edu.au