The Australian Bureau of Statistics decided the risks associated with asking about sexuality in the next census was too high to include the question, even though two federal departments and state and local governments said they need the data.
Staff at the bureau acknowledged that decisions made about the LGBTI community are made using research from overseas, and the information would improve visibility of a section of the population that experiences marginalisation, stigma, discrimination and heightened mental health issues and homelessness.
"Currently many of the programs and initiatives developed by governments and organisations supporting the LGBTI community are informed by anecdotal evidence, research outcomes from overseas or research which is insufficient due to sampling, research or scope."
The information could be used to effectively evaluate the Fifth National Mental Health and Suicide Prevention Plan, Aged Care Diversity Framework and Ageing Plan, the briefing said.
Despite an identified need for a reliable evidence base for decision-making, bureaucrats said that some members of the population could take offence at being asked a question about their sexuality, which could lead to them not answering other questions.
"It is also possible that groups against the inclusion of a sexual orientation topic on the Census, may lobby the against the ABS, and affect participation in the Census. This tactic was seen during the 2016 Census in regard to plans to retain name and address information."
Testing of the question found while the majority of people responded well, some people were surprised about being asked the question, mostly older people who said they would be unlikely to answer the question.
"The risk of statistical impact is considered too high at this point to include this topic in the recommendations," the briefing said.
The information is included in documents tabled in the Senate on Tuesday following a request from Greens Senator Janet Rice, after it was revealed a set of test Census forms were pulped instead of being used in a trial, after a call from the office of Minister Michael Sukkar.
According to the tabled documents, outgoing chief statistician David Kalisch recommended the government include new questions about chronic health conditions and defence force service, but made no recommendation on questions about gender identity and sexual orientation.
"There is not such a strong case for these topics as the other new topics proposed, and collection could be problematic given the nature of issues and associated sensitives," Dr Kalisch wrote to Mr Sukkar in July.
"However, claims have been made about the need for information on gender identity and sexual orientation, that government is best placed to judge."
The documents also show the decision-making process for asking a question about gender identity, and the effort that had been made to ensure data would be captured about sex without confusing respondents and still getting the necessary data. This risk was set to be tested at the October trial, but the questions were removed before then.
At Senate estimates earlier this year it was revealed two sets of forms were printed for trials ahead of the 2021 census, but only the ones without questions on sexuality were used.