Census forms about done; now to fill in gaps
"The data is processed as much as possible via automated processing.''
2018 census general manager Denise McGregor
Along with the emphasis on getting the 2018 census online, another concept that’s getting plenty of attention this time around is ‘‘imputation’’ – filling in the gaps.
By last Tuesday, more than 3.2 million people had completed the census online – up from the 2.98 million who had done so by midnight on census day, March 6. Statistics NZ was confident it would pass its target of having 70 per cent of forms done online.
By the end of last week, Stats had also scanned more than 300,000 individual paper forms. ‘‘Paper forms for dwellings and individual forms are scanned at a secure processing facility and added into our response store, which holds the completed online forms,’’ 2018 census general manager Denise McGregor said.
Field teams were now able to concentrate on following up on the remaining households and people who hadn’t taken part in the census yet.
Stats NZ’s systems could tell which households needed to be visited, at the same time as the forms were being processed, McGregor said. ‘‘Once the field visits are complete, we will concentrate purely on processing the data set.’’
Stats NZ’s report 2018 Census – a modernised, digital-first census – shows the visiting phase running through to April 16.
‘‘Running the processing systems at the same time as the field effort also allows the data to begin flowing through the system. The data coming through can be assessed for quality and changes made to the system to correct errors,’’ the report said.
The first release of data from the 2018 census is due in early October. That will include population and occupied dwelling counts for the country, and broken down into smaller areas.
There were three streams of processing work – automated, manual and automation, McGregor said.
‘‘The data is processed as much as possible via automated processing, but in some cases, data will need to be manually corrected so it can then be processed via the automated process.’’
If data could be processed easily, there was no need to look at it. It just became part of the big picture of census data.
‘‘We also have an evaluations team working with processed data to look at trends and patterns to assess that the processing system is working correctly, and that the quality of the data is suitable,’’ she said.
‘‘Once all the data has been evaluated, we can start to turn it into products that can be used by the public. This work will take a few months to complete.’’
Data needed to be processed and evaluated, and Stats NZ needed to confirm confidentiality was in place for all data before it could release it publicly, McGregor said.
‘‘One of the processes that we undertake is to evaluate the data, compare it to the 2013 census and assess whether the changes are attributable to real world change or errors in processing.’’
Stats NZ said it will carry out various procedures during the processing phase to ensure highquality data is produced.
Among the things Stats NZ will be doing during processing is ‘‘imputation’’ – ‘‘assigning a value where a response is missing, or unidentifiable’’.
‘‘Imputation is a key part of the processing phase, and we are going to do more of it in 2018 than in previous censuses.’’
Imputation would be used, where appropriate, to improve the quality of data, particularly to reduce non-response – where a census form has one or more unanswered questions, or where there’s no response from a person or a dwelling.
‘‘We will use imputation to ensure final census data reflects as closely as possible the population structure and its characteristics.
‘‘The 2018 census will benefit from an increasingly sophisticated statistical environment, particularly around the availability of administrative data,’’ Stats NZ said.
Administrative data is the information collected by government agencies and private organisations in the course of their operations. It is not collected primarily for statistical purposes.
Stats NZ said capability and understanding of the administrative data area had been strengthened, ‘‘meaning we can make more use of administrative and historical census data to impute a greater range of variables’’.
McGregor said imputation was a statistical process done to ensure the dataset was complete.
‘‘Incomplete data sets can carry bias, so we follow a methodology that imputes attributes that are as close a fit as possible. This methodology is automated – it is not conducted by people. An example of data that might be imputed is birth records or migration data.’’
A change from previous censuses is that people who are away from the place they live on census night will be assigned, or ‘‘repatriated’’ to their usual dwelling during processing.
In the past people were repatriated to their usual meshblock – the smallest geographic unit used in the census, with an ideal size range of 30-60 dwellings.