Scottish household survey 2017: methodology and fieldwork outcomes

Methodology of the Scottish household survey 2017 and information on fieldwork targets and outcomes.


6 Survey Response

Summary

  • The final number of social survey interviews achieved was 10,683, matching the target, a response rate of 64 per cent.
  • The SHS response rate has fallen in recent years, having fallen by 4% since 1999. The reduction in fieldwork performance resulted in a response rate of 64% in 2017. This was the same as in the previous year and 3 percentage points lower than the 2014 survey response rate of 67%.
  • The long-term average response rate for 1999-2011 was 67.9%. However, it should be noted that the calculation has changed slightly for 2012 as a portion of the addresses of unknown eligibility are considered to be eligible whereas previously they would all have been classed as ineligible. This calculation change would have led to a lower response rate in years prior to 2012, if it had been calculated on the same basis.
  • There was a target of at least 80 completed physical surveys for each local authority along with a target of 3,004 surveys for Scotland.

6.1 Introduction

This section presents the fieldwork outcomes for the sampled addresses.

The final number of social survey interviews achieved was 10,683 a response rate of 64 per cent. This was the first year since 2012 design came in that the target number of interviews had been met. The impact of this drop in response rate is discussed in section 6.3. The performance of the physical survey is described in section 6.5.

The SHS response rate has fallen in recent years, having fallen by 4% since 1999. The reduction in fieldwork performance resulted in a response rate of 64% in 2017. This was the same as in the previous year and 3 percentage points lower than the 2014 survey response rate of 67%. The maintenance of the response rate in a climate of declining response rates on other population surveys is commendable.

Survey response is an important indicator of survey quality as non-response can introduce bias into survey estimates. Standardised outcome codes (based on an updated version of those published in Lynn et al (2001)[16]) for survey fieldwork were applied across the SHS, SHeS and SCJS. The outcome codes paper includes guidance on the appropriate categorisation of interview outcomes. This will allow consistent reporting of fieldwork performance and effective comparison between the performance of the surveys.

6.2 Scotland-Level Summary

Table 6.1 shows a detailed breakdown of the SHS response for all sampled addresses for Scotland. The addresses of unknown eligibility have been allocated as eligible and ineligible proportional to the levels of eligibility for the remainder of the sample. This approach provides a conservative estimate of the response rate as it estimates a high proportion of eligible cases amongst the unknown eligibility addresses.

The table shows that the overall household response rate was 63.9 per cent. This is below the long-term (1999/2000 to 2011) average response rate for the SHS of 67.9 per cent and below the 2012 to 2014 rates. The effects of the drop in response rate are discussed in section 6.3. It should be noted that from 2012 the calculation was changed slightly and a portion of the addresses of unknown eligibility are now considered to be eligible, whereas previously they would all have been classed as ineligible.

Table 6.1: Fieldwork outcomes, Scotland

Fieldwork Outcome Sample % of issued % of eligible
Responding households 10,683 59.3% 63.9%
Random adult interview complete 9,812 54.4% 58.7%
Refused
Office refusal 445 2.5% 2.7%
Refusal at introduction/before interview 3,592 19.9% 21.5%
Broken appointment - no re-contact 202 1.1% 1.2%
Total refused 4,239 23.5% 25.3%
Non-contact
No contact with anyone at the address 688 3.8% 4.1%
Contact made at address, but not with target respondent 416 2.3% 2.5%
Total non-contact 1,104 6.1% 6.6%
Other non-response
Ill at home during field period 48 0.3% 0.3%
Away or in hospital throughout field period 108 0.6% 0.6%
Physically or mentally unable/incompetent 153 0.8% 0.9%
Language barrier 53 0.3% 0.3%
Lost interview 4 0.0% 0.0%
Other non-response (not covered by categories above) 28 0.2% 0.2%
394 2.2% 2.4%
Unknown eligibility
Inaccessible 5 0.0%
Unable to locate address 26 0.1%
Unknown if occupied, due to non-contact 289 1.6%
Other unknown eligibility 0 0.0%
Total unknown eligibility 320 1.8%
Estimated eligible addresses in set of unknown eligibility addresses 304 1.7% 1.8%
Total eligible addresses 16,724 92.8% 100.0%
Not eligible
Not yet built / under construction 8 0.0%
Demolished/derelict 51 0.3%
Vacant/empty 754 4.2%
Non-residential 222 1.2%
Address occupied but not resident household 214 1.2%
Communal establishment / institution 22 0.1%
Other ineligible 11 0.1%
Estimated ineligible addresses in set of unknown eligibility addresses 16 0.1%
Total not eligible 1,298 7.2%
All issued addresses 18,022 100.0%

6.3 Drop in response rate

The reduction in fieldwork performance resulted in a response rate of 64% for 2017. This was the same as in previous year and 3 percentage points lower than the 2014 survey response rate of 67%. Response rates are shown in the Table 6.2: Response rates over time below.

Table 6.2: Response rates over time

2001/02 2003/04 2005/06 2007/08 2009/10 2011 2012 2013 2014 2015 2016 2017
Response rate 67% 68% 69% 66% 69% 69% 67% 67% 67% 64% 64% 64%
Achieved sample 30,639 30,822 31,013 27,238 28,404 14,358 10,644 10,652 10,633 10,325 10,470 10,683

The long-term average response rate for 1999-2011 was 67.9%. However, it should be noted that the calculation has changed slightly for 2012 as a portion of the addresses of unknown eligibility are considered to be eligible whereas previously they would all have been classed as ineligible. This calculation change would have led to a lower response rate in years prior to 2012, if it had been calculated on the same basis.

Due to a drop in the response rate in 2015, an analysis was conducted to investigate the likely impact of this drop in response rates. This was undertaken by modeling the 2014 sample results to examine what the impact of a 3% drop in response rates would have been had a lower number of interviews been achieved, and comparing this against the 67% full sample results.

Analysis showed that the demographic composition of the sample was largely unchanged (age, gender), with only the most deprived SIMD quintile and other urban areas showing a 1 percentage point drop in their respective shares of the total sample.

Analysis of mean differences in the population estimates from the two samples for a basket of full sample questions from the SHS, and full one third sample questions from the SHCS module, were also undertaken.

This showed that the absolute mean differences for the total population estimates across the different basket of questions within the household and random adults part of the survey, including the Scotland Performs National Indicators, were very small, at around 0.10 percentage points.

A few sub-group categories had one or two 'maximum' differences in estimates of around 1 percentage points, including age and social and private-rented sector sub-groups. However, these differences are unlikely to be statistically significant due to small subgroup sample sizes.

Sub-national analysis was not considered. It is expected that there would be a greater impact of this lower response rate for Local Authorities and other sub-national geographies.

6.4 Local authority performance

Table 6.3: Response rate and eligibility by local authority shows levels of ineligible addresses, response rate and random adult conversion.

Table 6.3: Response rate and eligibility by local authority

Table 6.3: Response rate and eligibility by local authority

As in previous years, the highest levels of ineligible addresses were recorded in Argyll and Bute, Dumfries and Galloway, Na h-Eileanan Siar, Highland and Orkney Islands[17]. For both Na h-Eileanan Siar and Argyll and Bute, high levels of ineligible addresses were expected as both areas contain a high number of holiday and second homes which are not eligible for the survey. Expected levels of ineligible addresses for Dumfries and Galloway, Highland and Orkney were all above 10%.

The two lowest household response rates in 2017 were in Glasgow City and East Renfrewshire, with Glasgow City having the lowest household response rate. In addition, 5 other local authorities had a response rate under 60 per cent in 2016.

The conversion from household interview to random adult completion was 92 per cent in 2017(this was 92% in 2016, 91% in 2015, 92% in 2014 and 93% both in 2012 and 2013).

6.5 Physical survey

As described in section 2.2, over one third of the SHS sample was assigned to the physical survey module. For completion of the physical survey, respondents had to agree to make an appointment for a surveyor to make a follow-up visit and to complete the appointment. Table 6.4 shows the number of households assigned to the physical module which responded to the main SHS and the conversion to completion of the physical survey.

There was a target of at least 80 completed physical surveys for each local authority along with a target of 3,004 surveys for Scotland. The result of continuing lower than estimated levels of conversion from household survey to physical survey, meant that there were fewer than 80 physical survey responses for 9 of the local authorities in 2017[18]. However, the decision to 'open up' an additional stream to the physical survey gave a lower shortfall on the minimum target of 80 per local authority than in previous years (16 local authorities in 2016, 17 in 2015, 24 in 2014, 21 in 2013 and 20 in 2012). The 3,006 surveys achieved for Scotland in 2017 met the target, the first time since prior to 2012.

Table 6.4: Conversion to full physical survey

Table 6.4: Conversion to full physical survey

Contact

Email: shs@gov.scot

Back to top