We have a new website go to gov.scot

The Scottish Executive Central Heating Programme: Assessing Impacts on Health

Listen

APPENDIX A FIELDWORK REPORT

A.1 Introduction

As outlined in Section 3 the evaluation involved three waves of fieldwork, namely:

1) an initial wave of face-to-face interviews in respondents' homes (referred to hereafter as 'Wave 1')

2) a postal survey one year after the initial interviews ('Wave 2')

3) a second wave of face-to-face fieldwork two years after the initial interviews ('Wave 3')

This Appendix details the conduct of the fieldwork for each of the three waves.

A.2 Wave 1 fieldwork - research design, procedures and achieved sample

A.2.1 Research design

The fieldwork for the first wave of the evaluation commenced in November 2002 and was completed in February 2004. The evaluation design (described in Section 3.2) was informed by a previous study into the feasibility of measuring the health impact of the Warm Deal initiative. Households where central heating was to be installed under the CHP formed the intervention group. A comparison group of matched respondents, who did not receive work under the Programme, was also interviewed.

The design for the initial wave of fieldwork involved a number of elements:

  • Tenants of public sector properties which were potentially eligible for work under the CHP were contacted by Registered Social Landlords ( RSLs), while householders in eligible private sector properties were invited to apply for inclusion in the Programme through EAGA.
  • During the initial survey of potentially eligible properties, recipients' permission was sought to allow their details to be passed to the evaluation contractors ( TNS). This approach was used in the Warm Deal study and led to a high response rate.
  • Recipients were approached to be interviewed. One respondent was chosen in each household. In households without children, the respondent was chosen randomly from the householders. In households with children, the householder who was best placed to answer questions on the children's health was selected.
  • Where a successful interview with a recipient was achieved, an interview was sought with a representative from a matched comparison household. The criteria for matching were tenure (owned, privately rented, rented from a Local Authority or Housing Association); household type (single adult, single parent, single pensioner, couple with no children, couple with children, pensioner couple); socio-economic group (AB, C1C2, DE); and locality (postcode sector).

In a small number of cases, interviewers reported considerable difficulty in finding a suitable matched comparison respondent. These interviews were either reissued to another interviewer, or alternatively one of the matching criteria was relaxed (for example, by allowing a single pensioner to be matched with a pensioner couple).

Due to difficulties in obtaining the required amount of sample, a number of changes to the fieldwork procedures were made during the course of the Wave 1 fieldwork period. Before describing the outcome of the Wave 1 fieldwork sweep, details of the receipt of sample and the resulting changes to the initial procedures are discussed.

A.2.2 Receipt of sample

Several difficulties were encountered in the receipt of sample. Initially, the larger RSLs and EAGA did not collect consent details and thus had no sample to provide, leading to a delay in the start of fieldwork by a year. Following this delay, there were continuing difficulties in obtaining the necessary amount of useable sample, primarily from RSLs. As a result, the amount of sample available was much lower than expected. Considerable effort was required both by the research team and by representatives of the Scottish Executive to increase the flow of sample from RSLs.

Additionally, the short timescales between consent being given by recipients to be interviewed (obtained at the initial survey of the respondent's home) and the installation of central heating, coupled with the delay in receiving consent forms from RSLs, led to a high proportion of addresses being deemed either unusable on receipt or being found to have had central heating installed when visited by an interviewer. Difficulty in allocating sample in sufficient time to achieve an interview was exacerbated by the concentration of installations in a small number of RSLs - around 84% of the RSL sample were received from three RSLs. Thus, the concentration of installations in particular areas, coupled with short lead-in times to installation, led to some difficulty in allocating all useable sample to interviewers.

A.2.3 Changes to fieldwork procedures during Wave 1

In response to the challenges described above, several changes were made to the procedures for the sample collection and fieldwork during the course of 2003. First, information was obtained from RSLs on those properties they considered would be eligible to receive central heating, rather than waiting to obtain consent at the initial survey stage. This resulted in an increase in the number of addresses received, and, in the main, increased the period between receipt of sample and the date of installation of central heating. It also, however, led to a significant decrease in the proportion of addresses where an interview was obtained, for the following reasons:

  • Some properties were vacant.
  • There was a higher refusal rate as tenants had not 'opted in' to the survey, as they had done previously.
  • A proportion of tenants stated that they did not want work carried out under the Central Heating Programme, and were thus ineligible for interview.
  • In a proportion of properties the central heating work had been carried out some time prior to the interviewer's visit.

Second, the allocation of recipient and comparison interviews was 'decoupled'. At the start of the fieldwork period, each interviewer was required to achieve a comparison interview for every successful recipient interview. In order to maximise the number of interviews with recipients before installation, recipient and comparison interviews were split. This allowed interviewing capacity to be concentrated on recipient sample and reduced the proportion of addresses where work had been carried out before the visit of an interviewer. Although the increase in the elapsed time between recipient and comparison interviews was not ideal, this was considered preferable to losing a significant proportion of sample.

Third, on occasion the length of time between receipt of sample and the completion of work remained short. It was decided to proceed with interviews where installation had taken place in the preceding fortnight. If installation had taken place more than two weeks earlier, interviewers were instructed not to interview.

Finally, in order to maximise the available interviewer capacity, the questionnaire was converted from a CAPI format ( i.e. direct input of respondents' data to computer) to a pen-and-paper version, permitting the allocation of sample to additional interviewers. Given the intermittent pattern of receiving sample from the three RSLs who provided the majority of the LA sample, and the considerable size of such batches of sample, this maximised the number of interviews that could be achieved before installation took place.

A.2.4 Interviews obtained in Wave 1

Table A.2.1 provides details of the numbers of interviews achieved against the targets set. A total of 1136 interviews was achieved with tenants of RSLs who were due to get work done under the initiative, and a total of 1121 comparison interviews was achieved. In total, 842 interviews were completed with householder ( EAGA) recipients, and 751 comparison interviews were achieved for this group. Overall, the number of interviews with recipients marginally exceeded the target set (101%), while the number of comparison interviews fell only slightly short of the target set (95%).

Table A.2.1 Interviews achieved during Wave 1 fieldwork

recipient interviews

comparison interviews

target

achieved

target

achieved

RSL

1249

1136 10 (90%)

1249

1121 (90%)

EAGA

714

842 (118%)

714

751 (105%)

Total

1963

1978 (101%)

1963

1872 (95%)

A.3 Wave 2 fieldwork - procedures and achieved sample

This section details the conduct of the fieldwork for the second wave, the postal survey. The objectives of the postal wave were twofold. First, it was designed as an exercise in panel management, with contact through the postal survey designed to reduce attrition between Wave 1 and Wave 3. Second, it was intended to capture data on changes to property characteristics and the health of the respondents, and on occurrences of particular life events which may affect health. The questionnaire used in Wave 2 was much shorter than that employed at Wave 1.

The sample frame comprised those who had provided an interview at Wave 1. A target response rate of at least 50% was set, based on previous postal surveys. With 3,850 interviews achieved at Wave 1, this represented a target of 1,925 interviews. Fieldwork commenced in November 2003, one year after the start of the Wave 1 fieldwork. An initial questionnaire was sent out to Wave 1 respondents a year following their initial interview. A reminder letter and a further copy of the questionnaire were sent two weeks after the initial mailout to those who had not returned a completed postal questionnaire. Fieldwork was completed in March 2005 (just over a year after the completion of Wave 1).

Table A.3.1 provides details of the achieved response rate at Wave 2. In total, 2131 Wave 2 questionnaires were returned, representing a response rate of 55%. The response rate differed between the recipient and comparison groups, and between respondents in public sector stock ( RSLs) and the private sector stock ( EAGA). While the response rate among recipients from LA/ HAs was 45%, the corresponding response rate among EAGA recipients was 76%.

Table A.3.1 Interviews achieved during Wave 2 fieldwork

recipient interviews

comparison interviews

Total

RSLs

EAGA

RSLs

EAGA

Completed Wave 2 questionnaire

515 (45%)

641 (76%)

467 (42%)

508 (68%)

2131 (55%)

No Wave 2 questionnaire

621 (55%)

201 (24%)

654 (58%)

243 (32%)

1719 (45%)

Total

1136

842

1121

751

3850

For the vast majority of non-respondents to the postal survey, no information on reasons for non-response was received. This is usual for postal surveys. For a small number of postal questionnaires (n=22), however, feedback was available. Twelve respondents asked to withdraw from the study, including one respondent who asked for her Wave 1 data to be deleted. Five Wave 1 respondents had died, an additional four were too ill to take any further part in the study and one had moved to sheltered accommodation. These 22 Wave 1 respondents were not re-contacted as part of the Wave 3 fieldwork.

A.4 Wave 3 fieldwork - procedures and achieved sample

A.4.1 Fieldwork procedures at Wave 3

In order to permit an analysis of the impact of receipt of central heating on health status, Wave 1 respondents were re-interviewed around two years after they had undertaken the initial interview (Wave 3). The sample comprised those who had undertaken an interview at Wave 1, regardless of whether they had completed a postal questionnaire (bar the 22 who were actively withdrawn during Wave 2).

Fieldwork for Wave 3 commenced in December 2004 and was completed in March 2006. To ensure that the time lag between the initial interview and the re-interview was as near to two years as possible, the sample was divided into five-week periods, with interviewer schedules set around the midpoint of these periods. It should be noted that some sample with unsuccessful outcomes was reissued to increase the total number of achieved interviews. This invariably led to a gap of more than two years between the Wave 1 and Wave 3 interviews. Reissues are discussed in greater depth below.

The Wave 3 questionnaire was almost identical to the Wave 1 questionnaire. The only revisions to the questionnaire were the addition of questions relating to work conducted under the programme and minor changes to a small number of questions with pre-defined answer categories replacing "open" responses.

The fieldwork procedures were designed to maximise the number of interviews obtained. All Wave 1 respondents were visited on at least four occasions, on different days and at different times of the day to try to obtain an interview. This is reflected in the final fieldwork outcomes: only 7% of the total sample resulted in no contact after four or more visits. Indeed, among this sample, 26% were visited on six or more occasions. A further effort to maximise the response at Wave 3 was made by reissuing sample with an unsuccessful outcome. In total, 294 addresses were reissued to interviewers. The vast majority of this sample had been away during the initially specified fieldwork batch period, were 'soft' refusals, or had received four or more attempted contacts. This resulted in an additional 126 interviews being achieved.

A.4.2 Fieldwork outcomes at Wave 3

In total, 2,365 interviews were achieved at Wave 3. This represents 62% of the total sample available (all interviews achieved at Wave 1 minus those who actively withdrew at Wave 2). A summary of outcomes for Wave 3 is given in Table A.4.1, with a more detailed presentation - including a breakdown of responses by source ( i.e.RSL vs. EAGA) - in Table A.4.2. The proportions of interviews achieved differed between the recipient and comparison groups, and between respondents in public sector stock ( RSLs) and the private sector stock ( EAGA). In total, interviews were achieved with 59% of Wave 1 public sector stock recipients (n=669), 73% of private sector stock recipients (n=612), 52% of public sector comparison households (n=579), and 68% of private sector comparison households (n=505).

The total number of interviews achieved was less than the target of 70% of Wave 1 interviews. The most common unsuccessful fieldwork outcomes were sample members moving without trace (10%), refusal (9%), non-contact (7%), death (4%), hospitalisation (3%) and derelict/demolished/empty properties (2%). Survey response rates are generally calculated as the number of achieved interviews as a proportion of total sample minus ineligible sample ('deadwood'). Excluding deadwood, the overall achieved response rate among the full sample was 76%.

Table A.4.1 Fieldwork outcomes at wave 3 (summary)

all

recipients

comparison

Summary of sample

Total sample available

3828

1968

1860

Total sample allocated

3828

100.0%

1968

100%

1860

100%

Fieldwork outcomes

Successful

2365

62%

1281

65%

1084

58%

Unsuccessful

No reply after 4 or more visits

284

7%

131

7%

153

8%

Refused

358

9%

177

9%

181

10%

Empty/Demolished/Derelict

79

2%

26

1%

53

3%

Deceased

152

4%

88

4%

64

3%

In hospital/too ill

102

3%

49

2%

53

3%

Moved

365

10%

166

8%

199

11%

Other unsuccessful

123

3%

50

3%

73

4%

Total Unsuccessful

1463

38%

687

35%

776

42%

Total

3828

100%

1968

100%

1860

100%

Table A.4.2 Fieldwork outcomes at wave 3 (full version)

total

All recipient

All comparison

RSL recipients

EAGA recipients

RSL comparison

EAGA comparison

N

%

N

%

N

%

N

%

N

%

N

%

N

%

Total sample available

3828

1968

1860

1133

835

1115

745

Fieldwork outcomes

Successful

2365

62%

1281

65%

1084

58%

669

59%

612

73%

579

52%

505

68%

Unsuccessful

No reply after 4 or more visits

284

7%

131

7%

153

8%

96

8%

35

4%

124

11%

29

4%

Refused

358

9%

177

9%

181

10%

120

11%

57

7%

93

8%

88

12%

Empty/Demolished/Derelict

79

2%

26

1%

53

3%

17

2%

9

1%

47

4%

6

1%

Deceased

152

4%

88

4%

64

3%

46

4%

42

5%

40

4%

24

3%

In hospital/too ill

102

3%

49

2%

53

3%

24

2%

25

3%

30

3%

23

3%

Moved

365

10%

166

8%

199

11%

131

12%

35

4%

159

14%

40

5%

Other unsuccessful

123

3%

50

3%

73

4%

30

3%

20

2%

43

4%

30

4%

Total unsuccessful

1463

38%

687

35%

776

42%

464

41%

223

27%

536

48%

240

32%

Total

3828

100%

1968

100%

1860

100%

1133

100%

835

100%

1115

100%

745

100%