Sexual Orientation Research Phase 1: A Review of Methodological Approaches
3. Methodological issues
3.1 Why methodology matters
Appropriate and good quality methods are necessary if quantitative research is to be representative, reliable and valid, and if qualitative research is to have depth, capture diversity and be able to map associations. Without these, there is no way of asserting that research findings reflect the real needs of LGBT communities, are able to inform complex policy and funding decisions, and should be taken seriously by potential funders.(Webb and Wright, 2001; Bradford, 2001)
A consequence of poor or non-transparent methodology or data collection is the potential for bias in the results, which could lead to the damaging misrepresentation of LGBT communities or issues. For instance, the introduction on some general population surveys of a code to be used if the respondent 'spontaneously' responds to a standard marital status question that they are 'cohabiting in a same sex relationship,' has meant that some data on same sex cohabitation is likely to be a serious undercount of actual levels. Likewise, interpretation of administrative police statistics on reported male sexual assault rates needs to be informed by an awareness of the factors affecting reporting and the limitations of how the data is collected.
This Chapter explores a range of methodological issues, considering the range of current practice and suggesting preferred approaches. The aspects of methodology considered include approaches to sampling; qualitative and quantitative methods; non-response; analysis and reporting; community consultation and dissemination; and ethics.
Problems with obtaining an appropriate sample of respondents greatly disadvantage research with LGBT respondents in relation to research with other populations. In quantitative research there are few surveys that have employed probability sampling techniques and in qualitative research there is an absence of studies that have used sufficiently defined purposive or theoretical sampling procedures. In both sets of literature there is acknowledgement and extensive discussion of the problems involved in constructing representative or purposive samples of LGBT people. (e.g. GLEN and NEXUS, 1995; O'Connor and Molloy, 2001; Gonsiorek and Weinrich, 1991; Weston, 1991; Webb and Wright, 2001). This difficulty results primarily from the fact that, as a result of homophobic prejudice and discrimination, the LGBT community is largely concealed (as well as minority in numbers). The problems involved in selecting a sample are compounded further when a particular sub-sample, such as older or ethnic minority LGBT people, are sought.
The preferred approach to sampling will depend on the purpose, subject, methodology and resourcing of a project. Any attempt to measure the size of the LGBT community ideally requires a large, representative, randomly selected, general population sample; while an evaluation of the delivery of a particular service may be most efficient to recruit from users of, or people who have come into contact with, that service. Recruiting through the internet may make more sense for a web-based survey, recruiting in a venue for a face to face one, and using an organisation's mailing list for distributing a postal questionnaire. Combining several sampling approaches can help to maximise the diversity and number of people with a chance of being selected.
3.2.1 Sampling issues in quantitative research
The problems of definition and classification described in the previous Chapter impact enormously on attempts to delineate the appropriate population for a quantitative sexual orientation research project. Grossman and Kerner (1998) argue that it is impossible to obtain a 'representative' sample of gay male and lesbian youth because there can be no clear-cut definition of homosexuality. There are both theoretical and methodological issues impacting on the selection of an operational definition. Coxon (1993) describes how for a study of high-risk male same sex behaviour, the target groups "may well be 'all male-to-male sexual behaviour' (rather than 'homosexuals') or 'intravenous substance use activities' (rather than 'drug takers'), [but] it is unfeasible or grossly expensive to attempt to operationalise them."(Coxon, 1993) It is generally beyond the resources of conventional research funding agencies to carry out general population-wide screens to identify minority and socially invisible groups, though such an approach is necessary to measure prevalence and has been done (e.g. Snape, 1995 and Gadd, 2002).
Some studies report that, given the 'impossibility' of achieving a sample known to be representative of the LGBT community, the best that can be done is to simply achieve as large a sample as possible, drawn from as many sources as possible. While a larger sample size is not guaranteed to be any less biased than a smaller one, there is something to be said for this approach, and at least it allows for sub-group analysis to be carried out. Most of the surveys reviewed in summaries accompanying this report acknowledge that the sample size achieved was too small.
Research studies in the US have suggested that while 'representativeness' may be elusive to achieve, workable approximations may be possible which enhance the 'generalisability' of findings (e.g. Herek and Berrill, 1992). According to Gonsiorek and Weinrich (1991) to enhance representativeness "any homosexual sample should mimic the major demographic characteristics of the overall population. The other principle is that sampling should be diverse and the subjects should be drawn from as wide a variety of sources as possible. These sources and the demographic characteristics of the sample should then be described in considerable detail. While these procedures will not eliminate sampling problems, they should reduce them, and a clear and detailed description of procedures will make any sampling limitations apparent."(GLEN/NEXUS, 1995) However - to different degrees with different sampling approaches - the profile of achieved samples tend to be more representative of the gay community which is 'out' or 'on scene', than of the LGBT community as a whole.
Quantitative research is structured around dividing data into classifications in such a way as to be able to compare significant differences and similarities statistically. It seems strange therefore that there is a tendency in quantitative research into LGBT issues not to include a 'heterosexual' control or comparison group. The inclusion of such (where relevant and where alternative data sources are not already available) could enable disadvantages disproportionately experienced by LGBT people to be highlighted as well as similarities to be demonstrated.
3.2.2 Sampling issues in qualitative research
The ability to draw wider inference from qualitative research depends, to some extent, on the nature and quality of the sampling. The rationale in selecting those to be interviewed includes ensuring relevant diversity of coverage across certain key variables, rather than to select a sample that is statistically representative of the wider population. Purposive sampling of this kind provides the opportunity to identify a range of factors, influences and experiences underlying the research question. Theoretical sampling approaches include selecting those on the 'extremes' of a spectrum, rather than from across it.
Qualitative samples may be purposively selected to ensure sufficient diversity across variables such as gender, age, ethnicity and sexual orientation, but not all studies are transparent about their criteria, or even whether any selection criteria were used. There seems to be an assumption that because a qualitative sample does not need to be 'representative' of the population as a whole, that it also does not need to be systematic and deliberate.
3.2.3 Different sampling approaches
Random or probability samples
The purpose of quantitative research is to answer questions such as 'how many' and 'how often'. Any survey attempting to estimate the proportion and geographic spread of the general population that is LGBT, has need of LGBT services, or experienced harassment or bullying because they are perceived to be LGBT, should use a random sample where everyone in the population has a known and equal chance of selection.
For a survey attempting to estimate the proportion and geographic spread of LGBT people who have need of LGBT services or have experienced harassment or bullying, a random sample of LGBT people should ideally be selected. There are two main methods of selecting random samples of sub-groups: (i) selection from a comprehensive list of members of the relevant group or (ii) screening of the population at large. The former method is precluded because no comprehensive list of LGBT people exists (as membership lists of LGBT organisations are not comprehensive, and are likely to be biased towards particular types of people). The second random sample selection procedure, usually the screening of pre-selected addresses, involves drawing a very large general population sample and asking one or two brief questions (either on the 'doorstep' or as part of a survey) to establish who falls within the relevant group. This approach is also problematic because sensitivity about sexual orientation is likely to produce high levels of under-reporting of homosexuality.
The first National Survey of Sexual Attitudes and Lifestyles (Johnson et al., 1994) interviewed a randomly selected sample of 19,000 respondents living in Scotland, England and Wales. 14 It was able to provide reliable, accurate and valid estimates of the proportion of the population engaging in same sex activity or having ever cohabited with a same sex partner; and to describe this sub-group in terms of their socio-demographic characteristics, attitudes and behaviour. While the focus of that survey was not primarily on LGBT issues, it did yield a random sample of LGBT respondents that could be followed up for a subsequent study focusing specifically on LGBT issues.
Snape et al.'s 1995 survey followed up a random sample of 116 Natsal respondents who described themselves as 'homosexual' and 619 'heterosexuals'. While this unclustered sample is a great advance on most other approaches, it is still subject to non-response biases. These include the fact that the initial survey included only those living in private households 15; it achieved only a 66% response rate, there is the risk of under-reporting of homosexuality both on Natsal and the following up survey; and that not all Natsal respondents agreed to being recontacted for further research (10% refused). Only 19% of von Schulthess' "highly motivated" respondents to her survey on anti-lesbian harassment and violence agreed to be followed up.(1992)
In Scotland, Gadd et al. followed up victims of male domestic abuse (including those with male perpetrators) identified on the Scottish Crime Survey.(2002) As their study was qualitative the benefit of this sampling approach was not so much the random selection process, as the ability to identify a sub-group they would have otherwise been unlikely to locate.
The resources required for large-scale general population surveys are prohibitive. However, there is sometimes scope to 'piggy-back' a unit of questions on a large-scale random survey. Both GALOP (1997) and Mason and Palmer (1996) recommend a unit of questions on homophobic violence should be included on the British (or Scottish) Crime Survey, and ONS run an Omnibus survey of 2,000 respondents a month, the only regular omnibus in the UK which uses a random probability sample. Researchers working on the Omnibus survey are currently analysing data from an experimental pilot exploring the acceptability of asking about same-sex relationships. If such variables were incorporated onto the survey's standard classification, this would provide a rich and regular source of data on a wide range of issues disagreggated by sexual orientation, as well as demonstrate the plausibility of including such a question on other surveys.
Finally, in recent years not only has telephone coverage become more universal but also Random Digit Dialling (RDD) techniques have been improved, allowing for a random sample of telephone numbers to be generated for a telephone survey. This approach was used on the Urban Men's Health Survey (Stall, 2001). A future concern about this approach however is the increase in households that only have mobile telephones.
Quota or convenience samples
Quota sampling is the most appropriate approach for most qualitative research, as it allows for diversity across key pre-identified criteria to be achieved or specific types of individuals to be recruited. This approach is less useful for obtaining a representative sample of LGBT people for quantitative research however, given that the distribution of various quota criteria (e.g. gender, sexual orientation, social class, employment status) is unknown for the LGBT population as a whole. However quota sampling can be appropriate for affordable polls of general population attitudes where estimates do not need to be very accurate or have confidence intervals calculated. For instance, the MORI polls carried out to measure the attitudes of Scottish people to Section 28 used quota sampling, which meant that a large number of people could quickly be asked a small number of questions at a particular and significant moment in time.(Braunholtz, 2000) This kind of approach is likely to overestimate those respondents who are most easily accessible (e.g. for a telephone interview, those who go out little are most likely to be asked to take part).
On the qualitative Exploring Ethnicity and Sexual Health study (Elam et al., 1999) a quota sample was recruited by doorstep screening. This approach was used to avoid the bias that can result from recruiting through organisations. The more interlinking cells the quota specification has the more complicated the recruitment procedure becomes, and the final sample matrix is likely to be a compromise between the ideal breakdown and what is viable. Quota samples can be achieved through various means, including some of the approaches discussed below.
LGBT media, organisations and mailing lists
Whilst acknowledging that Snape et al.'s approach of following up a sub-sample of respondents from the National Survey of Sexual Attitudes and Lifestyles enabled a 'representative' and diverse sample to be drawn, Mason and Palmer also note that only a small sample was achieved in this way, and at great cost. Instead they argue that "the growth in the lesbian and gay media and groups has made it possible to contact large numbers of lesbians, gay men and bisexuals both using direct mail and anonymously using inserts in magazines."(Mason & Palmer, 1996) This approach is the one most frequently used in LGBT research using a postal questionnaire, as it is generally the most targeted, easily accessible and affordable means of generating a relatively large sample size.
The well-rehearsed concern about recruitment through LGBT organisations and media is that this approach tends to heavily over sample those who are most 'on the scene', and in particular those who are white, male, young, middle class, 'out', and most educated, literate, politically motivated and articulate. Gaining access through agencies, college classes, and advertisements, tends to weight a sample towards 'joiners', professional interviewees, the most educated, those with a particularly political perspective and people who perceive of themselves as central, as opposed to marginal, to the population surveyed.(GLEN/NEXUS, 1995)
LGBT venues, clubs and Pride events
While recruitment through LGBT media and mailing lists is often used for distribution of postal self-completion questionnaires, recruitment at LGBT venues, clubs and events may be more convenient for face to face interviewing (in practice the approaches are often combined). Many studies conducted by recruiting through pride style events, and in the US and Australia where ethnicity specific events are run this has proved invaluable for recruiting LGBT ethnic minorities.(Battle et al., 2002; Prestage et al., 2000) Recruitment in venues makes randomising the selection of respondents for surveys yet more problematic (unless a census is done) but can be better for including those requiring assistance due to literacy or comprehension difficulties. On the Gay Bars survey series (where respondents are recruited in pubs, clubs and bars for an AIDS awareness campaign evaluation survey) the data were weighted to adjust for the time of interview and how frequently the respondent visits the venue to try to ensure that those who are there least are not underrepresented in the data.(Samuels, 1997)
This approach, however, is subject to the same biases described above for recruitment through LGBT mailing lists and media: an over sampling of those who are white, male, young, middle class, 'out', and most educated, literate, politically motivated and articulate. One measure of the nature of any bias that might result from recruiting a sample exclusively from 'gay pubs, clubs and bars' can be established through secondary analysis of the data from the most recent Natsal survey.(Johnson et al., 2001) On Natsal, all respondents reporting having ever had any sexual experience with someone of the same sex was also asked how often they attend a gay pub, club or bar. Analysis of the data reveals that 22% of men and 39% of women who have had sex with a partner of the same gender in the last 5 years, have never been to a 'gay bar, pub or club.' 16 This only relates to sexual behaviour, not sexual identity, but if the variable of interest is sexual behaviour then recruitment exclusively through gay venues is clearly highly problematic, especially given that those whose attend gay venues are likely to be different from those who do not. A recent review of researching public sex venues suggests that this may pick up on some of those missing from venues.(Scott et al., 2001)
A health needs assessment of young gay, lesbian and bisexual people in Glasgow recently tried to overcome this bias by including 'non scene' venues and public places in their recruitment.(Coia, 2002) Other venues that different research projects have used for sampling include youth community centres.(Pilkington and D'Augelli, 1995).
A relatively new approach to recruitment is through the Internet, in particular by using 'banners' or 'pop-up boxes' on websites relating to LGBT groups or issues. This approach is primarily linked to carrying out web-based surveys (or surveys including a web-based component).
The 2001 National Gay Men's Sex Survey included (for the first time) web recruitment and completion of a short questionnaire on-line, alongside their usual methods of recruitment via Pride-style events and through mailing lists. The web version of the questionnaire, regarded as a pilot exercise to assess the viability of internet approaches to survey work, was available for completion on-line for 8 weeks, while the survey was promoted via gay.com, the largest gay-specific internet provider in the UK.(Reid et al., 2002) Given that 8,392 returns were received through the internet, this approach reached an enormous number of men relatively cheaply and anonymously in a short period of time. The authors express a note of caution however: out of this number 2,047 incoming responses were lost due to technical problems and a further 1,413 were from outside the eligible area. Given further development work, however, this approach may well prove successful in the future, particularly once Internet access is yet more widely available.
Because GMSS 2001 used three different recruitment approaches, it allows for comparison of the three sample profiles. A very interesting finding is that the sexual activity of their respondents varied with recruitment method. The web sample were less likely to have had sex with a man (94%) than the booklet (96%) or Pride samples (97%). Conversely they were more likely to have had sex with a woman in the last year (16%) than the other two recruitment sub-samples (7% and 4% respectively). (Reid et al., 2002) The authors conclude that the web sample was valuable for attracting younger men and behaviourally bisexual men to the survey. Although the web-recruited sample may have been a good complement to the other samples, it would be problematic if this was the only recruitment approach being used.
Where the purpose of a study is to evaluate experience of a particular service or intervention, the most appropriate approach to sampling will be to recruit from those who have had contact with the service or intervention (e.g. Williamson, 2001). If the purpose of the study is to assess the scope or reach of the service or evaluation, then clearly recruiting from those it has had contact with will produce biased results. Likewise, measuring sexual behaviour by obtaining a sample from those people whose visit STI clinics will overestimate risk behaviour for the population as a whole, as clinic attendees tend to constitute a higher risk group. (Fenton et al., 2001) As for every approach to obtaining a sample, it is crucial that the reporting of findings clearly indicates to what population the results apply. A recent study of gay bars and STI clinic attending men in London headlined that risk taking behaviour 'amongst gay men' was increasing; the study was not able to support this (Dodds & Mercey, 2000).
Savin-Williams (1994) has argued that recruiting gay, lesbian and bisexual respondents who are service users is likely to bias the sample towards those who are suffering, physically, socially and psychologically, the most. It could also be argued that it is biased towards those who are in receipt of help. Recruitment for a study of homelessness through those in contact with agencies is going to undersample those who are currently 'roofless'. O'Connor and Molloy recruited LGBT respondents through Housing Associations, which often lead to severe delays, although they found contacting a sample through service providers helpful regarding consent and ethical issues.
Most studies of transgender/transsexual people have recruited respondents from clinical records, leading to an emphasis on those who have either experienced severe psychological trauma or those who have decided to become operative. (While Wilson et al.'s recent study estimating the prevalence of transgender people was not clinic based, it also surveyed GPs and not transgender people).
Snowballing, pyramiding and networks
The 'snowball technique' or 'friendship-pyramiding' refers to asking a recruited respondent to suggest other people that they know who may also be eligible and agreeable to taking part. It is a useful approach for reaching people who may be mistrustful of being approached by someone they are not familiar with. Snowballing is often combined with other sampling methods, with some of the approaches described above being used to identify initial contacts. When the approach is cited in a report it is usually with the admission that it yielded fewer responses than anticipated (e.g. NBHA, 1999; O'Connor & Molloy, 2001). 17 Snowballing can be slow, unreliable, and requires highly motivated and involved first contacts who know many others.
Weston (1991) in San Francisco used personal connections developed over six years in an attempt to avoid organisational biases in her sample. However, despite efforts to select a varied sample "the sample remains weak in several areas, most notably the age range (which tends to cluster around the 20s and 30s), the inclusion of relatively few gay parents, and a bias toward fairly high levels of education. This partly reflects Weston's own personal situation as a well-educated lesbian with no children in her late twenties."(GLEN/NEXUS, 1995)
The primary concern with snowballing is that because people tend to know others who are similar to themselves, the additional sample generated is likely to be similar to the original respondent. This can be beneficial if the research design needs to identify respondents who have had particular and minority experiences (e.g. Deren et al.'s study of male intravenous drug users who have had sex with other men (2001)) or if the research aim is to map sexual networks, but it is inappropriate for any study that aims to estimate prevalence or explore diversity and difference. In a review of research into domestic violence in same-sex relationships, Burke and Follingstad (1999) argue that using personal networks to recruit participants may lead to inflated estimates of abuse because, for example, lesbians who have been abused might be likely to have friends who have also experienced abuse.
Cohort or panel studies
Cross-sectional survey research tends to be limited in how much data can be collected in the time available and in being able to look only at correlation, not causation. Cohort or panel studies, which involve returning to the same set of respondents over a period of time, allow for changes over time and causation to be explored. Fergusson's New Zealand Cohort study (1999) asked about sexual orientation at age 21, allowing for the experiences of LGB people to be compared with the rest of the sample in a variety of ways. The danger with this approach is that if being a part of the panel is a frequent and relatively demanding task, only the most motivated of respondents are likely to remain on it, which may bias the results.
3.3 Methodological approaches
The most appropriate research methodology will depend on a range of factors, including the sampling approach adopted; the nature of the research question; the subject area being investigated (and its sensitivity); the existence of baseline data; the nature of the available sample (e.g. how distributed or literate it is); and budgetary restraints. For questions of 'how many' and 'how often' a quantitative approach will be most appropriate, where a sufficient and suitable sample can be obtained. For questions of 'how' and 'why', qualitative approaches should be considered.
A survey uses a schedule of questions in a fixed order, which may make use of filtering to ensure that respondents are only asked the questions relevant to them. Most questions have pre-coded response options, and while there is some scope for including open questions which can generate useful verbatim quotes these should be kept to a minimum as they can be off-putting for respondents, time consuming to complete, difficult and expensive to codify for quantitative analysis and not always kept relevant to the question. The UK Gay and Lesbian Census was designed so that "many of the questions… give space for alternatives to the categories of response offered", but while this may feel more acceptable to the respondent, it is unlikely to have much impact on the actual analysis and reporting. In fact, 'other' answers are often simply 'back coded' into an existing category. Mason and Palmer included a fully open ended question in their postal self-completion-survey asking respondents to "tell us your story."(1996) This was completed by "several hundred people," but the authors do not give details of how this data was analysed. Inclusion of standardised sets of questions and measures on the questionnaire, for example quality of life or mental health scales, allows for comparison across studies, even internationally (e.g. Fergusson et al., 1999).
The wording and order of questions can exert an influence on the responses given. Just as project methods should undergo an initial pilot to test that they work as expected, new questions should be carefully pre-tested using cognitive techniques to ensure that they are unambiguous and are understood by respondents to mean what they are intended to mean. Kinsey suggested structuring questions around the assumption that an event had occurred, 'When did you last…?', to ask about stigmatised or sensitive behaviours. Coxon discusses 'how is sexual behaviour to be named?'(1993) "In SIGMA studies [vernacular] terminology has been elicited before questioning detail of sexual behaviour. The purpose of this is not only to gather information on 'street' terminology, but also to make the respondent more at ease in asking detailed information about what may be an embarrassing topic."(Coxon, 1993) Respondents are asked to give their 'preferred name' for a range of common sexual terms which were then substituted into the questions. Coxon argues that this "neatly combines the need to keep equivalence of meaning, ensuring compatibility and not unnecessarily embarrassing the respondent." Development work for Natsal found that general population respondents preferred precise and formal language to describe sexuality and sexual behaviour, and not casual or medicalised language.(Spencer et al., 1988)
The main methodological approaches to conducting a survey include questions being administered by an interviewer or being self-administered by the respondent. Mode of delivery includes face to face, over the telephone, or through the post. Computerised versions of all these modes are now widely used, including Computer Assisted Personal Interviewing (CAPI), Computer Assisted Self Interviewing (CASI) and Computer Assisted Telephone Interviewing (CATI), as well as web based studies. Audio versions of these computerised approaches are available to assist those with literacy or language problems, learning difficulties, or sight impairments (see Chapter 6 for how these approaches can be made more accessible to people with disabilities). Some computerised self-completion approaches are particularly appealing to younger respondents, although Natsal found that respondents were comfortable with using this technology irrespective of the level of their previous experience with computers. Face to face interviewing facilitates the use of tools such as sort and shuffle cards, vignette cards, visual stimuli (such as campaign or promotion materials for evaluation), and show cards (with 'concealed response' code letter lists which enable respondents to answer without having to verbalise their response out loud). Telephone interviewing is used widely in the US in sexual orientation research, however it excludes households without a landline telephone, including the growing number of younger people who live in households with only mobile phone cover.
In studies of sexual behaviour or other 'sensitive' issues it is necessary for data to be collected in an appropriate way in order to minimise underreporting. Both quantitative and qualitative interviews tend to be structured by starting on 'safe' and familiar territory, and building up to more sensitive topics as the rapport between interviewer and respondent develops. In face to face interviews combining interviewer and respondent administered components, it is best to give the self-completion module part way through or towards the end of the interview once trust and understanding has been established.(Johnson, 1994; Coxon, 1993)
In Snape et al.'s survey component of their research, standard questions were asked face to face of all respondents, and only in the self-completion booklet was sexual orientation established and relevant questions filtered on that basis. This means that interviewers never knew the sexual orientation of their respondents. Use of self-completion methods for sensitive issues is now standard practice. Paper booklets can be sealed in an envelope by the respondent before returning to the interviewer, with the only thing linking the booklets with the face to face questionnaire being a serial number. (Wellings et al., 1994; Snape et al., 1995) On Natsal II the self-completion was conducted directly onto the laptop, where by pressing certain keys as instructed the respondent 'locked' his/her answers into the computer so that they could not be accessed by the interviewer. A common approach in sexual orientation research (particularly in relation to sexual behaviour) in Scotland and the rest of Britain is self-completion paper questionnaires distributed face to face, which means that recruiters can explain the purpose of the study and assist with language (but should not advise on interpretation).
A disadvantage of self-completion methods is that they risk excluding those with poor literacy, whose first language is not English (unless the questionnaire is available in other languages) and who have visual impairments. Self-completions distributed by post are also subject to particularly poor response rates. Paper self-completions may have serious quality problems such as incorrectly followed filtering, and missing data for sensitive questions. If a question on sexual orientation is not answered, this could render the whole questionnaire useless. It is preferable therefore not to leave the most important variables to the end. Reid et al. noted that on the GMSS, which used three differently recruited samples, "men completing the web survey were most likely to be excluded for not completing enough of the questions to qualify (2.1% compared to 0.7% [from Pride events] and 0.5% [from postal booklet]). This occurred when men pressed 'submit' having completed less than 25% of the questionnaire content." (Reid et al. 2002) They assume this occurred because men deciding not to complete the survey at Pride or via the booklet would have just disposed of the paper version without returning it. Another issue for web or postal questionnaires is that it is difficult to establish whether the respondent is who they say they are or to control for the same person completing more than one questionnaire.
3.3.2 Qualitative approaches
Qualitative approaches are more exploratory and interactive in form than quantitative methods, and better for generating ideas and policy recommendations. There are a range of different qualitative approaches available. 75 hours of participant observation were used in a recent Sigma Research study; London's gay sex venues (backrooms and saunas): An HIV prevention research and development project.(Keogh, 1998) Although such an approach can produce very rich data it is problematic from the perspective of ethical requirements of informed consent.
Most current qualitative sexual orientation research employs either in-depth interviewing or focus groups, based on a broad topic guide to ensure some basic uniformity of coverage. In-depth interviewing uses no fixed questions and allows more scope for the interviewer to use and explore the language of the respondent. Topic guides allow for flexibility in structure and content, which facilitates exploration of individual circumstances and experiences in a way that is responsive to the accounts of individual respondents. Individual interviews tend to be most appropriate for sensitive or complicated issues, but interviews can also be carried out with pairs or triads, for instance a couple may be interviewed together. Interviewing two or three children of the same age together can be more fruitful in enabling them to open up. Other approaches include taking oral histories and use of key event techniques. Focus or discussion groups usually consist of between 6 and 10 people, sharing particular characteristics. Groups are good for allowing respondents to exchange and react to each other's attitudes and perspectives.
3.3.3 Other methodological approaches
There are an array of other research methodologies that may be appropriate for particular research issues, or may be combined with more standard survey and qualitative approaches to enrich the data. Coxon, and others, have made use of sexual diaries, which he found produced much more accurate data about sexual behaviour than could be obtained through retrospective probing. The approach, however, only succeeds with very motivated respondents as it requires a high level of commitment. (Coxon, 1993)
Coxon has also described the ethical, confidentiality, and procedural aspects of collecting and testing biological data as part of a sexual orientation research study. Project Sigma used interviewers (and the principle investigators) to collect blood samples from respondents to test for HIV, requiring training in taking blood and counselling. Blood is routinely collected by nurses on the Scottish and English Health Surveys, so a question on sexual orientation would make these data available for analysis by that variable. The second National Survey of Sexual Attitudes and Lifestyles (Johnson, 2001) used interviewers to collect urine samples to test for the sexually transmitted infection, chlamydia trachomatis (a guarantee was given to respondents that the samples would never be tested for HIV). Clearly a high response rate is achievable as 71% of eligible respondents agreed to provide a urine sample for testing. While the blood samples for Project Sigma could be tested anonymously, ethical consent requirements required treatment to be arranged for all Natsal respondents found to be positive. Collection of biological data as part of a quantitative survey allows for the relationship between social and biological variables to be analysed. This data enabled the Natsal researchers to discover that rates of chlamydia do not vary significantly between those with same-sex partners and those with opposite sex partners.
Campaign and promotion evaluations can be conducted in various ways, Keogh for example carried out 'content and product analysis' on 153 items of CHAPS health promotion literature.(2000) Legal, medical, psychiatric, and social work documents from professional interaction with a usually small number of 'subjects' can also provide data for research, though this approach may be subject to access restrictions. Martin (2000) looked at judicial judgements and Gadd et al. (2001) incorporated analysis of police statistics.
3.3.4 Combining methods and linking data
Given that qualitative and quantitative methods are best designed to answer different types of questions, the approaches can be combined for a richer and more rounded study. Many sexual orientation researchers have found that there is so little existing data currently available that they need to pursue an exploratory and multi-modal approach in order to establish a more reliable baseline picture (e.g. Hubbard and Rossington, 1995). Combining methodological approaches is often not done as much as it should be and not as well as it could be.
Gadd et al.'s (2002) study of domestic abuse against men in Scotland is an excellent example of a mixed method approach providing a more fully rounded research study. The combination of methods allowed flaws in one approach to be revealed and compensated for (e.g. qualitative follow-up work exposed survey respondents misinterpretation of a question).
Natsal data was greatly enriched by the combination of social survey and biological data. Project SIGMA used a set of inter-related studies to layer the data, including annual face-to-face interviews, HIV testing, and self-completed detailed sexual behaviour diaries completed on a daily basis.(Coxon, 1993)
Qualitative approaches have been used in the development stages of survey research, to generate subject areas to ask about and explore appropriate language and categories to use. The first Natsal survey was preceded by detailed qualitative work to explore issues, areas, concepts and language which informed the development of the survey.(Spencer et al., 1988) The second Natsal survey used qualitative work to assist with developing and testing the acceptability of new methodological approaches.(Mitchell et al., 1998)
There is also great scope for linking across survey datasets and administrative data where the same question has been asked or to identify where a population overlaps. Asking 'how often do you visit gay bars' on a national random survey (Natsal, 2001), a survey of gay bar attending gay men (Gay Bars, 1997), and on the UK Gay and Lesbian Census (2001), provides us with a measure of population coverage and context. The UK Gay and Lesbian Census also asks respondents what other recent gay and lesbian surveys they have participated in.
Very little research has been done on the impact of the interviewer in sexual orientation research.(Bancroft, 1997) Many studies make no reference to the processes that went into the selection, characteristics, supervision and training of interviewers.
Coxon (1993) and others have emphasised the need to allow respondents the clear right to refuse a given interviewer, either because they are known to them or on the basis of their sex. There has been discussion about the value of 'matching' interviewers and respondents by particular characteristics such as ethnicity, gender and sexual orientations. Ross used only gay male interviewers in his survey of MSM in the Grampian area.(2000) Spencer et al. found no preference regarding the sexual orientation of the interviewer from their gay male respondents, but found that some lesbian respondents preferred a lesbian interviewer (or at least not one 'anti-lesbian') (1988). Von Schulthess (1992), in her study of anti-lesbian assault and harassment notes that all the interviewing was carried out by women. Although she doesn't explore her reasons for this in the article, it can be assumed that the nature of the subject matter was a strong factor in the decision.
Other work has suggested that interviewers should remain 'neutral' and not disclose potentially 'hidden' characteristics such as their attitudes and sexual orientation to respondents in order not to influence responses. 18 The argument is that in qualitative work a respondent may assume that an interviewer already 'knows' what they are trying to explain if they are 'like' them, and so not verbalise their experiences so fully. Further qualitative work in 1998 found the manner of the interviewer to be the most important factor, "interviewers who were warm, friendly, yet detached were preferred by participants… Interviewer manner enhanced compliance, willingness to answer honestly and thoroughly, and reassurance of confidentially."(Mitchell, 1998) In addition the methodological work found that respondents feel less pressure to rush if the interviewer appears busy and occupied whilst the self-completion is being filled out.
Bradford extends the need for matching to the researchers working in the sexual orientation field, "although researchers of various sexual orientation and gender identities will contribute to this field, lesbian researchers have a unique perspective and an important role to play."(Bradford et al., 2001)
3.5 Analysis and reporting
3.5.1 Analysis of quantitative data
The main problems with analysis of sexual orientation survey data is that the sample size is often small and the sample selection is often non-random. Claims of statistical significance are reported, which are genuinely meaningful only for simple, random samples, and sub-groups are compared when the sample size cannot legitimately support it.
Great care should be taken with making comparisons across surveys and time series using different methodologies or differently defined sample populations (e.g. comparing men engaging in unprotected anal intercourse/UAI in the last month from one sample with self-identified 'gay men' from another). Reporting needs to clearly indicate to what population the results are generalisable to.
A long and careful process of data checking and cleaning should be undertaken before a dataset is used, this should include an edit check programme, consistency checking and a uniform response to how to respond to inconsistencies in the data.
3.5.2 Analysis of qualitative data
Qualitative groups and interviews should be tape recorded and verbatim transcripts produced. Various computer packages are available to assist with the thematic analysis of qualitative data, such as NU*DIST and Framework. The key topics and issues emerging from the data are identified through familiarisation with depth interview and group transcripts. A series of thematic charts are then drawn up and data from each summarised under each topic. Data from each stage of a multi-stage study can then be mapped within different - but linked - sets of thematic charts. These then form the basis for detailed exploration of the charted data, enabling examination of the range of views and experiences, comparing and contrasting individuals and groups and seeking explanations for similarities and differences within the data.
It is essential to recognise, however, that qualitative research samples are not designed to be statistically representative of the research population, and this means that statements about incidence or prevalence cannot be sustained. Similarly it is not possible to determine statistically discriminatory variables from qualitative data. O'Connor and Molloy note that where relationships are described in their study between, for example, circumstances and needs, "the purpose in doing so is to present explanations identified explicitly or implicitly by respondents and hypotheses for further research."(2001)
3.6.1 Confidentiality and anonymity
Confidentiality and anonymity of data are key issues in social research, especially when dealing with sensitive areas like sex and income. They become even more so when the behaviour may be stigmatised and/or illegal, or where the sexual orientation of a respondent is concealed.(Coxon, 1993) Since the publication of the Data Protection Legislation, this aspect of research has received more attention. Barry points out in her review of equality data sources in Ireland that the Census and other statistical and survey forms collect the name and address of the individual and household taking part. "While this information is for administrative rather than data protection purposes, to individuals and households it is frequently not evident that the data generated will not be linked back to those original forms. Security of data needs to be in place, but also to be seen to be in place."(2000) There is a need to reassure respondents of confidentiality in order to gain consent to take part, and have project procedures in place that guarantee that reassurance. Ross found that because "concerns were expressed around confidentiality issues by some members of Body positive Grampian" many declined to take part in the study.
In storage of data
These concerns are greater when the study is longitudinal, because actual identity is required for tracking purposes. In addition to being longitudinal, the Project Sigma study also had respondent's HIV status. Respondents were allowed to use a pseudonym, but were warned to remember it, as this would be the name used at the next re-contact stage. "Information linking identifiers and HIV status were kept on a separate machine, in a different location and fully accessible only by the Principle Investigator."(Coxon, 1993) Coxon discusses the temptation to disclose status to a positive respondent engaging in risky sexual practice or not taking up early intervention, and who is unaware of his own positive diagnosis.
Coxon (1993) discusses the problems of Section 25, Section 28 and 'Operation Spanner' to respondents' ability to report freely, honestly and without fear of retribution. "There has been widespread fear, then, that information about identity or behaviour could be accessed or obtained from Project files - and especially by the Police authorities. Was (and is) this a justified fear? Homosexual behaviour is illegal in Britain; which is not in private, or which involves more than two men, or where either partner is under 21 or with a member of the Armed Forces (or the Merchant Navy). On these criterion probably more than a quarter of the data of the Project could be construed as referring to illegal activity."(Coxon, 1993)
Despite the Project being funded by the MRC and the hope that this would mean the data had the status of 'medical records' there was serious risk of access or seizure, as the Project had had HM Customs and Excise seizing Project training videos and interview schedules being subject to confiscation by postal authorities. They went to the lengths of simulating raids on their offices as part of staff training, being able to destroy data at the touch of a button.
O'Connor and Molloy describe the need for recruitment of their sample of young homeless lesbians and gay men to be mediated through support organisations, with their making contact with, introducing the study to, obtaining consent and arranging contact with the research team for the young people. For ethical and confidentiality reasons this approach to sampling and recruitment was necessary, though it did mean that there was often a considerable delay before the interview could take place.
An aspect of confidentiality most relevant to the reporting of qualitative data is the treatment of verbatim passages from transcripts and case illustrations. To preserve the anonymity of respondents, specific details - such as names or places - which identify respondents may need to be omitted or changed. In many studies (e.g. O'Connor and Molloy (2001)) each respondent is given a fictitious name that is used consistently through out the report. Von Schulthess notes that the transcriptions of her taped interviews were "coded so that no names or identifying information appeared on the transcription."(1992)
3.6.2 Impact of evaluation on intervention
Some studies also hand out help lines, condoms, lubricant and information leaflets: "the steering group decided that it would be beneficial to have [condoms, lubricant and information] available, because taking part in the survey could raise issues or concerns which the respondent had not previously thought about."(Ross, 2000) The C apital Assets survey (Ndofor-Tah, 2000) was designed both to gather information from the communities and to provide information to them: "The HIV knowledge section of the interview consisted of giving participants statements and asking them if they thought they were true or false. In fact, all nineteen statements were true. This allowed the interviewer to state a number of facts about HIV, minimised confusion, and made addressing errors at the end of the interview easier. The interview process therefore also served as a simple educational intervention which could develop into a longer discussion if the participant wished to pursue it."
3.6.2 Informed consent
The Social Research Association and Market Research Society have ethical codes of conduct, although these have rarely, if ever, been enforced. There are different definitions of what constitutes 'informed consent', alongside informing respondents of who has funded a piece of research, what its purpose is and who will have access to the data, issues of what possible negative outcomes could results from their participation in the study in theory should be discussed up front. In practice this is problematic because what might cause a respondent 'upset', for example, can be unpredictable and is not always clear prior to an interview. Parental consent issues for interviewing young people are addressed in Chapter 6.
Appropriate and good quality methods are necessary if quantitative research is to be representative, reliable and valid, and if qualitative research is to have depth, represent diversity and be able to map associations. Without these, there is no way of asserting that research findings reflect the real needs of LGBT communities, are able to inform complex policy and funding decisions, and should be taken seriously by potential funders.
Problems with obtaining a representative and sufficiently large sample of LGBT respondents is the primary barrier to good quality sexual orientation research being carried out. Recruitment through LGBT venues, organisations and media tends to bias the sample towards the younger, well-educated, middle class, motivated and 'on scene' male respondents. Other sampling approaches are available, such as following up respondents from random sample general population surveys, though these may require greater resourcing.
Choice of methodology will depend largely on the purpose and subject area being researched and on the way in which the sample can be accessed. Where only a small sample can be obtained, qualitative approaches should be considered. Quantitative web based surveys are beginning to be used. While this may have benefits for reaching some LGBT people (for example in rural areas) careful methodological work is also required into who may be excluded from this approach.