New social model-based survey questions for disabled people

About the study

The Welsh Government is committed to the social model of disability, which views societal barriers as responsible for disabling people, rather than their impairments. To support this commitment, NatCen were commissioned to develop and test a set of survey questions that reflect the social model of disability.

In keeping with the Respondent Centred Design Framework, our research had separate ‘discovery’ and ‘alpha’ phases. The discovery phase, including deliberative workshops with disabled people, gathered insights to understand how current disability-related data is collected, identify gaps, and explore ways to improve the alignment of survey questions with the social model of disability.

An updated impairment question, based on the GSS harmonised impairment standard and new survey question on societal barriers that people with impairments experience underwent iterative testing during the alpha phase. Welsh and English versions of the survey questions underwent cognitive testing, with refined versions then included in a large-scale quantitative field test on the NatCen Panel. ‘Proof of concept’ cognitive testing of the survey questions in British Sign Language (BSL) and easy read was also carried out.

The survey questions should now be used in a wider range of surveys and data collection contexts. to build an evidence base on their use, the stability of estimates produced, and to inform whether further development work is needed and where it should be focused. Further testing of BSL and easy read versions of the questions is recommended.

Findings

Discovery phase

• Preference for the social model: Disabled people and data users support a shift from the medical model to the social model, emphasising the focus on societal barriers over individual impairments.
• Inclusive language: Participants highlighted the importance of using inclusive and non-stigmatising language in survey questions.
• Complex experiences: Existing survey questions are often too simplistic and fail to capture the nuanced and fluctuating nature of many impairments.
• Diverse data needs: Most data users require questions that can monitor trends over time, align with legislation like the Equality Act 2010, and provide detailed information for policymaking and service planning.
• Support for impairment questions: The GSS harmonised impairment question was preferred over the harmonised questions on long-lasting health conditions and activity restrictions but still requires refinement.
• Barriers-focused questions: There is strong support from data users and disabled people for developing new questions that focus on the societal barriers faced by disabled people, such as discrimination and accessibility issues.
• Practical considerations: Questions need to be clear, concise, and suitable for self-completion surveys to ensure high response rates and accurate data collection.
• Children and young people: Existing measures often overlook the experiences of disabled children and young people, indicating a need for more age-appropriate questions.
• Comparability versus inclusivity: Data comparability is particularly important to data users; however, many participants in the deliberative workshops prioritised inclusivity and the use of up-to-date, respectful language.
• Inconsistencies in definitions: The systematic review identified significant inconsistencies in how disability is defined and measured across different surveys, highlighting the need for standardisation.

Alpha phase

Social model impairment question 

Our final recommendations were that the social model-aligned impairment question developed and tested in NatCen Panel should now be taken forward and used in a wider range of surveys and data collection contexts.

Welsh Government were encouraged to review future findings to build an evidence base on use of the question, the stability of estimates it produces, and inform whether further development work is needed and where it should be focused.

Further research with deaf BSL signers was recommended to determine if, and how, the ‘Hearing’ response category of the social model impairment question might be improved. 

Experience of barriers survey questions 

A social model aligned approach to capturing people’s experience of barriers across life is novel in Wales, and the wider UK context. Two questions were developed and tested in this research, one measuring barriers experienced in different domains of life, and another measuring the different types of barriers experienced.  

Evidence indicated both questions had no obvious or major issues. As such, we recommended that both questions should be taken forward and used in a wider range of surveys and different data collection contexts.

Both questions returned similar prevalence estimates for disabled people (when used in combination with the new impairment question). Therefore, it was recommended that Welsh Government should consider how each may be applied, depending on the purpose and nature of the wider context in which they are used. 

Potential further testing could explore if changing the format of the questions reduces non-response, particularly for the question on barriers in different life domains. 

General recommendations

Parent version: Further testing of parent versions of the impairments and barriers questions should be undertaken prior to use.

Child version: This was beyond the revised scope of the research. Therefore, questions should be developed and cognitively tested with disabled children so data can be collected directly from older children, rather than parents.

BSL version: Testing was small in scope and further testing and consultation with deaf BSL signers should be carried out to improve data collection in a visual language. In particular, the impact of question structures and formats on the cognitive load for BSL signers, exploration of terminology issues, and the impact of how different identities, and social, cultural and language contexts among deaf BSL signers may shape responses.

Easy read version: Testing was small in scope, and the images and wording should be subject to further cognitive testing. In particular, making it explicit that the examples provided for each response option are not intended to be an exhaustive list; revising the ‘Something else’ response option to make it clearer that writing a comment is optional; and assessing whether all images are clear and help people to correctly understand the categories. 

Data collection mode: Field testing did not involve face-to-face data collection which should be tested further, alongside the impact of having a written reminder of the question and response options. The results of these tests should be published and guidance provided on what actions, if any, data producers should take to deal with any mode effects given.

Methodology

The discovery phase of the social model of disability research utilised a multi-faceted approach to gather comprehensive insights. The methodology comprised 4 main components.

• Firstly, a literature review was conducted to analyse academic and grey literature on various models of disability, including the medical, social, and biopsychosocial models. This review identified the strengths and criticisms of each model and their implications for survey design and data collection.
• Secondly, 3 online data-user workshops were held with Welsh Government employees, public sector workers, and third sector professionals. These workshops aimed to understand the current use of data about disabled people, limitations of existing measures, and the potential for new questions aligned with the social model. Discussions included practical data collection needs, legislative alignment, and the ability to monitor trends over time.
• Thirdly, a systematic review of existing surveys and questions related to disability was conducted. This phase aimed to evaluate the current instruments used for gathering data on disability, identifying gaps and areas for improvement to ensure closer alignment with the social model of disability.
• Finally, 2 online deliberative workshops were conducted with disabled people from across Wales. Participants shared their views on disability-related language, the appropriateness of existing survey questions, and the potential for new questions focused on societal barriers. Findings from these workshops will be central to developing new survey measures in the next phase of the research, which will include cognitive and field testing.

The alpha phase comprised

• Question design: A systematic iterative design process  to  draft, review, and refine a set of new survey questions, incorporating feedback from the Welsh Government and the steering group.
• Cognitive interviewing: This commonly used method enables researchers to assess whether survey questions are measuring what is intended, are clear and unambiguous, and whether participants are willing to answer them ¹ . Adult and parent versions of survey questions on impairments and barriers were tested. All participants recruited lived in Wales. Participants recruited had a range of impairments and health conditions. Questions were tested in Welsh, English, easy read Welsh, easy read English and British Sign Language (BSL). BSL cognitive testing was designed and carried out in collaboration with Social Research with Deaf people (SORD).
• Quantitative field test: Survey questions were refined after cognitive testing then included in the November wave of the NatCen Panel. A random sub-sample of 4,383 were selected, maintaining the probability-based design. The wave included a boost of 750 panel members living in Wales. Results for those living in Wales were compared with those living elsewhere in Britain to check for any significant differences in how the questions were answered. Fieldwork followed a sequential mixed-mode design. Participants were able to select whether to complete the survey in Welsh or English. Descriptive and bi-variate analysis were carried out. 

The study adhered to the principles of the social model by ensuring accessibility and inclusivity throughout the research process. Measures were taken to accommodate various language and access requirements, such as providing materials in multiple formats and offering support for participants with different needs.