AlphaPlus helps awarding organisations identify and interpret the evidence which demonstrates that their qualifications and assessments are valid, and that candidates’ results are fair and reliable.
Awarding organisations and certification agencies increasingly need to be able to demonstrate that their qualifications are both valid and reliable. Our team has substantial expertise in measurement science, and of the processes and workflow employed to assure assessment quality and validity. We deploy this expertise with clients to provide them with high quality, robust measurement solutions that fit their requirements and which are practical and can be implemented. Furthermore, we are a commercial consultancy with an in-depth understanding of the commercial realities that affect UK and international assessment organisations. We give practical consideration to the commercial implications of our recommendations.
Our work has involved graded/compensatory models of assessment, minimum competence assessments and criterion-based assessments. We undertake statistical performance monitoring both in-programme and post-hoc for a number of clients. We also monitor marking, moderation, standardisation and grading/awarding processes as part of a number of evaluation projects for high-stakes qualifications. This work includes comparability over time and cross-cohort monitoring for fairness – one of the major challenges facing awarding organisations today. International benchmarking of educational standards and candidate performance (based on qualitative and/or quantitative methods) is also often of interest. We are able to look at curriculum standards, assessment design processes, question and examination performance, scoring and grading processes.
A selection of relevant projects is listed below:
Project: Validity of vocational qualifications
We undertook a major research project for the Regulator in partnership with City & Guilds, and NCFE (two of the leading UK vocational qualification awarding organisations) to gain a better understanding of the major issues associated with the validity of regulated assessments in England. It involved consideration of the validity of a number of vocational qualifications, and a meta-analysis of the evidence vocational qualification awarding organisations are able to provide as part of a validity argument for their qualifications.
Client: The National College for Teaching and Leadership (formerly The Teaching Agency)
Project: Several projects relating to the development and standard setting for the new skills tests for people wishing to become teachers
We received a number of commissions from The National College for Teaching and Leadership (formerly The Teaching Agency) to develop the qualified teacher status (QTS) skills test and undertake statistical and qualitative analysis of test performance. Balanced tests were created using IRT analysis; standard setting procedures included designing and carrying out Angoff and Bookmark activities.
Client: Department for Business, Innovation & Skills (BIS)
Project: Quantitative programme of research for adult English and maths
AlphaPlus has been appointed by BIS, as part of a consortium, to deliver a major research project on the acquisition and retention of adult basic skills. Using a Randomised Controlled Trial, the research aims to test different models of teaching and learning provision of adult basic English and maths and assess which is most effective for different types of learners. One of AlphaPlus’ roles in the project is to develop new psychometric assessment tools to measure skills levels. We are using IRT to map candidates taking assessments at neighbouring levels onto a single scale in order that progression can be measured before and after the intervention.
Client: UK Driving Standards Agency
Project: Driving theory tests
We were appointed by the Driving Standards Agency (DSA) to provide educational consultancy for the driving theory tests (around two million tests are taken each year). We generated assessment performance data for their item bank and then assembled parallel test forms, using both classical and latent trait theory approaches, to ensure tests of comparable coverage, difficulty and measurement error.
Project: A Level Comparative Analysis of Student Work
This work for the Regulator included the scrutiny of samples of candidate's work and accompanying documents, collected as part of the Ofqual Awarding Body Data Archive (ABDA) project. The research looked at evidence of any difference, or similarities, with respect to stretch and challenge between the 2008 and 2010 specifications and students performance in the examinations across a range of subjects.