The Consistency of Question-order Bias in a Changing Political Context

Question-order bias is a well-known weakness of surveys commonly used in public administration research. However, most research on question-order bias uses question-order experiments that are relatively small, performed in one context, and rarely replicated. We carry out six question-order experiments in six large-scale Belgian surveys conducted during the COVID-19 pandemic. All experiments vary whether the respondents see questions regarding the effectiveness of pandemic governance or trust in different actors first. Results show that question-order effects are real and reasonably consistent across the high-powered replications, despite the changing political context of the pandemic. However, the direction of the effects largely changes when we flip the order of the trust outcome questions in the last three experiments, which sheds light on an underappreciated point: question-order bias also seems to exist within batteries of seemingly similar outcome questions.