We are all interested in what other people think, and why they do what they do. It matters especially to policymakers, politicians and commercial organisations, and as computers become more powerful, we have seen a huge expansion of research into opinions and attitudes. But, until recently, computerisation has had some clear limits. Artificial intelligence is now breaking some of those boundaries.
What and why?
There are two broad kinds of question in social research: what and why? The ‘what’ questions can be answered with quantitative tools like voting intention polls. Computing has made it relatively easy and cheap to collect that sort of data: you ask: ‘what party would you vote for in an election today?’, count the numbers, and hope they are telling the truth. You can understand a bit more by looking at the profile of respondents, for example: do older people give different answers from young ones?
But to understand why people think what they do, requires a different approach, a more open ended conversation which can probe ‘why do you think that?’, ‘would you change your mind if..?’. For this, the traditional tools are focus groups and one-to-one interviews. But these are expensive and time-consuming. So, they usually use very small numbers of people, which makes it difficult to generalise to whole populations. And, until now, computers have made little impact on this sort of research.
Neither traditional technique provides ideal results. Quantitative results are simplistic, and qualitative ones can’t tell us about most people. However, recent developments in artificial intelligence claim to overcome both problems. They make it possible to do something like a qualitative interview with much larger numbers of people, and to analyse much larger volumes of data.
AI arrives
The shift probably began in the world of personnel recruitment. An employer aiming to recruit a dozen new graduates may be faced with several hundred, fairly well qualified applicants, all with similar CVs. A number of artificial intelligence systems are now available which enable employers to ask more complex question, and analyse the typed answers more objectively. The process is simpler, much faster and more convenient for applicant and employer.
The next, and latest, step is to extend these techniques into broader social research. Now AIs are beginning to replicate the conversation which a good qualitative interviewer might have. Like that interviewer, they move beyond standardised questions to an interactive dialogue, probing motivation, alternatives and priorities.
This development is built on generative large language AI models like ChatGPT and Google’s Gemini. Having digested vast quantities of data, these are able to provide human-like responses to queries on almost any topic. They can summarise complex documents, answer questions and write reports drawing on multiple sources. They can write computer programmes for particular purposes, and rewrite texts in a different style or length. Importantly, they can engage in an online ‘conversation’ allowing the user to refine and expand a question. They can also, as recent experiments have shown, design and administer surveys. Anyone with a keyboard can talk to these systems, and get an intelligent and usually accurate response.
It is this ‘conversation’ which is now being trialled in social research. A (human) researcher determines whose views s/he is interested in, and what topics and broad questions are to be explored, but the computer designs the system, and carries out the interview, very much as a human would do. Like a human, it remembers previous answers, and modifies its questioning in response. It will move on to the next topic when one has been exhausted. It can steer away from issues which the respondent finds uncomfortable, and can probe when answers seem inconsistent. It can record the whole process with no need for transcription, and can analyse the responses with minimal effort from the researcher.
So far there have been only a few experiments with this. In the UK, Focaldata is now offering this as part of their menu of survey techniques, and it has been used to examine political attitudes in the UK, recently exploring the rise of the Reform Party.
Testing AI
To test this approach, a team of Scandinavian researchers carried out an experiment. They wanted to see whether an AI designed and delivered process could produce valid results and be acceptable to respondents. They chose a study of a problem which interests social scientists and economists: why do some Americans not invest savings in the stock market?
Having identified a group of people they wanted to interview, the researchers defined the issues they wanted to explore, in fairly general terms, and asked the AI to design a process to do this. The AI then contacted the respondents to invite them to take part. On the screen it explained the process, and carried out an online typed ‘interview’. Although the broad topics were defined, the AI evolved specific questions in response to the dialogue, exploring inconsistent answers and unclear areas. At the end, the AI checked its conclusions with the interviewee and thanked them. It produced a summary of each interview. It then searched all the records to identify common themes; measured how many people raised each theme; and analysed the relationships between them.
In only two weeks, nearly 400 people took part (a very large sample for a qualitative study), and almost all completed the interview, in around 30 minutes. Afterwards, the AI asked respondents about their experience. Four out of five gave it a positive score, and three quarters said that the conversation felt at least ‘somewhat natural’. To test its reliability, the interviewees were also asked to complete a conventional quantitative survey, which produced very similar answers.
The researchers concluded that the approach worked at least as well as conventional face to face interviewing. As expected, the follow up questions created by the AI provided a much richer explanation of the real attitudes of respondents than the initial ones. Respondents were happy to take part and the results were at least as good as traditional qualitative and quantitative research. The process was simpler, cheaper, and much faster than the traditional qualitative approach.
Giving AI control: are there risks?
The potential of this technology is stunning. It bridges the traditional gulf between qualitative and quantitative research. It can carry out much larger numbers of qualitative interviews than are usually affordable, it does so in much less time and with much less inconvenience for researcher and respondent. So, this opens the door to more thorough and extensive research in many fields.
So are there downsides? It enables much more work to be done by far fewer people, but it does not do away with humans altogether. An examination of the instructions given to the AI in the Danish/Norwegian study shows that small omissions or errors could substantially affect the process and the outcome, so serious research expertise will continue to be necessary. The ethical issues considered in all such research remain – in briefing the AI, is the researcher biasing the process, albeit at one remove?
A second concern is with access. The Scandinavian study was looking at a population who can be assumed to be comfortable with computers, but for other topics, that might be a problem. Some difficulties can probably be overcome with voice-based interfaces, but there is still a risk that these techniques might systematically exclude some kinds of potential respondents.
These are not trivial concerns, but arguably similar issues arise in more traditional approaches to research. We need to approach this with proper caution. But there is little doubt that we are on the brink of a revolution in social science research. We are about to learn a lot more about ourselves.
