Do you agree that Georgia should declare military neutrality while maintaining European integration? That would mean that Georgia will not be a member of any military alliance. The military bases of foreign countries will not be located in Georgia and military training with the participation of foreign countries will not be conducted. There are three possible answers: ‘yes’, ‘no’, or ‘I don't know/I haven't decided yet’.

This is a question being used by one anti-Western party with several MPs to ‘investigate’ public attitudes in the country. The leaders of the Alliance of Patriots Party do not trust any of the established research organisations, according to which the vast majority of Georgia’s population support membership of the European Union and NATO. 

One of the ‘arguments’ made by the Alliance of Patriots against research organisations NDI and IRI is that their polls of several thousand people carried out according to quantitative research standards do not really reflect public opinion. This while the numerous ‘surveys’ carried out by the party itself are apparently beyond criticism. 

The party — which sees NATO integration as unattainable and considers it necessary for Georgia to negotiate with Russia — argues that the public does not support the Euro-Atlantic course. Party members have maintained this rhetoric for some years. 

Several years ago, they decided to ‘strengthen their arguments’ by initiating their own polls and presenting the results as the opinion of Georgia’s population. 

At first, the number of respondents was in the tens of thousands. Now they’ve become more ambitious and have outlined plans to interview over a million citizens. 

In the meantime, the party is not limiting itself to rhetoric and is also staging protests against these organisations, which includes gathering in front of the US Embassy and demanding the country’s main strategic partner ‘respond to their “violation” of Georgian national interests’. 

The polls carried out by the Alliance of Patriots to counter those of established research organisations have many significant flaws.

These include violations of the general principles of selection, of research ethics, of the rights of respondents, and first and foremost, an anti-Western and pro-Russian narrative in the wording of questions.

To find out how such polls should be carried out and to investigate the threat of the ‘Survey of a Million Respondents’, we spoke to leading members of the Alliance of Patriots along with sociologists, statisticians, researchers, and political scientists.

Gocha Tevdoradze, the political secretary of the Alliance of Patriots and Irakli Chikhradze, the head of the party’s staff, spoke to On.ge about the content and technical details of the poll.

Research Methodology 

The party has selected three different methods for the survey: door to door, phone calls, and random polls taken at demonstrations and public gatherings. 

No specific age, sex, or demographic category has been selected for the study and they plan to ask ‘everyone’ whose phone numbers they can gather from databases on the Internet. Not every Georgian citizen has their phone number in these databases, but many do.

After previous research conducted by the Alliance of Patriots, a number of people submitted complaints to the State Inspector against the party due to the use of their phone numbers. 

The State Inspector’s office explained to On.ge that the party obtains data from various publicly available sources, which is not prohibited by law, however, the database used by the party should not be placed on the Internet. 

One of the study’s participants recorded the survey process on their phone and provided it to On.ge.

The objective and content of the research

‘Our research will interview more than a million people in Georgia, therefore it will no longer be necessary to ask which field was used and how this field was selected. When you interview more than a million people, such studies have a high degree of reliability’, the party’s political secretary, Gocha Tevdoradze, said.

When asked why he thinks that a poll of a large number of people is directly proportional to its representativeness, he answered:

‘When there are 100 people and we are interested in the attitude of these 100, we can interview 10-15 people and then expand it to 100 and say we have a 1.2% or 3% margin of error. But when we are interested in the attitude of 100 and interview 100 people, we cannot say that there is any margin of error.’

However, this example does not fit the Alliance of Patriots’ survey, as they will not be able to survey all the citizens or all voters in Georgia, even though more than a million (it’s unspecified, how many more) people are going to be polled. To this point Tevdoradze has the following answer:

‘How many people participate in the elections in Georgia? More than 1.8 million people never go to the polls in any election and when we say that we will interview more than a million people, it means that we will know the mindset of society with maximum confidence. That’s at least 50, 60, 70 per cent of those polled who are involved in the country’s political life and who participate in the elections’, states Tevdoradze. ‘Hence we can’t raise the issue of reliability’, he adds.

It is unclear what guarantee the party has that only those of electoral age will take part in the poll, as their phone survey does not provide for a monitoring mechanism.

When asked where those phone numbers come from, Irakli Chikhradze, the head of the party’s staff, answered that they come from public databases which are ‘everywhere on the Internet’. Chikhradze claims that they randomly dial different numbers from these databases.

Seven shortcomings in the Alliance of Patriots study

In order to find out about the shortcomings of the research method selected by the Alliance of Patriots and how this could affect the results, On.ge contacted several sociologists.

Number one: A survey cannot be carried out without error unless it is a general census.

The sociologists explained that polling cannot be done without error unless it is a universal survey or census. An increase in the number of respondents, they said, does not necessarily increase the accuracy of a study. 

To follow all professional standards it takes months to interview tens of thousands of respondents.

Number two: Mixed methods should not be used during a survey unless repeat questioning of the same respondents can be ruled out.

The second major mistake, the sociologists said, was to use a mix of different research methods. 

They said that it was inadmissible to combine phone survey and door-to-door methods to calculate the overall result. When using different methods, the possibility of respondents being interviewed twice should be considered.

Sociologist Teona Mataridze said that each method of obtaining information is based on a different principle of operation.

‘For door-to-door surveys, we use multi-stage, cluster selection. Household selection is based on another principle — selecting the respondent in the household. For a phone call, you no longer have a household unit, the access point there is a respondent’, she explained. 

Number three: Representativeness must be insured in quantitative research — you cannot question people who are called randomly or those met at public gatherings.

Sociologist Teona Mataradze explained to On.ge that one of the most important details in a phone survey is a perfect base, a method of selection, and not a survey of a million people.

‘The main characteristic of a quantitative survey is its representativeness. Selectivity is a determinant of representativeness. If the selection database used during a phone survey is complete and includes all potential respondents, i.e. all citizens of Georgia who are 18+ and a selective call will be made from this database, this is a representative study. A large number of respondents and more than a million people are not needed at this time’, explained Teona Mataradze.

Sociologist Nino Durglishvili explained why people shouldn’t be polled in a crowded place:

‘It is wrong to select people randomly by stopping them in the street. As a rule, most people in the street are not there by chance. For example, if you live on Vazha Pshavela and work on Chavchavadze, you will most likely be stopped in one of these streets, and not in Sololaki, unless you get there due to some [other] factor.’

According to the sociologists, any respondent should have the same chance of being selected, this is an essential factor during research. This method is not followed by the Alliance of Patriots. 

Ivane Kechakmadze, a selection specialist and statistician, said that if the database does not contain the numbers of everyone who has a phone, the selection will not be considered complete. 

‘In this case, I will receive definitive information only about the respondents whose numbers are included in the database and not full information about the population of Georgia’, said Kechakmadze.

Number four: The party did not disclose complete information about the research instruments used and thus violated research ethics.

As Nino Durglishvili explained, in addition to taking into account technical details, it is important to observe research ethics, which determine the reliability of the results. 

‘According to research ethics, if the results of the research are public, both the selection mechanisms and the research tools must also be public so that a person can make an adequate assessment. The database must also be public.’ 

According to the rules of ethics, whoever is paying for the research should also be public, because the interest in conducting the research may depend on this.

Number five: The Alliance of Patriots survey is not carried out by qualified personnel.

The sociologists explained that research results can be influenced by the interviewer’s qualifications and it is important to ask properly formulated questions. For this, interviewers who work with respondents should undergo special training. 

The respondents of the Alliance of Patriots survey reported they were given the same questions in different formulations which impacted the way they gave their answers. 

However, the party’s political secretary, Gocha Tevdoradze, insisted that their interviewers obeyed the rules. 

Irakli Chikhradze, head of the party’s staff, explained that the interviewers were volunteers and that this was an unpaid job. 

He said that hundreds of supporters were involved in the process and that the task was not difficult, with some making phone calls and others inviting respondents to the office to fill out a questionnaire (before the outbreak of COVID-19).

Number six: the respondents were not given sufficient information.

Based on the phone recording provided to On.ge, sociologist Nino Durglishvili singled out several violations that could affect the respondent’s decision and consequently the results of the research.

She said it was important for an interviewer to obtain the informed consent of a respondent before starting the research; the interviewer should explain to the respondent that answering the questions is not mandatory. 

A respondent should also be told they have the right not to answer all the questions or to end the survey at any desired time. 

Some respondents told On.ge that they did not receive any such information. 

Durglishvili also cited privacy protection and age monitoring as issues for this survey. 

In particular, she explained that minors could not be included in a survey without the permission of their parent or guardian. 

An organisation that conducts interviews should have a list of adult people in advance (selection frame). A respondent merely confirming to the interviewer that they are an adult is not sufficient. Without a personal ID number, there is no mechanism to verify their answer.

Number seven: equivocal questions.

As sociologist Durglishvili said, in one of the questions in the recording, the term ‘military neutrality’ is not used correctly — it can be interpreted differently (It should be noted that in the hardcopy version of the questionnaire the term ‘military non-alignment’ is used). 

According to the sociologist, ‘yes’ or ‘no’ answers are not appropriate in this case and for more accuracy it would be better to say: ‘I agree’ or ‘I do not agree’. 

‘It’s not a big violation, however, it does affect a respondent’s answer’, she added.

As Durglishvili explained, the second question was difficult for the respondent to understand because it raised additional questions: ‘Does direct dialogue, the reestablishment of economic relations and the restoration of the visa policy mean a policy of peace?’ 

The respondent may think that it is possible to renew the relationship with Russia in this way, but this does not mean that they support this method. 

‘It is difficult to assess [the answer] unequivocally unless we understand how they interpret the question’, she said. 

According to Durglishvili, the biggest violation can be seen in the respondent’s answer to the last question. 

From the recording, we learn that the respondent does not want to answer this question and said: ‘I will abstain from a specific answer’. The interviewer noted the response as: ‘I have not decided yet’.

The phrases — ‘I have decided and I will refrain from answering’ and ‘I do not know, I have not decided yet’ have absolutely different meanings. 

It appears that the respondent gave an answer, and then, in order to escape this interview, they agreed to the answer proposed by the interviewer. This is not an appropriate methodology, Durglishvili said. 

Why Russia needs this research and why its results are so dangerous

A number of researchers and political scientists investigating pro-Russian propaganda say the study has no academic value at all. According to them, the main problem is not the mistakes made in the research, but the attempt to use the survey to promote the Kremlin’s rhetoric. 

They say that negotiations with Russia and military non-alignment are presented as an alternative that people should consider as a ‘way out from a hopeless predicament’.

David Sikharulidze, a member of the Atlantic Council of Georgia, said that the formulation of the questions was in the interests of the Kremlin. He said the party was trying to instil suspicion in the population and manipulate public opinion. 

Sikharulidze explained the danger of so-called ‘military non-alignment’ and why Russia wishes for such an option for Georgia.

‘Georgia’s recent history, and not only ours, but also that of the other former states of the Soviet Union, proves that the status quo is not a guarantee of protection from Russian aggression.’ 

‘When a country is left without military allies, it immediately falls victim to Russian aggression. This happened in 1921, when communist aggression against an independent Georgia took place. A clear example of this in recent history is Moldova, which has never expressed a desire to join NATO but has been attacked by Russia.’

Members of the Alliance of Patriots have said that such criticism against them is politically motivated. They insist that not aligning with a military bloc is necessary because ‘NATO's door is open, but not open’ and they do not want the public to live with ‘false expectations’. 

In the meantime, they disagree with the criticism that this statement carries a pro-Russian narrative.

As for the results of the study, the party said they will be known in approximately one month. It is difficult to say for sure what type of information will be collected by the survey and how it will be processed or interpreted, which could include many violations in itself — perhaps it will be true that the Kremlin’s propaganda machine will use ‘a million voices’ prior to the elections.

This article is a partner post written by Tamuna Gegidze. The original version first appeared on on.ge, on 2 May 2020.