Matthew B Flynn
January 19, 2016
... is useful for including an international and/or cross-cultural component for research methods classes. Students learn
up of the students survey results, and finally 3) a report on the research methods, including sampling design
Students practice writing up survey results and interpreting their meaning.
Students use online software to analyze data, constructing tables of frequencies, cross-tabulations
and/or cross-cultural component for research methods classes. Students learn not only the basics of survey construction, administration, and analysis, but also the difficulties, limitations, and excitement in collecting data in a multicultural context in which English is typically a second language. After completing the different steps in the survey, students learn about survey design, construct and external validity, sampling, response coding, table construction, and basic statistical analysis (i.e. frequencies and Chi-square).
The survey is administered in India since the country has a large number of “Turkers” who complete “Human Intelligent Tasks” (HITs). Any country with a larger number of Turker is feasible. The details of this class activity are based on Qualtrics but other Internet-based survey software, like SuveyMonkey, should also work. The online survey could be a stand-alone activity, or part of a larger semester-long, research project.
Matthew B Flynn
Georgia Southern University
The online survey in a foreign country is part of a semester-long research project and takes approximately 5-weeks. Requirements include access to online survey tools (i.e. Qualtrics, Survey Monkey, etc.); an account with Amazon’s Mechanical Turk; and sufficient funds to recruit and incentivize subjects to complete the survey: e.g., 50 cents to pay each subject. Sample populations total about 10 subjects for the pilot survey and 300 for the main survey.
Instruction includes common mis-understandings of English in foreign contexts and different forms of spelling and use of English (e.g. British vs. US). Stress that survey questions must employ simple sentence structure using clear wording; avoiding idiomatic expressions; and include additional phrases that provide context or reiterate difficult phrases using different words to improve cross-cultural understanding (Harzing, Reiche and Pudelko 2012). Survey questions of potentially culturally offensive topics should not force responses and instead be left as optional for survey takers. Revising the survey can be somewhat time-consuming. Place a Qualtrics-generated weblink to the survey into Amazon’s Mechanical Turk. See http://brentcurdy.net/qualtrics-tutorials/link/ for linking Qualtrics with Amazon Mechanical Turk.
This assignment is not unlike other TRAIL posts about using online surveys for research methods classes (see Blankenship 2014). The main difference is its administration in a foreign country for which additional clarification focused on survey wording and results requiring additional in-class explication (see Harzing et al. 2012 for more details).
Blankenship, Chastity L. 2014 "Survey Monkey for Research Methods." Assignment published in TRAILS. Washington DC: ASA.
Harzing, A.W.; Reiche B.S.; Pudelko, M. 2012. “Challenges in International Survey Research: A review with illustrations and suggested solutions for best practice.” European Journal of International Management. (5) 4: 1-40.
Learning Goals and Assessments:
Students learn how to design survey questions for a foreign audience (i.e. English as second language) using survey software.
Students complete survey questions avoid bias, double-barreled questions, confused wording, culturally offensive or confusing terminology. Work is reviewed and suggestions provided by peers and instructor.
Students learn the strengths and weaknesses sampling methods (e.g. convenience sampling vs. random-sampling).
Students explain the strengths/weaknesses of sampling methods noting how certain groups are under- or over-represented vis-à-vis the population. The instructor grades the assignments and provides clarification.
Students practice writing up survey results and interpreting their meaning.
Students use online software to analyze data, constructing tables of frequencies, cross-tabulations, and present and interpret data to class and the instructor. Instructor grades written assignments and class discusses student presentations.
Additional Usage Notes.docx
Hypothesis Variables Survey Questions.docx
Methodology Section.docx INTS 3330 Research Methods in International Studies
Hypothesis, Variables & Survey Questions
This assignment includes the following student learning objectives:
1. State a hypothesis with measurable variables.
2. Demonstrate the construction of clear, unbiased survey questions and response categories for your research project.
3. Apply proper spelling, grammar, and citation style rules and guidelines.
4. Locate and use Qualtrics survey software licensed to the university.
A hypothesis consists of a proposition about the relationship of two variables that can be tested or observed. It is stated in value-neutral language and focuses on how the social world works. A strong hypothesis: (1) includes at least two variables; (2) expresses a causal relationship between two variables; (3) can be expressed as an expected outcome or prediction; (4) derives logically from theory, a research question, or qualitative studies, especially those using inductive logic; (5) and is falsifiable, that is, capable of being tested through the collection of empirical data. Complete parts 1-3 below. Itemize your answers as shown. Use complete sentences and correct course terminology.
a. State your hypothesis in correct form with your two variables in proper order. Remember that a variable is not the same as an indicator. A variable is simply something that varies (e.g. income), while an indicator is a measurement ($20,000). For example: “People who are more educated tend to prefer love marriages over arranged marriages.” You should also be able to state your hypothesis in opposite terms. “People with less education tend to prefer arranged marriages over love marriages.” In these examples, the variables are education and love marriage use. For the sake of simplicity, I strongly suggest that you use a socio-demographic variable: age, sex, social class, religion, income, caste, etc. as your independent variable in you hypothesis.
b. Diagram the relationship you hypothesize, using complete variable labels, arrows, and if necessary, (+) and (-) to indicate the direction of the relationship. The example below shows a positive relationship between two ratio-level variables. “More educated people opt for love marriage (as compared to arranged marriage).”
2. INDEPENDENT VARIABLE.
a. Define your variable. This is also referred to as the theoretical construct. It should be a simple definition. For example, I should be able to define “education” to my readers. What do we exactly mean by the concept “education”? Here is an example: “Education is defined as formal training and learning that takes place in an accredited institution.”
b. Statement of your progress in operationalizing the variable. How do you know it when you see it? How can it be measured? Write a complete operational definition, do so. For example: “Education is measured by the years of education in an accredited institution.” Remember to keep it simple and concise.
3. DEPENDENT VARIABLE
a. Repeat steps a-b from #2. In the example above, love marriage is the dependent variable. A conceptual definition would be the following: “A love marriage refers to a legally sanctioned union between two people based on their individual choices and romantic interests. It is the opposite of an arranged marriage in which the parents decide who their daughter/son will enter into a legally sanctioned union.” Operationalizing this definition, you can see that love versus arranged marriage can be observed by people who claim to have entered into either union or state their preference of one versus the other.
4. SURVEY QUESTIONS
1. Look for a survey questionnaire related to your research topic either online or using a library database. You probably even came across a survey in your literature review. Or, you could Google “questionnaire” and your research topic. Review the survey questions and consider whether or not you would like to adopt one or more of them for your research. Explain your rational for copying verbatim, adapting, or discarding these survey questions. Here are some good sources to consider:
General Social Survey
Demographic Health Surveys
Even if you do not want to use previously administered survey questions, you must find at least one survey and cite the survey according to ASA citation style guidelines. I expect both an in-text citation and full citation under a “Bibliography.”
2. Construct four survey questions, with response categories (i.e. answer choices), for your dependent variable (Y). Write a survey question for your independent variable (X). Again, I suggest that one of your independent variables should be a socio-demographic variable, such as age, gender, religion, caste, or social class. You should have a total of THREE survey questions in total. One of the best options is a Lickert question with the following answer choices: Strongly Agree, Agree, Neither Agree Nor Disagree, Disagree, Strongly Disagree. Where necessary, provide additional background information about your survey question especially if it is a topic that may be misunderstood for those who know English as a second language.
3. Now set up an account with Qualtrics survey software. Explore the program. Go to “Create Survey”. Create a test survey of your own and explore the possible survey item options. Which of the survey questions under “Create New Item” do you plan to use? Adjust your survey questions based on the options provided through Qualtrics. You should be able to complete this step after you watched the assigned "Survey Basics" and "Best Practices" videos provided by Qualtrics and available here: http://www.qualtrics.com/university/researchsuite/misc-pages/misc/learn-qualtrics-in-5-steps/. There are additional videos provided by Qualtrics detailing the use of the software.
LENGTH: No specific length requirement. Your work should be double-spaced using Times New Roman characters and size 12 font.
SOURCES: The reference list must include complete bibliographic information of each cited article, following the ASA style guide. Be sure you review how to cite interviews.
DUE DATE: Due Before Class on ---. Save the file with your last name and the word “survey questions” (for example: Smith_survey questions). In the top left corner include your name, the name of the course, and the date. Upload the file to the dropbox folder set up for this assignment and provide me with a hard copy. The homework is worth 15 points. INTS 3330 Research Methods in International Studies
Homework #7 (Methodology Section)
This assignment has the following student learning objectives:
1. State the steps involved in survey design so that others may replicate your efforts.
2. Apply proper spelling, grammar, and citation style rules and guidelines.
3. Demonstrate knowledge of reactivity, anonymity and confidentiality, bias, sample, population, percentages vs. percentage points.
4. Analyze sample validity by way of a table construction using sample vs. population percentages.
Write this section in paragraph form using the subheadings below. Use clear topic sentences at the start of each section and employ proper in-text citations where necessary with full bibliography at the end of the paper.
· Hypothesis: State your research hypothesis and include your diagram of dependent and independent variables.
· Survey Instrument: Describe the type of survey used. Specify if any survey software was used? Specify the number of items in the entire survey, whether they are open or closed; the overall organization of the survey, including item sequencing. Where in the survey are your survey questions placed and why?
· Administration: How and when was the survey administered (e.g. in-person, over the Internet, telephone, etc.? Describe the procedures of survey administration; discuss strategies for reducing respondent reactivity, ensuring confidentiality and/or anonymity, and using incentives. Demonstrate your knowledge of these terms.
· Sampling Design: State the target population. What type of sampling procedure used (e.g. convenience, random, snowballing, cluster, etc.) and why such a sampling procedure was used? Explain limitations of the sampling design related to representativeness. How many people completed the survey?
· Assess Sample Validity: Introduce Table 1. Place population values in left column and sample values on the right. Compare sample characteristics (using our data set) with population characteristics about India provided by official data sources (e.g. CIA Factbook, the World Bank, Census Bureau, etc.). Cite the source in your table and in the bibliography, using proper citation style guidelines. Choose two (2) socio-demographic variables to include in your table: sex, religion, age, income, etc. State the variable as a percentage in columns of your table with the total number (N) for the population and sample at the bottom of each column the table. Write one short paragraph per trait, so two (2) short paragraphs in total. Be specific about your assessment: compare each category and state the percentage-point difference between the population and the sample, including direction of error. Did we over-sample or under-sample this category? That is, is this category over-represented or under-represented in our sample compared to the official data about the population? See the sample Table 1 below. You can either state percentages as whole integers (e.g. 50%) or tenths (e.g. 51.1%)—just be consistent. Make sure your entire table is on the same page and not broken into two parts on two separate pages. Here is an example using made up figures about the population and sample:
Table 1: Socio-Demographic Characteristics
Be sure to complete two (2) different tables on two (2) separate socio-demographic characteristics. And write a paragraph about each.
· Conclusion: Provide a brief overall assessment of the sample representativeness (i.e. external validity). How does our sample compare to the overall population of India in terms of its socio-demographic characteristics: sex, religion, caste, income, age, education, local, etc.? You do not have to mention all of these demographic characteristics but you should provide a good overview. Basically, what should we keep in mind when making generalizations about our data to the entire population?
LENGTH: No specific length requirement. Your work should be double-spaced (except for tables) using Times New Roman characters and size 12 font.
SOURCES: The reference list must include complete bibliographic information of each cited article
DUE DATE: Due in class on ---. Save the file with your last name and the word “methodology” (for example: Smith_methodology). In the top left corner include your name, the name of the course, and the date. Upload the file to the dropbox folder set up for this assignment. The homework is worth fifteen (15) points. You will be evaluated on your understanding of research procedure and the correct use of course terminology. ..."
- Research Methods
- Any Level
This class activity consists of an online survey administered in a foreign country. Students write survey questions using Internet-based survey programs, and the instructor pastes the survey link to Amazon’s Mechanical Turk, an online platform for recruiting and...