- Non-experimental research designs
- Research design - the plan and structure of research to provide a credible
answer to a research question
- Purpose of non-experimental designs - describe current existing
characteristics such as achievement, attitudes, relationships, etc.
- Four types of designs
- Descriptive
- Relationships
- Comparative
- Correlational
- Causal-comparative
- Survey
- Descriptive designs
- Studies that describe a phenomena
- Statistical nature of the description (e.g., frequency, percentages,
averages, graphs, etc.)
- Importance of these designs in the early stages of the investigation
of an area
- Criteria for evaluating descriptive studies
- Conclusions about relationships should not be drawn
- Subjects and instruments should be described completely
- Relationship designs
- Designs describing the relationship between two or more variables
- Comparative designs
- These studies investigate the relationship of one variable to
another by examining differences on the dependent variable
between two groups of subjects
- If math scores for males are significantly higher than those
for females, a relationship exists between gender and math
achievement
- If the academic self-concept scores for ninth graders are
significantly different than those for twelfth graders, a
relationship exists between grade level and academic self-concept
- If the third grade achievement scores for whites are not
significantly different that those for non-whites, no
relationship exists between ethnicity and achievement
- Criteria for evaluating comparative designs
- Subjects and instruments are described completely
- Criteria for identifying the different groups is clearly stated
- No inferences are made about causation
- Graphs and images depict the results accurately
- Correlational designs
- Simple correlation - studies examine the relationship between two
variables
- Examples
- Math achievement and math attitudes
- Teacher effectiveness and teacher efficacy
- Cautions in interpreting correlations
- A relationship between two variables (e.g.,
achievement and attitude) does not mean one causes
the other (i.e., positive attitudes do not cause high
levels of achievement)
- Attenuation - the possibility of low reliability of the
instruments makes it difficult to identify relationships
- Restriction in range - a lack of variability in scores
makes it difficult to identify relationships
- Everyone scoring very, very low
- Everyone scoring very, very high
- Large sample sizes and/or using many variables can
identify significant relationships for statistical reasons
and not because the relationships really exist
- Prediction - studies examine the predictive nature of the
relationships between variables
- Simple predictive studies - performance on one variable (i.e.,
the predictor) is used to predict performance on a second
variable (i.e., the outcome or criterion)
- Examples
- Scholastic Aptitude Test (SAT) scores are
used to predict freshmen grade point averages
- Scores from a mathematical attitude scale are
used to predict math achievement scores
- Importance of the time interval between collecting the
predictor and criterion variable data
- Factors influencing correlations
- Attenuation - the possibility of low reliability of
the instruments measuring the predictor and
criterion variables makes it difficult to identify
relationships
- Length of time between the predictor and
criterion variable data collection
- Existence of many factors, not only the one
being examined, that influence the criterion
variable
- Multiple regression - studies that examine performance on
several variables (i.e., predictor variables) to predict
performance on a single variable (i.e., criterion)
- Examples
- Scholastic Aptitude Test (SAT) scores, high
school grade point average, and high school
rank in class are used to predict freshmen
grade point average
- Math attitude scale scores, academic self-esteem scale scores, and prior math grades
are used to predict math achievement scores
- Issues of concern
- Sample size of at least 10 subjects for each
predictor variable
- Relationships among the predictor variables
(i.e., colinearity)
- Significance of correlation coefficients
- Statistical significance
- Does a statistical relationship exist?
- Is the observed correlation significantly different from
zero?
- Practical significance
- Does a relationship of practical importance exist?
- Coefficient of determination (r2) - the percentage of
the criterion variable variation that can be explained
by the variation in the predictor variable
- Guidelines for interpreting the size of correlation coefficients
- Much larger correlations are needed for predictions with
individuals than with groups
- Crude group predictions can be made with
correlations as low as .40 to .60
- Predictions for individuals require correlations above
.75
- Exploratory studies
- Correlations of .25 to .40 indicate the need for further
research
- Much higher correlations are needed to confirm or
test hypotheses
- Multiple correlation coefficients (i.e., those resulting from
multiple regression analyses) of .20 - .40 are common and
usually indicate practical significance
- Criteria for evaluating correlational studies
- Causation should not be inferred from correlational studies
- The reported correlation should not be higher or lower than
the actual correlation
- Practical significance should not be confused with statistical
significance
- The size of the correlation should be sufficient for the use of
the results
- Prediction studies should report the accuracy of predictions
for new subjects
- Procedures for collecting data should be clearly indicated
- Comparing comparative and correlational designs
- Comparative - one variable and two or more groups
- Correlational - one group and two or more variables
- Causal-comparative designs
- Ex-post-facto designs
- Ex-post-facto designs investigate the relationships between
independent and dependent variables in situations where it is
impossible or unethical to manipulate the independent variable
- Example - what is the effect of pre-kindergarten (Pre-K)
attendance on first grade achievement
- Cannot mandate Pre-K attendance for children
- Characteristics and resources of families who do or
do not send their children to Pre-K may influence first
grade achievement
- Similarities with correlational and experimental research
designs
- Issues of concern
- Selecting subjects who are as similar as possible on all
characteristics except the independent variable
- Generalizing beyond the subjects studied
- Correlational causal-comparative studies
- Use of correlational models to investigate possible cause and effect
relationships
- Sophisticated statistical models
- Path analysis
- Structural equation modeling
- Fundamental limitations of all correlational research designs apply
- Criteria for evaluating causal-comparative studies
- Primary purpose is to investigate causal relationships when
experimental designs are not possible
- Presumed causal condition has already occurred
- Potential extraneous variables are considered
- Existing differences between groups being compared are controlled
- Causal conclusions are made with caution
- Using surveys in non-experimental designs
- Surveys represent a data collection method that is very useful in
descriptive and correlational studies because it is versatile, efficient, and
generalizable
- Types of surveys
- Cross sectional designs - information is collected from one or more
groups at the same time
- Examples
- Student's, teacher's, administrator's, and parent's
opinions regarding an extended school year
- Elementary, middle, and secondary teachers' feelings
toward a new school board policy
- Issue of concern - comparisons across groups can be the
result of differences between subjects within the groups
- Fifth and seventh graders opinions can be affected by
a change in the attendance zones of a school
- Longitudinal designs - information is collected from the same
subjects over time
- Example - changes in the academic self-concept of students
from the sixth to the twelfth grade
- Issues of concern
- Loss of subjects over time
- Difficulty tracking subjects over time
- Steps in designing a survey
- Define a purpose and objectives
- Identify the resources needed and the target population
- Costs
- Preparations
- Printing
- Mailing costs - sending and returning
- Analyzing
- Length of the survey
- Time needed to administer the survey
- Sample size
- Choose the method
- Paper
- Electronic
- Telephone
- Interview
- Develop the items - guidelines
- Use clear, unbiased, non-ambiguous language
- Keep it short and simple
- Use grammatically correct language
- Do not write leading items
- Use the same response scale for all items
- Be consistent with wording
- Design the format
- White space
- Font size
- Develop directions
- Make them clear with no ambiguity
- Indicate clearly how subjects are to respond
- Indicate where responses are recorded
- Indicate what subjects should do when finished
- Develop a letter of transmittal
- Keep it brief
- Include a statement of the purpose of the research
- Include a statement of the benefits of the research
- Pilot test
- 15-20 representative subjects
- Identify concerns
- Clarity
- Format
- Responding
- Directions
- Time to complete
- Response rates
- Low response rates are the major limitation of survey use
- Suggestions for increasing response rates
- Design the survey well
- Contact the subjects several times especially following-up on
non-respondents
- Include a self-addressed return envelope
- Use a good transmittal letter
- Use a telephone for follow-up
- Use incentives for completing the survey
- Using electronic surveys
- Internet based surveys
- E-mail attachments
- Web pages
- Advantages
- Reduced time and cost
- Easy access
- Quick responses
- Ease of creating data sets
- Disadvantages
- Limited to those with access to the technology
- Confidentiality and privacy issues
- Evaluations of online university survey research centers
- Criteria for evaluating online survey research centers
