- Experimental research
- Purpose: make causal inferences about the relationship between the independent and
dependent variables
- Characteristics
- Direct manipulation of the independent variable
- Control of extraneous variables
- Eliminate the variable from the study
- Statistically adjust for the effect of the variable
- Difficulties
- Difficult to carry out in educational settings
- Difficult to control all factors that might affect the outcome
- Consumers should know the limitations associated with a study to judge the usefulness of
the findings
- See the following websites
- A discussion of experimental designs
- A non-technical discussion of experimental designs
- Some examples of experimental and quasi-experimental designs
- Experimental validity
- Internal validity
- Internal validity is the extent to which the independent variable, and not other
extraneous variables, produce the observed effect on the dependent variable
- Threats
- History - extraneous events (e.g., the crash of the stock market, 9-11, the
invasion of Iraq, etc.) have an effect on the subjects' performance on the
dependent variable
- Selection - groups that are not equal due to differences in the subjects in
those groups (e.g., positive and negative attitudes, high and low
achievers, etc.)
- Maturation - subjects' maturation over the course of the study
- Pretesting - the effect of having taken a pretest
- Instrumentation - poor technical quality (i.e., validity, reliability) or
changes in instrumentation
- Treatment replications - insufficient replications of the treatment
- Subject attrition - differential loss of subjects from groups
- Statistical regression - the natural movement of extreme scores toward
the mean
- Diffusion of treatment - the treatment is given, usually inadvertently, to
the control group
- Experimenter effects - different characteristics or expectations of those
implementing the treatments across groups
- Subject effects - the effects of being aware one is involved in a study
- Hawthorne effect
- John Henry effect
- Resentful demoralization
- Novelty effect
- Control of effects through the researcher's choice of specific research designs
- Interpretation of the results is tempered by the existence of internal validity
concerns
- Internal validity
- Conceptual definition of internal validity
- External validity
- External validity in general is the extent to which the results are generalizable
- Five factors affecting external validity
- Subjects
- Representativeness of the sample in comparison to the
population
- Consistency of the results across subgroups within the sample
- Personal characteristics of the subjects
- Subject's awareness of being involved in a study
- Situations - characteristics of the setting (e.g., specific environment,
special situation, particular school, etc.)
- Time - explanations can change over time
- Treatments - specific way in which an experimental treatment is
conceptualized, operationalized, and administered
- Measures
- Different instruments measure content or constructs differently
- Measures change across studies
- Control of both types of threats through sampling procedures
- Generalization of results is tempered by external validity concerns
- Experimental designs
- Notation
- R indicates random selection or random assignment
- O indicates an observation (i.e., test, observation score, scale score, etc.)
- X indicates a treatment
- A, B, C, ... indicates a group
- Pre-experimental designs
- Types
- Single group pretest only
- A X O
- Internal validity threats
- History, maturation, attrition, experimenter effects,
subject effects and instrumentation are viable threats
- Useful only when the research is sure of the status of the
knowledge, skill, or attitude being changed and there are no
extraneous variables affecting the results
- Single group pretest posttest
- A O X O
- Internal validity threats
- Maturation and pretesting are threats
- History and instrumentation are potential threats
- Useful when subject effects will not influence the results, history
effects can be minimized, and multiple pretests and posttests are
used
- Non-equivalent groups posttest only
-
- Internal validity threats
- Selection is a threat
- History, maturation, and instrumentation are potential
threats
- Useful when groups are comparable and subjects can be
assumed to be about the same at the beginning of the study
- None of these designs control internal validity threats well
- Quasi-experimental designs
- Types
- Non-equivalent pretest-posttest, experimental control groups
-
- Non-equivalent pretest-posttest, multiple treatment groups
-
- Useful when subjects are in pre-existing groups (e.g., classes, schools, teams,
etc.)
- Threats to internal validity - see Table 9.2
- Selection is the major concern
- Controls for statistical regression
- Likely to control for most other threats provided the groups are not
significantly different from one another
- True experimental designs
- Terminology
- Random assignment
- Subjects are placed into groups using a random procedure
- This ensures equivalency of the groups
- Random selection of subjects
- Subjects are chosen from a population using random procedures
- This ensures generalizability to the population from which the
subjects were selected (i.e., external validity)
- Types
- Randomized posttest only experimental control groups
-
- Randomized posttest only multiple treatment groups
-
- Randomized pretest-posttest experimental control groups
-
- Randomized pretest-posttest multiple treatment groups
-
- Threats to internal validity - see Table 9.2
- Controls for selection, maturation, and statistical regression
- Likely to control for most other threats
- Factorial designs
- Research designs containing two or more independent variables (e.g., a study of
the effects of two instructional strategies on male and female students' math
achievement; a study of two counseling approaches on middle and secondary
students' self-esteem
- Effects
- Main effects - there is a main effect for each independent variable
- For the first example above, there is one main effect for
instructional strategy and one main effect for math
achievement
- For the second example above, there is one main effect
for counseling approach and one main effect for school
level
- Interaction
- A different effect for the level of the first independent
variable across the levels of the second independent
variable
- Examples
- For the first example above, the first instructional
strategy is effective for males but not females,
whereas the second instructional strategy is
effective for females but not males
- For the second example above, the first
counseling approach is effective for secondary
students and not so for middle school students,
while the second strategy is effective for middle
school students but not so for secondary
students
- In both examples one cannot state the
effectiveness of the instructional strategy or
counseling method without qualifying it relative to
the gender or school level respectively
- Examples of factorial designs
- Criteria for evaluating experimental research
- The primary purpose is to test causal hypotheses
- There should be direct manipulation of the independent variable
- There should be clear identification of the specific research design
- The design should provide maximum control of extraneous variables
- Treatments are substantively different from one another
- The number of subjects is dependent on equal to the number of treatment
replications
- Single subject research designs
- Designs in which the effect of an experimental treatment is studied for one subject
- Repeated measurement of the dependent variable before, during, and after
implementing treatment
- Not restricted to one (1) subject, but rarely involves more than three (3) subjects
- Used extensively in studies involving exceptional children or counseling
- Characteristics
- Reliable measurement
- Repeated measurement
- Clear description of the conditions
- Baseline and treatment conditions
- One variable investigated
- Specific designs
- Notation
- A indicates a baseline condition without treatment
- B indicates a treatment condition
- A B A
- Multiple observations are made during initial baseline time frame; multiple
observations during treatment implementation time frame; and treatment
withdraw and multiple observations during the second baseline time
frame
- Variations on this design include A B A B (i.e., including a second
treatment phase)
- Limitations
- Complicated statistical analysis of the data
- Interpretation of specific outcomes (e.g., a lasting effect of
treatment that does not diminish in the second baseline
observations)
- Multiple baseline designs
- Extension of the A B A design to include more than one subject,
behavior, or setting
- These designs enhance the generalizability of the results
- Criteria for evaluating single subject research
- Reliable measurement of the target behavior
- Target behavior is clearly defined in operational terms
- Sufficient measurements are made during each time frame to establish stability
- Full descriptions of the procedures, subjects, and settings are provided
- Use of one (1) standard treatment
- Control of experimenter and/or observer effects
- Results should have practical significance
|
|
