Discuss Research (In)Validity
Research Validity
1. External Validity
2. Internal Validity
a. Extrinsic Validity
b. Intrinsic Validity
- History
- Maturation
- Testing
- Instrumentation
- Statistical Regression
- Selection Biasis
- Mortality
- Diffusion of Treatments
- Causal Time-Order
- Compensation
- Compensatory Rivalry
- Demoralization
Nigem's not too concerned with details on External Validity. In fact, he said, "Now watch this. External Validity deals with generalization and representation. Can you generalize your findings to the outside or "real" world? That is all that you need to say for the essay."
Internal Validity on the other hand is EXTREMELY important (it even earned its own handout: See Sources of Invalidity for Designs 1 through 6).
Internal Invalidity refers to the possibility that the conclusions drawn from EXPERIMENTAL results may not accurately reflect what actually occured during the experiment. You MUST have control over all factors affecting your experiment.
Side Note for people who like the x,y stuff:
Independent Variable (x) = stimulus
Dependent Variable (y)
MAKE SURE "x" (and ONLY "x") influences "y"
Internal Validity has both INTRINSIC and EXTRINSIC components.
The Extrinsic bit just deals with the selection of people used in the experiment. (that's all i've got on this... gimme what ya' got)
INTERNAL INTRINSIC VALIDITY is what it's all about!!!
All we're talking about here, really, is "What could go wrong?" Hmmmmmm.....
These are the sources that can interfere with your Independent Variable (x)
(and of course, we need to BRIEFLY explain these a bit and give an example)
a. History
- something historical happens during the experiment that has some effect
- ie: an assassination of an African-American leader during an experiment on prejudice
b. Maturation
- people change
- ie: short-term=get tired, bored long-term=grow older and wiser
c. Testing
- the actual testing/retesting process
- ie: people become influenced by the tests and their responses change
d. Instrumentation
- what is used to do the measuring/testing
- ie: must use same survey and pre & post-test so one is not more sensitive than another
e. Statistical Regression (to the mean)
- when subjects are so extreme the results will erroneously be attributed to the stimulus
- ie: kids are so bad in math, they can't get any worse or tall people have shorter children
f. Selection Biases
- experimental and control groups MUST be comparable
- ie: use proper methods for subject selection (we studied this before)
- Note: when subjects come from all volunteer group you lose External Validity
g. Mortality
- subjects die or drop out of experiment
- ie: bigots leave during film leaving higer ratio of less prejudiced people for post-testing
h. Diffusion of Treatments
- the more groups you use the more complications arise
- common sense - more groups = more opportunities for Internal Invalidity
i. Causal Time-Order
j. Compensation
k. Compensatory Rivalry (thank you Alicia for noting/adding that... Compensatory Rivalry is when one group competes against the other in an attempt to better performance.)
(i've got nothing on these three... i think i was still recovering from the repeated use of the kittens in the dark experiment... I actually have written in my notes, "Why does he keep referring to the deprivation of light experiment with 1 mo/2, 3, 6, etc. w/the maze and the food? It is so upsetting and disturbing!")
l. Demoralization
- oh, i'm upset because I'm in the Control Group and I feel left out, boo-hoo
- demoralized kids in educational studies may stop studying, act up, or get angry
Now for the fun stuff... how are we supposed to remember all of the factors? I mean, I can tell you what each of them is if you ask me, but listing them all from memory to begin with? Not so much. Sooooo.....
Hairy Men Don't Do Much Shaving To Create Sexy Clean Cut Images
That's right... HMDDMSTCSCCI (I know that's a lot, but hell, there are 11+1 (so that's 12) types of Internal Intrinsic Invalidity Issues - I point that out as your "double checker"... the I's at the beginnning of the words "Internal Intrinsic Invalidity" should remind you that there are 11+ 1 (three III's = three 111's = 11+1 = 12) factors. That way you don't skip a word in Hairy Men Dont Do Much Shaving To Create Sexy Clean Cut Images.)
History
Maturation
Diffusion
Demoralization
Mortality
Selection Bias
Testing
Causal Time-Order
Statistical Regression
Compensation
Compensatory Rivalry
Instrumentation
If you're a cynic and have an easier time remembering negative things instead of funnier stuff, try this one:
How Many Deeply Troubled Marriages Can Improperly Supervised Couples Counseling Sessions Destroy?
Does he want more than this? What do you think?
1. External Validity
2. Internal Validity
a. Extrinsic Validity
b. Intrinsic Validity
- History
- Maturation
- Testing
- Instrumentation
- Statistical Regression
- Selection Biasis
- Mortality
- Diffusion of Treatments
- Causal Time-Order
- Compensation
- Compensatory Rivalry
- Demoralization
Nigem's not too concerned with details on External Validity. In fact, he said, "Now watch this. External Validity deals with generalization and representation. Can you generalize your findings to the outside or "real" world? That is all that you need to say for the essay."
Internal Validity on the other hand is EXTREMELY important (it even earned its own handout: See Sources of Invalidity for Designs 1 through 6).
Internal Invalidity refers to the possibility that the conclusions drawn from EXPERIMENTAL results may not accurately reflect what actually occured during the experiment. You MUST have control over all factors affecting your experiment.
Side Note for people who like the x,y stuff:
Independent Variable (x) = stimulus
Dependent Variable (y)
MAKE SURE "x" (and ONLY "x") influences "y"
Internal Validity has both INTRINSIC and EXTRINSIC components.
The Extrinsic bit just deals with the selection of people used in the experiment. (that's all i've got on this... gimme what ya' got)
INTERNAL INTRINSIC VALIDITY is what it's all about!!!
All we're talking about here, really, is "What could go wrong?" Hmmmmmm.....
These are the sources that can interfere with your Independent Variable (x)
(and of course, we need to BRIEFLY explain these a bit and give an example)
a. History
- something historical happens during the experiment that has some effect
- ie: an assassination of an African-American leader during an experiment on prejudice
b. Maturation
- people change
- ie: short-term=get tired, bored long-term=grow older and wiser
c. Testing
- the actual testing/retesting process
- ie: people become influenced by the tests and their responses change
d. Instrumentation
- what is used to do the measuring/testing
- ie: must use same survey and pre & post-test so one is not more sensitive than another
e. Statistical Regression (to the mean)
- when subjects are so extreme the results will erroneously be attributed to the stimulus
- ie: kids are so bad in math, they can't get any worse or tall people have shorter children
f. Selection Biases
- experimental and control groups MUST be comparable
- ie: use proper methods for subject selection (we studied this before)
- Note: when subjects come from all volunteer group you lose External Validity
g. Mortality
- subjects die or drop out of experiment
- ie: bigots leave during film leaving higer ratio of less prejudiced people for post-testing
h. Diffusion of Treatments
- the more groups you use the more complications arise
- common sense - more groups = more opportunities for Internal Invalidity
i. Causal Time-Order
j. Compensation
k. Compensatory Rivalry (thank you Alicia for noting/adding that... Compensatory Rivalry is when one group competes against the other in an attempt to better performance.)
(i've got nothing on these three... i think i was still recovering from the repeated use of the kittens in the dark experiment... I actually have written in my notes, "Why does he keep referring to the deprivation of light experiment with 1 mo/2, 3, 6, etc. w/the maze and the food? It is so upsetting and disturbing!")
l. Demoralization
- oh, i'm upset because I'm in the Control Group and I feel left out, boo-hoo
- demoralized kids in educational studies may stop studying, act up, or get angry
Now for the fun stuff... how are we supposed to remember all of the factors? I mean, I can tell you what each of them is if you ask me, but listing them all from memory to begin with? Not so much. Sooooo.....
Hairy Men Don't Do Much Shaving To Create Sexy Clean Cut Images
That's right... HMDDMSTCSCCI (I know that's a lot, but hell, there are 11+1 (so that's 12) types of Internal Intrinsic Invalidity Issues - I point that out as your "double checker"... the I's at the beginnning of the words "Internal Intrinsic Invalidity" should remind you that there are 11+ 1 (three III's = three 111's = 11+1 = 12) factors. That way you don't skip a word in Hairy Men Dont Do Much Shaving To Create Sexy Clean Cut Images.)
History
Maturation
Diffusion
Demoralization
Mortality
Selection Bias
Testing
Causal Time-Order
Statistical Regression
Compensation
Compensatory Rivalry
Instrumentation
If you're a cynic and have an easier time remembering negative things instead of funnier stuff, try this one:
How Many Deeply Troubled Marriages Can Improperly Supervised Couples Counseling Sessions Destroy?
Does he want more than this? What do you think?
posted by harvestorm at 3:37 PM
3 Comments:
Since the Almighty Handout doesn't even list all of the sources of Internal Invalidity we are expected to know, I am using it more as a review for the various Experimental Designs (listed on the left).
Otherwise, I think you might be able to get a decent paper airplane out of it! ;-)
Alright, Diffusion of Treatments was I THOUGHT like the tuskegee syphilis 'study' (to bring up an example used in my med anthro class) where people were being provided bogus 'treatment' as part of the (wrong for SO MANY reasons) 'study' of syphilis.
By the end of the study many people had found their own treatments - gone to outside Drs, taken someone else's meds, etc.
So what I'm picturing is that you're trying to test, say the effect of supplementary vitamin C on people - you think it'll make their colds shorter or less frequent or whatever. So you have your control and your test group, at min, but some of the folks in your control group (or both, actually) start taking something they either don't know contains large ammts of vit C, or obtains antibiotics for a cold from their MD. Or perhaps they just always drink 8 times their weight in OJ when they're sick.
Unknowingly, they f-up your results. You can't say that people's lower cold rates were due to the vit c because it could have been the antibiotics they snuck off to get from their dr.
-Or- you don't have statistically relevant results because your control group was getting mass vitC through other methods/treatments and their colds were shorter too.
What AM I describing, if not diff. of treatments (just checking, cause I can think of some options)
Soooooo.... I think I've got it. It's when the control group starts doing something meant only for the experimental group (coming into contact with the independent variable in some way) and totally blows the whole experiment?
Post a Comment
<< Home