The discussion of the construction, evaluation and application of psychological tests is beyond the scope of this course. However, issues of the reliability and validity of a psychological test are parallel to concerns that one may have about any measure. A psychological test is simply an approach to measurement often used in psychology.
So here we provide a brief overview of some of the traditional ways of thinking about the reliability and validity of tests. Be warned, though, that when we discuss the threats to validity in experiementation we shall be using a rather different conceptual framework.
I. Test construction: Introduction and Overview
II. Reliability
III. Validity
IV. Item Analysis
V. Test Interpretation
Showing posts with label C آزمون سازی. Show all posts
Showing posts with label C آزمون سازی. Show all posts
Friday, April 10, 2009
III. Validity
III. Validity
Three major categories:
content, criterion-related, and construct validity
1) content validity:
A test has content validity if it measures knowledge of the content domain of which it was designed to measure knowledge. Another way of saying this is that content validity concerns, primarily, the adequacy with which the test items adequately and representatively sample the content area to be measured. For e.g., a comprehensive math achievement test would lack content validity if good scores depended primarily on knowledge of English, or if it only had questions about one aspect of math (e.g., algebra). Content validity is primarily an issue for educational tests, certain industrial tests, and other tests of content knowledge like the Psychology Licensing Exam.
Expert judgement (not statistics) is the primary method used to determine whether a test has content validity. Nevertheless, the test should have a high correlation w/other tests that purport to sample the same content domain.
This is different from face validity: face validity is when a test appears valid to examinees who take it, personnel who administer it and other untrained observers. Face validity is not a technical sense of test validity; i.e., just b/c a test has face validity does not mean it will be valid in the technical sense of the word. "just cause it looks valid doesn’t mean it is."
2) criterion-related validity:
Criterion-related validity is a concern for tests that are designed to predict someone’s status on an external criterion measure. A test has criterion-related validity if it is useful for predicting a person’s behavior in a specified situation.
2a) Concurrent vs. predictive validation:
First the term "validation" refers to the procedures used to determine how valid a predictor is. There are two types.
In concurrent validation, the predictor and criterion data are collected at or about the same time. This kind of validation is appropriate for tests designed to asses a person’s current criterion status. It is good diagnostic screening tests when you want to diagnose.
In Predictive validation, the predictor scores are collected first and criterion data are collected at some later/future point. this is appropriate for tests designed to asses a person’s future status on a criterion.
2b) Standard Error of estimate
The standard error of estimate (s est) is used to estimate the range in which a person’s true score on a criterion is likely to fall, given his/her score as estimated by a predictor.
2c) Decision-Making
In many cases when using predictor tests, the goal is to predict whether or not a person will meet or exceed a minimum standard of criterion performance — the criterion cutoff point. When a predictor is to be used in this manner, the goal of the validation study is to set an optimal predictor cutoff score; an examinee who scores at or above the predictor cutoff is predicted to score at or above the criterion cutoff.
We then get
true positives (or valid acceptance): accurately identified by the predictor as meeting the criterion standard
False positives (or false acceptance): incorrectly identified by the predictor as meeting the criterion standard.
True negative (valid rejection): accurately identified by the predictor as not meeting the criterion standard.
False negative (invalid rejection): meets the criterion standard, even though the predictor indicated s/he wouldn’t .
2d) Factors affecting the criterion-related validity coefficient:
This is about factors that potentially affect the magnitude of the criterion-related validity coefficient. We will not go into this as it relates to correlation.
3) Construct validity
a test has construct validity if it accurately measures a theoretical, non-observable construct or trait. The construct validity of a test is worked out over a period of time on the basis of an accumulation of evidence. There are a number of ways to establish construct validity.
Two methods of establishing a test’s construct validity are convergent/divergent validation and factor analysis.
3a) Convergent/divergent validation
A test has convergent validity if it has a high correlation with another test that measures the same construct. By contrast, a test’s divergent validity is demonstrated through a low correlation with a test that measures a different construct. Note this is the only case when a low correlation coefficient (b/w two test that measure different traits) provides evidence of high validity.
The multitrait-multimethod matrix is one way to assess a test’s convergent and divergent validity. We will not get into this right now.
3b) Factor analysis
Factor analysis is a complex statistical procedure which is conducted for a variety of purposes, one of which is to assess the construct validity of a test or a number of tests. We will get there.
3c) Other methods of assessing construct validity:
we can asses the test’s internal consistency. That is, if a test has construct validity, scores on the individual test items should correlate highly with the total test score. This is evidence that the test is measuring a single construct
also developmental changes. tests measuring certain constructs can be shown to have construct validity if the scores on the tests show predictable developmental changes over time.
and experimental intervention, that is if a test has construct validity, scores should change following an experimental manipulation, in the direction predicted by the theory underlying the construct.
4) Relationship between reliability and validity
If a test is unreliable, it cannot be valid.
For a test to be valid, it must reliable.
However, just because a test is reliable does not mean it will be valid.
Reliability is a necessary but not sufficient condition for validity!
IV. Item Analysis
There are a variety of techniques for performing an item analysis, which is often used, for example, to determine which items will be kept for the final version of a test. Item analysis is used to help "build" reliability and validity are "into" the test from the start. Item analysis can be both qualitative and quantitative. The former focuses on issues related to the content of the test, eg. content validity. The latter primarily includes measurement of item difficulty and item discrimination.
item difficulty: an item’s difficulty level is usually measured in terms of the percentage of examinees who answer the item correctly. This percentage is referred to as the item difficulty index, or "p".
Item discrimination refers to the degree to which items differentiate among examinees in terms of the characteristic being measured (e.g., between high and low scorers). This can be measured in many ways. One method is to correlate item responses with the total test score; items with the highest test correlation with the total score are retained for the final version of the test. This would be appropriate when a test measures only one attribute and internal consistency is important.
Another way is a discrimination index (D).
V. Interpretation of Test Scores
1. norm-referenced interpretation:
involves comparing an examinee’s score to that of others in a normative sample. It provides an indication of where the examinee stands in relation to others who have taken the test. Norm-referenced scores include developmental scores (eg, mental age scores and grade equivalents), which indicate how far along the normal developmental path an individual has progressed, and within-group norms, which provide a comparison to the scores other individuals whom the examinee most resembles.
2. Criterion-referenced interpretation:
involves interpreting an examinee’s test score in terms of an external pre-established standard of performance.
Three major categories:
content, criterion-related, and construct validity
1) content validity:
A test has content validity if it measures knowledge of the content domain of which it was designed to measure knowledge. Another way of saying this is that content validity concerns, primarily, the adequacy with which the test items adequately and representatively sample the content area to be measured. For e.g., a comprehensive math achievement test would lack content validity if good scores depended primarily on knowledge of English, or if it only had questions about one aspect of math (e.g., algebra). Content validity is primarily an issue for educational tests, certain industrial tests, and other tests of content knowledge like the Psychology Licensing Exam.
Expert judgement (not statistics) is the primary method used to determine whether a test has content validity. Nevertheless, the test should have a high correlation w/other tests that purport to sample the same content domain.
This is different from face validity: face validity is when a test appears valid to examinees who take it, personnel who administer it and other untrained observers. Face validity is not a technical sense of test validity; i.e., just b/c a test has face validity does not mean it will be valid in the technical sense of the word. "just cause it looks valid doesn’t mean it is."
2) criterion-related validity:
Criterion-related validity is a concern for tests that are designed to predict someone’s status on an external criterion measure. A test has criterion-related validity if it is useful for predicting a person’s behavior in a specified situation.
2a) Concurrent vs. predictive validation:
First the term "validation" refers to the procedures used to determine how valid a predictor is. There are two types.
In concurrent validation, the predictor and criterion data are collected at or about the same time. This kind of validation is appropriate for tests designed to asses a person’s current criterion status. It is good diagnostic screening tests when you want to diagnose.
In Predictive validation, the predictor scores are collected first and criterion data are collected at some later/future point. this is appropriate for tests designed to asses a person’s future status on a criterion.
2b) Standard Error of estimate
The standard error of estimate (s est) is used to estimate the range in which a person’s true score on a criterion is likely to fall, given his/her score as estimated by a predictor.
2c) Decision-Making
In many cases when using predictor tests, the goal is to predict whether or not a person will meet or exceed a minimum standard of criterion performance — the criterion cutoff point. When a predictor is to be used in this manner, the goal of the validation study is to set an optimal predictor cutoff score; an examinee who scores at or above the predictor cutoff is predicted to score at or above the criterion cutoff.
We then get
true positives (or valid acceptance): accurately identified by the predictor as meeting the criterion standard
False positives (or false acceptance): incorrectly identified by the predictor as meeting the criterion standard.
True negative (valid rejection): accurately identified by the predictor as not meeting the criterion standard.
False negative (invalid rejection): meets the criterion standard, even though the predictor indicated s/he wouldn’t .
2d) Factors affecting the criterion-related validity coefficient:
This is about factors that potentially affect the magnitude of the criterion-related validity coefficient. We will not go into this as it relates to correlation.
3) Construct validity
a test has construct validity if it accurately measures a theoretical, non-observable construct or trait. The construct validity of a test is worked out over a period of time on the basis of an accumulation of evidence. There are a number of ways to establish construct validity.
Two methods of establishing a test’s construct validity are convergent/divergent validation and factor analysis.
3a) Convergent/divergent validation
A test has convergent validity if it has a high correlation with another test that measures the same construct. By contrast, a test’s divergent validity is demonstrated through a low correlation with a test that measures a different construct. Note this is the only case when a low correlation coefficient (b/w two test that measure different traits) provides evidence of high validity.
The multitrait-multimethod matrix is one way to assess a test’s convergent and divergent validity. We will not get into this right now.
3b) Factor analysis
Factor analysis is a complex statistical procedure which is conducted for a variety of purposes, one of which is to assess the construct validity of a test or a number of tests. We will get there.
3c) Other methods of assessing construct validity:
we can asses the test’s internal consistency. That is, if a test has construct validity, scores on the individual test items should correlate highly with the total test score. This is evidence that the test is measuring a single construct
also developmental changes. tests measuring certain constructs can be shown to have construct validity if the scores on the tests show predictable developmental changes over time.
and experimental intervention, that is if a test has construct validity, scores should change following an experimental manipulation, in the direction predicted by the theory underlying the construct.
4) Relationship between reliability and validity
If a test is unreliable, it cannot be valid.
For a test to be valid, it must reliable.
However, just because a test is reliable does not mean it will be valid.
Reliability is a necessary but not sufficient condition for validity!
IV. Item Analysis
There are a variety of techniques for performing an item analysis, which is often used, for example, to determine which items will be kept for the final version of a test. Item analysis is used to help "build" reliability and validity are "into" the test from the start. Item analysis can be both qualitative and quantitative. The former focuses on issues related to the content of the test, eg. content validity. The latter primarily includes measurement of item difficulty and item discrimination.
item difficulty: an item’s difficulty level is usually measured in terms of the percentage of examinees who answer the item correctly. This percentage is referred to as the item difficulty index, or "p".
Item discrimination refers to the degree to which items differentiate among examinees in terms of the characteristic being measured (e.g., between high and low scorers). This can be measured in many ways. One method is to correlate item responses with the total test score; items with the highest test correlation with the total score are retained for the final version of the test. This would be appropriate when a test measures only one attribute and internal consistency is important.
Another way is a discrimination index (D).
V. Interpretation of Test Scores
1. norm-referenced interpretation:
involves comparing an examinee’s score to that of others in a normative sample. It provides an indication of where the examinee stands in relation to others who have taken the test. Norm-referenced scores include developmental scores (eg, mental age scores and grade equivalents), which indicate how far along the normal developmental path an individual has progressed, and within-group norms, which provide a comparison to the scores other individuals whom the examinee most resembles.
2. Criterion-referenced interpretation:
involves interpreting an examinee’s test score in terms of an external pre-established standard of performance.
Reliability and Validity
Objectives
TOPICS
A. Formative vs. Summative Assessments
B. Setting Targets and Writing Objectives
C. Reliability and Validity
Assignments
C. Reliability and Validity
In order for assessments to be sound, they must be free of bias and distortion. Reliability and validity are two concepts that are important for defining and measuring bias and distortion.
Reliability refers to the extent to which assessments are consistent. Just as we enjoy having reliable cars (cars that start every time we need them), we strive to have reliable, consistent instruments to measure student achievement. Another way to think of reliability is to imagine a kitchen scale. If you weigh five pounds of potatoes in the morning, and the scale is reliable, the same scale should register five pounds for the potatoes an hour later (unless, of course, you peeled and cooked them). Likewise, instruments such as classroom tests and national standardized exams should be reliable – it should not make any difference whether a student takes the assessment in the morning or afternoon; one day or the next.
Another measure of reliability is the internal consistency of the items. For example, if you create a quiz to measure students’ ability to solve quadratic equations, you should be able to assume that if a student gets an item correct, he or she will also get other, similar items correct. The following table outlines three common reliability measures.
Type of Reliability How to Measure
Stability or Test-Retest Give the same assessment twice, separated by days, weeks, or months. Reliability is stated as the correlation between scores at Time 1 and Time 2.
Alternate Form Create two forms of the same test (vary the items slightly). Reliability is stated as correlation between scores of Test 1 and Test 2.
Internal Consistency (Alpha, a) Compare one half of the test to the other half. Or, use methods such as Kuder-Richardson Formula 20 (KR20) or Cronbach's Alpha.
The values for reliability coefficients range from 0 to 1.0. A coefficient of 0 means no reliability and 1.0 means perfect reliability. Since all tests have some error, reliability coefficients never reach 1.0. Generally, if the reliability of a standardized test is above .80, it is said to have very good reliability; if it is below .50, it would not be considered a very reliable test.
Validity refers to the accuracy of an assessment -- whether or not it measures what it is supposed to measure. Even if a test is reliable, it may not provide a valid measure. Let’s imagine a bathroom scale that consistently tells you that you weigh 130 pounds. The reliability (consistency) of this scale is very good, but it is not accurate (valid) because you actually weigh 145 pounds (perhaps you re-set the scale in a weak moment)! Since teachers, parents, and school districts make decisions about students based on assessments (such as grades, promotions, and graduation), the validity inferred from the assessments is essential -- even more crucial than the reliability. Also, if a test is valid, it is almost always reliable.
There are three ways in which validity can be measured. In order to have confidence that a test is valid (and therefore the inferences we make based on the test scores are valid), all three kinds of validity evidence should be considered.
Type of Validity Definition Example/Non-Example
Content The extent to which the content of the test matches the instructional objectives. A semester or quarter exam that only includes content covered during the last six weeks is not a valid measure of the course's overall objectives -- it has very low content validity.
Criterion The extent to which scores on the test are in agreement with (concurrent validity) or predict (predictive validity) an external criterion. If the end-of-year math tests in 4th grade correlate highly with the statewide math tests, they would have high concurrent validity.
Construct The extent to which an assessment corresponds to other variables, as predicted by some rationale or theory. If you can correctly hypothesize that ESOL students will perform differently on a reading test than English-speaking students (because of theory), the assessment may have construct validity.
So, does all this talk about validity and reliability mean you need to conduct statistical analyses on your classroom quizzes? No, it doesn’t. (Although you may, on occasion, want to ask one of your peers to verify the content validity of your major assessments.) However, you should be aware of the basic tenets of validity and reliability as you construct your classroom assessments, and you should be able to help parents interpret scores for the standardized exams.
Try This
Reflect on the following scenarios.
A parent called you to ask about the reliability coefficient on a recent standardized test. The coefficient was reported as .89, and the parent thinks that must be a very low number. How would you explain to the parent that .89 is an acceptable coefficient?
Your school district is looking for an assessment instrument to measure reading ability. They have narrowed the selection to two possibilities -- Test A provides data indicating that it has high validity, but there is no information about its reliability. Test B provides data indicating that it has high reliability, but there is no information about its validity. Which test would you recommend? Why?
Continue to Section Assignments
Home Basic Concepts Selected Response Constructed Response
SITE MAP
This course was developed in partnership between the Pinellas School
District and FCIT at USF. Please send comments or corrections.
TOPICS
A. Formative vs. Summative Assessments
B. Setting Targets and Writing Objectives
C. Reliability and Validity
Assignments
C. Reliability and Validity
In order for assessments to be sound, they must be free of bias and distortion. Reliability and validity are two concepts that are important for defining and measuring bias and distortion.
Reliability refers to the extent to which assessments are consistent. Just as we enjoy having reliable cars (cars that start every time we need them), we strive to have reliable, consistent instruments to measure student achievement. Another way to think of reliability is to imagine a kitchen scale. If you weigh five pounds of potatoes in the morning, and the scale is reliable, the same scale should register five pounds for the potatoes an hour later (unless, of course, you peeled and cooked them). Likewise, instruments such as classroom tests and national standardized exams should be reliable – it should not make any difference whether a student takes the assessment in the morning or afternoon; one day or the next.
Another measure of reliability is the internal consistency of the items. For example, if you create a quiz to measure students’ ability to solve quadratic equations, you should be able to assume that if a student gets an item correct, he or she will also get other, similar items correct. The following table outlines three common reliability measures.
Type of Reliability How to Measure
Stability or Test-Retest Give the same assessment twice, separated by days, weeks, or months. Reliability is stated as the correlation between scores at Time 1 and Time 2.
Alternate Form Create two forms of the same test (vary the items slightly). Reliability is stated as correlation between scores of Test 1 and Test 2.
Internal Consistency (Alpha, a) Compare one half of the test to the other half. Or, use methods such as Kuder-Richardson Formula 20 (KR20) or Cronbach's Alpha.
The values for reliability coefficients range from 0 to 1.0. A coefficient of 0 means no reliability and 1.0 means perfect reliability. Since all tests have some error, reliability coefficients never reach 1.0. Generally, if the reliability of a standardized test is above .80, it is said to have very good reliability; if it is below .50, it would not be considered a very reliable test.
Validity refers to the accuracy of an assessment -- whether or not it measures what it is supposed to measure. Even if a test is reliable, it may not provide a valid measure. Let’s imagine a bathroom scale that consistently tells you that you weigh 130 pounds. The reliability (consistency) of this scale is very good, but it is not accurate (valid) because you actually weigh 145 pounds (perhaps you re-set the scale in a weak moment)! Since teachers, parents, and school districts make decisions about students based on assessments (such as grades, promotions, and graduation), the validity inferred from the assessments is essential -- even more crucial than the reliability. Also, if a test is valid, it is almost always reliable.
There are three ways in which validity can be measured. In order to have confidence that a test is valid (and therefore the inferences we make based on the test scores are valid), all three kinds of validity evidence should be considered.
Type of Validity Definition Example/Non-Example
Content The extent to which the content of the test matches the instructional objectives. A semester or quarter exam that only includes content covered during the last six weeks is not a valid measure of the course's overall objectives -- it has very low content validity.
Criterion The extent to which scores on the test are in agreement with (concurrent validity) or predict (predictive validity) an external criterion. If the end-of-year math tests in 4th grade correlate highly with the statewide math tests, they would have high concurrent validity.
Construct The extent to which an assessment corresponds to other variables, as predicted by some rationale or theory. If you can correctly hypothesize that ESOL students will perform differently on a reading test than English-speaking students (because of theory), the assessment may have construct validity.
So, does all this talk about validity and reliability mean you need to conduct statistical analyses on your classroom quizzes? No, it doesn’t. (Although you may, on occasion, want to ask one of your peers to verify the content validity of your major assessments.) However, you should be aware of the basic tenets of validity and reliability as you construct your classroom assessments, and you should be able to help parents interpret scores for the standardized exams.
Try This
Reflect on the following scenarios.
A parent called you to ask about the reliability coefficient on a recent standardized test. The coefficient was reported as .89, and the parent thinks that must be a very low number. How would you explain to the parent that .89 is an acceptable coefficient?
Your school district is looking for an assessment instrument to measure reading ability. They have narrowed the selection to two possibilities -- Test A provides data indicating that it has high validity, but there is no information about its reliability. Test B provides data indicating that it has high reliability, but there is no information about its validity. Which test would you recommend? Why?
Continue to Section Assignments
Home Basic Concepts Selected Response Constructed Response
SITE MAP
This course was developed in partnership between the Pinellas School
District and FCIT at USF. Please send comments or corrections.
Subscribe to:
Posts (Atom)