Psychology 1025: Tests and Measurement Contact Information:

Professor: Irene Hanson Frieze email: frieze@pitt.edu

Fall 2002 office: 3329 Sennott Square

Office hours: Thursday 9:30 to 10:30 am Telephone: (412) 624-4336

Course Description.

This upper level course discusses the interpretation of standard psychological tests and the development of new scales for measuring personality, attitudes, or abilities. Topics to be covered include scale reliability and validity, and methods of assessment. We will briefly discuss testing in business and educational settings. The course will provide practical experience in scale construction and scale assessment.

Course Objectives:

Upon completion of this course, students should be able to:

1. Develop a valid and reliable scale for psychological measurement.

2. Be aware of the major methods of evaluating scales and tests.

3. Apply the concepts of validity and reliability to actual examples.

4. Administer and interpret scales in business and other settings.

Prerequisites:

· Understanding of basic statistics (generally through completing at least one course in Statistics) and Research Methods.

· Completion of at least three previous courses in Psychology before taking this course.

Required Textbook:

Cohen, R.J., & Swerdlik, M.E. (2002). Psychological testing and assessment: An introduction to tests and measurement. (Fifth Edition). McGraw Hill.

Grading and Course Requirements.

Grading is based on a 1000 point system, with 900 and above an A, 800 and above a B, 700 and above a C, 600 and above a D, and below 600 an F. The first 25 points within each group are minus grades [i.e., 800 to 824 is a B-] and the last 25 points are plus grades [i.e., 875-899 is a B+].

• Exams. There will be two exams, a midterm and a final exam. Each is worth 250 points. Each exam will have 40 multiple choice questions for 200 points and two essays for 50 points. Possible essay questions will be distributed before the exam.

• Attendance and Participation. 100 points possible. Points are awarded for attending class and participating in class discussion. There are no excused absences. It is the student’s responsibility to sign the attendance sheet at each class to verify attendance.

• Group Project. A total of 200 points are possible, for a project plan (150), and project manager/group rating (50 points).

• Final Paper. A total of 200 points possible for a paper outlining how to develop a scale. Not required for project managers, who will do class presentations instead.

Late exams or papers. Late exams can be taken up to two weeks after the scheduled exam. They will be all essay exams, and are generally considered harder than the regular exams. Late papers will be penalized ½ grade for each week late.

……………………………………..

Group Project – Development of a Scale

Each group will be responsible for developing a scale of 10 to 20 items for measuring some personality concept or some type of attitude or for evaluating some type of knowledge or ability. For example, the scale might measure motives for work, or burnout, or attitudes about student health services, or typing skill on the computer or knowledge of sexually transmitted diseases. During the semester, the group will develop items for the scale and administer the scale to a sample large enough for scale validation and reliability assessment. In general, scales should be anonymously administered. Data from the scale will be analyzed using SPSS, a widely used statistical program.

There are a number of tasks to be done in scale development. Each member of the group should select one of these tasks. The tasks include:

• Project Manager. Makes sure each part of the project is completed on time. Prepares a draft and final copy of the complete survey, identifying scale items (noting reversals). Reports on measures to be used for validation and any demographic questions and how they might be used. Reports on assessment of reliability. Ensures that ethical requirements of the project are met. Prepares a summary of the final project and discusses how well the scale worked and what modifications could be made to improve the scale.

• Theorist. Searches the published literature for two journal articles outlining the relevant theory and/or studies looking at similar types of scales and shares this with group members. Reviews the definitions used for the published scales and the reliability and validity information for these scales. Reviews the methods used to collect data for the published scales and the samples used. Discusses relevant demographic variables. Two articles should be reviewed in detail, outlining samples, methods of data collection, reliability and validity data

• Reliability and Validity Assessor. Data analysis. Develops plans for determining reliability and validity. Identifies the types of reliability that will be evaluated. Provides a table showing how items developed for the group scale relate to the concept definition (content validity). Identifies the types of validation to be attempted, and the specific measures to be used for concurrent, discriminate and/or predictive validity or construct validity. Enters the data from the scale into the computer and computes the planned reliability coefficients and validity coefficients, after cleaning the data. [Procedures for data cleaning should be described in detail]. Conducts item analysis to improve reliability and validity, if needed. Prepares a frequency distribution of scores to be used by the Data Collection Manager for developing norms.

• Sampling and Data Collection Manager. Defines the size and type of sample being used. Decides on the method of collecting data on the group scale. Develops specific plans for finding a sample. Supervises collection of data. Prepares the introductory information for the scale and feedback information. [All group members are responsible for helping with the actual data collection as needed.] Reports on item means and standard deviations. Defines the norms that will be used for interpreting test scores. Develops instructions for scoring the scale and interpreting the results

………………………………………………

Due Dates and Assignments

Project manager:

• October 29 or 31. Oral presentation to the class of group plans. Include planned scale introduction, actual survey, feedback, and reliability and validity plans. (175 points)
• December 3-5. Oral presentation of data collection, analysis of survey data, and ideas for improvement. (175 points)
• No paper required. Oral presentations replace paper.
• Group members’ rating. (50 points)

Theorist:

• On or before October 15. Meeting with Dr. Frieze or prepare a written report describing at least two related scales. Information to be shared with group. (150 points)
• December 5. Paper due. (200 points)
• Project manager rating. (50 points)

Reliability and Validity Assessor:

• Provides plans to Group for validity and reliability assessment by October 24.
• Develops plan for data cleaning
• Before December 3, meets with Dr. Frieze to clean data, and to analyze validity and reliability. Shares data with group. (150 points)

• December 5. Paper due. (200 points)
• Project manager rating. (50 points)

Sampling and Data Collection Manager:

• Develops plans for who to sample and how many people are needed by October 24.
• Supervises data collection.
• Shares plans with Dr. Frieze before data collection is initiated and reports on actual data collection afterwards. Prepares formal feedback (150 points)

• December 5. Paper due. (200 points)
• Project manager rating. (50 points)

………………………………………………

Tentative Schedule for Lectures and Group Meetings

August 27. Course overview. Discussion of syllabus.

August 29 – September 3. Overview of psychological testing and assessment. Chapter 1 in text.

September 5 – 12. History of psychological testing. Legal and ethical considerations. Chapter 2.

September 12 – 17 . Defining the sample for scale development. Standardized norms. Chapter 4 [pages 100-112].

September 19. Scale reliability. Chapter 5 [pages 128-148].

September 24-26. Scale validity. Chapter 6. Review for exam.

October 1. Midterm exam.

October 3-8. First group meetings. Discussion of tasks. Plans for the type of scale to be developed.

October 10 – October 24. Development of a scale. Chapter 7. [pages 188-203]. Preparing a report. Group meetings on October 17 and 22.

October 15. Reports from Theorists due.

October 24. Reports from Reliability/Validity Assessors and from Sampling and Data Collection Managers due.

October 29 – 31. Oral presentations on group plans from Project Managers.

November 5 - 7. Overview of personality assessment. Chapters 11, 12 and 13.

November 12. Group meetings.

November 14 – 19. Overview of assessment in business, IQ testing, educational assessment, and other types of testing. Chapter 16 and 17.

November 21. Group meetings.

November 26. No class. Thanksgiving break.

December 3 – 5. Final reports from Project Managers.

December 5. Papers Due.

December 11. Scheduled date for final exam [2 to 3:30 pm].

………………………………………………

Paper. 200 points possible. Due in class on December 5.

Assume you have been asked by your company to develop a scale to measure consumer reactions. Discuss how you would do this, including each section below. Lengths listed should be considered as minimum length guidelines:

1. Definition of the concept being tested. Describe the product or service and define what it is about this product or service that you are testing. 10 points. 1 paragraph.

2. Describe the sample to be used for collecting scale data. How would you find them and encourage them to participate? How large a group should be sampled? What type of people should they be? 40 points. 2 paragraphs.

3. How would you assess reliability and validity? Describe the data that would be collected to test reliability and validity? Explain why you have selected these methods and not other possible methods. 40 points. 1 page.

4. Description of ethical concerns and how each is addressed. 30 points. 1 page.

5. Describe your data analysis. How would data be cleaned? How would the score be computed? What statistics would be done to determine reliability and validity? 30 points. 2 paragraphs.

6. What type of report would be prepared for management? What information would be provided to those completing your test? 30 points. 1 page.

7. Include a complete draft of your survey, including any validity items and demographic questions you would include. 20 points.