Cognitive Methods

PSCH 459: Cognitive Research Methods (Lecture and Lab)
Fall 2006 CRNs 23202 and 23469

Thursdays 1-4pm 1075 BSB
Instructor: Jennifer Wiley

WEBSITE: http://litd.psch.uic.edu:8888/psch459/index.html

Overview
This course will cover both the logic and practical methodological considerations of several standard cognitive methods, including “online” probe and RT paradigms, “offline” open-ended response paradigms, trace methods and correlational/simulation methods. Grades will be based on course participation, knowledge of the readings, and weekly exercises (these are the lab portion). Weekly readings will be two or three papers that describe or discuss the method in question, or serve as an example of the paradigm.

Many readings are taken from Puff, CR ed (1982), Handbook of research methods in human memory and cogntion. New York : Academic Press.

Course outline:

Jan 12 Introductions

Jan 19 Overview of Cognitive Methods

Bower GH & Clapper JP (1989) Experimental Methods in Cognitive Science
in M. Posner, Ed., Foundations of Cognitive Science, MIT Press. pp 245-300.
Miller, JR, Polson, PG, & Kintsch, W. Problems of methodology in cognitive science.
In W. Kintsch, JR Miller, and PG Polson (Eds.), Method and tactics in cognitive science, pp. 1–18.

Jan 26      Probe Word/RT Methodology (Part 1)
               Logic of the Paradigm, Presentation and Timing issues, Baseline Conditions, Randomized vs. Ordered lists, Counterbalancing
               Data Transformations, Outlier Analyses

Ratcliff, R. (1993). Methods for dealing with reaction time outliers. Psychological Bulletin, 114, 510-532.
Masson, M. E. J., & Bodner, G. E. (2003). A retrospective view of masked priming: Toward a unified account of masked and long-term repetition priming. S. Kinoshita & S. J. Lupker (Eds.), Masked priming: The state of the art (pp. 57-94). New York: Psychology Press.

Word Frequency and other helpful tools for matching target lists: http://www.psy.uwa.edu.au/mrcdatabase/uwa_mrc.htm

Feb 2       Probe Word/RT (Part 2)
                Alternative methods, Interpretation of Probe/RT data
                Speed Accuracy Tradeoffs, d'

Outlier.sav
REVISED Outlier Assignment
Murdock (1982) in Puff, CR ed (1982), Handbook of research methods in human memory and cogntion. New York : Academic Press.
Pachella 1974 (not assigned) but a good and oft-cited reference on reaction time experiments

Feb 9 Text/Reading Time Methodology (Part 1)
Subject and Item analyses (F1 and F2)

Dprime assignment
Dprime.sav
Clark, H. (1973). The language-as-fixed-effect fallacy: A critique of language statistics in psychological research. Journal of Verbal Learning and Verbal Behavior, 12, 335-359.

Raajimakers (the rest of the story)

Feb 16     Text/Reading Time Methodology (Part 2)
                Presentation, Timing, and Response Issues
                Comparison conditions, Alternative Dependent Measures (eyetracking as as reading methodology)

F1, F2 assignment
Math.sav
Eyetracking Methods Paper (Richardson & Spivey)
Keenan, Potts, Golding and Jennings from Graesser and Bower 1990

Feb 23 Card Sort Analyses

Collect card sort data
Hubert & Pellegrino (Puff)Puff, CR ed (1982), Handbook of research methods in human memory and cogntion. New York : Academic Press.

Lecture on Card Sort Analysis

Mar 3 Free Recall Clustering Analysis

            Do Sorting Analysis For your Data:
                        Create a Tree Diagram for your data and 2 models.
                        Create a distance matrix for your data and 2 models.
                        Convert to 3 columns of data.
                        Create Difference Score between models.
                        Compute correlations for data with each model and difference score.
                        Conclude which model if any fits the data better.
                        Turn in tree diagrams, distance matricies and correlation table with conclusion sentence.

Collect word list recall for 1 male between the ages of 18 and 40 (Instructions and Answer Sheet)

Re-read cluster analysis section of Murphy & Puff (From Puff) especially how to compute RR, MRR, SO and ARC'

Mar 10 Analysis of Open-ended responses, Instrusions, Strategies, Errors
Developing a coding rubric

Cluster/Seriation Assignment (female data) (first 6 males) (1 more male)
Goldman, S. R. & Wiley, J. (2004) Discourse Analysis: Written Text. In N. Duke & M. Malette (Eds.) Literacy Research Methods (pp.62-91). New York: Guilford.

Koedinger, K. R. & Nathan, M. J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. Journal of the Learning Sciences, 13(2), 129-164.

Alternative Reference on Verbal Analysis:
Chi, M.T.H. (I997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 6, 271-315.

Mar 17 Determining reliability and dealing with categorical and frequency data

Problem Solving Coding Exercise (for each student, figure out if they got the correct answer, and try to identify what strategy they used to solve the problem)
We will develop a coding scheme and test reliability in class.

Testing Interrater Reliability

Cohen 1960 citation classic
http://www.temple.edu/mmc/reliability/ (especially section on how to compute Kappa with SPSS)

Mar 24 No class, Spring Break

Mar 30 Think-Aloud Protocols as a Trace Method

         Everyone needs to pick one of these sets of readings to be responsible for discussion with the rest of the class.
            Everyone should look over all readings, but the read the one they need to convey the closest. You are responsible for telling us conclusions
            and also providing expanation of how they are reached (i.e. what evidence or arguments are offered)

1) What are think aloud protocols? How does one do it correctly?(Margret, Travis)

Ericsson, K. A., & Simon, H. A. (1977). Verbal reports as data . Psych Review

                 van Someren, Maarten W., Yvonne F. Barnard, and Jacobin AC Sandberg. The. Think Aloud Method: A Practical Guide to
                 Modelling Cognitive Processes.
               http://hcs.science.uva.nl/usr/maarten/Think-aloud-method.pdf

                Chapters 3 & 4
                The think aloud method
                Practical procedures in obtaining think aloud protocols

2) What is self explanation? What is the difference between Self Explanation and Think Aloud? (Leah, Lauren, Kari)

Ericsson, K. A., & Simon, H. A. (1998). How to study thinking in everyday life: Contrasting think-aloud protocols with descriptions and
explanations of thinking. Mind, Culture, & Activity, 5(3), 178-186.

Chi, M.T.H. (2000). Self-explaining expository texts: The dual processes of generating inferences and repairing mental models.
In R. Glaser (Ed.), Advances in Instructional Psychology, Hillsdale, NJ: Lawrence Erlbaum Associates. 161-238. (PDF).

3) What phenomena might not be good to study with verbal reports? (Pat, Hinze, Aliza)

Nisbett, R. E., & Wilson, T. D. 1977. Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84: 231-259.

Schooler, J. W., Ohlsson, S., & Brooks, K. (1993). Thoughts beyond words: When language overshadows insight.
Journal of Experimental Psychology: General, 122, 166-183.

Apr 6 No class, AERA (collect and transcribe think aloud protocols)

Apr 13 Eyetracking/Video-coding as a Trace Method (reading patterns, gesture, actions, progress)

Transana

Apr 20 Protocol Coding and Analysis (continued)

Bakeman Observational Data Analysis

Transcriptions for coding:
subject 1

Apr 27 Calibration/Correlational Methods, Modelling as a means of articulating theory and inferring process
(and eyemovements as a source of trace data and data for modelling)

Hintzman, D. L. (1991). Why are formal models useful in psychology? In W. E. Hockley & S. Lewandowsky (eds.), Relating theory and data: Essays on human memory in honor of Bennet B. Murdock (pp. 39-56). Hillsdale, NJ: Erlbaum.

Roberts, S. & Pashler, H. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107, 358-367.

Carpenter, Just & Shell

Final assignment: Develop Protocol/Interview Study Materials