What Is Typically Used To Calculate Interobserver Agreement

Fiss, J. L. (1971). Measure of the scale rated correspondence between many advisors. Psychological Bulletin, 76, 378-382. A procedure to improve the credibility of the data by comparing the independent observations of two or more people from the same event. The IOA is calculated by calculating the number of agreements between independent observers and divided by the total number of agreements plus disagreements. The coefficient is then multiplied by 100 to calculate the percentage (%) Consent. Hawkins, R.

P., and Dotson, V. A. Reliability scores that delude: An Alice in Wonderland trip through the misleading characteristics of interobserver agreement scores in interval recording. In E. Ramp and G. Semb (Eds.), behavioral analysis: areas of research and application. N.J.: Prentice Hall, 1975, 539-376. Suen, H. K., Lee, P.S. (1985). Impact of the use of a percentage agreement on behavioural observation: a reassessment.

Journal of Psychopathology and Behavioral Assessment, 7, 221-234. Fleiss, J. L. Bespoke agreement between two judges on the existence or absence of ownership. Biometrics 1975,31, 651-659. Yelton, A. R., Wildman, B.G., and Erickson, M. T. A probability-based formula for calculating the Interobserver agreement. Journal of Applied Behavior Analysis 1977,10, 127-131.

Repp, A.C., Deitz, D. E., Boles, S.M., Deitz, S.M., and Repp, C. F. Differences between the current methods of calculating the Interobserver agreement. Journal of Applied Behavior Analysis 1976,9, 109-113 Cohen, J. Weighted kappa: Nominal scale agreement with provision for scaled disagrely or partial credit. Psychological Bulletin 1968,70, 213-220. Taylor, D.R.

A useful method for calculating Harris and Lahey`s weighted agreement formula. Behavioural Therapist 1980,3, 3. Mitchell, S.K. Interobserver Accord, reliability and generalization of data collected in observational studies. Psychological Bulletin 1979,86, 376-390. Langenbucher, J., Labouvie, E., Morgenstern, J. (1996). Methodological evolution: measurement of the diagnostic agreement. Journal of Consulting and Clinical Psychology, 64, 1285-1289. Berk, R. A.

(1979). Generalization of behavioural observations: a clarification of the Interobserver agreement and the reliability of the inter-observer. American Journal of Mental Deficiency, 83, 460-472. Harris, F.C. and Lahey, B.B. A method to combine event and non-attendance assessments. Journal of Applied Behavior Analysis 1978,11, 523-527. House, A.E., House, B.J. – Campbell, M.B. Measures of interobserver agreement: Calculation formulas and distribution effects. Journal of Behavioral Assessment 3, 37-57 (1981).

doi.org/10.1007/BF01321350 House, A. E., Farber, J. and Nier, L. L. Accuracy and speed of reliability calculation using different measures of the Interobserver agreement.