Kappa, a statistic that gauges inter-observer agreement corrected for chance, is generally regarded as superior to the percentage agreement sometimes reported. But when is kappa big enough? No one value of kappa can be regarded as universally acceptable. Any published article that claims otherwise is problematic. Whether a given value of kappa is acceptable depends on the percentage accuracy researchers require of their observers.  How accurate observers need to be to achieve a particular value of kappa depends on several factors, the most important of which is the number of codes or ratings observers are asked to assign to events.

We have prepared a computer program, KappaAcc.exe, that computes various kappa statistics and estimates observer percentage accuracy. Whenever a kappa is reported, estimated observer accuracy should be reported as well, as a way to judge the adequacy of the computed value.  The rationale and method for computing estimated accuracy and the KappaAcc program are described in:

Bakeman, R. (2022). KappaAcc: A program for assessing the adequacy of kappa. Behavior Research Methods. https://doi.org/10.3758/s13428-022-01836-1

Click here to download KappaAcc.zip. The ZIP file contains the Behavior Research Methods article as a PDF file and the single executable file, KappaAcc.exe. Once this file is placed in a folder on your computer it can be invoked, like any other program, with a double click; you could also create a shortcut and place it on your desktop. If your computer’s security measures block running of “unknown” executable files, you may need help from your local IT people. KappaAcc is written in Embarcadero® Delphi 10 Seattle. It will run on Windows computers or on Apple computers with a Windows simulator.

Note, the Behavior Research Methods article stated that this ZIP file could be downloaded from https://bakeman.gsucreate.org/KappaAcc. For reasons unknown, Georgia State University decided to “decommission” this site. The correct URL is now https://sites.gsu.edu/bakeman/files/2023/03/KappaAcc.zip