Table 1 Multi-rater reliability for reviewers rating of 30 studies of diagnostic reliability using QAREL
From: The reliability of a quality appraisal tool for studies of diagnostic reliability (QAREL)
Item | Item description (abbreviated) | Subsequent evaluation | |
|---|---|---|---|
κ | 95% CI | ||
1 | Was the sample of subjects representative? | 0.27 | (0.11, 0.42) |
2 | Was the sample of raters representative? | 0.59 | (0.43, 0.74) |
3 | Were raters blinded to the findings of other raters? | 0.92 | (0.76, 1.00) |
4 | Were raters blinded to their own prior findings? | 0.78 | (0.62, 0.94) |
5 | Were raters blinded to the accepted reference standard? | 0.66 | (0.49, 0.82) |
6 | Were raters blinded to clinical information not part of test | 0.51 | (0.37, 0.64) |
7 | Were raters blinded to additional non-clinical cues? | 0.59 | (0.39, 0.78) |
8 | Was the order of examination varied? | 0.71 | (0.58, 0.84) |
9 | Was the time interval between repeated measures appropriate? | 0.69 | (0.50, 0.88) |
10 | Was the test applied correctly and interpreted appropriately? | 0.35 | (0.18, 0.51) |
11 | Were appropriate statistical measures of agreement used? | 0.73 | (0.54, 0.92) |