Suppose you have 30 subjects whose diagnosis is rated by two raters on a scale t

ID: 3046631 • Letter: S

Question

Suppose you have 30 subjects whose diagnosis is rated by two raters on a scale that rates the subject’s disorder as being 1 to 5. The data are given below:

how reliable this coding scheme is in practice. Another way to say this is, “what is the inter-coder reliability?”

Coder1   Coder2
5.00   4.00
4.00   4.00
3.00   3.00
1.00   1.00
4.00   4.00
3.00   3.00
1.00   2.00
4.00   4.00
3.00   3.00
5.00   5.00
1.00   1.00
1.00   1.00
2.00   2.00
3.00   3.00
4.00   4.00
5.00   4.00
2.00   1.00
2.00   2.00
1.00   1.00
3.00   3.00
1.00   1.00
2.00   2.00
3.00   3.00
4.00   4.00
5.00   5.00
5.00   5.00
1.00   1.00
2.00   2.00
2.00   2.00
2.00   2.00

Explanation / Answer

Let X = rating of patient’s disorder by Rater 1 and Y = rating of patient’s disorder by Rater 1.

Concept Base

If the rating system is reliable, the ratings given by the raters must be close to each other since both the raters are rating the same 30 patients.

How close the two ratings are will be decided by the Paired t-test as shown below.

Testing

Let D = Y – X.

Then, D ~ N(µ, 2) where 2 is unknown.

Claim:

The rating system is reliable.

Hypotheses:

Null: H0: µ = µ0 Vs Alternative: HA: µ µ0

Test Statistic:

t = (n) (Dbar - µ0)/s where

Dbar and s are respectively, sample average and sample standard deviation based on n observations on X and Y.

Calculations

Summary of Excel calculations is given below:

Given

µ µ0

n =

dbar

-0.0667

s(d)

1.0807

µ0

tcal

-0.3379

sqrt(n)

5.4772

0.05

tcrit =

2.04523

Raw data is given at the bottom.

Distribution, Critical Value:

Under H0, t ~ tn - 1. Hence, for level of significance %, Critical Value = upper (/2)% point of tn.

Using Excel Functions, the above is found to be: 2.045

Decision Criterion (Rejection Region):

Reject H0 if | tcal | > tcrit

Decision:

Since | tcal | < tcrit, H0 is accepted.

Conclusion:

There is sufficient evidence to suggest that the claim is valid,

that the rating system is reliable. ANSWER

DONE

RAW DATA

Raw data

Coder 1

Coder 2

-1

-2

-1