Unidimensionality and Reliability

Another group of statistical procedures routinely employed in scale development were used to maximize confidence that the resulting scales were both valid and reliable (De Vaus 2002: 184-5; Kidder 1981:126-8). Like factor analysis, these techniques are utilized once the responses to the questions have been collected. These complementary procedures assess whether each question included in a scale really ought to belong to that scale by measuring whether each particular question included on a scale measures the same underlying concept (such as spirituality), and whether each question, in respect to the other questions of a particular scale, reliably measure the concept under investigation. Together, these procedures are known as item analysis. While there are several types of procedures available to do item analysis, I utilized the two most commonly employed in scale construction: ‘unidimensionality’ and ‘reliability’ (Kidder 1981:126-132).

In essence, these two procedures produce correlation coefficients ranging between 0 and 1 (or 0 and –1). Table 6.1 lists the standard interpretations of linear coefficient values for the social sciences adopted throughout this study. While certain coefficients detect non-linear coefficients (e.g. Lamda or Goodman and Kruskal’s tau), the majority of correlations utilized in this study are designed to measure linear relationships. Exceptions are noted in the correlation tables. Correlation coefficients are extremely useful for exploratory, bivariate analysis. This is because they measure the strength and direction of relationships between two variables (e.g. between ‘age’ and ‘education level’). As an index of strength, the higher the correlation coefficient (whether positive or negative) the stronger the relationship. A positive or negative sign bears no relation to the strength of a relationship; it simply indicates the direction of the relationship. Direction depends upon the coding of the data; or more precisely, the direction of coding. So to properly interpret a coefficient, we need to know if both variables are coding in the same direction. To continue the example, are ‘education’ and ‘age’ both coded so that higher values signal greater education and greater age. A positive coefficient obtaining between ‘education’ and ‘age’ would then index a positive relationship: the greater the ‘age’ the greater the ‘education’. Likewise, a negative relationship (a negative correlation) indexes the opposite relationship. Linear correlation is best visualized in terms of a scatterplot, as indeed it is a numerical shorthand for the graphical information displayed in a scatterplot. Using a two axis graph (bivariate), plotting ‘education’ by ‘age’ would produce a series of points corresponding to the values of these two variables. A positive linear coefficients signals that a line could be drawn with an upward slope moving left to right. A negative linear coefficient would indicate the opposite (downward slope moving right to left). Thought of in these visual terms, the more concentrated the points, the stronger the relationship. And points approximating a straight line indicate a near perfect relationship. Where there is no patterning to the points, there is no relationship.

[Could not fetch included URL]

For purposes of unidimensionality, the set of coefficients for unidimensionality are arrived at by correlating item-to-scale, or correlating the responses to a particular question with the responses to all of the other questions on that scale. As a rule of thumb, a question ought to have a coefficient of at least 0.3 in order to be confidently included on a scale. This ensures consistency amongst the scale questions, with the operating assumption being that if a set of questions all measure or tap or ‘load onto’ the same explanatory concept, then the responses to these questions should be correlated beyond that possible by random error or systematic error in the question design (Kidder 1981:130-1).

So for example, I constructed a scale for explanatory concept or association of heritage by including four questions conceptually related to heritage and identity (Appendix 1, Figure 3, questions 11,12,15 and 23). These ranged from asking whether respondents agreed that “Teotihuacan is part of national identity”, to if “Teotihuacan should be respected as a place of the ancestors”. I then took the total number of responses to these four questions and subjected them to the unidimensionality, or item-to-scale operation. The results are presented in Table A2.4. In the statistical package I utilized (SPSS), unidimensionality results are included with reliability results (below), so for now we want to look at the numbers listed under the third column entitled “corrected item-total correlation”. Each row lists the results for each of the four questions. The correlations for these four questions satisfy the minimum correlation standard of >0.3 (rounded to the nearest tenth decimal place), with values of 0.5, 0.4, 0.4 and 0.3.

While this indicates that the four questions consistently measure the same underlying concept (heritage), indicating that I have a ‘unidimensional scale’, I still want to be certain that the scale is reliable. Since classical measurement theory begins with the assumption that any measurement contains some error (or ‘observed score = true score + error’), a reliable measure is one which has a small error component and so does not fluctuate randomly (Kidder 1982:126). For purposes of this study, this means that respondents answer consistently and would (hypothetically) obtain the same scale score on different occasions. Since it is rarely possible to have respondent’s answer the same questions twice, the alternative is to look at the consistency of a person’s response on any single question (‘item’) as compared to each other question on the same scale (the remaining ‘items’). In statistical jargon, this is termed ‘item-item correlation’ (De Vaus 2002:184). Like item-scale correlation, it is looking for consistency across responses. The assumption being that a set of good (reliable) questions will elicit the same responses from the same individual on two different occasions. A specific statistic, called the ‘Cronbach’s alpha’, is used for these correlations and, like other correlations, ranges from 0-1; the higher the correlation, the greater the reliability of the scale. Again, while no hard and fast rule, most social science researchers stipulate that alpha for a set of questions should be 0.5 or greater; and that ideally it should be upwards of 0.6. So to continue the example, Table A2.4 presents the overall scale alpha in the ‘Reliability Statistics’ box under ‘Cronbach’s Alpha’. For the heritage scale, the statistic is 0.6 at the tenth decimal place. (Additionally, for purposes of scale construction, SPSS also calculates the alpha if a particular question (item) was not included on the scale. These are listed in the final column of the Item-Total Statistics box of Table A2.4). With this alpha, I can be reasonably confident that my heritage scale reliably measures the concept.

In all, these two procedures were performed for each set of questions grouped under an explanatory concept to ensure that they individually and collectively measured the concept (De Vaus 2002:184-6, Fowler 1995, Tables A2.3-6). Again, this was to mitigate the caveat listed above in regards to constructing scales. I needed to be reasonably confident that my questions included in the final scales reliably and accurately measure the concepts or associations that I am interested in. Additionally, as these techniques are conceptually related to factor analysis, in that they assess whether variables (responses to questions) ‘belong together’, I employed these procedures in tandem with exploratory factor analysis (discussed above) to crosscheck that the questions did indeed load onto the targeted concepts (De Vaus 2002:186; Tables A2.1-2).

All of the final questions included on each scale are listed under the discussion of the survey results in the following chapter. Here, there is one more consideration that must be addressed in constructing scale scores. This is the problem of having meaningful upper and lower limits to scales so that they can be compared. I want to be able to say something informative both about a particular person’s overall associations with Teotihuacan and how their associations differ or are similar to other individuals’ associations. To be able to move from the micro to the macro or general level of analysis, I need some standardization amongst the scales. Otherwise, depending upon the number of questions included on each scale and the minimum and maximum scores possible for each question, a scale score can potentially vary between any two values. So how would a score, say, of 4.75 on the spirituality scale score compare with 9.0 on the heritage scale score? Creating equivalence of scores on these ‘new scales’ is easily achieved using the formula:

‘new scale’ = ((old scale score – minimum scale value) / range) x (desired upper limit of new scale)

This is best illustrated with the example of the heritage scale. As coded (above), the values of questions 11, 12, 15 and 23 (those included on the heritage scale) range from 0 (don’t know/no opinion) to 4 (strongly agree). So any respondent can potentially have a scale score ranging from 0 (minimum scale value) to 16 (representing strong agreement on all four questions). So 16 equals the ‘range’ for the ‘old scale score’ or uncalibrated scale score. To make the scale scores comparable, I chose 10 to represent the maximum potential value for all of the scales. This is arbitrary, and could easily have been 1 or 100. 10, then, represents the desired upper limit of the ‘new scale’ for heritage. So Señora Rodriguez (respondent #415) from Otumba, State of Mexico, had a score of 3, 3, 4 and 4 for questions (respectively) 11, 12, 15 and 23, giving her a heritage score of 14 on the uncalibrated scale score. Converting her score with the above formula gives here a new score on a range of 1-10 of 8.75; or ((14 – 0)/16)) x 10 = 8.75. Luckily, the SPSS program allows for the automatic transformation of every individual’s score into these calibrated scale scores by entering the equation under the ‘compute’ function. Now, with defined upper and lower limits for all of the scale scores, both intra- and inter-personal comparisons can be made with some measure of equivalency.

In summary, once compiled, these scales enabled a ‘scale score’ to be assigned to each respondent for each explanatory category allowing cross-comparison of amplitudes and frequencies. To better get at causal factors predisposing certain scale scores and move beyond description, additional questions were developed to measure demographics and background attributes of the respondents. Responses to these questions could then be compared to scale scores and inferences made as to causal factors underlying specific associations with Teotihuacan.

Forward to Sampling Strategy

Containments

Return to Factor Analysis