Sampling Strategy

A stratified, systematic sampling procedure was utilized for the administration of the final questionnaires (De Vaus 2002:72-5). In designing the sampling strategy, I wanted to collected opinions from those individuals most directly involved with Teotihuacan on a local level. Residents, visitors, and those employed at the archaeological zone, whether in an official capacity (such as INAH archaeologists or guards) or unofficial capacity (such as the numerous ‘wandering crafts vendors’ or vendedores), were whom I wished to elicit opinions from. These individuals, I assumed, would have developed the strongest associations with the site. So while Teotihuacan is certainly a part of ‘global heritage’ as a UNESCO site, I decided to restrict consideration to more local networks. Further studies could certainly be undertaken to assay global associations, but including foreign tourists visiting the site, for example, would detract from the comparative purpose of examining the local ‘archaeological ecosystem’ of Teotihuacan and its centrality to the everyday life of Mexicans.

Figure 6.5: Satellite image of Teotihuacan Valley with archaeological zone and proximate pueblos labeled.

With this goal in mind, I developed a ‘sampling frame’, or an identification of potential respondents, to focus on these individuals. If differences arose between these ‘sub-groups’ of individuals at Teotihuacan, it would be during analysis of the data and not due to an arbitrary distinction beforehand (such as the differences between visitors, students and workers discussed in Chapter 7). Within this ambit, a distinction was necessary to make, however, due to potential respondents being physically on-site and those off-site. Thus the sampling frame was broken into two ‘stratifying variables’: Mexican nationals visiting the site and Mexican nationals residing in the pueblos most proximate to the archaeological zone (San Juan, San Martín, San Francisco, San Lorenzo, San Sebastian and Santa María). For this reason, a simple random sample (SRS) was not utilized. That is, while a SRS is the most basic method for probabilistic sampling in the social sciences (De Vaus 2002:72-5, Kidder 1986:77-80), and has become of increased importance in archaeological methods (Orton 2000, Shennan 1988: 298-331, and see e.g. in Merriman 1991), it must be applied universally. I could have undertaken a SRS of the local inhabitants utilizing Mexican census information and randomly selecting houses for potential respondents, but I then would have had to apply a different method for the ‘itinerant population’ of the archaeological zone itself. This relates to the primary difficulty in using SRS. A good sampling frame must be available for a ‘population’ in order to randomly select cases; where it is not, as with mobile vendors or visitors to an archaeological zone, a SRS may neither be appropriate nor feasible (De Vaus 2002:71, Orton 2000:206-10). Indeed, the vast majority of sampling utilized in archaeology has been described as ‘pragmatic sampling’, where research goals, combined with considerations of accessibility and efficiency of drawing the sample, determine the method of investigation (Cowgill 1970, Orton 2000, Shennan 1988:300). To avoid this methodological inconsistency, combined with my research goal of being more interested in the local population as opposed to drawing statistical inferences to the whole population, I opted to use the less reliable but also probabilistically based stratified, systematic sampling procedure which could be consistently applied whether in the pueblos or on the site.

To do this, a sampling fraction was created for each stratifying category (Mexican national residents and ‘population’ of the archaeological zone) by calculating an estimate of the total local populations in each of these pueblos using the Mexican Instituto Nacional de Estadística, Geografía e Informática (INEGI) and the average number of yearly visitors to the site through the Instituto Nacional de Antropología e Historia (INAH) (INEGI 2007). A sampling fraction is determined by dividing the population estimates by the desired sample size. With an estimate of the population of the two stratifying categories, I needed to determine my desired sample size. Contrary to intuition, the size of the population is largely irrelevant to for the accuracy of the sample (De Vaus 2002:81). Instead, probability theory confines consideration to two key factors: degree of accuracy which we desire for the results; the variation in the population with respect to key characteristics of the study. As a 95% confidence level is standard for most social surveys, I needed to estimate the variation of opinion that the questionnaire might elicit in order to arrive at my sample size. I adopted a very conservative approach to potential variation. That is, I based my estimate of variation upon the expectation of the most heterogeneous response variance (i.e. 50/50 percent split on any particular question). As a rule of thumb, the greater the population variance, the greater the potential sample error; and thus the larger the sample size required for reliability of results. At the 95% confidence level and with a conservative estimate of a great degree of diversity amongst response rates, the desired sampling size for the survey was a minimum of 400 (De Vaus 2002:Tables 6.3-6.4). However, two additional considerations forced me to consider a larger sample size. First, these sample size estimates are for SRS. Not using SRS tends to increase sample error; and so I thought it best to insure reliability by increasing the sample size beyond 400. Furthermore, ‘non-response’, or selected potential respondents who chose not to participate, also posed a potential source of sampling bias (increasing sampling error). To mitigate the effects of non-response I would need to increase the sample size by roughly 20%, or 40 respondents, to counterbalance the average non-response rate of 20% for most social research surveys . Additionally, for the case of non-response, I recorded rudimentary information about the would-be respondent based upon observation. That way, if sampling bias was pronounced during analysis, I could make adjustments based upon this background content of the non-responders (such as weighting the sample based upon background variables). Though this did not become apparent in the analysis, these two additional considerations suggested that a sample size of at least 450 would be prudent.

With my population estimates and my desired sample size, a sampling fraction was then calculated for each stratigraphic category. These numbers are summarized in Table 6.1. For ease, these fractions were rounded to the nearest whole tenth place. Thus, the sampling fraction for the contiguous pueblos on the west side of Teotihuacan = 31,730(population)/450 (desired sample size) = 71. This was rounded down to 70 for ease of counting. Likewise with the west pueblos, or 27,788/450 = 62. Again this was rounded to 60. Finally, the archaeological zone ‘population’ = 11,000/450= 24, rounded up to 25.

[Could not fetch included URL]

Table 6.1: Comparison of population of municipalities of Teotihuacan Valley and average, daily visitors to site with respect to sample sizes of survey

With a sampling percentage at the archaeological zone of 1.2%, I ended up with a disproportionate number of completed questionnaires from individuals on-site. This had to due with my concerns over the political relations of the administration of the site. As federal property, all activities within the zone are closely monitored by INAH administrative staff and security. When I was initially granted permission to begin the survey within the site’s boundary, the then zone director was keen on amassing public input concerning perceptions of Teotihuacan. “Something such as this has not been done on a large scale before here [Teotihuacan]. It will be interesting to understand what individuals [la gente] think about the archaeological zone. And right now, a priority for INAH and the Consejo Nacional [administrative agency overseeing permitting of activities at archaeological sites] is to make changes to the federal law over heritage [la Ley Federal sobre patrimonio cultural] so that we better communicate with the public [divulgación al público]” (director of the archaeological zone, 2004). However, once I began collecting questionnaires, I quickly heard from other INAH employees that the site’s administration was uneasy over the fact that the questionnaire solicited opinion concerning the looting of artifacts from Teotihuacan, the recent controversy of Walmart and the impact upon the site, as well as ideas for better management of the site (for example, Appendix 1, Figure 3, questions 16, 24, 26). In the spring of 2005, several months after I began the survey, the former director vacated his position and a new director was appointed. Shortly after this, an INAH archaeologist at the Center for Teotihuacan Studies [el Centro de Estudios Teotihuacanos], informed me that he had been commissioned by the new director to begin an ‘official questionnaire’.

Figure 6.6: INAH archaeologist with the Center for Teotihuacan Studies, commissioned to distribute an ‘official questionnaire’.

“The director thought it was a good idea to have an official questionnaire [encuesta] done at the site. I came across yours [questionnaire for this study] and I have used several of the same questions. I hope it’s OK…There aren’t any questions about Walmart, though…I am giving it to INAH employees too…I don’t think it is controversial” (INAH archaeologist). With these changes in administration and a similar survey being implemented ‘officially’, my hunch was that I was going to be asked to stop distributing my questionnaire. While I had secured permission to work on the site and in the towns, I knew that the director would only have the authority to rescind my on-site permit. I immediately stopped working in the pueblos and spent all of my time canvassing the archaeological zone. I wanted to be certain that, irrespective of how large my overall sample would be, I would have secured ample questionnaires from the site, particularly as permissions for such a public-oriented project may not be granted again (or at the least, not without INAH oversight). Indeed, several weeks later after I had collected 135 questionnaires, the new director had me abruptly collected in the back of a pick-up truck by the head of security and a cadre of guards with rifles. The director explained to me that “…it was no longer alright for me to be collected questionnaires [encuestas]…Being new [as the director] I want to make sure that INAH and the administrators in Mexico City [el Consejo Nacional] are not worried with respect to what is going on at the site. And while you are not excavating, it does not seem appropriate that you continue to talk to people in the zone…[when asked about working in the towns]…I can’t tell you not too – they are public places. But people are asking about your intentions [preocupaciones por] for the study” (director of archaeological zone, 2005). The director has since vacated his post and yet a new director has replaced him. So simply due to the volatile and unpredictable context of Teotihuacan’s administration, I wanted to make certain I completed as many questionnaires as possible – taking precedence over my original sampling design.

Figure 6.7: Sampling in the San Juan

Sampling was restricted to public places in the pueblos and on the site. In the pueblos, I stationed myself in the principal plazas and adjoining thoroughfares (Figure 6.7). At the archaeological site, I alternated between the five primary entrances. Using the sampling fraction for each locale, I randomly selected the first case using a table of random numbers. Thereafter, every ‘nth’ individual was then asked to complete a questionnaire. So, while at the site I asked every 25th person; while in the pueblos on the east side of the site, I approached every 50th person; and while in San Juan and San Lorenzo, I asked every 100th person. So, for example, every time I began sampling at the site (sampling fraction of 25), a random number between one and twenty-five was drawn. Then the individual corresponding to that number (whether the 2nd or 23rd to pass) was selected as the first respondent. Then I waited until the next 25th person passed and I asked them to complete a questionnaire. This was accomplished using a simple hand-held counter. Precision was not the ultimate goal; but rather to ensure that everyone at the site or in the pueblos had a reasonably equal chance of being selected. This ‘cycling’ effect, or systematically asking only every ‘nth’ individual after the initial random selection is precisely what differentiates systematic sampling from SRS. And it also underscores why it may be inappropriate, though it seems a most reasonable and logical procedure, for certain contexts, especially in suburban residential streets or in cities. Because of the ‘cycling’ of asking only every ‘nth’ individual, a systematic procedure does not give all combinations of cases equal chance of being included. Once the first number is drawn, say at the site, thereafter only every 25th individual is eligible to participate. If I selected only every 25th case in a systematically planned residential area, the results could be biased as every 25th case may correspond to the same type of dwelling (say corner houses or houses adjacent to markets) (for a discussion see Kidder 1986:155-56).

In the setting of pedestrians in public spaces, the concern was with ‘periodicity’, or the possibility that certain types of individuals may reoccur at regular intervals within the sampling frame. However, the geographically distributed nature of the sampling (six removed locations in addition to the five primary entrances to site which are all dispersed) combined with the temporally dispersed nature of the collection (between April and September 2005) mitigates somewhat against these concerns (De Vaus 2002:72).

The more serious caveat of not using a SRS is that the distributions of individual samples may not as closely mirror population distributions (Kidder 1986:159). However, utilized effectively, and with a large enough sample, a stratified, systematic sample should approximate characteristics of the population at large. A comparison between a social breakdown of the national population along the key demographic categories of age and sex with the corresponding breakdown of the sample is presented in Figure 6.2. Except for a higher representation of individuals aged 18-29, the proportions of the sample categories are fairly close to the population proportions (the parameters of these age categories were defined by the Mexican Instituto Nacional de Estadística, Geografía e Informática (INEGI). The proportions of gender are reasonably close, though the sample underrepresented females (51.2% of the total population versus 45.0% of the sample, with 1.1% not reporting their gender).

Table 6.2: Comparison of social breakdown of sample with population profiles (by age and sex)

Overall, the distribution of age categories of the sample resembles the distribution of the population as a whole. There is a ‘bottleneck distribution’ with a much larger representation of young adults and an incremental decrease of each successive age group. The sample, however, augmented this feature of the Mexican population as a whole by over representing Mexicans aged 18-29 (with a sample proportion of 48.0% as compared to a population proportion of 30.4%). While sampling error (chance) routinely skews samples of whatever design away from precisely reproducing population proportions (De Vaus 2002:74), these divergences are most likely due to two factors: where sampling was conducted (in public places where a larger percent of the potential respondents were young adults); selected respondents misrepresenting their age (my sampling frame excluded minors or those younger than 18 years of age, yet most selected young adults were enthusiastic to participate, possibly identifying themselves as over eighteen). A consequence of this divergence is a large proportion of respondents who identified themselves as ‘students’. This lead to breaking down the sample into subgroups during analysis (into ‘visitors’, ‘workers’ and ‘students’; see Chapter 7) which produced some interesting and unintended comparative results. However, this skewing of the sample towards 18-29 year olds diminishes the ‘representativeness’ of the sample in comparison to the national population and means that inferences should only cautiously be made with respect to the Teotihuacan valley’s population as a whole.

In keeping with the calculation of the sample size, a total of 471 questionnaires were completed. The responses were hand-coded by developing a codebook (Appendix 3) and entered into a SPSS 13.0 database (see ‘Coding’ above). All subsequent statistical analysis was conducted using the industry standard statistical software package SPSS 13.0 (SPSS 2006). As Chapter 5 discussed, a wiki, or a particular type of ‘social software’, with customized features for the display of statistical information was developed by myself in conjunction with the Metamedia lab. It was utilized to create an easy user-interface database of all of the original documents, data files, codebook, photographs, and interviews, as well as the descriptive and inferential statistics. The ideal is to make the project’s methods and results transparent and to allow distributed and accelerated access to the primary data. The wiki’s capacity for quickly establishing and easily managing databases, combined with the straightforward navigation through, and information retrieval from, these databases potentially offers a great boon to studies using the tool of statistics. The credibility of statistical studies comes not from the conferral of objective status through arcane procedures and mathematical formulae, but from their possibility for replication (De Vaus 2002:69; Kidder 1986:36). Thus transparency and access to primary data is essential. As argued in Chapter 5, digital archives, while of increasing importance for archaeology, museums and other institutions of cultural heritage, are likewise critically vital for accessing statistical databases which otherwise are often difficult or costly to access and often bear the transitory fate of the primary researcher her/himself. For analysis of the data itself, having the data displayed visually in the medium of the wiki allows rapid comparison of results within the multiple frames of the computer graphical user interface (GUI) (see Chapter 5). It is accessible at: http://humanitieslab.stanford.edu/Teotihuacan/Home.

All told, the responses to the 37 questions potentially present more than 700 possible cross-tabulations for analysis (for example, the responses to every question can be cross-tabulated with the responses to each of the other 36 questions). Factor analysis and making composite scale scores for the key explanatory concepts reduce this number somewhat. Nevertheless, it must be prefaced that many more analyses, informed by alternate research goals, could be asked of the data presented in this chapter. To make data explication reasonable it has been necessary to concentrate only on those relationships pertinent to identifying the major associations with Teotihuacan.

Forward to Design and Analytical Methods Synopsis

Containments

Return to Unidimensionality and Reliability