DO NOT DISSEMINATE Brookes Publishing | www.brookespublishing.com | 1-800-638-3775 | © 2009 | All rights reserved

ASQ-3 Technical Report

This report offers a range of technical information about the Ages & Stages Questionnaires ® : A Parent-Completed Child Monitoring System, Third Edition (ASQ®-3).development of the The Ages & Stages Questionnaires (ASQ®) system, including item selection and readability, are reviewed, as are the revisions that have been made to the questionnaires. Since publishing the second edition in 1999, new data have been collected on more than 18,000 questionnaires. These data have been used to examine selected psychometric parameters of the questionnaires. In addition to de-scribing the demographic characteristics of the samples, analyses included in this report address interobserver and test–retest reliability and measures of internal consistency. A comparison of questionnaire performance by groups of risk and nonrisk children is presented, as is the ration-ale for combining groups to derive the revised cutoff points for the ASQ-3. Validity analyses in-clude descriptions of how the cutoff points were determined and of measures of concurrent va-lidity. A final section presents a comparison between the English and Spanish versions of the questionnaires.

DEVELOPMENT OF THE ASQ

Item Selection

ASQ items were developed using a variety of sources, including standardized developmental tests, nonstandardized tests focused on early development, textbooks, and other literature con- taining information about early developmental milestones. Using these sources, the following criteria were used to develop items:

1. Skills were selected that easily could be observed or elicited by parents.
2. Skills were selected that were highly likely to occur in a variety of homes and child care settings.
3. Skills were selected that indexed important developmental milestones. Excerpted from ASQ¨-3 User's Guide by Jane Squires, Ph.D., Elizabeth Twombly, M.S., Diane Bricker, Ph.D., & LaWanda Potter, M.S. Brookes Publishing | www.brookespublishing.com | 1-800-638-3775 | © 2009 | All rights reserved

Once skills had been chosen, items were written using familiar, concrete words that did not exceed a sixth-grade reading level, and illustrations and examples were provided for as many items as possible. Using this process, a large pool of potential items was created. From this pool, the six items

Using this process, a large pool of potential items was created. From this pool, the six items that composed each developmental area (Communication, Gross Motor, Fine Motor, Problem Solving, and Personal-Social) for each age interval were selected. Item selection for each questionnaire interval was restricted by allowing only items that targeted a skill that occurred at the middle to low end of the developmental range for that particular chronological age interval (i.e., the developmental range of 75–100 was targeted). This range was chosen for two reasons. First, many standardized tests use 1.5–2.0 standard deviations below the mean as the lower end of the typical developmental range; therefore, it was reasoned that any child who was generally unable to perform items at a developmental quotient of 75–100 should be referred for further assessment. Second, it was reasoned that items above a developmental quotient of 100 would identify primarily children who were developing without problem, and, thus, the inclusion of such items would be of little help. By targeting a restricted developmental range of 75–100, it was possible to keep the questionnaires brief. To determine the developmental quotient for each item, the following formula was used:

To determine the developmental quotient for each item, the following formula was used:

\ \mathrm

The age equivalent was obtained from the source(s) of the item such as the Gesell (Knobloch, Stevens, & Malone, 1980), the Bayley Scales of Infant Development (Bayley, 1969, 2002, 2006), the Battelle Developmental Inventory (BDI; Newborg, Stock, Wnek, Guidubaldi, & Svinicki, 2004), and Developmental Resources: Behavioral Sequences for Assessment and Program Planning (Cohen & Gross, 1979). When sources varied, a developmental range was used. Table 1 contains the age equivalent and developmental quotient for each item by area for each of 20 questionnaires. As shown in Table 1, to the extent possible, each area has two items with developmental quotients of approximately 75, two items with developmental quotients of approximately 85, and two items with developmental quotients of approximately 100.

REVISIONS OF THE ASQ

The ASQ was designed for use with a range of parents and other caregivers (e.g., varying income and educational levels); therefore, the reading level was kept low, and illustrations and examples were added to clarify items when possible. To ascertain the reading level of the ASQ-3 questionnaires, the Flesch Reading Ease and Flesch-Kincaid Grade Level readability measures (Microsoft Word 2007) were used. Average readability grade levels of the ASQ-3 were 4.9 for Communication, 5.6 for Gross Motor, 5.3 for Fine Motor, 5.4 for Problem Solving, 4.7 for Personal-Social, and 4.7 for the Overall section.

Excerpted from ASQ¨-3 User’s Guide by Jane Squires, Ph.D., Elizabeth

Table 1. Age equivalent and developmental quotient of items by area for each questionnaire

(continued)

Table 1. (continued)

(continued)

Excerpted by Jane from Squires, ASQ¨-3 Ph.D., User’s Elizabeth Guide

Table 1. (continued)

Excerpted by Jane from Squires, ASQ¨-3 Ph.D., User’s Elizabeth Guide

First Revision

In response to validity and utility data gathered on the questionnaires (e.g., Brinker, Franzier, Lancelot, & Norman, 1989), the questionnaires were first revised in 1991. Six types of changes were made. First, a number of items were reworded to clarify meaning. These modifications were made based on feedback from project staff, interventionists, parents, nurses, and pediatricians using the questionnaires in clinic and research environments. In most cases, the modifications entailed minimal word changes. For example, “reach for a toy” was changed to “try to get a toy”; “couch or adult chair” was changed to “furniture”; and “being able to stop” was changed to “stopping.” In a few cases, examples were added or modified. For example, “Does your baby play ball with you by either rolling or throwing the ball to you?” was changed to “Does your baby either roll or throw a ball back to you so that you can return it to him?” For some items, examples were changed to more available household items to facilitate completion of the questionnaires by parents. For example, “toy” and “four objects like blocks or cars” were substituted for “block” in several instances. Second, modifications of a more extensive nature were made. In some cases, an item that

Second, modifications of a more extensive nature were made. In some cases, an item that was difficult to interpret was eliminated and replaced with another item. In all cases, the substituted items appeared on an ASQ at the previous or next age interval. For example, on the 20 month questionnaire, an item in the Fine Motor area was eliminated and replaced with an item from the Fine Motor area on the 24 month questionnaire. The third change made to the questionnaires was the elimination of items with develop-

The third change made to the questionnaires was the elimination of items with developmental quotients of 125–150. On the initial version of the questionnaires, each developmental area included one item with a developmental range of 125–150. These items were added to provide information on parents’ reported tendency to overestimate their children’s developmental status (cf. Gradel, Thompson, & Sheehan, 1981; Hunt & Paraskevopoulos, 1980). An analysis of parental responses to these items did not support parental overestimation of children’s developmental achievements, so these items were eliminated from the questionnaires. A fourth change was ordering the items within each developmental area according to level

A fourth change was ordering the items within each developmental area according to level of difficulty. Initially, items were not arranged in developmental order on the questionnaires; however, with this revision, the items in each developmental area were arranged according to level of difficulty, beginning with the easiest items and ending with the most advanced. A fifth modification was the addition of the 6, 18, and 48 month questionnaires. The 6 and

The first type of revisions focused on minor wording changes and deletions to increase the clarity of items. For example, qualifying words such as generally or usually were eliminated. The second category of revisions centered on minor modification of the questionnaire format to be more user friendly.

Second Revision

A fifth modification was the addition of the 6, 18, and 48 month questionnaires. The 6 and 18 month questionnaires were constructed by taking developmentally appropriate items from the adjacent questionnaires and adding items when necessary. The 48 month questionnaire was developed by examining a variety of tests and other developmental resources and constructing test items. The same criteria for the development of the previous questionnaires were applied to items for the 48 month questionnaire. Finally, the sixth type of revision entailed changing the name of the questionnaires from

Excerpted from ASQ¨-3 by Jane Squires,

Excerpted from ASQ¨-3 User’s Guide by Jane Squires, Ph.D., Elizabeth

Another type of revision involved adding eight new age intervals to the questionnaire system. From 1997 to 1998, additional intervals were completed at the 10, 14, 22, 27, 33, 42, 54, and 60 month age intervals. These intervals were added to make the ASQ series more comprehensive and to ensure that children could be screened using the ASQ at any age between 4 and 66 months.

Third Revision

The ASQ-3 was completed in 2008. For the ASQ-3, two additional questionnaire intervals were added to the ASQ series, making a total of 21 intervals across the 1- to 66-month age span. Second, the age range for administration of each questionnaire was modified so that children of any age could be continuously screened from 1 to 66 months. Administration age ranges for the ASQ-3 are listed in Table 2. Third, minor revisions were made to the existing 19 questionnaires. Fourth, additional questions were added to the Overall section to ask about behavioral concerns when applicable. Fifth, the Information Summary sheets for all intervals were revised, and a monitoring zone was added.

Addition of 2 and 9 Month Questionnaires

To assist programs in the screening of young children from birth, a 2 month ASQ-3 was developed, and data were gathered on its validity, reliability, and utility during a 2.5-year period. These data are reported in this technical report. Second, a 9 month ASQ-3 was developed primarily for use in pediatric settings, based on the American Academy of Pediatrics recommendations (2006) for screening at 9, 18, and 24 or 30 months. The 9 month ASQ-3 was derived from the items on the 10 month interval (i.e., identical items), with cutoff scores delineated for 9- month-old children (i.e., children from 9 months 0 days through 9 months 30 days). Data for the 9 month ASQ-3 are reported as appropriate in this technical report. For some analyses, the

Table 2. ASQ-3 age administration chart

aMay use the 9 or 10 month ASQ-3 with children in this age range.

Excerpted from ASQ¨-3 by Jane Squires,

9 and 10 month questionnaires are reported separately, whereas for other analyses, the 9 and 10 month questionnaire data are combined. When combined, there are 20 questionnaire intervals; when reported separately, there are 21 questionnaire intervals.

Revisions to Items

The items are the heart of the ASQ, and changes made were carefully considered and generally did not alter meanings. Item revisions such as minor rewording and inclusion of additional examples to items were made based on ASQ user feedback and statistical analyses. Statistical analyses included item response theory (IRT) modeling, in which mathematical models that scaled items according to the statistical probability of response to each item and a child’s ability to com-

items according to the statistical probability of response to each item and a child’s ability to complete the item were computed. Items that did not fit a developmental model were examined, and

plete the item were computed. Items that did not fit a developmental model were examined, and minor revisions were made to clarify items. Item changes were made across all developmental areas and age intervals (with the exception of the 2 month interval) and are of four types. The numbers of changes across questionnaire intervals are shown in Table 3. The most frequent type of change was wording adjustments to improve the clarity of items.

The most frequent type of change was wording adjustments to improve the clarity of items. For example, in the Personal-Social area, the item, “Can your child put on a coat, jacket, or shirt by himself?” was changed to “Does your child put on a coat, jacket, or shirt by himself?” In the Fine Motor area, the item, “Does your baby usually pick up a small toy with only one hand?” was changed to “Does your baby pick up a small toy with only one hand?” A second type of item revision involved deleting or adding examples that accompanied

A second type of item revision involved deleting or adding examples that accompanied items. For example, in the Gross Motor area, the item, “While standing, does your child throw a ball overhand by raising his arm to shoulder height and throwing the ball forward? (Dropping the ball, letting the ball go, or throwing the ball underhand does not count)” was changed to “While standing, does your child throw a ball overhand by raising his arm to shoulder height and

Table 3. Number of items per questionnaire with revisions Questionnaire Minor wording

throwing the ball forward? (Dropping the ball or throwing the ball underhand should be scored as ‘not yet’).” In the Fine Motor area, the item, “Does your child thread a shoelace through either a bead or an eyelet of a shoe?” was changed to “Can your child string small items such as beads, macaroni or pasta ‘wagon wheels’ onto a string or shoelace?” A third type of revision entailed changing illustrations accompanying items. For example,

A third type of revision entailed changing illustrations accompanying items. For example, the illustration in the 30 month Fine Motor area for item number 4 was deleted and replaced with the correct illustration for the item: “After your child watches you draw a line from one side of the paper to the other side, ask her to make a line like yours. Do not let your child trace your line. Does your child copy you by drawing a single line in a horizontal direction?” The final type of revision was the addition of one item to the Communication area to as-

The final type of revision was the addition of one item to the Communication area to assist in identifying children who may have delays in expressive language. On the 12 month questionnaire interval, the following item was added: “Does your baby make two similar sounds like ‘ba-ba,’ ‘da-da,’ or ‘ga-ga’? (The sounds do not need to mean anything).”

Revisions to Overall Section

The Overall section of the ASQ asks a series of general questions about children’s early development. In this section, two types of changes were made. First, changes in the wording of some existing questions were made to improve the clarity of the items. For example, in the 4 month through 14 month intervals, the question, “Does your baby use both hands equally well?” was changed to “Does your baby use both hands and both legs equally well?” Second, new questions were added to the Overall section. For example, on the 30 month

Second, new questions were added to the Overall section. For example, on the 30 month through 60 month intervals, “Can other people understand most of what your child says?” was added to help ensure that infants and children who might require further assessment were identified. A question about behavioral concerns was added on all intervals that was based on our research and other studies regarding the early identification of autism spectrum disorders (ASDs). Our research suggests that parents of young children often notice anomalies in their child’s behavior early on—in the first few months—far ahead of when professionals diagnose ASD in these children. Thus, asking specifically about concerns regarding a child’s behavior may alert professionals to parental concerns early on and assist in the early identification of ASDs and other developmental disorders. A summary of changes to the Overall questions on the ASQ-3 can be found in Table 4.

Table 4. Number and wording of new and revised items added to Overall section in the ASQ-3

Figure 1. Portions of the ASQ-3 Information Summary sheet, with monitoring zone shown in light gray shading.

Some sections on the Information Summary sheet were reordered to more closely follow the order of sections on the ASQ. Second, a monitoring zone (i.e., questionable area) was added to the summary profile of children’s scores. A lightly shaded area located just to the right of the cutoff points (i.e., representing scores that are 1 and 2 standard deviations from the mean) was included, as shown in Figure 1. It may be important to closely track the development of children whose ASQ-3 scores fall in this monitoring range and to provide parents/caregivers with activities to practice with these children.

PSYCHOMETRIC STUDIES OF THE ASQ

With the passage of the amendments to the Individuals with Disabilities Education Act (IDEA) of 1990 (PL 101-476) came a call for early detection of social or emotional problems in young children. As a complement to the ASQ, the Ages & Stages Questionnaires: Social-Emotional (ASQ:SE) was developed and published in 2002. This screening tool should be used in conjunction with the ASQ to identify children between 3 and 66 months of age who may need in-depth assessments of their social and emotional behavior. Eight questionnaires are available (in either English or Spanish) that address seven behavioral areas: self-regulation, compliance, communication, adaptive functioning, autonomy, affect, and interaction with people. An accompanying User’s Guide also is available to assist professionals in the use of the ASQ:SE questionnaires, as is a DVD, ASQ:SE in Practice (Squires, Twombly, & Munkres, 2004).

This section presents a range of empirical information collected on the questionnaires since 2004. These data were used to guide the ASQ-3 revisions. The data include 18,572 completed questionnaires for children between 1 and 66 months of age. This entire data set was used to derive new cutoff scores for the questionnaires. Subsamples of the data set were used to examine reliability and validity of the questionnaires. The respective ns are provided for each analysis.

Data Collection Procedures

Revisions for the ASQ-3 were based on 18,572 questionnaires completed by parents of children between 1 and 66 months of age. The numbers of questionnaires by interval are shown in Table 5 along with methods of completion (paper and web based). Questionnaire data were collected using two methods: 1) completion of paper question-

Questionnaire data were collected using two methods: 1) completion of paper questionnaires and 2) completion of online, web-based questionnaires. Data were gathered between January 2004 and June 2008, across 20 ASQ intervals, as shown in Table 5. Paper questionnaires (52.4%) were completed by parents from an array of community-based programs. Web-based questionnaires (47.6%) were completed by parents who logged onto the ASQ research web site and completed demographic and research forms and questionnaires. The online questionnaires had wide geographic distribution, with the sample representing families from all 50 states and several U.S. territories.

Paper Questionnaire Completion

Paper questionnaires were completed by parents whose children attended programs for young children, including child care centers, preschools, infant programs, nonprofit organizations serving young children, medical offices conducting well-child screening, Head Start and Early Head Start programs, Healthy Start programs, home visiting programs, nonprofit organizations such as the Urban League and United Way, and IDEA Child Find programs throughout the United

Excerpted from ASQ¨-3 by Jane Squires,

by Jane Squires, Ph.D., Elizabeth

States. In addition, questionnaires were completed as part of large-scale screening projects for monitoring and identifying developmental delays in young children (e.g., at-risk monitoring projects in Idaho; Head Start and Migrant Head Start in Oregon, California, Ohio, New York, and Washington; subsidized child care programs in Florida). For the paper questionnaires, recruit-

For the paper questionnaires, recruitment procedures included inquiries made by the ASQ-3 research staff to 1) EI/ECSE programs in Washington, Oregon, Hawaii, and California and 2) Healthy Start, nurse home visiting, child care, and Head Start/Early Head Start programs in Ohio, Washington, California, Colorado, Minnesota, and Florida. As part of providing services, parents received a questionnaire from the provider along with a form asking for the child’s demographic information and a research consent form. The questionnaire was completed either independently by the parent or with assistance from providers. The completed questionnaires were usually scored by the provider,

assistance from providers. The completed questionnaires were usually scored by the provider, and the results were shared with the parent or caregiver. The provider assisted the parent with referrals to community evaluation services as needed. Either hard copies of the questionnaires or deidentified computer files were sent to the research site and entered into an ASQ database. Procedures ensuring protection of human participants were approved by the University of Oregon institutional review board and were followed in all research phases.

Web-Based Questionnaire Completion

With the premise of a parent-friendly tool, the mediated ASQ research web site was designed using best practices of recruitment, data collection, and data management. The web site was produced in a hypertext markup language form that used PHP: Hypertext Preprocessor version 4.4.3 scripting to process and save data. It was tested through a variety of Windows- and Maccompatible web browsers (i.e., Internet Explorer, Netscape, Firefox, Safari). After consenting to participate in the online completion of the questionnaires, parents pro- completion of the questionnaires was posted on sites such as http://www.daycareresource.com and http://www.craigslist.com.

Paper and Web-Based Questionnaire Completion Comparison

To test variations between the web-based and paper versions of the ASQ, statistical analyses including IRT modeling were used (for a full research report, see Yovanoff, McManus, & Squires, 2009). IRT involves mathematical models that statistically characterize the probability of response to each item in a test and the participant’s ability to endorse the item. Such probabilistic response to the item is depicted by an item response function (i.e., item characteristic curve). The item characteristic curve characterizes one item from other items with three location parameters: 1) a, item discrimination; 2) b, item difficulty; and 3) c, guessing—the probability of correctly endorsing the item. IRT models include one-, two-, or three-parameter logistics models (Embretson & Reise, 2000; Ferrando & Lorenzo-Seva, 2005; Fraley, Waller, & Brennan, 2000). The Rasch model, a one-parameter logistic model, was applied in studying differences

The Rasch model, a one-parameter logistic model, was applied in studying differences (i.e., differential items functioning, or DIF) between web-based completed and paper completed questionnaires. The one-parameter logistic model contains only one item parameter— b, difficulty—which is free to vary between groups. It was used to examine the item response of the web-based group (focal group) in relation to that of the paper group (reference group). DIF occurs when the groups at the same ability level differ in their likelihood of endorsing an item. The WINSTEPS Rasch Measurement version 3.64.2 computer program (Linacre, 2007)

The WINSTEPS Rasch Measurement version 3.64.2 computer program (Linacre, 2007) was used to analyze the questionnaire data. WINSTEPS DIF statistics imply the following:

1. web and paper groups represent the same ability—ability constant—on the same scale, and
2. item calibrations have been made at the item and ability levels. The implementation of a statistical test with a p value of .01 indicated items displaying DIF. Out of 570 questionnaire items, the statistical findings indicated that only 60 items

tistical test with a p value of .01 indicated items displaying DIF. Out of 570 questionnaire items, the statistical findings indicated that only 60 items (10.5%) exhibited significant DIF when comparing the web-based (focal group) and paper (referent group) groups. DIF items were equally spread among all intervals (for this analysis, the 19 intervals from 4 to 60 months were used—the 2 and 9 month intevals were still under development), and, within each developmental area, they did not show all positive or negative t values, indicating no consistent pattern of differences between the same items on the web-based and paper questionnaires. Several factors may explain the differences between web-based and paper questionnaire

Several factors may explain the differences between web-based and paper questionnaire completion of these 60 items. First, the stationary location of computers may have interfered with parents’ ability to directly observe their children demonstrating skills. Also, the location of online connections may have posed further mobility limitations. In contrast, the paper method allowed parents to go to their children either for direct observation or for hands-on activity to address specific items. Second, it is possible that parents or caregivers navigated the web-based ASQ when their

by Jane Squires, Ph.D., Elizabeth

Table 6. Gender of children

Overall, the IRT analyses found few significant differences between web-based and papercompleted questionnaires. Only 10% of items (60 items out of 570 total) seemed to function differently when these two completion methods were compared. Differences also were both positive and negative, indicating no consistent pattern between the completion methods. Therefore, web-based and paper questionnaire data were combined for all 21 questionnaire intervals to derive the cutoff scores.

Population Sample

The data analyses that are contained in this report are based on 18,572 completed questionnaires. However, within this sample, 3,434 children had more than one completed questionnaire (e.g., 4, 8, and 12 months); therefore, the total demographic sample was 15,138 individual children. Each parent or caregiver who completed a questionnaire was asked to complete a demo-

Each parent or caregiver who completed a questionnaire was asked to complete a demographic form. Demographic data included information on gender, ethnicity, mother’s education, family income, who had completed the questionnaire, and whether the child was known to have any medical or environmental risk. The demographic data for the population sample are displayed in Tables 6–11. As shown in Table 6, the gender distribution for the sample was 53% male and 47% female.

As shown in Table 6, the gender distribution for the sample was 53% male and 47% female. The distribution of mother’s level of education is displayed in Table 7. The greatest percentage of mothers in this sample (54%) had at least 4 years of college, whereas 12% had an associate’s degree, 23% had a high school education, and only 3.5% had not completed high school. Data on family income were collected and are displayed in Table 8. The majority of the re-

Data on family income were collected and are displayed in Table 8. The majority of the reporting caregivers indicate incomes greater than $40,000 (57%), whereas 36% reported incomes below that figure, and 7% reported not knowing. Table 9 contains data on the person completing the questionnaires. The majority of indi-

Table 7. Level of mother’s education

Table 8. Family income level

Data on the risk status of each child were collected and are displayed in Table 10. Seventysix percent of the sample had only one or no known risk factor, whereas 19% had two risk factors, and 4% had three or more known risk factors; there are missing data for 4% of the sample. The number of children with two or more risk factors may seem high; however, it may be that children exposed to risk conditions may be referred for screening more often than children who are not. Table 11 contains data on the ethnicity of the sample. The greatest percentage of children

Table 11 contains data on the ethnicity of the sample. The greatest percentage of children were white (66%); 12% were African American, and 15% were Hispanic. Asian, Native American, Hawaiian, Pacific Islander, other, and mixed ethnicity composed 5% of the sample. Table 11 also contains data from the U.S. Census estimates for 2007. These comparisons suggest that the ethnicity of this sample of children is representative of the general U.S. population. Table 12 contains information on the number of questionnaires completed for each child

Table 12 contains information on the number of questionnaires completed for each child in the sample. One questionnaire was completed for 86.5% of the children, 8.5% had two questionnaires completed, and the remaining 5% of the sample had three or more questionnaires completed.

Reliability Studies

Table 9. Person completing ASQ-3

Reliability studies completed on the ASQ-3 include test–retest reliability and interobserver reliability. In addition, internal consistency of ASQ-3 items was examined using correlational analyses and the Cronbach coefficient alpha (Cronbach, 1951). Each of these analyses is presented next.

by Jane Squires, Ph.D., Elizabeth

Table 10. Risk status of child

Table 11. Ethnicity of children in sample compared with 2007 U.S. Census estimates

Table 12. Number of questionnaires completed for each child

aSource: Population Division, U.S. Census Bureau, May 1, 2008.

by Jane Squires, Ph.D., Elizabeth Twombly, M.S., Diane Bricker, Ph.D., & LaWanda Potter, M.S.

Test–Retest Reliability

Test–retest reliability is designed to help determine the stability of test outcomes over time. Test–retest reliability of the ASQ was examined by comparing two questionnaires completed by the same parent at a 2-week time interval. That is, parents were asked to complete the same questionnaire interval for their child twice within a 2-week time period between completions. Questionnaires completed by 145 parents were included in this analysis. Forty-two parents completed two questionnaires online, and 103 parents completed two paper questionnaires. Parents were blind to the results of the first questionnaire when they completed the second one. The two questionnaires completed by parents were then compared for agreement on classifications (i.e.,

questionnaires completed by parents were then compared for agreement on classifications (i.e., screened or not screened). The percent agreement for the 145 parents was 92%. Intraclass correlations ranged from .75 to .82, indicating strong test–retest reliability across ASQ developmen-

Interobserver Reliability

Interobserver reliability refers to the agreement of test outcomes that have been completed by at least two independent test administrators. The interobserver reliability of the ASQ was examined by comparing questionnaires completed by parents with questionnaires completed by trained test examiners for the same children. Trained test examiners filled out a questionnaire on a child immediately after completing a standardized assessment (e.g., BDI). Interobserver reliability was derived by comparing the agreement between the classifications (i.e., screened or not screened) of 107 children based on the parents’ and trainer examiners’ completion of ASQ. The percent agreement between ASQ classifications between parents and trainer examiners was 93%. Intraclass correlations by area ranged from .43 to .69, suggesting robust agreement between parents and trained examiners when completing the ASQ for this group of 107 children. The Personal-Social area had the strongest agreement (.69), and the Communication area had the lowest agreement (.43). Parents and professionals may observe different samples of behavior while completing the Communication Area, thus accounting for fair intraclass correlations between parents and test administrators.

Internal Consistency

Cronbach coefficient alphas were calculated for developmental area scores for 20 age intervals. Alphas are presented in Table 15 and range from .51 to .87. These alphas indicate that ASQ items have good to acceptable internal consistency. The reliability of the questionnaires has been studied by examining the internal consistency,

The reliability of the questionnaires has been studied by examining the internal consistency, test–retest reliability, and interobserver reliability of the questionnaires. Internal consistency

Pearson product moment correlation coefficients were calculated for developmental area scores with an overall ASQ score for 20 questionnaire age intervals. As shown in Table 13, the correlations by developmental area and overall ASQ score are consistent and generally range from .60 to .85. The one exception is the Gross Motor area, in which two correlations are below .60. All correlations are significant at p < .01. These findings suggest moderate to strong internal consistency between developmental areas and total test score. Table 14 contains the correlations between developmental area scores that have been col-

Table 13. Correlations between developmental area and overall ASQ-3 score

9^{\mathrm{a}}

10^{\mathbf{a}}

Note: All correlations are significant at p < .01. Total number of questionnaires completed = 18,572. a9 month interval data are a subset of 10 month data.

analyses have indicated strong relationships across items and within areas on the questionnaires. The questionnaires also achieved substantial test–retest and interobserver reliability. Parents’ evaluations of their children using the questionnaires were consistent over time. In addition, professional examiners’ agreement with parental evaluations of children on the questionnaires was consistently high.

Validity

Studies of the validity of the ASQ-3 are described here, beginning with a comparison of performance on the questionnaires by nonrisk and risk groups. The next section describes the procedures used to determine the screening cutoff points for each interval. This section is followed by de-

p<.01

Note: All correlations are significant at p < .01.

Excerpted from ASQ¨-3 by Jane Squires,

Table 15. Standardized alphas by area and age interval

Note: Analyses include only questionnaires with no missing items.

tailing the investigation of concurrent validity. The final section addresses a preliminary comparison between the English- and Spanish-language versions of the ASQ-3.

Figure 2. Risk and nonrisk samples for the Communication area.

Nonrisk and Risk Groups

A sample of 18,572 questionnaires for children between the ages of 1 and 66 months (shown in Table 5) was used to determine the cutoff scores for the ASQ-3. The demographic information on this sample is presented in Tables 6–11. This sample contains both nonrisk and risk children. Subjects in the risk sample included infants and young children from families who met one or more of the following criteria: 1) extreme poverty (according to family income level, as defined

Communication

Figure 3. Risk and nonrisk samples for the Gross Motor area.

by federal guidelines, 100% poverty level); 2) maternal age of 19 years or younger at the time of the infant’s birth; 3) maternal education less than 12th grade; 4) parents who had experienced involvement with child protective services for abuse and/or neglect of their children; 5) medical risk, including prematurity ( 39 weeks’ gestation); and 6) infant’s birth weight less than 3 pounds, 5 ounces. As noted in Table 10, 19% of this sample had two or more risk factors, and 4% had three

As noted in Table 10, 19% of this sample had two or more risk factors, and 4% had three or more known risk factors. Figures 2–6 contain graphic comparisons of the mean scores by de-

velopmental area for the nonrisk, risk, and combined samples. As expected, an examination of these graphs shows that the means for the risk group were generally, but not always, lower than the means for the nonrisk group. Some anomalies did occur. For example, at the 2 month age

the means for the nonrisk group. Some anomalies did occur. For example, at the 2 month age interval, the risk group had consistently higher means than the nonrisk group did; however, this is likely attributable to the small number of risk infants at this age interval (n 14 month interval, large mean differences in favor of the nonrisk group occurred, which, again, may be a function of a small number of risk children in this interval (n From studies on the second edition of the ASQ (Squires, Potter, & Bricker, 1999), it was

the means for the nonrisk group. Some anomalies did occur. For example, at the 2 month age interval, the risk group had consistently higher means than the nonrisk group did; however, this is likely attributable to the small number of risk infants at this age interval (n 5). Also, at the 14 month interval, large mean differences in favor of the nonrisk group occurred, which, again, may be a function of a small number of risk children in this interval (n 52). From studies on the second edition of the ASQ (Squires, Potter, & Bricker, 1999), it was

Figure 4. Risk and nonrisk samples for the Fine Motor area.

From studies on the second edition of the ASQ (Squires, Potter, & Bricker, 1999), it was determined that including both the risk and nonrisk samples was more representative of a gen-

Figure 5. Risk and nonrisk samples for the Problem Solving area.

eral population and provided the most accurate cutoff scores (Squires et al., 1999). The method used to test this question was an analytic technique called relative (or receiver) operating characteristic (ROC). The ROC, based on statistical decision theory, has been used in a variety of disciplines, including human perception and decision making (Green & Swets, 1966). The ROC provides estimates of the probabilities of decision outcomes by revealing the reciprocal relationship between the true positive, true negative, false positive, and false negative probabilities that can be attained by shifting the decision criteria (i.e., cutoff points). Based on the reported range of ROCs by group, it was determined that points derived by

Figure 6. Risk and nonrisk samples for the Personal-Social area.

can be attained by shifting the decision criteria (i.e., cutoff points). Based on the reported range of ROCs by group, it was determined that points derived by using means and standard deviations from the combined risk and nonrisk groups provided the most accurate cutoff scores (Squires et al., 1999). This decision had practical implications because agencies responsible for screening often do not know the risk status of the population to be screened. Adopting the combined risk and nonrisk referral cutoffs seemed the most appropriate for screening programs.

Determining Cutoff Scores

For the ASQ-3, risk and nonrisk groups were combined for all analyses and determination of age interval cutoff scores. The challenge, as with all screening measures, was to select scores that maximized accuracy and minimized error. This challenge was addressed in two ways. The first strategy was to develop a matrix showing the conditional probabilities that existed at each cutoff score for 2, 1.5, and 1 standard deviations and to generate a ROC curve that displayed the probabilities at each cutoff score. A matrix for each questionnaire interval was created using the cutoff scores at 2, 1.5, and 1

A matrix for each questionnaire interval was created using the cutoff scores at 2, 1.5, and 1 standard deviations from the mean. This matrix included several computations that were generated using a contingency table. Cutoff scores for developmental areas were included, along with conditional probabilities that were computed. A sample matrix for the 48 month questionnaire can be found in Table 16 with the following conditional probabilities: 1) sensitivity, 2) specificity, 3) false positive proportion, 4) false negative proportion, 5) underidentification, and 6) overidentification. (For definitions and computational formulas, see Chapter 6.) Table 17 shows combined conditional probabilities across the intervals. As expected, when

Table 17 shows combined conditional probabilities across the intervals. As expected, when the cutoff became less conservative (i.e., 1.5 or 1 standard deviations from the mean), the overreferral rate increased as the underidentification rate decreased. The cutoff score of 2 standard deviations, although not perfect, seemed the most balanced cutoff point in terms of the true positive and false positive proportions. The second strategy entailed determining the percentage of children identified at each of

The second strategy entailed determining the percentage of children identified at each of the cutoff scores that were 2, 1.5, and 1 standard deviations below the mean. Targets of 12%–16% of children identified in one developmental area (i.e., one area below the cutoff score) and 2%–7% identified in two or more areas were adopted as the desired percentages to be identified for further assessment at each age interval. These figures were based on U.S. Census Bureau and Centers for Disease Control and Prevention prevalence data related to developmental disabilities in young children (Cornell University, 2003–2009; U.S. Census Bureau, 2004).

Note: Values are cutoff points by standard deviation units and accompanying conditional probabilities for the 48 month questionnaire (n = 45 for validity analyses).

Table 17. Conditional probabilities across questionnaire intervals by cutoff point for all questionnaires (n = 579 for validity analyses)

For every questionnaire interval, ROC analyses and percentages of children identified in one and two domains were studied. Based on these comparisons, a referral cutoff point of 2 standard deviations below the mean across all 21 questionnaire intervals was chosen.

Monitoring Zone

For the ASQ-3, a monitoring zone of 1–2 standard deviations below the mean score in each domain was highlighted on the ASQ-3 Information Summary sheet forms, as shown previously in Figure 1. This monitoring zone was added to alert ASQ-3 users that children who are not identified as needing follow-up assessment (i.e., with scores that were 2 standard deviations below the mean) might, nevertheless, benefit from targeted interventions. Children whose scores are 1–2 standard deviation units below the mean score in any developmental area should be monitored and given follow-up activities for practicing skills in these areas. In addition, these children should be rescreened at regular intervals. The monitoring zones and cutoff scores are presented in Table 18. (Scores that fall within the monitoring zone are 1 but 2 standard deviations from the mean.)

Concurrent Validity

Concurrent validity was measured by comparing the classification of children based on their performance on a standardized test with their classification based on their performance on the ASQ-3. Agreement meant that the ASQ-3 had assigned a child to the same classification as the standardized test had; disagreement meant that the ASQ-3 classification did not match the standardized test’s classification. Two groups of children were included in this analysis: those tested for eligibility for IDEA

Two groups of children were included in this analysis: those tested for eligibility for IDEA services and those not receiving services and presumed to be developing without problems— hereafter called the typical group. The identified group (n 257) participated in EI/ECSE programs in Oregon, New York, and California, and the typical group (n 322) was recruited from child care centers, preschool programs, and Internet advertising for research participants. In most cases, the standardized measure used was the BDI, first and second editions (Newborg, Stock, Wnek, Guidubaldi, & Svinicki, 1984, 2004). The BDI was administered and scored by trained examiners. For the identified group, test data were gathered from children’s files and included scores and

by Jane Squires, Ph.D., Elizabeth

Table 18. ASQ-3 means, standard deviations, and cutoff scores

b Brookes DO Publishing NOT DISSEMINATE | www.brookespublishing.com

DO Publishing NOT DISSEMINATE

Excerpted 10 12 by 14 16 Jane 18 20 from 22 24 Squires, 27 ASQ¨-3 30 33 36 42

Excerpted by Jane 18 from Squires, ASQ¨-3 Ph.D., User’s Elizabeth Guide established by many states for entrance into EI/ECSE programs (http://www.NECTAC.org). (Although 75 is 2.5 points less than the BDI 1.5-SD cutoff score of 77.5, 75 was the mean cutoff score used for BDI eligibility decisions.) A child’s performance on the ASQ was considered identified if his or her score was at or

A child’s performance on the ASQ was considered identified if his or her score was at or below the cutoff score of 2 standard deviations below the mean in one or more developmental areas. One of the following four outcomes was possible:

1. Both tests classified the child as typical.

2. Both tests classified the child as identified.

3. Both tests classified the child as identified.

4. The standardized measure classified the child as typical, and the questionnaire classified him

5. The standardized measure classified the child as typical, and the questionnaire classified him or her as identified.

6. The questionnaire classified the child as typical, and the standardized measure classified him

or her as identified. 4. The questionnaire classified the child as typical, and the standardized measure classified him or her as identified.

Agreement between the BDI and ASQ-3 classifications for the total group of 579 children across all questionnaires is shown in Figure 7. An examination of these data suggests that the ASQ-3 has moderate to high agreement with BDI classifications. These findings are important because they provide objective evidence that, in most cases, ASQ-3 results will accurately identify children who require further assessment but will not identify those who are developing typically as needing further assessment. Users of the ASQ-3 can be relatively confident that ASQ-3 results will identify those children whose development is suspect and those whose development is falling within typical developmental norms. Contingency tables containing agreement between the BDI and the ASQs by combined age

Contingency tables containing agreement between the BDI and the ASQs by combined age intervals (2–12, 14–24, 27–36, and 42–60 months) can be found in Figure 8. Age intervals are combined for this report to simplify the reporting of results because of the numerous intervals in the ASQ-3. Contingency tables showing validity agreement for the 20 separate intervals (9 and 10 month intervals were combined) can be found at http://www.agesandstages.com.

Figure 7. Overall concurrent validity for aggregated 20 questionnaire intervals (9- and 10-month intervals were combined for this analysis).

Status determined by ASQ-3

Extensive review and revision have been undertaken in the development of the Spanish translation of the ASQ-3. The Spanish version of the second edition of the ASQs was reviewed by several experts in pediatrics and also by developmental pediatricians and practitioners working with

Eligibility status

2–12 months

Eligibility status

Status determined by ASQ-3

14–24 months

Eligibility status

Status determined by ASQ-3

27–36 months

Eligibility status

Status determined by ASQ-3

Status determined by ASQ-3

Figure 8. Agreement between ASQ-3 and eligibility status, collapsed by age grouping.

42–60 months

Eligibility status

young children and families who speak numerous Spanish dialects (e.g., Nicaraguan, Mexican, Argentinean). Translation errors that were found in the Spanish second edition of the ASQ were corrected, and minor wording changes and substitutions were made. The experts suggested these changes and revisions to develop a translation that could be used across a variety of Spanish dialects. Items also were reworded to match the changes in items and format made in the ASQ-3 English version. To date, analyses suggest similar cutoff scores when comparing the Spanish risk and English risk samples, with a few exceptions; differences appeared in both positive and negative directions. Data will continue to be gathered on the ASQ-3 and will be posted at www .agesandstages.com.

CONCLUSION

This report describes the latest revisions that have been made to the ASQ and also presents the most recent empirical information gathered on the questionnaires. The second edition of the ASQ appeared in 1999, and during the ensuing years, the authors and research staff have accumulated information that suggested minor changes to the questionnaires would improve their accuracy and functionality. These changes have necessitated the collection of additional data on the psychometric characteristics of the modified questionnaires. Examining the validity and reliability of a screening measure is essential to understanding

Examining the validity and reliability of a screening measure is essential to understanding and appreciating its strengths and weaknesses. All screening measures make errors; however, users should have confidence in the measure’s accuracy (i.e., underidentification and overidentification of children is low). The data present in this report should provide the user with the assurance that in most cases, children will be accurately screened and that the screening can be conducted at a modest cost. The ASQ system is built on the premise that early identification is essential to maximally

Examining the validity and reliability of a screening measure is essential to understanding and appreciating its strengths and weaknesses. All screening measures make errors; however, users should have confidence in the measure’s accuracy (i.e., underidentification and overidentification of children is low). The data present in this report should provide the user with the assurance that in most cases, children will be accurately screened and that the screening can be The ASQ system is built on the premise that early identification is essential to maximally

American Academy of Pediatrics. (2006). Identifying infants and young children with developmental disorders in the medical home: An algorithm for developmental surveillance and screening. Pediatrics, 118(1), 405–420. Available online at http://aappolicy.aappublications.org/cgi/content/full/pediatrics;118/1/405 Bayley, N. (1969). Bayley Scales of Infant Development. San Antonio, TX: The Psychological Corporation. Bayley, N. (2002). Bayley III Scales of Infant Development. San Antonio, TX: The Psychological Corporation. Bayley, N. (2006). Bayley Scales of Infant Development (3rd ed.). San Antonio, TX: The Psychological Corporation. Brinker, R., Franzier, W., Lancelot, B., & Norman, J. (1989). Identifying infants from the inner city for early intervention. Infants and Young Children, 2(1), 49–58. Cohen, M., & Gross, P. (1979). Developmental resources: Behavioral sequences for assessment and program planning (Vols. I–II). New York: Grune & Stratton. Cornell University. (2003–2009). Disability statistics. Retrieved from http://www.ilr.cornell.edu/edi/ disabilitystatistics/links.cfm Cronbach, L. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. Embretson, S., & Reise, S. (2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum Associates.

REFERENCES

conducted at a modest cost. The ASQ system is built on the premise that early identification is essential to maximally effective intervention with young children and their families. A critical feature of early identification is universal and ongoing developmental screening of young children. The ASQ provides practitioners and researchers a measure that is low cost to use and reliable in its identification of children who require further in-depth assessment.

Ferrando, P., & Lorenzo-Seva, U. (2005). IRT-related factor analytic procedures for testing the equivalence of paper-and-pencil and internet-administered questionnaires. Psychological Methods, 10(2), 193–205. Fraley, R., Waller, N., & Brennan, K. (2000). An item-response theory analysis of self-report measures of adult attachment. Journal of Personality and Social Psychology, 78(2), 350–365. Gradel, K., Thompson, M., & Sheehan, R. (1981). Parental and professional agreement in early childhood assessment. Topics in Early Childhood Special Education, 1(2), 31–39. Green, D., & Swets, J. (1966). Signal detection theory and psychophysics. New York: Wiley. Hunt, J., & Paraskevopoulos, J. (1980). Children’s psychological development as a function of the accuracy of their mothers’ knowledge of their abilities. Journal of Genetic Psychology, 136(2), 285–298. Individuals with Disabilities Education Act (IDEA) of 1990, PL 101-476, 20 U.S.C. §§ 1400 et seq. Knobloch, H., Stevens, F., & Malone, A.F. (1980). Manual of developmental diagnosis: The administration and interpretation of the revised Gesell and Amatruda developmental and neurological examination. New York: Harper & Row. Linacre, J.M. (2007). WINSTEPS (Version 3.64.2) [Computer software]. Chicago: Winsteps.com. Newborg, J., Stock, J.R., Wnek, L., Guidubaldi, J., & Svinicki, J. (1984). Battelle Developmental Inventory. Itasca, IL: Riverside. Newborg, J., Stock, J., Wnek, L., Guidubaldi, J., & Svinicki, J. (2004). Battelle Developmental Inventory. Chicago, IL: Riverside Publishing. Squires, J., Potter, L., & Bricker, D. (1999). Ages and Stages Questionnaires user’s guide (2nd ed.). Baltimore: Paul H. Brookes Publishing Co. Squires, J., Twombly, E., & Munkres, A. (2004). ASQ:SE in practice [DVD]. Baltimore: Paul H. Brookes Publishing Co. U.S. Census Bureau. (2004). American Community Survey. Available online at http://www.census.gov/ acs/www/index.html Yovanoff, P., McManus, S., & Squires, J. (2009). Web-based and paper-pencil administration of a developmental questionnaire for young children: Score interpretation using item response theory. Available from University of Oregon, Early Intervention Program, Eugene.

DO NOT DISSEMINATE Brookes Publishing | www.brookespublishing.com

Excerpted from ASQ¨-3 User’s Guide by Jane Squires, Ph.D., Elizabeth