phb-asqse2-user01.indd

ASQ:SE-2 Technical Report

Technical Report

Jane Squires, Diane Bricker, Elizabeth Twombly,
Kimberly Murphy, and Rob Hoselton,

This technical report focuses on the development and study of the psychometric properties of the Ages & Stages Questionnaires®: Social-Emotional, Second Edition (ASQ:SE-2), including ™ revisions and additions to the first edition of ASQ:SE. Psychometric studies completed on

revisions and additions to the first edition of ASQ:SE. Psychometric studies completed on ASQ:SE-2 are also described in detail, including results across the nine questionnaire intervals. The first section addresses the development of the first edition, including item selection and revision, and is followed by a description of the iterative process used for developing, revising, and studying ASQ:SE-2. The next section describes ASQ:SE-2 data collection procedures, including participant recruitment, as well as procedures for the validity and reliability studies. Next, this technical report offers an overview of the demographic characteristics of the ASQ:SE-2 normative sample. The following section reports psychometric findings, including internal consistency, test– retest reliability, and convergent and known group validity. Finally, findings on the utility of ASQ:SE-2 are reported.

DEVELOPMENT OF ASQ:SE

Development of ASQ:SE

Excerpted from ASQ:SE-2ª Users

182 ASQ:SE-2 Technical Report ASQ:SE items initially were developed using a variety of sources, including standardized social-emotional and developmental assessments, research studies, textbooks and other resources in developmental and abnormal psychology, education and intervention resources,

and language and communication materials. We ensured that items are as follows:

1. Representative of critical adaptive and maladaptive behaviors at targeted age intervals

2. Representative of critical adaptive and maladaptive behaviors at targeted age intervals

3. Easy for parents/caregivers to understand and recognize

4. Appropriate for a variety of cultural groups and families Each item was written using common words that did not exceed a sixth-grade reading level. When possible, quantitative descriptors (e.g., 15 minutes, within 24 hours) and concrete examples (e.g., smiles, kicks, bites other children) were provided to assist with interpretation of

the item meanings.

Iterative Item Development and Testing Items were assembled into a field test version, which was titled the Behavior-Ages & Stages Questionnaires (B-ASQ; Bricker, Squires, Twombly, Yockelson, & Kim, 1996). The field test version contained seven age intervals. The number of items per interval varied from 21 items at 6 months to 33 items at 48 months. The items in this field test version were reviewed by experts in psychology, psychiatry, education, early childhood development, pediatrics, nursing, and mental health. Experts provided feedback on the items regarding their appropriateness, ease of

mental health. Experts provided feedback on the items regarding their appropriateness, ease of understanding, scoring format, and content validity. Concurrently, practitioners in approximately 50 programs across the United States used the B-ASQ with a diverse population of young children and parents, including families served by Healthy Start in Hawaii and Oregon; inner-city families in Cincinnati, Ohio, Portland, Oregon, and San Francisco, California; Head Start and Migrant Head Start families in California, Texas, and Washington; Child Development and Rehabilitation Center professors in Eugene, Oregon; and families with young children identified with social-emotional delays in Arizona, California, Oregon, Utah, and Washington. Utility surveys completed by service providers and parents offered feedback on the clarity of item meaning, appropriateness of items,

DEVELOPMENT OF ASQ:SE-2 Beginning in late 2009, we began the process of revising and updating ASQ:SE based on data and comments from professionals and parents who had used the tool for more than 6 years. Early intervention program faculty, staff, and doctoral students again delved into an updated literature base on social-emotional development, including studies documenting the rise in the awareness and prevalence of autism spectrum and related disorders in young children,

DEVELOPMENT OF ASQ:SE-2

viders and parents offered feedback on the clarity of item meaning, appropriateness of items, missing content, and suggestions for revisions or additional items. The B-ASQ was revised and renamed the Ages & Stages Questionnaires: Social-Emotional (ASQ:SE) based on the input gathered from experts, parents, and practitioners and on preliminary data analyses. Revisions included eliminating or combining items with overlapping and similar content; rewording items that were difficult to understand; adding items to fill content gaps (e.g., items were added to target so-called “red flags” for autism and adding a questionnaire

ASQ:SE-2 Technical Report delays and disabilities in the preschool population. We began to develop and revise items for the second edition of ASQ:SE with a focus on capturing symptoms related to autism and communication disorders and developing additional items to identify competence and problem behaviors in infants and toddlers. Our main goals for the second edition revisions included the

following:

1. Extend the age range of ASQ:SE in both directions— to include infants from 1 month to facilitate enrolling and monitoring children from birth, and to expand items to cover children’s behavioral repertoire to age 6 (i.e., 72 months) when children generally enter

children’s behavioral repertoire to age 6 (i.e., 72 months) when children generally enter school settings 2. Develop new questionnaire items that directly target social-emotional, early communication, regulatory, and autism spectrum disorder (ASD) behaviors, and do so for children at

tion, regulatory, and autism spectrum disorder (ASD) behaviors, and do so for children at a younger age 3. Update the format, including response options, to enhance the user friendliness of

3. Update the format, including response options, to enhance the user friendliness of ASQ:SE

4. Update the normative sample to reflect the current, more diverse population of young

5. Update the normative sample to reflect the current, more diverse population of young children in in the United States

6. Incorporate the addition of a monitoring zone similar to that used in ASQ-3 to aid in

score interpretation and follow-

Extended Age Range Extending the ASQ:SE-2 age windows included testing and modifying the 6 month interval to include infants as young as 1 month of age, as well as expanding the upper age range of the

60 month interval to include children to age 6 years.

Development of the 2 Month Age Interval The 2  month questionnaire was developed in response to requests from program personnel monitoring the development of newborns and older infants, who wanted an initial ASQ:SE to give parents soon after the birth of their children. With a 2 month questionnaire, professionals would be able to screen those infants whose parents had great concerns shortly after

is hungry, tired, or uncomfortable? For example, does she fuss or cry?”

sionals would be able to screen those infants whose parents had great concerns shortly after birth. Analyses indicated that infants 1– 2 months of age were inconsistently performing several items on the ASQ:SE-2 6 month questionnaire. A 2 month questionnaire for children from 1 to 3 months was developed and was composed of 16 scored items from the 6 month questionnaire, which were revised to be more developmentally appropriate. For example, “Does your baby let you know when she is hungry or sick?” was revised to “Does your baby let you know when she

184

Item Additions Additional ASQ:SE-2 items were formulated and field-tested in a series of eight iterative versions between 2010 and 2013. Items were added primarily to target early communication and behaviors associated with ASD, as well as internalizing behaviors in young children that might go undetected, such as excessive shyness and lack of reciprocity in social situations. We identified internalizing and communication behaviors and other more subtle behaviors associated with ASD in children 14 months of age and older, as these behaviors become easier to identify at this time. For infants, we looked at what behaviors parents would notice that are indicative of the communication, social, cognitive, and behavioral impairments at young ages (e.g., no back-and-forth sharing of sounds, no or few social smiles, no babbling or gesturing by

no back-and-forth sharing of sounds, no or few social smiles, no babbling or gesturing by 12 months) that might lead to early identification and improved outcomes. Between three and seven additional items were added to each of the previously existing age intervals, and minor wording changes were made to existing ASQ:SE items. We made minor edits to existing ASQ:SE items based on item response modeling analyses and feedback

minor edits to existing ASQ:SE items based on item response modeling analyses and feedback from users. Examples of new items include the following:

• “Does your baby make sounds and look at you while playing with you?” (6 months) • “When you copy sounds your baby makes, does your baby repeat the same sounds back to

• “When you copy sounds your baby makes, does your baby repeat the same sounds back to you?” (12 months) • “Does your child play with objects by pretending? For example, does your child pretend to

• “Does your child play with objects by pretending? For example, does your child pretend to talk on the phone, feed a doll, or fly a toy airplane?” (18–

talk on the phone, feed a doll, or fly a toy airplane?” (18– • “Is your child too worried or fearful?” (24–

Information Summary Sheet The third format change was to revise the Information Summary sheet of each ASQ:SE-2 ques-

ASQ:SE-2 Technical Report and indicate next steps for referral and evaluation. The Information Summary revisions were based on feedback from five focus groups with students, four focus groups with ASQ:SE users, as well as a final design review with professionals representing six programs with extensive experience using ASQ:SE. The revised Information Summary includes a scoring graphic illustrating the range of normative scores from 0% to 90% (the range of children’s scores is too wide to accurately illustrate in the space on each questionnaire— often up to 350 points— the 90th percentile score is indicated). The child’s total score can be reviewed on the scoring

graphic to get a visual representation of where the score falls on the distribution.

Monitoring Zone We developed a monitoring zone, or “questionable” area, just under the empirically derived cutoff score for each of the nine age intervals. The monitoring zone cutoff for each interval was set above the median at the 65th percentile, and each monitoring zone includes a range of 10– 30 points below the at-risk cutoff score, depending on the specific age interval. The monitoring zone will identify an additional 14% of children across the ASQ:SE-2 intervals. It is depicted on the Information Summary sheet as a light gray area on the scoring graphic, as it is on

depicted on the Information Summary sheet as a light gray area on the scoring graphic, as it is on ASQ-3. This monitoring zone was included to assist programs in talking to parents about results and the relationship between the child’s score and normative data, and to assist with decisions regarding referral options. Significant parent concerns paired with a child’s total score in the monitoring zone may indicate a need for immediate referral to a mental health agency. See Chapter 6, Table 6.3, for additional information to guide professionals on aspects to consider

regarding referral, including parent concerns.

Participant Recruitment Children between the ages of 1 and 72 months and their parents were recruited for the normative study. Approximately 10% were recruited through newspaper and magazine advertisements; 25% through agency personnel who attended national conferences and agreed to field- ASQ:SE-2; and 65% through recruiting efforts via electronic bulletin boards and parenting web sites. Additional recruiting methods included posting advertisements on other web sites (e.g., Craigslist), social media, and in parenting resource guides; sending recruitment letters to

ASQ:SE-2 Pilot Version Once initial content revisions were finalized in 2010, we began piloting an expanded version of ASQ:SE-2 to investigate 1) potential cutoff scores and psychometric properties with a new and larger normative population; 2) the accuracy of items extending the age range of ASQ:SE down to 1 month (from the original 3 months) and up to 72 months, or 6 years (from the original 66 months); and 3) how this expanded version identified children with and without ASD diagnoses. Parents of children from 1 month to 72 months of age were invited to complete the

DATA COLLECTION PROCEDURES

Data Collection Procedures

Excerpted from ASQ:SE-2ª Users By Jane Squires, Ph.D.,

186 ASQ:SE-2 Technical Report high-risk families and young children with disabilities in several states (including California,

high-risk families and young children with disabilities in several states (including California, Connecticut, Florida, Hawaii, Michigan, North Carolina, Ohio, Oregon, and Washington). An attempt was made to stratify the normative sample so that children/families would be representative of the U.S. population in terms of race/ethnicity, geographic region, parent education and income, and gender of children. Recruitment letters and research protocols were approved by the University of Oregon Human Subjects Compliance Committee prior to beginning data collection. Data from the normative sample were first collected, followed by recruitment of samples for assessing convergent validity— the agreement of the classification outcomes of ASQ:SE-2 (i.e., risk, okay) with other screening and evaluation measures— and ASQ:SE-

internal consistency, test–

Measures ASQ:SE-2 and a demographic form were the primary measures that were completed by parents. ASQ:SE-2 is a series of questionnaires at nine age intervals from 1 month to 72 months, as described previously. The questionnaires are designed to be completed by parents or other care-

described previously. The questionnaires are designed to be completed by parents or other caregivers who can provide information on a child’s social-emotional competence across situations. The demographic form asked parents to provide information on the child’s age, date of birth, gender, and race/ethnicity. Information about mother’s age at child’s birth, mother’s

education level, and family income also were requested.

Convergent Validity Several measures were used as comparators for ASQ:SE-2 outcomes to measure convergent validity. Due to the lack of “gold standard” assessments for the evaluation of general socialemotional difficulties in children from birth to 6 years (DelCarmen-Wiggins & Carter, 2004), we chose to compare ASQ:SE-2 outcomes with several evidence-based measures that focus on social-emotional outcomes in one or more age groups within the 1- to 7 2-month age range. We chose the term convergent validity to suggest that we were investigating the correspondence between ASQ:SE-2 and these measures rather than comparing ASQ:SE-2 with one gold standard of diagnosis. Convergent measures included the Devereux Early Childhood Assessment for Infants and Toddlers (DECA-IT; Mackrain, LeBuffe, & Powell, 2007), the Infant Toddler Social Emotional Assessment (ITSEA; Carter & Briggs-Gowan, 2006), and the Child Behav-

Social Emotional Assessment (ITSEA; Carter & Briggs-Gowan, 2006), and the Child Behavior Checklist (CBCL; Achenbach & Rescorla, 2000). DECA-IT (Mackrain et al., 2007) is an assessment designed for parent or teacher completion and measures protective and risk factors for children’s social-emotional development. Reliability studies by the DECA-IT authors indicated internal consistency ranged from .90 to .94, test– retest reliability from .83 to .94, and interrater reliability from .68 to .72. The Infant Interval, designed for children from birth to 18 months, was used with a subset of infants and

ASQ:SE-2 Technical Report Three subscales (i.e., Compliance, Negative Emotionality, and Prosocial Peer Relations) were used as comparators for ASQ:SE-2, based on recommendations from the ITSEA first author

used as comparators for ASQ:SE-2, based on recommendations from the ITSEA first author (Carter) as measures critical for social-emotional assessment (Beeber et al., 2007). The CBCL (Achenbach & Rescola, 2000) was used as a convergent measure for a subset of children whose ages were in the 18 months to 5 years range in the ASQ:SE-2 normative sample. CBCL is a well-standardized and validated checklist that has been used widely with diverse populations. CBCL is completed by parents and/or teachers, has solid psychometric evidence, and contains approximately 100 items targeting problem behaviors in young children

(Whitcomb & Merrell, 2013).

Diagnostic Status Additional children with a formal diagnosis of social-emotional disability, developmental disability, or ASD were recruited. Multidisciplinary teams in both educational and medical settings evaluated these children who received a diagnosis for eligibility for specialized early intervention/early childhood special education services. The disability status of these children was then compared with their overall classification on ASQ:SE-2 (at risk/okay) as a measure of

convergent validity.

Procedures with Parents Parents willing to participate in the study were presented with a packet of materials containing a consent form, ASQ:SE-2 questionnaire, demographic form, and one or more convergent validity measures, depending on the child’s age and the research phase. Packets were distributed in one of three ways: by mailing packets to parents (e.g., those who were contacted through advertisements, social network sites, online parenting sites) who agreed to participate; by preschool teachers directly to parents; and via the Internet through a research web portal. The forms were automatically scored for those using the portal, and the parents immediately received results online after completion. Parents were asked to complete paper research forms

within 1 week and return them by mail to the researchers.

Iterative Questionnaire Development In order to develop ASQ:SE-2, an initial version was developed and distributed to parents during a 2-year period, between 2009 and 2011. Between 1 and 7 items were initially added to each ASQ:SE interval to test the ability of these new items to differentiate between children with social-emotional difficulties and those without. Data were collected in an ongoing fashion and analyzed every 2– 3 months to test item functioning, utility, and agreement with convergent measures. Item response modeling and correlational and chi-square analyses were performed to test the functioning of the new items and their agreement with other measures and clusters of items. Every 5– 6 months, focus groups of students, researchers, practitioners, and parents were assembled to review the ASQ:SE-2 questionnaires, make comments, suggest revisions, and evaluate utility. Eight different field test versions with differing numbers of new items were tested and analyzed; items that did not differentiate children with social-emotional

188 ASQ:SE-2 Technical Report

Demographic Characteristics of Normative Sample Children between the ages of 1 month and 72 months were recruited to examine the psychometric properties of ASQ:SE-2. Data for demographic variables such as race/ethnicity, family income, and mother’s education level were not always provided by parents or primary caregivers for a variety of reasons (e.g., privacy, information unknown). The number of questionnaires

ers for a variety of reasons (e.g., privacy, information unknown). The number of questionnaires completed without corresponding demographic information is noted for each analysis. The ASQ:SE-2 total sample included 14,074 children. The total number of ASQ:SEassessments completed on these children was 16,424. The distribution of these questionnaires by age interval and gender is shown in Table C.1. The majority of parents completed one ASQ:SE-2 interval; however, some parents completed more than one ASQ:SE-2 as their child

ASQ:SE-2 interval; however, some parents completed more than one ASQ:SE-2 as their child aged (e.g., contributing questionnaires for the same child at 12, 24, and 36 months). For those children born 3 or more weeks premature, an adjusted age was used to correct for prematurity up to 24 months of age. This adjusted age corresponded to the expected due date. Note that ASQ-3 uses the same procedure for determining which questionnaire a care-

date. Note that ASQ-3 uses the same procedure for determining which questionnaire a caregiver should complete for the child. Table C.2 contains a comparison of U.S. Census Bureau estimates of the race/ethnicity distribution with those of the ASQ:SE-2 normative sample. This comparison is not straightforward, given that the U.S. Census did not use the category of “mixed” race/ethnicities on the 2010 Census and used a two-part question that separated categories of race from ethnicity (e.g.,

2010 Census and used a two-part question that separated categories of race from ethnicity (e.g., Hispanic or Latino/White not Hispanic or Latino). According to data provided by the U.S. Census, the ASQ:SE-2 normative sample had a higher percentage of well-educated mothers than found generally in the United States (see Table C.3). Comparisons are not straightforward, however, given differing categories of analysis. A comparison between the U.S. Census data and the ASQ:SE-2 sample on annual family income level indicates the ASQ:SE-2 sample was composed of a similar percentage of families

Total 16,394

Excerpted from ASQ:SE-2 $ ^{ \mathrm{TM}} $ User's Guide

Excerpted from ASQ:SE-2ª Users By Jane Squires, Ph.D.,

ASQ:SE-2 Technical Report Table C.2. Race/ethnicity comparison of ASQ:SE-2 normative sample (N = 14,137) with 2010 U.S. Census Bureau

Two or more races

Two or more races

aThere are race/ethnicity data missing for 2,287 cases. bFrom U.S. Census Bureau (2010b).

bFrom U.S. Census Bureau (2010b). cU.S. Census Bureau (2010b) reported Hispanic or Latino 16.3%, White Not Hispanic or Latino 83.7%; our demographic data were collected in different categories.

Do not know

Do not know

Table C.3. Education level comparison of ASQ:SE-2 normative sample (N = 14,290)a with 2010 U.S. Census Bureau

total scores across groups. We hypothesized that children with fewer risk factors would have lower scores on ASQ:SE-2, indicating social-emotional development in the typical range, and that as risk factors increased, ASQ:SE-2 scores would increase, indicating more potential social-emotional difficulties. The four groups are as follows: 1) No and Low Risk (i.e., children with no or one identified environmental/medical risk factor); 2) Risk (i.e., children with two or more environmental/medical risk factors); 3) Developmental Disability (i.e., children with established developmental disabilities who were receiving early intervention/early childhood

aThere are level of education data missing for 2,134 caregivers. Based on U.S. Census Bureau (2010a).

190 ASQ:SE-2 Technical Report Table C.4. Income level comparison of ASQ:SE-2 normative sample (N = 14,647) with 2010 U.S. Census Bureau

More than $40,000 60.0

More than $40,000

with identified social-emotional disabilities, according to IDEA Part B eligibility guidelines and Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition [DSM-IV] diagnostic classifications). Variables used to determine level of risk for the Risk group included the

following:

1. Family income less than $12,000 per year

2. Family income less than $12,000 per year

3. Mother younger than 18 years old when child was born

4. Mother younger than 18 years old when child was born

5. Mother’s highest level of education less than high school diploma

6. Mother’s highest level of education less than high school diploma

7. Involvement of child protective services with family; or a child in foster care

Establishing Reliability This section explains how we established reliability, including internal consistency and test–

4. Involvement of child protective services with family; or a child in foster care
5. Child’s birth weight less than 3 pounds, 5 ounces

ASQ:SE-2 Technical Report

Total/overall 10,385 7,916 40

Total/overall 10,385

aDevelopmental status data are missing for 6,039 children.

Developmental status data are missing for 6,039 children. bNo identified risk factors.

No identified risk factors. cOne or more identified risk factors.

Internal Consistency Internal consistency measures the extent to which the different items on the assessment tool measure the same underlying construct (Saliva, Ysseldyke, & Bolt, 2012). High internal consistency reflects items that assess the same characteristic or behavioral area. To measure internal consistency, coefficient alpha was calculated for each ASQ:SE-2 age interval using the variances of individual items and the variance of the total test scores (N = 11,489). Cronbach’s coefficient alphas for the ASQ:SE-2 intervals are shown in Table C.6. Alphas ranged from .71 to .90, with an overall alpha of .84. An alpha of .70 is considered to be an adequate measure of internal

Test– etest Reliability R Test– retest reliability measures the stability of child performance over time. Test– retest reliability for ASQ:SE-2 was determined by comparing the results of two of the same questionnaires completed by parents at 1- to 3-week intervals. A random sample of parents (N = 281) was asked to complete the same ASQ:SE-2 after returning the first completed questionnaire. The percent agreement between classifications of the child’s performance on ASQ:SE at Time 1 (first

consistency (Nunnally, 1978).

ASQ:SE-2 Technical Report

Overall

the cutoff point and further evaluation of their social-emotional status was needed. Test– retest agreement between classifications was 89% between Time 1 and Time 2 questionnaires. Intra-

class correlations also were calculated and were .91 across intervals.

of individuals who do not require further testing (i.e., false positives) and minimizing the nonidentification of individuals who should receive further testing (i.e., false negatives). After tentative cutoff scores are selected, the second step is to determine their accuracy and, thus, the validity of the screening measure. Determining the accuracy of the cutoff scores is done by comparing children’s classification on the screening measure with their classification on a selected criterion or convergent measure(s). ASQ:SE-2 was examined by comparing children’s classification (i.e., developmentally Okay, At Risk) on ASQ:SE-2 with their clas-

ASQ:SE-2 Technical Report that included the DECA-IT (Mackrain et  al., 2007), ITSEA (Carter & Briggs-Gowan, 2006), CBCL (Achenbach & Rescola, 2000), and professional diagnosis of a social-emotional

2006), CBCL (Achenbach & Rescola, 2000), and professional diagnosis of a social-emotional disability. Receiver operating characteristic (ROC) curves were used to calculate optimal cutoff scores (i.e., those that yield high true positives, low false positives, and low false negatives for ASQ:SE-2). ROC analysis permits the systematic comparison of true positive probabilities against false positive probabilities for a range of possible cutoff scores (Swets & Picket, 1982).

Formulas for calculating:

\frac{\mathsf{A}+\mathsf{B}}{\mathsf{A}+\mathsf{B}+\mathsf{C}+\mathsf{D}}

A + B A + B + C + D

\frac{\mathsf{A+D}}{\mathsf{A+B+D}}\times100

A + D × 100 A + B + C + D Sensitivity: The proportion of children correctly identified by the questionnaires as needing further

A A + C

\frac{C}{A+B+C+C}

B A + B + C + D Underidentification: The proportion of children (of the total number of children for whom a questionnaire

\frac{\mathsf{B}}{\mathsf{A}+\mathsf{B}+\mathsf{C}+\mathsf{D}}

\frac{textsf A{\textsf{B}}}{\textsf{A}\textsf{B}}

194 ASQ:SE-2 Technical Report criterion measure— DECA-IT (Mackrain et  al., 2007), ITSEA (Carter & Briggs-Gowan, 2006), CBCL (Achenbach & Rescola, 2000)— or had a professional diagnosis of a socialemotional disability to create these comparison data. Each child’s classification (i.e., Okay, At Risk) on ASQ:SE-2 was then compared with the classification (i.e., Okay, At Risk/Disabled) of the child by one of the criterion measures. Figure C.1 shows a four-cell contingency table used to assess the agreement between the screening measure (i.e., ASQ:SE-2) and the criterion measure (i.e., DECA-IT, ITSEA, CBCL, diagnosis of social-emotional disability). In addition, this figure shows the formulas for calculating percentage of children identified as needing further assessment and the percent agreement, sensitivity, specificity, overidentification, underidentifi-

assessment and the percent agreement, sensitivity, specificity, overidentification, underidentification, and positive predictive value of ASQ:SE-2. Comparison of range of scored points, medians, interquartile ranges, and ROC cutoffs is shown in Table C.7. Note that ROC cutoff scores for most intervals were similar to scores derived from adding 1.5 semi-interquartile ranges to medians. The general trend of increasingly higher scores as children develop is reflected in median scores, except at 48 and

increasingly higher scores as children develop is reflected in median scores, except at 48 and 60 months. Cutoff scores for screening tools frequently are set by using means and standard deviations. That is, the mean score plus one standard deviation is a likely choice for a cutoff score. Using means to calculate cutoff scores presumes a normal distribution of scores, however. Score distribution for ASQ:SE-2 questionnaires was positively skewed— that is, the majority of children obtained low scores (i.e., indicating no problem or Okay) and relatively few children obtained high scores (i.e., indicating a potential problem or risk). The majority of children had scores below 25. Figure C.2 shows the positively skewed distribution of scores for the 48 month ASQ:SE-2; other age intervals showed similar score distributions. Means and standard deviations were not used for determining cutoff points because of the positive skew of ASQ:SEscores across intervals. Instead, ROC analyses were conducted to determine the best cutoff

point for each interval.

Gender Differences Scores for males and females were compared in order to examine gender differences. Mean and median scores by gender are presented in Table C.9. Box plots were then derived to examine the score distributions by gender. Box plots provide a visual picture of a distribution. Box plots for the 30 month and 36 month ASQ:SE-2 male and female score distributions are shown in Figure C.3. The bottom line of the box is the 25th percentile, or Quartile 1. The top line of

Modifying Cutoff Scores If programs want to modify cutoff scores, then semi-interquartile ranges [i.e., median + (Quartile 1 – Quartile 3)/2] should be used as the basis for modification. See www.agesandstages.com

Monitoring Zone

ASQ:SE-2 Technical Report

60 month 1,516 0–310 46

mean is indicated by the “x.” Whiskers (lines) extend to the highest and lowest observations but not further than 1.5 interquartile ranges. Outliers beyond 1.5 interquartile ranges are indicated

ASQ:SE-2 total scores

not further than 1.5 interquartile ranges. Outliers beyond 1.5 interquartile ranges are indicated by “ ”; outliers beyond 3 interquartile ranges are indicated by “*.” As shown in Figure C.3, the majority of scores for males at the 30 month interval range between 25 and 75, with the 1.5 interquartile ranges extending to 180. Outliers extend upward to 370. For females, the range is between 20 and 61, with the 1.5 interquartile ranges extending to 150. Outliers extend upward to 325. A similar distribution for both males and females can be seen at the 36 month interval. Similar distribution patterns occurred at all test intervals and indicate, in general, that males tended to have greater dispersal of scores and more extreme

Figure C.2. ASQ:SE-2 total scores by number of children for the 48 month questionnaire (N=2,530), showing a positively skewed distribution.

Excerpted from ASQ:SE-2 $ ^{ \mathrm{TM}} $ User's Guide

ASQ:SE-2 Technical Report

If gender group differences are compared using nonparametric tests (i.e., Kruskal-Wallis Test [Heiman, 1992]), significant differences are found at 12, 18, 24, 30, 36, 48, and 60 months, as shown in Table C.9. It is important to note that the validity sample did not have adequate numbers of girls identified with social-emotional problems to determine if separate cutoff scores

Figure C.3. Box plots for 30 month and 36 month ASQ:SE-2 by gender. Box plots illustrate the distribution of scores. The bottom line of the box is the 25th percentile, or Quartile 1. The top line of the box is the 75th percentile, or Quartile 3. The middle line is the median, or Quartile 2. Mean is indicted by an “x.” Whiskers (lines) extend to the highest and lowest observations, but not further

numbers of girls identified with social-emotional problems to determine if separate cutoff scores for females are needed, especially at younger ages. Gender differences also were compared using item response modeling (IRT), specifically differential item functioning (DIF). Differences between scores of boys and girls were analyzed across the nine ASQ:SE-2 age intervals using the Mantel (1963) approach (Dorans & Holland, 1993); 22 out of 295 items were identified with DIF or differential functioning for boys and

Excerpted from ASQ:SE-2 $ ^{\mathrm {TM}} $ User's Guide

ASQ:SE-2 Technical Report

Total 9,460

Total

*Significant total at p < .001. the phone?” Based on the analysis, we revised this item for subsequent versions to “Does your child play with objects by pretending? For example by talking on the phone, feeding a doll, or flying a toy airplane?” The order of items within the questionnaires was changed so that they followed the developmental order identified by the IRT analysis. Second, and perhaps most important, we included additional examples that might have more appeal to boys, with materi-

when their scores are near as well as above the screening cutoff points.

Examining Convergent Validity A comparison with selected social-emotional measures was necessary to determine how accurately ASQ:SE-2 discriminates between children whose social-emotional development is proceeding without a problem from those children who have or are at risk for developing a social-emotional problem. DECA-IT (Mackrain et al., 2007), ITSEA (Carter & Briggs- Gowan, 2006), and CBCL (Achenbach & Rescola, 2000) were convergent measures chosen to

important, we included additional examples that might have more appeal to boys, with materials such as balls. Furthermore, items with differential functioning indicated specific patterns in socialemotional competence areas across intervals by gender. For example, the scores of boys indicated a greater frequency of problem behaviors than girls on several items in social-communication and self-regulation areas; girls’ scores indicated a greater frequency of problem behaviors than boys on items assessing sleeping and autonomy. The different patterns of scoring may have resulted from potential differences between genders or the parents’ varying expectations for

198 ASQ:SE-2 Technical Report Parents or other primary caregivers of children in the validity sample (N = 2,862) completed the DECA-IT (Mackrain et al., 2007), ITSEA (Carter & Briggs-Gowan, 2006), and/or CBLC (Achenbach & Rescola, 2000) (depending on the age of their child) within 2– 3 weeks of also completing ASQ:SE-2. Included in the validity sample was a group of 146 children ranging in age from 6 to 60 months who had been professionally diagnosed as having a socialemotional disability and were receiving intervention services. Each diagnosis was based on a multidisciplinary battery of diagnostic assessments. Table C.10 summarizes the numbers of

multidisciplinary battery of diagnostic assessments. Table C.10 summarizes the numbers of children by convergent measure. Children in the validity sample were classified as either Okay or At Risk based on their ASQ:SE-2 score using the established cutoffs and were independently classified as either Okay or At Risk/Disabled using their score on the DECA-IT (Mackrain et al., 2007), ITSEA (Carter & Briggs-Gowan, 2006), and/or CBLC (Achenbach & Rescola, 2000) or professional diagnosis. These two independent classifications were then compared for all children in the validity sample. One of four outcomes was possible: 1) ASQ:SE-2 and the criterion measure both classified the child as Okay (i.e., true negatives), 2)  ASQ:SE-2 and the criterion measure both classified the child as At Risk/Disabled (i.e., true positives), 3) ASQ:SE-2 classified the child as Okay while the criterion measure classified the child as At Risk/Disabled (i.e., false negatives), and 4) ASQ:SE-2 classified the child as At Risk, whereas the criterion measure clas-

negatives), and 4) ASQ:SE-2 classified the child as At Risk, whereas the criterion measure classified the child as Okay (i.e., false positives). Contingency tables containing four cells (i.e., A = true positives, B = false positives, C  = false negatives, and D = true negatives, as shown in Figure C.1) were developed for each ASQ:SE-2 age interval using the ROC cutoff scores (see Table C.7) to conduct these comparisons. Each contingency table contained in Figure C.4 shows the agreement for true positives, false positives, false negatives, and true negatives. Using the data contained in the contingency table, the sensitivity, specificity, false positive rate, false negative rate, percent agreement, under identified rate, over identified rate, and positive predictive value were calculated for each ASQ:SE-2 age interval. An overall comparison across all intervals is shown in

Table C.10. Measures and N used for ASQ:SE-2 convergent validity analyses

Total 133 375 782 1,426 146 2,862

ASQ:SE-2 Technical Report Table C.11 presents a comparison of the sensitivity, specificity, percent agreement, false positive rate, false negative rate, underidentified rate, and overidentified rate across ASQ:SE-

positive rate, false negative rate, underidentified rate, and overidentified rate across ASQ:SEage intervals. Sensitivity ranged from a low of 77.8% at 2  months to a high of 84% at 24 months. Specificity ranged from 76.2% at 18 months to 98% at 60 months. Percent agreement ranged from 77% at 18 months to 89.1% at 60 months. The underidentified rate ranged from 1.8% at 6 months to 10.9% at 48 months, whereas the overidentified rate ranged from .8% at 60 months to 19.4% at 12 months. These findings suggest that ASQ:SEaccurate in discriminating between children who are Okay and those who need follow- Because of the overidentification rate at 12 months (19.4%) and 18 months (19.0%), programs

2 month Criterion measure classification $ ^{a} $

Figure C.4. Contingency tables showing agreement between ASQ:SE-2 classification, criterion measure classification, and ASQ:SE-2 sensitivity, specificity, false positive rate, false negative rate, percent agreement, percent underidentified, percent over identified, and positive predictive value by age interval (definitions and formulas are contained in Figure C.1). Criterion measure classification includes CBCL, DECA-IT, ITSEA, and professional diagnoses. (Key:aIncludes DECA-IT; bIncludes DECA-IT and professional

200

ASQ:SE-2 Technical Report

Known Groups Validity

This section describes the validity of known groups: risk and disability status, and ASD.

Risk and Disability Status Examining the differences in scores across groups is another approach to assessing the validity of a screening measure suggested by Spector (1992). For this analysis, a subsample of children in the validity sample were divided into one of three groups based on developmental status— No Risk,

ASQ:SE-2 Technical Report

80.7% 83.5% 16.5% 19.3%

aIncludes CBCL, DECA-IT, ITSEA, and professional diagnosis. Figure C.5. Contingency table showing overall agreement (combined across age intervals) between ASQ:SE-2 classification with criterion measure classification and ASQ:SE-2 sensitivity, specificity, false positive rate, false negative rate, percent agreement,

Criterion measure classification includes CBCL, DECA-IT, ITSEA, and professional diagnosis. No Risk group if caregivers reported no risk factors (N = 7,916); children were assigned to the Developmentally Disabled group if they were receiving general early intervention services (N = 454); and children were assigned to the Socially-Emotionally Disabled group if they had been diagnosed with a behavior or emotional problem and were receiving intervention services (N = 224). Risk factors included the following: 1) family income less than $12,000; 2) mother

IT, ITSEA, and professional diagnosis. No Risk group if caregivers reported no risk factors (N = 7,916); children were assigned to the Developmentally Disabled group if they were receiving general early intervention services (N = 454); and children were assigned to the Socially-Emotionally Disabled group if they had been diagnosed with a behavior or emotional problem and were receiving intervention services (N = 224). Risk factors included the following: 1) family income less than $12,000; 2) mother

(N = 224). Risk factors included the following: 1) family income less than $12,000; 2) mother younger than 18 years old when child was born; 3) mother’s level of education less than high

school diploma; 4) involvement of child protective services with family or a child in foster care; 5) birth weight less than 3 pounds, 5 ounces; and 6) child enrolled in Early Head Start or Head

Overall 2,862 — 80.7 83.5

ASQ:SE-2 age interval

Figure C.6. Median ASQ:SE-2 scores by group risk status.

distribution of risk groups for the 48 month ASQ:SE-2 is presented in Figure C.7. The box plots show mean (marked with a “x”) and median (middle horizontal line in each box) scores are elevated for children with developmental and social-emotional disabilities. In addition, there is almost no overlap in the distribution of scores between the No Risk and Social-Emotional Disability groups. Children with diagnosed social-emotional disabilities had the highest scores. There is an overlap in scores for the Risk and No Risk groups, possibly due to a selection factor involving a large number of parents with concerns for their child’s social-emotional development who completed the ASQ:SE-2 online. The majority of these families experienced low or

Figure C.7. Box plot distribution of total scores on the 48 month ASQ:SE-2 by developmental status. Box plots illustrate the spread of distribution. The bottom line of the box is the 25th percentile, or Quartile 1. The top line of the box is the 75th percentile, or Quartile 3. The middle line is the median, or Quartile 2. Mean is indicted by a “x.” Whiskers (lines) extend to the highest and lowest observations but not further than 1.5 interquartile ranges. Outliers beyond 1.5 interquartile

\tt{a}^{()}^{(}^{{X})^{2}}

Excerpted from ASQ:SE-2 $ ^{2} $ User's Guide

\mathrm{a}^{\prime\prime}\mathrm{

204

Autism Spectrum Disorder For comparisons related to ASD, the classification of children on ASQ:SE-2 was compared for 133 children between 18 and 60 months of age with a diagnosis of ASD. Overall, the ASQ:SEclassification (Okay, Risk) agreed 83.5% with the child’s classification of ASD (Risk). These children had received a diagnosis of ASD through educational or medical diagnostic proce-

children had received a diagnosis of ASD through educational or medical diagnostic procedures, and their parents had completed an ASQ:SE-2 questionnaire.

dures, and their parents had completed an ASQ:SE-2 questionnaire. These preliminary analyses suggest that ASQ:SE-2 will identify the majority of children with ASD. Ongoing research is being conducted in which in-depth ASD measures such as the Autism Diagnostic Observation Schedule, Second Edition (Lord, Rutter, DiLavore, & Risi, 2012) are used concurrently with ASQ:SE- These results may assist programs in making decisions regarding referral for ASD testing based on ASQ:SE-2 results. In addition, a summary of ASQ:SE-2 and ASQ-3 items related to behaviors associated with ASD appears in Table C.12. ASQ-3 items pertaining to autism are included to provide a better understanding of how to use the

SUMMARY

UTILITY SURVEY

Utility SURVEY Utility of a screening tool measures the usefulness or practicality of the test or procedure (Bricker & Squires, 1989). A random sample of parents (N = 332) who completed a paper version of ASQ:SE-2 was asked to complete a utility survey that requested their opinion about the length, appropriateness, and ease of completion of ASQ:SE-2. Summary results of the utility survey can be found in Table C.13.

of the utility survey can be found in Table C.13. Thirty-one percent of the respondents indicated that it took less than 10  minutes to complete; thirty-nine percent indicated that it took 10– 20 minutes to complete. Ninety-three percent of the respondents indicated that ASQ:SEwas easy to understand, and eighty percent felt that question content was appropriate for their child. Seventy-three percent reported that ASQ:SE-2 helped them think about their child’s behavior, and ninety-four percent said that they would like to fill out another questionnaire when their child is older. Thus, parents reported that ASQ:SE-2 was easy to understand, it took little time to complete, questions were appropriate, and they enjoyed the process. In addition, parents felt that completing ASQ:SE-2 was interesting and helped them think about the social-

ASQ:SE-2 Technical Report

ASQ:SE-2 Technical Report

27

Excerpted from ASQ:SE-2ª Users By Jane Squires, Ph.D.,

ASQ:SE-2 Technical Report

206

Can your child draw a picture of a Social reciprocity — 54, 60 person/girl/boy?**

generally less than specificity (i.e., the ability of ASQ:SE-2 to correctly identify typically developing children). Research is continuing on ASQ:SE-2. Results of ASQ:SE-2 questionnaires completed by parents are being compared with in-depth ASD diagnostic evaluations to study the sensitivity and specificity of ASQ:SE-2 related to identifying young children with ASD. Additional research findings will be posted on the ASQ web site as they become available

slightly between intervals and across measures. See questionnaires for exact wording. children. Internal consistency measured by coefficient alpha was found to be high across intervals, ranging from .71 to .90, with an overall alpha of .84. Test– the agreement between two ASQ:SE-2 questionnaires completed by parents at 1 day to 60 days apart was .89. Sensitivity ranged from .78 at 2 months to .84 at 24 months, with .81 overall sensitivity. Specificity of the questionnaires ranged from .76 at 18 months to .98 at 60 months, with .84 overall. Percent agreement between questionnaires and standardized assessments/disability status ranged from .77 at 18 months to .89 at 60 months with overall agreement at .83. The ability of ASQ:SE-2 to detect atypical socialgenerally less than specificity (i.e., the ability of ASQ:SE-

slightly between intervals and across measures. See questionnaires for exact wording. children. Internal consistency measured by coefficient alpha was found to be high across intervals, ranging from .71 to .90, with an overall alpha of .84. Test– retest reliability, measured as 2 questionnaires completed by parents at 1 day to 60 days apart was .89. Sensitivity ranged from .78 at 2 months to .84 at 24 months, with .81 overall sensitivity. Specificity of the questionnaires ranged from .76 at 18 months to .98 at 60 months, with .84 overall. Percent agreement between questionnaires and standardized assessments/disability status ranged from .77 at 18 months to .89 at 60 months with overall agreement at 2 to detect atypical social-emotional development (sensitivity) was generally less than specificity (i.e., the ability of ASQ:SE-2 to correctly identify typically devel-

ASQ:SE-2 Technical Report

ASQ:SE-2 Technical Report

REFERENCES

REFERENCES Achenbach, T., & Rescorla, L. (2000). Manual for the ASEBA preschool forms and profiles. Burlington: University of Vermont, Research Center for Children, Youth, and Families. Beeber, L., Chazan Cohen, R., Squires, J., Jones Harden, B., Boris, N., Heller, S., & Malik, N. (2007). The early promotion and intervention research consortium (E-PIRC): Five approaches to improving infant/ toddler mental health in Early Head Start. Infant Mental Health Journal, 28(2), 151–170. Bricker, D., & Squires, J. (1989). The effectiveness of screening atnaire. Topics in Early Special Childhood Education, 3(9), 67–85. Bricker, D., Squires, J., Twombly, E., Yockelson, S., & Kim, Y. (1996). Behavior– naires. Unpublished manuscript.

Achenbach, T., & Rescorla, L. (2000). Manual for the ASEBA preschool forms and profiles. Burlington: University of Vermont, Research Center for Children, Youth, and Families. Beeber, L., Chazan Cohen, R., Squires, J., Jones Harden, B., Boris, N., Heller, S., & Malik, N. (2007). The early promotion and intervention research consortium (E-PIRC): Five approaches to improving infant/ toddler mental health in Early Head Start. Infant Mental Health Journal, 28(2), 151–170. Bricker, D., & Squires, J. (1989). The effectiveness of screening at-risk infants: Infant monitoring question- Bricker, D., Squires, J., Twombly, E., Yockelson, S., & Kim, Y. (1996). Behavior– Ages and Stages Question-

naires. Unpublished manuscript. Carter, A., & Briggs-Gowan, M. (2006). Manual for the Infant-Toddler Social & Emotional Assessment (ITSEA) and Brief-ITSEA (BITSEA). San Antonio, TX: Psychological Corporation. DelCarmen-Wiggins, R., & Carter, A. (Eds.). (2004). Handbook of infant, toddler, and preschool mental health assessment. New York, NY: Oxford University Press. Dorans, N.J., & Holland, P.W. (1993). DIF detection and description: Mantel-Haenszel and standardization.

Dorans, N.J., & Holland, P.W. (1993). DIF detection and description: Mantel-Haenszel and standardization. In P.W. Holland & H. Wainer (Eds.), Differential item functioning (pp. 35– 66). Hillsdale, NJ: Lawrence

208 ASQ:SE-2 Technical Report Krogstad, J., & Cohn, D. (2014). U.S. Census looking at big changes in how it asks about race and ethnicity. Retrieved from Pew Research Center web site: http://www.pewresearch.org/fact-tank/2014/03/14 /u-s-census-looking-at-big-changes-in-how-it-asks-about-race-and-ethnicity/ Lord, C., Rutter, M., DiLavore, P., & Risi, S. (2012). Autism Diagnostic Observation Schedule. Los Angeles, CA: Western Psychological Services. Mackrain, M., LeBuffe, P., & Powell, G. (2007) Devereux Early Childhood Assessment for Infants and Toddlers (DECA-IT). Lewisville, NC: Kaplan Early Learning. Printz, P.H., Borg, A., & Demarree, M.A. (2003). A look at social, emotional, and behavioral screening tools for Head Start and Early Head Start. Newton, MA: Education Development Center, Center for Children & Families. Saliva, J., Ysseldyke, J., & Bolt, S. (2012). Assessment. In Special and inclusive education (12th ed.). Belmont, CA: Wadsworth. Spector, P.E. (1992). A consideration of the validity and meaning of self-report measures of job conditions (Paper 567). Psychology Faculty Publications. Retrieved from http://scholarcommons.usf.edu/psy_facpub /567 Swets, J.A., & Pickett, R.M. (1982). Evaluation of diagnostic systems: Methods from signal detection theory. San Diego, CA: Academic Press. U.S. Census Bureau. (2010a). Educational attainment in the United States: 2010— Detailed tables. Retrieved from http://www.census.gov/hhes/socdemo/education/data/cps/2010/tables.html U.S. Census Bureau. (2010b). Overview of race and Hispanic origin: 2010. Retrieved from http://www.census .gov/prod/cen2010/briefs/c2010br-02.pdf Whitcomb, S., & Merrell, K. (2013). Behavioral, social, and emotional assessment of children and adolescents