NSCH Data Anomaly: Investigating Age Correlations

by Alex Johnson 50 views

Introduction

The National Survey of Children's Health (NSCH) is a valuable resource for understanding the well-being of children across the United States. However, recent analyses of the 2021 and 2022 NSCH datasets have revealed a concerning anomaly: systematic negative age correlations across several developmental items. This means that, contrary to expectations, some skills and abilities appear to decrease with age in these datasets. This finding led to the exclusion of NSCH 2021 and 2022 data from IRT (Item Response Theory) calibration, a process used to establish national benchmarks for child development. In this article, we will delve into the investigation of the root causes behind these negative correlations, explore potential solutions to salvage this valuable data, and discuss the implications for future national benchmarking efforts. Understanding and addressing this issue is crucial for ensuring the accuracy and reliability of NSCH data and its use in informing policies and programs that support the healthy development of children nationwide.

The exclusion of NSCH 2021 and 2022 data from IRT calibration highlights the importance of rigorous data quality checks and the potential challenges in combining data from different sources or survey designs. By carefully examining the potential root causes of the negative age correlations, we can gain insights into the factors that can affect data quality and develop strategies to mitigate these issues in future surveys. This investigation also underscores the need for ongoing collaboration and communication between researchers, data analysts, and survey designers to ensure that data collection and analysis methods are aligned and that data are interpreted appropriately.

The goal of this article is to provide a comprehensive overview of the investigation into the negative age correlations in NSCH 2021 and 2022 data, outlining the problem, exploring potential causes, detailing the investigation tasks, and discussing the expected outcomes. By sharing this information, we hope to inform the broader research community about the challenges and complexities of working with large-scale survey data and to encourage further research and discussion on data quality issues in child development research. Ultimately, our aim is to ensure that the NSCH data can be used to its full potential to improve the lives of children and families across the nation.

Background: NSCH and IRT Calibration

The National Survey of Children’s Health (NSCH) is a critical data source providing nationally representative information on the physical and emotional health, access to healthcare, and family and community context of children aged 0-17 years in the United States. The survey, conducted by the U.S. Census Bureau and sponsored by the Health Resources and Services Administration (HRSA), aims to provide insights into factors influencing children's well-being. Researchers and policymakers use NSCH data to monitor trends, identify disparities, and inform policies and programs designed to improve the health and development of children.

IRT calibration is a statistical process used to estimate the parameters of items in a test or survey. In the context of child development, IRT calibration involves analyzing responses to developmental items to create a standardized scale that can be used to measure children's abilities and skills. The calibrated items can then be used to assess individual children's development, track progress over time, and compare children's performance to national benchmarks. The accuracy and reliability of IRT calibration depend on the quality of the data used, including the representativeness of the sample, the validity of the items, and the absence of systematic biases or errors.

To create national benchmarks for IRT calibration, a sample of 1,000 children aged 0-6 from each of the 2021 and 2022 NSCH datasets was initially selected. However, after quality validation, the decision was made to exclude both NSCH21 and NSCH22 from the calibration dataset due to the discovery of systematic negative age correlations. This decision, documented in Commit 59816a4, highlights the importance of rigorous data quality checks in ensuring the validity of research findings and the potential challenges in using data from different sources or survey designs. The exclusion of NSCH 2021 and 2022 data underscores the need for a thorough investigation into the root causes of the negative age correlations and the development of strategies to mitigate these issues in future surveys.

Problem Description: Unveiling the Negative Correlations

The core issue at hand involves multiple developmental items within the NSCH 2021/2022 datasets exhibiting negative correlations with age. This is a significant deviation from the expected positive correlation, where a child's skills and abilities typically improve with age. Instead, the data suggests that, for certain items, older children are performing worse than younger children, an implausible developmental trend.

This unusual pattern was observed across several developmental domains, indicating a widespread data quality concern rather than an isolated issue within a specific area of child development. The consistency of this negative correlation across multiple domains suggests that the problem is not due to a genuine developmental reversal but rather to systematic errors or biases in the data collection or processing methods. This finding underscores the importance of carefully examining the data collection and processing procedures used in the NSCH to identify the source of the negative correlations and develop strategies to correct or mitigate the issue.

To illustrate the scope of the problem, many items that showed positive age correlations in other datasets such as NE20, NE22, NE25, and USA24, displayed negative correlations in NSCH21/NSCH22. This discrepancy further emphasizes the need for a thorough investigation into the factors that may be contributing to the negative correlations in the NSCH data. Understanding the reasons for these discrepancies is crucial for ensuring the accuracy and reliability of the NSCH data and its use in informing policies and programs that support the healthy development of children nationwide.

Potential Root Causes: Exploring the Possibilities

Several potential factors could be contributing to the observed negative age correlations in the NSCH 2021/2022 data. These can be broadly categorized as:

1. Missing Data Patterns

Similar to the issues encountered in NE25 (Issue #7), missing data patterns could be influencing the correlations. If items are administered to incorrect age groups or if age-dependent missingness exists, it could create spurious correlations. The specific survey routing in NSCH might differ from the Kidsights survey design, leading to inconsistencies in how items are presented and answered. For example, if older children are less likely to be asked certain questions because it is assumed that they have already mastered those skills, this could lead to an underestimation of their abilities and a negative correlation with age. Understanding and addressing these missing data patterns is crucial for ensuring the accuracy of the NSCH data and its use in informing policies and programs that support the healthy development of children.

2. Missing Value Codes

The use of specific codes (94-98) in NSCH to denote missing or skipped patterns, as highlighted in Issue #6, could be contaminating threshold estimates and affecting correlation calculations. Even after addressing the threshold issue, the presence of these codes might still be influencing the overall data analysis. The way these missing value codes are handled during data processing can have a significant impact on the results, and it is essential to ensure that they are treated appropriately to avoid introducing biases or errors. For example, if missing values are simply ignored without accounting for the reasons why they are missing, this could lead to an underestimation of the true relationships between variables.

3. Survey Design Differences

NSCH, being a parent-report survey for children aged 0-17, has a much broader age range compared to Kidsights, which focuses on ages 0-6. This difference in age range could lead to items performing differently due to varying interpretations or reporting biases across age groups. Parent reporting bias might differ significantly for older versus younger children, potentially skewing the data. Parents of older children may have different expectations or recall biases compared to parents of younger children, which could affect their responses to the survey questions. Understanding and addressing these survey design differences is crucial for ensuring the comparability of data across different studies and for drawing accurate conclusions about child development.

4. Sampling or Weighting Issues

NSCH employs complex survey weights to ensure national representativeness. The random sample of 1,000 children selected for analysis might not preserve the correct age distribution, and unweighted analysis could introduce bias. The survey weights are designed to account for differences in the probability of selection and to adjust for non-response bias, but if these weights are not properly applied or if the sample is not representative of the population, it could lead to inaccurate estimates. It is important to carefully consider the sampling and weighting procedures used in the NSCH to ensure that the data are representative of the population and that the results are not biased by sampling or weighting issues.

5. Response Scale Differences

Variations in response scales between NSCH and Kidsights studies, along with incorrect reverse coding logic or value transformations in helper functions, could introduce artifacts. If the response scales are not consistent across studies, it could lead to differences in how respondents answer the questions, making it difficult to compare the data. Similarly, if the reverse coding logic is incorrect, it could lead to misinterpretation of the data and inaccurate conclusions. It is important to carefully review the response scales and coding logic used in both NSCH and Kidsights studies to ensure that they are consistent and that the data are interpreted correctly.

Investigation Tasks: A Phased Approach

To systematically investigate the potential root causes of the negative age correlations, a phased approach has been outlined:

Phase 1: Descriptive Analysis

This phase focuses on understanding the scope and nature of the problem.

  • Identify affected items: Calculate age correlations for all NSCH21/NSCH22 items and compare them to pooled correlations from other datasets. This will help pinpoint which specific items are exhibiting the negative correlation pattern.
  • Age distribution analysis: Create a crosstab of age by study for NSCH21/NSCH22. This will help identify any irregularities in the age distribution of the sample that might be contributing to the problem.
  • Missing data patterns: For each problematic item, calculate the response rate by age. This will help determine if missing data patterns are related to the negative correlations.

Phase 2: Data Quality Checks

This phase aims to verify the accuracy and consistency of the data.

  • Verify missing code recoding: Confirm that the NSCH helper functions properly recode the missing codes (94-98) to NA (Not Available). This ensures that missing values are handled correctly during data analysis.
  • Validate reverse/forward coding: Verify that the direction of coding is correct for items with negative correlations. This ensures that the items are interpreted correctly and that the negative correlations are not due to coding errors.
  • Response scale validation: Check that the unique values for problematic items match the expected scales. This ensures that the response scales are consistent across items and that there are no unexpected values that might be contributing to the problem.

Phase 3: Comparative Analysis

This phase involves comparing the NSCH data to other datasets to identify discrepancies.

  • Study-specific correlation table: Extend the Age Gradient Explorer tool to include NSCH21/NSCH22. This will allow for a direct comparison of age correlations across different studies.
  • Domain-level analysis: Check if negative correlations cluster in specific developmental domains. This will help identify if the problem is specific to certain areas of child development.
  • Cross-study item overlap: Compare correlation patterns for the same items across different studies. This will help determine if the negative correlations are unique to the NSCH data or if they are also present in other datasets.

Phase 4: Root Cause Determination

This final phase synthesizes the findings from the previous phases to determine the primary cause of the negative age correlations and whether the NSCH data can be salvaged.

  • Synthesize findings from Phases 1-3: This involves reviewing all the data and analysis from the previous phases to identify the most likely cause of the negative correlations.
  • Determine primary cause and whether NSCH data can be salvaged: Based on the findings, a decision will be made whether the NSCH data can be corrected and used for IRT calibration or whether it must be excluded.
  • Document decision for future reference: The decision and the rationale behind it will be documented for future reference. This will ensure that the decision-making process is transparent and that the reasons for excluding or including the NSCH data are clearly understood.

Expected Outcomes: Two Possible Scenarios

The investigation into the negative age correlations in the NSCH 2021/2022 data may lead to one of two possible outcomes:

Scenario A: Fixable Issue

In this scenario, the investigation identifies a specific data processing error that is causing the negative correlations. This error could be related to missing data patterns, missing value codes, survey design differences, sampling or weighting issues, or response scale differences. If the error can be identified and corrected, the NSCH data can be salvaged and used for IRT calibration.

  • Identify specific data processing error: The investigation identifies the specific error that is causing the negative correlations.
  • Implement correction in NSCH helper functions: The error is corrected in the NSCH helper functions.
  • Re-integrate NSCH21/NSCH22 into calibration: The NSCH data is re-integrated into the IRT calibration dataset.
  • Improves national representativeness of IRT parameters: The inclusion of the NSCH data improves the national representativeness of the IRT parameters.

Scenario B: Unfixable Design Issue

In this scenario, the investigation reveals a fundamental incompatibility between the NSCH and Kidsights measurement approaches. This incompatibility could be due to differences in survey design, response scales, or other factors that cannot be easily corrected. In this case, the NSCH data cannot be used for IRT calibration and must be excluded.

  • Document fundamental incompatibility between NSCH and Kidsights measurement: The investigation documents the reasons why the NSCH and Kidsights data cannot be combined.
  • Formalize exclusion decision with clear rationale: A formal decision is made to exclude the NSCH data from IRT calibration, and the rationale behind the decision is clearly documented.
  • Consider alternative national benchmarking sources: Alternative sources of national benchmarking data are considered.
  • Update documentation to explain why NSCH is excluded: The documentation is updated to explain why the NSCH data was excluded from IRT calibration.

Impact: Implications for National Benchmarking

The outcome of this investigation has significant implications for national benchmarking efforts.

Current State:

  • The calibration dataset currently includes 7,512 records from NE20, NE22, NE25, and USA24 only.
  • No NSCH data is included, despite the availability of approximately 2,000 age-eligible records.

If Salvageable:

  • The inclusion of the NSCH data would add approximately 2,000 records to the calibration dataset, representing a 26% increase.
  • The national representativeness of the IRT parameters would improve significantly.
  • The IRT parameters would be more generalizable beyond Nebraska.

If Not Salvageable:

  • An alternative national benchmarking strategy would be needed.
  • NSCH 2023+ data with improved data quality may be needed.
  • The limitations of the current calibration sample would need to be documented.

Related Issues & Commits

  • Issue #6: Fixed NSCH missing codes causing invalid thresholds (CLOSED).
  • Issue #7: NE25 age-inappropriate item administration (OPEN) - similar investigation pattern.
  • Commit 59816a4: Excluded NSCH 2021/2022 from calibration.
  • Commit 20e3cf5, 25d2b47: Fixed NSCH helper functions to recode missing codes before transformations.

Files to Review

  • scripts/irt_scoring/helpers/recode_nsch_2021.R - Data transformation logic.
  • scripts/irt_scoring/helpers/recode_nsch_2022.R - Data transformation logic.
  • scripts/irt_scoring/prepare_calibration_dataset.R - Dataset creation (currently excludes NSCH21/22).
  • scripts/shiny/age_gradient_explorer/ - QA tool for visualizing correlations.
  • docs/irt_scoring/quality_flags.csv - Quality validation flags.

Priority

This investigation is considered High priority due to the significant potential value of the NSCH data for national benchmarking. Understanding why the NSCH data fails quality checks is critical for deciding whether to pursue NSCH 2023+ data and for informing future multi-study IRT calibration efforts.

Next Steps

The following next steps have been identified:

  1. Extend the Age Gradient Explorer to include NSCH21/NSCH22 in the correlation table.
  2. Run a systematic correlation analysis to quantify how many items are affected.
  3. Investigate the top 10 most discrepant items (largest positive to negative correlation flips).
  4. Document the findings and make a go/no-go decision on NSCH inclusion.

In conclusion, the investigation into the negative age correlations in the NSCH 2021/2022 data is a critical undertaking that will have significant implications for national benchmarking efforts. By systematically exploring the potential root causes of the negative correlations and carefully evaluating the quality of the NSCH data, we can ensure that our national benchmarks are accurate, reliable, and representative of the diverse population of children in the United States.

For further information on the National Survey of Children's Health, please visit the HRSA website.