Comm 141 | deewilsoncon

'Noise' in Child Welfare Judgment

(Originally published October 2021)

Noise: A Flaw in Human Judgment (2021) by Daniel Kahneman, Oliver Sibony and Cass Sunstein is about the variability of expert human judgment in responding to the same information regarding a specific case and/or large variations in rates of medical diagnoses, lengthy criminal sentences, foster care placement and many other professional judgments when making decisions on similar types of cases.

Examples of “noise” in child welfare include:

An intake caseworker screens out a CPS report in the morning, while another screener in the same office accepts the same report for investigation in the afternoon.
An intake unit in one office consistently screens out 45-50% of CPS reports; another intake unit 30 miles away usually screens out 65-70% of reports.
CPS caseworkers in the same unit or office assigned investigations on rotation often have rates of dependency filing and child removal that vary by 2-1 or greater. Child welfare offices in the same region often have dependency filing rates that vary by 3-1 or higher.
Substantiation decisions are extremely “noisy;” decisions to substantiate allegations of child maltreatment often depend less on evidence than on the anticipation of future legal action, or a caseworker’s desire to collaborate with parents on a voluntary basis, and other factors as well.
Some CPS caseworkers and units rarely open cases for in-home services, while other caseworkers or units provide in-home services on 50% or more of investigations/assessments.
Reunification rates among offices in the same state are “noisy,” in part because some (but not all) communities have Family Treatment Drug Courts or Safe Baby Courts, and/or parent mentors while other offices lack services/ resources to support safe reunification.
Foster care units in the same state and different states and large counties may have vastly different rates of termination of parental rights (TPR) and adoption. A recently released study (Edwards, et al, 2021) found that Maricopa County, Arizona (Phoenix) had a TPR rate 20 times higher than New York City.
Child welfare experts from various professions given common child protection scenarios have modest (at best) levels of agreement regarding out-of-home placement decisions, with the greatest level of agreement when the scenarios describe severe abuse and the least agreement in marginal cases.

“Noise” in professional judgment occurs in other professions as well:

“Medicine is noisy. Faced with the same patient, different doctors make different judgments about whether patients have skin cancer, breast cancer, heart disease, tuberculosis, pneumonia, depression and a host of other conditions.”
Professional forecasters of all types “offer highly variable predictions …”
Personnel decisions are noisy, and “Performance ratings of the same employees are highly variable and depend more on the person doing the assessment than on the performance being assessed.”
Bail decisions are noisy. “Judges also differ markedly in their assessment of which defendants present the highest risk of reoffending.”
Forensic science is noisy, including fingerprint identification, according to Kahneman, et al.
When judges have discretion in the sentencing of persons convicted of crime, their decisions are shockingly noisy. In one famous study involving hypothetical scenarios, “A heroin dealer could be incarcerated from one to ten years. Punishments for a bank robber ranged from five to eighteen years in prison.” (Kahneman, et al, 2021)

Furthermore, “All these noisy situations are the tip of a large iceberg. Wherever you look at human judgment you are likely to find noise” Kahneman and his co-authors assert.

Is “noise” in human judgment always bad?

“Noise” in human judgment violates the bureaucratic ethos that values consistent application of eligibility criteria and the provision of services or benefits based on policies and procedures. Consistency in screening decisions is a “no brainer;” how can it be acceptable for one caseworker to screen out a CPS report at 10 AM and another caseworker screen in the same report a few hours later? However, consistency is not necessarily a good thing. It is possible for child welfare offices to make consistently bad decisions regarding certain types of cases, e.g., alleged abuse of adolescents, or types of child maltreatment, e.g., emotional abuse and neglect.

Some degree of variability in judgment is reasonable when the best option is uncertain. Regardless of frequent claims to the contrary, it is currently unknown whether a low rate of out-of-home placement is better than a higher rate, given that placement decisions are (and ought to be) affected by available safety services and community resources, e.g., access to residential care treatment programs that accept both a mother and baby. The assumption of some child advocates that a lower entry-into-care rate is always better than a higher rate is based on values, not research, which is not to assert that a higher rate of child removal is better. The one study from Illinois often cited to support the view that a “lower foster care rate is better” (Doyle, 2007) had a flawed methodology, and has been contradicted by a more recent study completed in Michigan. In addition, no conclusion regarding foster care outcomes should be based on a single study when there have been dozens of studies on this subject, with mixed results.

It is doubtful that the “noise” in substantiation decisions has any demonstrable effect on other child welfare outcomes except in states that require substantiation to make families eligible for services. On the other hand, a 20-1 difference in TPR rates between two of the largest U.S. counties suggests that permanency planning policy frameworks embodied in law have less influence on decision making than the values and beliefs of child welfare leaders and judges in various jurisdictions and on the social milieu of communities.

Causes of “noise”

Kahneman, et al, make a strict distinction between bias and “noise,” but nevertheless acknowledge that biases are a source of variability in human judgment. The influence of bias is far from the whole story, however. Human judgment is affected by mood, stress level, social pressure, community norms, resources, as well as by information, beliefs, values and goals. Regarding the influence of mood on judgment, Kahneman, et al, maintain that “you are not the same person at all times. As your mood varies … some features of your cognitive machinery vary with it …” For example, people in a good mood are more likely to trust their intuition and make quick confident decisions. “Physicians are significantly more likely to prescribe opioids at the end of a long day,” based on a study of nearly 700,000 primary care visits, according to Kahneman.

The widespread idea among developers of assessment tools that caseworkers gather information, then interpret information and apply their understanding to achieve well defined program goals is a misunderstanding of how decision-making works much of the time. Caseworkers may decide to provide services to a family, and then complete a risk assessment instrument in a way that makes a family eligible for services; or decide to close a case and document risk factors to justify this decision. In past years, studies of family preservation services (FPS) found that caseworkers sometimes asserted that a child was at imminent risk of foster care placement to make the family eligible for FPS, when only a small percentage of children in the control group referred for FPS services were ever placed out-of-home, regardless of the lack of FPS services.

A caseworker’s values and goals often affect the gathering and interpretation of evidence regarding child maltreatment. For this reason (and others) use of

risk assessment tools have had small effect (at best) at reducing “noise” in child protection decisions. The assumption of program developers and scholars that a model of information processing that moves logically from information to analysis and interpretation of information to application of this information and analysis to practice misrepresents how caseworkers and supervisors process information during interactions with families.

In Thinking Fast and Slow (2011), Kahneman discuses the importance and impact of stories in System 1 ( i.e., quick automatic) decision making. System 1 takes a small amount of information ( “What You See Is All There Is”), converts this information into a narrative that connects the dots, and then quickly jumps to a conclusion. System 1 is connected to intuition, which is accompanied by a strong sense of conviction (“I’m sure I’m right”). System 1 is bad at statistics, but loves stories, with the result that System 1 is given to unlikely predictions. Risk assessments generated from use of actuarially based assessment tools are likely to have far less influence on CPS decision making than impressions and stories based on interactions with children and parents, and on emotional reactions to information regarding child maltreatment.

Beliefs and attitudes regarding child protection, children’s and parent’s rights, substance abuse, mental health, DV, foster care, social justice have great influence on decision making independent of information gathered during investigations or assessments. Caseworkers’ feelings of like or dislike of parents sometimes affect their willingness to listen to and assimilate new information incongruent with their feelings. Halo effect, ( i.e., “people I like can do no wrong and people I dislike have no virtues”), one of twenty or so heuristic biases discussed in Thinking Fast and Slow, can have a large effect on CPS decision making. The willingness and capacity to see people as they are regardless of personal feelings requires constant practice and self-reflection, even for experienced professionals. Few, if any, caseworkers come to child protection or other casework positions with this ability. For this reason, critical thinking is as much about the capacity for self-reflection as it is about analytical skills and extraordinary deductive prowess, per detective fiction.

How to become a better decision maker

Postpone judgment until there is enough credible information to make a sound decision. Kahneman, et al, assert that “In general, we jump to conclusions, then stick to them.” Once a decision maker has reached an initial conclusion, he/she is likely to look for evidence that they’re right and ignore evidence that they may be wrong. Confirmation bias makes it difficult to reconsider a mistaken assessment. Furthermore, high IQ is not a protection against confirmation bias. No one can overcome confirmation bias without the help of others who interpret evidence differently.
Do not get stuck with either/or decisions, e.g., place or don’t place a child. Widen your options, per the advice of Chip and Dan Heath in Decisive: How to Make Better Decisions in Life and Work.
Separate facts from values, predictive judgment from wish fulfillment fantasies and judgments regarding specific behaviors from feelings about the people involved.
Prepare to be wrong. The Heath brothers and Kahneman underline the fallibility of experts whose predictions of the future are unlikely to be much better than those of non-experts. Skilled decision makers identify “trip wires”, i.e., indicators that a decision was a mistake instead of engaging in denial and refusing to admit error.

The exercise of dispassionate intelligence is a great asset in decision making, but this does not mean that Mr. Spock-like cold rationality results in the best judgments. The most effective professionals combine cool rationality with passionate commitment to program goals or worthy causes. There is also a small number of professionals and advocates who combine rational intelligence, strong commitment with an engaged heart that responds empathetically to the suffering of vulnerable people. It is common for decision makers in the human services to have one of these virtues, uncommon (but not rare) to have two of them, and exceedingly rare to have all three, i.e., the habit of dispassionate judgment, strong commitment to a cause or vulnerable population and a consistent empathetic response to the suffering of strangers and other persons who are not members of their extended family.

How to reduce unwanted “noise” in child welfare

The variability of judgments in child protection and child welfare cannot be

eliminated or significantly reduced solely through policy and procedural manuals, assessment tools, safety frameworks or other written guidelines. “Noise” reduction requires repeated practice with the application of screening criteria, assessment tools, placement guidelines, etc., preferably in group meetings that allow caseworkers and supervisors to interact regarding their differences of opinion regarding specific intakes or cases. Consensus building exercises with intake caseworkers and CPS caseworkers regarding screening decisions or placement decisions can be an effective approach to reducing “noise.”

Child welfare systems that seek to reduce inconsistency of decision making and improve practice must provide timely feedback to caseworkers and supervisors regarding the outcomes of their decisions. Absent timely feed-back regarding the outcomes of decisions, it’s difficult to learn from experience. However, feedback is not useful unless the outcome in question, e.g., child safety, is measured in a meaningful way. Currently, child welfare agencies have no idea whether they are improving, staying the same, or becoming worse at child protection, a ridiculous situation given the amount administrative data available in child welfare systems.

Kahneman is a fan of “back of the envelope” actuarial models that apply a few simple rules consistently to predictive judgments, rather than utilizing complex rules tailored to individual differences. Kahneman acknowledges the trade-off: the use of a few simple rules reduces “noise” but at the cost of ignoring potentially important differences among cases. Tolerating a greater degree of “noise” may increase the usefulness of assessment tools when applying them to specific cases.

Child welfare agencies should disseminate summaries of important research

and studies and (on demand) the studies themselves as a means of grounding decision making in knowledge that is widely disseminated in the agency. It is not enough to measure compliance with policies and manage to performance indicators to reduce “noise.” Every metric used to evaluate the performance of units, offices and regions can and will be “gamed” to impress managers.

It is also possible to make use of “noise” among units to compare the effectiveness of different practices. Some variability in key judgments is useful; but too much “noise” makes a mockery of policy frameworks. ©

References

Doyle, J., “Child Protection and Child Outcomes: Measuring the Effects of Foster Care,” American Economic Review, vol. 897, No. 5, December 2007.

Edwards, F., Wakefield, S., Healy, K. & Wildeman, C., “Contact with Child Protective Services is pervasive but unequally distributed by race/ ethnicity in large US counties,” Proceedings of the National Academy of Science, vol. 118, No. 30, 2021.

Gross, M. & Baron, E.J., “Temporary Stays and Persistent Gains: The Causal Effects of Foster Care,” 2020, available on-line at SSRN, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3576640.

Heath, C. & Heath, D., Decisive: How to Make Better Choices in Life and Work, (2013), Crown Business, New York City.

Kahneman, D., Sibony, O. & Sunstein, C., Noise: A Flaw in Human Judgment, (2021), Little, Brown Spark, New York City.

Kahneman, D, Thinking Fast and Slow, (2011), Farrar, Straus & Girard, New York, New York.

©Dee Wilson

For past Sounding Boards click here

deewilson13@aol.com