Study retention as bias reduction in a hard-to-reach population

This contribution is part of the special series of Inaugural Articles by members of the National Academy of Sciences elected in 2015.
Contributed by Bruce Western, March 29, 2016 (sent for review October 23, 2015; reviewed by David Harding and Becky Pettit)
May 9, 2016
113 (20) 5477-5485
Profile
Profile of Bruce Western
Paul Gabrielsen

Significance

Very poor and otherwise disadvantaged populations pose a challenge for research. The circumstances of extreme disadvantage—homelessness, acute economic insecurity, untreated addiction or mental illness, for example—are urgent problems but also make research subjects elusive or unwilling. We examine study retention in a deeply disadvantaged sample of newly released prisoners, achieving a 94% response rate over five survey waves over 1 y of follow-up. The high response rate indicates strategies for study retention and points to biases associated with nonresponse. Small-scale data collections can illuminate very harsh conditions of contemporary poverty by observing repeated and co-occurring episodes of homelessness, institutionalization, mental illness, and physical disability that are largely unobserved with conventional methods.

Abstract

Collecting data from hard-to-reach populations is a key challenge for research on poverty and other forms of extreme disadvantage. With data from the Boston Reentry Study (BRS), we document the extreme marginality of released prisoners and the related difficulties of study retention and analysis. Analysis of the BRS data yields three findings. First, released prisoners show high levels of “contact insecurity,” correlated with social insecurity, in which residential addresses and contact information change frequently. Second, strategies for data collection are available to sustain very high rates of study participation. Third, survey nonresponse in highly marginal populations is strongly nonignorable, closely related to social and economic vulnerability. The BRS response rate of 94% over a 1-y follow-up period allows analysis of hypothetically high nonresponse rates. In this setting, nonresponse attenuates regression estimates in analyses of housing insecurity, drug use, and unemployment. These results suggest that in the analysis of very poor and disadvantaged populations, methods that maximize study participation reduce bias and yield data that can usefully supplement large-scale household or administrative data collections.
Severe poverty and related disadvantage pose a fundamental challenge for social research. Deeply disadvantaged populations are underenumerated and at high risk of survey nonresponse (13). If markers of disadvantage—such as unstable housing or untreated drug addiction—are themselves causes of nonresponse, analysis of disadvantage may be biased.
Under current conditions of historically high imprisonment rates, incarceration has become a regular part of the hardship associated with American poverty (4, 5). Men and women who go to prison present a range of co-occurring disadvantages and vulnerabilities that make them difficult to contact or unwilling to participate in research studies. Involvement in crime and the criminal justice system also makes study subjects elusive for researchers. As a result, follow-up studies of the formerly incarcerated have suffered from high rates of attrition that are likely correlated with outcomes of key interest. The incarcerated population has been called “invisible men” because of their underenumeration in social survey estimates of socioeconomic well-being (6). Because of their deep marginality in social and economic life, invisibility in social science data collection also extends to those who have moved from incarceration to communities.
We explore survey nonresponse in a highly disadvantaged population using data from the Boston Reentry Study (BRS), a small longitudinal survey of newly released prisoners. Earlier longitudinal studies of released prisoners suffered from 30% to 60% attrition over a 1- to 2-y follow-up period (7, 8). In contrast, the BRS followed Massachusetts state prisoners through the first year of prison release, achieving a response rate of 94%.
The unusually high response rate allows us to explore three main challenges to research and data collection in highly disadvantaged populations. First, we document extreme social insecurity in the BRS sample that is closely associated with a high level of contact insecurity—the frequent changing of residential addresses and phone numbers that makes nonresponse more likely. Second, we describe a variety of strategies for data collection that yield a high rate of study participation. Third, we show that the risk of survey nonresponse is associated with deep social disadvantage including life histories of drug addiction and mental illness. In this setting, nonresponse contributes to a loss of statistical precision in data analysis and nonresponse is nonignorable, biasing estimates of risk and vulnerability in the population.
Our analysis focuses on the formerly incarcerated but the methods and findings reported here are generally relevant to other hard-to-reach populations that are unstably housed, difficult to contact, or otherwise weakly attached to private households. Although large household surveys have been central to the study of poverty and related topics, our results suggest the utility of specialized, and often small-scale, data collections for highly marginal populations.

Nonresponse in Disadvantaged Populations

Extreme poverty and other deprivation produce social insecurity and distrust of authorities that frustrate conventional survey methods. Social insecurity describes the unpredictability of daily life associated, for example, with housing and financial instability or poor health. Homelessness, unpaid phone bills, and other everyday catastrophes make it difficult for study subjects to be contacted or to keep appointments. Disability or mental illness can damage normal functioning, impeding a subject’s capacity to arrange or complete interviews. Although study subjects may be willing to participate despite the obstacles associated with social insecurity, distrust of authorities may erode such willingness. Ethnographers have described distrust of authority in high-poverty field sites and the subsequent necessity of gaining trust to successfully collect data (9, 10). Writing about the poor residents in New York’s Spanish Harlem, Philippe Bourgois (9) describes their skepticism about assurances of confidentiality and suspicions that researchers are in fact undercover police. Distrust escalates to the active evasion of survey researchers where local residents are involved in drug dealing and other illegal activity.
Social insecurity and distrust of authorities create problems of underenumeration in a cross-section, and attrition in longitudinal studies. Marginalized groups—such as the homeless, drug users, or sex workers—are significantly underrepresented in cross-sectional household surveys. Researchers have thus used methods such as snowball and respondent-driven sampling to extend coverage of hard-to-reach groups (11, 12).
This paper focuses on attrition in longitudinal surveys. Even when they can be adequately sampled, disadvantaged subjects are at high risk of nonresponse in follow-up studies (13, 14). Researchers report low rates of follow-up among drug users (1517), low-income minority parents (3, 18), women at risk for HIV infection (19), the homeless mentally ill (14), and youth engaged in crime and delinquency (13, 20).
Prison releasees are unusually disadvantaged. Besides their involvement in crime and low levels of schooling, people who serve time in prison suffer from high rates of mental illness and drug addiction (21, 22). After prison release, the formerly incarcerated are often homeless or insecurely housed (23). Michigan parolees, for example, were found to average over six residential moves over about 2.5 y after prison release (24). A criminal record itself can contribute to residential mobility where a rental application is denied after a background check or where a residential program is mandated following a parole violation (25). Thus, former prisoners have been found to be more likely to be living in group quarters (shelters or other transitional housing) than to be continuously resident in stable households (26). Study subjects may also be deliberately elusive when on the run with outstanding arrest warrants, evading both researchers and police (27, 28). The stigma of a criminal record also contributes to unstable and undocumented employment and reduces access to social programs that can provide a source of administrative data (29, 30). In short, the formerly incarcerated are weakly connected to mainstream social institutions, making them inaccessible to standard data collections and hard to follow even in specialized field studies.
Nonresponse related to incarceration has been observed in medical and demographic research, and the consequences are likely wide-ranging in analyses of poor, prime-age, men (6, 31, 32). The difficulty of data collection can be seen from levels of nonresponse in studies of incarcerated populations. Table 1 lists nine studies since 1999 that have collected data from samples of newly released prison and jail inmates. The challenges of study participation are reflected in the retention rates at exit interviews. In the earliest study, in New York in 1999, only around half of a sample of released prison and jail inmates were reinterviewed after 30 d. Most reentry studies record retention rates between 50% and 70%. The Michigan study provides an important exception, retaining 86% of a small sample of parolees over a 2-y follow-up period.
Table 1.
Retention rates of longitudinal studies of released prisoners
Study siteYear(s)Sample sizeFollow-up, moRetained at exit, %
1. New York199988156.0
2. Baltimore2001–03324632.1
3. Chicago2002–034001649.5
4. Cleveland2002–034241269.0
5. Houston2004–056768–1055.9
6. 14 states2004–072,3911568.5
7. Southeastern Michigan2007–09222486.4
8. Newark, NJ2012–13152370.0
9. Boston2012–141221291.0
Studies are described in refs. 19 and 3339.
The final study in Table 1, the BRS, forms the focus of this paper. Following a sample of Massachusetts state prisoners over a year after prison release, the BRS achieved a retention rate of 91%. With a baseline sample size of N=122, and five scheduled interviews for each respondent, the design included 610 interviews. A total of 576 interviews were completed, yielding an overall response rate of 94.4%.

Bias and Nonresponse

High rates of nonresponse reduce the precision of estimates and may be a source of bias. With the small sample sizes typical of the studies of hard-to-reach populations, even low rates of nonresponse produce relatively large increases in sampling variance. Because survey nonresponse in disadvantaged populations is related to the conditions of disadvantage themselves, nonresponse is called nonignorable and can be a source of selection bias (40). Say drug users are difficult to contact for interviews and substantive interest focuses on drug use after incarceration for respondents with a history of addiction. For respondent i at wave t, drug use measured by yit can be written as a function of xi, a dummy variable indicating drug addiction that is measured at baseline and is completely observed. Addiction may also be a risk factor for nonresponse.
The regression of interest is
yit=β0+β1xi+eit,
[1]
where eit is a random error. To see the effects of nonresponse we write a response indicator, Rit, that equals 1 if a survey is completed and 0 in the case of nonresponse. A sample selection model writes
Rit={1ifηit00ifηit<0,
where ηit describes the respondent’s propensity to be observed,
ηit=δ0+δ1xi+δ2zi+uit,
[2]
zi is another completely observed covariate predicting nonresponse, and uit is a random error. The dependence of nonresponse on the level of drug use yit is usually modeled by allowing the errors eit and uit to be correlated. Note that Eq. 1 cannot be fit directly because yit is unobserved when Rit=0. Analyzing just the observed data, the conditional expectation of the dependent variable depends on covariates and an adjustment factor reflecting the predicted probability of being observed,
E(yit|xi,zi,Rit=1)=β0+β1xi+h(δ0+δ1xi+δ2zi),
[3]
where the sample selection adjustment, h, is a function whose form depends on assumptions about the bivariate distribution of eit and uit (41, 42).
In the naive regression of the observed values of drug use on addiction—a regression of yit on xi—bias results from the correlation between the covariate, xi, and the residual that includes the omitted term, h. Below, we assess the magnitude of selection bias in a (near) completely observed dataset reporting estimates that exclude observations that are at high risk of nonresponse. The key intuition is that if drug users with a history of addiction are difficult to contact for interviews, the true level of drug use among drug addicts will be understated by the observed data and the regression coefficients will be underestimated. In general, where measures of insecurity are outcomes of interest and insecurity is a mechanism for survey nonresponse, regression relationships will be attenuated.*
In the study of marginal populations, risks and adversities such as untreated addiction, mental illness, or housing insecurity may all contribute to attrition from panel surveys and are also outcomes of key interest. In these cases, nonignorable nonresponse biases sample statistics but can be difficult to diagnose in the absence of additional information such as a refreshment sample or population statistics for comparison (43).

Study Retention in a Cohort of Prison Releasees

The BRS was designed to collect data intensively in the early stages of prison release. All prisoners who planned to live in the Boston area were eligible for the study and subjects were recruited with an information sheet distributed by prison staff. The sample included men and women, many with long histories of incarceration. Some respondents were released to probation or parole supervision, but others had completed sentences and returned to their communities without supervision.
The research design involved five face-to-face interviews over a year. The first interview was held in prison a week before release, and follow-up interviews were scheduled at 1 wk after release, then at 2, 6, and 12 mo. The in-prison baseline interview was conducted by a university researcher with a staff member from the research unit at the Department of Correction (DOC). At the baseline interview, contact information was obtained for the first follow-up interview 2 wk later in the community. All follow-up interviews were conducted by university research staff who worked in pairs. If a respondent was reincarcerated, a follow-up face-to-face interview was conducted inside a correctional facility.
The offense profile, criminal history, level of risk, and demographic characteristics of the BRS sample are similar to those for the Boston-area reentry population as a whole, suggesting little nonrandom selection based on observable characteristics (39). After release, the 1-y rate of reincarceration in the BRS sample (22%) was the same as that for the state’s whole reentry population (44). The BRS respondents represent a one-quarter sample returning to the Boston area from Massachusetts state prisons. Like the national prison population, the sample is black or Latino with an average age in the mid-30s, having been incarcerated 2 to 3 y.
Characteristics of the BRS sample, reported in Table S1, place respondents at high risk of interview nonresponse. The sample was entirely urban, living in and around Boston, mostly male, African American and Hispanic, with very low levels of schooling. Many had histories of mental illness and heavy drug use and had previously lived in unstable housing, either dividing time between multiple residences or living on the streets or in shelters or other group quarters. In contrast to most prison reentry studies, BRS respondents were not recruited from probation or parole agencies and over a third of the sample was released to the street with a completed sentence and no postrelease supervision. After release, nearly a quarter of the sample returned to incarceration in the 1-y follow-up period, two-thirds lived in some kind of unstable housing, four out of five were enrolled in a food-stamp program, and a quarter were continuously unemployed. In short, like most newly released prisoners, the BRS respondents experienced a high level of social insecurity and were often difficult to contact for follow-up interviews.
Table S1.
Means and percentage distribution for selected characteristics of BRS respondents (N = 122)
Respondent characteristicsPercentage
Demographics 
 Average age, y36.5
 Female12.3
 Black50.1
 Hispanic18.9
 High-school dropout59.8
Baseline characteristics 
 Substance abuse history54.1
 Mental illness history44.3
 Average time in prison, mo32.4
 Unemployed before arrest41.0
 Unstable housing before arrest19.7
Postrelease characteristics 
 No probation/parole after release38.5
 Incarcerated after release22.1
 Unstable housing after release62.0
 Enrolled in food-stamps program after release83.5
 Continuously unemployed after release24.8
Baseline characteristics are based on self-reports measured at the first, in-prison, interview.
Table 2 reports the pattern of study participation in the BRS. Five out of 122 respondents were missed at the first follow-up survey and nonresponse grows to 11 respondents by the 12-mo exit interview. Only a minority of nonrespondents were dropouts in the sense of being permanently lost to follow up. Although the study sustained a high response rate, data on the days to interview indicate the increasing difficulty of completing interviews. The median days to interview shows that about half the sample were interviewed on the scheduled follow-up day or earlier. Latecomers, at the 75th percentile of the days-to-interview distribution, got progressively later even though nonresponse did not markedly increase after the 2-mo follow-up. (The full distribution of the days-to-interview for each survey wave is shown in Fig. S1.)
Table 2.
Study participation across five waves of the BRS
    Days late for scheduled interview
Survey waveNo. of interviewsResponse rate, %Respondents attritted, NMedian75th percentile
Baseline122100.0
1 wk11795.9003
2 mo11392.6249.5
6 mo11392.64618
12 mo11191.05830
 Total57694.411
Survey nonresponse includes all those who could not be contacted or scheduled for a follow-up interview and those unreachable through incarceration or hospitalization as a percentage of those eligible to be interviewed. Attrition in a given wave is defined as missing the current interview and all subsequent interviews. Dashes for some table entries indicates not applicable.
Fig. S1.
Density plot showing time to interview from prison release for four follow-up interviews. Interviews were scheduled for 1 wk and 2, 6, and 12 mo after prison release (BRS, n = 122).
Survey methodologists distinguish between nonrespondents and “reluctant respondents” who are either difficult to contact or hesitant to consent to an interview (45). We can think of the latecomers, in the top quartile of the time-to-interview distribution, as reluctant respondents who are at risk for nonresponse without additional efforts at study retention. Latecomers and nonrespondents consist mostly of those who were just intermittently missing or late for one or two waves (85%) and a smaller number who were persistently late or missing in at least three of the four waves (15%).

Social Insecurity and Contact Insecurity.

The social insecurity of hard-to-reach study subjects is linked to nonresponse by contact insecurity. When contact information—phone numbers and residential addresses—is intermittently unavailable or changes frequently, survey nonresponse is more likely. Table 3 reports the cumulative percentage of respondents who experienced contact insecurity by the time of each scheduled follow-up interview.
Table 3.
Cumulative percentage of respondents experiencing contact insecurity by 1 wk and 2, 6, and 12 mo after prison release
 Percent experiencing event by
Type of contact insecurity1 wk2 mo6 mo12 mo
No phone (excluding incarcerated)5.08.212.321.3
Changed phone56.882.890.295.9
Unstable/unknown residence36.840.250.159.0
Changed residence40.254.170.584.4
New charge/arraignment0.05.713.933.6
Entered prison or jail0.81.69.022.1
Data from the BRS (n = 122). Unstable residence includes staying in multiple residences, treatment programs, transitional housing, shelters, correctional facilities, and homeless or on the street. Data on new charges and prison or jail stays are drawn from administrative records and thus are complete for the whole sample.
We mainly contacted respondents by phone. Whereas most respondents provided the number of a family home or program at the baseline interview, half of the sample obtained a cell phone within seven days of release. Respondents often informed researchers of their new number when we called to schedule the 1-wk interview. After the first week out, 23 out of 122 respondents maintained the same phone number throughout the year out of prison. These 23 respondents with a high level of contact security completed all postrelease interviews and were more likely to be interviewed on schedule at each wave of the survey.
More commonly, phone status was unstable. Respondents changed phone numbers eight times on average over the course of the study. Primary phones failed to work when bills went unpaid or when a respondent left a residential program or returned to incarceration. Some respondents obtained new phones when they started working or changed their phone numbers every few months. Others reported that regularly changing phones was a habit picked up when they were involved in illegal activity and feared their phones were being tracked. In addition, a number of phones went out of use when they were lost or stolen, particularly in the chaotic environments of homeless shelters and transitional housing. About a fifth of respondents were without phones at some point during the year-long follow-up period, and nearly all changed their phone numbers at least once (Table 3).
In addition to being hard to telephone, three-quarters of respondents changed residence at least once and over half lived in shelters or other insecure housing. It was sometimes difficult for us to verify residence for respondents in transitional or other institutional housing because program staff could not disclose the names of housing occupants. Criminal justice involvement may also contribute to nonresponse for respondents with outstanding warrants or who are detained while awaiting trial. By the end of the follow-up period a third of respondents had been charged for a new offense and a fifth had spent some time back in custody.

Strategies for Study Retention.

In this challenging setting for study retention, a variety of strategies were adopted to reduce nonresponse. In a context of extreme contact insecurity, we collected an extensive array of supplementary contact information, maintained a high level of contact between interviews, and tried to increase the willingness of respondents to participate. Four specific features of the BRS research design were intended to promote study participation.

Interview incentives.

Respondents were paid $50 for each interview. Incentives of about this size were previously found to increase participation among low-income respondents and parolees (37, 46). Similar to another recent study of parolees in Michigan, BRS respondents were paid in cash at the conclusion of an interview (37). In the case of reincarceration, research staff deposited $50 into the respondent’s prison commissary account. At the first meeting in the community, respondents received a total of $100, $50 for the baseline interview in prison and $50 for the 1-wk follow-up. One hundred dollars was a significant payment for people with little income, particularly for a fifth of the sample who were released with less than $100 in their commissary accounts. When asked why they were participating in the study, a quarter of respondents cited the interview payment as a strong incentive at the 1-wk interview. The interview incentive seemed to be less crucial to study retention over time. By the 12-mo interview, only about 10% of respondents indicated they were participating for the money.
Besides the cash incentives, interviewers presented a nonjudgmental, conversational posture in interactions with respondents. This involved unscripted probes and digressions in interviews motivated by a curiosity about the respondent’s life and welfare. The approach was echoed in unstructured communications at the conclusion of the interview (see also ref. 47). Respondents reported that this approach contrasted with official interactions with criminal justice authorities and that the attitude of interviewers provided an additional incentive for participation. Reflecting the quality of the interviewer–respondent relationship, a third of the sample reported they were participating in the research because they were committed to completing the study or because they liked the interviewer. Another 20% said it was helpful for them to be able to share their experiences after incarceration.§

Phone check-ins and letters.

Research staff also conducted regular phone check-ins with study respondents. Between the baseline, 1-wk, 2-mo, and 6-mo interviews, interviewers phoned respondents every few weeks. Between the 6-mo and the 12-mo waves, interviewers checked in each month by phone. Phone check-ins were used to update the respondents’ residential information and to maintain contact and rapport. Phone contact involved both conversations and texting. Where phone contact was lost, we emailed or wrote letters to the respondents, often using address information previously provided by family members and friends. A strongly nonjudgmental approach in interviews combined with frequent phone check-ins were intended to build rapport and trust between interviewers and respondents.

Secondary contacts and interviews.

At the baseline interview, we asked respondents for a list of family members or friends who could help us to stay in contact after prison release. Similar protocols were used elsewhere (13, 15, 37). The list of secondary contacts was updated at each interview, sometimes adding new names or new phone numbers for existing contacts. Secondary contacts were particularly useful when their contact information was stable, when they were supportive of the respondent, and when we developed trust through repeated interaction. Thus, mothers and other older women were particularly helpful because their own phone numbers rarely changed. Mothers and other supportive secondary contacts also tended to stay in touch with respondents, particularly in moments of crisis when the risk of attrition was acute. Repeated inquiries into the whereabouts and well-being of respondents also helped reduce suspicion and promoted active support of the study retention effort.

Justice agencies and community contacts.

When conventional retention strategies were exhausted, a variety of justice agencies and community contacts helped reestablish and maintain contact with the study subjects. Each week, the DOC notified the study team of respondents with new charges, violations, or reimprisonment. We also queried probation, parole, and police departments for phone numbers or addresses for hardest-to-reach respondents. Respondents had earlier consented to such contact and the queries were designed to be unobtrusive both for the agencies and the respondents. Parolees had been the focus of earlier reentry studies and below we study whether parole/probation status is associated with study retention.
For those without probation or parole supervision, we tried to reestablish contact through a variety of street and community workers operating in inner-city neighborhoods where study respondents resided. Streetworkers (gang outreach workers drawn from local communities) were especially helpful for reestablishing contact with younger respondents who were associated with gangs or other street groups. In some cases, streetworkers were able to provide a direct link—through a working phone number or current address—for a study respondent who had gone missing. More often, however, streetworkers connected the research team to a third party (say an ex-girlfriend, neighborhood acquaintance, or a former youth worker) that could arrange contact with a missing respondent. When subjects dropped out, community contacts also offered explanations for nonresponse. For example, we learned that one subject had developed extreme paranoia about being “set up” following a shooting in his area in between survey waves and was reluctant to meet or talk with all but a small group of trusted friends.
Besides the four main strategies for study retention, survey records were also linked to court data on criminal records. Administrative data from courts and parole agencies have been valuable sources of information in other research on the transition from prison to community (25). Administrative data yield the considerable advantages of large samples and long follow-up periods at low cost but lack the rich information on covariates and outcomes characteristic of primary data collection. Because administrative data are collected for operational and not scientific purposes, their validity can often only be assessed with additional observation. Whether a drug arrest record reflects criminal conduct or the enforcement efforts of police, for example, requires more information than the administrative record provides. Primary and administrative data collection are thus complementary strategies for studying hard-to-reach populations.

Social and Contact Insecurity in Qualitative Perspective.

In the day-to-day operation of the reentry study, the retention strategies were applied continuously and concurrently, so much of the work of the data collection revolved around maximizing study participation. Consistent with nonignorable nonresponse, contact insecurity was often linked to insecurity in daily life. Omar, a Puerto Rican man in his mid-40s, changed his phone number 17 times over the course of the study. Omar had been a regular heroin user for much of his life. At 16 y old, he suffered brain damage after a car accident and reported it had been difficult to function without drugs since that time. His physical and mental health problems compromised his ability to maintain a steady phone number and to remember dates of medical appointments and study interviews. Still, Omar was eager to participate and on days when he was clear-headed and had a working cell phone he would sometimes call to check in.
For the first 3 mo after incarceration, Omar resided at several addresses and was sometimes homeless on the streets. He rekindled a relationship with a former girlfriend, Jessie, who was just released from prison herself. Jessie had been a heavy drug user but was in better health and helped Omar attend appointments, including survey interviews. She stayed with him on the streets and moved with Omar into a friend’s apartment 4 mo after his release. When Omar’s cell phone was out of service, we would often call Jessie, and she would hand her phone to him so that we could check in.
Omar was sentenced to probation upon release but never reported to his probation officer due to his drug use. Just before his 6-mo interview, he received a warrant for not reporting. Although Omar mentioned his trouble with probation during interviews and phone calls, we received further information from the DOC. Omar’s warrant was dropped on the condition that he complete a residential drug treatment program. However, when we called the program a staff member reported he was no longer there. Thus, while Omar’s legal status was compromised and he was no longer living with Jessie we struggled to reach him to schedule the 12-mo interview.
When Omar and Jessie were both out of contact, we would call Omar’s sister Lena. An older woman in her late 50s, Lena was 16 y old when the family moved from Puerto Rico to Boston. She was more comfortable communicating with Spanish-speaking members of the research staff and had lived in public housing in Boston’s inner city for nearly 15 y. Lena was the most stable person in Omar’s life and he used her residence as his mailing address. Lena would get updated contact information from him every so often when he stopped by her place for meals. After we learned that Omar had left his treatment program, we contacted Lena, who gave us his new cell phone number. Two days later, we conducted Omar’s 12-mo interview. Although the interview took place 2.5 mo late, it provided valuable information about Omar’s housing insecurity, drug use, and probation late in the year after incarceration.
Between the 6- and 12-mo interviews, we completed five phone check-ins with Omar, failed to reach him on 18 call attempts, spoke to his sister or girlfriend five times, and received information on his legal and residential status from local criminal justice agencies. His case points to the necessity of a range of redundant strategies to retain vulnerable study participants and the high risk of nonresponse for people who are struggling with housing insecurity, drug relapse, mental illness, and criminal involvement.
Efforts at study retention are reported for the whole sample in Table S2. Dividing the sample into respondents who were consistently interviewed on time, respondents who were interviewed late, and those who missed interviews shows that the hardest to reach respondents (a third of the sample) received about 50% more retention effort than those who were interviewed on time.
Table S2.
Average number of contact attempts for each respondent, by nonresponse status, between each survey wave of the BRS
Interview statusSuccessful contactsFailed contactsCalls to secondary contactsLetters sentJustice system queriesN
Baseline to 1 wk      
 On-time completion1.480.630.680.030.0090
 Late completion2.001.301.110.190.0427
 Missed1.001.400.600.400.605
1 wk to 2 mo      
 On-time completion3.310.470.150.000.0085
 Late completion2.651.380.270.150.0428
 Missed2.507.501.830.830.339
2 mo to 6 mo      
 On-time completion3.551.280.170.000.0483
 Late completion2.801.630.270.100.2730
 Missed0.756.251.751.501.009
6 mo to 12 mo      
 On-time completion3.773.100.450.050.1586
 Late completion2.927.242.040.760.8025
 Missed1.808.601.200.400.6011
  Total, N1,3318902505358122

A Model of Reluctance and Nonresponse

We can link respondent characteristics to nonresponse across the whole sample with a multinomial logit regression that takes late and missing interviews as response categories. Completely observed covariate characteristics can be taken from the baseline interview and from linked administrative data on arrests and incarceration. Such a model allows us to test two hypotheses. First, we test for an enduring propensity to nonresponse by including lagged dummy variables for late and missed interviews. Coefficients for lagged measures also suggest whether late respondents are at high risk of nonresponse, that is, whether reluctant respondents are potential nonrespondents. Second, we can also assess the association of lateness and nonresponse to measures of extreme disadvantage that are closely related to contact insecurity. These markers of social insecurity include histories of marginal housing, mental illness, and addiction, all measured at the first, in-prison, interview. Contact insecurity is also closely related to time-varying measures of new charges and reincarceration. Finally, to study the risk of nonresponse for those who were unsupervised, a dummy variable is included for respondents released with completed sentences and no parole or probation supervision. (Controls are also included for age, sex, race, and ethnicity.)
More formally, for respondent i in wave t, we can write the probability of being late or missing, pj (j=L,M) relative to being interviewed on time, pO,
log(pjitpOit)=αoj+α1jlit1+α2jmit1+xiβ1j+zitβ2j,
where dummy variables indicating lagged interview status are included for respondents who were late (lit1=1) or missing (mit1=1) in the prior wave, xi is a vector of time-invariant covariates, and zit is a vector of time-varying covariates. (With lagged predictors, analysis is confined to the last three waves of the survey, t= 2 mo, 6 mo, and 12 mo.) Note that the multinomial logit model yields two sets of coefficients (j=L,M), one showing the odds of late respondents relative to those who were on time, and another showing the odds of missing respondents relative to those who were on time.
Multinomial logit results show that respondents who were late or missing in a given wave were likely to be missing at the following wave, suggesting that the propensity to nonresponse is in part an enduring trait that is observable from wave to wave (Table 4). Nonresponse risk is also related to the respondent’s social insecurity. Respondents with histories of drug addiction and mental illness are only tenuously retained in the study. Unexpectedly, living in unstable housing before incarceration is associated with a relatively low risk of nonresponse, when other covariates are controlled. Unusually, the BRS sample also includes a relatively large number of people who have completed sentences and are not supervised by probation or parole. Coefficients for unsupervised status are relatively small and insignificant, suggesting the utility of the sample retention strategies that do not directly involve assistance from community corrections agencies. Estimates also show that the odds of being late for an interview are more than doubled by a new arrest. Respondents who return to prison or jail are also at relatively high risk of a noninterview, even in the current context with a very low nonresponse rate. The positive association of new charges and incarceration with the risk of nonresponse suggests that, in a sample with less retention effort and a lower response rate, survey nonresponse may be strongly nonignorable.
Table 4.
Multinomial logit regression results in an analysis of late and missed interviews compared with on-time interviews
 Model 1Model 2
Independent variableLateMissingLateMissing
Intercept−0.356 (0.66)−2.179** (2.80)−0.675 (1.17)−2.667** (3.21)
Lag late interview0.474 (1.35)1.213* (2.52)0.495 (1.37)1.270* (2.49)
Lag missed interview1.880** (3.30)3.470** (5.95)1.858** (3.30)3.647** (6.40)
Age−0.048** (2.94)−0.031 (1.50)−0.047** (2.76)−0.025 (1.12)
Female−0.056 (0.16)−1.642 (1.78)−0.122 (0.32)−1.673* (2.05)
Addiction0.420 (1.29)1.067* (2.53)0.415 (1.20)0.919* (1.96)
Mental illness0.773* (2.51)0.0554 (0.13)0.850* (2.54)0.0172 (0.04)
Prior unstable housing−0.863* (2.38)−0.818 (1.59)−0.823* (2.09)−0.926 (1.60)
Unsupervised0.253 (0.85)0.326 (0.72)0.239 (0.75)0.529 (1.14)
New charge  0.971* (2.50)−0.144 (0.17)
New incarceration  1.122 (1.91)2.581* (2.38)
Log likelihood−250.246−237.502
Respondent-waves, N366366
Three panels of the BRS. Numbers in parentheses are z statistics. Late respondents are those in the top quartile of the days-to-interview distribution. SEs have been adjusted for clustering. The 1-wk survey wave is omitted to allow lagged measures of interview status. *P < 0.05; **P < 0.01.

Consequences of Reluctance and Nonresponse

We study the effect of survey nonresponse on sample statistics directly by constructing a measure of the risk of nonresponse and examining the sensitivity of sample quantities to different hypothetical levels of missing data. Late interviews are completed with additional retention effort, so treating late interview subjects as potential nonrespondents yields a sample that would have been observed with less retention effort. A summary index of the risk of nonresponse can be constructed using information on the days late for a scheduled interview and actual nonresponse. Because the distribution of days late varies across waves, we transform the measures to percentiles, Pit, within each wave and assign nonrespondents to the highest percentile. An average across the four follow-up waves of the survey yields a time-invariant measure of the risk of nonresponse, P¯i, for each respondent. High values of P¯i indicate respondents who were consistently very late or who missed follow-up interviews.
To study the sensitivity of analysis to hypothetical levels of nonresponse, we pool together the four follow-up waves from the BRS and fit logistic regressions to the pooled data using as dependent variables measures of postincarceration unstable housing, drug use, and unemployment. Regression analysis focuses on the coefficients for a history of drug addiction, mental illness, and unstable housing before arrest. Regressions also control for the wave of the survey, age, sex, and race and ethnicity.
A simulation experiment analyzes subsets of the data where observations are included in the analysis conditional on the nonresponse risk, P¯i. The nonresponse risk, P¯i, varies from 10 to 97 with a mean of 50. We define a nonresponse threshold, T, that is used to drop all observations from respondent i for P¯i>T. With the reduced dataset, we estimate new regression coefficients, βT. These coefficients can be interpreted as the estimates that would have been calculated with a response rate at a level of retention effort at threshold T. Allowing the retention effort to vary from T=97,96,,50, we can examine how coefficients change as nonresponse increases. With T varying over this range, the simulated nonresponse rate varies from 0% to 45%, approximately the range of nonresponse reported in recent longitudinal prisoner reentry studies.
Fig. 1 graphs results from the simulation experiment. Results for each dependent variable—unstable housing, drug use, and unemployment—are shown down the columns. The coefficients—for prearrest measures of addiction, mental illness, and unstable housing—are shown across the rows. Hypothetical response rates are reported on the horizontal axis and coefficients and confidence intervals are shown on the vertical axis. In the model of housing instability, the effects of prearrest mental illness and unstable housing behave consistently with classical sample selection bias in which estimates are significantly attenuated with high rates of nonresponse (Fig. 1, column 1). Neither effect would be detectable with an 80% response rate, but both coefficients become substantively large and significant as the response rate approaches 100%. For postincarceration drug use, the coefficient for prearrest addiction grows by about a third as the response increases from 55% to 100% (Fig. 1, column 2). In the model for unemployment, the coefficient for mental illness illness becomes larger as sample selectivity declines and sample size increases (Fig. 1, column 3). These results indicate how high rates of nonresponse in hard-to-reach populations contribute to sample selection bias, driving null results in the analysis of social disadvantage and vulnerability. Although null results are often incorrectly interpreted as effects of size zero, the current example illustrates how null results can be produced by biased and inefficient data designs that arise for very disadvantaged groups.
Fig. 1.
Logistic regression coefficients for prearrest addiction, prearrest unstable housing, and probation or parole status in panel regressions on postincarceration housing instability, drug use, and unemployment. Coefficients vary with the response rate sorting sample respondents from low to high risk of nonresponse. (Regressions fit an overdispersion parameter adjusting SEs for clustering by survey wave. Dashed lines indicate 95% confidence intervals. A zero coefficient is marked by the solid horizontal line.)

Discussion

Three main conclusions can be drawn from this analysis. First, formerly incarcerated men and women show a very high level of contact insecurity. Most of the BRS respondents experienced significant residential instability, were weakly attached to stable households, changed phones frequently, were periodically out of phone contact, and in some cases returned to incarceration. These respondents would typically be overlooked in conventional household surveys and would experience high attrition in panel surveys without special measures for study retention. High rates of contact insecurity help explain the high rates of nonresponse reported in prior studies of prisoner reentry.
Second, the challenges to study retention in hard-to-reach populations can be successfully addressed through research design. Successful retention requires a variety of redundant strategies for minimizing attrition. Key strategies in the BRS included significant interview incentives, a nonjudgmental approach to interviews along with frequent phone check-ins that helped build rapport, lists of secondary contacts, and in the last resort, justice system and community streetworker queries. These measures provided strong incentives for participation and helped quickly find those whose contact information was lost.
Third, contact insecurity and the risk of survey nonresponse were closely linked to the social risks and vulnerabilities of scientific interest. Thus, the most likely nonrespondents had histories of drug addiction and mental illness and were also more likely to be arrested and incarcerated after prison release. Because the probability of noninterview is related to extreme social disadvantages, nonresponse is strongly nonignorable. Regression estimates were highly sensitive to survey nonresponse in analyses of housing insecurity, drug use, and unemployment. Regression estimates were attenuated at hypothetically high rates of nonresponse, often around 35–45%.
Earlier studies of nonresponse in hard-to-reach populations report that significant resources are required to collect data from the most elusive respondents (13, 18). Researchers have suggested that the benefits in data quality that result from a higher retention rate may not be worth the cost of tracking down those who are most difficult to contact (18).
Our findings point to three additional considerations when evaluating the benefits of a high rate of study retention. First, retaining the most elusive respondents does require significant effort, but developing a culture within the research project that promotes rapport and connectedness with study subjects can improve retention at little extra financial cost. Second, the effort to sustain 100% retention likely improves the quality of interviews that are completed. Where interviews are conducted in a climate of trust with an interviewer that a respondent has come to know through numerous interactions, survey responses are likely to be more forthcoming and complete, particularly in sensitive areas. Third, as our field period progressed, 100% study retention became as much a humanistic as a methodological objective. A single missing observation out of nearly 600 interviews would barely increase bias or SEs, but an opportunity would be missed to register the voice of a respondent who has rarely been heard in science or policy. In very poor or socially marginal research sites, building trust and connection with respondents yields the benefits of accurate measurement and a high response rate but also gives voice to those who are largely invisible with conventional survey methods.
Despite the challenges, the analysis of extreme poverty and other deprivation relies heavily on national surveys and administrative data (4951). The problems of underenumeration and survey nonresponse are well-documented with these data (45, 52). Large samples offer no relief when the data design is biased. Our analysis shows that large-scale data collection can be usefully supplemented by specialized studies that focus on those who are missed by a household sampling frame and are at risk for attrition in a panel study. Counting those who are the most difficult to count holds the promise of observing the deepest disadvantage. In these very marginal social spaces—where nonresponse is strongly nonignorable—special strategies for study retention are critical for reducing bias and accurately observing extreme material hardship and social insecurity.

Materials and Methods

Informed consent was obtained from all research subjects participating in the Boston Reentry Study. This research was approved under protocol number 21354 by Harvard University's Committee on the Use of Human Subjects.

Acknowledgments

We thank David Harding, Becky Pettit, and Emily Wang, who provided excellent comments on earlier drafts of the paper. We gratefully acknowledge the hundreds of men and women who shared their time and life experience to improve our understanding of incarceration in America, and the Massachusetts Department of Correction, which provided access to correctional facilities and advice and collaboration throughout the research. The data for this paper are from the Boston Reentry Study, a research project conducted by Bruce Western, Anthony Braga, and Rhiana Kohl. This research was supported by Grant 5R21HD073761-02 from the NIH/Eunice Kennedy Shriver National Institute of Child Health and Human Development, Grant SES-1259013 from the National Science Foundation, a grant from the Russell Sage Foundation, the Radcliffe Institute for Advanced Study, and the Rappaport Institute for Greater Boston at Harvard University.

Supporting Information

Supporting Information (PDF)
Supporting Information

References

1
J Hunter, M de la Puente, M Salo, Comparative ethnographic research on mobile populations (US Census Bureau, Washington, DC). (2003).
2
M de la Puente, Census 2000 ethnographic studies. Technical Report 15 (US Census Bureau, Washington, DC). (2004).
3
V Martinez-Ebers, Using monetary incentives with hard-to-reach populations in panel surveys. Int J Public Opin Res 9, 77–86 (1997).
4
B Pettit, B Western, Mass imprisonment and the life course: Race and class inequality in US incarceration. Am Sociol Rev 69, 151–169 (2004).
5
J Travis, B Western, S Redburn The Growth of Incarceration in the United States: Exploring Causes and Consequences (National Academy, Washington, DC, 2014).
6
B Pettit Invisible Men: Mass Incarceration and the Myth of Black Progress (Russell Sage Foundation, New York, 2012).
7
M Nelson, P Deess, C Allen, The first month out: Post-Incarceration experiences in New York City (Vera Institute of Justice, New York). (1999).
8
CA Visher, V Kachnowski, N La Vigne, J Travis, Baltimore prisoners’ experiences returning home (Urban Institute, Washington, DC). (2004).
9
PI Bourgois Hypothesis and Ethnographic Analysis of Concealment in the Underground Economy: The Economic and Idealogical Dynamics of the Census Undercount (Undercount and Behavioral Research Group, US Bureau of the Census, Washington, DC, 1990).
10
K Edin, L Lein Making Ends Meet: How Single Mothers Survive Welfare and Low-Wage Work (Russell Sage Foundation, New York, 1997).
11
J Faugier, M Sargeant, Sampling hard to reach populations. J Adv Nurs 26, 790–797 (1997).
12
MJ Salganick, DD Heckathorn, Sampling and estimation in hidden populations using respondent-driven sampling. Sociol Methodol 34, 193–240 (2004).
13
DP Farrington, B Gallagher, L Morley, RJ St. Ledger, DJ West, Minimizing attrition in longitudinal research: Methods of tracing and securing cooperation in a 24-year follow-up study. Data Quality in Longitudinal Research, eds D Magnusson, L Bergman (Cambridge Univ Press, Cambridge, UK), pp. 122–147 (1990).
14
RL Hough, H Tarke, V Renker, P Shields, J Glatstein, Recruitment and retention of homeless mentally ill participants in research. J Consult Clin Psychol 64, 881–891 (1996).
15
DP Desmond, JF Maddux, TH Johnson, BA Confer, Obtaining follow-up interviews for treatment evaluation. J Subst Abuse Treat 12, 95–102 (1995).
16
LB Cottler, WM Compton, A Ben-Abdallah, M Horne, D Claverie, Achieving a 96.6 percent follow-up rate in a longitudinal study of drug abusers. Drug Alcohol Depend 41, 209–217 (1996).
17
A Messiah, H Navaline, A Davis-Vogel, D Tobin-Fiore, D Metzger, Sociodemographic and behavioral characteristics associated with timeliness and retention in a 6-month follow-up study of high-risk injection drug users. Am J Epidemiol 157, 930–939 (2003).
18
JO Teitler, NE Reichman, S Sprachman, Costs and benefits of improving response rates for a hard-to-reach population. Public Opin Q 67, 126–138 (2003).
19
P Brown-Peterside, et al., Retaining hard-to-reach women in HIV prevention and vaccine trials: Project ACHIEVE. Am J Public Health 91, 1377–1379 (2001).
20
RB Cotter, JD Burke, R Loeber, JL Navratil, Innovative retention methods in longitudinal research: A case study of the developmental trends study. J Child Fam Stud 11, 485–498 (2002).
21
SJ Prins, Prevalence of mental illnesses in US State prisons: A systematic review. Psychiatr Serv 65, 862–872 (2014).
22
CJ Mumola, JC Karberg, Drug use and dependence, state and federal prisoners, 2004 (Bureau of Justice Statistics, Washington, DC). (2006).
23
S Metraux, CG Roman, RS Cho, Incarceration and homelessness. The 2007 National Symposium on Homelessness Research, eds D Dennis, G Locke, J Khadduri (US Department of Housing and Urban Development, Washington, DC), pp. 9-1–9-31 (2007).
24
CW Herbert, JD Morenoff, DJ Harding, Homelessness and housing insecurity among former prisoners. RSF 1, 44–79 (2015).
25
DJ Harding, JD Morenoff, CW Herbert, Home is hard to find: Neighborhoods, institutions, and the residential trajectories of returning prisoners. Ann Am Acad Pol Soc Sci 647, 214–236 (2013).
26
C Sirois, Household dynamics in the year after prison (Boston Reentry Study, Cambridge, MA). (2015).
27
A Goffman On the Run: Fugitive Life in an American City (Univ of Chicago Press, Chicago, 2014).
28
S Brayne, Surveillance and system avoidance: Criminal justice contact and institutional attachment. Am Sociol Rev 79, 367–391 (2014).
29
D Pager Marked: Race, Crime, and Finding Work in an Era of Mass Incarceration (Chicago Univ Press, Chicago, 2007).
30
Government Accountability Office (2005) Drug offenders: Various factors may limit the impacts of federal laws that provide for denial of selected benefits (US Government Accountability Office, Washington, DC).
31
LM Lopoo, B Western, Incarceration and the formation and stability of marital unions. J Marriage Fam 67, 721–734 (2005).
32
EA Wang, JA Aminawung, C Wildeman, JS Ross, HM Krumholz, High incarceration rates among black men enrolled in clinical studies may compromise ability to identify disparities. Health Aff (Millwood) 33, 848–855 (2014).
33
V Kachnowski, Employment and prisoner reentry. Returning Home Illinois Policy Brief (Urban Institute, Washington, DC). (2005).
34
CA Visher, SME Courtney, One year out: Experiences of prisoners returning to Cleveland (Urban Institute, Washington, DC). (2007).
35
NG La Vigne, TL Shollenberger, SA Debus, One year out: Tracking the experiences of male prisoners returning to Houston, Texas (Urban Institute, Washington, DC). (2009).
36
PK Lattimore, DM Steffey, The multi-site evaluation of SVORI: Methodology and analytic approach (RTI International, Research Triangle Park, NC and Urban Institute, Washington, DC). (2009).
37
DJ Harding, JJ Wyse, C Dobson, JD Morenoff, Making ends meet after prison. J Policy Anal Manage 33, 440–470 (2014).
38
NF Sugie, Finding work: A smartphone study of job searching, social contacts, and wellbeing after prison. PhD dissertation (Princeton University, Princeton). (2014).
39
B Western, AA Braga, J Davis, C Sirois, Stress and hardship after prison. Am J Sociol 120, 1512–1547 (2015).
40
RJA Little, DB Rubin Statistical Analysis of Missing Data (Wiley, New York, 1987).
41
J Fitzgerald, P Gottschalk, RA Moffitt, An analysis of sample attrition in panel data: The Michigan Panel Study of Income Dynamics. J Hum Resour 33, 251–299 (1998).
42
JJ Heckman, Sample selection bias as a specification error. Econometrica 47, 153–161 (1979).
43
K Hirano, GW Imbens, G Ridder, DB Rubin, Combining panel data sets with attrition and refreshment samples. Econometrica 69, 1645–1659 (2001).
44
D Silva, C Savelli, Massachusetts Department of Correction 2013 annual report (Massachusetts Department of Correction, Milford, MA). (2013).
45
RM Groves, MP Couper Nonresponse in Household Interview Surveys (Wiley, New York, 1998).
46
E Martin, D Abreu, F Winters, Money and motive: Effects of incentives on panel attrition in the Survey of Income and Program Participation. J Off Stat 17, 267–284 (2001).
47
H Houtkoop-Steenstra Interaction and the Standardized Survey Interview: The Living Questionnaire (Cambridge Univ Press, Cambridge, UK, 2000).
48
JR Warren, A Halpern-Manners, Panel conditioning in longitudinal social science surveys. Sociol Methods Res 41, 491–534 (2012).
49
J Brooks-Gunn, GJ Duncan, The effects of poverty on children. Future Child 7, 55–71 (1997).
50
R Haveman, R Blank, R Moffitt, T Smeeding, G Wallace, The war on poverty: Measurement, trends, and policy. J Policy Anal Manage 34, 593–638 (2015).
51
L Fox, et al., Trends in deep poverty from 1968 to 2011: The influence of family structure, employment patterns, and the safety net. RSF 1, 14–34 (2015).
52
R Kornfeld, H Bloom, Measuring program impacts on earnings and employment: Do unemployment insurance wage reports from employers agree with surveys of individuals? J Labor Econ 17, 168–197 (1999).

Information & Authors

Information

Published in

The cover image for PNAS Vol.113; No.20
Proceedings of the National Academy of Sciences
Vol. 113 | No. 20
May 17, 2016
PubMed: 27162332

Classifications

Submission history

Published online: May 9, 2016
Published in issue: May 17, 2016

Keywords

  1. hard-to-reach populations
  2. poverty
  3. study retention
  4. survey methods
  5. sampling bias

Acknowledgments

We thank David Harding, Becky Pettit, and Emily Wang, who provided excellent comments on earlier drafts of the paper. We gratefully acknowledge the hundreds of men and women who shared their time and life experience to improve our understanding of incarceration in America, and the Massachusetts Department of Correction, which provided access to correctional facilities and advice and collaboration throughout the research. The data for this paper are from the Boston Reentry Study, a research project conducted by Bruce Western, Anthony Braga, and Rhiana Kohl. This research was supported by Grant 5R21HD073761-02 from the NIH/Eunice Kennedy Shriver National Institute of Child Health and Human Development, Grant SES-1259013 from the National Science Foundation, a grant from the Russell Sage Foundation, the Radcliffe Institute for Advanced Study, and the Rappaport Institute for Greater Boston at Harvard University.

Notes

*It is possible too, though less common in our experience, that greater social integration and security may contribute to nonresponse. If survey interviews are uncomfortable reminders of past episodes of personal crisis, respondents may be reluctant to participate when they are doing well. Nonresponse in this case would lead to overestimates of social insecurity.
Probation is a period of community supervision determined at sentencing and administered by the court. Parole is a type of early release and community supervision, determined after a period of incarceration, and administered by an executive department.
There were no deaths in the follow-up period, although one respondent was critically injured and was hospitalized near the end of the follow-up period. Another was released straight from prison to county jail custody for the entire year after prison release.
§
Repeated contact with interviewers can induce behavioral and attitudinal change (48). In particular, social support in BRS interviews might improve outcomes for respondents. Improved outcomes would be welcome news for released prisoners and policymakers, but insignificant differences in recidivism between the BRS sample and the state as whole suggests study participation yields few positive effects.

Authors

Affiliations

Bruce Western1 [email protected]
Department of Sociology, Harvard University, Cambridge, MA 02138;
Anthony Braga
School of Criminal Justice, Rutgers, The State University of New Jersey, Newark, NJ 07102;
Present address: School of Criminology and Criminal Justice, Northeastern University, Boston, MA 02115.
David Hureau
Department of Sociology, Harvard University, Cambridge, MA 02138;
Catherine Sirois
Department of Sociology, Stanford University, Stanford, CA 94305

Notes

1
To whom correspondence should be addressed. Email: [email protected].
Author contributions: B.W., A.B., and C.S. designed research; B.W., A.B., D.H., and C.S. performed research; B.W. and C.S. analyzed data; and B.W., D.H., and C.S. wrote the paper.
Reviewers: D.H., University of California, Berkeley; and B.P., University of Texas at Austin.

Competing Interests

The authors declare no conflict of interest.

Metrics & Citations

Metrics

Note: The article usage is presented with a three- to four-day delay and will update daily once available. Due to ths delay, usage data will not appear immediately following publication. Citation information is sourced from Crossref Cited-by service.


Citation statements




Altmetrics

Citations

Export the article citation data by selecting a format from the list below and clicking Export.

Cited by

    Loading...

    View Options

    View options

    PDF format

    Download this article as a PDF file

    DOWNLOAD PDF

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Personal login Institutional Login

    Recommend to a librarian

    Recommend PNAS to a Librarian

    Purchase options

    Purchase this article to access the full text.

    Single Article Purchase

    Study retention as bias reduction in a hard-to-reach population
    Proceedings of the National Academy of Sciences
    • Vol. 113
    • No. 20
    • pp. 5459-E2872

    Figures

    Tables

    Media

    Share

    Share

    Share article link

    Share on social media