More Information

Published Year: 2017
Survey/Program(s): Current Population Survey (CPS)
Catalog: Statistical Survey Paper

Renee Ellis, Matt Virgile, Jessica Holzberg, Dawn V. Nelson, Jennifer Edgar, Polly A. Phipps, and Robin L. Kaplan

Assessing the Feasibility of Asking About Sexual Orientation and Gender Identity in the Current Population Survey: Results From Cognitive Interviews

PDF version

by Renee Ellis, Matt Virgile, Jessica Holzberg, Dawn V. Nelson, Center for Survey Measurement, U.S. Census Bureau
     Jennifer Edgar, Polly Phipps, Robin Kaplan, Office of Survey Methods Research, Bureau of Labor Statistics

Acknowledgement: The authors sincerely thank Joanne Pascale, Research Social Science Analyst at the Census Bureau, for her significant contributions to the analysis, structure and writing of this report.

Disclaimer: This report is released to inform interested parties of research and to encourage discussion. The views expressed are those of the authors and not necessarily those of the U.S. Census Bureau or the Bureau of Labor Statistics.

Executive Summary

This report presents findings from cognitive interviews that were conducted by the U.S. Census Bureau and the Bureau of Labor Statistics (BLS). Cognitive interviews were conducted in four cities to explore the feasibility of asking about sexual orientation and gender identity (SOGI) in the Current Population Survey (CPS). In the test questionnaire, we selected 46 questions from the basic CPS to both set the general context of the actual CPS, and to include items known to be challenging in some way for respondents to answer, either for themselves or others in the household, such as income, and disability. This gave a basis for comparing the difficulty and sensitivity of the SOGI items to others already in the questionnaire. Data from 132 respondents with diverse demographic characteristics were collected, in both "individual" interviews with one member of a household, and "paired" interviews with two members of a household interviewed separately. Both lesbian, gay, bisexual and transgender (LGBT) and non-LGBT respondents were included. The cognitive interviews were used to answer five research questions:

1. How difficult are the SOGI questions for respondents to understand and answer? Do respondents have the knowledge to answer for other people in their household and are they willing to provide those answers?

Overall, a majority of respondents found the SOGI questions clear and did not have difficulty answering for either themselves or for other people in their households. Respondents judged income, employment, and disability questions to be more difficult than the SOGI items for both self-response and proxy response (response for other household members). For proxy response, the date of birth item was found to be more difficult than both SOGI items, and gender identity was among the least difficult. The observed difficulty for SOGI questions tended to be related to the response categories, having a fluid identity, questioning one's identity, and/or lacking knowledge of the sexual orientation or gender identity of others in the household.

The SOGI responses collected from household members via the paired interviews matched for almost all the respondents, suggesting respondents are able and willing to provide the information about other people in their household. Additionally, data collected during a card sort exercise indicates that while many respondents categorized the SOGI questions as difficult, the income, disability, and race questions were more frequently identified as difficult.

2. How sensitive do the respondents perceive the SOGI questions to be when answering for themselves and for others in their household, and how does that sensitivity relate to willingness to answer the questions or complete the survey?

Most respondents did not find the SOGI questions sensitive, but relative to other items in the questionnaire, the sexual orientation and disability items were found to be the most sensitive items for both self and proxy response,. Respondents who found the SOGI questions sensitive to report for themselves were both LGBT and non-LGBT respondents, with the former finding their own identity personal to talk about, and the latter finding sexual orientation culturally sensitive and more private. Fewer respondents indicated that they found gender identify sensitive, for either self or proxy reporting. Those who did indicate sensitivity had similar reasons as for sexual orientation, indicating the topic is personal or culturally sensitive. A few respondents (most of whom were LGB) indicated they found SO or GI questions sensitive because they were uncomfortable with the question wording and response options.

Despite some indications of sensitivity, all respondents were willing to answer the SOGI questions for themselves and other people in their households.

3. Do difficulty and sensitivity differ based on demographics - such as geography, household structure, race and/or Hispanic ethnicity, educational attainment, sexual orientation or gender identity?

While there were not many trends by demographics, the main difference that emerged was that most of the respondents who expressed difficulty or sensitivity with the SOGI questions were lesbian, gay, or bisexual (LGB). Furthermore, the few transgender respondents in the study almost universally expressed both difficulty and sensitivity with the SOGI questions as worded.

The demographic profile of the LGBT and non-LGBT samples were different, reducing our ability to detect demographic differences in the results between those groups. For some comparisons (e.g., difficulty for proxy reporting, sensitivity for self-reporting) however, some differences by household size, age, urbanicity, educational attainment, and race were identified. These differences were not consistent across comparisons, and so are difficult to interpret.

4. Are respondents willing to answer SOGI questions for themselves and others in their household in the context of a Federal government survey on employment?

Results suggest that respondents are generally willing to answer SOGI questions for themselves and others in their household in the context of a Federal employment survey, although we note this study included only paid, volunteer respondents who were comfortable enough with the Federal government to participate. When asked directly for their thoughts on being asked SOGI questions in the context of the CPS, a majority said that they had no concerns with the idea, and no respondents refused to answer any SOGI question. A few respondents did raise issues about SOGI questions, discussing concerns over confidentiality, or mentioning that their responses could be less protected and/or used for discrimination in the current political climate. Additionally, the idea of cultural sensitivity was raised by a few respondents.

5. What feedback do respondents have on wording of SOGI questions?

While question wording was not a main focus of this study, respondents provided some valuable feedback on the wording that was tested. In general, respondents understood the SOGI questions but some respondents, particularly LGBT respondents, had difficulty with or concerns about the response options in both questions. A variety of feedback and suggestions are detailed in the report, and will be a valuable guide for future research.

Conclusion

Overall, feedback in these cognitive interviews suggests that most respondents do not find SOGI questions difficult or sensitive to report for themselves or for others in their households, and that few raised objections to these questions in the context of the CPS. However, perceptions of difficulty for self and others in the household were more frequent among LGBT respondents. These findings suggest that while collection of SOGI information on the CPS may be feasible, extensive further testing is needed. Additionally, beyond the problems identified in this study, there are a variety of issues that will need to be addressed before any implementation decisions can be made.

These interviews were just one part of a larger study on the feasibility of asking about sexual orientation and gender identity on the CPS. A decision on overall feasibility of collecting SOGI information in the CPS should consider the findings of the cognitive interviews as well as those of the focus groups with transgender respondents (Holzberg et al., 2017).

1. Introduction

In 2015, the Office of Management and Budget (OMB) convened a working group of more than 100 representatives from 35 Federal agencies across 14 departments and 7 independent Federal agencies (Park, 2016). The purpose of the group is to share knowledge on the development and testing of questions on sexual orientation and gender identity (SOGI) in both Federal and non-Federal surveys in the United States (Federal Interagency Working Group, 2016a). The goal of including SOGI data in Federal surveys is to allow researchers to estimate the size and distribution of the sexual and gender minority populations in the U.S., and to identify disparities between people who identify as lesbian, gay, bisexual, and transgender (LGBT) and those who do not in domains such as health, crime, or employment. Currently, there are 11 Federal surveys that collect data on sexual orientation, and of these, seven also ask about gender identity (Federal Interagency Working Group, 2016a). These surveys vary on features such as question wording, mode of survey response, and population being surveyed. In terms of context (i.e., primary topic of the survey), most are health surveys; the exceptions are the National Inmate Survey (NIS), the Survey of Prison Inmates (SPI), and the National Crime Victimization Survey (NCVS). All of these surveys ask respondents to report SOGI for only themselves — not other household members.

The OMB working group has so far issued three working papers on the landscape of SOGI questions in Federal surveys: one on current SOGI measurement in Federal surveys, a second on evaluations of these Federal SOGI measures, and a third on research priorities moving forward (Federal Interagency Working Group, 2016a, 2016b, 2016c). In this latter paper, the group identified proxy reporting and question wording as primary research priorities for SOGI questions and survey context as a secondary research priority (Federal Interagency Working Group, 2016c). The working group has also recently taken an interest in asking about SOGI on the Current Population Survey (CPS), which serves as the primary source of labor force statistics for the U.S. population. The CPS is sponsored jointly by the Bureau of Labor Statistics (BLS) and the Census Bureau. The CPS differs from the surveys currently collecting SOGI information in two important ways. First, the context is employment, as opposed to health, which may affect respondents' perceptions of the relevance of SOGI questions. Second, a single household respondent answers questions about all other household members. Thus, in households with two or more members, household respondents provide self-reports for themselves, and also proxy reports for other household members. It is unknown whether respondents are able to report SOGI accurately by proxy and whether they feel comfortable doing so.

Because of these differences, the U.S. Department of Labor's Office of the Assistant Secretary for Policy sponsored research to explore the feasibility of asking about SOGI in the CPS. The overall goal was to assess the feasibility of collecting SOGI data in the CPS setting—that is, an employment survey context with proxy reporting. More specifically, we wanted to examine: (1) the sensitivity of the questions in general; (2) whether household members have the knowledge about each other with regard to SOGI questions and are willing to provide those answers, and (3) reactions to the inclusion of SOGI questions in an employment survey. To address these goals, cognitive interviews were conducted with both LGBT and non-LBGT populations, and focus groups were conducted with individuals who identified as transgender. Results of the focus groups are provided in a separate report (Holzberg et al, 2017); the purpose of this report is to document the results of the cognitive interviews.

Cognitive interviews are one-on-one sessions between a researcher and respondent. The researcher presents the respondent with the verbatim survey questions and then probes the respondent on various aspects, such as interpretation and comprehension of the question, difficulty in formulating an answer, sensitivity in providing a response, and other issues that may arise during the response process. A total of 132 cognitive interviews were conducted with roughly equal numbers of GBT and non-LGBT individuals. These groups were further divided between "individual" interviews with one member of a household (80 respondents), and "paired" interviews with two members of a household interviewed separately (52 respondents). The cognitive interviews were used to answer five research questions:

1. How difficult are the SOGI questions for respondents to understand and answer? Do respondents have the knowledge to answer for other people in their household and are they willing to provide those answers?
2. How sensitive do the respondents perceive the SOGI questions to be when answering for themselves and for others in their household, and how does that sensitivity relate to willingness to answer the questions or complete the survey?
3. Do difficulty and sensitivity differ based on demographics — such as geography, household structure, race and/or Hispanic ethnicity, educational attainment, sexual orientation, or gender identity?
4. Are respondents willing to answer SOGI questions for themselves and others in their household in the context of a Federal government survey on employment?
5. What feedback do respondents have on wording of SOGI questions?

2. Background

2.1 Current Population Survey (CPS)

The CPS is conducted monthly by the Census Bureau on behalf of BLS with an annually-selected probability sample of about 60,000 occupied households in all 50 states and the District of Columbia. The survey excludes those living in institutions, such as prisons, long-term care facilities, and nursing homes. One person (age 15 or older), known as the "household respondent," generally responds for all household members and is usually a person who owns or rents the housing unit. Households are in the survey for four consecutive months, out of sample for eight months, and then return for another four months before leaving the sample permanently, for a total of eight interviews. In terms of content, the CPS consists of a basic monthly survey, to which a supplemental or topical module is added in most months. The basic monthly survey is divided into two parts: household and demographic information (e.g., date of birth, marital status) and labor force information (e.g., employment during the past week). Supplements cover a range of topic areas, including annual work activity and income, veteran status, school enrollment, and volunteerism, among other topics (U.S. Census Bureau, 2017a).

Most of the household and demographic questions are asked only once, in the first interview. In subsequent interviews, respondents are asked to confirm that members of the household previously reported are still living there. If someone has left the household, they are removed from the household roster; if someone has moved into the household, demographic questions are asked for that person. Respondents are also re-asked demographic questions if they previously reported they did not know the answer (but if they refused initially, the question is not re-asked). Some questions (such as educational attainment and disability) are re-asked of all respondents in select subsequent interviews regardless of prior responses because the answers could change over time. The CPS is administered through both computer-assisted personal interviewing (CAPI) and computer-assisted telephone interviewing (CATI). The majority of the first and fifth CPS interviews are conducted through personal visit, with other waves often being conducted over the telephone.

Currently, there are no questions on SOGI in the CPS. There is a question on sex, in the demographics section of the basic monthly survey. Household respondents are asked to respond for themselves and all other household members, regardless of age. The question is worded as shown below (note that for this and all items tested, responses of "don't know" and "refused" are not explicitly displayed or read to the respondent, but they do exist as response categories):

CPS Current Question on Sex
What is [NAME's] sex?

• Male
• Female

Interviewers are instructed to ask this question "only if necessary" (U.S. Census Bureau, 2017b). If SOGI questions were implemented in the CPS, gender identity questions would likely replace the current sex question, and the sexual orientation question would likely be placed in the demographic section of the basic CPS. Because of this placement at the beginning of the interview, there is concern that respondents may refuse to answer SOGI questions and break-off from the rest of the CPS.

2.2 Sexual Orientation and Gender Identity

According to a multidisciplinary expert panel convened by the Williams Institute at the University of California, Los Angeles, known as the Sexual Minority Assessment Research Team (SMART), sexual orientation has three main dimensions: sexual attraction, sexual behavior, and sexual identity (SMART, 2009). In this report, we base "sexual orientation" on only sexual identity, rather than sexual attraction or behavior. The most commonly used terms to describe different sexual orientations are lesbian, gay, bisexual, and heterosexual/straight. In general, people who self-identify as gay or lesbian are primarily attracted to and/or have sex with people of the same sex. However, the concepts of sexual identity, attraction, and behavior do not always follow these patterns, or individuals may not want to identify as gay or lesbian even if they are attracted to the same sex or only occasionally have same sex relations (Federal Interagency Working Group, 2016a). As for gender identity, SMART defines this as "A person's internal sense of gender (e.g., being a man, a woman, or genderqueer) and potential affiliation with a gender community (e.g., women, trans-women, genderqueer)"¹ (Federal Interagency Working Group, 2016a). Throughout this report, we use the acronym "LGBT" to refer to sexual and gender minorities, encompassing both sexual orientation and gender identity. For sexual orientation, we use "lesbian, gay or bisexual" or "LGB" as an umbrella term to refer to anyone who self-identifies as anything other than straight. For gender identity, we use "transgender" as an umbrella term to refer to "anyone whose gender identity differs from their sex assigned at birth." (GLAAD, 2017) It is important to note that any one individual may be LGB, transgender, both LGB and transgender, or neither LGB nor transgender.

2.3 Measurement of Sexual Orientation and Gender Identity in Surveys

2.3.1 Sexual Orientation

Since at least the late 2000s, researchers have been exploring alternative methods for asking about sexual orientation in surveys. In 2009, the Williams Institute issued a report from SMART on best practices for question wording on sexual orientation as:

SMART Best Practices Recommendation
Do you consider yourself to be:

• Heterosexual or straight;
• Gay or lesbian; or
• Bisexual?

In the past few years, sexual orientation questions have been added to a number of state and Federal surveys. These generally use the wording that first appeared in the National Health Interview Survey (NHIS) in 2013, or a variation of it. The NHIS question is asked of all sample adults age 18 and over, and response categories vary depending on sex:

National Health Interview Survey
Which of the following best represents how you think of yourself?
[for male respondents]

• Gay
• Straight, that is, not gay
• Bisexual
• Something else
• I don't know the answer

[for female respondents]

• Lesbian or gay
• Straight, that is, not lesbian or gay
• Bisexual
• Something else
• I don't know the answer

For this study, we maintained the question stem used in the 2013 NHIS for the self-response version of the question, but for the proxy-response version, we added the precursor "To the best of your knowledge," For both versions, we consolidated the response categories as shown.

CPS Test Question on Sexual Orientation
[Self-response]: Which of the following best represents how you think of yourself?
[Proxy response]: To the best of your knowledge, which of the following best represents how [NAME] thinks of themselves?

• Gay or Lesbian
• Straight, that is not gay, lesbian, or bisexual
• Bisexual
• Something else

2.3.2 Gender Identity

There has been limited testing of question wording on gender identity, but studies thus far have demonstrated that transgender respondents are generally able to understand and answer questions on the subject (Baker & Hughes, 2016; Lombardi & Banik, 2016; Reisner et al., 2014; Cahill et al., 2014). In 2014, the Williams Institute issued a report addressing best practices for asking about gender identity in a survey based on research by a multidisciplinary expert panel known as the Gender Identity in U.S. Surveillance (GenIUSS) Group. The group recommended using a two-question approach by first asking for a respondent's assigned sex at birth, and then asking for their current gender identity (GenIUSS group, 2014). Other organizations, such as the Center of Excellence for Transgender Health (CoE) and the World Professional Association for Transgender Health (WPATH) also recommend using a two-question approach, but they suggest asking about current gender identity before sex assigned at birth (Federal Interagency Working Group, 2016b). Of the seven Federal surveys that ask about gender identity, three of them — the National Crime Victimization Survey (NCVS), the Survey of Prison Inmates (SPI), and the National Adult Tobacco Survey (NATS) — use the two-question approach. All ask the sex-at-birth question first, and all use slightly different wording and response categories, as shown:

Question 1: Sex at Birth
NCVS and SPI: What sex were you assigned at birth, on your original birth certificate?
NATS: What sex were you at birth? [Field Interviewer Note: "Did they tell you that you were born male or female?"]

• Male
• Female

Question 2: Current Gender Identity
NCVS: Do you currently describe yourself as male, female, or transgender?

• Male
• Female
• Transgender²

SPI: How do you describe yourself (select one)?¦

• Male
• Female
• Transgender
• Do not identify as male, female or transgender

NATS: Do you currently consider yourself to be:

• Male
• Female

In terms of measuring prevalence, under a two-question approach respondents would be identifiable as transgender if they selected different options for their sex at birth and for their current gender identity. In the NCVS and SPI, those who chose "transgender" would also be added to the tally. The NCVS also includes a follow-up question that has interviewers verify they are recording the correct answer if there is a mismatch between the two questions:

[IF Q1 ≠ Q2] Just to confirm, you were assigned [male/female] at birth and now describe yourself as [male/female]. Is that correct?

• Yes
• No

The other four Federal surveys — the Behavioral Risk Factor Surveillance System (BRFSS), the National Inmate Survey (NIS), the Population Assessment of Tobacco and Health (PATH), and the Health Care Patient Survey (HCPS) — use a one-question approach that asks directly about transgender status (though the PATH survey first asks a yes/no on transgender, then a follow-up on type of transgender). The HCPS also has a follow-up question for those who identify as genderqueer or "other" to determine skip patterns for subsequent survey questions. As with the sex-at-birth question, all three surveys use different question wording for gender identity, as shown:

BRFSS: Do you consider yourself to be transgender?

• Yes, Transgender, male-to-female
• Yes, Transgender, female-to-male
• Yes, Transgender, gender non-conforming
• No

NIS: Are you male, female, or transgender?

• Male
• Female
• Transgender

PATH:
(a) Some people describe themselves as transgender when they experience a different gender identity from their sex at birth. For example, a person born into a male body, but who feels female or lives as a woman, would be transgender. Do you consider yourself to be transgender?

• Yes ask (b)
• No
• Not sure

[if R answers YES]
(b) Do you consider yourself to be male-to-female, female-to-male, or non-conforming?

• Yes, Transgender, male to female
• Yes, Transgender, female to male
• Yes, Transgender, gender nonconforming
• No
• Not sure

HCPS: What is your gender?

• Male
• Female

[If age >13]:

• Female to male transgender male/trans male/female to male
• Male to female transgender female/trans woman/male to female
• Genderqueer
• OTHER, specify

[If Genderqueer or other]: We have entered your gender as [Genderqueer OR other]. In this interview, questions will appear based on gender. For example, we only ask questions about mammograms to females of a specific age. Since this is a research study collecting medical-related data, could you tell us your biological sex at birth?

• Male
• Female

Under the one-question approach, respondents would be identifiable as transgender only if they selected that answer category explicitly, which could risk under-reporting compared to the two-question approach (Tate et al., 2012).

For purposes of this CPS testing, we followed the recommendation of the GenIUSS report and used the two-question approach. We also adopted the approach we used for the sexual orientation question when a respondent is asked to report about another household member by adding the precursor "To the best of your knowledge,"

CPS Test Questions on Gender Identity
QUESTION 1: SEX AT BIRTH
[Self-response]: Was your sex recorded as male or female at birth?
[Proxy response]: To the best of your knowledge, was [NAME's] sex recorded as male or female at birth?

• Male
• Female

QUESTION 2: CURRENT GENDER IDENTITY
[Self-response]: Do you describe yourself as male, female, or transgender?
[Proxy response]: To the best of your knowledge, does [NAME] describe themselves as male, female, or transgender?

• Male
• Female
• Transgender

With regard to age, we follow suit from the sexual orientation protocol and ask the question about those 15 and older. For those under age 15, we used the CPS current question on sex shown above.

In addition to issues of question wording, the OMB working group expressed concerns about "cultural and non-binary inclusivity" ³ noting that "Some individuals who vary in age, cultural and linguistic groups, etc., may not endorse terms such as 'transgender' when responding to Federal gender identity questions because they do not identify with this term" (Federal Interagency Working Group, 2016c). To address this, we recruited on a wide range of demographic characteristics and aimed to include respondents from various minority and non-minority groups, on characteristics such as race, ethnicity, and urban or rural residency. We also recruited roughly equal numbers of LGBT and non-LGBT respondents, and we included eight respondents who identified as transgender in our study.

Another measurement concern is how surveys should use pronouns such as "he" or "she" in subsequent questions. Transgender respondents vary in their preferred pronouns; for example, some respondents prefer the gender neutral "ze" (University of California, Berkeley, 2017). Surveys collecting gender identity need to decide whether to use the preferred pronoun based on gender identity, the pronoun aligning with sex at birth, or "they" as a singular, non-gendered pronoun (Federal Interagency Working Group, 2016c). After deciding how to use pronouns, surveys then need to implement this change throughout the entire survey, a task that is operationally complex and time-consuming. In this study, the option of using the gender-neutral pronoun "themselves" was used when a pronoun was used in the current CPS demographic questions, though for most questions the household member's name was used.

Finally, the working group expressed concerns about terminology evolving over time. We acknowledge this, and below we discuss findings on the fluidity of some of these categorizations even in a static context (that is, even from day to day, self-identities may change).

2.4 Proxy Reporting

Many Federal surveys use proxy response, in which one person generally responds for all eligible household members, primarily to reduce costs and nonresponse (Tamborini and Kim, 2013). It can be very time-consuming and difficult to collect survey responses when all members of the household are required to answer for themselves (Pierce et al., 1993; Park, 2015). However, proxy response involves other tradeoffs, as proxy answers may differ from those provided by other household members. Across survey topics, evidence on the quality of proxy response in surveys is mixed. Respondents may use less precise question answering strategies such as estimation when answering about other people in the household (Bickart, Blair, Menon and Sudman, 1990). Data quality can depend on the question topic and the relationship between the respondent and others in the household. For example studies have found that respondents most familiar with other household members, such as spouses, tend to be better proxies (Kojetin and Mullin, 1995; Tamborini and Kim 2013; Grieco and Armstrong, 2014; Pascale 2016). Small differences in agreement between proxy and self-response answers are more common than large differences (Mellow and Sider, 1983; Boehm, 1989; Moore, 1988; Tamborini and Kim 2013).

With regard to SOGI questions in particular, very little is known about whether respondents have the knowledge necessary to report for other household members, and whether they would be willing to report the information if they do have it. Surveys in New Zealand and the United Kingdom do not permit proxy reporting for sexual orientation due to concerns about accuracy and confidentiality (Joloza et al., 2010, Park, 2015). One of the only studies conducted on this issue employed an online nonprobability panel to test SOGI questions similar in wording to the questions we use in this study (Ortman et al., 2017). That study, while it could not determine the accuracy of the responses, found overall low rates of nonresponse for the SOGI questions. Item nonresponse for these questions was lower than for income, which is also considered to be a sensitive question. However, nonresponse to the SOGI questions was also significantly higher for proxy reports compared to self-reports. Nonresponse rates also varied by relationship categories; rates of nonresponse were higher when respondents were reporting for children of respondents (age 16 or older) and roommates than for other household members, such as spouses and unmarried partners.

2.5 Survey Context

As noted above, the OMB working group raised the issue of survey context — that is, asking SOGI questions in the context of an employment survey — versus the more common context of health surveys. The concern was that if respondents view SOGI questions as irrelevant to the subject matter of the survey, they may refuse to answer the questions or break-off from the survey entirely. Thus far, there is very little evidence available in this area. In one study, after SOGI questions were added to the NCVS, researchers conducted a debriefing survey with interviewers about issues they experienced when administering SOGI and other questions. A relatively small percentage of interviewers reported that at least one respondent asked why sexual orientation or gender identity was relevant to crime (Truman et al., 2017). Nonresponse to these questions was low and only a few respondents broke off from the survey after being asked these questions.

Feedback from respondents in the present study provides insight into whether SOGI questions are seen as irrelevant in surveys about topics other than health.

3. Study Methodology

We conducted 132 cognitive interviews in Washington, DC; Portland, OR; Nashville, TN; and Fargo, ND. The Washington interviews were conducted from September through December 2016, while the Portland, Nashville, and Fargo interviews were held in March, April, and June 2017, respectively. These cities were selected to represent different geographic regions of the country, with the assumption that these regions would also vary on attitudes, political experiences, and other factors that would impact respondent experiences and opinions. Interviews were conducted by staff from the Census Bureau, BLS, and Community Marketing, Inc. (CMI). Each interview was allotted one hour, but many were shorter, with an average interview length of 39 minutes. All interviews were conducted in person, and respondents received $40 each to compensate them for their time. Following standard Federal research procedures, OMB provided clearance for this study prior to the start of recruitment.

In order to have a more direct measure of the accuracy of proxy reporting, we also included 52 individuals from 26 unique households for "paired interviews." Each respondent in the pair was interviewed separately, providing information both about themselves, each other and (in households with three or more members) all the other members of their household. This allowed us to directly compare survey responses between household members to gauge accuracy based on match rates.

3.1 Recruitment

For the research carried out in Washington, DC, recruitment methods consisted of posted flyers, advertisements through Craigslist.com, a broadcast message sent to all U.S. Census Bureau employees who work in the Suitland, Maryland headquarters, and posts on Facebook pages for LGBT groups that featured a telephone number and email address to contact a recruiter for screening.

For the other sites, the Census Bureau and BLS established a contract with CMI to handle recruiting, onsite logistics, and administration of some interviews. CMI maintains a nationwide research panel of LGBT individuals who were recruited for this study. CMI also recruited new respondents using targeted Facebook advertisements to transgender Facebook users, flyers, and Craigslist.com advertisements. Print recruiting materials featured a telephone number, while digital advertisements directed participants to a CMI intake survey first. Intake survey responses were used to evaluate whether the respondent was potentially appropriate for the study.

All prospective respondents were screened via telephone prior to being scheduled for interviews. Screening included questions on respondents' age, race, ethnicity, employment, geographic area (urban versus rural⁴), household composition, and LGBT status. On the latter, to identify LGBT individuals we used screener questions that differed from the SOGI questions being tested.⁵ Single-person households were excluded because they would not be able to inform proxy reporting. The recruitment goals were as follows:

• At least 50 percent in households with three or more members,
• At least 50 percent in households with someone age 15 to 25,
• At least 25 percent in households with non-relatives,
• At least 33 percent non-White or Hispanic respondents,
• An equal or nearly equal balance of males and females,
• An equal or nearly equal balance of urban and rural households, and
• At least 50 percent with less than a Bachelor's Degree.

All goals were met or reached at least ninety percent of the target rate. The next section provides details on the characteristics of those successfully recruited.

3.2 Respondents

Tables 1 and 2 below show respondent and household characteristics, respectively, by site and interview type (individual or paired). In total, 71 interviews were conducted with respondents who were LGBT or had a household member who identified as LGBT, and 61 were conducted with respondents in non-LGBT households.

3.3 Interview Protocols

All respondents were provided with a consent form before beginning the interview session. They were also told that information they provided would be confidential and audio-recorded, and were notified if there were any observers. Interviewers then explained that the purpose of the study was to test new questions developed for the CPS — the primary source of labor force statistics, like employment and unemployment, in the nation. No mention was made at the start about testing of SOGI questions. Interviewers then administered several different interview protocols shown in Table 3 and explained in more detail below.

3.3.1 Standardized Questionnaire

Researchers administered 46 questions from five different questionnaire topic areas of the basic CPS interview (with the SOGI questions embedded in the demographics section) as if it were a regular interview. See Table 4 for a list of topic areas and question content. Employment and unemployment questions were included in the Standardized Questionnaire in order to convey the context of the CPS as a survey on the labor force. Disability and income questions were also included since these are generally more difficult and/or sensitive than other CPS questions. This gave us the chance to evaluate the relative difficulty or sensitivity of SOGI questions by comparing them to disability and income questions. No probes were asked during administration of the CPS questions, but a respondent's spontaneous, volunteered feedback was captured by the interviewer.

During personal interviews in the actual CPS, "show cards" are used for certain questions (relationship, educational attainment, race and income). Show cards include the full text of the question and response options and are presented to the respondent as the questions are asked. We used show cards for the questions that use them in actual CPS interviews, and we also included a show card for the sexual orientation question due to the length and complexity of the response options. A show card was not used for the gender identity question because it was brief and had fairly simple response options.

3.3.2 General Debriefing

Interviewers debriefed respondents about their overall thoughts on the interview, ease or difficulty of answering the questions, and whether anything stood out or was bothersome. At this point respondents were still not told that SOGI questions were of particular interest, because we wanted to see what was salient to the respondent without prompting. Since this section was open-ended, some respondents offered their reactions at the question-level (e.g., they found the family income or the gender identity question to be sensitive), while others offered their reactions at the topic-level (e.g., they noted the disability questions were difficult, but did not specify a particular disability question). Interviewers did not probe for more specific details at this point, since this would be done in later sections of the protocol.

Often, but not always, a topic area consisted of just one question, as shown in Table 5. Thus, comparison between topic areas is imprecise because topic areas with more than one question provide more opportunity for the respondent to find the questions difficult or sensitive. However, we note that the range of the number of questions per topic area is small (one to three), with the notable exception of the employment topic area, which contained 17 questions. In the debriefings, however, only three specific items (on work last week, class of worker, and employer name) within this topic area were mentioned frequently in respondents' reactions. Nevertheless, the difference in number of questions per topic area will impact interpretation of the coding results, described in Section 3.4.

3.3.3 Card Sort Exercise

Respondents were presented with index cards listing 15 of the 46 administered CPS questions and asked to complete two sorting tasks for themselves — first sorting cards into two piles that were either "sensitive" or "not sensitive," then ranking the cards in the sensitive pile from most to least sensitive. A second Card Sort task was then administered, with respondents sorting cards into "difficult" and "not difficult" groups, ranking the difficult cards by most to least difficult. Thus, the same question could be categorized as both difficult and sensitive.

3.3.4 Card Sort Debriefing

After both Card Sort Exercises, respondents were asked debriefing probes about why they found the questions sensitive/difficult when answering for themselves. They also were asked if they thought other members in their household would sort the cards differently, and if so, why. Finally, they were asked if they would sort the cards differently based on answering for others in the household instead of themselves, and if so, why. For these probes, respondents were not asked to re-sort, or move the cards, though a few did so.

3.3.5 Question Specific Probing

Using a semi-structured protocol, respondents were asked about a subset of specific questions from the Standardized Questionnaire: disability,⁷ sex at birth, gender identity, income, and sexual orientation. For each of these, respondents were asked about their reaction to the question, the meaning of the question and/or specific terms within the question (such as "transgender"), their certainty and comfort in answering for themselves and for other household members, and whether they thought these other household members would be certain or comfortable answering for themselves and others.

3.3.6 Paired Interview Match Rate

In the 26 households where two respondents ("Person A" and "Person B") were sampled, each was administered the full Standardized Questionnaire protocol. Thus both respondents were asked the same questions about each other, the household as a whole, and, in households with three or more people, about other household members.

3.3.7 Paired Interview Debriefing

A debriefing was conducted among each person in the pair to explore both SOGI and non-SOGI items from two perspectives: Person A's reactions to being asked to provide information about Person B, and also Person A's reactions to Person B providing information about them. During their individual interviews, both respondents in the pair were asked how comfortable they felt answering questions about the other person in the pair, and how accurate they thought their own answers would be about the other person. Respondents were also asked how they felt about the other person answering survey questions on their behalf, and how accurate they thought the person would be. Both in the pair were also asked whether they had any concerns about the other person answering on their behalf, and whether they thought their answers to any particular questions might not match.

3.3.8 Context Debriefing

Interviewers explained that the Federal government was considering adding SOGI questions to an employment survey. Respondents were asked what they thought about this idea generally, why they thought the government would be interested in collecting SOGI in an employment survey, and how they and other household members would respond if asked to provide this information in a government survey about employment.

3.4 Analysis

The interview protocols produced both qualitative and quantitative data. The analysis of each is described below.

3.4.1 Qualitative Data Analysis

Six different protocols produced qualitative data: spontaneous comments made during administration of the Standardized Questionnaire, responses to Question Specific Probing, and comments from four Debriefings (General, Card Sort, Paired Interview, and Context). These data were analyzed for evidence of recurring themes and patterns. We also developed a coding scheme to flag the data as sensitive or difficult, and we coded whether the difficulty/sensitivity was reported or indicated in relation to a self-report or to a report about someone else. The coding schemes are described below. Note that in some instances, the difficulty/sensitivity was directly reported and other times the difficulty/sensitivity was indirectly indicated by the nature of respondents comments during debriefings. For convenience we use the term "reported" interchangeably with "indicated" but we note that "reported" does not necessarily mean a direct report.

For reliability, two staff members, working independently, coded the data from each of the protocols (shown in Table 3). Once the independent coding was completed, final codes were assigned through adjudication, either by a third staff member or by a discussion among the coders.

Results from each of the six protocols are shown in multiple tables below, specific to each section (e.g.: Difficulty for Self-Reporting). In addition to the individual protocol results, these tables show an "Interview Protocol Summary" column that is an indicator of difficulty/sensitivity in one or more of the six individual qualitative protocols. For the sake of simplicity, as we discuss results we draw primarily from the Interview Protocol Summary column. There are two caveats to keep in mind with this approach. The first, mentioned earlier, is the differing numbers of questions in each topic; for example, 17 employment questions were included in the test questionnaire, versus one income question. This discrepancy creates unequal opportunities for respondents to have difficulty or sensitivity within a topic area.

The second caveat is the design of the Question Specific probing protocol. Based on their expected level of difficulty and/or sensitivity, only four topics were selected for this protocol (SO, GI, income and disability). These topics received probes that other topics did not receive, again creating additional opportunities for respondents to have difficulty within a topic area. Thus, while the figures from the Interview Protocol Summary column represent an imperfect metric, it is convenient for making multiple comparisons, and results from individual protocols are maintained in the main results tables along with the summary column.

3.4.1.1 Coding Difficulty and Sensitivity

In general, across the six protocols, questions were coded as difficult if there was any evidence that respondents thought that they or others (within or outside the household) would be unable to answer the question, because either they lacked the relevant knowledge or they did not understand the question and/or certain terms in the question. Questions were coded as sensitive if there was any evidence that respondents thought they or others would be unwilling to answer them or if the respondent had a negative emotional reaction to the question. The evidence used for the coding was generally verbal — that is, respondents' responses to a debriefing probe or a spontaneous comment made during the Standardized Questionnaire.

In the Standardized Questionnaire protocol, respondents were not asked directly about the difficulty or sensitivity of any questions, but interviewers were instructed to make question specific notes for the SOGI questions as they were administering the questionnaire, including observations of verbal and non-verbal reactions from respondents. Interviewers were also instructed to make notes on demographics, employment, disability, and income topics. Coding of difficulty and sensitivity was based on respondents' verbal reactions while they were answering the questions. For example, a question was coded as difficult if respondents explicitly commented that it was hard for them to understand or answer, asked for clarification, answered "don't know," or voiced uncertainty about their answer. For sensitivity, if respondents volunteered that a question was upsetting or personal, or refused to answer, the question was coded as sensitive. In addition to volunteered comments like this, questions were flagged as difficult or sensitive based on interviewer notes about nonverbal behavior. For example, if an interviewer noted that a respondent was visibly agitated or upset when a particular question was asked, this would be flagged as a sensitive question.

3.4.1.2 Coding Type of Report (Self, Proxy, Non-Household Member)

A key goal at the beginning of the project was to identify and compare issues for self-reporting versus proxy reporting for others within the household. However, although we did not directly ask for respondents' opinions about people outside of the household, many respondents offered comments about this during administration of one or more protocols. Respondents frequently volunteered that they expected that a given question would be sensitive or difficult for members of a particular group (e.g., people who identify as LGBT), even if no one from this group lived in their household. For example, some respondents said the sexual orientation question was not sensitive for themselves or anyone in their non-LGBT household, but they thought the question would be sensitive to someone who is gay or lesbian. Respondents also said that unspecified "others" (not in the household) would, hypothetically, find a question difficult or sensitive, without specifying whether those others belong to a particular group. We coded these mentions of others as non-household members and distinguish them from proxy (i.e., within-household) responses. Ultimately we developed three categories to describe who the comment on difficulty/sensitivity referred to: self, proxy (within household), and non-household members.

For three of the Debriefings (General, Paired Interview, and Context) and the Question Specific Probing, coding of self, proxy and non-household was fairly straightforward and was based on how the respondent characterized the difficulty/sensitivity. Coding of the Standardized Questionnaire protocol was based on who the question referred to; respondents were asked separate questions for themselves and other household members.⁸ If issues arose when the respondent was answering standardized questions for themselves, the difficulty/sensitivity was coded under self-response; if the issues arose when answering questions for others the difficulty/sensitivity was assigned as either proxy or non-household member. For the Card Sort Debriefing, respondents' comments on their own rationale for their sorting and ranking were coded as related to self-response unless the respondent explicitly mentioned that their rationale was in relation to other people. When debriefed on whether other household members would sort the cards the same way, these comments were coded as related to proxy reporting.

The disability and family income questions were asked at the household level (i.e., "Because of a physical, mental, or emotional condition, does anyone have difficulty [insert task]?"). Thus, sometimes it was not clear whether the respondent found the question difficult/sensitive for themselves, or for other household members. For difficulty, we coded comments as pertaining to proxy response by default unless the respondent (1) expressed comprehension issues with the question or certain terms in the question, or (2) was explicit that their reaction was based on their own status, and not the status of anyone else. For example, if a respondent said the income question was difficult because they work multiple jobs and had to add up the income across jobs, responses were coded as difficult for self-response. But, if a respondent said they were unsure of their roommates' income amount, they were coded as difficulty related to proxy response. With regard to sensitivity, we coded comments as pertaining to self-response by default unless respondents were explicit that their reaction was based on the income or disability status of someone else, and not themselves. For example, in response to a disability question, if the respondent disclosed that a household family member has a physical or mental condition, and that this is upsetting to talk about, this would be coded as pertaining to a proxy response.

3.4.2 Quantitative Analysis

3.4.2.1 Card Sort Exercise

The outcome of a Card Sort Exercise is the list of cards included in each pile and the ranking each card was given (e.g., most and least difficult of the difficult questions). Interviewers captured both these data elements, and they were analyzed to identify the questions most and least often considered sensitive and/or difficult.

3.4.2.2 Paired Interview Match Rate

The answers from Person A and Person B were evaluated as a "match" or a "mismatch." If the answers given by each person in the pair were not the same, it was coded as a mismatch. For questions that had multiple response categories (e.g., marital status with categories of single, divorced, etc., or income which had 16 response categories) if the exact same category was not chosen by both respondents in the pair, it was considered a mismatch. The match rate was interpreted as a measure of accuracy.
The unique outcomes from these paired interviews were:

• How often do Person A and Person B match on the answers to questions for each other?
• How often do Person A and Person B match on the answers they provide about the household (e.g., family income)?
• In households with three or more members, how often do Person A and Person B provide the same answers to questions when asked about Persons C, D, and so on?

4. Findings

4.1 Difficulty

Table 6 provides a summary of results for difficulty across question topic areas from the six qualitative protocols. Each cell shows the number of respondents who expressed difficulty with each topic area and reporting type (self, proxy, or non-household). For example, during the Standardized Questionnaire protocol, two respondents expressed difficulty with self-reporting for the sexual orientation question, three expressed difficulty with proxy-reporting for someone within the household, and one indicated that "other people" would have difficulty. Note that a given respondent could report difficulty for multiple types of reports; that is, a respondent could find a question difficult to report for themselves, and also for someone else in the household. Also, note that the summary column is an indication of difficulty from one or more individual protocols.

Table 7 provides results on the demographic characteristics of those reporting difficulty for the sexual orientation (SO) and/or the gender identity (GI) items.⁹ Results are shown for each type of report (self, proxy or non-household members), and for LGBT versus non-LGBT respondents.

We first discuss results about difficulty in self-reporting, concentrating on these four areas: the sexual orientation question, the gender identity question, demographic differences of respondents reporting difficulty with the SOGI questions, and differences in difficulty between SOGI and non-SOGI questions. We then discuss results in the same four areas for proxy reporting. Finally, we discuss respondents' perceptions of non-household members' reactions to SOGI questions.

4.1.1 Self-Reporting

4.1.1.1 Sexual Orientation

CPS Test Question on Sexual Orientation
[Self-response]: Which of the following best represents how you think of yourself?
[Proxy response]: To the best of your knowledge, which of the following best represents how [NAME] thinks of themselves?

• Gay or Lesbian
• Straight, that is not gay, lesbian, or bisexual
• Bisexual
• Something else

There was little evidence from the qualitative protocols of respondents having any difficulty answering the SO question for themselves (see Table 6). Of 132 respondents, only 14 had any difficulty in any of the qualitative protocols. All respondents answered the question in the Standardized Questionnaire protocol, with only two indicating difficulty during that protocol. Most of the difficulty indicated during the Question Specific Probing protocol (10), followed by the Card Sort protocol (7).

When we asked what the question was asking in their own words during Question Specific Probing, all respondents were able to give an answer that indicated understanding of the question. Typical responses were:

• "Who are you attracted to."
• "Your sexual and romantic preference in partners."
• "It wants to know if I'm heterosexual or consider myself bisexual, or wanting to be with a same-sex partner, or if I consider myself to be something that is not documented already."
• "How you view yourself. Not how the world categorizes you."

In terms of choosing an answer, most indicated it was an easy question to answer.

• "No problem for me, I'm straight."
• "It's a good question. It's pretty straightforward."

Some respondents noted that it was an expected question:

• "What [is] expected from Census."
• "It's just something that's common nowadays."
• "It was simple, straightforward, just general demographic data."

However one respondent noted, "These are not so much difficult as unexpected." They caused her to hesitate and think about the answer, but she could provide an answer without a problem.

Most of the difficulty issues indicated throughout the interview protocols were expressed by LGBT individuals; 13 of the 14 respondents who indicated difficulty with the SO question were LGB and four were transgender. This amounted to half of the transgender respondents (four of eight) ¹¹, and 13 of the 64 LGB respondents expressing difficulty. For the most part the difficulty had to do with the response categories.

One main reason for the difficulty was among those who felt sexual orientation is an inherently fluid concept. For example, one said they were "Uncertain about where I fit in the choices, I know my identity but it can be fluid but primarily I am gay." A related issue came from a respondent who, in the end, chose "bisexual" but said they are still questioning their orientation: "No one has asked me, still figuring out identity." Another issue was respondents saying the response categories were too limited. Three transgender respondents had a hard time reconciling ideas of straight or gay with gender fluidity, and they all chose "something else" as the category:

• "If you identify as like, a non-binary, or some people, trans people, have issue with these words. See, take me for instance. I'm dating a woman. But some trans men, they would not be sure whether in my case if I would be straight, or if I would be in some way, like gay or queer."
• "There are many ways to identify. This is only giving three categories."

With regard to response categories being limited, some would have preferred something that fit them better than the catch-all "something else," such as "queer," "pansexual," or "asexual."

• "Something else' is the only option I can say as I'm not represented 100 percent by the other three options."
• "Something else is what I'd answer, but it also doesn't give you an identity at all."

In six cases, respondents had trouble selecting from among the LGB categories (gay/lesbian, bisexual or "something else"). However, the categories they said they would choose would be one of the three LGB categories, and not the straight/heterosexual category, so the difficulty would not result in a misclassification as long as the intent was to distinguish LGBT and non-LGBT respondents.

4.1.1.2 Gender Identity

CPS Test Questions on Gender Identity
QUESTION 1: SEX AT BIRTH
[Self-response]: Was your sex recorded as male or female at birth?
[Proxy response]: To the best of your knowledge, was [NAME's] sex recorded as male or female at birth?

• Male
• Female

QUESTION 2: CURRENT GENDER IDENTITY
[Self-response]: Do you describe yourself as male, female, or transgender?
[Proxy response]: To the best of your knowledge, does [NAME] describe themselves as male, female, or transgender?

• Male
• Female
• Transgender

Gender identity (GI) results were similar to those for the SO question. Only 11 of 132 respondents reported difficulty in any of the qualitative protocols when answering for themselves. All but one respondent answered the question for themselves during the Standardized Questionnaire, and only three indicted difficulty. Again, most difficulty was reported during the Question Specific Probing (9), followed by the Card Sort Debriefing (8).

When asked about ease/difficulty throughout the interview protocols, respondents tended to say things like, "Not much, these questions seem pretty easy to be answered, just straight up: male or female?" A few respondents expressed surprise at the addition of "at birth" to the sex question, and some also found gender identify question surprising, but they did not find them difficult, just unexpected: "I haven't seen that much before in surveys or questions."

When asked to define transgender in their own words, three respondents conflated transgender with sexual orientation (e.g., "If you feel comfortable with your sexuality."). But more typical responses were:

• "I think about someone who was born the wrong sex that might not line up with his or her perceived identity."
• "That you wish to live as a different gender than you were born."
• "It is an umbrella term for someone who does not identify with, or falls outside of, the gender dichotomy."

Those who did indicate difficulty were dominated by LGBT respondents; only one respondent was non LGBT (see Appendix Table A2), and seven of the 11 were transgender.¹² One respondent found it difficult because they were currently questioning their own gender identity, but most of the difficulty expressed had to do with response options rather than an understanding of the concepts. Respondents either did not see themselves in the list of response categories offered (e.g., because they were gender-fluid), and/or they wanted to choose more than one category. Some respondents did not like the limitation of choosing only one category because "transgender" was excluded from "male" and "female." Others noted that male and female were biological concepts of sex and not exclusive from gender identities. Typical responses:

• "When people ask, I just say 'trans male.' Um, I don't know, I think, it doesn't, I wouldn't say male directly, unless it was a government agency or somebody who didn't really matter to me. But, yeah, for me personally, I identify as trans male. Are there other options, or are those the only three options I'm given?"
• "I see myself as female and identify as female, but sometimes I have to put male or transgender."
• "I would answer it as 'other' because I am not listed yet."
• "I don't like using the term … I think as far as it goes the question is fine but it doesn't really fit. The description of male and female aren't consistent with transgender. They don't mean the same things and aren't mutually exclusive."
• "I think it could have been phrased better. Transgender sort of, separates trans women and trans men from male and female."

Difficulty did not result in unwillingness to answer. Respondents expressed that surveys were a place where they would be more willing to answer this question, even if they did not like the wording. One respondent, after thinking about the issues at length, concluded that there may not be a better a way and asking was more important than having a perfect question.

4.1.1.3 Demographic Characteristics of Respondents Reporting Difficulty with SOGI Questions

In total, 21 of 132 respondents expressed difficulty for themselves with the SO and/or GI questions (Table 7). For the most part, the demographic profile of those who did and did not have difficulty with the SO and/or GI question was similar.

Nevertheless, some demographic differences were notable, even when considering the demographic distribution within the samples. Among the 21 respondents indicating difficulty, most were LGBT (19), White/Non-Hispanic (14), and/or urban (15). Furthermore, seven of the eight transgender respondents in the sample found the GI question to be difficult, and four also found the SO question difficult (data not shown).¹³

4.1.1.4 Differences in Difficulty for SOGI versus non-SOGI Questions

Despite the caveats noted in Section 3.4.1 on design differences by protocols, and thus potential for some CPS questions to provide more opportunities for respondents to identify difficulty concerns, we did some high-level comparisons to understand the relative difficulty of the SOGI questions compared to the non-SOGI questions.

Results from the qualitative protocols show that across topics, the income item stood out as the most difficult, with 31 respondents expressing difficulty at some point across all protocols (see Table 6). Disability and employment questions were next (24 each). Race/ethnicity, sexual orientation, relationship/marital status and gender identify were next and fairly close, with between 11 and 14 respondents expressing difficulty across these topics.

For questions other those about SOGI, respondents who reported difficulty most often said that they had to think about before answering.

• [On income]: "More difficult only because I had to think about them."
• [On disability]: "It sounds a little general to me…I didn't know what they were looking for."

For some questions (such as educational attainment, and relationship to other household members), respondents reported having difficulty with the response categories, and these required more thought.

• Educational attainment: "Associate's degree options were confusing"
• Relationship: "None of the relationship responses represent our partnership"

The question topic that most resembles the SOGI questions in relation to the reason for difficulty was race/ethnicity, where some respondents did not see themselves in the categories, and some had difficulty defining or deciding on the categories that fit them:

• "I don't really know. I mean, this is restrictive. I'm mixed."
• "I put race here [in the difficulty Card Sort Exercise] because my option wasn't there." During the initial CPS interview portion, this respondent answered White to the race question and Hispanic to the ethnicity question. Later the respondent said there should be options for Latino identity in the race question.

Quantitative results from the Card Sort Exercise were generally consistent with results from the qualitative protocols. Respondents put an average of 1.05 of the 15 cards into the difficult pile, with a zero card median. Sixty-eight respondents said that none of the questions provided on the cards were difficult, and an additional 29 and 12 respondents put only one or two cards in the difficult pile, respectively. Only 11 of the 132 respondents put four or more cards in the difficult pile. This indicates that most of the questions were not seen as difficult.

As shown in Table 8, the card most frequently sorted into the difficult pile was the question about income, with 46 respondents considering it to be difficult to answer. This was followed by two disability items, and then race and sexual orientation. Patterns from the card sort ranking task are similar. The item ranked most difficult for self-response by far was income (32). Race, and the questions about disability trailed far behind, with very few respondents ranking them as the most difficult to answer question.

4.1.2 Proxy Reporting

4.1.2.1 Sexual Orientation

When asked about SO for others in the household, only 20 of 132 respondents expressed any difficulty, all but one provided answers to the question, and only three respondents indicated some difficulty during the Standardized Questionnaire protocol. Reasons for the difficulty echoed the issues found for self-reporting in terms of choosing from among the response categories. Typical responses were:

• "He is asexual. Straight, and something else."
• "[I] would feel bad for [my] roommate that was not included in identity categories."
• "He does identify as something not listed on there, but 'bisexual' represents well enough as what he represents himself as."
• "They would want to answer the sexual orientation question as 'queer.'"
• "They see sexuality [as] more fluid. They might answer it 'lesbian', might answer 'bisexual.'"

about the identity of the younger people in the household (generally teenagers). These respondents thought they knew the answers, but were reluctant to rule out the possibility that the person had not decided on a sexual orientation:

• "…except my son. I don't want to label him if I don't know. Until then I assume he's straight."
• "My daughter is in college, she could be experimenting. My son is 18 and not talking to me about sex…I know the answer to the best of my knowledge but they are just starting to discover themselves, they are exploring…I don't want to assume or judge anything."
• "My son is still young and society is still not 100 percent accepting, so it is still possible that my son may be bisexual or something rather than straight and not told me."
• "It's something that we've talked about, but I don't think that she definitely identifies as bisexual or lesbian or anything. I'm pretty sure she would share that with me because we're pretty open like that."
• "Partner's sister is in college, so who knows what she's up to."

One respondent who reported difficulty was a teenager and did not know their parents' sexual orientation because they had not talked to them about it. Other respondents thought that non-relatives or roommates who lived in their household would not have the knowledge to answer the sexual orientation about another household member: "The roommate does not know she is bisexual. He would probably say 'straight.'" A few respondents also speculated that other household members would have difficulty understanding the terminology; this was generally in reference to older household members: "She would not relate to the language used in the SOGI questions. Concepts would be foreign to her."

4.1.2.2 Gender Identity

Qualitative results indicate that only six of 132 respondents had any difficulty reporting GI for other household members. No respondent indicated difficulty with proxy reporting during the Standardized Questionnaire protocol, and all respondents answered the GI question for all of their household members age 15 and up.

Three of the six who had difficulty had older household members who they thought would have trouble answering this question about themselves and about others because of the language in the questions. For example: "The older people in the house might be thrown off by the transgender questions." One of these respondents remarked that the older people in her household would have difficulty with the question, and also said that older people in general would have trouble with this question. Another respondent said their older mother would be confused by the terms used in this question.

Two other respondents said they lacked the knowledge about their roommates. One respondent thought he could answer for his roommate but was not 100 percent sure because they had not talked about it. The other found it difficult because she had new roommates that had moved in that week: "Think I know the answer but I don't know the roommates very well. They could be transgender, but I don't think so."

One respondent felt the questions would be difficult for other household members to answer because either the response options were not inclusive enough, or because they could not select more than one option. Two transgender respondents expressed uncertainty over whether other household members would identify their transgender identity in the same way they did: "I don't know that they would answer that I am transgender or male."¹⁴

4.1.2.3 Demographic Characteristics of Respondents Reporting Difficulty with SOGI Questions

As shown in Table 7, total of 24 respondents found SO and/or GI questions difficult when asked about someone in the household, and most often the difficulty was with SO (20) rather than GI (6).¹⁵ These 24 respondents were most often LGBT (18), from households with 3 or more people (19), and/or urban (17). Of the eight transgender respondents, three indicated difficulty with the SO and/or GI questions.

4.1.2.4 Differences in Difficulty for SOGI versus non-SOGI Questions

Despite the caveats noted in Section 3.4.1 on design differences by protocols, and thus potential for some CPS questions to provide more opportunities for respondents to identify difficulty concerns, we did some high-level comparisons to understand the relative difficulty of the SOGI questions compared to the non-SOGI questions.

Findings from the qualitative protocols indicate that by far the most difficult topic area for proxy reporting was income, with 85 respondents reporting difficulty (see Table 6).¹⁶ Next was date of birth (28), followed by disability (24). The employment and SO questions were similar (21 and 20, respectively). Next were educational attainment and race, and GI was low with only six respondents reporting difficulty.

Across questions/topics, the most frequent reason for difficulty was response categories. For example, as with the SOGI questions, respondents had difficulty choosing among answer categories for the race question for other household members:

• "There isn't quite an option because she's completely Hispanic."
• "With (daughter) Hispanic and White, it's weird."

For disability, the difficulty was more about the question's intended meaning: "More explanation and examples, if you want to get into the natty gritty… Nowadays so many people have mental and emotional difficulties… Give examples, or define it a little bit more."

Lack of knowledge was sometimes a reason for difficulty, but answers of "don't know" for at least one other household member were mainly concentrated in the date of birth item (16). Education was next (5), followed by race/Hispanic origin (4). Only one respondent reported, "don't know" to the sexual orientation question for at least one other household member, and for gender identity there were no answers of "don't know."

The Card Sort Debriefing provided another angle on this same issue, because respondents were asked if other household members would sort and rank the cards differently than they did. About half of respondents (71 of 132) said other household members would identify only a subset of the cards they selected as difficult, or none at all. Some respondents (25) said that other household members would choose the same cards. The remainder (36) said other household members would select different or additional cards. Of these, the questions that respondents thought others would have difficulty with but they themselves did not have difficulty with were most often questions on income, disability, and sexual orientation. However, each were mentioned eight or fewer times in total across the 36 respondents.

The quantitative results on match rate complement the qualitative findings. The income item had the highest number of mismatched answers between pairs of respondents¹⁷ (18 of 26) (See Table 9). Educational attainment was the next most-often mismatched item (13). Contrary to some expectations, only two mismatched answers to each of these questions were between non-relatives; most of the mismatches were among family members (data not shown). Two of these family pairs were between parents and their teenage children. SOGI questions ranked toward the bottom in terms of mismatches; SO had five, and GI had the lowest number of mismatches with only two.

For the SO question in particular, two of five mismatched cases were opposite-sex partners where females identified themselves as "bisexual" or "something else" and males identified their spouse/partner as straight. In both cases, the females noted that their spouse/partner would report them (the females) as non-LGB. Also in both cases, the male partner at some point during the debriefing protocols said that their female partner might not identify herself as straight. In three cases, members of the pair answered the SO question about another household member differently. In two cases, both members of the pair identified a different household member as LGB but did not match on categories ("bisexual" versus "something else"). In one case, a parent and teenager were paired and were asked about another teenager in the household. The parent identified the other teenager as "bisexual," while the teenager in the pair identified the sibling as "straight". The teenager also answered "don't know" about the parent's sexual orientation.

For the GI question, all respondents matched on sex at birth and GI for the other person in the pair. Two pairs mismatched when reporting GI for other members of the household. One pair was reporting for a roommate.¹⁸ The second pair mismatched on GI categories for a third household member.¹⁹