Electronically mediated work: new questions in the Contingent Worker Supplement

The Contingent Worker Supplement

The Contingent Worker Supplement (CWS) is a set of questions that has periodically been appended to the nation’s monthly labor force survey, the Current Population Survey (CPS).¹ The CWS, first fielded in 1995, is designed to measure the number and characteristics of contingent workers and workers in four alternative employment arrangements—independent contractors, on-call workers, temporary help agency workers, and workers provided by contract firms. The survey was fielded four more times—in 1997, 1999, 2001, and 2005—with a largely unchanged questionnaire.

In 2016, the U.S. Bureau of Labor Statistics (BLS) obtained funding to field the CWS in May 2017. One major goal of the 2017 CWS was to see how the number of contingent workers and workers in alternative employment arrangements had changed since 2005. Therefore, in order to maintain data comparability over time, the 2017 questionnaire was largely the same as that used when the data were last collected.

Many stakeholders were interested in adding questions to the CWS to collect information about a variety of other topics. However, the development of new questions can be a lengthy process, and BLS had limited time to make changes if the survey was to be fielded in May 2017. First, to comply with Office of Management and Budget (OMB) guidance, all substantive changes to federal survey questionnaires are evaluated, and proposed changes are announced and provided to the public for comment, which can take considerable time. Additionally, the U.S. Census Bureau—which conducts the survey for BLS—had adopted new software for its data collection instrument since the 2005 survey, so the CWS needed to be completely reprogrammed in the new software and tested thoroughly. (The data collection instrument is the custom-designed software used by Census Bureau interviewers to conduct the survey and collect responses.) Because the existing survey had an extremely complicated questionnaire, many rounds of systematic testing would be necessary to ensure that the survey instrument was programmed correctly. Developing and adding new questions and ensuring that these questions were programmed correctly would strain an already ambitious schedule.

After consulting with the Census Bureau, BLS determined that, given the time constraints and the need to minimize respondent burden, it was not possible to add more than four straightforward questions—that is, four questions with limited skip-and-fill patterns and with limited response options.² Also, the four questions would need to be added to the end of the CWS questionnaire so that there was no impact on responses to earlier questions. In addition, placing questions at the end of the questionnaire would simplify the programming of the data collection instrument.

Developing a draft set of questions to add to the CWS

In early 2016, BLS formed a team of economists and survey methodologists to investigate the possibility of adding four questions to the CWS. While the group considered several topics—such as second jobs, flexibility of work, advance notification of work schedule, and contingent work and alternative employment arrangements over a longer time span than the previous week—consensus coalesced around obtaining data about work arrangements that have emerged since 2005.

New terms are being used in relation to this emerging type of work, such as “gig workers” and “gig economy.” BLS does not have a definition for these terms, and there is no generally accepted definition among researchers. Many definitions of gig workers include people in temporary jobs, independent contractors, on-call workers, and day laborers—all of which can be estimated with CWS data. However, many definitions also include people in types of work arrangements that did not exist when the survey was last fielded. Many researchers and policymakers have expressed a need for additional data on emerging work arrangements to paint a more complete picture of gig workers, especially since anecdotal evidence suggests a sharp rise in the number of these workers in recent years.

The CWS seemed an appropriate survey for collecting these new data. BLS decided to focus on one emerging type of work that most researchers consider to be a type of gig work—one that is sometimes referred to as “electronically mediated work” or “online platform work.”³ In this type of employment arrangement, workers

·       use a company’s website or mobile app to connect to clients or customers and obtain short jobs, projects, or tasks;

·       are paid by or through the company that owns the website or mobile app;

·       choose when and whether to work; and

·       may do these short jobs, projects, or tasks in person or online.

There are many examples of this type of work. For instance, some people use their own cars to transport people from place to place, having obtained customers through a mobile app that facilitates payment of the ride. (Companies that currently enable this kind of work include Uber and Lyft.) Others do household chores or yardwork after finding clients through a mobile app or website that later arranges for work payment. (Examples of companies that focus on these types of short-term jobs include TaskRabbit and Handy.) Additionally, some workers do work entirely online, such as taking surveys, adding descriptive keywords to photos or documents, or designing webpages for businesses. (Companies such as Amazon Mechanical Turk and Clickworker enable this type of work.)

Note that workers are not considered electronically mediated workers simply because they use a website or mobile app to do their work. The website or app must be used to connect them directly to customers or short-term jobs or tasks, and workers must also be paid by or through the company that owns the website or app. People who find customers or jobs through online ads but are not paid by the company that owns the website where they posted the ad are not considered electronically mediated workers. For example, work found through a Craigslist.com ad is not considered electronically mediated work.

Moreover, many businesses have websites or mobile apps that their employees may use to carry out their work. For example, a company’s driver may use a mobile app to map a route when making deliveries; however, this alone does not constitute electronically mediated work. Similarly, some businesses—such as coffee shops or fast-food restaurants—allow customers to order through a website or app. These businesses typically have a dedicated staff to complete orders. Thus, a barista at a coffee shop is not considered an electronically mediated worker just because a customer may order a beverage through a mobile app.

Electronically mediated workers often have the ability to choose when and how much they work. Some people do electronically mediated work as their only source of income, while others do this type of work as a second job or “on the side.”

Though the measurement of electronically mediated work, like that of other emerging types of work, had been little researched, the BLS team began by reviewing the existing literature.⁴ The concepts used to define electronically mediated work are quite complicated, and many decisions had to be made in developing questions to identify this type of work.

Self-reporting versus proxy reporting

In both the CPS and CWS, one person answers the survey questions about everyone living in the household. Thus, respondents provide data about themselves (self-reports) and others living with them (proxy reports). People who report about other household members may not be able to answer all questions about others’ employment arrangements. While it would be possible to have all household members report only about themselves, this would be quite expensive because interviewers would have to contact some households multiple times in order to obtain responses from all household members. Therefore, respondents need to be able to answer all questions both about themselves and others.

Reference period

The CPS and the CWS use a “last week” reference period—that is, the week including the 12th of the month. Because the four new questions would follow the CPS and the CWS, it would make sense that they too focus on “last week.” However, while some people may do electronically mediated work as a full-time job, there is evidence that many do this type of work sporadically.⁵ Thus, focusing on “last week” might understate the number of people engaging in such work, as it would fail to capture those who regularly perform electronically mediated work but did not do so during the past week.

Therefore, BLS team members considered using a longer reference period, such as the past month or past year, but were concerned that this would confuse respondents because so many previous questions in both the CPS and CWS focus on “last week.” Further, a long reference period can sometimes be difficult for respondents because they may not remember when certain activities occurred. For example, if respondents are asked how many times they did a particular activity in the last year, they may include activities from 2 or 3 years ago. In addition, if a longer reference period were used for the new questions, the number of workers doing electronically mediated work would not be comparable with other CWS estimates. With the same reference week, the interaction of electronically mediated work with contingent work and alternative employment arrangements could be explored. For example, BLS could estimate the number of electronically mediated workers who were also contingent workers or independent contractors. To both avoid respondent confusion and keep measures on a comparable basis, BLS decided to use the reference period of “last week” for these new questions.

Question universe

The CWS questions are asked of employed people.⁶ However, some researchers have suggested that people do not consider electronically mediated work to be a job.⁷ If true, people who only did electronically mediated work might be undercounted if the questions were limited to those classified as employed through answers to CPS questions.

Although keeping the same universe for the new questions would be practical, the BLS team evaluated whether the question universe should be expanded to include those who were not employed—that is, either unemployed or not in the labor force. However, the most basic of the labor force questions in the CPS asks “LAST WEEK, did you do ANY work for pay?”⁸ People with responses of “no” to this question (and who were not temporarily absent from a job) would be classified as unemployed or not in the labor force. In order to expand the question universe to those who were not employed, the CWS would essentially have to repeat this question or a variant of it to people who had already answered “no.” This could frustrate respondents who felt that they had already answered the question. In addition, BLS research suggests that the effect of missed informal work on total employment estimates is likely to be small.⁹ Given these concerns, BLS decided to restrict the universe of the new questions to the employed.

Question scope

The existing questions in the CWS apply only to a person’s main job. For the relatively few people with more than one job (about 1 in 20 workers in 2017), this is the job in which they usually work the most hours. Because anecdotal evidence suggests that many people do electronically mediated work in addition to a regular job, the team investigated expanding the scope of the questions to include all reported jobs. Also, as mentioned above, some researchers have suggested that some people may not view electronically mediated work done on the side as a job, which could cause electronically mediated work to be underrepresented in measures of second jobs. One way to expand the scope of the questions would be to ask about any work done in the reference week, not just work for the main job.

BLS team members feared that respondents would be confused by a sudden shift to questions asking about any work after having answered so many questions about their main job. In addition, asking about any work, rather than the main job, would mean that data on electronically mediated work would not be on a comparable basis with the other data collected in the CWS.

Despite concerns about respondent confusion, BLS thought that it was important to expand the scope to provide more information about the relatively little-studied topic of electronically mediated work. To address concerns about comparability, information could be collected about whether electronically mediated work had been done for the main job, a second job, or additional work for pay. The additional work for pay category would hopefully capture electronically mediated work done by people who do not consider such work to be part of a job.  

Question subject

Some researchers were interested in distinguishing between electronically mediated work done entirely online and that done in person, speculating that their effects on the labor market might be different.¹⁰ In particular, in-person electronically mediated work would more likely affect local labor markets, while electronically mediated work done entirely online would more likely impact the global labor market and be influenced by international regulations and trends. In addition, some researchers suggested that in-person electronically mediated work was more likely to be a sole source of income because it may require a greater time commitment. By contrast, people might be more likely to do online electronically mediated work on an intermittent basis to supplement their incomes.¹¹ For this reason, the BLS team decided to distinguish between these two types of work.

Question wording

There were considerable challenges to designing a set of only four questions that would be clearly understood by respondents. One of the easiest ways to ask questions about electronically mediated work would be to ask whether respondents (or members of their household) had done work through specific companies, such as Uber, Lyft, or TaskRabbit. However, BLS survey questions, by longstanding tradition, do not use specific company names because companies can change, especially in emerging industries or fields. Companies popular at the time of initial survey development may no longer exist when the survey is fielded. BLS attempts to minimize changes to questionnaires because even small changes to question wording can affect responses and, thus, data comparability over time. In addition, respondents may focus only on the company named and omit similar companies. For example, respondents may fail to respond about ride-share companies other than Uber and Lyft if only those companies’ names were included in a question.¹² Given these concerns, BLS decided the questions should describe the characteristics of the work itself but not use company names.

BLS knew it would be difficult to design four questions about electronically mediated work that would be clear to respondents without using company names. Respondents might interpret questions about finding jobs through websites or mobile apps as questions about online job search. In addition, use of websites and mobile apps is widespread, and they are used for many different reasons. Writing questions so that respondents could clearly identify when they had used websites or apps only to facilitate electronically mediated work would be a challenge.

Difficult concepts can often be clarified by including examples in questions. The risk of using examples is that respondents may focus on the example rather than on the actual question, which may lead to incorrect answers if a respondent’s experience does not align with the example chosen. For instance, a respondent who did electronically mediated chores might answer “no” to a question that included an example about electronically mediated ride sharing. Because of this danger, BLS is cautious about including examples in questions. However, BLS believed that the advantages of using examples would outweigh the disadvantages as long as the examples were chosen carefully.

Draft questions

After much discussion of the previously mentioned topics, the team proposed the wording of the four questions. One question asked about in-person electronically mediated work. Another asked about online electronically mediated work. Both used examples to clarify the concepts. The in-person and online questions were each followed by a question about which job this work was done for—that is, whether the work was for their main job, a second job, or additional work for pay.

Stakeholder outreach

Throughout the question development process, BLS actively sought feedback about the proposed new questions. BLS staff gave many presentations and briefings about the CWS to outside groups, including congressional staff, industry groups, academics, nonprofit organizations, and other government agencies. While these presentations tended to focus on the CWS as a whole, BLS efforts to add new questions to collect more information were also described.

In addition, an early draft of the new questions was circulated to many academics, industry experts, special interest groups, and other data users. The draft questions were also discussed with the Department of Labor’s Structure of Work Policy Working Group, which had emphasized the need for up-to-date data that could be used to study how Americans’ work arrangements have changed over time. Furthermore, the new questions were reviewed and cleared by OMB. The clearance process included two periods of public comment, during which BLS received suggestions from the public.

Through these outreach efforts, BLS received considerable feedback, all of which was evaluated. BLS staff made several wording changes to the questions based on specific suggestions received. Some suggestions were not feasible given the tight timeline, such as overhauling the CWS questionnaire or developing an alternative set of four questions on a different topic. Likewise, expanding the scope of the questions to cover a longer timeframe was not deemed practical.

Cognitive testing

In accordance with OMB guidelines for statistical surveys, BLS typically cognitively tests proposed new questions before they are added to surveys.¹³ Cognitive testing involves administering a sample questionnaire to recruited participants and then asking a series of debriefing questions.¹⁴ These debriefing questions collect information about the response process, providing insight into whether participants understand the questions as intended, have difficulty formulating their answers, and respond “correctly” given the measurement objectives. This type of testing can be valuable in ensuring that questions measure the intended concepts.

Two cognitive testing methods were used to evaluate the electronically mediated work questions—laboratory testing and online testing. For both the laboratory and online modes, the goals of the cognitive testing were as follows:

·       To ensure that the proposed questions worked as intended—that is, that they maximized the number of true positives and minimized the number of false positives

·       To test the wording of the draft questions

·       To determine if introductory or transition language was necessary between the existing CWS questions and the electronically mediated work questions

·       To determine whether interviewer instructions or help screens were necessary to explain the key concepts

Laboratory testing

BLS staff conducted 24 interviews in their Washington, DC, cognitive testing laboratory. Participants were recruited through advertisements on Craigslist.com and through flyers handed out at a DC taxi stand and a pizza restaurant. The ads targeted workers who were employed by specific companies, such as Uber, Lyft, TaskRabbit, or GrubHub, or in specific professions. The professions selected include a relatively large number of both electronically mediated workers and traditional workers in the same occupation (for example, Uber drivers and taxi drivers). People who responded to the advertisements were asked several screening questions to ensure they had relevant experience before being invited to participate in the cognitive testing.

A trained cognitive interviewer administered an abbreviated version of the CPS and the CWS, along with the four new questions. The interviewer then debriefed participants to gain insight into their response process in order to uncover any sources of error in what was reported and ways to improve the questions.

Online testing

BLS also conducted 138 online interviews through the Amazon Mechanical Turk (mTurk) platform. While online interviews differ from how the CPS is conducted—that is, in person or by telephone—they allowed BLS to recruit participants in a broad variety of professions and outside the DC area. Also, since mTurk is itself an example of a platform that facilitates online electronically mediated work, online interviews allowed BLS to recruit a large number of individuals for whom the online question would be relevant.

Results for the in-person electronically mediated work question

The in-person question asked about short, in-person jobs or tasks that people find through companies that connect them with customers through a website or mobile app and also coordinate payment for the service. The in-person question performed differently in the two cognitive testing modes. The cognitive interviews conducted in the laboratory contained some false positive responses. Through the debriefing questions, BLS survey methodologists determined that 4 (out of 14) participants who said they had done in-person electronically mediated work had not actually done so. Instead, they had obtained clients through websites (such as Craigslist.com) but were not paid through those websites. Additionally, two of the three proxy responses of “yes” to the in-person question were found to have similar errors. However, most responses to the in-person question were correct, and participants seemed to understand the question as intended.

The in-person question performed better in the laboratory testing than it did in the online mTurk testing. During the mTurk testing, there were 18 (out of 57) false positive “yes” responses and 13 (out of 81) false negative “no” responses. The false positives were determined by evaluating open-ended text descriptions of jobs. Seven false positives were due to participants identifying mTurk tasks performed in the previous week—which were done entirely online—as in-person electronically mediated work. Other false positives were made by respondents who obtained clients through a website but were not paid through that site. The false negative determinations were made by having participants select from a list of electronically mediated work platforms through which they had worked during the previous week.

Results for the online electronically mediated work question

The online question asked about short, paid tasks done entirely online that people find through companies that maintain online lists of tasks. Very few people responded “yes” to this question in the cognitive testing interviews conducted in the laboratory. (Because the mTurk testing was planned, and mTurk is a platform through which people do electronically mediated work entirely online, BLS focused its efforts on recruiting cognitive test participants for the laboratory who were likely to have done in-person electronically mediated work.) All three “yes” responses to this question collected in the laboratory were determined through the debriefing to be false positives. These participants said “yes” either because they (or their household members) found clients online or because they did some of their work online.

The mTurk testing yielded mixed results for the online question. There were very few false positives but many false negatives (31 out of 42 “no” responses). Many participants who answered “no” did not include mTurk tasks they had done in the previous week. Most of these participants did not think the online question was intended to include mTurk tasks. This could be a result of administering the testing via mTurk; participants may have excluded their mTurk work because BLS knew they were on mTurk.

Results of the “which job” questions

Both the in-person and online questions were followed by a “which job” question; if respondents said “yes” to either the in-person or online question, they were asked if that work had been done for the main job, a second job, or additional work for pay. In the laboratory, some participants found it difficult to distinguish between a second job and additional work for pay, but the cognitive testing did not probe specifically about participants’ answers to these two questions. Interviewers did probe when participants displayed obvious difficulty with either of the questions. Some participants found it difficult to distinguish between a second job and additional work for pay because they did not think of electronically mediated work as a job. In the mTurk testing, most participants said they did electronically mediated work—particularly online work—as additional work for pay.

In the CPS, main job and second job concepts are communicated through the survey questions. However, the truncated version of the CPS interview given during the cognitive testing asked only about the main job and did not include any questions about the second job. Therefore, BLS believed that some of the confusion that occurred during testing would not occur in an actual field interview. Similarly, BLS thought that CWS respondents would understand the difference between main and second jobs if they had been administered the full CPS interview.

Issues identified and final recommendations

Although there were some participants who provided incorrect responses, both types of testing indicated that the four questions generally measured what they were intended to measure. BLS survey methodologists analyzed all participant interviews to determine why incorrect answers had occurred. They identified several issues:

In-person and online questions

·       Some participants thought websites that advertised goods and services but did not facilitate payment, such as Craigslist.com, were applicable to both the in-person and online questions.

In-person question

·       In both testing modes, participants who relied on the internet or mobile apps for their work thought the in-person question applied to them. Specifically, participants who found clients through social media and participants who worked for businesses that allow customers to place their orders through mobile apps or websites thought the in-person question applied to their situation. They appeared to miss the reference to “in person.”

·       Many participants in the mTurk testing reported online electronically mediated work (in particular, tasks done through mTurk) as part of their answer to the in-person question.

Online question

·       Several participants with data entry jobs at traditional companies believed that the online question applied to them.

·       Many mTurk participants did not include their experience with mTurk as part of the online question. This may be because they were tested using mTurk and assumed that mTurk tasks should be excluded.

“Which job” questions

·       Participants had some difficulty differentiating between second job and additional work for pay.

The final report on the cognitive testing made several recommendations designed to improve the questions.¹⁵ To stress the difference between in-person and online work, the report made two suggestions: (1) to add introductory, clarifying language and (2) to emphasize the words “in person” and “online” in the questions. The report also suggested revising the examples to better represent the type of work being asked about. In addition, the report suggested highlighting that BLS was interested in learning about all work, not just the main job.

The question wording was finalized based on these recommendations. However, because of time and funding constraints, BLS adopted the revised questions without additional cognitive testing.

Final question wording

After making changes based on the cognitive testing results and stakeholder comments, BLS finalized the question wording in July 2016. Before being asked the questions about electronically mediated work, respondents were given a short introduction:

I now have a few questions related to how the internet and mobile apps have led to new types of work arrangements. I will ask first about tasks that are done in person and then about tasks that are done entirely online.

This introduction was intended to alert respondents to the fact that the following questions would touch on the internet and mobile apps. It also aimed to signal respondents to distinguish between in-person work and work done entirely online. The hope was that this introduction would clarify what might otherwise appear to be repetitive language.

Final wording of the in-person question and follow-up “which job” question was as follows:

Q1       Some people find short, IN-PERSON tasks or jobs through companies that connect them directly with customers using a website or mobile app. These companies also coordinate payment for the service through the app or website.

For example, using your own car to drive people from one place to another, delivering something, or doing someone’s household tasks or errands.

Does this describe ANY work (you/NAME) did LAST WEEK?

Q1a    Was that for (your/NAME’s) (job/(main job, (your/NAME’s) second job)) or (other) additional work for pay?

Note that names are used if the question is asked about others in the household. If respondents answer “yes” to the in-person question (Q1), they are asked the follow-up “which job” question (Q1a). People with only one job are asked whether the in-person electronically mediated work was for their job or additional work for pay. Multiple jobholders are asked whether this work was for their main job, a second job, or other additional work for pay.

As recommended in the cognitive testing report, the words “in person” were capitalized in the question. Interviewers are instructed that capitalized words are important in questions and must be emphasized when conducting interviews. To reduce the underreporting of paid activities that participants think of as “not a job,” respondents were asked to describe any work they did during the reference period. It should be noted that the basic CPS questions inquiring about work already include the emphasis on any work, so respondents would have heard this emphasis in prior questions. The words “last week” are also emphasized, which is done in other questions throughout the CWS and the CPS that refer to the reference week.

The questions about online electronically mediated work and about which job were very similar to the questions for in-person work, though with emphasis on the word “online” and with different examples:

Q2       Some people select short, ONLINE tasks or projects through companies that maintain lists that are accessed through an app or a website. These tasks are done entirely online, and the companies coordinate payment for the work.

For example, data entry, translating text, web or software development, or graphic design.

Does this describe ANY work (you/NAME) did LAST WEEK?

Q2a   Was that for (your/NAME’s) (job/(main job, (your/NAME’s) second job)) or (other) additional work for pay?

Collecting and processing the data

As mentioned earlier, the software used to program the data collection instrument—that is, the custom-designed software used by Census Bureau interviewers to collect the data—had changed since the CWS was last collected. Because of the change, Census Bureau staff reprogrammed the instrument for the 2017 CWS, adding the four new questions to the end. Staff at both the Census Bureau and BLS performed many rounds of extensive instrument testing to ensure that CWS questions appeared on the screen as expected and that all skip-and-fill patterns were correct. In addition, the Census Bureau tested the processing system before fielding.

It is cost prohibitive to do in-person training for CPS supplements like the CWS because interviewers are based all over the country. Instead, interviewers are typically trained about supplements through 1-hour self-study materials. BLS updated and augmented the 2005 training materials to include information about the new questions on electronically mediated work. The final May 2017 self-study materials covered not only the new questions but all questions on the CWS, which collects data about a number of different topics. Reflecting the order of the questions in the survey, the information about the new questions appeared at the very end of the self-study. The training materials were provided before the fielding of the CWS, and interviewers were instructed to complete the self-study materials as part of their preparations for the month.

The CWS was fielded in May 2017. No major problems with either the existing questions or the new questions were reported by interviewers during the data collection period. Considerable time was needed to process the data. Just as the data collection instrument had to be reprogrammed, all edits had to be completely reprogrammed. In addition, supplement weights needed to be developed.

Evaluating the data: monitoring interviews

Interviews conducted by telephone from one of the Census Bureau’s three data collection centers are taped for quality assurance purposes and are retained for a short period. It is standard practice for BLS staff to monitor a handful of interviews after new CPS questions are fielded. Monitoring allows staff to hear the entire interview, including apparent respondent confusion, requests for clarification, and verbatim responses to the questions. From listening to interviews, it is often possible to determine whether the questions were easily understood by respondents, whether answers were correct, and whether breakdowns in communication occurred.

While the Census Bureau was processing the data, the BLS team monitored many interviews to assess the data quality of the new questions. To enable a qualitative analysis of how the questions worked, the team used the unprocessed data to select cases with a variety of characteristics—such as occupation, self-response versus proxy response, and multiple-jobholding status. The selected cases included both those in which respondents said “yes” to at least one of the new CWS questions and those in which respondents answered “no” to both questions. Three or four team members attended each of several monitoring sessions and recorded their observations. Team members independently noted interactions between respondents and interviewers based on predetermined guidelines and assessed the correctness of answers to the new questions. The group discussed each interview immediately after listening to it, and team members were almost always in complete agreement about their assessments of cases.

In all, the CWS team monitored about 100 interviews. It was clear that there were many false positives to both the in-person and online electronically mediated questions. Respondents had described the main job earlier in the interview, and they often mentioned additional details about the work when answering the in-person and online questions. For most “yes” responses, it was obvious that the reported work could not have been obtained through a website or app that also coordinated payment of the work. Staff monitoring interviews observed some common patterns.

Many respondents focused on the examples

Many respondents focused on the examples rather than on the definitions of electronically mediated work given in the questions. Additionally, if respondents hesitated, interviewers sometimes repeated only the examples. Consequently, many said “yes” to the question if any of their job duties resembled any of the examples included in the questions. For example, monitors heard the following responses:

·       “Yes, I drive my car to work.”

·       “Yes, I sometimes use a computer at work.”

·       “Yes, that describes part of what I do at work.”

·       “Yes, I’m a graphic designer.”

Also, many respondents who said they did in-person electronically mediated work for their main job also said they did online electronically mediated work for that same job. It is highly unlikely that people did both electronically mediated work in person and entirely online for the same job.¹⁶

Many respondents said “yes” if they used websites or mobile apps in their work

Some respondents with traditional jobs used websites or mobile apps in their work. Some of these websites and apps did not facilitate electronically mediated work, but respondents gave affirmative answers to the questions anyway. Many answered “yes” if they obtained clients or jobs using a website or mobile app even if they were not paid through that website or app. Examples of respondents in this type of situation include the following:

·       A real estate agent who obtained customers through the web

·       A gravel delivery person who used an app to obtain route directions

·       A fast-food worker who prepared orders that customers placed through an app

Many respondents said “yes” to the questions if they used a computer for work

Some respondents appeared to think the questions were asking about whether they used a computer in their work. A number of respondents said “yes” to the questions and listed as examples work that was clearly not electronically mediated. Examples of respondents in this type of situation include the following:

·       A university lecturer who did all work online (lectures, student interactions, etc.)

·       A technical support person who was connected to people to help through the internet

·       A receptionist in a doctor’s office who scheduled appointments using a computer

Many interviewers did not seem to understand the goal of the questions

By asking unscripted probes, the interviewer can help respondents determine the response option that best fits. However, BLS staff rarely observed interviewers probing when necessary for correct answers or providing explanations to confused respondents. Instead, many interviewers simply repeated the examples in the questions. In addition, interviewers sometimes could not interpret the respondents’ answers. In a few cases, interviewers intervened to change previously correct answers, saying things such as “but you do use a computer, don’t you?” for the in-person question. In response to these types of inquiries from interviewers, respondents’ correct “no” answers were occasionally converted to incorrect “yes” answers.

Conclusions from the monitoring

The team observed many false “yes” answers to both the in-person and online questions. In general, both questions appear to have been too complicated. In order for a “yes” answer to be accurate, several conditions needed to be true. Many respondents did not seem to consider all of the necessary conditions and instead responded “yes” when only one of the conditions was true. While the team concluded that both questions had a high number of false positive responses, they observed no false negatives.

Evaluating the data: microdata review

The CWS team then turned to examining records on the confidential microdata file. While this file does not contain as much information about each case as a taped interview, it does include answers for other questions in the CPS and the CWS, including respondents’ verbatim descriptions of job duties, employer name, industry, and occupation. In addition, the file contains information about usual work hours; whether the person worked for the government, a for-profit firm, or a nonprofit firm; and self-employment status. The file also contains CWS information about whether people were independent contractors or in other alternative employment arrangements on their main job.

The electronically mediated work questions were asked about more than 46,000 people, and there were relatively few “yes” responses—about 1,600 for the in-person question, the online question, or both. Most of these answers indicated that the work was done for a person’s main job, and BLS could obtain information about those jobs using the confidential microdata file. A quick review reinforced what had been observed in the monitoring—that many of these “yes” answers were clearly false positives. For example, the file showed that “yes” answers for the in-person question had been recorded for the following main jobs:

·       Vice president of a major bank

·       Manager of a fast-food restaurant

·       Local police officer

·       Surgeon at a large hospital

For the online question, “yes” answers were often given for people who used computers or mobile apps in their work, even though not all of them had done electronically mediated work. Many people with “yes” answers, though not all, clearly could not have done all of their work entirely online. Examples of cases with likely false positives for the online question include the following (again, these are people who said they did this work for their main job):

·       Medical assistant administering medication to patients

·       Hair stylist

·       Railroad engineer

·       Front desk clerk at a motel

BLS also examined records with “no” responses for the electronically mediated questions. A quick review reinforced the conclusions from the monitoring—that is, the vast majority of negative answers for both the in-person and online questions appeared to be correct.

Given that both the monitoring and evaluation revealed that the questions had not worked as intended, BLS considered whether the data were too flawed to release. Although there were many false positives, false negatives did not seem to be a problem. Therefore, the team decided to use information on the confidential microdata file to see whether incorrectly coded cases could be identified using the verbatim information on the confidential microdata file.

The team devised a test to determine whether false positives could be identified, first creating guidelines to help identify whether electronically mediated work had been done. For example, respondents who worked for the federal, state, or local government were unlikely to be electronically mediated workers. The team agreed that unclear cases should be assumed to be correct. (See appendix A for a complete list of the guidelines.)

Using the guidelines the team had developed, a group of 5 staff members evaluated 100 records with “yes” answers to the in-person question. Key information about each case was read aloud, and each of the five team members independently evaluated whether the respondents’ answers were compatible with electronically mediated work, assigning answers of “yes,” “no,” or “maybe” for each case. Team members’ determinations were not discussed during the evaluation session. After all 100 cases had been evaluated, team members’ responses were compared. For a substantial number of records, the team members had unanimously agreed that the “yes” answer was incorrect. They also agreed unanimously that a few cases definitely had correct answers. The test confirmed to the team that many false positives could be identified through a recoding process.

Recoding the data

Based on the results of the recoding test, the team decided to evaluate all records with affirmative answers to the in-person and online questions and recode erroneous answers when possible. Information is collected for both main and second jobs in the CPS, so any evaluation of answers needed to consider the job for which the electronically mediated work was done. The team devised three approaches for reviewing data that depended on respondents’ answers to the “which job” questions—that is, work done for the main job, a second job, or additional work for pay. The team also reviewed a sample of “no” answers to check for false negatives. 

Electronically mediated work for the main job

The vast majority of respondents who said “yes” to either the in-person or online questions reported that the electronically mediated work had been done for their main job (or their household members’ main job). Of the 912 “yes” answers for the in-person question, 826 (91 percent) were for the main job. Of the 963 “yes” answers for the online question, 917 (95 percent) were for the main job.

The CWS team reviewed each record with a “yes” answer to the in-person or online question and an answer of “main job” to the corresponding “which job” question. The review was done in a systematic fashion by groups of five team members, and the in-person and online questions were evaluated separately. As with the recoding test, key information about each case was read aloud, and each of the five team members independently evaluated whether they thought the respondent had done electronically mediated work, assigning answers of “yes,” “no,” or “maybe” for each case. Team members’ determinations were not discussed during the evaluation sessions.

Cases with four “no” answers and one “maybe” answer were assumed to be false positives, as were cases that had unanimously been assigned “no” by all five team members. Once this review was completed, the number of records with “yes” answers for the in-person question had been reduced from 826 on the main job to 184, and the number of “yes” answers to the online question had been reduced from 917 on the main job to 167. Team members believed that, while they had identified many false positives, there were likely additional false positives that could not be identified given the available data.

Electronically mediated work for the second job

Almost all respondents who answered “yes” to either the in-person or online questions said that the work was done for the main job. However, a small number of people said this work was done for the second job. Five percent of “yes” responses for the in-person question and 2 percent for the online question were for a second job.

Because of the survey design, the CPS has less information on second jobs than it does for main jobs. Each month, information about job duties, employer name, occupation, and industry for the second job is collected of only about one-fourth of multiple jobholders.¹⁷ For the records with detailed information about second jobs, the team did an evaluation similar to that done for the main job. The team evaluated respondents’ verbatim descriptions and other information to determine whether a “yes” answer for the electronically mediated work questions should have been coded as a “no.” For the three-fourths of records for which no additional information was available, the response provided was accepted without recoding.

As with the exercise done for main job, the team evaluated answers for the in-person and online questions independently and identified a small number of false positives. Because so few records were evaluated, the team was not able to conclude whether “yes” answers were more likely to be correct for the second job than for the main job. After recoding the records for which information was available, the affirmative answers for the in-person question decreased from 50 to 48 and the affirmative answers for the online question decreased from 24 to 19. It is likely that, had information about second jobs been available for the other three-fourths of multiple jobholders, more “yes” answers would have been recoded to “no.”

Electronically mediated work as additional work for pay

A small number of respondents reported that they or their household members did electronically mediated work for “additional work for pay”—4 percent of the “yes” responses for in-person work and 2 percent for online work. The confidential microdata file does not contain any information about what respondents did for additional work for pay. Because the CWS team had no additional information about these respondents, their answers were accepted and were not reviewed. In addition, answers were assumed to be correct for the very small number of respondents who said that they or their household members did electronically mediated work but did not answer the “which job” questions.

False negatives

Although the monitoring suggested there was not a problem with incorrect “no” answers, BLS used the microdata to look for false negatives in two ways. First, staff looked at cases with “no” answers in occupations in which anecdotal evidence suggests there may be high numbers of electronically mediated workers. Staff members saw no evidence of a substantial problem with false negatives in these occupations.

Second, BLS identified records containing selected keywords. Keywords included businesses that commonly facilitate electronically mediated work, such as Uber, Lyft, TaskRabbit, Handy, Amazon mTurk, and Crowdflower. Words associated with electronically mediated work, such as taxi, freelance, and ride share, were also included.¹⁸ Using the verbatim descriptions of job duties, employer name, occupation, and industry, a team of five staff members evaluated each case to determine whether an incorrect answer of “no” had been recorded for the in-person and online questions. Team members identified a handful of incorrect “no” answers—9 records out of about 175 cases with the selected keywords. (Many of the correct “no” answers were taxi drivers and handymen who clearly had not done electronically mediated work.) Note that this was not a random sample. Rather, these records were chosen as being the most likely to have false negatives among all those with “no” answers. Because the number of false negatives identified was so small, the team concluded that false negatives were of little concern overall.

Comparing collected data with recoded data

Final results of the recoding, taking into account both the false positives and the handful of false negatives found by the team, served to lower the number of observations with “yes” answers for the in-person electronically mediated work question from 912 to 277. Recoding lowered the number of “yes” answers to the online question from 963 to 208. For both the in-person and online questions, most of the answers that were changed were for workers who had done electronically mediated work for their main job. This was partly because most cases were for the main job, and partly because BLS had information about virtually all main jobs. False positives for cases in which the electronically mediated work was done for the second job were less common, but the BLS team could only evaluate about one-fourth of those cases because of the lack of information about second jobs on the microdata file. Lacking any information about what was done as additional work for pay, the BLS team could not recode any additional-work-for-pay cases.

The recoding of in-person and online electronically mediated work was done independently. There was no attempt to ensure that “yes” answers did not occur for both questions, even though the BLS team agreed that someone was highly unlikely to do electronically mediated work both in-person and entirely online for the same job. Despite this fact, the number of cases with “yes” answers for both in-person and online electronically mediated work was reduced sharply—from 293 in the collected data to 23 in the recoded data.

Weighted estimates showed that the broad demographic characteristics of electronically mediated workers were similar for both the collected and recoded data. However, there were a number of differences by industry. (See table 1.)

Most notably, 22 percent of electronically mediated workers in the recoded data were in the transportation and utilities industry on their main job, over twice the share of transportation and utilities workers found in the collected data (10 percent). Reflecting the relatively large share of electronically mediated workers who were ride-share drivers, the difference was particularly great for electronically mediated workers who did their jobs in person—35 percent as recoded and 14 percent as collected. In addition, the share of electronically mediated workers in professional and business services was higher for the recoded data (31 percent) than for the collected data (20 percent). Many technical jobs in the industry are sometimes electronically mediated, such as graphic design, copy editing, and computer programming. Among electronically mediated workers who did their work entirely online, the recoded data share was about double that of the collected data—51 percent versus 25 percent.

By class of worker—that is, whether people were wage and salary workers or self-employed—the characteristics are somewhat different for the recoded and collected data. In the recoded data, 4 percent of electronically mediated workers were employed in government on their main job, compared with 11 percent in the collected data. (Although people are unlikely to do electronically mediated work for a government job, some workers employed by the government on their main job did electronically mediated work for a second job or for additional work for pay. In addition, data were not recoded for a small number of cases because there was insufficient verbatim information on the confidential microdata file.) Moreover, the share of electronically mediated workers who were self-employed workers with unincorporated businesses was 30 percent in the recoded data, twice the share as in the collected data (15 percent). (Detailed estimates showing the impact of recoding on in-person and online electronically mediated work are available in appendix B.)

For both the collected and recoded data, table 2 shows the numbers and percentages of in-person and online electronically mediated workers who did this work for their main job, a second job, or additional work for pay. The share who did in-person electronically mediated work for their main job was 91 percent in the collected data, higher than the 72 percent found in the recoded data. The difference was similar for online workers—94 percent in the collected data and 78 percent in the recoded data. However, this difference reflects the fact that BLS had more information about main jobs than about other jobs or additional work for pay, and BLS could recode many cases in which work was done for the main job. As mentioned earlier, the confidential file contains information for second jobs for only about one-fourth of multiple jobholders, and contains no information about the work done for additional work for pay. Consequently, BLS could recode very few cases in which work was done for the second job. The data reflect this, showing little difference in the number of people who did electronically mediated work for their second job in the collected and recoded data. None of the cases reporting additional work for pay were recoded. Thus, there are likely to be more false positives in the recoded data among those who did electronically mediated work for a second job or as additional work for pay. While BLS is confident in estimates of the number of people who did electronically mediated work for their main job, the number of people who did this work for a second job or for additional work for pay may be overstated. Because the team could not recode as many second-job cases or any additional-work-for-pay cases, percent distributions from the “which job” questions should be viewed with caution.

Conclusions from the recoding

BLS is confident that the recoded data provides a better picture of the number and characteristics of in-person and online electronically mediated workers than does the collected data. While the confidential file provided additional detail about jobs and was used to identify clear false positives, team members did not recode ambiguous cases. In addition, there was little or no information to recode people who did electronically mediated work for a second job or for additional work for pay, but relatively few responses were in these categories. Thus, while there are likely still some false positives in the recoded data, BLS believes that measures using recoded data more accurately represent the number and characteristics of electronically mediated workers than do measures using the collected data.

However, BLS does not recommend using data from the “which job” questions. In particular, percent distributions of in-person and online electronically mediated work done for the main job, second job, or additional work for pay are likely to be misleading.

Characteristics of the employed who did electronically mediated work

All estimates in this section are based on recoded data. BLS believes these data to be superior because they exclude the obvious false positives in the collected data. However, because the questions did not work as intended and there was not enough information to recode all cases, the recoded data may still have limitations. In addition, some of these estimates are based on relatively few observations, so variances may be large.

In May 2017, there were 1.6 million electronically mediated workers, accounting for 1.0 percent of total employment. (See table 3.) These workers obtained short jobs or tasks through websites or mobile apps that both connected them with customers and facilitated payment for the tasks. The estimates include all people who did electronically mediated work, whether for their main job, a second job, or additional work for pay. Of all workers, 0.6 percent did electronically mediated work in person and 0.5 percent did electronically mediated work entirely online. Note that some people did electronically mediated work both in person and entirely online. This can occur when people do electronically mediated work for two different jobs.

Electronically mediated workers were slightly more likely to be men than women, reflecting the fact that, overall, a higher percentage of the employed were men. (See table 4.) Compared with workers overall, electronically mediated workers were more likely to be in the prime-working-age category (25 to 54) and less likely to be in the oldest age category (55 and over). They were also more likely than workers overall to work part time.¹⁹

Blacks or African Americans accounted for 17 percent of electronically mediated workers, higher than their share of overall employment (12 percent). By contrast, Whites made up 75 percent of electronically mediated workers, slightly lower than their share of workers overall (79 percent). Hispanics or Latinos made up 16 percent of electronically mediated workers, and Asians accounted for 6 percent. Blacks were overrepresented among in-person electronically mediated workers (23 percent), while Whites were overrepresented among online workers (84 percent).

Educational attainment data are restricted to those age 25 and over because most people have completed their education by that age. Compared with workers overall, people who did electronically mediated work were more likely to have a bachelor’s degree or higher. This was driven by people who did their tasks entirely online; 67 percent of online electronically mediated workers had a bachelor’s degree or higher.

Self-employed workers were more likely than wage and salary workers to do electronically mediated work (4 percent versus 1 percent). (See table 3.) Five percent of self-employed workers whose businesses were unincorporated did such work, as did 2 percent of the self-employed with incorporated businesses.

By industry, workers in transportation and utilities (main job) were the most likely to have done electronically mediated work, with 5 percent of workers in this industry having done such work. Those employed in professional and business services, information, and other services were also more likely to do electronically mediated work, at 3 percent, 2 percent, and 2 percent, respectively. Workers in transportation and utilities and in other services were more likely to do in-person work, and those in professional and business services and in information were more likely to do online work.

Workers in the four alternative employment arrangements measured in the CWS—independent contractors, temporary help agency workers, on-call workers, and workers provided by contract firms—were more likely than workers in traditional arrangements to have done electronically mediated work. Independent contractors were the most likely to do electronically mediated work—6 percent did so in May 2017, compared with 3 percent of temporary help agency workers, 3 percent of on-call workers, and 2 percent of workers provided by contract firms. By contrast, less than 1 percent of workers in traditional arrangements were electronically mediated workers.

Lessons learned

BLS should not again attempt to collect data about electronically mediated work using the four new questions fielded in the May 2017 CWS. If BLS were to collect data about electronically mediated work in the future, the questions would need to be substantially revised. It may simply be that the concepts are too complicated for four questions to properly identify all the information BLS was attempting to measure.

BLS recognizes that a number of steps could be taken to improve future questions on all subjects, not simply those that concern electronically mediated work. Such strategies address ways to improve question development, cognitive testing, and interviewer training.

Cognitively test participants who are not in the population of interest

For complicated questions that are intended to measure a small portion of the population, it is important to test participants who do not fit the characteristic of interest as well as those who do. For the laboratory testing, BLS made the strategic decision to recruit people who had done in-person electronically mediated work or were in occupations with a large number of electronically mediated workers. These laboratory participants may have been more familiar with the concepts than people in occupations with little or no electronically mediated work. For example, even if they do no electronically mediated work, many taxi drivers may be familiar with such work because they know about Uber and Lyft drivers. They may be able to answer the questions more easily than people in occupations that have few electronically mediated workers, such as schoolteachers or firefighters. For the mTurk testing, all participants were at least somewhat familiar with electronically mediated work since they were using the mTurk platform. Thus, mTurk testing participants were doubtless more likely to be familiar with electronically mediated work than some others would have been. An increased sample size with a wider variation of occupations and work arrangements may have provided increased insight about the potential for false positives.

Conduct additional rounds of cognitive testing

Conducting multiple rounds of cognitive testing is a recommended best practice, particularly when questions involve complicated concepts or revisions are made during testing. Because of time and funding constraints, BLS committed to a tight timeframe for cognitive testing and conducted only one round of testing. Improvements may have been realized if—after analyzing the cognitive interview results and revising the questions—additional interviews had been conducted to evaluate the revisions.

Improve interviewer training

Interviewers clearly could have benefited from additional training. Because CPS interviewers are located all over the country, in-person classroom training would be prohibitively expensive. Less expensive ways to improve training include the following:

·       Additional time for the self-study material

·       Computer-based training modules with graded quizzes

·       Online training sessions

·       Web-based training or teleconferences

·       Practice interviews

Increase interviewer involvement in questionnaire and training design

CPS interviewers have a keen sense about what might be confusing to other interviewers and respondents. Meeting with interviewers prior to finalizing the questionnaire to identify possible problems with questions could improve both the questionnaire and training material. It could have an added benefit of increasing interviewer engagement.

Continue to learn from research

Electronically mediated work is an area that is being increasingly studied. Since the 2017 CWS was fielded, new findings have emerged that could have been helpful in the question design process. For example, Statistics Finland, when attempting to measure “platform jobs,” found that they needed to include company names in the question in order to get accurate responses.²⁰ BLS should keep abreast of new research, continue to work with outside experts, and leverage the efforts of others when designing questions.

Appendix A. Guidelines for recoding

Each record was reviewed by multiple staff members. In order to determine whether responses were correct, they looked at the following pieces of information: 

·       Verbatim descriptions of industry, occupation, and duties on the job

·       Company name

·       Class of worker (whether wage and salary worker or self-employed)

·       Whether the person is an independent contractor, an on-call worker, a temporary help agency worker, or a worker provided by a contract firm on the main job

·       The number of hours a person usually works

Answers were not recoded if there was not enough information to determine, with a high probability, that a person was not doing electronically mediated work.

Appendix B. Detailed information on the effect of the BLS data recoding process