1. Expert Crowdsourcing with Flash Teams
Daniela Retelny, Sébastien Robaszkiewicz, Alexandra To, Michael Bernstein
{dretelny, robi, ato1120, msb}@cs.stanford.edu
Crowdsourcing is Plateauing in Complexity Flash Team Examples
Crowd workers struggle to complete open ended and highly Our evaluation of flash teams to date includes the successful completion of eight different flash team examples,
utilizing different team structures and workflows. The flash teams varied in size, complexity and expertise and
interdependent tasks.
they produced outputs that ranged in completion time and quality. The images below depict the team structures,
Despite good intentions, teams of crowd experts are
workflows and output from three napkin sketch, animation and MOOC flash team examples we completed.
typically neither effective nor efficient at these tasks.
Expert crowdsourcing lacks generalizable techniques for
guiding crowds through complex tasks.
Flash Teams
We introduce flash teams, modular team structures that
computationally guide a team of crowd experts reproducibly
and at scale. These modular structures enable crowd expert
teams to grow, adapt, and recombine into larger organizations.
Defining attributes of flash teams:
MODULARITY: self-contained and replicable team
structures composed of crowd experts that are
computationally guided.
ELASTICITY: parameterizable team growth on demand,
allowing teams to grow and shrink as expertise are needed.
PIPELINING: parallelism enabled by defining
interdependencies and handoffs.
Methods:
To explore the feasibility and scope of flash teams, and
understand the effectiveness of modularity, elasticity and
pipelining, we gathered thirteen different flash teams to
complete complex tasks ranging from design to education.
Foundry, a Flash Team Authoring Platform
To encapsulate the foundations of the flash teams formalism and lessen
We recruit experts through oDesk, a crowdsourcing platform the threshold to creating flash teams, we created Foundry, a web platform
that allows requesters to create, reuse and recombine flash team
for expert work that allows requesters to post a request
under a specific expertise category (e.g., Python development, structures and workflows and allows team members to coordinate tasks
and track progress. The image to the right shows a screenshot of Foundry.
sound production, or visual design).
http://hci.stanford.edu
STANFORD HCI GROUP