‘More Shots on Goal’
Prizes, Challenges Spur Innovation, Tap Global Talent
Photo: Bill Branson
If you want to get something done in this world, and your current workforce isn’t exactly delivering the mail, open up your challenge to the crowd; you might just find cheaper, faster, more powerful solutions, especially if the problem involves bludgeoning big data with sharper algorithms.
That was the take-home message at an NIH symposium on challenges held Sept. 30 in the Porter Neuroscience Research Center, sponsored by the Office of the Associate Director for Data Science.
The usefulness of the carrot as motivation was presented from two angles: Tom Kalil, deputy director for technology and innovation at the White House’s Office of Science and Technology Policy, outlined the long history of challenges as a road to innovation and explained why the Obama administration has backed the America COMPETES Act and launched its own Strategy for American Innovation. Dr. Karim Lakhani, professor of business administration at Harvard Business School, who runs a Crowd Innovation Lab, offered astonishing examples of the success of global crowd-sourcing, especially for data science problems.
Finally, an NIH panel offered home-grown examples of challenges involving NHLBI, NCI, NIBIB, NEI and NINDS. “NIH has run about two dozen challenges since 2010,” said session moderator Elizabeth Kittrie of OD. More than 700 incentive prizes have been established government-wide; visit www.challenge.gov, a site administered by the General Services Administration, for a complete list.
“The use of challenges and prizes is not a new idea,” said Kalil; the British Parliament in the 1700s offered a prize for solving the longitude problem “because they were sick of losing ships.” The first X Prize, in 1996, offered $10 million to a team that could rocket the equivalent of 3 people into space and bring them back safely twice within 2 weeks. “They have demonstrated that space is not just for governments anymore,” he noted.
When DARPA held a self-driving car competition, the CEO of Google was at the finish line, to acquire the winning team, Kalil reported.
“One of the big advantages [of challenges] is that the sponsor only pays if someone is successful,” Kalil explained. “The sponsor simply sets a goal and is agnostic about what approaches are used.”
He recalled that Charles Lindbergh’s successful solo trans-Atlantic flight was spurred by a $25,000 purse offered by a New York hotelier. “Everyone thought [Lindbergh] was crazy.”
Photo: Bill Branson
Challenges “broaden the pool of expertise” to address problems, Kalil explained, citing Sun Microsystems co-founder Bill Joy’s dictum, “No matter who you are, most of the smartest people work for someone else.”
“All agencies have been given the authority to sponsor prizes, up to $50 million [with larger prizes dependent on congressional approval],” Kalil said, “and the ability to do public/private partnerships.”
Kalil urged NIH to make prizes and challenges part of its investment strategy, given the “explosion of data in the biomedical area.” He cited advances in artificial intelligence and machine learning that already show promise in breast cancer diagnosis.
“We need to take this field [of incentive prizes] from the craft, the artisanal and the boutique to more like ordering a laptop online,” he said.
Kalil suggested several next steps for NIH to consider, such as providing professional development on incentive prizes for program managers, investing in a team similar to the NASA Tournament Lab that could help NIH and NIH-funded principal investigators design and manage prizes, and making it easier for the NIH community to access “innovation marketplaces” such as Kaggle and TopCoder.
Photo: Bill Branson
Kalil also encouraged NIH to replicate NEI’s Audacious Goals Initiative. “I think a great question to pose to the research community would be, ‘What data sets would accelerate progress in the application of AI and machine learning to biomedical research?’”
Lakhani said he became an academic when customers while he was employed at GE Medical were out-innovating GE engineers; he was keen to know if the best solutions were hiding somewhere out of reach. His research at Harvard has proven that crowds often provide cheaper, faster and better solutions than those created internally, and that a “crowd” can be as small as two dozen people.
Like Kalil, he cited historical evidence for the value of crowd-sourcing: it resulted in construction of Il Duomo (The Dome) in Florence, Italy, in 1418, and the development of canned food, which Napoleon commissioned, in order to feed distant armies.
Lakhani’s lab works with NASA, Harvard Medical School, the Broad Institute and a 1-million-member online platform called TopCoder and has been involved in more than 700 contests in the past 7 years, for a variety of software applications. Of 20 challenges involving computational algorithms, 18 submissions exceeded benchmarks, he said.
Three years ago, his lab ran a 3-week contest with NASA to find a way to power the International Space Station more efficiently. The prize? “Decimal dust,” said Lakhani—$30,000. Four-hundred fifty-nine people from around the world submitted solutions, more than half of which met or exceeded the NASA solution. Ten prizes were awarded.
“We were stunned,” said Lakhani. “This works.” NASA now has a Center of Excellence for Collaborative Innovation that uses task orders to run contests, he said. [And on Oct. 11, the NASA Tournament Lab announced the “Space Poop Challenge,” which offers $30,000 in prizes to anyone who can develop a system inside a space suit that collects human waste for up to 144 hours and routes it away from the body, without the use of hands.]
Photo: Bill Branson
The life sciences are ripe for crowd-sourcing due to a data explosion that is “already in place,” said Lakhani. “We are moving from petabytes (1015) to exabytes (1018) of information. The Broad is already generating 25 terabytes of information per day…I tell my students we are never going to be in a world with less data.”
Lakhani said the labor market is already short 1.8 million data scientists. “There is a war for talent, intense competition, from places like Uber and Lyft to space exploration…With the rise of deep learning, we need new approaches to solve data challenges.”
To illustrate technology’s transformational speed, and economic impact, he used a personal example: As a kid growing up in Toronto, he convinced his parents to buy him the Encyclopedia Brittanica, which cost about $2,000 and is updated every 10 years. Only a few years later, Microsoft introduced Encarta, which cost $100 and is updated annually. Then came Wikipedia, which is free, and is updated continuously.
“The models for organizing creative effort are changing,” Lakhani said, noting that the Brittanica stopped publishing in 2012.
Five years ago, his group wondered, “Are crowds smarter than Harvard Medical School?” A 2-week competition offering a $6,000 prize drew 654 submissions from 122 coders. “Thirty-four exceeded the state of the art by orders of magnitude,” Lakhani said. Eighty-nine different ways to solve the challenge—annotating 10 million sequences in under 3 minutes, and 0.25 billion sequences in less than an hour on a laptop—arrived from Russia, France, Egypt, Belgium and the U.S.
A genomic data processing challenge involving the Broad pitted one of its premier scientists—“a highly skeptical participant,” Lakhani noted—lasted 3 weeks and offered a $20,000 prize. Twelve hundred people registered, 96 solutions were submitted and the contest was won on its first day by a person who exceeded the benchmark by 87 percent, increased the speed of the computation 15-fold and boosted accuracy from 91 to 97 percent.
“Incentives in academia are not set up to produce this kind of result,” Lakhani observed.
He offered more examples of the power of crowds; interestingly, winners and top finishers included high school students, the unemployed, and people whose study of biology ended in high school. Learning was a crucial motivation for participating, along with enjoyment and a taste for problem-solving. “It’s a big, big driver,” Lakhani said. Cash, prestige and job-seeking were lesser factors, he noted.
In sum, contests provide incentives of varying weight, along with parallel search, with the result that “we get more shots on goal,” said Lakhani. “It goes back to STAT 101—you get extreme value if you do it a lot of times.”
He concluded, “Maybe we should start with the crowd first, not last. It is a cheap, efficient and fast way to uncover the frontier.” NIH and its grantees should have access to the crowd, too, he said.
During the panel session, all agreed that contests should augment, but not take the place of, grants and contracts; they are not a substitute for internal efforts. “We pick winners before the job is done with traditional grants and contracts,” said Sandeep Patel of the HHS Idea Lab. “With contests, it’s hard to predict where the solution will come from.”
Each IC at NIH has challenge managers, said Lora Kutkat, who has formulated NIH policy on challenges since 2011. Current prizes here range from $500 to $20 million, for the Anti-Microbial Resistance Challenge, she said. Reaching non-traditional audiences, the so-called “solvers and seekers,” is her favorite aspect of the mechanism. She cautioned, however, that NIH’ers can’t compete, either on or off-duty, in NIH-sponsored challenges.
“The best starting point,” said Kutkat, “is to clearly define the problem and the kind of solution you’d like.”
Said Chris Nelson of OSTP, “Challenges are a way to bring unusual suspects into this game.”
Quoting an economist, Lakhani noted, “We all have what is known as ‘cognitive surplus’—from the person doing Sudoku on the Metro to the guy watching Game of Thrones…there are many ways to spend it. I wouldn’t see [challenge participation] as a waste of time or leisure…These platforms are hungry for demand.”