DREAM’s First Ever Hackathon! AML Challenge Organizers Run Hackathon to Foster Collaboration

Although the DREAM Challenges do an excellent job bringing together researchers from several different areas of science and several different institutions to work on the same problem, the competitive setting provides little incentive for these great minds to work together. While the cornerstone of crowdsourcing is in fact the application of several different approaches to the same issue, the organizers of the Amyloid Myeloid Leukemia (AML) Outcome Prediction DREAM Challenge would still like to see participants working on similar or complementary approaches collaborating and possibly even joining teams. To encourage these partnerships we decided to hold the first ever DREAM Hackathon! The Hackathon took place on July 26-27 at Rice University and was simultaneously broadcast over the web. The Hackathon was designed to catalyze conversations about the AML Challenge in two ways:

First, we wanted to use the Hackathon to encourage Challenge participants to share a little about their general approach to model building. While we wanted participants to present their ideas, we were also mindful that the AML Outcome Prediction DREAM Challenge is a competition, and we didn’t want participants to present their approach at a level of detail that could make their methodology available to others. Considering this fine line, we encouraged participants to present only as much as they were comfortable with: no details about the approach or efficacy of their methods had to be presented.

For this part of the Hackathon, two Challenge teams signed up and presented an overview of their model-building methodologies via live webinar to other Challenge participants around the world. Both teams had excellent presentations and received some valuable feedback from Hackathon-Challenge participants! While both presenters had very good approaches, it was clear that Hackathon spectators with different expertise and “fresh eyes” had numerous good ideas on how these two teams could improve even further. In the end, while the turnout for this section of the Hackathon was low, the exercise was very constructive and the main goal was achieved for those who participated.

The second goal we had for the Hackathon was to invite a few “experts in the field” — for both model building methodology as well as approaches to predict AML outcome — to present during the Hackathon in order to get participants discussing new ideas that could help their model building efforts.   These talks along with their Q&A sessions were very constructive, particularly the one by Dr. Kenneth Hess from MD Anderson Cancer Center, who presented on general statistical analysis of survival time data tailored to the DREAM 9 dataset. Given the diverse nature of these talks, we believe participants of the AML Outcome Prediction DREAM Challenge were able to get an insight on different approaches that could be incorporated into their methods. These talks could also give them different perspectives on the Challenge, opening new horizons not considered before. These presentation were recorded and are available in the Synapse website (https://www.synapse.org/#!Synapse:syn2455683/wiki/64687).

Overall I believe this was a really successful first DREAM Hackathon! We will continue to follow up with the participants about what they liked or didn’t like about this event, and we are open to ideas on how we can improve. With your help we hope to make this event more and more successful in subsequent editions of the DREAM Challenges.

Thank you,
André Schultz: member of the AML DREAM Challenge Organizing Team



FasterCures Webinar On Crowdsourcing and DREAM Challenges

“The way biomedical research is carried out is changing fundamentally,” Sage Bionetworks President Stephen Friend declared at the beginning of a webinar about the crowdsourced computational challenges Sage is facilitating in partnership with the DREAM (Dialogue for Reverse Engineering Assessment and Methods) project that originated at IBM. Friend laid out five opportunities he believes are giving rise to new ways to generate, analyze, and support new research models:

– It’s now possible to generate massive amounts of human “omic’s” data.
– Network modeling approaches for diseases are emerging.
– Information technology infrastructure and cloud computing capacity allow an open approach to biomedical problem solving.
– There’s an emerging movement for patients to control their own sensitive information, allowing sharing.
– Open social media allows citizens and experts to use gaming to solve problems.

“The usual rule of anointed experts being the only ones who can solve problems has really been shattered,” said Friend.

For several years, Sage has been grappling with how to bring about a better understanding of the complexity of biology, given these trends. One initiative central to their efforts has been the creation of a technology platform for data sharing and analysis called Synapse, built on the model of “github” from the open-source software world, which allows distributed projects to get done and provides the foundation for running the DREAM Challenges.

Friend noted that computational biology has been driven by crowdsourcing for a long time, and challenges like those that DREAM has been running for many years have been integral to its successes. There are increasingly large and powerful sets of data in the public domain, and putting them out for many people to look at (some of them from outside the field of biology) and make predictions and unbiased evaluations based on the data is critical to solving complex problems in biology in this day and age. Data is getting so complex that it’s impossible for any single researcher or institution to analyze it effectively. As John Wilbanks, Sage’s Chief Commons Officer and a FasterCures Senior Fellow, noted, “One of the hardest things to do in the emerging Big Data world is to get your data analyzed.”

An important aim of these challenges is to foster a new culture in research. As Friend argues, “We have a serious need not just to solve specific problems, but … to build communities so that people begin to think of each other as colleagues and collaborators.” DREAM Challenges are carefully constructed to provide opportunities for publications in journals and for other forms of recognition that are important to researchers, often more important than the promise of a monetary prize.

First of the four past challenges run by Sage and DREAM (along with partners from academia, industry, government, and patient groups) was the Breast Cancer Prognosis Challenge, created to forge a computational model that accurately predicts breast cancer survival. The winning team was from the academic lab that invented the MP3 format for digital audio, bringing their expertise in data compression to the task. Hundreds of teams comprised of thousands of individuals have participated, and a number of publications have resulted, along with other opportunities for professional advancement for “solvers.”

Challenges currently open include:

– The Somatic Mutation Calling Challenge, to predict cancer-associated mutations from whole-genomic sequencing data;
– The Rheumatoid Arthritis Responder Challenge (in partnership with the Arthritis Foundation, among others), to predict which patients will not respond to anti-TNF therapy – a clinical trial could follow if a powerful classifier emerges from the Challenge for validation; and
– The Alzheimer’s Disease Big Data Challenge, which seeks to predict early AD-related cognitive decline and the mismatch between high amyloid levels and cognitive decline. Massive amounts of data in the public domain has been aggregated, collated, massaged and curated for the task.

Two more are set to open this summer, in partnership with the Broad Institute and MD Anderson Cancer Center, and several more are being considered for launch by the end of 2014. All stakeholders – including and perhaps especially patient groups – are invited to participate by proposing ideas for challenges, contributing data, recruiting teams to participate. The Sage-DREAM Challenges are looking for partners who want not only to find the answers to tough questions in their fields, but who want to help create the conditions for the real collaboration necessary to bring about “the next generation of biomedical research.”

For more information on how to get involved with an open DREAM Challenge, click here.

View webinar slides and recording

(Cross posted from http://fastercures.tumblr.com/post/81603549119/crowdsourcing-data-challenges-to-speed-the-search-for)

ICGC-TCGA SMC DREAM Challenge highlighted in Nature Genetics

A great correspondence was published in Nature Genetics today regarding the ICGC-TCGA DREAM Somatic Mutation Calling (SMC) Challenge¹. Organizers highlight the unique nature of this challenge including its possible impact on the broad research community, the ability for challenge infrastructure to assist in the peer review process, and the resulting ‘living benchmark’ for the bioinformatics community.

To get more information or to sign up for the Somatic Mutation Calling Challenge, visit the SMC Challenge Project in Synapse or watch the kickoff webinar.


¹Butros P, Ewing A, Ellrott K, Norman T, Dang K, Hu Y, Kellen M, Suver C, Bare C, Stein L, Spellman P, Stolovitzky G, Friend S, Margolin A, Stuart J. Global optimization of somatic variant identification in cancer genomes with a global community challenge. Nature Genetics 46, 318–319 (2014). doi:10.1038/ng.2932

ICGC-TCGA Mutation Calling Challenge Webinar

The ICGC-TCGA DREAM Genomic Mutation Calling Challenge (open for participation Nov 2013 — Summer 2014) is an international effort to improve standard methods for identifying cancer-associated mutations and rearrangements in whole-genome sequencing (WGS) data. The goal of this somatic mutation calling (SMC) Challenge is to identify the most accurate mutation detection algorithms, and establish the state-of-the-art. The algorithms in this Challenge must use as input WGS data from tumour and normal samples and output mutation calls associated with cancer.

In this January 29, 2014 webinar, Challenge participants were invited to hear presenations and participate in a live Q&A session about the Challenge. The webinar video consists of the following three sections:

  1. Background and motivation for the Challenge (Paul Boutros: SMC Challenge Leader)
  2. Demo of Challenge web services to show you how to participate (Chris Bare: Sage Bionetworks)
  3. Answering your questions in real-time



For more information about all DREAM Challenges, please visit the DREAM web presence on Synapse.