User:Syksyinen/sandbox

= DREAM Challenges =

DREAM Challenges (Dialogue for Reverse Engineering Assessment and Methods) is a non-profit initiative for advancing biomedical and systems biology research via crowd-sourced competitions. Started in 2006, DREAM challenges collaborate with Sage Bionetworks to provide a platform for competitions run on the Synapse platform. Over 60 DREAM challenges have been conducted over the span of over 15 years, resulting in a large quantity of scientific publications.

Overview
DREAM Challenges were founded in 2006 by Gustavo Stolovizky from IBM Research and Andrea Califano from Columbia University. Current chair of the DREAM organization is Paul Boutros from University of California. Further organization spans emeritus chairs Justin Guinney and Gustavo Stolovizky, and multiple DREAM directors.

Motivation for DREAM Challenges is that via crowd-sourcing data analysis in specific biomedical applications to a larger audience via competitions, better models and insight is gained than if the analysis was conducted by a single entity. Prior results have been published in such scientific venues as the flagship journals of the Nature Portfolio and PLOS. Results of DREAM challenges are announced via web platforms, and the top performing participants are invited to present their results in the annual RECOMB/ISCB Conferences with RSG/DREAM organized by the International Society for Computational Biology.

While DREAM Challenges have emphasized open science and data, in order to mitigate issues rising from highly sensitive data such as genomics in patient cohorts, there have been multiple instances of "model to data" approach. In such challenges participants submit their models via containers such as Docker or Singularity. This allows retaining confidentiality of the original data as these containers are then run by the organizers on the confidential data. This differs from the more traditional open data model, where participants submit predictions directly for the provided open data.

Challenge organization
Each DREAM challenge typically comprises of a core DREAM/Sage Bionetworks organization group as well as a scientific expert group, who may have contributed to creation and conception of the challenge or by providing key data. Additionally, new DREAM challenges may be proposed by the wider research community. Pharmaceutical companies or other private entities may also be involved in DREAM challenges, for example in providing data.

Challenge structure
Timelines for key stages (such as introduction webinars, model submission deadlines, and final deadline for participation) are provided in advance. After the winners are announced, organizers start collaborating with the top performing participants to conduct post hoc analyses for a publication describing key findings from the competition.

Challenges may be split into sub-challenges, each addressing a different subtopic within the research question. For example, regarding cancer treatment efficacy predictions, these may be separate predictions for progression-free survival, overall survival, best overall response according to RECIST, or exact time until event (progression or death).

Incentives
The main incentives offered by DREAM challenges for top performing participants are by-line authorships in scientific publications presenting the key results. In addition, there have been challenges where monetary rewards have been offered by sponsors.

Participation
During DREAM challenges, participants typically build models on provided data, may utilize outside sources as additional data, and submit predictions or model that are then validated on held-out data. While DREAM challenges avoid leaking validation data to participants, there are typically mid-challenge submission leaderboards available to assist participants in evaluating their performance on a sub-sampled or scrambled dataset.

DREAM challenges are free for participants. During the open phase anybody can register via Synapse to participate either individually or as a team of participants. A person may only only register once and may not use any aliases.

There are some exceptions, which disqualify an individual from participating, for example:


 * Person has privileged access to the data for the particular challenge, thus providing them with an unfair advantage.
 * Person has been caught or is under suspicion of cheating or abusing previous DREAM Challenges.
 * Person is a minor (under age 18 or the age of majority in jurisdiction of residence). This may be alleviated via parental consent.

Evaluation of submitted models
Final submissions are validated based on a suitable performance metric tailored for the challenge, for example:


 * Condordance index for survival outcomes.
 * ROC-AUC for binary classification tasks.

The final submission performance metrics are typically bootstrapped to identify top performers. Multiple top performers may be announced, as based on sampling their relative ranking can be considered equal after accounting for random variation in the performance metric.

List of past or on-going challenges
As challenges open and close, below list may be inaccurate and non-exhaustive.