ICML 2013 Challenges in Representation Learning

Organizers: Ian Goodfellow, Dumitru Erhan, Yoshua Bengio

Welcome to the website for the ICML 2013 Workshop in Challenges in Representation Learning. The workshop will be held on Friday, June 21 in Atlanta, GA. For those at the conference, it is in rooms L401-3.


There has been a great deal of recent work on learning useful representations of data, much of it emerging from researchers interested in training deep architectures. Deep learning methods such as deep belief networks, sparse coding­-based methods, convolutional networks, deep Boltzmann machines, and dropout have shown promise as a means of learning invariant representations of data and have already been successfully applied to a variety of tasks in computer vision, audio processing, natural language processing, information retrieval, robotics, drug design and finance. Bayesian nonparametric methods and other hierarchical graphical model­-based approaches have also been recently shown the ability to learn rich representations of data.


The full schedule for the workshop is available here.

Invited Speakers

Our workshop will feature four invited speakers who have been chosen for the influence they have had on the field of representation learning:

Arthur Szlam, CCNY. “Some variations on K-Subspaces”
Ilya Sutskever, University of Toronto, Google. “Learning Control Laws with Recurrent Neural Networks”
Grégoire Montavon, Berlin T.U. “Deep Learning of Molecular Electronic Properties in Chemical Compound Space”
Ruslan Salakhutdinov, University of Toronto. “Annealing Between Distributions by Averaging Moments”


For this workshop we invited the community to compete in three challenges intended to advance the field of representation learning. See the challenges page for more information.

Accepted papers

Representation Learning papers

These papers are about representation learning, but do not necessarily relate directly to the specific challenges we highlighted with contests.

Nitish Srivastava and Ruslan Salakhutdinov. “Discriminative Transfer Learning with Tree-based Priors

Yichuan Tang and Ruslan Salakhutdinov. “A New Learning Algorithm for Stochastic Feedforward Neural Nets

Misha Denil, Babak Shakibi, Laurent Dinh, Marc’Aurelio Ranzato, Nando de Freitas. “Predicting Parameters in Deep Learning

Roger Grosse, Chris Maddison, Ruslan Salakhutdinov. “Annealing Between Distributions by Averaging Moments

Olgert Denas and James Taylor. “Deep modeling of gene expression regulation in an Erythropoiesis model

James Bergstra and David D. Cox. “Hyperparameter Optimization and Boosting for Classifying Facial Expressions: How good can a “Null” Model be?

Alexander Grubb and J. Andrew Bagnell. “Stacked Training for Overfitting Avoidance in Deep Networks

Sumit Chopra, Suhrid Balakrishnan, and Raghuraman Gopalan. “DLID: Deep Learning for Domain Adaptation by Interpolating between Domains

Kaggle contest papers

These papers are about methods that performed well in the Kaggle contests that highlight challenges in representation learning. All of these accepted papers correspond to methods that performed extremely well in the contests–either getting perfect accuracy in the multimodal learning challenge, roughly human-level performance in the facial expression recognition challenge, or in the top 3% of entrants to the black box learning challenge.

Yichuan Tang. “Deep Learning using Linear Support Vector Machines

Lukasz Romaszko. “A Deep Learning Approach with an Ensemble-Based Neural Network Classifier for Black Box ICML 2013 Contest

Fangxiang Feng, Ruifan Li, Xiaojie Wang. “Constructing Hierarchical Image-tags Bimodal Representations for Word Tags Alternative Choice

Jingjing Xie, Bing Xu, Zhang Chuang. “Horizontal and Vertical Ensemble with Deep Representation for Classification

Radu Tudor Ionescu, Marius Popescu, Cristian Grozea. “Local Learning to Improve Bag of Visual Words Model for Facial Expression Recognition

Dong-Hyun Lee. “Pseudo-Label : The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks