Package pylearn :: Package sandbox :: Package sparse_random_autoassociator :: Module main
[hide private]

Module main

source code

An autoassociator for sparse inputs, using Ronan Collobert + Jason Weston's sampling trick (2008).

The learned model is:

  h   = sigmoid(dot(x, w1) + b1)
  y   = sigmoid(dot(h, w2) + b2)

We assume that most of the inputs are zero, and hence that we can separate x into xnonzero, x's nonzero components, and xzero, a sample of the zeros. We sample---randomly without replacement---ZERO_SAMPLE_SIZE zero columns from x.

The desideratum is that every nonzero entry is separated from every zero entry by margin at least MARGIN. For each ynonzero, we want it to exceed max(yzero) by at least MARGIN. For each yzero, we want it to be exceed by min(ynonzero) by at least MARGIN. The loss is a hinge loss (linear). The loss is irrespective of the xnonzero magnitude (this may be a limitation). Hence, all nonzeroes are equally important to exceed the maximum yzero.

(Alternately, there is a commented out binary xent loss.)

LIMITATIONS:

Variables [hide private]
  nonzero_instances = []
  model = model.Model()

Imports: numpy