Module main
source code
An autoassociator for sparse inputs, using Ronan Collobert + Jason
Weston's sampling trick (2008).
The learned model is:
h = sigmoid(dot(x, w1) + b1)
y = sigmoid(dot(h, w2) + b2)
We assume that most of the inputs are zero, and hence that we can
separate x into xnonzero, x's nonzero components, and xzero, a sample of
the zeros. We samplerandomly without replacementZERO_SAMPLE_SIZE
zero columns from x.
The desideratum is that every nonzero entry is separated from every
zero entry by margin at least MARGIN. For each ynonzero, we want it to
exceed max(yzero) by at least MARGIN. For each yzero, we want it to be
exceed by min(ynonzero) by at least MARGIN. The loss is a hinge loss
(linear). The loss is irrespective of the xnonzero magnitude (this may be
a limitation). Hence, all nonzeroes are equally important to exceed the
maximum yzero.
(Alternately, there is a commented out binary xent loss.)
LIMITATIONS:

Only does pure stochastic gradient (batchsize = 1).

Loss is irrespective of the xnonzero magnitude.

We will always use all nonzero entries, even if the training instance
is very nonsparse.

nonzero_instances = []


model = model.Model()

Imports:
numpy