Guide
The scan functions provides the basic functionality needed to do loops
in Theano. Scan comes with many whistles and bells, that can be easily
introduced through a few examples :
Basic functionality : Computing 
Assume that, given k you want to get A**k using a loop.
More precisely, if A is a tensor you want to compute
A**k elemwise. The python/numpy code would loop like
result = 1
for i in xrange(k):
result = result * A
The equivalent Theano code would be
# Symbolic description of the result
result,updates = theano.scan(fn = lambda x_tm1,A: x_tm1*A,\
outputs_info = T.ones_like(A),\
non_sequences = A, \
n_steps = k)
# compiled function that returns A**k
f = theano.function([A,k], result[-1], updates = updates)
Let us go through the example line by line. What we did is first to
construct a function (using a lambda expression) that given x_tm1 and
A returns x_tm1*A. Given the order of the parameters, x_tm1
is the value of our output at time step t-1. Therefore
x_t (value of output at time t) is A times value of output
at t-1.
Next we initialize the output as a tensor with same
shape as A filled with ones. We give A to scan as a non sequence parameter and
specify the number of steps k to iterate over our lambda expression.
Scan will return a tuple, containing our result (result) and a
dictionary of updates ( empty in this case). Note that the result
is not a matrix, but a 3D tensor containing the value of A**k for
each step. We want the last value ( after k steps ) so we compile
a function to return just that. Note that there is an optimization, that
at compile time will detect that you are using just the last value of the
result and ensure that scan does not store all the intermediate values
that are used. So do not worry if A and k are large.
Multiple outputs, several taps values - Recurrent Neural Network with Scan
A more practical task would be to implement a RNN using scan. Assume
that our RNN is defined as follows :
Note that this network is far from a classical recurrent neural
network and might be useless. The reason we defined as such
is to better ilustrate the features of scan.
In this case we have a sequence over which we need to iterate u,
and two outputs x and y. To implement this with scan we first
construct a function that computes one iteration step :
def oneStep(u_tm4, u_t, x_tm3, x_tm1, y_tm1, W, W_in_1, W_in_2, W_feedback, W_out):
x_t = T.tanh( theano.dot(x_tm1, W) + \
theano.dot(u_t, W_in_1) + \
theano.dot(u_tm4, W_in_2) + \
theano.dot(y_tm1, W_feedback))
y_t = theano.dot(x_tm3, W_out)
return [x_t, y_t]
As naming convention for the variables we used a_tmb to mean a at
t-b and a_tpb to be a at t+b.
Note the order in which the parameters are given, and in which the
result is returned. Try to respect cronological order among
the taps ( time slices of sequences or outputs) used. For scan is crucial only
for the variables representing the different time taps to be in the same order
as the one in which these taps are given. Also, not only taps should respect
an order, but also variables, since this is how scan figures out what should
be represented by what. Given that we have all
the Theano variables needed we construct our RNN as follows :
u = T.matrix() # it is a sequence of vectors
x0 = T.matrix() # initial state of x has to be a matrix, since
# it has to cover x[-3]
y0 = T.vector() # y0 is just a vector since scan has only to provide
# y[-1]
([x_vals, y_vals],updates) = theano.scan(fn = oneStep, \
sequences = dict(input = u, taps= [-4,-0]), \
outputs_info = [dict(initial = x0, taps = [-3,-1]),y0], \
non_sequences = [W,W_in_1,W_in_2,W_feedback, W_out])
# for second input y, scan adds -1 in output_taps by default
Now x_vals and y_vals are symbolic variables pointing to the
sequence of x and y values generated by iterating over u. The
sequence_taps, outputs_taps give to scan information about what
slices are exactly needed. Note that if we want to use x[t-k] we do
not need to also have x[t-(k-1)], x[t-(k-2)],.., but when applying
the compiled function, the numpy array given to represent this sequence
should be large enough to cover this values. Assume that we compile the
above function, and we give as u the array uvals = [0,1,2,3,4,5,6,7,8].
By abusing notations, scan will consider uvals[0] as u[-4], and
will start scaning from uvals[4] towards the end.
Using shared variables - Gibbs sampling
Another useful feature of scan, is that it can handle shared variables.
For example, if we want to implement a Gibbs chain of length 10 we would do
the following:
W = theano.shared ( W_values ) # we assume that ``W_values`` contains the
# initial values of your weight matrix
bvis = theano.shared( bvis_values)
bhid = theano.shared( bhid_values)
trng = T.shared_randomstreams.RandomStreams(1234)
def OneStep( vsample) :
hmean = T.nnet.sigmoid( theano.dot( vsample, W) + bhid)
hsample = trng.binomial( size = hmean.shape, n = 1, prob = hmean)
vmean = T.nnet.sigmoid( theano.dot( hsample. W.T) + bvis)
return trng.binomial( size = vsample.shape, n = 1, prob = vsample)
sample = theano.tensor.vector()
values, updates = theano.scan( OneStep, outputs_info = sample, n_steps = 10 )
gibbs10 = theano.function([sample], values[-1], updates = updates)
Note that if we use shared variables ( W, bvis, bhid) but
we do not iterate over them ( so scan doesn’t really need to know
anything in particular about them, just that they are used inside the
function applied at each step) you do not need to pass them as
arguments. Scan will find them on its on and add them to the graph. Of
course, if you wish to (and it is good practice) you can add them, when
you call scan (they would be in the list of non sequence inputs).
The second, and probably most crucial observation is that the updates
dictionary becomes important in this case. It links a shared variable
with its updated value after k steps. In this case it tells how the
random streams get updated after 10 iterations. If you do not pass this
update dictionary to your function, you will always get the same 10
sets of random numbers. You can even use the updates dictionary
afterwards. Look at this example :
a = theano.shared(1)
values,updates = theano.scan( lambda : {a:a+1}, n_steps = 10 )
In this case the lambda expression does not require any input parameters
and returns an update dictionary which tells how a should be updated
after each step of scan. If we write :
b = a+1
c = updates[a] + 1
f = theano.function([], [b,c], updates = updates)
print b
print c
print a.value
We will see that because b does not use the updated version of
a, it will be 2, c will be 12, while a.value is 11.
If we call the function again, b will become 12, c will be 22
and a.value 21.
If we do not pass the updates dictionary to the function, then
a.value will always remain 1, b will always be 2 and c
will always be 12.
Reference
This module provides the Scan Op
Scanning is a general form of recurrence, which can be used for looping.
The idea is that you scan a function along some input sequence, producing
an output at each time-step that can be seen (but not modified) by the
function at the next time-step. (Technically, the function can see the
previous K time-steps of your outputs and L time steps (from the past and
future of the sequence) of your inputs.
So for example, sum() could be computed by scanning the z+x_i
function over a list, given an initial state of z=0.
Special cases:
- A reduce operation can be performed by returning only the last
output of a scan.
- A map operation can be performed by applying a function that
ignores each previous output.
Often a for-loop can be expressed as a scan() operation, and scan is
the closest that theano comes to looping. The advantage of using scan
over for loops is that it allows the number of iterations to be a part of the symbolic graph.
The Scan Op should typically be used by calling the scan() function.
-
theano.map(fn, sequences, non_sequences=[], truncate_gradient=-1, go_backwards=False, mode=None, name=None)
Similar behaviour as python map
| Parameters: |
- fn – the function to be applied over the elements in
sequences ( see scan fn for more info)
- sequences – list of arrays over which map should
iterate (see scan for more info)
- non_sequences – list of other arguments of fn over which
map shouldn’t iterate (see scan for more info)
- truncate_gradient – see scan for more info
- go_backwards – set to true if you want map to start at the end of the
provided arrays in sequences going towards 0 (back in time)
- mode – see scan
- name – see scan
|
|---|
-
theano.reduce(fn, sequences, outputs_info, non_sequences=[], go_backwards=False, mode=None, name=None)
Similar behaviour as python reduce
| Parameters: |
- fn – the function to be applied over the elements in
sequences ( see scan fn for more info)
- outputs_info – information about outputs (mainly the initial state
of each, but other options are available ), see scan for more
info
- sequences – list of arrays over which reduce should
iterate (see scan for more info)
- non_sequences – list of other arguments of fn over which
reduce shouldn’t iterate (see scan for more info)
- go_backwards – set to true if you want map to start at the end of the
provided arrays in sequences going towards 0 (back in time)
- mode – see scan
- name – see scan
|
|---|
-
theano.foldl(fn, sequences, outputs_info, non_sequences=[], mode=None, name=None)
Similar behaviour as haskell foldl
| Parameters: |
- fn – the function to be applied over the elements in
sequences ( see scan fn for more info)
- sequences – list of arrays over which foldl should
iterate (see scan for more info)
- outputs_info – information about outputs (mainly the initial state
of each )
- non_sequences – list of other arguments of fn over which
foldl shouldn’t iterate (see scan for more info)
- mode – see scan
- name – see scan
|
|---|
-
theano.foldr(fn, sequences, outputs_info, non_sequences=[], mode=None)
Similar behaviour as haskell foldr
| Parameters: |
- fn – the function to be applied over the elements in
sequences ( see scan fn for more info)
- sequences – list of arrays over which foldr should
iterate (see scan for more info)
- outputs_info – information about outputs (mainly the initial state
of each )
- non_sequences – list of other arguments of fn over which
foldr shouldn’t iterate (see scan for more info)
- truncate_gradient – see scan for more info
- mode – see scan
- name – see scan
|
|---|
-
theano.scan(fn, sequences=[], outputs_info=[], non_sequences=[], n_steps=None, truncate_gradient=-1, go_backwards=False, mode=None, name=None)
Function that constructs and applies a Scan op
| Parameters: |
- fn –
Function that describes the operations involved in one step of scan
Given variables representing all the slices of input and past values of
outputs and other non sequences parameters, fn should produce
variables describing the output of one time step of scan. The order in
which the argument to this function are given is very important. You
should have the following order:
- all time slices of the first sequence (as given in the
sequences list) ordered in the same fashion as the time taps provided
- all time slices of the second sequence (as given in the
sequences list) ordered in the same fashion as the time taps provided
- ...
- all time slices of the first output (as given in the
initial_state list) ordered in the same fashion as the time taps provided
- all time slices of the second otuput (as given in the
initial_state list) ordered in the same fashion as the time taps provided
- ...
- all other parameters over which scan doesn’t iterate ordered accordingly
If you are using shared variables over which you do not want to iterate,
you do not need to provide them as arguments to fn, though you can if you
wish so. The function should return the outputs after each step plus the updates
for any of the shared variables. You can either return only outputs or only
updates. If you have both outputs and updates the function should return
them as a tuple : (outputs, updates) or (updates, outputs).
Outputs can be just a theano expression if you have only one output or
a list of theano expressions. Updates can be given either as a list of tuples or
as a dictionary. If you have a list of outputs, the order of these
should match that of their initial_states.
- sequences –
list of Theano variables or dictionaries containing Theano variables over which
scan needs to iterate. The reason you might want to wrap a certain Theano
variable in a dictionary is to provide auxiliary information about how to iterate
over that variable. For example this is how you specify that you want to use
several time slices of this sequence at each iteration step. The dictionary
should have the following keys :
- input – Theano variable representing the sequence
- taps – temporal taps to use for this sequence. They are given as a list
of ints, where a value k means that at iteration step t scan needs to
provide also the slice t+k The order in which you provide these int values
here is the same order in which the slices will be provided to fn.
If you do not wrap a variable around a dictionary, scan will do it for you, under
the assumption that you use only one slice, defined as a tap of offset 0. This
means that at step t scan will provide the slice at position t.
- outputs_info –
list of Theano variables or dictionaries containing Theano variables used
to initialize the outputs of scan. As before (for sequences) the reason
you would wrap a Theano variable in a dictionary is to provide additional
information about how scan should deal with that specific output. The dictionary
should contain the following keys:
- initial – Theano variable containing the initial state of the output
- taps – temporal taps to use for this output. The taps are given as a
list of ints (only negative .. since you can not use future values of outputs),
with the same meaning as for sequences (see above).
- inplace – theano variable pointing to one of the input sequences; this
flag tells scan that the output should be computed in the memory space occupied
by that input sequence. Note that scan will only do this if allowed by the
rest of your computational graph and if you are not using past taps of the
input.
- return_steps how many steps to return from your output. If not given, or
0 scan will return all steps, otherwise it will return the last return_steps.
Note that if you set this to something else then 0, scan will try to be smart
about the amount of memory it allocates for a given input.
If the function applied recursively uses only the
previous value of the output, the initial state should have
same shape as one time step of the output; otherwise, the initial state
should have the same number of dimension as output. This is easily
understood through an example. For computing y[t] let us assume that we
need y[t-1], y[t-2] and y[t-4]. Through an abuse of
notation, when t = 0, we would need values for y[-1], y[-2]
and y[-4]. These values are provided by the initial state of y,
which should have same number of dimension as y, where the first
dimension should be large enough to cover all the required past values, which in
this case is 4. If init_y is the variable containing the initial state
of y, then init_y[0] corresponds to y[-4], init_y[1]
corresponds to y[-3], init_y[2] corresponds to y[-2],
init_y[3] corresponds to y[-1]. The default behaviour of scan is
the following :
- if you do not wrap an output in a dictionary, scan will wrap it for you
assuming that you use only the last step of the output ( i.e. it makes your tap
value list equal to [-1]) and that it is not computed inplace
- if you wrap an output in a dictionary and you do not provide any taps but
you provide an initial state it will assume that you are using only a tap value
of -1
- if you wrap an output in a dictionary but you do not provide any initial state,
it assumes that you are not using any form of taps
- if you provide a None instead of a variable or a dictionary scan assumes
that you will not use any taps for this output (this would be the case for map)
If you did not provide any information for your outputs, scan will assume by
default that you are not using any taps for any of the outputs. If you provide
information for just a subset of outputs, scan will not know to which outputs
these correspond and will raise an error.
- non_sequences – Parameters over which scan should not iterate. These parameters are
given at each time step to the function applied recursively.
- n_steps – Number of steps to iterate. If the input sequences are not long enough, scan
will produce a warning and run only for the maximal amount of steps allowed by
the input sequences. If the value is 0, the outputs will have 0 rows. If the
value is negative, scan will run backwards (or if the flag go_backwards is
already set to true it will run forward in time). If n_steps is not provided,
or evaluetes to None, inf or nan, scan will figure out the maximal amount of
steps it can run given the input sequences and do that.
- truncate_gradient – Number of steps to use in truncated BPTT. If you compute gradients
through a scan op, they are computed using backpropagation through time.
By providing a different value then -1, you choose to use truncated BPTT
instead of classical BPTT, where you only do truncate_gradient
number of steps.
- go_backwards – Flag indicating if you should go backwards through the sequences ( if you
think as the sequences being indexed by time, this would mean go backwards
in time)
- name – The name of the theano function compiled by the Scan op. It will show in the
profiler output.
- mode – The mode used when compiling the theano function in the Scan op.
If None, it will use the config mode. If None and the config mode is set to
profile mode, it we will create a new instance of the ProfileMode in order
to compute the timming correctly.
If no new instance is created the time spend in Scan will show up twice in the
profiling, once as the time taken by scan, and the second time as the time
taken by the ops inside scan. This will be even worse for multiple cascading
scans.
The new profiler instance will be printed when python exits.
|
|---|
| Return type: | tuple
|
|---|
| Returns: | tuple of the form (outputs, updates); outputs is either a
Theano variable or a list of Theano variables representing the
outputs of scan. updates is a dictionary specifying the
updates rules for all shared variables used in the scan
operation; this dictionary should be pass to theano.function
|
|---|
