module – a theano object system

Note

Module addresses similar needs to shared. New code is encouraged to use shared variables.

Now that we’re familiar with the basics, we introduce Theano’s more advanced interface, Module. This interface allows you to define Theano “files” which can have variables and methods sharing those variables. The Module system simplifies the way to define complex systems such as a neural network. It also lets you load and save these complex systems using Python’s pickle mechanism.

Remake of the “state” example

Let’s use Module to re-implement the example using state.

>>> m = Module()
>>> m.state = T.dscalar()
>>> m.inc = T.dscalar('inc')
>>> m.new_state = m.state + m.inc
>>> m.add = Method(m.inc, m.new_state, {m.state: m.new_state})
>>> m.sub = Method(m.inc, None, {m.state: m.state - m.inc})
>>> acc = m.make(state = 0)
>>> acc.state, acc.inc
(array(0.0), None)
>>> acc.add(2)
array(2.0)
>>> acc.state, acc.inc
(array(2.0), None)
>>> acc.state = 39.99
>>> acc.add(0.01)
array(40.0)
>>> acc.state
array(40.0)
>>> acc.sub(20)
>>> acc.state
array(20.0)

This deserves to be broken up a bit...

>>> m = Module()

Here we instantiate an empty Module. If you can imagine that Theano is a way of generating code (expression graphs), then a Module() is like a fresh blank file.

>>> m.state = T.dscalar()
>>> m.inc = T.dscalar('inc')

Then we declare Variables for use in our Module. Since we assign these input Variables as attributes of the Module, they will be member Variables of the Module. Member Variables are special in a few ways, which we will see shortly.

Note

There is no need to name the Variable explicitly here. m.state will be given the name 'state' automatically, because it is being assigned to the attribute named 'state'.

Note

Since we made it a member of m, the acc object will have an attribute called inc. This attribute will keep its default value of None throughout the example.

>>> m.new_state = m.state + m.inc

This line creates a Variable corresponding to some symbolic computation. Although this line also assigns a Variable to a Module attribute, it does not become a member Variable like state and inc because it represents an expression result.

>>> m.add = Method(m.inc, m.new_state, {m.state: m.new_state})

Here we declare a Method. The three arguments are as follow:

• inputs: a list of input Variables
• outputs: a list of output Variables, or None. None is equivalent to returning an empty list of outputs.
• updates: a dictionary mapping member Variables to Variables. When we call the function that this Method compiles to, it will replace (update) the values associated with the member Variables.

>>> m.sub = Method(m.inc, None, {m.state: m.state - m.inc})

We declare another Method, that has no outputs.

>>> acc = m.make(state = 0)

This line is what does the magic (well, compilation). The m object contains symbolic things such as Variables and Methods. Calling make on m creates an object that can do real computation and whose attributes contain values such as numbers and numpy ndarrays.

At this point something special happens for our member Variables too. In the acc object, make allocates room to store numbers for m‘s member Variables. By using the string 'state' as a keyword argument, we tell Theano to store the number 0 for the member Variable called state. By not mentioning the inc variable, we associate None to the inc Variable.

>>> acc.state, acc.inc
(array(0.0), None)

Since state was declared as a member Variable of m, we can access it’s value in the acc object by the same attribute. Ditto for inc.

Note

Members can also be accessed using a dictionary-like notation. The syntax acc['state'] is equivalent to acc.state.

>>> acc.add(2)
array(2.0)
>>> acc.state, acc.inc
(array(2.0), None)

When we call the acc.add method, the value 2 is used for the symbolic m.inc. The first line evaluates the output and all the updates given in the updates argument of the call to Method that declared acc. We only had one update which mapped state to new_state and you can see that it works as intended, adding the argument to the internal state.

Note also that acc.inc is still None after our call. Since m.inc was listed as an input in the call to Method that created m.add, when acc.add was created by the call to m.make, a private storage container was allocated to hold the first parameter. If we had left m.inc out of the Method input list, then acc.add would have used acc.inc instead.

>>> acc.state = 39.99

The state can also be set. When we manually set the value of a member attribute like this, then subsequent calls to the methods of our module will use the new value.

>>> acc.add(0.01)
array(40.0)
>>> acc.state
array(40.0)
>>> acc.sub(20)
>>> acc.state
array(20.0)

Here, note that acc.add and acc.sub share access to the same state value but update it in different ways.

Using Inheritance

A friendlier way to use Module is to implement your functionality as a subclass of Module:

from theano.compile import Module, Method
import theano.tensor as T

class Accumulator(Module):

    def __init__(self):
        super(Accumulator, self).__init__() # don't forget this
        self.inc = T.dscalar()
        self.state = T.dscalar()
        self.new_state = self.inc + self.state
        self.add = Method(inputs = self.inc,
                          outputs = self.new_state,
                          updates = {self.state: self.new_state})
        self.sub = Method(inputs = self.inc,
                          outputs = None,
                          updates = {self.state: self.state - self.inc})

if __name__ == '__main__':
    m = Accumulator()
    acc = m.make(state = 0)

This is just like the previous example except slightly fancier.

Warning

Do not forget to call the constructor of the parent class! (That’s the call to super().__init__ in the previous code block.)

If you forget it, you’ll get strange behavior :(

Extending your Module with Python methods

Let’s say we want to add a method to our accumulator to print out the state and we want to call it print_state. There are two mechanisms to do this: let’s call them _instance_method and InstanceType.

Mechanism 1: _instance_method

This is the preferred way of adding a few instance methods with a minimum of boilerplate code.

All we need to do to use this mechanism is to give a method called _instance_print_state to our Module class.

Any method called like _instance_XXX will cause the object obtained through a call to make to have a method called XXX. Note that when we define _instance_print_state there are two “self” arguments: self which is symbolic and obj which is the compiled object (the one that contains values).

Hint: self.state is the symbolic state variable and prints out as “state”, whereas obj.state is the state’s actual value in the accumulator and prints out as “0.0”.

Mechanism 2: InstanceType

If a number of instance methods are going to be defined, and especially if you will want to inherit from the kind of class that gets instantiated by make, you might prefer to consider using the InstanceType mechanism.

Adding custom initialization

As was said in the previous section, you can add functionality with _instance_XXX methods. One of these methods is actually special: _instance_initialize will be called with whatever arguments you give to make. There is a default behavior which we have used, where we give the states’ initial values with keyword arguments (acc.make(state = 0)). If you want more personalized behavior, you can override the default with your own method, which has to be called _instance_initialize.

Nesting Modules

Probably the most powerful feature of Theano’s Modules is that one can be included as an attribute to another so that the storage of each is available to both.

As you read through examples of Theano code, you will probably see many instances of Modules being nested in this way.