Table Of Contents

Previous topic

Profiling Theano function

Next topic

Frequently Asked Questions

This Page

Extending Theano

Theano Graphs

  • Theano works with symbolic graphs.
  • Those graphs are bi-partite graphs (graphs with 2 types of nodes).
  • The two types of nodes are Apply and Variable nodes.
  • Each Apply node has a link to the op that it executes.

Inputs and Outputs are lists of Theano variables.

../_images/apply_node.png

Note

This tutorial does not cover how to make an op that returns a view or modifies the values in its inputs. Thus, all ops created with the instructions described here MUST return newly allocated memory or reuse the memory provided in the parameter output_storage of the perform() function. See Views and inplace operations for an explanation on how to do this.

If your op returns a view or changes the value of its inputs without doing as prescribed in that page, Theano will run, but will return correct results for some graphs and wrong results for others.

It is recommended that you run your tests in DebugMode (Theano flag mode=DebugMode) since it verifies if your op behaves correctly in this regard.

Note

See the Developer Start Guide for information regarding the versioning framework, namely about git and GitHub, regarding the development workflow and how to make a quality contribution.

Op Structure

This an overview of the methods you typically have to implement to make a new op. It does not provide extensive coverage of all the possibilities you may encounter or need. For that refer to Op’s contract.

import theano

class MyOp(theano.Op):
    __props__ = ()

    def make_node(self, *inputs):
        pass

    # Python implementation:
    def perform(self, node, inputs_storage, output_storage):
        pass

    # C implementation: [see theano web site for other functions]
    def c_code(...):
        # ...
        pass

    # Other implementations (pycuda, ...):
    def make_thunk(self, node, storage_map, _, _2):
        pass

    # optional:
    check_input = True

    def __init__(self, ...):
        pass

    def grad(self, inputs, g):
        pass

    def R_op(self, inputs, eval_points):
        pass

    def infer_shape(node, (i0_shapes, ...)):
        pass

There are two mandatory methods that one needs to implement. The first one is make_node(). The second one would describe the computations that are required to be done at run time. Currently there are 2 different possibilites: implement the perform() and/or c_code methods (and other related c methods), or the make_thunk() method. perform allows to easily wrap an existing Python function into Theano. c_code and the related methods allow the op to generate C code that will be compiled and linked by Theano. On the other hand, make_thunk will be called only once during compilation and should generate a thunk: a standalone function that when called will do the wanted computations. This is useful if you want to generate code and compile it yourself. For example, this allows you to use PyCUDA to compile GPU code.

The __props__ attribute serves to make Op generate an appropriate __eq__() and __hash__() for your Op. It must be a tuple that lists the properties that influence how the computation is performed (Ususally these are those that you set in __init__()). If you don’t have any properties, then you should set this attribute to the emtpy tuple (). It will also generate a suitable __str__() for your op. This requires development version after September 1st, 2014 or version 0.7.

__eq__() and __hash__() will be used by the optimization phase to merge nodes that are doing a equivalent compuation (same inputs, same operation). It is especially important that two Ops that compare equal (have the same values for all the properties listed in __props__ and the same type) compute the same thing when presented with the same inputs.

Also note that this attribute will also generate a suitable __str__() method for your Op. You may override this default with a custom one if you want another format for the output.

The infer_shape() method allows to infer the shape of some variable, somewhere in the middle of the computational graph without actually computing the outputs (when possible). This could be helpful if one only needs the shape of the output instead of the actual outputs.

The grad() method is required if you want to differentiate some cost whose expression includes your op.

The __str__() method is useful in order to provide a more meaningful string representation of your op.

The R_op() method is needed if you want theano.tensor.Rop to work with your op.

The optional boolean check_input attribute is used to specify if you want the types used in your op to check their inputs in their c_code. It can be used to speed up compilation, reduce overhead (particularly for scalars) and reduce the number of generated C files.

Op Example

import theano

class DoubleOp(theano.Op):
    __props__ = ()

    def make_node(self, x):
        # check that the theano version has support for __props__
        assert hasattr(self, '_props')
        x = theano.tensor.as_tensor_variable(x)
        return theano.Apply(self, [x], [x.type()])

    def perform(self, node, inputs, output_storage):
        x = inputs[0]
        z = output_storage[0]
        z[0] = x * 2

    def infer_shape(self, node, i0_shapes):
        return i0_shapes

    def grad(self, inputs, output_grads):
        return [output_grads[0] * 2]

    def R_op(self, inputs, eval_points):
        # R_op can receive None as eval_points.
        # That mean there is no diferientiable path through that input
        # If this imply that you cannot compute some outputs,
        # return None for those.
        if eval_points[0] is None:
            return eval_points
        return self.grad(inputs, eval_points)

You can try it as follows:

x = theano.tensor.matrix()
f = theano.function([x], DoubleOp()(x))
import numpy
inp = numpy.random.rand(5, 4)
out = f(inp)
assert numpy.allclose(inp * 2, out)
print inp
print out

How To Test it

Theano has some functionalities to simplify testing. These help test the infer_shape, grad and R_op methods. Put the following code in a file and execute it with the theano-nose program.

Basic Tests

Basic tests are done by you just by using the op and checking that it returns the right answer. If you detect an error, you must raise an exception. You can use the assert keyword to automatically raise an AssertionError.

from theano.tests import unittest_tools as utt
from theano import config
class test_Double(utt.InferShapeTester):
    def setUp(self):
        super(test_Double, self).setUp()
        self.op_class = DoubleOp
        self.op = DoubleOp()

    def test_basic(self):
        x = theano.tensor.matrix()
        f = theano.function([x], self.op(x))
        inp = numpy.asarray(numpy.random.rand(5, 4), dtype=config.floatX)
        out = f(inp)
        # Compare the result computed to the expected value.
        utt.assert_allclose(inp * 2, out)

We call utt.assert_allclose(expected_value, value) to compare NumPy ndarray.This raise an error message with more information. Also, the default tolerance can be changed with the Theano flags config.tensor.cmp_sloppy that take values in 0, 1 and 2. The defaul value do the most strict comparison, 1 and 2 make less strict comparison.

Testing the infer_shape

When a class inherits from the InferShapeTester class, it gets the self._compile_and_check method that tests the op’s infer_shape method. It tests that the op gets optimized out of the graph if only the shape of the output is needed and not the output itself. Additionally, it checks that the optimized graph computes the correct shape, by comparing it to the actual shape of the computed output.

self._compile_and_check compiles a Theano function. It takes as parameters the lists of input and output Theano variables, as would be provided to theano.function, and a list of real values to pass to the compiled function. It also takes the op class as a parameter in order to verify that no instance of it appears in the shape-optimized graph.

If there is an error, the function raises an exception. If you want to see it fail, you can implement an incorrect infer_shape.

When testing with input values with shapes that take the same value over different dimensions (for instance, a square matrix, or a tensor3 with shape (n, n, n), or (m, n, m)), it is not possible to detect if the output shape was computed correctly, or if some shapes with the same value have been mixed up. For instance, if the infer_shape uses the width of a matrix instead of its height, then testing with only square matrices will not detect the problem. This is why the self._compile_and_check method prints a warning in such a case. If your op works only with such matrices, you can disable the warning with the warn=False parameter.

from theano.tests import unittest_tools as utt
from theano import config
class test_Double(utt.InferShapeTester):
    # [...] as previous tests.
    def test_infer_shape(self):
        x = theano.tensor.matrix()
        self._compile_and_check([x],  # theano.function inputs
                                [self.op(x)],  # theano.function outputs
                                # Always use not square matrix!
                                # inputs data
                                [numpy.asarray(numpy.random.rand(5, 4),
                                               dtype=config.floatX)],
                                # Op that should be removed from the graph.
                                self.op_class)

Testing the gradient

The function verify_grad verifies the gradient of an op or Theano graph. It compares the analytic (symbolically computed) gradient and the numeric gradient (computed through the Finite Difference Method).

If there is an error, the function raises an exception. If you want to see it fail, you can implement an incorrect gradient (for instance, by removing the multiplication by 2).

def test_grad(self):
    theano.tests.unittest_tools.verify_grad(self.op,
                                            [numpy.random.rand(5, 7, 2)])

Testing the Rop

The class RopLop_checker defines the functions RopLop_checker.check_mat_rop_lop(), RopLop_checker.check_rop_lop() and RopLop_checker.check_nondiff_rop(). These allow to test the implementation of the Rop method of a particular op.

For instance, to verify the Rop method of the DoubleOp, you can use this:

import numpy
import theano.tests
from theano.tests.test_rop import RopLop_checker
class test_DoubleRop(RopLop_checker):
    def setUp(self):
        super(test_DoubleRop, self).setUp()
    def test_double_rop(self):
        self.check_rop_lop(DoubleRop()(self.x), self.in_shape)

Testing GPU Ops

Ops to be executed on the GPU should inherit from the theano.sandbox.cuda.GpuOp and not theano.Op. This allows Theano to distinguish them. Currently, we use this to test if the NVIDIA driver works correctly with our sum reduction code on the GPU.

Running Your Tests

To perform your tests, you may select either one of the three following methods:

theano-nose

The method of choice to conduct tests is to run the file theano-nose. In a regular Theano installation, the latter will be on the operating system’s path and directly accessible from any folder. Otherwise, it can be accessed in the Theano/bin folder. The following command lines may be used for the corresponding purposes:

  • theano-nose --theano: Run every test found in Theano’s path.
  • theano-nose folder_name: Run every test found in the folder folder_name.
  • theano-nose test_file.py: Run every test found in the file test_file.py.

The following are particularly useful for development purposes since they call for particular classes or even for particular tests:

  • theano-nose test_file.py:test_DoubleRop: Run every test found inside the class test_DoubleRop.
  • theano-nose test_file.py:test_DoubleRop.test_double_op: Run only the test test_double_op in the class test_DoubleRop.

Help with the use and functionalities of theano-nose may be obtained by running it with the command line parameter --help (-h).

nosetests

The command nosetests can also be used. Although it lacks the useful functionalities that theano-nose provides, nosetests can be called similarly to theano-nose from any folder in Python’s path like so:

nosetests [suffix similar to the above].

More documentation on nosetests is available here: nosetests.

In-file

One may also add a block of code similar to the following at the end of the file containing a specific test of interest and run the file. In this example, the test test_DoubleRop in the class test_double_op would be performed.

if __name__ == '__main__':
   t = test_DoubleRop("test_double_rop")
   t.setUp()
   t.test_double_rop()

We recommend that when we execute a file, we run all tests in that file. This can be done by adding this at the end of your test files:

if __name__ == '__main__':
    unittest.main()

Exercise

Run the code of the DoubleOp example above.

Modify and execute to compute: x * y.

Modify and execute the example to return two outputs: x + y and x - y.

You can omit the Rop functions. Try to implement the testing apparatus described above.

(Notice that Theano’s current elemwise fusion optimization is only applicable to computations involving a single output. Hence, to gain efficiency over the basic solution that is asked here, the two operations would have to be jointly optimized explicitly in the code.)

as_op

as_op is a python decorator that converts a python function into a basic Theano op that will call the supplied function during execution.

This isn’t the recommended way to build an op, but allows for a quick implementation.

It takes an optional infer_shape() parameter that must have this signature:

def infer_shape(node, input_shapes):
     # ...
     return output_shapes
  • input_shapes and output_shapes are lists of tuples that represent the shape of the corresponding inputs/outputs.

Note

Not providing the infer_shape method cause shapes-related optimization to not work with that op. For example your_op(inputs, ...).shape will need the op to be executed just to get the shape.

Note

As no grad is defined, this means you won’t be able to differentiate paths that include this op.

Note

It converts the python function to a callable object that takes as inputs Theano variables that were declared.

as_op Example

import theano
import numpy
from theano.compile.ops import as_op

def infer_shape_numpy_dot(node, input_shapes):
    ashp, bshp = input_shapes
    return [ashp[:-1] + bshp[-1:]]

@as_op(itypes=[theano.tensor.fmatrix, theano.tensor.fmatrix],
       otypes=[theano.tensor.fmatrix], infer_shape=infer_shape_numpy_dot)
def numpy_dot(a, b):
   return numpy.dot(a, b)

You can try it as follows:

x = theano.tensor.fmatrix()
y = theano.tensor.fmatrix()
f = function([x, y], numpy_dot(x, y))
inp1 = numpy.random.rand(5, 4)
inp2 = numpy.random.rand(4, 7)
out = f(inp1, inp2)

Exercise

Run the code of the numpy_dot example above.

Modify and execute to compute: numpy.add and numpy.subtract.

Modify and execute the example to return two outputs: x + y
and x - y.

Random numbers in tests

Making tests errors more reproducible is a good practice. To make your tests more reproducible, you need a way to get the same random numbers. You can do this by seeding NumPy’s random number generator.

For convenience, the classes InferShapeTester and RopLop_checker already do this for you. If you implement your own setUp function, don’t forget to call the parent setUp function.

For more details see Using Random Values in Test Cases.

Solution

Documentation

See Documentation Documentation AKA Meta-Documentation, for some information on how to generate the documentation.

Here is an example how to add docstring to a class.

import theano

class DoubleOp(theano.Op):
""" Double each element of a tensor.

:param x: input tensor.

:return: a tensor of the same shape and dtype as the input with all
    values doubled.

:note:
    this is a test note

:seealso:
    You can use the elemwise op to replace this example.
    Just execute `x * 2` with x being a Theano variable.

.. versionadded:: 0.6
"""

This is how it will show up for files that we auto-list in the library documentation:

class theano.misc.doubleop.DoubleOp(use_c_code='g++')

Double each element of a tensor.

Parameters:x – input tensor.
Returns:a tensor of the same shape and dtype as the input with all values doubled.
Note :this is a test note
Seealso :You can use the elemwise op to replace this example. Just execute x * 2 with x being a Theano variable.

New in version 0.6.

Final Note

A more extensive discussion of this section’s content may be found in the advanced tutorial Extending Theano.

The section Other ops includes more instructions for the following specific cases: