Use Cases (Functional Requirements)

These use cases exhibit pseudo-code for some of the sorts of tasks listed in the requirements (requirements.txt)

Evaluate a classifier on MNIST

The evaluation of a classifier on MNIST requires iterating over examples in some set (e.g. validation, test) and comparing the model’s prediction with the correct answer. The score of the classifier is the number of correct predictions divided by the total number of predictions.

To perform this calculation, the user should specify: - the classifier (e.g. a function operating on weights loaded from disk) - the dataset (e.g. MNIST) - the subset of examples on which to evaluate (e.g. test set)

For example:

vm.call(classification_accuracy(
function = classifier, examples = MNIST.validation_iterator))

The user types very few things beyond the description of the fields necessary for the computation, no boilerplate. The MNIST.validation_iterator must respect a protocol that remains to be worked out.

The vm.call is a compilation & execution step, as opposed to the symbolic-graph building performed by the classification_accuracy call.

Train a linear classifier on MNIST

The training of a linear classifier requires specification of

  • problem dimensions (e.g. n. of inputs, n. of classes)
  • parameter initialization method
  • regularization
  • dataset
  • schedule for obtaining training examples (e.g. batch, online, minibatch, weighted examples)
  • algorithm for adapting parameters (e.g. SGD, Conj. Grad)
  • a stopping criterion (may be in terms of validation examples)

Often the dataset determines the problem dimensions.

Often the training examples and validation examples come from the same set (e.g. a large matrix of all examples) but this is not necessarily the case.

There are many ways that the training could be configured, but here is one:

vm.call(
  halflife_stopper(
      # OD: is n_hidden supposed to be n_classes instead?
      initial_model=random_linear_classifier(MNIST.n_inputs, MNIST.n_hidden, r_seed=234432),
      burnin=100,
      score_fn = vm_lambda(('learner_obj',),
          classification_accuracy(
              examples=MNIST.validation_dataset,
              function=as_classifier('learner_obj'))),

      step_fn = vm_lambda(('learner_obj',),
          sgd_step_fn(
              parameters = vm_getattr('learner_obj', 'params'),
              cost_and_updates=classif_nll('learner_obj',
                  example_stream=minibatches(
                      source=MNIST.training_dataset,
                      batchsize=100,
                      loop=True)),
              momentum=0.9,
              anneal_at_iter=50,
              n_iter=100)))  #step_fn goes through lots of examples (e.g. an epoch)

Although I expect this specific code might have to change quite a bit in a final version, I want to draw attention to a few aspects of it:

  • we build a symbolic expression graph that contains the whole program, not just the learning algorithm
  • the configuration language allows for callable objects (e.g. functions, curried functions) to be arguments
  • there is a lambda function-constructor (vm_lambda) we can use in this language
  • APIs and protocols are at work in establishing conventions for parameter-passing so that sub-expressions (e.g. datasets, optimization algorithms, etc.) can be swapped.
  • there are no APIs for things which are not passed as arguments (i.e. the logic of the whole program is not exposed via some uber-API).

OD comments: I didn’t have time to look closely at the details, but overall I like the general feel of it. At least I’d expect us to need something like that to be able to handle the multiple use cases we want to support. I must say I’m a bit worried though that it could become scary pretty fast to the newcomer, with ‘lambda functions’ and ‘virtual machines’. Anyway, one point I would like to comment on is the line that creates the linear classifier. I hope that, as much as possible, we can avoid the need to specify dataset dimensions / number of classes in algorithm constructors. I regularly had issues in PLearn with the fact we had for instance to give the number of inputs when creating a neural network. I much prefer when this kind of thing can be figured out at runtime:

  • Any parameter you can get rid of is a significant gain in user-friendliness.
  • It’s not always easy to know in advance e.g. the dimension of your input dataset. Imagine for instance this dataset is obtained in a first step by going through a PCA whose number of output dimensions is set so as to keep 90% of the variance.
  • It seems to me it fits better the idea of a symbolic graph: my intuition (that may be very different from what you actually have in mind) is to see an experiment as a symbolic graph, which you instantiate when you provide the input data. One advantage of this point of view is it makes it natural to re-use the same block components on various datasets / splits, something we often want to do.

K-fold cross validation of a classifier

splits = kfold_cross_validate(
    # OD: What would these parameters mean?
    indexlist = range(1000)
    train = 8,
    valid = 1,
    test = 1,
)

trained_models = [
    halflife_early_stopper(
        initial_model=alloc_model('param1', 'param2'),
        burnin=100,
        score_fn = vm_lambda(('learner_obj',),
            classification_error(
                function=as_classifier('learner_obj'),
                dataset=MNIST.subset(validation_set))),
        step_fn = vm_lambda(('learner_obj',),
                sgd_step_fn(
                    parameters = vm_getattr('learner_obj', 'params'),
                    cost_and_updates=classif_nll('learner_obj',
                        example_stream=minibatches(
                            source=MNIST.subset(train_set),
                            batchsize=100,
                            loop=True)),
                    n_iter=100)))
    for (train_set, validation_set, test_set) in splits]

vm.call(trained_models, param1=1, param2=2)
vm.call(trained_models, param1=3, param2=4)

I want to draw attention to the fact that the call method treats the expression tree as one big lambda expression, with potentially free variables that must be assigned - here the ‘param1’ and ‘param2’ arguments to alloc_model. There is no need to have separate compile and run steps like in Theano because these functions are expected to be long-running, and called once.

Analyze the results of the K-fold cross validation

It often happens that a user doesn’t know what statistics to compute before running a bunch of learning jobs, but only afterward. This can be done by extending the symbolic program, and calling the extended function.

vm.call(
[pylearn.min(pylearn_getattr(model, ‘weights’)) for model in trained_models], param1=1, param2=2)

If this is run after the previous calls:

vm.call(trained_models, param1=1, param2=2) vm.call(trained_models, param1=3, param2=4)

Then it should run very quickly, because the vm can cache the return values of the trained_models when param1=1 and param2=2.