LIP0003

LIP 3 - Parameter Assignments

LIP	3
Title	Parameter Assignments
Author	A. Pronobis
Status	Draft
Type	Standard
Discussion	Issue #26
PR
Created	Sep 1, 2017

Introduction

Currently, the interface of Weights does not allow for assigning a value to the internal variable without creating additional assign TF operations. This is problematic when transferring weight values between TensorFlow and numpy, e.g. in cases when weights values are loaded from a file or modified by an external algorithm.

This proposal offers changes to the interface that allow for cheap assignments without creating new ops.

Technical Background

Parameter values are stored using TensorFlow variables. TensorFlow variables come with an assign operation that is created when a variable is created to assign the initial value of the variable. That assign operation can be "re-used" by executing it in a session with a feed_dict feeding a different value of the tensor that provides the initial value. This way, it is possible to assign different values to a variable without creating additional assign. This works only if we are assigning a fixed value instead of a value that requires additional TF operations to compute. See this IPython Notebook for an example.

The value of a variable can be read at any time by running the variable in a session.

The Weights node permits specification of initial variable value in terms of a constant, a tensor, or an instance of a member class of ValueType, for more complex initializations (e.g. random).

The Weights node provide the following members for access to the variable:

• variable property - returns the variable itself, can be used to obtain variable value
• initialize method - returns the assign op assigning the initial value to the variable (includes parsing of ValueType, broadcasting and normalization)
• assign method - creates an assignment operation with parsing of ValueType, broadcasting and normalization

Currently these are only available as part of the specific Weights interface and not part of the general ParamNode interface.

Proposal

First, in order to introduce similar semantics as in TensorFlow, we propose to retain the assign method and variable property, but to convert the initialize method into initializer property. This way the interface of Weights will become similar to the interface of tf.Variable.

Second, in order to enable assignment of variable value without creating new assign operations, we propose to add two new properties:

• init_value - exposes the initial value of the variable (as set in the constructor) which is used when the op returned by the initializer property is run
• init_tensor - provides the tensor that can be fed to assign a different value when running the operation returned by the initializer property

Finally, we propose to make the variable, initializer, init_value, and init_tensor properties a part of the general interface of a ParamNode. We can safely assume that every parameter node should be associated with a variable. However, the way that variable is created as well as any pre-processing of assigned values might differ. Therefore, we propose that the specific node should handle the creation of new operations, but the general interface of a ParamNode should permit access to the underlying variable. This way, it is possible to access the ParamNode variable while being agnostic to the specific type of the parameter.

As a consequence of the last change, we should modify the functions initialize_weights and assign_weights to operate on any parameter node and rename them to initialize_params and assign_params.

Examples

First, we create a param node:

w = spn.Weights(init_value=1, num_weights=5)

We can initialize it in a session:

sess = tf.Session()
sess.run(w.initializer)

To assign a value to a variable without adding a new op:

sess.run(w.initializer, feed_dict={w.init_tensor: [10, 11, 12, 13, 14]})

Decision