Layer

Layer

Activation

class deepreg.model.layer.Activation(*args: Any, **kwargs: Any)

Layer wraps tf.keras.activations.get().

Parameters
  • identifier – e.g. “relu”

  • kwargs

AdditiveUpSampling

class deepreg.model.layer.AdditiveUpSampling(*args: Any, **kwargs: Any)

Layer up-samples 3d tensor and reduce channels using split and sum.

Parameters
  • output_shape – (out_dim1, out_dim2, out_dim3)

  • strides – int, 1-D Tensor or list

  • kwargs

call(inputs, **kwargs)
Parameters
  • inputs – shape = (batch, dim1, dim2, dim3, channels)

  • kwargs

Returns

shape = (batch, out_dim1, out_dim2, out_dim3, channels//stride]

Conv3d

class deepreg.model.layer.Conv3d(*args: Any, **kwargs: Any)

Layer wraps tf.keras.layers.Conv3D.

Parameters
  • filters – number of channels of the output

  • kernel_size – int or tuple of 3 ints, e.g. (3,3,3) or 3

  • strides – int or tuple of 3 ints, e.g. (1,1,1) or 1

  • padding – str, same or valid

  • activation – str, defines the activation function

  • use_bias – bool, whether add bias to output

  • kernel_initializer – str, defines the initialization method, defines the initialization method

Conv3dBlock

class deepreg.model.layer.Conv3dBlock(*args: Any, **kwargs: Any)

A conv3d block having conv3d - norm - activation.

Parameters
  • filters – number of channels of the output

  • kernel_size – int or tuple of 3 ints, e.g. (3,3,3) or 3

  • strides – int or tuple of 3 ints, e.g. (1,1,1) or 1

  • padding – str, same or valid

call(inputs, training=None, **kwargs)
Parameters
  • inputs – shape = (batch, in_dim1, in_dim2, in_dim3, channels)

  • training – training flag for normalization layers (default: None)

  • kwargs

Returns

shape = (batch, in_dim1, in_dim2, in_dim3, channels)

Conv3dWithResize

class deepreg.model.layer.Conv3dWithResize(*args: Any, **kwargs: Any)

A layer contains conv3d - resize3d.

Parameters
  • output_shape – tuple, (out_dim1, out_dim2, out_dim3)

  • filters – int, number of channels of the output

  • kernel_initializer – str, defines the initialization method

  • activation – str, defines the activation function

  • kwargs

call(inputs, **kwargs)
Parameters
  • inputs – shape = (batch, dim1, dim2, dim3, channels)

  • kwargs

Returns

shape = (batch, out_dim1, out_dim2, out_dim3, channels)

Deconv3d

class deepreg.model.layer.Deconv3d(*args: Any, **kwargs: Any)

Layer wraps tf.keras.layers.Conv3DTranspose and does not requires input shape when initializing.

Parameters
  • filters – number of channels of the output

  • output_shape – (out_dim1, out_dim2, out_dim3)

  • kernel_size – int or tuple of 3 ints, e.g. (3,3,3) or 3

  • strides – int or tuple of 3 ints, e.g. (1,1,1) or 1

  • padding – str, same or valid

  • kwargs

Deconv3dBlock

class deepreg.model.layer.Deconv3dBlock(*args: Any, **kwargs: Any)

A deconv3d block having deconv3d - norm - activation.

Parameters
  • filters – number of channels of the output

  • output_shape – (out_dim1, out_dim2, out_dim3)

  • kernel_size – int or tuple of 3 ints, e.g. (3,3,3) or 3

  • strides – int or tuple of 3 ints, e.g. (1,1,1) or 1

  • padding – str, same or valid

  • kwargs

call(inputs, training=None, **kwargs)
Parameters
  • inputs – shape = (batch, in_dim1, in_dim2, in_dim3, channels)

  • training – training flag for normalization layers (default: None)

  • kwargs

Return output

shape = (batch, in_dim1, in_dim2, in_dim3, channels)

Dense

class deepreg.model.layer.Dense(*args: Any, **kwargs: Any)

Layer wraps tf.keras.layers.Dense and flattens input if necessary.

Parameters
  • units – number of hidden units

  • bias_initializer – str, default “zeros”

  • kwargs

call(inputs, **kwargs)
Parameters
  • inputs – shape = (batch, *vol_dim, channels)

  • kwargs – (not used)

Returns

shape = (batch, units)

DownSampleResnetBlock

class deepreg.model.layer.DownSampleResnetBlock(*args: Any, **kwargs: Any)

A down-sampling resnet conv3d block, with max-pooling or conv3d.

  1. conved = conv3d_block(inputs) # adjust channel

  2. skip = residual_block(conved) # develop feature

  3. pooled = pool(skip) # down-sample

Parameters
  • filters – number of channels of the output

  • kernel_size – int or tuple of 3 ints, e.g. (3,3,3) or 3

  • padding – str, same or valid

call(inputs, training=None, **kwargs)
Parameters
  • inputs – shape = (batch, in_dim1, in_dim2, in_dim3, channels)

  • training – training flag for normalization layers (default: None)

  • kwargs

Returns

(pooled, skip)

  • downsampled, shape = (batch, in_dim1//2, in_dim2//2, in_dim3//2, channels)

  • skipped, shape = (batch, in_dim1, in_dim2, in_dim3, channels)

IntDVF

class deepreg.model.layer.IntDVF(*args: Any, **kwargs: Any)

Layer calculates DVF from DDF.

Reference:

Parameters
  • fixed_image_size – tuple, (f_dim1, f_dim2, f_dim3)

  • num_steps – int, number of steps for integration

  • kwargs

call(inputs, **kwargs)
Parameters
  • inputs – dvf, shape = (batch, f_dim1, f_dim2, f_dim3, 3), type = float32

  • kwargs

Returns

ddf, shape = (batch, f_dim1, f_dim2, f_dim3, 3)

LocalNetResidual3dBlock

class deepreg.model.layer.LocalNetResidual3dBlock(*args: Any, **kwargs: Any)

A resnet conv3d block, simpler than Residual3dBlock.

  1. conved = conv3d(inputs)

  2. out = act(norm(conved) + inputs)

Parameters
  • filters – number of channels of the output

  • kernel_size – int or tuple of 3 ints, e.g. (3,3,3) or 3

  • strides – int or tuple of 3 ints, e.g. (1,1,1) or 1

  • kwargs

LocalNetUpSampleResnetBlock

class deepreg.model.layer.LocalNetUpSampleResnetBlock(*args: Any, **kwargs: Any)

Layer up-samples tensor with two inputs (skipped and down-sampled).

Parameters
  • filters – int, number of output channels

  • use_additive_upsampling – bool to used additive upsampling (default is True)

  • kwargs

build(input_shape)
Parameters

input_shape – tuple (nonskip_tensor_shape, skip_tensor_shape)

call(inputs, training=None, **kwargs)
Parameters
  • inputs – list = [inputs_nonskip, inputs_skip]

  • training – training flag for normalization layers (default: None)

  • kwargs

Returns

MaxPool3d

class deepreg.model.layer.MaxPool3d(*args: Any, **kwargs: Any)

Layer wraps tf.keras.layers.MaxPool3D

Parameters
  • pool_size – int or tuple of 3 ints

  • strides – int or tuple of 3 ints or None, if None default will be pool_size

  • padding – str, same or valid

  • kwargs

Norm

class deepreg.model.layer.Norm(*args: Any, **kwargs: Any)

Class merges batch norm and layer norm.

Parameters
  • name – str, batch_norm or layer_norm

  • axis – int

  • kwargs

Residual3dBlock

class deepreg.model.layer.Residual3dBlock(*args: Any, **kwargs: Any)

A resnet conv3d block.

  1. conved = conv3d(conv3d_block(inputs))

  2. out = act(norm(conved) + inputs)

Parameters
  • filters – int, number of filters in the convolutional layers

  • kernel_size – int or tuple of 3 ints, e.g. (3,3,3) or 3

  • strides – int or tuple of 3 ints, e.g. (1,1,1) or 1

  • kwargs

call(inputs, training=None, **kwargs)
Parameters
  • inputs – shape = (batch, in_dim1, in_dim2, in_dim3, channels)

  • training – training flag for normalization layers (default: None)

  • kwargs

Return output

shape = (batch, in_dim1, in_dim2, in_dim3, channels)

UpSampleResnetBlock

class deepreg.model.layer.UpSampleResnetBlock(*args: Any, **kwargs: Any)

An up-sampling resnet conv3d block, with deconv3d.

Parameters
  • filters – number of channels of the output

  • kernel_size – int or tuple of 3 ints, e.g. (3,3,3) or 3

  • concat – bool,specify how to combine input and skip connection images. If True, use concatenation if false use sum (default=False).

  • kwargs

build(input_shape)
Parameters

input_shape – tuple, (downsampled_image_shape, skip_connection image_shape)

call(inputs, training=None, **kwargs)
Parameters
  • inputs

    tuple

    • down-sampled

    • skipped

  • training – training flag for normalization layers (default: None)

  • kwargs

Returns

shape = (batch, *skip_connection_image_shape, filters]

Warping

class deepreg.model.layer.Warping(*args: Any, **kwargs: Any)

A layer warps an image using DDF.

Reference:

Parameters
  • fixed_image_size – shape = (f_dim1, f_dim2, f_dim3) or (f_dim1, f_dim2, f_dim3, ch) with the last channel for features

  • kwargs

call(inputs, **kwargs)
Parameters
  • inputs

    (ddf, image)

    • ddf, shape = (batch, f_dim1, f_dim2, f_dim3, 3), dtype = float32

    • image, shape = (batch, m_dim1, m_dim2, m_dim3), dtype = float32

  • kwargs

Returns

shape = (batch, f_dim1, f_dim2, f_dim3)

Util

Module containing utilities for layer inputs

deepreg.model.layer_util.get_n_bits_combinations(num_bits: int) → list

Function returning list containing all combinations of n bits. Given num_bits binary bits, each bit has value 0 or 1, there are in total 2**n_bits combinations.

Parameters

num_bits – int, number of combinations to evaluate

Returns

a list of length 2**n_bits, return[i] is the binary representation of the decimal integer.

Example
>>> from deepreg.model.layer_util import get_n_bits_combinations
>>> get_n_bits_combinations(3)
[[0, 0, 0], # 0
 [0, 0, 1], # 1
 [0, 1, 0], # 2
 [0, 1, 1], # 3
 [1, 0, 0], # 4
 [1, 0, 1], # 5
 [1, 1, 0], # 6
 [1, 1, 1]] # 7
deepreg.model.layer_util.get_reference_grid(grid_size: (<class 'tuple'>, <class 'list'>)) → tensorflow.Tensor

Generate a 3D grid with given size.

Reference:

Note:

for tf.meshgrid, in the 3-D case with inputs of length M, N and P, outputs are of shape (N, M, P) for ‘xy’ indexing and (M, N, P) for ‘ij’ indexing.

Parameters

grid_size – list or tuple of size 3, [dim1, dim2, dim3]

Returns

shape = [dim1, dim2, dim3, 3], grid[i, j, k, :] = [i j k]

deepreg.model.layer_util.pyramid_combination(values: list, weights: list) → tensorflow.Tensor

Calculates linear interpolation (a weighted sum) using values of hypercube corners in dimension n.

For example, when num_dimension = len(loc_shape) = num_bits = 3 values correspond to values at corners of following coordinates

[[0, 0, 0], # even
 [0, 0, 1], # odd
 [0, 1, 0], # even
 [0, 1, 1], # odd
 [1, 0, 0], # even
 [1, 0, 1], # odd
 [1, 1, 0], # even
 [1, 1, 1]] # odd

values[::2] correspond to the corners with last coordinate == 0

[[0, 0, 0],
 [0, 1, 0],
 [1, 0, 0],
 [1, 1, 0]]

values[1::2] correspond to the corners with last coordinate == 1

[[0, 0, 1],
 [0, 1, 1],
 [1, 0, 1],
 [1, 1, 1]]

The weights correspond to the floor corners. For example, when num_dimension = len(loc_shape) = num_bits = 3, weights = [w1, w2, w3] (ignoring the batch dimension).

So for corner with coords (x, y, z), x, y, z’s values are 0 or 1

  • weight for x = w1 if x = 0 else 1-w1

  • weight for y = w2 if y = 0 else 1-w2

  • weight for z = w3 if z = 0 else 1-w3

so the weight for (x, y, z) is

W_xyz = ((1-x) * w1 + x * (1-w1)) * ((1-y) * w2 + y * (1-w2)) * ((1-z) * w3 + z * (1-w3))

= (W_xy * (1-z)) * w3 + (W_xy * z) * (1-w3)

where W_xy is the weight for (x, y), let

  • W_xy0 = W_xy * w3

  • W_xy1 = W_xy * (1-w3)

So, the final sum V equals

sum over x,y,z (V_xyz * W_xyz)

= sum over x,y ( V_xy0 * W_xy0 + V_xy1 * W_xy1 )

= sum over x,y ( V_xy0 * W_xy * w3 + V_xy1 * W_xy * (1-w3) )

= sum over x,y ( W_xy * (V_xy0 * w3 + V_xy1 * W_xy * (1-w3)) )

That’s why we call this pyramid combination. It calculates the linear interpolation gradually, starting from the last dimension. The key is that the weight of each corner is the product of the weights along each dimension.

Parameters
  • values – a list having values on the corner, it has 2**n tensors of shape (*loc_shape) or (batch, *loc_shape) or (batch, *loc_shape, ch) the order is consistent with get_n_bits_combinations loc_shape is independent from n, aka num_dim

  • weights – a list having weights of floor points, it has n tensors of shape (*loc_shape) or (batch, *loc_shape) or (batch, *loc_shape, 1)

Returns

one tensor of the same shape as an element in values (*loc_shape) or (batch, *loc_shape) or (batch, *loc_shape, 1)

deepreg.model.layer_util.random_transform_generator(batch_size: int, scale: float, seed: (<class 'int'>, None) = None) → tensorflow.Tensor

Function that generates a random 3D transformation parameters for a batch of data.

for 3D coordinates, affine transformation is

[[x' y' z' 1]] = [[x y z 1]] * [[* * * 0]
                                [* * * 0]
                                [* * * 0]
                                [* * * 1]]

where each * represents a degree of freedom, so there are in total 12 degrees of freedom the equation can be denoted as

new = old * T

where

  • new is the transformed coordinates, of shape (1, 4)

  • old is the original coordinates, of shape (1, 4)

  • T is the transformation matrix, of shape (4, 4)

the equation can be simplified to

[[x' y' z']] = [[x y z 1]] * [[* * *]
                              [* * *]
                              [* * *]
                              [* * *]]

so that

new = old * T

where

  • new is the transformed coordinates, of shape (1, 3)

  • old is the original coordinates, of shape (1, 4)

  • T is the transformation matrix, of shape (4, 3)

Given original and transformed coordinates, we can calculate the transformation matrix using

x = np.linalg.lstsq(a, b)

such that

a x = b

In our case,

  • a = old

  • b = new

  • x = T

To generate random transformation, we choose to add random perturbation to corner coordinates as follows: for corner of coordinates (x, y, z), the noise is

-(x, y, z) .* (r1, r2, r3)

where ri is a random number between (0, scale). So

(x’, y’, z’) = (x, y, z) .* (1-r1, 1-r2, 1-r3)

Thus, we can directly sample between 1-scale and 1 instead

We choose to calculate the transformation based on four corners in a cube centered at (0, 0, 0). A cube is shown as below, where

  • C = (-1, -1, -1)

  • G = (-1, -1, 1)

  • D = (-1, 1, -1)

  • A = (1, -1, -1)

            G — — — — — — — — H
          / |               / |
        /   |             /   |
      /     |           /     |
    /       |         /       |
  /         |       /         |
E — — — — — — — — F           |
|           |     |           |
|           |     |           |
|           C — — | — — — — — D
|         /       |         /
|       /         |       /
|     /           |     /
|   /             |   /
| /               | /
A — — — — — — — — B
Parameters
  • batch_size – int

  • scale – a float number between 0 and 1

  • seed – control the randomness

Returns

shape = (batch, 4, 3)

deepreg.model.layer_util.resample(vol, loc, interpolation='linear')

Sample the volume at given locations.

Input has

  • volume, vol, of shape = (batch, v_dim 1, …, v_dim n), or (batch, v_dim 1, …, v_dim n, ch), where n is the dimension of volume, ch is the extra dimension as features.

    Denote vol_shape = (v_dim 1, …, v_dim n)

  • location, loc, of shape = (batch, l_dim 1, …, l_dim m, n), where m is the dimension of output.

    Denote loc_shape = (l_dim 1, …, l_dim m)

Reference:

Parameters
  • vol – shape = (batch, *vol_shape) or (batch, *vol_shape, ch) with the last channel for features

  • loc – shape = (batch, *loc_shape, n) such that loc[b, l1, …, ln, :] = [v1, …, vn] is of shape (n,), which represents a point in vol, with coordinates (v1, …, vn)

  • interpolation – linear only, TODO support nearest

Returns

shape = (batch, l_dim 1, …, l_dim n)

deepreg.model.layer_util.resize3d(image: tensorflow.Tensor, size: (<class 'tuple'>, <class 'list'>), method: str = tensorflow.image.ResizeMethod.BILINEAR) → tensorflow.Tensor

Tensorflow does not have resize 3d, therefore the resize is performed two folds.

  • resize dim2 and dim3

  • resize dim1 and dim2

Parameters
  • image – tensor of shape = (batch, dim1, dim2, dim3, channels) or (batch, dim1, dim2, dim3) or (dim1, dim2, dim3)

  • size – tuple, (out_dim1, out_dim2, out_dim3)

  • method – str, one of tf.image.ResizeMethod

Returns

tensor of shape = (batch, out_dim1, out_dim2, out_dim3, channels) or (batch, dim1, dim2, dim3) or (dim1, dim2, dim3)

deepreg.model.layer_util.warp_grid(grid: tensorflow.Tensor, theta: tensorflow.Tensor) → tensorflow.Tensor

Perform transformation on the grid.

  • grid_padded[i,j,k,:] = [i j k 1]

  • grid_warped[b,i,j,k,p] = sum_over_q (grid_padded[i,j,k,q] * theta[b,q,p])

Parameters
  • grid – shape = (dim1, dim2, dim3, 3), grid[i,j,k,:] = [i j k]

  • theta – parameters of transformation, shape = (batch, 4, 3)

Returns

shape = (batch, dim1, dim2, dim3, 3)

deepreg.model.layer_util.warp_image_ddf(image: tensorflow.Tensor, ddf: tensorflow.Tensor, grid_ref: tensorflow.Tensor, None) → tensorflow.Tensor

Warp an image with given DDF.

Parameters
  • image – an image to be warped, shape = (batch, m_dim1, m_dim2, m_dim3) or (batch, m_dim1, m_dim2, m_dim3, ch)

  • ddf – shape = (batch, f_dim1, f_dim2, f_dim3, 3)

  • grid_ref – shape = (1, f_dim1, f_dim2, f_dim3, 3) or None, if None grid_reg will be calculated based on ddf

Returns

shape = (batch, f_dim1, f_dim2, f_dim3) or (batch, f_dim1, f_dim2, f_dim3, ch)