Layer¶
Layer¶
Activation¶
-
class
deepreg.model.layer.
Activation
(*args: Any, **kwargs: Any)¶ Layer wraps tf.keras.activations.get().
- Parameters
identifier – e.g. “relu”
kwargs –
AdditiveUpSampling¶
-
class
deepreg.model.layer.
AdditiveUpSampling
(*args: Any, **kwargs: Any)¶ Layer up-samples 3d tensor and reduce channels using split and sum.
- Parameters
output_shape – (out_dim1, out_dim2, out_dim3)
strides – int, 1-D Tensor or list
kwargs –
-
call
(inputs, **kwargs)¶ - Parameters
inputs – shape = (batch, dim1, dim2, dim3, channels)
kwargs –
- Returns
shape = (batch, out_dim1, out_dim2, out_dim3, channels//stride]
Conv3d¶
-
class
deepreg.model.layer.
Conv3d
(*args: Any, **kwargs: Any)¶ Layer wraps tf.keras.layers.Conv3D.
- Parameters
filters – number of channels of the output
kernel_size – int or tuple of 3 ints, e.g. (3,3,3) or 3
strides – int or tuple of 3 ints, e.g. (1,1,1) or 1
padding – str, same or valid
activation – str, defines the activation function
use_bias – bool, whether add bias to output
kernel_initializer – str, defines the initialization method, defines the initialization method
Conv3dBlock¶
-
class
deepreg.model.layer.
Conv3dBlock
(*args: Any, **kwargs: Any)¶ A conv3d block having conv3d - norm - activation.
- Parameters
filters – number of channels of the output
kernel_size – int or tuple of 3 ints, e.g. (3,3,3) or 3
strides – int or tuple of 3 ints, e.g. (1,1,1) or 1
padding – str, same or valid
-
call
(inputs, training=None, **kwargs)¶ - Parameters
inputs – shape = (batch, in_dim1, in_dim2, in_dim3, channels)
training – training flag for normalization layers (default: None)
kwargs –
- Returns
shape = (batch, in_dim1, in_dim2, in_dim3, channels)
Conv3dWithResize¶
-
class
deepreg.model.layer.
Conv3dWithResize
(*args: Any, **kwargs: Any)¶ A layer contains conv3d - resize3d.
- Parameters
output_shape – tuple, (out_dim1, out_dim2, out_dim3)
filters – int, number of channels of the output
kernel_initializer – str, defines the initialization method
activation – str, defines the activation function
kwargs –
-
call
(inputs, **kwargs)¶ - Parameters
inputs – shape = (batch, dim1, dim2, dim3, channels)
kwargs –
- Returns
shape = (batch, out_dim1, out_dim2, out_dim3, channels)
Deconv3d¶
-
class
deepreg.model.layer.
Deconv3d
(*args: Any, **kwargs: Any)¶ Layer wraps tf.keras.layers.Conv3DTranspose and does not requires input shape when initializing.
- Parameters
filters – number of channels of the output
output_shape – (out_dim1, out_dim2, out_dim3)
kernel_size – int or tuple of 3 ints, e.g. (3,3,3) or 3
strides – int or tuple of 3 ints, e.g. (1,1,1) or 1
padding – str, same or valid
kwargs –
Deconv3dBlock¶
-
class
deepreg.model.layer.
Deconv3dBlock
(*args: Any, **kwargs: Any)¶ A deconv3d block having deconv3d - norm - activation.
- Parameters
filters – number of channels of the output
output_shape – (out_dim1, out_dim2, out_dim3)
kernel_size – int or tuple of 3 ints, e.g. (3,3,3) or 3
strides – int or tuple of 3 ints, e.g. (1,1,1) or 1
padding – str, same or valid
kwargs –
-
call
(inputs, training=None, **kwargs)¶ - Parameters
inputs – shape = (batch, in_dim1, in_dim2, in_dim3, channels)
training – training flag for normalization layers (default: None)
kwargs –
- Return output
shape = (batch, in_dim1, in_dim2, in_dim3, channels)
Dense¶
-
class
deepreg.model.layer.
Dense
(*args: Any, **kwargs: Any)¶ Layer wraps tf.keras.layers.Dense and flattens input if necessary.
- Parameters
units – number of hidden units
bias_initializer – str, default “zeros”
kwargs –
-
call
(inputs, **kwargs)¶ - Parameters
inputs – shape = (batch, *vol_dim, channels)
kwargs – (not used)
- Returns
shape = (batch, units)
DownSampleResnetBlock¶
-
class
deepreg.model.layer.
DownSampleResnetBlock
(*args: Any, **kwargs: Any)¶ A down-sampling resnet conv3d block, with max-pooling or conv3d.
conved = conv3d_block(inputs) # adjust channel
skip = residual_block(conved) # develop feature
pooled = pool(skip) # down-sample
- Parameters
filters – number of channels of the output
kernel_size – int or tuple of 3 ints, e.g. (3,3,3) or 3
padding – str, same or valid
-
call
(inputs, training=None, **kwargs)¶ - Parameters
inputs – shape = (batch, in_dim1, in_dim2, in_dim3, channels)
training – training flag for normalization layers (default: None)
kwargs –
- Returns
(pooled, skip)
downsampled, shape = (batch, in_dim1//2, in_dim2//2, in_dim3//2, channels)
skipped, shape = (batch, in_dim1, in_dim2, in_dim3, channels)
IntDVF¶
-
class
deepreg.model.layer.
IntDVF
(*args: Any, **kwargs: Any)¶ Layer calculates DVF from DDF.
Reference:
integrate_vec of neuron https://github.com/adalca/neurite/blob/legacy/neuron/utils.py
- Parameters
fixed_image_size – tuple, (f_dim1, f_dim2, f_dim3)
num_steps – int, number of steps for integration
kwargs –
-
call
(inputs, **kwargs)¶ - Parameters
inputs – dvf, shape = (batch, f_dim1, f_dim2, f_dim3, 3), type = float32
kwargs –
- Returns
ddf, shape = (batch, f_dim1, f_dim2, f_dim3, 3)
LocalNetResidual3dBlock¶
-
class
deepreg.model.layer.
LocalNetResidual3dBlock
(*args: Any, **kwargs: Any)¶ A resnet conv3d block, simpler than Residual3dBlock.
conved = conv3d(inputs)
out = act(norm(conved) + inputs)
- Parameters
filters – number of channels of the output
kernel_size – int or tuple of 3 ints, e.g. (3,3,3) or 3
strides – int or tuple of 3 ints, e.g. (1,1,1) or 1
kwargs –
LocalNetUpSampleResnetBlock¶
-
class
deepreg.model.layer.
LocalNetUpSampleResnetBlock
(*args: Any, **kwargs: Any)¶ Layer up-samples tensor with two inputs (skipped and down-sampled).
- Parameters
filters – int, number of output channels
use_additive_upsampling – bool to used additive upsampling (default is True)
kwargs –
-
build
(input_shape)¶ - Parameters
input_shape – tuple (nonskip_tensor_shape, skip_tensor_shape)
-
call
(inputs, training=None, **kwargs)¶ - Parameters
inputs – list = [inputs_nonskip, inputs_skip]
training – training flag for normalization layers (default: None)
kwargs –
- Returns
MaxPool3d¶
-
class
deepreg.model.layer.
MaxPool3d
(*args: Any, **kwargs: Any)¶ Layer wraps tf.keras.layers.MaxPool3D
- Parameters
pool_size – int or tuple of 3 ints
strides – int or tuple of 3 ints or None, if None default will be pool_size
padding – str, same or valid
kwargs –
Norm¶
-
class
deepreg.model.layer.
Norm
(*args: Any, **kwargs: Any)¶ Class merges batch norm and layer norm.
- Parameters
name – str, batch_norm or layer_norm
axis – int
kwargs –
Residual3dBlock¶
-
class
deepreg.model.layer.
Residual3dBlock
(*args: Any, **kwargs: Any)¶ A resnet conv3d block.
conved = conv3d(conv3d_block(inputs))
out = act(norm(conved) + inputs)
- Parameters
filters – int, number of filters in the convolutional layers
kernel_size – int or tuple of 3 ints, e.g. (3,3,3) or 3
strides – int or tuple of 3 ints, e.g. (1,1,1) or 1
kwargs –
-
call
(inputs, training=None, **kwargs)¶ - Parameters
inputs – shape = (batch, in_dim1, in_dim2, in_dim3, channels)
training – training flag for normalization layers (default: None)
kwargs –
- Return output
shape = (batch, in_dim1, in_dim2, in_dim3, channels)
UpSampleResnetBlock¶
-
class
deepreg.model.layer.
UpSampleResnetBlock
(*args: Any, **kwargs: Any)¶ An up-sampling resnet conv3d block, with deconv3d.
- Parameters
filters – number of channels of the output
kernel_size – int or tuple of 3 ints, e.g. (3,3,3) or 3
concat – bool,specify how to combine input and skip connection images. If True, use concatenation if false use sum (default=False).
kwargs –
-
build
(input_shape)¶ - Parameters
input_shape – tuple, (downsampled_image_shape, skip_connection image_shape)
-
call
(inputs, training=None, **kwargs)¶ - Parameters
inputs –
tuple
down-sampled
skipped
training – training flag for normalization layers (default: None)
kwargs –
- Returns
shape = (batch, *skip_connection_image_shape, filters]
Warping¶
-
class
deepreg.model.layer.
Warping
(*args: Any, **kwargs: Any)¶ A layer warps an image using DDF.
Reference:
transform of neuron https://github.com/adalca/neurite/blob/legacy/neuron/utils.py
where vol = image, loc_shift = ddf
- Parameters
fixed_image_size – shape = (f_dim1, f_dim2, f_dim3) or (f_dim1, f_dim2, f_dim3, ch) with the last channel for features
kwargs –
-
call
(inputs, **kwargs)¶ - Parameters
inputs –
(ddf, image)
ddf, shape = (batch, f_dim1, f_dim2, f_dim3, 3), dtype = float32
image, shape = (batch, m_dim1, m_dim2, m_dim3), dtype = float32
kwargs –
- Returns
shape = (batch, f_dim1, f_dim2, f_dim3)
Util¶
Module containing utilities for layer inputs
-
deepreg.model.layer_util.
get_n_bits_combinations
(num_bits: int) → list¶ Function returning list containing all combinations of n bits. Given num_bits binary bits, each bit has value 0 or 1, there are in total 2**n_bits combinations.
- Parameters
num_bits – int, number of combinations to evaluate
- Returns
a list of length 2**n_bits, return[i] is the binary representation of the decimal integer.
- Example
>>> from deepreg.model.layer_util import get_n_bits_combinations >>> get_n_bits_combinations(3) [[0, 0, 0], # 0 [0, 0, 1], # 1 [0, 1, 0], # 2 [0, 1, 1], # 3 [1, 0, 0], # 4 [1, 0, 1], # 5 [1, 1, 0], # 6 [1, 1, 1]] # 7
-
deepreg.model.layer_util.
get_reference_grid
(grid_size: (<class 'tuple'>, <class 'list'>)) → tensorflow.Tensor¶ Generate a 3D grid with given size.
Reference:
volshape_to_meshgrid of neuron https://github.com/adalca/neurite/blob/legacy/neuron/utils.py
neuron modifies meshgrid to make it faster, however local benchmark suggests tf.meshgrid is better
Note:
for tf.meshgrid, in the 3-D case with inputs of length M, N and P, outputs are of shape (N, M, P) for ‘xy’ indexing and (M, N, P) for ‘ij’ indexing.
- Parameters
grid_size – list or tuple of size 3, [dim1, dim2, dim3]
- Returns
shape = [dim1, dim2, dim3, 3], grid[i, j, k, :] = [i j k]
-
deepreg.model.layer_util.
pyramid_combination
(values: list, weights: list) → tensorflow.Tensor¶ Calculates linear interpolation (a weighted sum) using values of hypercube corners in dimension n.
For example, when num_dimension = len(loc_shape) = num_bits = 3 values correspond to values at corners of following coordinates
[[0, 0, 0], # even [0, 0, 1], # odd [0, 1, 0], # even [0, 1, 1], # odd [1, 0, 0], # even [1, 0, 1], # odd [1, 1, 0], # even [1, 1, 1]] # odd
values[::2] correspond to the corners with last coordinate == 0
[[0, 0, 0], [0, 1, 0], [1, 0, 0], [1, 1, 0]]
values[1::2] correspond to the corners with last coordinate == 1
[[0, 0, 1], [0, 1, 1], [1, 0, 1], [1, 1, 1]]
The weights correspond to the floor corners. For example, when num_dimension = len(loc_shape) = num_bits = 3, weights = [w1, w2, w3] (ignoring the batch dimension).
So for corner with coords (x, y, z), x, y, z’s values are 0 or 1
weight for x = w1 if x = 0 else 1-w1
weight for y = w2 if y = 0 else 1-w2
weight for z = w3 if z = 0 else 1-w3
so the weight for (x, y, z) is
W_xyz = ((1-x) * w1 + x * (1-w1)) * ((1-y) * w2 + y * (1-w2)) * ((1-z) * w3 + z * (1-w3))
= (W_xy * (1-z)) * w3 + (W_xy * z) * (1-w3)
where W_xy is the weight for (x, y), let
W_xy0 = W_xy * w3
W_xy1 = W_xy * (1-w3)
So, the final sum V equals
sum over x,y,z (V_xyz * W_xyz)
= sum over x,y ( V_xy0 * W_xy0 + V_xy1 * W_xy1 )
= sum over x,y ( V_xy0 * W_xy * w3 + V_xy1 * W_xy * (1-w3) )
= sum over x,y ( W_xy * (V_xy0 * w3 + V_xy1 * W_xy * (1-w3)) )
That’s why we call this pyramid combination. It calculates the linear interpolation gradually, starting from the last dimension. The key is that the weight of each corner is the product of the weights along each dimension.
- Parameters
values – a list having values on the corner, it has 2**n tensors of shape (*loc_shape) or (batch, *loc_shape) or (batch, *loc_shape, ch) the order is consistent with get_n_bits_combinations loc_shape is independent from n, aka num_dim
weights – a list having weights of floor points, it has n tensors of shape (*loc_shape) or (batch, *loc_shape) or (batch, *loc_shape, 1)
- Returns
one tensor of the same shape as an element in values (*loc_shape) or (batch, *loc_shape) or (batch, *loc_shape, 1)
-
deepreg.model.layer_util.
random_transform_generator
(batch_size: int, scale: float, seed: (<class 'int'>, None) = None) → tensorflow.Tensor¶ Function that generates a random 3D transformation parameters for a batch of data.
for 3D coordinates, affine transformation is
[[x' y' z' 1]] = [[x y z 1]] * [[* * * 0] [* * * 0] [* * * 0] [* * * 1]]
where each * represents a degree of freedom, so there are in total 12 degrees of freedom the equation can be denoted as
new = old * T
where
new is the transformed coordinates, of shape (1, 4)
old is the original coordinates, of shape (1, 4)
T is the transformation matrix, of shape (4, 4)
the equation can be simplified to
[[x' y' z']] = [[x y z 1]] * [[* * *] [* * *] [* * *] [* * *]]
so that
new = old * T
where
new is the transformed coordinates, of shape (1, 3)
old is the original coordinates, of shape (1, 4)
T is the transformation matrix, of shape (4, 3)
Given original and transformed coordinates, we can calculate the transformation matrix using
x = np.linalg.lstsq(a, b)
such that
a x = b
In our case,
a = old
b = new
x = T
To generate random transformation, we choose to add random perturbation to corner coordinates as follows: for corner of coordinates (x, y, z), the noise is
-(x, y, z) .* (r1, r2, r3)
where ri is a random number between (0, scale). So
(x’, y’, z’) = (x, y, z) .* (1-r1, 1-r2, 1-r3)
Thus, we can directly sample between 1-scale and 1 instead
We choose to calculate the transformation based on four corners in a cube centered at (0, 0, 0). A cube is shown as below, where
C = (-1, -1, -1)
G = (-1, -1, 1)
D = (-1, 1, -1)
A = (1, -1, -1)
G — — — — — — — — H / | / | / | / | / | / | / | / | / | / | E — — — — — — — — F | | | | | | | | | | C — — | — — — — — D | / | / | / | / | / | / | / | / | / | / A — — — — — — — — B
- Parameters
batch_size – int
scale – a float number between 0 and 1
seed – control the randomness
- Returns
shape = (batch, 4, 3)
-
deepreg.model.layer_util.
resample
(vol, loc, interpolation='linear')¶ Sample the volume at given locations.
Input has
volume, vol, of shape = (batch, v_dim 1, …, v_dim n), or (batch, v_dim 1, …, v_dim n, ch), where n is the dimension of volume, ch is the extra dimension as features.
Denote vol_shape = (v_dim 1, …, v_dim n)
location, loc, of shape = (batch, l_dim 1, …, l_dim m, n), where m is the dimension of output.
Denote loc_shape = (l_dim 1, …, l_dim m)
Reference:
neuron’s interpn https://github.com/adalca/neurite/blob/legacy/neuron/utils.py
Difference
they dont have batch size
they support more dimensions in vol
TODO try not using stack as neuron claims it’s slower
- Parameters
vol – shape = (batch, *vol_shape) or (batch, *vol_shape, ch) with the last channel for features
loc – shape = (batch, *loc_shape, n) such that loc[b, l1, …, ln, :] = [v1, …, vn] is of shape (n,), which represents a point in vol, with coordinates (v1, …, vn)
interpolation – linear only, TODO support nearest
- Returns
shape = (batch, l_dim 1, …, l_dim n)
-
deepreg.model.layer_util.
resize3d
(image: tensorflow.Tensor, size: (<class 'tuple'>, <class 'list'>), method: str = tensorflow.image.ResizeMethod.BILINEAR) → tensorflow.Tensor¶ Tensorflow does not have resize 3d, therefore the resize is performed two folds.
resize dim2 and dim3
resize dim1 and dim2
- Parameters
image – tensor of shape = (batch, dim1, dim2, dim3, channels) or (batch, dim1, dim2, dim3) or (dim1, dim2, dim3)
size – tuple, (out_dim1, out_dim2, out_dim3)
method – str, one of tf.image.ResizeMethod
- Returns
tensor of shape = (batch, out_dim1, out_dim2, out_dim3, channels) or (batch, dim1, dim2, dim3) or (dim1, dim2, dim3)
-
deepreg.model.layer_util.
warp_grid
(grid: tensorflow.Tensor, theta: tensorflow.Tensor) → tensorflow.Tensor¶ Perform transformation on the grid.
grid_padded[i,j,k,:] = [i j k 1]
grid_warped[b,i,j,k,p] = sum_over_q (grid_padded[i,j,k,q] * theta[b,q,p])
- Parameters
grid – shape = (dim1, dim2, dim3, 3), grid[i,j,k,:] = [i j k]
theta – parameters of transformation, shape = (batch, 4, 3)
- Returns
shape = (batch, dim1, dim2, dim3, 3)
-
deepreg.model.layer_util.
warp_image_ddf
(image: tensorflow.Tensor, ddf: tensorflow.Tensor, grid_ref: tensorflow.Tensor, None) → tensorflow.Tensor¶ Warp an image with given DDF.
- Parameters
image – an image to be warped, shape = (batch, m_dim1, m_dim2, m_dim3) or (batch, m_dim1, m_dim2, m_dim3, ch)
ddf – shape = (batch, f_dim1, f_dim2, f_dim3, 3)
grid_ref – shape = (1, f_dim1, f_dim2, f_dim3, 3) or None, if None grid_reg will be calculated based on ddf
- Returns
shape = (batch, f_dim1, f_dim2, f_dim3) or (batch, f_dim1, f_dim2, f_dim3, ch)