The Symplectic Transformer

The symplectic transformer is, like the standard transformer, a combination of attention layers and feedforward layers. The difference is that the attention layers are not multihead attention layers and the feedforward layers are not standard GeometricMachineLearning.ResNetLayers, but symplectic attention layers and sympnet layers.

Library Functions

GeometricMachineLearning.SymplecticTransformer — Type

SymplecticTransformer <: TransformerIntegrator

Constructors

SymplecticTransformer(sys_dim)

Make an instance of SymplecticTransformer for a specific system dimension and sequence length. Also see LinearSymplecticTransformer.

Arguments

You can provide the additional optional keyword arguments:

n_sympnet::Int = 2: The number of sympnet layers in the transformer.
upscaling_dimension::Int = 2*dim: The upscaling that is done by the gradient layer.
L::Int = 1: The number of transformer units.
sympnet_activation = tanh: The activation function for the SympNet layers.
attention_activation = GeometricMachineLearning.MatrixSoftmax(): The activation function for the Attention layers.
init_upper::Bool = true: Specifies if the first layer is a $q$-type layer (init_upper=true) or if it is a $p$-type layer (init_upper=false).
symmetric::Bool = true: Specifies whether the weighting matrix $A$ of the SymplecticAttention layers is symmetric (symmetric=true) or arbitrary (symmetric=false).

Each transformer unit has n_sympnet gradient layers after its $q$ attention layer and n_sympnet after its $p$ attention layer, so one unit has 2n_sympnet SympNet layers. In each group of n_sympnet layers the type alternates between GradientLayerQ and GradientLayerP, starting with the type that init_upper selects. For n_sympnet = 1 and init_upper = true each group is one GradientLayerQ.

source