Prototype embeddings are recreated every forward pass instead of being optimized

[jorgelerre]

Description

Prototype embeddings when CCM is active are updated using:

cluster_emb = self.p2c(
    cluster_emb_,
    x_emb_,
    x_emb_,
    mask=mask.transpose(0,1)
)

They are then assigned as:

self.cluster_emb = nn.Parameter(cluster_emb, requires_grad=True)

This operation creates a new nn.Parameter object at every forward pass.

Consequence

As a result, the prototype embeddings:

• Do not appear to be optimized via standard gradient descent.
• Are instead overwritten at each iteration by the output of the attention module (which seems to have no weights, as described in this other issue, so this module produces a simple weighted sum of the time series embeddings).
• Become strongly dependent on the current batch rather than accumulated training signal.

Observed behavior

During experiments on ETTh1 with two clusters:

• Clustering metrics exhibit strong oscillations across epochs.
• Channel assignments frequently switch between epochs.
• Cluster-specific linear layers progressively receive information from all channels.

This suggests that cluster specialization is not being preserved.

Question

Is this behavior intentional, or should prototype embeddings be updated using a smoother method (like standard gradient-based optimization or a learning rate) instead of being re-instantiated at every forward pass?