Minimal example with custom RNNs cells and sliding window support
❓ Questions and Help
Hello,
I would like to ask you about the support of pytorch-lighting regarding custom RNN cells and sliding window predictions of a sequence ( i.e. video)
I am implementing a kind of conv-LSTM variant for video prediction, where each frame is estimated from the previous frames in a sliding window manner.
Before asking:
I have searched the issues and the docs and I have found this:
# backprop every 5 steps in a batch
trainer = Trainer(truncated_bptt_steps=5)

.....

# Truncated back-propagation through time
def training_step(self, batch, batch_idx, hiddens):
    # hiddens are the hiddens from the previous truncated backprop step
    out, hiddens = self.lstm(data, hiddens)

    return {
        "loss": ...,
        "hiddens": hiddens  # remember to detach() this
    }
I have stored all the information I need (across timesteps) in the class (as members) so this is not an issue I think.
What is your question?
How can I provide my sequence as a sliding window and train the model for each step of the sequence (grads+backprop)?
Does the implemented supported function for truncated back-prop through time support this?
What have you tried?
I have tried to overwrite:
def tbptt_split_batch(self, batch, split_size):
....

but is seems it is not called on the Validation sanity check
What's your environment?

OS: Linux
Packaging: pip

It would be nice to create a minimal example on how the lib could support this kind of recurrent model predictions with sliding window support. If I can make this to work I can provide a minimal collab notebook.
Any ideas?
Thanks a lot,
N.A.