Any way to make Lightning work with petastorm custom DataLoaders?
Is it possible to use petastorm (https://github.com/uber/petastorm) pytorch data loaders with pytorch lightning?
This issue is that petastorm's DataLoaders need to be re initiated for each epoch.
A sample code looks like this:
for epoch in range(1, loop_epochs + 1):
        train_loader = DataLoader(...)
        train(model, device, train_loader, args.log_interval, optimizer, epoch)
        test_loader = DataLoader(...)
        test(model, device, test_loader)
The dataloader keeps it's state, so refactoring the snippet as below breaks for epochs > 1:
train_loader = DataLoader(...)
test_loader = DataLoader(...)
for epoch in range(1, loop_epochs + 1):
        train(model, device, train_loader, args.log_interval, optimizer, epoch)
        test(model, device, test_loader)
Thanks for you help guys.