How to implement CNN+LSTM using cudnn torch

[Kevinpsk]

Hi guys,

I want to implement a CNN+LSTM model using cudnn torch for a sequence classification problem, but it seems I can not find the correct way to connect CNN with LSTM so that I can train with sequences. So I would really appreciate it if you can help me. My sequential data is 2D with dimension something like this: 500 * 100 * 3 * 20 * 9 (SeqLen * NumSeq * NumPlane * Height * Width), and my targets are like this 500 * 100 * 1 (SeqLen * NumSeq * TargetDim). So in this particular case, I have 100 sequences, each with 500 time steps. My features are 2D images with 3 channels and my targets are just 1D labels.

I build my model like this,

features = features:view(50000, 3, 20, 9) -- I reshape the feature to be able to feed into CNN
model = nn.Sequential()
model:add(cudnn.SpatialConvolution(3, 10, 9, 9, 1, 1))
model:add(cudnn.ReLU())
model:add(cudnn.SpatialMaxPooling(1, 2, 1, 2)) -- after pooling output size is at each time step is 60
model:add(nn.View(500, 100, 60)) -- I reshape the outputs from CNN to SeqLen * NumSeq * FeatSize
model:add(cudnn.LSTM(20, 20)) -- first LSTM layer
model:add(cudnn.LSTM(20, 20)) -- second LSTM layer
model:add(nn.Sequencer(nn.Linear(20, 1)))
model:add(nn.Sequencer(nn.Sigmoid())) -- finally get a scalar output

After I build this model, the forward path works fine, but the backward path breaks with the following error,

cudnnConvolutionBackwardFilter failed: 9 convDesc=[mode : CUDNN_CROSS_CORRELATION datatype : CUDNN_DATA_FLOAT]

So what could be wrong with my code then? Or more generally, how to build CNN+LSTM model in using cudnn torch? Thanks a lot for your help,

Kevin