Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cuDNN.torch not thread safe when input size changes #155

Open
jonathanasdf opened this issue Mar 29, 2016 · 0 comments · May be fixed by #160
Open

cuDNN.torch not thread safe when input size changes #155

jonathanasdf opened this issue Mar 29, 2016 · 0 comments · May be fixed by #160

Comments

@jonathanasdf
Copy link

I'm not very familiar with how cudnn.SpatialConvolution works, but it seems when an input of a different size comes in, output gets a new reference or something, and so copies of it on a different thread end up referring to an outdated copy of output or something?

Here's a snippet:

require 'cunn'
require 'cudnn'
local model = cudnn.SpatialConvolution(1,1,1,1):cuda()
local model2 = nn.SpatialConvolution(1,1,1,1):cuda()

local nThreads = 8
torch.setnumthreads(nThreads)
local Threads = require 'threads'
Threads.serialization('threads.sharedserialize')
local mutex_id = Threads.Mutex():id()
local threads = Threads(nThreads,
  function()
    require 'cunn'
    require 'cudnn'
  end,
  function()
    _model = model
    _model2 = model2
    _mutex = (require 'threads').Mutex(mutex_id)
  end
)

print("Start")
for t=1,100 do
  for i=2,10 do
    threads:addjob(
      function()
        _mutex:lock()
        local inputs = torch.rand(i, 1, 1, 1):cuda()
        local outputs = _model:forward(inputs)
        --local outputs = _model2:forward(inputs)
        if i ~= outputs:size(1) then
          print("mismatch!", inputs:size(1), outputs:size(1))
        end
        _mutex:unlock()
      end
    )
  end
  threads:synchronize()
end
print("Done")

When I run it, I see mismatch being printed. The cunn SpatialConvolution doesn't seem to have this problem (use model2 instead of model in that snippet).

jonathanasdf added a commit to jonathanasdf/cudnn.torch that referenced this issue Apr 5, 2016
Use the same instance of iSize to fix soumith#155
jonathanasdf added a commit to jonathanasdf/cudnn.torch that referenced this issue Apr 6, 2016
jonathanasdf added a commit to jonathanasdf/cudnn.torch that referenced this issue Apr 6, 2016
@jonathanasdf jonathanasdf changed the title SpatialConvolution doesn't work nicely with threads cuDNN.torch not thread safe when input size changes Apr 8, 2016
jonathanasdf added a commit to jonathanasdf/cudnn.torch that referenced this issue Sep 16, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant