Is the background image necessary?

[Tangshitao]

I use the model to train my own dataset consisting of 2 classes(manhole cover and bg). When I use the original imagenet resnet50 model to initialize, it can't detect anything. However, When I use the rfcn model, it can detect what I want, but also it's very responsive to the objects the original rfcn model can detect(car, person, dog and etc.) What I plan to do is to use some images in pascal voc dataset as background to train the model. Do you have any suggestions?

Here are some parameters I use in training:
iter: 50000
lr: 0.00001(increasing it will result in nan loss)
number of images: 300-400