Hang on exit with disconnected name server

[mdavidsaver]

For some time now I've occasional had phoebus hang on exit. This time I've had the presence of mind to get a thread dump. I think the relevant fact is that I have a org.phoebus.pv.ca/name_servers entry which is not connected, and CAJ is blocking on a connect() while holding a lock.

"JavaFX Application Thread" #21 prio=5 os_prio=0 cpu=6773.83ms elapsed=140.75s tid=0x00007fb8a416e0c0 nid=0x5c15e waiting for monitor entry  [0x00007fb892578000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at com.cosylab.epics.caj.CAJChannel.destroyChannel(CAJChannel.java:403)
        - waiting to lock <0x00000000e22e4330> (a com.cosylab.epics.caj.CAJChannel)
        at com.cosylab.epics.caj.CAJContext.destroyChannel(CAJContext.java:1108)
        at com.cosylab.epics.caj.CAJChannel.destroy(CAJChannel.java:383)
        at com.cosylab.epics.caj.CAJChannel.destroy(CAJChannel.java:368)
        at gov.aps.jca.Channel.dispose(Channel.java:710)
        at org.phoebus.pv.ca.JCA_PV.close(JCA_PV.java:634)
        at org.phoebus.pv.PVPool.releasePV(PVPool.java:236)
        at org.csstudio.display.builder.runtime.pv.RuntimePV.close(RuntimePV.java:168)
        at org.csstudio.display.builder.runtime.pv.PVFactory.releasePV(PVFactory.java:54)
        at org.csstudio.display.builder.runtime.PVNameToValueBinding.disconnect(PVNameToValueBinding.java:112)
        at org.csstudio.display.builder.runtime.PVNameToValueBinding.dispose(PVNameToValueBinding.java:120)
        at org.csstudio.display.builder.runtime.WidgetRuntime.stop(WidgetRuntime.java:474)
        at org.csstudio.display.builder.runtime.RuntimeUtil.stopRuntime(RuntimeUtil.java:168)
        at org.csstudio.display.builder.runtime.RuntimeUtil.stopChildRuntimes(RuntimeUtil.java:194)
        at org.csstudio.display.builder.runtime.internal.DisplayRuntime.stop(DisplayRuntime.java:64)
        at org.csstudio.display.builder.runtime.RuntimeUtil.stopRuntime(RuntimeUtil.java:168)
        at org.csstudio.display.builder.runtime.ActionUtil.handleClose(ActionUtil.java:197)
        at org.csstudio.display.builder.runtime.app.DisplayRuntimeInstance.disposeModel(DisplayRuntimeInstance.java:452)
        at org.csstudio.display.builder.runtime.app.DisplayRuntimeInstance.onClosed(DisplayRuntimeInstance.java:505)
        at org.csstudio.display.builder.runtime.app.DisplayRuntimeInstance$$Lambda$757/0x0000000800fa5078.run(Unknown Source)
        at org.phoebus.ui.docking.DockItem.handleClosed(DockItem.java:664)
        at org.phoebus.ui.docking.DockItemWithInput.handleClosed(DockItemWithInput.java:450)
        at org.phoebus.ui.docking.DockItem.close(DockItem.java:699)
        at org.phoebus.ui.docking.DockStage.closeItems(DockStage.java:294)
        at org.phoebus.ui.application.PhoebusApplication.lambda$closeMainStage$48(PhoebusApplication.java:1209)
        at org.phoebus.ui.application.PhoebusApplication$$Lambda$1869/0x000000080128df18.run(Unknown Source)
        at com.sun.javafx.application.PlatformImpl.lambda$runLater$10(PlatformImpl.java:457)
        at com.sun.javafx.application.PlatformImpl$$Lambda$99/0x0000000800cae240.run(Unknown Source)
        at java.security.AccessController.executePrivileged([email protected]/AccessController.java:776)
        at java.security.AccessController.doPrivileged([email protected]/AccessController.java:399)
        at com.sun.javafx.application.PlatformImpl.lambda$runLater$11(PlatformImpl.java:456)
        at com.sun.javafx.application.PlatformImpl$$Lambda$98/0x0000000800caafb0.run(Unknown Source)
        at com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:96)
        at com.sun.glass.ui.gtk.GtkApplication._runLoop(Native Method)
        at com.sun.glass.ui.gtk.GtkApplication.lambda$runLoop$11(GtkApplication.java:316)
        at com.sun.glass.ui.gtk.GtkApplication$$Lambda$87/0x0000000800c991f8.run(Unknown Source)
        at java.lang.Thread.run([email protected]/Thread.java:833)

...


"pool-4-thread-1" #56 prio=5 os_prio=0 cpu=9.67ms elapsed=138.35s tid=0x00007fb8752459e0 nid=0x5c19c runnable  [0x00007fb8621f4000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.Net.connect0([email protected]/Native Method)
        at sun.nio.ch.Net.connect([email protected]/Net.java:579)
        at sun.nio.ch.Net.connect([email protected]/Net.java:586)
        at sun.nio.ch.SocketChannelImpl.connect([email protected]/SocketChannelImpl.java:853)
        at java.nio.channels.SocketChannel.open([email protected]/SocketChannel.java:285)
        at com.cosylab.epics.caj.impl.CAConnector.tryConnect(CAConnector.java:179)
        at com.cosylab.epics.caj.impl.CAConnector.connect(CAConnector.java:94)
        at com.cosylab.epics.caj.CAJContext.getTransport(CAJContext.java:1662)
        at com.cosylab.epics.caj.CAJContext.searchResponse(CAJContext.java:1634)
        - locked <0x00000000e22e4330> (a com.cosylab.epics.caj.CAJChannel)
        at com.cosylab.epics.caj.impl.handlers.SearchResponse.internalHandleResponse(SearchResponse.java:108)
        at com.cosylab.epics.caj.impl.handlers.AbstractCAResponseHandler.handleResponse(AbstractCAResponseHandler.java:110)
        at com.cosylab.epics.caj.impl.CAResponseHandler.handleResponse(CAResponseHandler.java:139)
        at com.cosylab.epics.caj.impl.BroadcastTransport.processRead(BroadcastTransport.java:217)
        at com.cosylab.epics.caj.impl.BroadcastTransport.handleEvent(BroadcastTransport.java:166)
        at com.cosylab.epics.caj.impl.reactor.lf.LeaderFollowersHandler.handleEvent(LeaderFollowersHandler.java:77)
        at com.cosylab.epics.caj.impl.reactor.Reactor.processInternal(Reactor.java:404)
        at com.cosylab.epics.caj.impl.reactor.Reactor.process(Reactor.java:288)
        at com.cosylab.epics.caj.CAJContext$2.run(CAJContext.java:827)
        at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1136)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:635)
        at java.lang.Thread.run([email protected]/Thread.java:833)