Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crash when exporting a pool with mounted dataset #289

Closed
EchterAgo opened this issue Oct 13, 2023 · 5 comments
Closed

Crash when exporting a pool with mounted dataset #289

EchterAgo opened this issue Oct 13, 2023 · 5 comments
Assignees
@lundman

Comments

@EchterAgo
Copy link

EchterAgo commented Oct 13, 2023 •
edited
Loading

This is from the snapshot hang test in tests.py:

9: kd> k
 # Child-SP          RetAddr               Call Site
00 ffff9c84`8edfd880 fffff803`4b49e85a     OpenZFS!vnode_recycle_int+0x638 [H:\dev\openzfs\module\os\windows\spl\spl-vnode.c @ 1219] 
01 ffff9c84`8edfd930 fffff803`4b7a09a2     OpenZFS!vflush+0x2da [H:\dev\openzfs\module\os\windows\spl\spl-vnode.c @ 1649] 
02 ffff9c84`8edfd9c0 fffff803`4b7e8e2f     OpenZFS!zfs_vfs_unmount+0xc2 [H:\dev\openzfs\module\os\windows\zfs\zfs_vfsops.c @ 1608] 
03 ffff9c84`8edfdb40 fffff803`4b79c545     OpenZFS!zfs_windows_unmount+0x41f [H:\dev\openzfs\module\os\windows\zfs\zfs_vnops_windows_mount.c @ 1581] 
04 ffff9c84`8edfe430 fffff803`4b4a3906     OpenZFS!zfs_ioc_unmount+0x55 [H:\dev\openzfs\module\os\windows\zfs\zfs_ioctl_os.c @ 916] 
05 ffff9c84`8edfe470 fffff803`4b79c375     OpenZFS!zfsdev_ioctl_common+0x816 [H:\dev\openzfs\module\zfs\zfs_ioctl.c @ 7866] 
06 ffff9c84`8edfe550 fffff803`4b77f0ad     OpenZFS!zfsdev_ioctl+0x2c5 [H:\dev\openzfs\module\os\windows\zfs\zfs_ioctl_os.c @ 866] 
07 ffff9c84`8edfe640 fffff803`4b77e9b6     OpenZFS!ioctlDispatcher+0x32d [H:\dev\openzfs\module\os\windows\zfs\zfs_vnops_windows.c @ 6395] 
08 ffff9c84`8edfe710 fffff803`41210665     OpenZFS!dispatcher+0x1e6 [H:\dev\openzfs\module\os\windows\zfs\zfs_vnops_windows.c @ 7307] 
09 ffff9c84`8edfe800 fffff803`4160142c     nt!IofCallDriver+0x55
0a ffff9c84`8edfe840 fffff803`41601081     nt!IopSynchronousServiceTail+0x34c
0b ffff9c84`8edfe8e0 fffff803`416003f6     nt!IopXxxControlFile+0xc71
0c ffff9c84`8edfea20 fffff803`41410ef8     nt!NtDeviceIoControlFile+0x56
0d ffff9c84`8edfea90 00007ff9`5656d0c4     nt!KiSystemServiceCopyEnd+0x28
0e 000000c5`5c77cf08 00007ff9`53e9591b     ntdll!NtDeviceIoControlFile+0x14
0f 000000c5`5c77cf10 00007ff9`55895921     KERNELBASE!DeviceIoControl+0x6b
10 000000c5`5c77cf80 00007ff6`c14995d7     KERNEL32!DeviceIoControlImplementation+0x81
11 000000c5`5c77cfd0 00000000`00000000     zpool!zcmd_ioctl_compat+0xe7 [H:\dev\openzfs\lib\libzfs_core\os\windows\libzfs_core_ioctl.c @ 54]

the code for the test looks like this:

        preTest("snapshot hang")
        with zpool_create(ctx, "testsn02", paths_to_unc(bf[:1])) as pool:
            log_dl("after testsn02 pool create")

            allocate_file(pool.mount_path / "test01.file", 1 * Size.KIB)
            run_cmd(ctx.ZFS, ["snapshot", "testsn02@friday"])
            allocate_file(pool.mount_path / "test02.file", 1 * Size.KIB)
            run_cmd(ctx.ZFS, ["mount", "testsn02@friday"])
            run_cmd(ctx.ZPOOL, ["export", "-a"])
            pool.destroy = False  # already exported
@EchterAgo EchterAgo mentioned this issue Oct 13, 2023
Merged
@EchterAgo
Copy link
Author 

9: kd> dt flags
Local var @ 0xffff9c848edfd920 Type int
0n256
9: kd> dt node
Local var @ 0xffff9c848edfd8d0 Type vnode_fileobjects*
(null) 
9: kd> dt -b vp
Local var @ 0xffff9c848edfd918 Type vnode*
0xffff8907`4149e608 
   +0x000 FileHeader       : _FSRTL_ADVANCED_FCB_HEADER
      +0x000 NodeTypeCode     : 0n0
      +0x002 NodeByteSize     : 0n0
      +0x004 Flags            : 0x40 '@'
      +0x005 IsFastIoPossible : 0x1 ''
      +0x006 Flags2           : 0x2 ''
      +0x007 Reserved         : 0y0000
      +0x007 Version          : 0y0001
      +0x008 Resource         : 0xffff8907`4149e748 
      +0x010 PagingIoResource : 0xffff8907`4149e7b0 
      +0x018 AllocationSize   : _LARGE_INTEGER 0x0
         +0x000 LowPart          : 0
         +0x004 HighPart         : 0n0
         +0x000 u                : _LARGE_INTEGER::<unnamed-tag>
            +0x000 LowPart          : 0
            +0x004 HighPart         : 0n0
         +0x000 QuadPart         : 0n0
      +0x020 FileSize         : _LARGE_INTEGER 0x0
         +0x000 LowPart          : 0
         +0x004 HighPart         : 0n0
         +0x000 u                : _LARGE_INTEGER::<unnamed-tag>
            +0x000 LowPart          : 0
            +0x004 HighPart         : 0n0
         +0x000 QuadPart         : 0n0
      +0x028 ValidDataLength  : _LARGE_INTEGER 0x0
         +0x000 LowPart          : 0
         +0x004 HighPart         : 0n0
         +0x000 u                : _LARGE_INTEGER::<unnamed-tag>
            +0x000 LowPart          : 0
            +0x004 HighPart         : 0n0
         +0x000 QuadPart         : 0n0
      +0x030 FastMutex        : 0xffff8907`4149e660 
      +0x038 FilterContexts   : _LIST_ENTRY [ 0xffff8907`4149e640 - 0xffff8907`4149e640 ]
         +0x000 Flink            : 0xffff8907`4149e640 
         +0x008 Blink            : 0xffff8907`4149e640 
      +0x048 PushLock         : 0
      +0x050 FileContextSupportPointer : (null) 
   +0x058 AdvancedFcbHeaderMutex : _FAST_MUTEX
      +0x000 Count            : 0n1
      +0x008 Owner            : (null) 
      +0x010 Contention       : 0
      +0x018 Event            : _KEVENT
         +0x000 Header           : _DISPATCHER_HEADER
            +0x000 Lock             : 0n393217
            +0x000 LockNV           : 0n393217
            +0x000 Type             : 0x1 ''
            +0x001 Signalling       : 0 ''
            +0x002 Size             : 0x6 ''
            +0x003 Reserved1        : 0 ''
            +0x000 TimerType        : 0x1 ''
            +0x001 TimerControlFlags : 0 ''
            +0x001 Absolute         : 0y0
            +0x001 Wake             : 0y0
            +0x001 EncodedTolerableDelay : 0y000000 (0)
            +0x002 Hand             : 0x6 ''
            +0x003 TimerMiscFlags   : 0 ''
            +0x003 Index            : 0y000000 (0)
            +0x003 Inserted         : 0y0
            +0x003 Expired          : 0y0
            +0x000 Timer2Type       : 0x1 ''
            +0x001 Timer2Flags      : 0 ''
            +0x001 Timer2Inserted   : 0y0
            +0x001 Timer2Expiring   : 0y0
            +0x001 Timer2CancelPending : 0y0
            +0x001 Timer2SetPending : 0y0
            +0x001 Timer2Running    : 0y0
            +0x001 Timer2Disabled   : 0y0
            +0x001 Timer2ReservedFlags : 0y00
            +0x002 Timer2ComponentId : 0x6 ''
            +0x003 Timer2RelativeId : 0 ''
            +0x000 QueueType        : 0x1 ''
            +0x001 QueueControlFlags : 0 ''
            +0x001 Abandoned        : 0y0
            +0x001 DisableIncrement : 0y0
            +0x001 QueueReservedControlFlags : 0y000000 (0)
            +0x002 QueueSize        : 0x6 ''
            +0x003 QueueReserved    : 0 ''
            +0x000 ThreadType       : 0x1 ''
            +0x001 ThreadReserved   : 0 ''
            +0x002 ThreadControlFlags : 0x6 ''
            +0x002 CycleProfiling   : 0y0
            +0x002 CounterProfiling : 0y1
            +0x002 GroupScheduling  : 0y1
            +0x002 AffinitySet      : 0y0
            +0x002 Tagged           : 0y0
            +0x002 EnergyProfiling  : 0y0
            +0x002 SchedulerAssist  : 0y0
            +0x002 ThreadReservedControlFlags : 0y0
            +0x003 DebugActive      : 0 ''
            +0x003 ActiveDR7        : 0y0
            +0x003 Instrumented     : 0y0
            +0x003 Minimal          : 0y0
            +0x003 Reserved4        : 0y00
            +0x003 AltSyscall       : 0y0
            +0x003 Emulation        : 0y0
            +0x003 Reserved5        : 0y0
            +0x000 MutantType       : 0x1 ''
            +0x001 MutantSize       : 0 ''
            +0x002 DpcActive        : 0x6 ''
            +0x003 MutantReserved   : 0 ''
            +0x004 SignalState      : 0n0
            +0x008 WaitListHead     : _LIST_ENTRY [ 0xffff8907`4149e680 - 0xffff8907`4149e680 ]
               +0x000 Flink            : 0xffff8907`4149e680 
               +0x008 Blink            : 0xffff8907`4149e680 
      +0x030 OldIrql          : 0
   +0x090 SectionObjectPointers : _SECTION_OBJECT_POINTERS
      +0x000 DataSectionObject : (null) 
      +0x008 SharedCacheMap   : (null) 
      +0x010 ImageSectionObject : (null) 
   +0x0a8 oplock           : (null) 
   +0x0b0 v_mutex          : kmutex
      +0x000 m_lock           : mutex_t
         +0x000 opaque           : _KEVENT
            +0x000 Header           : _DISPATCHER_HEADER
               +0x000 Lock             : 0n393217
               +0x000 LockNV           : 0n393217
               +0x000 Type             : 0x1 ''
               +0x001 Signalling       : 0 ''
               +0x002 Size             : 0x6 ''
               +0x003 Reserved1        : 0 ''
               +0x000 TimerType        : 0x1 ''
               +0x001 TimerControlFlags : 0 ''
               +0x001 Absolute         : 0y0
               +0x001 Wake             : 0y0
               +0x001 EncodedTolerableDelay : 0y000000 (0)
               +0x002 Hand             : 0x6 ''
               +0x003 TimerMiscFlags   : 0 ''
               +0x003 Index            : 0y000000 (0)
               +0x003 Inserted         : 0y0
               +0x003 Expired          : 0y0
               +0x000 Timer2Type       : 0x1 ''
               +0x001 Timer2Flags      : 0 ''
               +0x001 Timer2Inserted   : 0y0
               +0x001 Timer2Expiring   : 0y0
               +0x001 Timer2CancelPending : 0y0
               +0x001 Timer2SetPending : 0y0
               +0x001 Timer2Running    : 0y0
               +0x001 Timer2Disabled   : 0y0
               +0x001 Timer2ReservedFlags : 0y00
               +0x002 Timer2ComponentId : 0x6 ''
               +0x003 Timer2RelativeId : 0 ''
               +0x000 QueueType        : 0x1 ''
               +0x001 QueueControlFlags : 0 ''
               +0x001 Abandoned        : 0y0
               +0x001 DisableIncrement : 0y0
               +0x001 QueueReservedControlFlags : 0y000000 (0)
               +0x002 QueueSize        : 0x6 ''
               +0x003 QueueReserved    : 0 ''
               +0x000 ThreadType       : 0x1 ''
               +0x001 ThreadReserved   : 0 ''
               +0x002 ThreadControlFlags : 0x6 ''
               +0x002 CycleProfiling   : 0y0
               +0x002 CounterProfiling : 0y1
               +0x002 GroupScheduling  : 0y1
               +0x002 AffinitySet      : 0y0
               +0x002 Tagged           : 0y0
               +0x002 EnergyProfiling  : 0y0
               +0x002 SchedulerAssist  : 0y0
               +0x002 ThreadReservedControlFlags : 0y0
               +0x003 DebugActive      : 0 ''
               +0x003 ActiveDR7        : 0y0
               +0x003 Instrumented     : 0y0
               +0x003 Minimal          : 0y0
               +0x003 Reserved4        : 0y00
               +0x003 AltSyscall       : 0y0
               +0x003 Emulation        : 0y0
               +0x003 Reserved5        : 0y0
               +0x000 MutantType       : 0x1 ''
               +0x001 MutantSize       : 0 ''
               +0x002 DpcActive        : 0x6 ''
               +0x003 MutantReserved   : 0 ''
               +0x004 SignalState      : 0n1
               +0x008 WaitListHead     : _LIST_ENTRY [ 0xffff8907`4149e6c0 - 0xffff8907`4149e6c0 ]
                  +0x000 Flink            : 0xffff8907`4149e6c0 
                  +0x008 Blink            : 0xffff8907`4149e6c0 
      +0x018 m_owner          : 0xffff8907`36e4a080 
      +0x020 m_destroy_lock   : 0
      +0x028 m_initialised    : 0x23456789
   +0x0e0 v_mount          : 0xffff93aa`bfd97ea0 
   +0x0e8 v_parent         : (null) 
   +0x0f0 v_data           : (null) 
   +0x0f8 v_reparse        : 0xffff8907`416ac298 
   +0x100 security_descriptor : (null) 
   +0x108 v_flags          : 0x43
   +0x10c v_iocount        : 2
   +0x110 v_usecount       : 0
   +0x114 v_type           : 2
   +0x118 v_unlink         : 0
   +0x11c v_unused         : 0
   +0x120 v_reparse_size   : 0x92
   +0x128 v_id             : 0x3c
   +0x130 v_easize         : 0
   +0x138 v_age            : 0n46109375000
   +0x140 resource         : _ERESOURCE
      +0x000 SystemResourcesList : _LIST_ENTRY [ 0xffff8907`4149e7b0 - 0xffff8907`40f7f7c0 ]
         +0x000 Flink            : 0xffff8907`4149e7b0 
         +0x008 Blink            : 0xffff8907`40f7f7c0 
      +0x010 OwnerTable       : (null) 
      +0x018 ActiveCount      : 0n0
      +0x01a Flag             : 0
      +0x01a ReservedLowFlags : 0 ''
      +0x01b WaiterPriority   : 0 ''
      +0x020 SharedWaiters    : (null) 
      +0x028 ExclusiveWaiters : (null) 
      +0x030 OwnerEntry       : _OWNER_ENTRY
         +0x000 OwnerThread      : 0
         +0x008 IoPriorityBoosted : 0y0
         +0x008 OwnerReferenced  : 0y0
         +0x008 IoQoSPriorityBoosted : 0y0
         +0x008 OwnerCount       : 0y00000000000000000000000000000 (0)
         +0x008 TableSize        : 0
      +0x040 ActiveEntries    : 0
      +0x044 ContentionCount  : 0
      +0x048 NumberOfSharedWaiters : 0
      +0x04c NumberOfExclusiveWaiters : 0
      +0x050 Reserved2        : (null) 
      +0x058 Address          : (null) 
      +0x058 CreatorBackTraceIndex : 0
      +0x060 SpinLock         : 0
   +0x1a8 pageio_resource  : _ERESOURCE
      +0x000 SystemResourcesList : _LIST_ENTRY [ 0xffff8907`414bbbe0 - 0xffff8907`4149e748 ]
         +0x000 Flink            : 0xffff8907`414bbbe0 
         +0x008 Blink            : 0xffff8907`4149e748 
      +0x010 OwnerTable       : (null) 
      +0x018 ActiveCount      : 0n0
      +0x01a Flag             : 0
      +0x01a ReservedLowFlags : 0 ''
      +0x01b WaiterPriority   : 0 ''
      +0x020 SharedWaiters    : (null) 
      +0x028 ExclusiveWaiters : (null) 
      +0x030 OwnerEntry       : _OWNER_ENTRY
         +0x000 OwnerThread      : 0
         +0x008 IoPriorityBoosted : 0y0
         +0x008 OwnerReferenced  : 0y0
         +0x008 IoQoSPriorityBoosted : 0y0
         +0x008 OwnerCount       : 0y00000000000000000000000000000 (0)
         +0x008 TableSize        : 0
      +0x040 ActiveEntries    : 0
      +0x044 ContentionCount  : 0
      +0x048 NumberOfSharedWaiters : 0
      +0x04c NumberOfExclusiveWaiters : 0
      +0x050 Reserved2        : (null) 
      +0x058 Address          : (null) 
      +0x058 CreatorBackTraceIndex : 0
      +0x060 SpinLock         : 0
   +0x210 lock             : _FILE_LOCK
      +0x000 CompleteLockIrpRoutine : (null) 
      +0x008 UnlockRoutine    : (null) 
      +0x010 FastIoIsQuestionable : 0 ''
      +0x011 SpareC           :  ""
       [00] 0 ''
       [01] 0 ''
       [02] 0 ''
      +0x018 LockInformation  : (null) 
      +0x020 LastReturnedLockInfo : _FILE_LOCK_INFO
         +0x000 StartingByte     : _LARGE_INTEGER 0x0
            +0x000 LowPart          : 0
            +0x004 HighPart         : 0n0
            +0x000 u                : _LARGE_INTEGER::<unnamed-tag>
               +0x000 LowPart          : 0
               +0x004 HighPart         : 0n0
            +0x000 QuadPart         : 0n0
         +0x008 Length           : _LARGE_INTEGER 0x0
            +0x000 LowPart          : 0
            +0x004 HighPart         : 0n0
            +0x000 u                : _LARGE_INTEGER::<unnamed-tag>
               +0x000 LowPart          : 0
               +0x004 HighPart         : 0n0
            +0x000 QuadPart         : 0n0
         +0x010 ExclusiveLock    : 0 ''
         +0x014 Key              : 0
         +0x018 FileObject       : (null) 
         +0x020 ProcessId        : (null) 
         +0x028 EndingByte       : _LARGE_INTEGER 0x0
            +0x000 LowPart          : 0
            +0x004 HighPart         : 0n0
            +0x000 u                : _LARGE_INTEGER::<unnamed-tag>
               +0x000 LowPart          : 0
               +0x004 HighPart         : 0n0
            +0x000 QuadPart         : 0n0
      +0x050 LastReturnedLock : (null) 
      +0x058 LockRequestsInProgress : 0n0
   +0x270 share_access     : _SHARE_ACCESS
      +0x000 OpenCount        : 0
      +0x004 Readers          : 0
      +0x008 Writers          : 0
      +0x00c Deleters         : 0
      +0x010 SharedRead       : 0
      +0x014 SharedWrite      : 0
      +0x018 SharedDelete     : 0
   +0x290 v_list           : list_node
      +0x000 list_next        : 0xffff8907`420c42a0 
      +0x008 list_prev        : 0xffff8907`4149eb58 
   +0x2a0 v_fileobjects    : avl_tree
      +0x000 avl_root         : (null) 
      +0x008 avl_compar       : 0xfffff803`4b49f8b0 
      +0x010 avl_offset       : 0
      +0x014 avl_numnodes     : 0
      +0x018 avl_size         : 0

@andrewc12 andrewc12 assigned andrewc12 and lundman and unassigned andrewc12 Oct 13, 2023
@derritter88
Copy link

I have a similar issue with external SSDs.
I can create a pool, can copy data on it but I cannot export the pool to unmount the drive itself.

@lundman
Copy link

lundman commented Oct 16, 2023

run_cmd(ctx.ZFS, ["mount", "testsn02@friday"])

Does it happen with mounted snapshots? What is supposed to happen is that zpool export goes and unmounts any snapshots, then datasets, then export. If something is still mounted, it will definitely hang. Although, a EISBUSY error would be better

@andrewc12
Copy link

Yeah it does. i had to comment out the test because it crashed.
Want me to move it into its own test?

andrewc12 added a commit that referenced this issue Oct 19, 2023
@andrewc12
Add more snapshot tests
15c62a7 
Test that crashes in #289
Signed-off-by: Andrew Innes <[email protected]>
Co-Authored-By: Axel Gembe <[email protected]>
@andrewc12
Copy link

Fixed in b508b29
Test added in #299

lundman pushed a commit that referenced this issue Dec 11, 2023
@andrewc12 @lundman
Add more snapshot tests
4cc1248 
Test that crashes in #289
Signed-off-by: Andrew Innes <[email protected]>
Co-Authored-By: Axel Gembe <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lundman lundman

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@lundman @EchterAgo @andrewc12 @derritter88