Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SPDK vhost-blk w/ poll queues enabled fails #43

Open
enterJazz opened this issue Jan 7, 2024 · 1 comment
Open

SPDK vhost-blk w/ poll queues enabled fails #43

enterJazz opened this issue Jan 7, 2024 · 1 comment

Comments

@enterJazz
Copy link
Contributor

enterJazz commented Jan 7, 2024 •
edited
Loading

reproduce:

# start vhost backend
inv spdk.setup-vhost-target --huge-mem=8192 & # or start in separate tty
inv spdk.setup-vhost-blk-backend

# start qemu
inv run.run-debug-vhost-blk-poll-qemu --num-mem-gb=8 --num-poll-queues=2
# fails

# this works:
inv run.run-debug-vhost-blk-poll-qemu --num-mem-gb=8 --num-poll-queues=0

err message:

kernel log:

...
[    0.579497] virtio_blk virtio3: 2/0/2 default/read/poll queues
[    0.580864] virtio_blk virtio3: [vda] 390703446 4096-byte logical blocks (1.60 TB/1.46 TiB)
qemu-system-x86_64: Failed to read msg header. Read -1 instead of 12. Original request 40.
qemu-system-x86_64: Failed to write msg. Wrote -1 instead of 20.
qemu-system-x86_64: Failed to set msg fds.
qemu-system-x86_64: vhost VQ 3 ring restore failed: -22: Invalid argument (22)
qemu-system-x86_64: Failed to set msg fds.
qemu-system-x86_64: vhost VQ 2 ring restore failed: -22: Invalid argument (22)
qemu-system-x86_64: Failed to set msg fds.
qemu-system-x86_64: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
qemu-system-x86_64: Failed to set msg fds.
qemu-system-x86_64: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
qemu-system-x86_64: Failed to set msg fds.
qemu-system-x86_64: vhost_set_vring_call failed 22
qemu-system-x86_64: Failed to set msg fds.
qemu-system-x86_64: vhost_set_vring_call failed 22
qemu-system-x86_64: Failed to set msg fds.
qemu-system-x86_64: vhost_set_vring_call failed 22
qemu-system-x86_64: Failed to set msg fds.
qemu-system-x86_64: vhost_set_vring_call failed 22
qemu-system-x86_64: vhost-user-blk: vhost start failed: Error starting vhost: Input/output error

vhost log:

[2024-01-07 18:39:09.878689] Starting SPDK v23.09 / DPDK 23.07.0 initial
ization...                                                              
[2024-01-07 18:39:09.878801] [ DPDK EAL parameters: vhost --no-shconf -c
 0x3 --huge-unlink --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --l
og-level=user1:6 --iova-mode=pa --base-virtaddr=0x200000000000 --match-a
llocations --file-prefix=spdk_pid20848 ]                                
EAL: No free 2048 kB hugepages reported on node 1                       
TELEMETRY: No legacy callbacks, legacy socket not created               
[2024-01-07 18:39:10.034977] app.c: 786:spdk_app_start: *NOTICE*: Total 
cores available: 2                                                      
[2024-01-07 18:39:10.061302] reactor.c: 937:reactor_run: *NOTICE*: React
or started on core 1                                                    
[2024-01-07 18:39:10.061305] reactor.c: 937:reactor_run: *NOTICE*: React
or started on core 0                                                    
VHOST_CONFIG: (/var/tmp/vhost.1) vhost-user server: socket created, fd: 
234                                                                     
VHOST_CONFIG: (/var/tmp/vhost.1) binding succeeded                      
VHOST_CONFIG: (/var/tmp/vhost.1) new vhost user connection is 227       
VHOST_CONFIG: (/var/tmp/vhost.1) new device, handle is 0                
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_GET_FEATURES   
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_GET_PROTOCOL_FE
ATURES                                                                  
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_PROTOCOL_FE
ATURES                                                                  
VHOST_CONFIG: (/var/tmp/vhost.1) negotiated Vhost-user protocol features
: 0x11ebf                                                               
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_GET_QUEUE_NUM  
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_BACKEND_REQ
_FD                                                                     
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_OWNER      
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_GET_FEATURES   
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: (/var/tmp/vhost.1) vring call idx:0 file:239
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_VRING_ERR
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: (/var/tmp/vhost.1) vring call idx:1 file:238
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_VRING_ERR
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: (/var/tmp/vhost.1) vring call idx:2 file:240
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_VRING_ERR
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: (/var/tmp/vhost.1) vring call idx:3 file:241
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_VRING_ERR
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_GET_CONFIG
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_FEATURES
VHOST_CONFIG: (/var/tmp/vhost.1) negotiated Virtio features: 0x150007646
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_GET_STATUS
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_STATUS
VHOST_CONFIG: (/var/tmp/vhost.1) new device status(0x00000008):
VHOST_CONFIG: (/var/tmp/vhost.1)        -RESET: 0
VHOST_CONFIG: (/var/tmp/vhost.1)        -ACKNOWLEDGE: 0
VHOST_CONFIG: (/var/tmp/vhost.1)        -DRIVER: 0
VHOST_CONFIG: (/var/tmp/vhost.1)        -FEATURES_OK: 1
VHOST_CONFIG: (/var/tmp/vhost.1)        -DRIVER_OK: 0
VHOST_CONFIG: (/var/tmp/vhost.1)        -DEVICE_NEED_RESET: 0
VHOST_CONFIG: (/var/tmp/vhost.1)        -FAILED: 0
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_GET_INFLIGHT_FD
VHOST_CONFIG: (/var/tmp/vhost.1) get_inflight_fd num_queues: 4
VHOST_CONFIG: (/var/tmp/vhost.1) get_inflight_fd queue_size: 128
VHOST_CONFIG: (/var/tmp/vhost.1) send inflight mmap_size: 8448
VHOST_CONFIG: (/var/tmp/vhost.1) send inflight mmap_offset: 0
VHOST_CONFIG: (/var/tmp/vhost.1) send inflight fd: 242
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_INFLIGHT_FD
VHOST_CONFIG: (/var/tmp/vhost.1) set_inflight_fd mmap_size: 8448
VHOST_CONFIG: (/var/tmp/vhost.1) set_inflight_fd mmap_offset: 0
VHOST_CONFIG: (/var/tmp/vhost.1) set_inflight_fd num_queues: 4
VHOST_CONFIG: (/var/tmp/vhost.1) set_inflight_fd queue_size: 128
VHOST_CONFIG: (/var/tmp/vhost.1) set_inflight_fd fd: 243
VHOST_CONFIG: (/var/tmp/vhost.1) set_inflight_fd pervq_inflight_size: 21
12
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: (/var/tmp/vhost.1) vring call idx:0 file:242
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: (/var/tmp/vhost.1) vring call idx:1 file:239
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: (/var/tmp/vhost.1) vring call idx:2 file:238
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: (/var/tmp/vhost.1) vring call idx:3 file:240
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_FEATURES
VHOST_CONFIG: (/var/tmp/vhost.1) negotiated Virtio features: 0x150007646
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_GET_STATUS
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_MEM_TABLE
VHOST_CONFIG: (/var/tmp/vhost.1) guest memory region size: 0xc0000000
VHOST_CONFIG: (/var/tmp/vhost.1)         guest physical addr: 0x0
VHOST_CONFIG: (/var/tmp/vhost.1)         guest virtual  addr: 0x7f6917e0
0000
VHOST_CONFIG: (/var/tmp/vhost.1)         host  virtual  addr: 0x7fe44000
0000
VHOST_CONFIG: (/var/tmp/vhost.1)         mmap addr : 0x7fe440000000
VHOST_CONFIG: (/var/tmp/vhost.1)         mmap size : 0xc0000000
VHOST_CONFIG: (/var/tmp/vhost.1)         mmap align: 0x200000
VHOST_CONFIG: (/var/tmp/vhost.1)         mmap off  : 0x0
VHOST_CONFIG: (/var/tmp/vhost.1) guest memory region size: 0x140000000
VHOST_CONFIG: (/var/tmp/vhost.1)         guest physical addr: 0x10000000
0
VHOST_CONFIG: (/var/tmp/vhost.1)         guest virtual  addr: 0x7f69d7e0
0000
VHOST_CONFIG: (/var/tmp/vhost.1)         host  virtual  addr: 0x7fe30000
0000
VHOST_CONFIG: (/var/tmp/vhost.1)         mmap addr : 0x7fe240000000
VHOST_CONFIG: (/var/tmp/vhost.1)         mmap size : 0x200000000
VHOST_CONFIG: (/var/tmp/vhost.1)         mmap align: 0x200000
VHOST_CONFIG: (/var/tmp/vhost.1)         mmap off  : 0xc0000000
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: (/var/tmp/vhost.1) vring base idx:0 last_used_idx:0 last_a
vail_idx:0.
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_VRING_ADDR
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_VRING_KICK
VHOST_CONFIG: (/var/tmp/vhost.1) vring kick idx:0 file:245
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: (/var/tmp/vhost.1) vring base idx:1 last_used_idx:0 last_a
vail_idx:0.
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_VRING_ADDR
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_VRING_KICK
VHOST_CONFIG: (/var/tmp/vhost.1) vring kick idx:1 file:246
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: (/var/tmp/vhost.1) vring base idx:2 last_used_idx:0 last_a
vail_idx:0.
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_VRING_ADDR
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_VRING_KICK
VHOST_CONFIG: (/var/tmp/vhost.1) vring kick idx:2 file:247
VHOST_CONFIG: (/var/tmp/vhost.1) read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: (/var/tmp/vhost.1) vring call idx:2 file:-1
VHOST_CONFIG: (/var/tmp/vhost.1) vhost message handling failed.

looks like we found some nasty bug -> specifically an issue of guest-side completion polling
@enterJazz
Copy link
Contributor Author

Will ask in SPDK slack tomorrow and see if they have any ideas

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@enterJazz