Using pin_memory=true in the WSL2 environment will cause an error
#1032
liuwake
started this conversation in
Show and tell
Replies: 2 comments
-
|
Full error log see: Click to expand/collapseTraining command is /opt/miniconda3/envs/mm/bin/python /home/bc/code/mmyolo/mmyolo/.mim/tools/train.py configs/ppyoloe/ppyoloe_plus_s_fast_8xb8-80e_coco.py --launcher none.
09/26 10:12:09 - mmengine - WARNING - Failed to search registry with scope "mmyolo" in the "log_processor" registry tree. As a workaround, the current "log_processor" registry in "mmengine" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmyolo" is a correct scope, or whether the registry is initialized.
09/26 10:12:09 - mmengine - INFO -
------------------------------------------------------------
System environment:
sys.platform: linux
Python: 3.10.14 (main, May 6 2024, 19:42:50) [GCC 11.2.0]
CUDA available: True
MUSA available: False
numpy_random_seed: 1010925727
GPU 0: NVIDIA GeForce RTX 4090
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 11.8, V11.8.89
GCC: gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
PyTorch: 2.0.1+cu118
PyTorch compiling details: PyTorch built with:
- GCC 9.3
- C++ Version: 201703
- Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v2.7.3 (Git Hash 6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- LAPACK is enabled (usually provided by MKL)
- NNPACK is enabled
- CPU capability usage: AVX2
- CUDA Runtime 11.8
- NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
- CuDNN 8.7
- Magma 2.6.1
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=8.7.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF,
TorchVision: 0.15.2+cu118
OpenCV: 4.10.0
MMEngine: 0.10.5
Runtime environment:
cudnn_benchmark: False
mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0}
dist_cfg: {'backend': 'nccl'}
seed: 1010925727
Distributed launcher: none
Distributed training: False
GPU number: 1
------------------------------------------------------------
09/26 10:12:10 - mmengine - INFO - Config:
_backend_args = None
_multiscale_resize_transforms = [
dict(
transforms=[
dict(scale=(
640,
640,
), type='YOLOv5KeepRatioResize'),
dict(
allow_scale_up=False,
pad_val=dict(img=114),
scale=(
640,
640,
),
type='LetterResize'),
],
type='Compose'),
dict(
transforms=[
dict(scale=(
320,
320,
), type='YOLOv5KeepRatioResize'),
dict(
allow_scale_up=False,
pad_val=dict(img=114),
scale=(
320,
320,
),
type='LetterResize'),
],
type='Compose'),
dict(
transforms=[
dict(scale=(
960,
960,
), type='YOLOv5KeepRatioResize'),
dict(
allow_scale_up=False,
pad_val=dict(img=114),
scale=(
960,
960,
),
type='LetterResize'),
],
type='Compose'),
]
backend_args = None
base_lr = 0.001
custom_hooks = [
dict(
ema_type='ExpMomentumEMA',
momentum=0.0002,
priority=49,
strict_load=False,
type='EMAHook',
update_buffers=True),
]
data_root = 'data/coco/'
dataset_type = 'YOLOv5CocoDataset'
deepen_factor = 0.33
default_hooks = dict(
checkpoint=dict(
interval=5, max_keep_ckpts=3, save_best='auto', type='CheckpointHook'),
logger=dict(interval=50, type='LoggerHook'),
param_scheduler=dict(
min_lr_ratio=0.0,
start_factor=0.0,
total_epochs=96,
type='PPYOLOEParamSchedulerHook',
warmup_epochs=5,
warmup_min_iter=1000),
sampler_seed=dict(type='DistSamplerSeedHook'),
timer=dict(type='IterTimerHook'),
visualization=dict(type='mmdet.DetVisualizationHook'))
default_scope = 'mmyolo'
env_cfg = dict(
cudnn_benchmark=False,
dist_cfg=dict(backend='nccl'),
mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0))
img_scale = (
640,
640,
)
img_scales = [
(
640,
640,
),
(
320,
320,
),
(
960,
960,
),
]
launcher = 'none'
load_from = 'https://download.openmmlab.com/mmyolo/v0/ppyoloe/ppyoloe_pretrain/ppyoloe_plus_s_obj365_pretrained-bcfe8478.pth'
log_level = 'INFO'
log_processor = dict(by_epoch=True, type='LogProcessor', window_size=50)
max_epochs = 80
model = dict(
backbone=dict(
act_cfg=dict(inplace=True, type='SiLU'),
attention_cfg=dict(
act_cfg=dict(type='HSigmoid'), type='EffectiveSELayer'),
block_cfg=dict(
shortcut=True, type='PPYOLOEBasicBlock', use_alpha=True),
deepen_factor=0.33,
norm_cfg=dict(eps=1e-05, momentum=0.1, type='BN'),
type='PPYOLOECSPResNet',
use_large_stem=True,
widen_factor=0.5),
bbox_head=dict(
bbox_coder=dict(type='DistancePointBBoxCoder'),
head_module=dict(
act_cfg=dict(inplace=True, type='SiLU'),
featmap_strides=[
8,
16,
32,
],
in_channels=[
192,
384,
768,
],
norm_cfg=dict(eps=1e-05, momentum=0.1, type='BN'),
num_base_priors=1,
num_classes=80,
reg_max=16,
type='PPYOLOEHeadModule',
widen_factor=0.5),
loss_bbox=dict(
bbox_format='xyxy',
iou_mode='giou',
loss_weight=2.5,
reduction='mean',
return_iou=False,
type='IoULoss'),
loss_cls=dict(
alpha=0.75,
gamma=2.0,
iou_weighted=True,
loss_weight=1.0,
reduction='sum',
type='mmdet.VarifocalLoss',
use_sigmoid=True),
loss_dfl=dict(
loss_weight=0.125,
reduction='mean',
type='mmdet.DistributionFocalLoss'),
prior_generator=dict(
offset=0.5, strides=[
8,
16,
32,
], type='mmdet.MlvlPointGenerator'),
type='PPYOLOEHead'),
data_preprocessor=dict(
batch_augments=[
dict(
interval=1,
keep_ratio=False,
random_interp=True,
random_size_range=(
320,
800,
),
size_divisor=32,
type='PPYOLOEBatchRandomResize'),
],
bgr_to_rgb=True,
mean=[
0.0,
0.0,
0.0,
],
pad_size_divisor=32,
std=[
255.0,
255.0,
255.0,
],
type='PPYOLOEDetDataPreprocessor'),
neck=dict(
act_cfg=dict(inplace=True, type='SiLU'),
block_cfg=dict(
shortcut=False, type='PPYOLOEBasicBlock', use_alpha=False),
deepen_factor=0.33,
drop_block_cfg=None,
in_channels=[
256,
512,
1024,
],
norm_cfg=dict(eps=1e-05, momentum=0.1, type='BN'),
num_blocks_per_layer=3,
num_csplayer=1,
out_channels=[
192,
384,
768,
],
type='PPYOLOECSPPAFPN',
use_spp=True,
widen_factor=0.5),
test_cfg=dict(
max_per_img=300,
multi_label=True,
nms=dict(iou_threshold=0.7, type='nms'),
nms_pre=1000,
score_thr=0.01),
train_cfg=dict(
assigner=dict(
alpha=1,
beta=6,
eps=1e-09,
num_classes=80,
topk=13,
type='BatchTaskAlignedAssigner'),
initial_assigner=dict(
iou_calculator=dict(type='mmdet.BboxOverlaps2D'),
num_classes=80,
topk=9,
type='BatchATSSAssigner'),
initial_epoch=30),
type='YOLODetector')
num_classes = 80
optim_wrapper = dict(
optimizer=dict(
lr=0.001,
momentum=0.9,
nesterov=False,
type='SGD',
weight_decay=0.0005),
paramwise_cfg=dict(norm_decay_mult=0.0),
type='OptimWrapper')
param_scheduler = None
persistent_workers = True
resume = False
save_epoch_intervals = 5
strides = [
8,
16,
32,
]
test_cfg = dict(type='TestLoop')
test_dataloader = dict(
batch_size=1,
dataset=dict(
ann_file='annotations/instances_val2017.json',
data_prefix=dict(img='val2017/'),
data_root='data/coco/',
filter_cfg=dict(filter_empty_gt=True, min_size=0),
pipeline=[
dict(backend_args=None, type='LoadImageFromFile'),
dict(
height=640,
interpolation='bicubic',
keep_ratio=False,
type='mmdet.FixShapeResize',
width=640),
dict(_scope_='mmdet', type='LoadAnnotations', with_bbox=True),
dict(
meta_keys=(
'img_id',
'img_path',
'ori_shape',
'img_shape',
'scale_factor',
),
type='mmdet.PackDetInputs'),
],
test_mode=True,
type='YOLOv5CocoDataset'),
drop_last=False,
num_workers=2,
persistent_workers=True,
pin_memory=True,
sampler=dict(shuffle=False, type='DefaultSampler'))
test_evaluator = dict(
ann_file='data/coco/annotations/instances_val2017.json',
metric='bbox',
proposal_nums=(
100,
1,
10,
),
type='mmdet.CocoMetric')
test_pipeline = [
dict(backend_args=None, type='LoadImageFromFile'),
dict(
height=640,
interpolation='bicubic',
keep_ratio=False,
type='mmdet.FixShapeResize',
width=640),
dict(_scope_='mmdet', type='LoadAnnotations', with_bbox=True),
dict(
meta_keys=(
'img_id',
'img_path',
'ori_shape',
'img_shape',
'scale_factor',
),
type='mmdet.PackDetInputs'),
]
train_batch_size_per_gpu = 8
train_cfg = dict(max_epochs=80, type='EpochBasedTrainLoop', val_interval=5)
train_dataloader = dict(
batch_size=8,
collate_fn=dict(type='yolov5_collate', use_ms_training=True),
dataset=dict(
ann_file='annotations/instances_train2017.json',
data_prefix=dict(img='train2017/'),
data_root='data/coco/',
filter_cfg=dict(filter_empty_gt=True, min_size=0),
pipeline=[
dict(backend_args=None, type='LoadImageFromFile'),
dict(type='LoadAnnotations', with_bbox=True),
dict(type='PPYOLOERandomDistort'),
dict(mean=(
103.53,
116.28,
123.675,
), type='mmdet.Expand'),
dict(type='PPYOLOERandomCrop'),
dict(prob=0.5, type='mmdet.RandomFlip'),
dict(
meta_keys=(
'img_id',
'img_path',
'ori_shape',
'img_shape',
'flip',
'flip_direction',
),
type='mmdet.PackDetInputs'),
],
type='YOLOv5CocoDataset'),
num_workers=8,
persistent_workers=True,
pin_memory=True,
sampler=dict(shuffle=True, type='DefaultSampler'))
train_num_workers = 8
train_pipeline = [
dict(backend_args=None, type='LoadImageFromFile'),
dict(type='LoadAnnotations', with_bbox=True),
dict(type='PPYOLOERandomDistort'),
dict(mean=(
103.53,
116.28,
123.675,
), type='mmdet.Expand'),
dict(type='PPYOLOERandomCrop'),
dict(prob=0.5, type='mmdet.RandomFlip'),
dict(
meta_keys=(
'img_id',
'img_path',
'ori_shape',
'img_shape',
'flip',
'flip_direction',
),
type='mmdet.PackDetInputs'),
]
tta_model = dict(
tta_cfg=dict(max_per_img=300, nms=dict(iou_threshold=0.65, type='nms')),
type='mmdet.DetTTAModel')
tta_pipeline = [
dict(backend_args=None, type='LoadImageFromFile'),
dict(
transforms=[
[
dict(
transforms=[
dict(scale=(
640,
640,
), type='YOLOv5KeepRatioResize'),
dict(
allow_scale_up=False,
pad_val=dict(img=114),
scale=(
640,
640,
),
type='LetterResize'),
],
type='Compose'),
dict(
transforms=[
dict(scale=(
320,
320,
), type='YOLOv5KeepRatioResize'),
dict(
allow_scale_up=False,
pad_val=dict(img=114),
scale=(
320,
320,
),
type='LetterResize'),
],
type='Compose'),
dict(
transforms=[
dict(scale=(
960,
960,
), type='YOLOv5KeepRatioResize'),
dict(
allow_scale_up=False,
pad_val=dict(img=114),
scale=(
960,
960,
),
type='LetterResize'),
],
type='Compose'),
],
[
dict(prob=1.0, type='mmdet.RandomFlip'),
dict(prob=0.0, type='mmdet.RandomFlip'),
],
[
dict(type='mmdet.LoadAnnotations', with_bbox=True),
],
[
dict(
meta_keys=(
'img_id',
'img_path',
'ori_shape',
'img_shape',
'scale_factor',
'pad_param',
'flip',
'flip_direction',
),
type='mmdet.PackDetInputs'),
],
],
type='TestTimeAug'),
]
val_batch_size_per_gpu = 1
val_cfg = dict(type='ValLoop')
val_dataloader = dict(
batch_size=1,
dataset=dict(
ann_file='annotations/instances_val2017.json',
data_prefix=dict(img='val2017/'),
data_root='data/coco/',
filter_cfg=dict(filter_empty_gt=True, min_size=0),
pipeline=[
dict(backend_args=None, type='LoadImageFromFile'),
dict(
height=640,
interpolation='bicubic',
keep_ratio=False,
type='mmdet.FixShapeResize',
width=640),
dict(_scope_='mmdet', type='LoadAnnotations', with_bbox=True),
dict(
meta_keys=(
'img_id',
'img_path',
'ori_shape',
'img_shape',
'scale_factor',
),
type='mmdet.PackDetInputs'),
],
test_mode=True,
type='YOLOv5CocoDataset'),
drop_last=False,
num_workers=2,
persistent_workers=True,
pin_memory=True,
sampler=dict(shuffle=False, type='DefaultSampler'))
val_evaluator = dict(
ann_file='data/coco/annotations/instances_val2017.json',
metric='bbox',
proposal_nums=(
100,
1,
10,
),
type='mmdet.CocoMetric')
val_num_workers = 2
vis_backends = [
dict(type='LocalVisBackend'),
]
visualizer = dict(
name='visualizer',
type='mmdet.DetLocalVisualizer',
vis_backends=[
dict(type='LocalVisBackend'),
])
widen_factor = 0.5
work_dir = './work_dirs/ppyoloe_plus_s_fast_8xb8-80e_coco'
/opt/miniconda3/envs/mm/lib/python3.10/site-packages/mmcv/cnn/bricks/hsigmoid.py:35: UserWarning: In MMCV v1.4.4, we modified the default value of args to align with PyTorch official. Previous Implementation: Hsigmoid(x) = min(max((x + 1) / 2, 0), 1). Current Implementation: Hsigmoid(x) = min(max((x + 3) / 6, 0), 1).
warnings.warn(
09/26 10:12:10 - mmengine - INFO - Distributed training is not used, all SyncBatchNorm (SyncBN) layers in the model will be automatically reverted to BatchNormXd layers if they are used.
09/26 10:12:10 - mmengine - INFO - Hooks will be executed in the following order:
before_run:
(VERY_HIGH ) RuntimeInfoHook
(49 ) EMAHook
(BELOW_NORMAL) LoggerHook
--------------------
after_load_checkpoint:
(49 ) EMAHook
--------------------
before_train:
(9 ) PPYOLOEParamSchedulerHook
(VERY_HIGH ) RuntimeInfoHook
(49 ) EMAHook
(NORMAL ) IterTimerHook
(VERY_LOW ) CheckpointHook
--------------------
before_train_epoch:
(VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(NORMAL ) DistSamplerSeedHook
--------------------
before_train_iter:
(9 ) PPYOLOEParamSchedulerHook
(VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
--------------------
after_train_iter:
(9 ) PPYOLOEParamSchedulerHook
(VERY_HIGH ) RuntimeInfoHook
(49 ) EMAHook
(NORMAL ) IterTimerHook
(BELOW_NORMAL) LoggerHook
(VERY_LOW ) CheckpointHook
--------------------
after_train_epoch:
(9 ) PPYOLOEParamSchedulerHook
(NORMAL ) IterTimerHook
(VERY_LOW ) CheckpointHook
--------------------
before_val:
(VERY_HIGH ) RuntimeInfoHook
--------------------
before_val_epoch:
(49 ) EMAHook
(NORMAL ) IterTimerHook
--------------------
before_val_iter:
(NORMAL ) IterTimerHook
--------------------
after_val_iter:
(NORMAL ) IterTimerHook
(NORMAL ) DetVisualizationHook
(BELOW_NORMAL) LoggerHook
--------------------
after_val_epoch:
(9 ) PPYOLOEParamSchedulerHook
(VERY_HIGH ) RuntimeInfoHook
(49 ) EMAHook
(NORMAL ) IterTimerHook
(BELOW_NORMAL) LoggerHook
(VERY_LOW ) CheckpointHook
--------------------
after_val:
(VERY_HIGH ) RuntimeInfoHook
--------------------
before_save_checkpoint:
(49 ) EMAHook
--------------------
after_train:
(VERY_HIGH ) RuntimeInfoHook
(VERY_LOW ) CheckpointHook
--------------------
before_test:
(VERY_HIGH ) RuntimeInfoHook
--------------------
before_test_epoch:
(49 ) EMAHook
(NORMAL ) IterTimerHook
--------------------
before_test_iter:
(NORMAL ) IterTimerHook
--------------------
after_test_iter:
(NORMAL ) IterTimerHook
(NORMAL ) DetVisualizationHook
(BELOW_NORMAL) LoggerHook
--------------------
after_test_epoch:
(VERY_HIGH ) RuntimeInfoHook
(49 ) EMAHook
(NORMAL ) IterTimerHook
(BELOW_NORMAL) LoggerHook
--------------------
after_test:
(VERY_HIGH ) RuntimeInfoHook
--------------------
after_run:
(BELOW_NORMAL) LoggerHook
--------------------
loading annotations into memory...
Done (t=6.26s)
creating index...
index created!
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stem.0.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stem.0.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stem.1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stem.1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stem.2.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stem.2.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage1.0.conv_down.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage1.0.conv_down.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage1.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage1.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage1.0.conv2.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage1.0.conv2.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage1.0.blocks.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage1.0.blocks.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage1.0.blocks.0.conv2.rbr_dense.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage1.0.blocks.0.conv2.rbr_dense.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage1.0.blocks.0.conv2.rbr_1x1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage1.0.blocks.0.conv2.rbr_1x1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage1.0.conv3.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage1.0.conv3.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.conv_down.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.conv_down.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.conv2.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.conv2.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.blocks.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.blocks.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.blocks.0.conv2.rbr_dense.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.blocks.0.conv2.rbr_dense.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.blocks.0.conv2.rbr_1x1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.blocks.0.conv2.rbr_1x1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.blocks.1.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.blocks.1.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.blocks.1.conv2.rbr_dense.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.blocks.1.conv2.rbr_dense.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.blocks.1.conv2.rbr_1x1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.blocks.1.conv2.rbr_1x1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.conv3.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.conv3.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.conv_down.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.conv_down.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.conv2.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.conv2.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.blocks.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.blocks.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.blocks.0.conv2.rbr_dense.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.blocks.0.conv2.rbr_dense.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.blocks.0.conv2.rbr_1x1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.blocks.0.conv2.rbr_1x1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.blocks.1.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.blocks.1.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.blocks.1.conv2.rbr_dense.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.blocks.1.conv2.rbr_dense.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.blocks.1.conv2.rbr_1x1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.blocks.1.conv2.rbr_1x1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.conv3.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.conv3.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage4.0.conv_down.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage4.0.conv_down.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage4.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage4.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage4.0.conv2.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage4.0.conv2.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage4.0.blocks.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage4.0.blocks.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage4.0.blocks.0.conv2.rbr_dense.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage4.0.blocks.0.conv2.rbr_dense.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage4.0.blocks.0.conv2.rbr_1x1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage4.0.blocks.0.conv2.rbr_1x1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage4.0.conv3.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage4.0.conv3.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.reduce_layers.2.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.reduce_layers.2.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.reduce_layers.2.0.conv2.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.reduce_layers.2.0.conv2.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.reduce_layers.2.0.blocks.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.reduce_layers.2.0.blocks.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.reduce_layers.2.0.blocks.0.conv2.rbr_dense.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.reduce_layers.2.0.blocks.0.conv2.rbr_dense.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.reduce_layers.2.0.blocks.0.conv2.rbr_1x1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.reduce_layers.2.0.blocks.0.conv2.rbr_1x1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.reduce_layers.2.0.blocks.spp.conv2.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.reduce_layers.2.0.blocks.spp.conv2.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.reduce_layers.2.0.conv3.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.reduce_layers.2.0.conv3.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.upsample_layers.0.0.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.upsample_layers.0.0.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.upsample_layers.1.0.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.upsample_layers.1.0.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.0.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.0.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.0.0.conv2.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.0.0.conv2.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.0.0.blocks.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.0.0.blocks.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.0.0.blocks.0.conv2.rbr_dense.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.0.0.blocks.0.conv2.rbr_dense.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.0.0.blocks.0.conv2.rbr_1x1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.0.0.blocks.0.conv2.rbr_1x1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.0.0.conv3.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.0.0.conv3.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.1.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.1.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.1.0.conv2.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.1.0.conv2.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.1.0.blocks.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.1.0.blocks.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.1.0.blocks.0.conv2.rbr_dense.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.1.0.blocks.0.conv2.rbr_dense.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.1.0.blocks.0.conv2.rbr_1x1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.1.0.blocks.0.conv2.rbr_1x1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.1.0.conv3.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.1.0.conv3.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.downsample_layers.0.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.downsample_layers.0.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.downsample_layers.1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.downsample_layers.1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.0.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.0.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.0.0.conv2.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.0.0.conv2.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.0.0.blocks.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.0.0.blocks.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.0.0.blocks.0.conv2.rbr_dense.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.0.0.blocks.0.conv2.rbr_dense.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.0.0.blocks.0.conv2.rbr_1x1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.0.0.blocks.0.conv2.rbr_1x1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.0.0.conv3.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.0.0.conv3.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.1.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.1.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.1.0.conv2.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.1.0.conv2.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.1.0.blocks.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.1.0.blocks.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.1.0.blocks.0.conv2.rbr_dense.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.1.0.blocks.0.conv2.rbr_dense.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.1.0.blocks.0.conv2.rbr_1x1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.1.0.blocks.0.conv2.rbr_1x1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.1.0.conv3.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.1.0.conv3.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- bbox_head.head_module.cls_stems.0.conv.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- bbox_head.head_module.cls_stems.0.conv.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- bbox_head.head_module.cls_stems.1.conv.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- bbox_head.head_module.cls_stems.1.conv.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- bbox_head.head_module.cls_stems.2.conv.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- bbox_head.head_module.cls_stems.2.conv.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- bbox_head.head_module.reg_stems.0.conv.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- bbox_head.head_module.reg_stems.0.conv.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- bbox_head.head_module.reg_stems.1.conv.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- bbox_head.head_module.reg_stems.1.conv.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- bbox_head.head_module.reg_stems.2.conv.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- bbox_head.head_module.reg_stems.2.conv.bn.bias:weight_decay=0.0
loading annotations into memory...
Done (t=0.26s)
creating index...
index created!
loading annotations into memory...
Done (t=0.26s)
creating index...
index created!
Loads checkpoint by http backend from path: https://download.openmmlab.com/mmyolo/v0/ppyoloe/ppyoloe_pretrain/ppyoloe_plus_s_obj365_pretrained-bcfe8478.pth
The model and loaded state dict do not match exactly
size mismatch for bbox_head.head_module.cls_preds.0.weight: copying a param with shape torch.Size([365, 96, 3, 3]) from checkpoint, the shape in current model is torch.Size([80, 96, 3, 3]).
size mismatch for bbox_head.head_module.cls_preds.0.bias: copying a param with shape torch.Size([365]) from checkpoint, the shape in current model is torch.Size([80]).
size mismatch for bbox_head.head_module.cls_preds.1.weight: copying a param with shape torch.Size([365, 192, 3, 3]) from checkpoint, the shape in current model is torch.Size([80, 192, 3, 3]).
size mismatch for bbox_head.head_module.cls_preds.1.bias: copying a param with shape torch.Size([365]) from checkpoint, the shape in current model is torch.Size([80]).
size mismatch for bbox_head.head_module.cls_preds.2.weight: copying a param with shape torch.Size([365, 384, 3, 3]) from checkpoint, the shape in current model is torch.Size([80, 384, 3, 3]).
size mismatch for bbox_head.head_module.cls_preds.2.bias: copying a param with shape torch.Size([365]) from checkpoint, the shape in current model is torch.Size([80]).
missing keys in source state_dict: backbone.stage1.0.blocks.0.conv2.alpha, backbone.stage2.0.blocks.0.conv2.alpha, backbone.stage2.0.blocks.1.conv2.alpha, backbone.stage3.0.blocks.0.conv2.alpha, backbone.stage3.0.blocks.1.conv2.alpha, backbone.stage4.0.blocks.0.conv2.alpha
The model and loaded state dict do not match exactly
size mismatch for bbox_head.head_module.cls_preds.0.weight: copying a param with shape torch.Size([365, 96, 3, 3]) from checkpoint, the shape in current model is torch.Size([80, 96, 3, 3]).
size mismatch for bbox_head.head_module.cls_preds.0.bias: copying a param with shape torch.Size([365]) from checkpoint, the shape in current model is torch.Size([80]).
size mismatch for bbox_head.head_module.cls_preds.1.weight: copying a param with shape torch.Size([365, 192, 3, 3]) from checkpoint, the shape in current model is torch.Size([80, 192, 3, 3]).
size mismatch for bbox_head.head_module.cls_preds.1.bias: copying a param with shape torch.Size([365]) from checkpoint, the shape in current model is torch.Size([80]).
size mismatch for bbox_head.head_module.cls_preds.2.weight: copying a param with shape torch.Size([365, 384, 3, 3]) from checkpoint, the shape in current model is torch.Size([80, 384, 3, 3]).
size mismatch for bbox_head.head_module.cls_preds.2.bias: copying a param with shape torch.Size([365]) from checkpoint, the shape in current model is torch.Size([80]).
missing keys in source state_dict: backbone.stage1.0.blocks.0.conv2.alpha, backbone.stage2.0.blocks.0.conv2.alpha, backbone.stage2.0.blocks.1.conv2.alpha, backbone.stage3.0.blocks.0.conv2.alpha, backbone.stage3.0.blocks.1.conv2.alpha, backbone.stage4.0.blocks.0.conv2.alpha
09/26 10:12:23 - mmengine - INFO - Load checkpoint from https://download.openmmlab.com/mmyolo/v0/ppyoloe/ppyoloe_pretrain/ppyoloe_plus_s_obj365_pretrained-bcfe8478.pth
09/26 10:12:23 - mmengine - WARNING - "FileClient" will be deprecated in future. Please use io functions in https://mmengine.readthedocs.io/en/latest/api/fileio.html#file-io
09/26 10:12:23 - mmengine - WARNING - "HardDiskBackend" is the alias of "LocalBackend" and the former will be deprecated in future.
09/26 10:12:23 - mmengine - INFO - Checkpoints will be saved to /home/bc/code/mmyolo/work_dirs/ppyoloe_plus_s_fast_8xb8-80e_coco.
/opt/miniconda3/envs/mm/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3483.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
Traceback (most recent call last):
File "/home/bc/code/mmyolo/mmyolo/.mim/tools/train.py", line 123, in <module>
main()
File "/home/bc/code/mmyolo/mmyolo/.mim/tools/train.py", line 119, in main
runner.train()
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/mmengine/runner/runner.py", line 1777, in train
model = self.train_loop.run() # type: ignore
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/mmengine/runner/loops.py", line 98, in run
self.run_epoch()
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/mmengine/runner/loops.py", line 114, in run_epoch
for idx, data_batch in enumerate(self.dataloader):
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 633, in __next__
data = self._next_data()
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1345, in _next_data
return self._process_data(data)
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1371, in _process_data
data.reraise()
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/torch/_utils.py", line 644, in reraise
raise exception
RuntimeError: Caught RuntimeError in pin memory thread for device 0.
Original Traceback (most recent call last):
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/torch/utils/data/_utils/pin_memory.py", line 34, in do_one_step
data = pin_memory(data, device)
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/torch/utils/data/_utils/pin_memory.py", line 60, in pin_memory
return type(data)({k: pin_memory(sample, device) for k, sample in data.items()}) # type: ignore[call-arg]
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/torch/utils/data/_utils/pin_memory.py", line 60, in <dictcomp>
return type(data)({k: pin_memory(sample, device) for k, sample in data.items()}) # type: ignore[call-arg]
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/torch/utils/data/_utils/pin_memory.py", line 70, in pin_memory
return type(data)([pin_memory(sample, device) for sample in data]) # type: ignore[call-arg]
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/torch/utils/data/_utils/pin_memory.py", line 70, in <listcomp>
return type(data)([pin_memory(sample, device) for sample in data]) # type: ignore[call-arg]
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/torch/utils/data/_utils/pin_memory.py", line 55, in pin_memory
return data.pin_memory(device)
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Traceback (most recent call last):
File "/opt/miniconda3/envs/mm/bin/mim", line 8, in <module>
sys.exit(cli())
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/mim/commands/train.py", line 100, in cli
is_success, msg = train(
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/mim/commands/train.py", line 261, in train
ret = subprocess.check_call(
File "/opt/miniconda3/envs/mm/lib/python3.10/subprocess.py", line 369, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/opt/miniconda3/envs/mm/bin/python', '/home/bc/code/mmyolo/mmyolo/.mim/tools/train.py', 'configs/ppyoloe/ppyoloe_plus_s_fast_8xb8-80e_coco.py', '--launcher', 'none']' returned non-zero exit status 1. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
However I have not modified the pin_memery in val_dataloader, it remained |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
mmyoloor so.pin_memory=truein theWSLenviroment will cause an error. So putting it toFalsewill fix it.Pytorchandpin_memory, see got the error: out of memory ,when invoke cuda in wsl2. microsoft/WSL#8447 (comment)This is the process I followed to discover and solve the problem
mim train mmyolo configs/ppyoloe/ppyoloe_plus_s_fast_8xb8-80e_coco.py --cfg-options train_dataloader.batch_size=16 train_dataloader.num_workers=8 optim_wrapper.optimizer.lr=0.00025Click to expand/collapse
pin memory.Torch,mmlab,mmyolowithpin memoryorTORCH_USE_CUDA_DSA.pin_memory=truein theWSL2environment will cause an error.mim train mmyolo configs/ppyoloe/ppyoloe_plus_s_fast_8xb8-80e_coco.py --cfg-options train_dataloader.batch_size=16 train_dataloader.num_workers=8 optim_wrapper.optimizer.lr=0.00025 train_dataloader.pin_memory=FalseBeta Was this translation helpful? Give feedback.
All reactions