Skip to content

Latest commit

 

History

History
322 lines (273 loc) · 17 KB

COMPARISON.md

File metadata and controls

322 lines (273 loc) · 17 KB
Raw

Zimtohrli correlation report

Created at 2024-06-12

Revision v0.1.10-42-g214796d, branch work, origin [email protected]:google/zimtohrli.git

PEASS_DB

Spearman correlation table for all score types

2f AQUA CDPAM DPAM GVPMOS MOS PARLAQ PEAQB PESQ ViSQOL WARP-Q Zimtohrli
2f 1.00 0.73 0.74 0.37 0.27 0.53 0.65 0.65 0.64 0.78 0.49 0.82
AQUA 0.73 1.00 0.79 0.52 0.20 0.33 0.85 0.81 0.63 0.56 0.75 0.67
CDPAM 0.74 0.79 1.00 0.47 0.19 0.47 0.85 0.78 0.65 0.56 0.65 0.74
DPAM 0.37 0.52 0.47 1.00 0.12 0.40 0.57 0.50 0.33 0.36 0.34 0.51
GVPMOS 0.27 0.20 0.19 0.12 1.00 0.17 0.18 0.09 0.27 0.19 0.32 0.33
MOS 0.53 0.33 0.47 0.40 0.17 1.00 0.45 0.36 0.34 0.52 0.25 0.56
PARLAQ 0.65 0.85 0.85 0.57 0.18 0.45 1.00 0.83 0.68 0.49 0.78 0.72
PEAQB 0.65 0.81 0.78 0.50 0.09 0.36 0.83 1.00 0.70 0.40 0.71 0.65
PESQ 0.64 0.63 0.65 0.33 0.27 0.34 0.68 0.70 1.00 0.43 0.59 0.57
ViSQOL 0.78 0.56 0.56 0.36 0.19 0.52 0.49 0.40 0.43 1.00 0.35 0.70
WARP-Q 0.49 0.75 0.65 0.34 0.32 0.25 0.78 0.71 0.59 0.35 1.00 0.49
Zimtohrli 0.82 0.67 0.74 0.51 0.33 0.56 0.72 0.65 0.57 0.70 0.49 1.00

Score type MOS Spearman correlation in order

Score type Spearman correlation
Zimtohrli 0.56
2f 0.53
ViSQOL 0.52
CDPAM 0.47
PARLAQ 0.45
DPAM 0.40
PEAQB 0.36
PESQ 0.34
AQUA 0.33
WARP-Q 0.25
GVPMOS 0.17

SAOC

Spearman correlation table for all score types

2f AQUA CDPAM DPAM GVPMOS MOS PARLAQ PEAQB PESQ ViSQOL WARP-Q Zimtohrli
2f 1.00 0.82 0.76 0.62 0.24 0.78 0.88 0.74 0.85 0.84 0.76 0.90
AQUA 0.82 1.00 0.74 0.54 0.13 0.69 0.87 0.72 0.67 0.79 0.77 0.84
CDPAM 0.76 0.74 1.00 0.62 0.30 0.61 0.87 0.62 0.61 0.73 0.71 0.77
DPAM 0.62 0.54 0.62 1.00 0.53 0.50 0.55 0.55 0.48 0.47 0.66 0.60
GVPMOS 0.24 0.13 0.30 0.53 1.00 0.18 0.23 0.01 0.19 0.05 0.31 0.21
MOS 0.78 0.69 0.61 0.50 0.18 1.00 0.73 0.59 0.76 0.74 0.62 0.72
PARLAQ 0.88 0.87 0.87 0.55 0.23 0.73 1.00 0.67 0.76 0.85 0.75 0.88
PEAQB 0.74 0.72 0.62 0.55 0.01 0.59 0.67 1.00 0.60 0.71 0.70 0.75
PESQ 0.85 0.67 0.61 0.48 0.19 0.76 0.76 0.60 1.00 0.80 0.65 0.81
ViSQOL 0.84 0.79 0.73 0.47 0.05 0.74 0.85 0.71 0.80 1.00 0.70 0.81
WARP-Q 0.76 0.77 0.71 0.66 0.31 0.62 0.75 0.70 0.65 0.70 1.00 0.74
Zimtohrli 0.90 0.84 0.77 0.60 0.21 0.72 0.88 0.75 0.81 0.81 0.74 1.00

Score type MOS Spearman correlation in order

Score type Spearman correlation
2f 0.78
PESQ 0.76
ViSQOL 0.74
PARLAQ 0.73
Zimtohrli 0.72
AQUA 0.69
WARP-Q 0.62
CDPAM 0.61
PEAQB 0.59
DPAM 0.50
GVPMOS 0.18

SASSEC

Spearman correlation table for all score types

2f AQUA CDPAM DPAM GVPMOS MOS PARLAQ PEAQB PESQ ViSQOL WARP-Q Zimtohrli
2f 1.00 0.84 0.84 0.61 0.22 0.79 0.88 0.87 0.85 0.88 0.74 0.86
AQUA 0.84 1.00 0.76 0.53 0.09 0.68 0.82 0.79 0.72 0.84 0.76 0.80
CDPAM 0.84 0.76 1.00 0.56 0.21 0.67 0.88 0.79 0.73 0.82 0.74 0.79
DPAM 0.61 0.53 0.56 1.00 0.51 0.50 0.52 0.56 0.46 0.56 0.66 0.64
GVPMOS 0.22 0.09 0.21 0.51 1.00 0.24 0.22 0.09 0.15 0.15 0.35 0.26
MOS 0.79 0.68 0.67 0.50 0.24 1.00 0.74 0.68 0.76 0.73 0.65 0.68
PARLAQ 0.88 0.82 0.88 0.52 0.22 0.74 1.00 0.75 0.82 0.88 0.73 0.83
PEAQB 0.87 0.79 0.79 0.56 0.09 0.68 0.75 1.00 0.78 0.81 0.72 0.75
PESQ 0.85 0.72 0.73 0.46 0.15 0.76 0.82 0.78 1.00 0.83 0.67 0.77
ViSQOL 0.88 0.84 0.82 0.56 0.15 0.73 0.88 0.81 0.83 1.00 0.76 0.82
WARP-Q 0.74 0.76 0.74 0.66 0.35 0.65 0.73 0.72 0.67 0.76 1.00 0.74
Zimtohrli 0.86 0.80 0.79 0.64 0.26 0.68 0.83 0.75 0.77 0.82 0.74 1.00

Score type MOS Spearman correlation in order

Score type Spearman correlation
2f 0.79
PESQ 0.76
PARLAQ 0.74
ViSQOL 0.73
AQUA 0.68
PEAQB 0.68
Zimtohrli 0.68
CDPAM 0.67
WARP-Q 0.65
DPAM 0.50
GVPMOS 0.24

SiSEC08

Spearman correlation table for all score types

2f AQUA CDPAM DPAM GVPMOS MOS PARLAQ PEAQB PESQ ViSQOL WARP-Q Zimtohrli
2f 1.00 0.86 0.85 0.62 0.15 0.79 0.92 0.83 0.87 0.89 0.89 0.96
AQUA 0.86 1.00 0.70 0.55 0.05 0.74 0.87 0.85 0.71 0.80 0.82 0.87
CDPAM 0.85 0.70 1.00 0.53 0.16 0.70 0.86 0.72 0.80 0.79 0.75 0.83
DPAM 0.62 0.55 0.53 1.00 0.47 0.54 0.53 0.56 0.46 0.49 0.66 0.58
GVPMOS 0.15 0.05 0.16 0.47 1.00 0.13 0.13 0.04 0.06 0.01 0.28 0.12
MOS 0.79 0.74 0.70 0.54 0.13 1.00 0.78 0.66 0.73 0.78 0.75 0.76
PARLAQ 0.92 0.87 0.86 0.53 0.13 0.78 1.00 0.72 0.84 0.86 0.82 0.93
PEAQB 0.83 0.85 0.72 0.56 0.04 0.66 0.72 1.00 0.70 0.78 0.79 0.79
PESQ 0.87 0.71 0.80 0.46 0.06 0.73 0.84 0.70 1.00 0.86 0.79 0.85
ViSQOL 0.89 0.80 0.79 0.49 0.01 0.78 0.86 0.78 0.86 1.00 0.81 0.83
WARP-Q 0.89 0.82 0.75 0.66 0.28 0.75 0.82 0.79 0.79 0.81 1.00 0.84
Zimtohrli 0.96 0.87 0.83 0.58 0.12 0.76 0.93 0.79 0.85 0.83 0.84 1.00

Score type MOS Spearman correlation in order

Score type Spearman correlation
2f 0.79
PARLAQ 0.78
ViSQOL 0.78
Zimtohrli 0.76
WARP-Q 0.75
AQUA 0.74
PESQ 0.73
CDPAM 0.70
PEAQB 0.66
DPAM 0.54
GVPMOS 0.13

coresvnet

Spearman correlation table for all score types

2f AQUA CDPAM DPAM GVPMOS MOS PARLAQ PEAQB PESQ ViSQOL WARP-Q Zimtohrli
2f 1.00 0.53 0.73 0.31 0.26 0.77 0.65 0.87 0.71 0.77 0.14 0.74
AQUA 0.53 1.00 0.62 0.60 0.20 0.40 0.68 0.33 0.69 0.29 0.14 0.58
CDPAM 0.73 0.62 1.00 0.49 0.17 0.59 0.68 0.60 0.67 0.54 0.19 0.67
DPAM 0.31 0.60 0.49 1.00 0.21 0.21 0.58 0.13 0.38 0.17 0.23 0.40
GVPMOS 0.26 0.20 0.17 0.21 1.00 0.17 0.37 0.24 0.13 0.13 0.32 0.29
MOS 0.77 0.40 0.59 0.21 0.17 1.00 0.54 0.85 0.57 0.80 0.10 0.71
PARLAQ 0.65 0.68 0.68 0.58 0.37 0.54 1.00 0.55 0.67 0.51 0.29 0.73
PEAQB 0.87 0.33 0.60 0.13 0.24 0.85 0.55 1.00 0.63 0.86 0.14 0.77
PESQ 0.71 0.69 0.67 0.38 0.13 0.57 0.67 0.63 1.00 0.51 0.06 0.70
ViSQOL 0.77 0.29 0.54 0.17 0.13 0.80 0.51 0.86 0.51 1.00 0.04 0.72
WARP-Q 0.14 0.14 0.19 0.23 0.32 0.10 0.29 0.14 0.06 0.04 1.00 0.18
Zimtohrli 0.74 0.58 0.67 0.40 0.29 0.71 0.73 0.77 0.70 0.72 0.18 1.00

Score type MOS Spearman correlation in order

Score type Spearman correlation
PEAQB 0.85
ViSQOL 0.80
2f 0.77
Zimtohrli 0.71
CDPAM 0.59
PESQ 0.57
PARLAQ 0.54
AQUA 0.40
DPAM 0.21
GVPMOS 0.17
WARP-Q 0.10

furball_mushra

Spearman correlation table for all score types

2f AQUA CDPAM DPAM GVPMOS MOS PARLAQ PEAQB PESQ ViSQOL WARP-Q Zimtohrli
2f 1.00 0.91 0.75 0.71 0.75 0.60 0.94 0.76 0.79 0.53 0.69 0.76
AQUA 0.91 1.00 0.73 0.72 0.80 0.64 0.91 0.76 0.81 0.55 0.68 0.81
CDPAM 0.75 0.73 1.00 0.61 0.84 0.66 0.67 0.63 0.85 0.62 0.77 0.87
DPAM 0.71 0.72 0.61 1.00 0.64 0.54 0.71 0.61 0.67 0.66 0.57 0.64
GVPMOS 0.75 0.80 0.84 0.64 1.00 0.74 0.69 0.63 0.96 0.62 0.78 0.95
MOS 0.60 0.64 0.66 0.54 0.74 1.00 0.56 0.54 0.74 0.52 0.58 0.73
PARLAQ 0.94 0.91 0.67 0.71 0.69 0.56 1.00 0.79 0.71 0.56 0.65 0.67
PEAQB 0.76 0.76 0.63 0.61 0.63 0.54 0.79 1.00 0.67 0.58 0.65 0.65
PESQ 0.79 0.81 0.85 0.67 0.96 0.74 0.71 0.67 1.00 0.64 0.75 0.95
ViSQOL 0.53 0.55 0.62 0.66 0.62 0.52 0.56 0.58 0.64 1.00 0.73 0.59
WARP-Q 0.69 0.68 0.77 0.57 0.78 0.58 0.65 0.65 0.75 0.73 1.00 0.72
Zimtohrli 0.76 0.81 0.87 0.64 0.95 0.73 0.67 0.65 0.95 0.59 0.72 1.00

Score type MOS Spearman correlation in order

Score type Spearman correlation
PESQ 0.74
GVPMOS 0.74
Zimtohrli 0.73
CDPAM 0.66
AQUA 0.64
2f 0.60
WARP-Q 0.58
PARLAQ 0.56
PEAQB 0.54
DPAM 0.54
ViSQOL 0.52

minimodal_cmaudio_mushra

Spearman correlation table for all score types

2f AQUA CDPAM DPAM GVPMOS MOS PARLAQ PEAQB PESQ ViSQOL WARP-Q Zimtohrli
2f 1.00 0.65 0.44 0.24 0.58 0.54 0.53 0.61 0.61 0.70 0.07 0.67
AQUA 0.65 1.00 0.30 0.25 0.47 0.46 0.77 0.55 0.73 0.56 0.18 0.69
CDPAM 0.44 0.30 1.00 0.19 0.44 0.44 0.20 0.21 0.35 0.41 0.08 0.38
DPAM 0.24 0.25 0.19 1.00 0.20 0.19 0.28 0.18 0.27 0.28 0.02 0.20
GVPMOS 0.58 0.47 0.44 0.20 1.00 0.91 0.50 0.34 0.65 0.61 0.09 0.64
MOS 0.54 0.46 0.44 0.19 0.91 1.00 0.47 0.28 0.60 0.61 0.07 0.64
PARLAQ 0.53 0.77 0.20 0.28 0.50 0.47 1.00 0.55 0.81 0.49 0.26 0.63
PEAQB 0.61 0.55 0.21 0.18 0.34 0.28 0.55 1.00 0.66 0.28 0.22 0.46
PESQ 0.61 0.73 0.35 0.27 0.65 0.60 0.81 0.66 1.00 0.57 0.22 0.74
ViSQOL 0.70 0.56 0.41 0.28 0.61 0.61 0.49 0.28 0.57 1.00 0.02 0.73
WARP-Q 0.07 0.18 0.08 0.02 0.09 0.07 0.26 0.22 0.22 0.02 1.00 0.14
Zimtohrli 0.67 0.69 0.38 0.20 0.64 0.64 0.63 0.46 0.74 0.73 0.14 1.00

Score type MOS Spearman correlation in order

Score type Spearman correlation
GVPMOS 0.91
Zimtohrli 0.64
ViSQOL 0.61
PESQ 0.60
2f 0.54
PARLAQ 0.47
AQUA 0.46
CDPAM 0.44
PEAQB 0.28
DPAM 0.19
WARP-Q 0.07

perceptual_audio

Maximal audibility classification accuracy and threshold per score type

Score type Accuracy Threshold
Zimtohrli 0.73 0.11
CDPAM 0.71 0.049
ViSQOL 0.71 4.2
DPAM 0.69 0.73
GVPMOS 0.51 1.5
WARP-Q 0.49 1.3
PEAQB 0.49 -3.9
PARLAQ 0.48 30
AQUA 0.48 0.018
2f 0.48 -14
PESQ 0.48 1

tcd-voip

Spearman correlation table for all score types

2f AQUA CDPAM DPAM GVPMOS MOS PARLAQ PEAQB PESQ ViSQOL WARP-Q Zimtohrli
2f 1.00 0.72 0.65 0.77 0.53 0.70 0.73 0.84 0.76 0.72 0.61 0.65
AQUA 0.72 1.00 0.63 0.48 0.45 0.50 0.90 0.75 0.59 0.49 0.57 0.57
CDPAM 0.65 0.63 1.00 0.51 0.74 0.73 0.54 0.55 0.64 0.76 0.79 0.86
DPAM 0.77 0.48 0.51 1.00 0.36 0.57 0.44 0.59 0.57 0.71 0.48 0.47
GVPMOS 0.53 0.45 0.74 0.36 1.00 0.63 0.41 0.41 0.49 0.63 0.63 0.68
MOS 0.70 0.50 0.73 0.57 0.63 1.00 0.52 0.54 0.84 0.65 0.78 0.65
PARLAQ 0.73 0.90 0.54 0.44 0.41 0.52 1.00 0.81 0.70 0.36 0.56 0.59
PEAQB 0.84 0.75 0.55 0.59 0.41 0.54 0.81 1.00 0.74 0.45 0.58 0.68
PESQ 0.76 0.59 0.64 0.57 0.49 0.84 0.70 0.74 1.00 0.46 0.77 0.68
ViSQOL 0.72 0.49 0.76 0.71 0.63 0.65 0.36 0.45 0.46 1.00 0.57 0.64
WARP-Q 0.61 0.57 0.79 0.48 0.63 0.78 0.56 0.58 0.77 0.57 1.00 0.80
Zimtohrli 0.65 0.57 0.86 0.47 0.68 0.65 0.59 0.68 0.68 0.64 0.80 1.00

Score type MOS Spearman correlation in order

Score type Spearman correlation
PESQ 0.84
WARP-Q 0.78
CDPAM 0.73
2f 0.70
ViSQOL 0.65
Zimtohrli 0.65
GVPMOS 0.63
DPAM 0.57
PEAQB 0.54
PARLAQ 0.52
AQUA 0.50

Global leaderboard across all studies

Mean square error (1 - Spearman correlation, or 1 - accuracy) per score type

Score type MSE Min score Max score Mean score
Zimtohrli 0.10 0.56 0.76 0.69
ViSQOL 0.12 0.52 0.80 0.68
2f 0.13 0.48 0.79 0.67
PESQ 0.15 0.34 0.84 0.65
CDPAM 0.15 0.44 0.73 0.62
PARLAQ 0.19 0.45 0.78 0.59
AQUA 0.22 0.33 0.74 0.55
PEAQB 0.23 0.28 0.85 0.55
DPAM 0.32 0.19 0.69 0.46
WARP-Q 0.34 0.07 0.78 0.48
GVPMOS 0.43 0.13 0.91 0.41