Ranking not always according to "aggregate_score" (Chai-1 server)

[smg3d]

I thought ranking was based on aggregate scores (0.2 * iptm + 0.8 * ptm - 100 * clash)?

If so, model_idx_2 (with "aggregate_score": 0.28714820742607117) should be rank_0, but it is rank_3.

The results from the .json files (with their aggregate_score):

- model_idx_2 (0.28714820742607117) is rank_3      (should be rank_0)
- model_idx_4 (0.24289762973785400) is rank_1
- model_idx_3 (0.24094939231872559) is rank_0      (should be rank_2)
- model_idx_0 (0.23602318763732910) is rank_2      (should be rank_3)
- model_idx_1 (0.21727515757083893) is rank_4

Here are the .json files:

::::::::::::::
scores.model_idx_0.rank_2.json
::::::::::::::
{
  "aggregate_score": 0.2360231876373291,
  "ptm": 0.6798261404037476,
  "iptm": 0.1250724345445633,
  "per_chain_ptm": [
    [
      0.6851804256439209,
      0.18351513147354126
    ]
  ],
  "per_chain_pair_iptm": [
    [
      [
        0.6851804256439209,
        0.010061381384730339
      ],
      [
        0.1250724345445633,
        0.18351513147354126
      ]
    ]
  ],
  "has_inter_chain_clashes": false,
  "chain_chain_clashes": [
    [
      [
        0,
        0
      ],
      [
        0,
        0
      ]
    ]
  ]
}
::::::::::::::
scores.model_idx_1.rank_4.json
::::::::::::::
{
  "aggregate_score": 0.21727515757083893,
  "ptm": 0.679915189743042,
  "iptm": 0.10161515325307846,
  "per_chain_ptm": [
    [
      0.6894143223762512,
      0.18856889009475708
    ]
  ],
  "per_chain_pair_iptm": [
    [
      [
        0.6894143223762512,
        0.0038339924067258835
      ],
      [
        0.10161515325307846,
        0.18856889009475708
      ]
    ]
  ],
  "has_inter_chain_clashes": false,
  "chain_chain_clashes": [
    [
      [
        0,
        0
      ],
      [
        0,
        0
      ]
    ]
  ]
}
::::::::::::::
scores.model_idx_2.rank_3.json
::::::::::::::
{
  "aggregate_score": 0.28714820742607117,
  "ptm": 0.6902005672454834,
  "iptm": 0.1863851100206375,
  "per_chain_ptm": [
    [
      0.689195454120636,
      0.18162894248962402
    ]
  ],
  "per_chain_pair_iptm": [
    [
      [
        0.689195454120636,
        0.029202930629253387
      ],
      [
        0.1863851100206375,
        0.18162894248962402
      ]
    ]
  ],
  "has_inter_chain_clashes": false,
  "chain_chain_clashes": [
    [
      [
        0,
        0
      ],
      [
        0,
        0
      ]
    ]
  ]
}
::::::::::::::
scores.model_idx_3.rank_0.json
::::::::::::::
{
  "aggregate_score": 0.24094939231872559,
  "ptm": 0.6850574016571045,
  "iptm": 0.12992237508296967,
  "per_chain_ptm": [
    [
      0.691375732421875,
      0.1818333864212036
    ]
  ],
  "per_chain_pair_iptm": [
    [
      [
        0.691375732421875,
        0.008454011753201485
      ],
      [
        0.12992237508296967,
        0.1818333864212036
      ]
    ]
  ],
  "has_inter_chain_clashes": false,
  "chain_chain_clashes": [
    [
      [
        0,
        0
      ],
      [
        0,
        0
      ]
    ]
  ]
}
::::::::::::::
scores.model_idx_4.rank_1.json
::::::::::::::
{
  "aggregate_score": 0.242897629737854,
  "ptm": 0.685787558555603,
  "iptm": 0.13217514753341675,
  "per_chain_ptm": [
    [
      0.6912361979484558,
      0.18199051916599274
    ]
  ],
  "per_chain_pair_iptm": [
    [
      [
        0.6912361979484558,
        0.00896384660154581
      ],
      [
        0.13217514753341675,
        0.18199051916599274
      ]
    ]
  ],
  "has_inter_chain_clashes": false,
  "chain_chain_clashes": [
    [
      [
        0,
        0
      ],
      [
        0,
        0
      ]
    ]
  ]
}

[jackdent]

Can you share your job id (e.g. https://lab.chaidiscovery.com/dashboard/jobs/{JOB_ID})?

[smg3d]

Can you share your job id (e.g. https://lab.chaidiscovery.com/dashboard/jobs/{JOB_ID})?

https://lab.chaidiscovery.com/dashboard/jobs/cm527t8hw07bxpm3ve32unt0y

[jackdent]

We merged a change that we're hoping fixes this issue. Please let us know if this issues recurs (N.B. we cache jobs, so if you specify the exact same inputs as last time, there won't be any change to the outputs--you'll need to test with new inputs).