You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Right now, the rating of most of the bots just doesn't make sense.
The current rating system is designed to include everyone, bots, and humans but in reality, it doesn't. Bots play lots of rated games against other bots, making the rating of newer bots not provisional relatively fast. Humans, on the other hand, don't play rated games against bots, and why is that? Because they are pretty much guaranteed to lose even against bots that are much lower rated than them.
How it shouldn't be
On the one hand we have one of the worst examples: SimpleEval https://lichess.org/SExc3smv. It had a blitz rating of 2100 when that match happened, and its opponent, a human NM, had a rating of 2700. The human should've been able to win comfortably most of the games, but what happened is that it got demolished 70.5 - 1.5. The human obviously didn't want to play rated games against the bot because it would've lost a lot of rating points because the rating of the bot was not provisional, it was considered real and it was way lower that what it should be. If SimpleEval had a rating of, say 3500, maybe the human would've considered playing rated games against it, because losing so many games was expected.
How it should be
And on the other hand we have the best examples of how the system should be: Maia and LeelaLevel5. These bots play lots of rated games against humans. Humans know that the rating of these bots is accurate so they don't fear playing rated games as much. The rating of these bots is so good that I think they should even replace the current "computer" system used.
What to do about it
I don't really know what the best solution to this problem would be, but I think that using other "random" bots to figure out the rating of newer bots is not the solution. Maybe bots shouldn't be allowed to play rated games against other bots at all, or at least, not against bots that haven't played the majority of their games against humans.
I understand that fixing this now might be hard, but please, consider it.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
The problem
Right now, the rating of most of the bots just doesn't make sense.
The current rating system is designed to include everyone, bots, and humans but in reality, it doesn't. Bots play lots of rated games against other bots, making the rating of newer bots not provisional relatively fast. Humans, on the other hand, don't play rated games against bots, and why is that? Because they are pretty much guaranteed to lose even against bots that are much lower rated than them.
How it shouldn't be
On the one hand we have one of the worst examples: SimpleEval https://lichess.org/SExc3smv. It had a blitz rating of 2100 when that match happened, and its opponent, a human NM, had a rating of 2700. The human should've been able to win comfortably most of the games, but what happened is that it got demolished 70.5 - 1.5. The human obviously didn't want to play rated games against the bot because it would've lost a lot of rating points because the rating of the bot was not provisional, it was considered real and it was way lower that what it should be. If SimpleEval had a rating of, say 3500, maybe the human would've considered playing rated games against it, because losing so many games was expected.
How it should be
And on the other hand we have the best examples of how the system should be: Maia and LeelaLevel5. These bots play lots of rated games against humans. Humans know that the rating of these bots is accurate so they don't fear playing rated games as much. The rating of these bots is so good that I think they should even replace the current "computer" system used.
What to do about it
I don't really know what the best solution to this problem would be, but I think that using other "random" bots to figure out the rating of newer bots is not the solution. Maybe bots shouldn't be allowed to play rated games against other bots at all, or at least, not against bots that haven't played the majority of their games against humans.
I understand that fixing this now might be hard, but please, consider it.
Beta Was this translation helpful? Give feedback.
All reactions