Skip to content
This repository has been archived by the owner on May 11, 2020. It is now read-only.

Home

Jump to bottom
JoWa edited this page Mar 6, 2013 · 38 revisions

Component Rating System

This Component Rating System can be used to evaluate individual components of a software system by considering performance evaluations of component combinations. It provides a default implementation that is based on Microsoft's TrueSkill(tm) player rating system and also a generic interface for alternative implementations.

Development

The Component Rating System (CRS) is written in Scala, which is bytecode-compatible with Java, i.e. it runs on the JVM. The CRS can thus be easily used in any Scala or Java application. One easy way of setting up a Scala developement enviroment is the Scala IDE for Eclipse. Scala 2.9.2 is required for compilation.

Simple Example Setup of Component Rating System

To simply use the Component Rating System you need to instantiate the class

val crs: ComponentRatingSystem = alesia.componentrating.TrueSkillRatingSystem()

This instance holds the component ratings and receives updates in the form of component combination comparisons. Initially, nothing is known about your components. This changes as more comparison results are submitted.

A comparison result is an ordering of components resembling their relative results at the time of the comparison. A simple example will help understand comparison results:

val comparisonResult = List("Pentium", "486", "C64")

Components are identified by a string. In the above example, a comparison of three components resembling computer models has resulted in an ordering according to their computation power with the "Pentium" model scoring first place, the 468 second and the C64 last.

The TrueSkill Rating System was build to rank compound "teams" formed out of one or more components each. This means, the three components above need to be in teams of one member each:

val compResult = List( Set("Pentium"), Set("468"), Set("C64") )

For the comparison result to take effect and the CRS updating the knowledgebase accordingly, the comparison result needs to be submitted to the instance of the ranking system:

crs.submitResults(compResult)

Each submission changes the knowledgebase and updates what the CRS believes to be the real strength of each component. After several updates, if the submitted comparison results are not too noisy and each component show up in at least some updates, the CRS comes up with a ranking of the components somewhat resemling their "real" (with regards to the submitted comparison results) strength.

The knowledge base can be accessed in two ways. Firstly, the components can be compared with regards to their current strength using the comparator interface:

val comparisonOfTwoComponents = crs.compare("486", "C64") // will yield some double < 0 here

Secondly, the strength and uncertainty of each component in the knowledge base can be accessed directly:

val strength = crs.getPoints("Pentium") // will yield some double around 30.0 here
val uncertainty = crs.getUncertainty("Pentium") // will yield some double around 9.0 here

The meaning of these values are straight forward: the component with higher points is more likely to win a comparison and a higher uncertainty means that the lately submitted comparison results may have been contradictory.

Advanced Features

Setting the Rating of a Component

To overwrite the rating of one component or to add a component with a certain rating instead of the default one, injectRating can be used:

val componentName = "MyComponent"
val componentPoints = 40.0 // mean
val componentUncertainty = 8.0 // sigma
crs.injectRating(componentName, componentPoints, componentUncertainty)

Partially Reusing the Rating of an old Component after it was updated

It may happen that a component has been updated, that is, its strength has supposedly changed greatly, but the old rating should still be taken into account. In that case use componentUpdated:

crs.componentUpdated("ComponentName")

The uncertainty of the Component is increased in the knowledge base to take account of the update. At the point of writing this, it is multiplied by 1.5.

TrueSkill's Default Values

The TrueSkill algorithm relies on some basic variables that influence the updates. For those variables default values are given in the papers that introduced TrueSkill. The Alesia Component Rating System holds these variables in the class TrueSkillDefaultValues, an instance of which can be implicitly passed to TrueSkillRatingSystem. For most purposes nothing needs to be passed and the default values as given by the paper of Herbrich et al. are used.

If you want to change a variable refer to the following example (setting the "skill chain" beta to 500):

val myDefaultValues = new TrueSkillDefaultValues(beta = 500.0)
val crs = alesia.componentrating.TrueSkillRatingSystem()(myDefaultValues, new AdvancedOptions)

Note that new AdvancedOptions part is necessary here due to a limitation of Scala (both parameters need to be stated). The default values you can change are: the default skill consisting of mean and uncertainty (example: defaultSkill = new NormalDist( myMean, myUncertainty )), draw probability pDraw, skill chain beta, uncertainty additive factor tau tau, accuracy of the approximative algorithm delta (note: the closer to 0 the more accurate) and the debug flag debug.

Partial Play Component Factors

The original inventors of TrueSkill considered a scenario, where a player might not participate in the whole game but rather leave or join after some time. To take that into account partial play was developed. When submitting game results to the TrueSkillRatingSystem a player can be marked as contributing only partly to his teams performance in this game. To set the partial play factor for a component to 50%:

crs.setPartialPlayFactor("myComponentName", 0.5)

This is persistent within the TrueSkillRatingSystem instance.

The Influence of Team Size

One of the more controversial features of TrueSkill is the influence it gives to team size on the supposed team strength: the strenght of the team equals the sum of the strength of its components. The result is, that components, which play in bigger teams more often, are ranked lower.

There are some applications, for which a different approach is prefered. For example, the team strength could be around the average strength of this teams components, so the influence of team size (almost) removed. This can be set in the advancedOptions, which are then passed to the TrueSkillRatingSystem on construction:

val crs: ComponentRatingSystem = TrueSkillRatingSystem()(new DefaultOptions, new AdvancedOptions(useVPBalancing=true))

Note that, if any options are passed, both must be present. The above example uses the Virtual Player Team Size Balancing which was observed to be somewhat closer to the desired effect than the Partial Play Team Size Balancing.

Further Reading

Clone this wiki locally