Note: This is a write-up for a project that I had the pleasure of doing during the spring of 2023 with Parimarjan Negi, chess grandmaster and MIT PhD student.
Introduction
During this UROP, Pari and I attempted to classify chess positions by using machine learning techniques. Our project was motivated by a question: are human-understandable chess concepts captured by machine learning models trained to predict human-like moves? In natural language processing, there is a common task called Next Word Prediction, which inspired some of our methods. We trained two move prediction models (the chess analog to word prediction), one on a dataset of games from 1100-rated players and another on games from 2600-rated games. Then, we extracted embeddings from the last few layers of the move prediction models, and attempted to use those embeddings to classify positions by theme. We were not able to see any obvious trends, so we concluded that human-understandable chess concepts were not clearly distinguished in the embeddings learned from our models.
Methodology
The first part of our project was to train the models that we would use to produce our embeddings. We followed the training methodology used in the Maia Chess [1] project, and we decided to train two different versions of the Maia Chess model on two different datasets. The first dataset contained online games from 1100 rated players (excluding games played with very low time), and the second dataset contained online bullet (~1 minute per side, little to no increment) games from 2600 rated players. Intuitively, the idea of including bullet games from higher rated players was that strong players can quickly play logical / human like moves, and in longer time controls, their calculation skills would probably be more critical than their intuition. With our ML model, it should be harder to learn decisions that depend on calculations, therefore, we consider bullet games of the stronger players. Our motivation for this was to see if there were any differences in embeddings created using lower rated players’ moves vs. higher rated players, and if chess concepts could be distinguished better using one or the other.
We then manually collected about 50 positions from a few chess books. The positions were tagged by chapter, and each chapter contained positions with a different “theme,” as defined by the book. Then, we generated embeddings for each position. This was done by running the position through the move-prediction models, and then extracting one of the last layers (before the activation layer) that contained the model’s internal representation of the position. This technique is commonly used in other machine learning settings as well. After generating the embeddings, we used TSNE [2] to graph the embeddings in two dimensions, and looked for trends/clusters in the graph. We hoped for relatively abstract chess concepts, like the isolated queen’s pawn shown in Figure 1, to be clustered together, even if the number or position of other pieces changed from position to position.
Figure 1: an example of an Isolated Queen’s Pawn position
Results
First, in Figures 2 and 3, we can see the accuracy curves for the models trained on 1100-rated games and 2600-rated games, respectively. Both models reached ~40% accuracy before we stopped training them because of signs of plateauing.
Figure 2: Accuracy curves for model trained using 1100-rated games
Figure 3: Accuracy curves for model trained using 2600-rated games
Then, in Figures 4 and 5, we see the embeddings for different positions in each chapter of the book plotted in two dimensions after using TSNE to reduce their dimensionality.
Figure 4: Embeddings for 1100 rated games, graphed
Figure 5: Embeddings for 2600 rated games, graphed We also produced embeddings for positions gathered from Lichess puzzle databases. Theoretically, puzzles should cluster together more easily, since the tactical themes should play a large role in determining what the next move should be in a given position; also these tactical themes are often of a geometrical nature, so it is reasonable to assume that a model may extract such themes from the position for making its decision. In this case, the embeddings should reflect that. However, as seen in Figures 6 and 7, there were no discernable clustering or trends from the Lichess puzzles either.
Figures 6 and 7: Embeddings for puzzles using 1100-trained model and 2600-trained model
From the graphs, we can see that in general, there are no distinguishable clusters or trends for games from the same chapter. It appears as if each position generally gets its own cluster. Therefore, we concluded that different chess concepts were not obviously represented in our embeddings.
Discussion
One potential issue with our project could be that the concepts we chose from the chess book were too abstract. It’s possible that simpler concepts, like few-move tactics (forks, pins, mate in 1, mate in 2, etc.), would have been better clustered by our methods. There are a lot of differences between isolated queen’s pawn positions, for example, that likely added a lot of noise to the embeddings we generated. However, the lack of clustering from Lichess puzzle positions suggests that there wouldn’t have been much clustering from simpler positions either.
This suggests that there may be better methods to learn about these concepts than simply next move prediction. We will consider exploring other embedding methods, such as siamese nets which uses pair of similar / or dissimilar positions to create embeddings, or better pretraining objectives than next move prediction for these models in the future.