Jetson Project of the Month “LiveChess2FEN” is a fully-functional framework that automatically digitizes a chessboard in real time using NVIDIA Jetson Nano.
American chess grandmaster and the eleventh World Chess Champion Bobby Fischer once said, “All that matters on the chessboard is good moves.”
Easy for him to say. But, if you are new to the game or looking to get better, how can you know that the moves you are making are in fact good ones? Wouldn’t it be great if you could analyze your game, play by play?
To address this problem, the latest Jetson Project of the Month winners developed LiveChess2FEN—a framework that automatically digitizes the configuration of a chessboard in real time—optimized on an NVIDIA Jetson Nano.
“This is the first attempt at deploying a chess digitization framework onto an embedded platform, obtaining similar if not better execution times than other approaches, which at least used a midrange laptop,” LiveChess2FEN creators David Mallasén Quintana, Alberto A. Del Barrio, and Manuel Prieto-Matias state in their paper.
LiveChess2FEN digitizes a chess game using the Forsyth–Edwards Notation (FEN), which is a standard notation using letters and numbers that represent a particular board position during a chess game. The project was created for amateur players or tournaments, where a webcam watches the board continuously and checks if the position of any of the pieces on the board has changed. The video is streamed through the learning models and returns any changing board position as a new FEN string.
In the NVIDIA Developer Forum, the developers state, “Previous work has shown promising results, but the recognition accuracy and the latency of state-of-the-art techniques still need further enhancements to allow their practical and affordable deployment.”
Additionally, current hardware solutions such as specialized boards or robotic arms can be expensive and difficult to deploy. Using the Jetson Nano, the team found a way to address these shortcomings.
The problem can be divided into two main parts. The first is recognizing the chessboard and its orientation, and then identifying the chess pieces and their precise position afterward.
To tackle the challenge, the team used a variety of software, including Keras API on top of TensorFlow to define and train the convolutional neural networks (CNNs) to test piece classification. To train their models, they used two labeled datasets of chess pieces that consisted of nearly 55,000 images.
They also used ONNX Runtime, TensorRT, ONNX-TensorRT, and several Python libraries. To help monitor compute resource usage on a Jetson Nano, they used the system monitoring utility jtop.
Using the Jetson Nano and optimizing the CNNs with ONNX Runtime and TensorRT they reduced the inference latency from 16.38 seconds down to 3.84 seconds.
Adding domain-specific knowledge to the inference of the models provided an edge when it came to accuracy. By testing different CNNs on TensorRT, the team achieved latency times ranging from 6 seconds down to 0.46 seconds. There were some tradeoffs between accuracy and speed, which are shown in the graph below.
The team then devised a way to check if the pieces on the chessboard are in the same location by calculating if the grid of squares lines up with what the AI previously detected. This helps avoid constant recalculations based on observing the board, in order to determine the coordinates of the chess pieces more quickly. By checking if the chessboard pieces moved since the last inference, which takes 150 ms, the team saw a massive improvement in inference times.
In addition to a dedicated NVIDIA GPU, the system integrates a quad-core ARM CPU capable of performing the sequential computation necessary to detect the chessboards. This type of architecture has been widely used successfully for more than a decade in tasks related to image processing.
The result is a framework that can automatically digitize a chess position from an image in less than 1 second, with an accuracy of 92% classifying a piece, and 95% detecting changes on the board.
The source code of the LiveChess2FEN framework is available with an open-source license in GitHub.