Project Joseph Hoane Season II - Robotic Chess Player
Autonomous Robotic Chess Player
A fully autonomous robotic chess system that perceives a physical chessboard, computes optimal moves using a chess engine, and executes those moves using an industrial robotic arm - without human intervention
Team Members
Project Information
- Client: Dr. Bowen Weng
- Course: COM S 402C Senior Design
- Semester: Fall 2025
- Project Type: Autonomous Robotic Chess System
Executive Summary
This project implements a complete autonomous robotic chess system that integrates computer vision, chess AI, and robotic manipulation. Using real chess pieces on a physical board, the system detects the board state through vision, computes optimal moves using the Stockfish chess engine, and executes those moves using a UR10e robotic arm. A modular architecture combining ROS, a web-based interface, and MuJoCo simulation enables robust, real-world autonomous gameplay and extensibility for research and education.
System Capabilities
- Real-time physical chessboard perception
- AI-based move computation using Stockfish
- Autonomous pick-and-place execution of chess moves
- Robust recovery from vision and calibration errors
- Integrated simulation and real-world execution
Tech Stack
Operating System:
- Local: MacOS / Windows
- Production: Ubuntu 20.04
Hardware:
- UR10e Robotic Arm
- Robotiq Hand-e (2-arm) gripper
- Logitech WebCam
- Tripod
- Standard Chess Board & Pieces
Software:
- Chess Engine: Stockfish
- Programming Language: Python
- Robot Control: ROS Noetic (ROS 1), rospy
- Vision: OpenCV, YOLO
- Simulation: MuJoCo
Architecture
| Term | Description |
| UR10e | An industrial robotic arm with 6 degrees of freedom. This is the main robot that allows the program to move the chess pieces. |
| Hand-e gripper | A gripper that allows the UR10e to pick up a chess piece. |
| ROS | Acronym for Robotic Operating System. It provides an interface to program the UR10e and the Hand-e gripper. Our system uses ROS Noetic. |
| Chess Board + Pieces | Standard Chess board and pieces |
| Vision System | Camera -> Board Detection -> Homography Transformation -> YOLO Piece Recognition -> Square Mapping -> FEN |
| Webapp | Integrates our Vision System, ROS, Robotic Arm and Chess Engine |
| Inverse Kinematics | Mathematical model to convert 3d coordinates to list of angles for the joints of robots. Transformation is solved using numerical methods. |
The system operates as a closed-loop pipeline connecting perception, decision-making, and actuation:
Camera → Board Detection → Homography Transformation → YOLO Piece Recognition → Square Mapping → FEN Generation → Stockfish → Motion Planning → ROS → UR10e Robot
Core Components
Vision System
Detects the chessboard and pieces from a live camera feed. Homography transforms the camera view into a stable bird’s-eye perspective, YOLO performs piece recognition, and the detected board state is converted into FEN notation.
Web Application
Acts as the central control interface, integrating the vision system, ROS, robotic arm, and chess engine. Provides live board visualization, robot control, and system status.
Inverse Kinematics
Uses numerical optimization methods to convert 3D end-effector positions into joint angles for the UR10e robotic arm, enabling precise pick-and-place motion.
Why This Project Matters
This project demonstrates a comprehensive real-world autonomy pipeline that bridges computer vision, artificial intelligence, and robotics. It highlights challenges encountered outside controlled simulations and presents a robust, modular solution suitable for future research, education, and extension.
Acknowledgments
We would like to thank Dr. Bowen Weng, Dr. Simanta Mitra, Shaiqur Mohammed Rahman (Head of TA), and Scott Song (TA) for their guidance and support, as well as the Computer Science Department for access to robotics laboratory resources.
