Project: Real Steel

Problem

Based on the movie Real Steel released in 2012, our project aims at solving shadow boxing (main reason). Shadow boxing, or better put mimicking human motion, can be used to "Box" or to improve capabilities and functions of autonomous systems in dangerous environments or to complete menial jobs. Our project focuses on Copycat learning, which in the future could be improved to a Reinforcement Learning model better improve the autonomous systems capabilities. 

This project is also loosely based on the Naturalistic Control of Naturalistic Mobility project from Fall 2024.

Project Idea & Functionality

Our projects main goal will allow a Unitree-G1 humanoid robot's upper body to mimic actions by a human (There are also stretch goals including hand tracking and locomotion (walking)). Using the standard communication system for robotics ROS (Robot Operating System) and MediaPipe Pose Estimation (skeleton pose tracking model) our project aims at tracking human movement from video or still frame images and translating them into robot capable motions using inverse kinematics.

Mediapipe skeleton overlaying person

After MediaPipe identifies locations of joints, our project will gather the 'z' axis (depth) from the depth camera for each tracked point. Using ROS, our system will publish those points to display with MatPlotLib (debugging) and a inverse kinematics ROS Node which will then update the robot to move to that position. (See the following chart for a more in-depth flowchart Flowchart of system

Note: While we will not always have access to the humanoid robot, we will also work on a MuJoCo simulation environment.

The main deliverable requested over the course of this project was CSV file containing all used joint positions timestamped every .100 ms (10hz). To ensure safety of robot, a speed limit was imposed on our system. Our self imposed speed limit was 1 radian per second or roughly 57 degrees per second. To prevent breaking the speed limit, point interpolation was developed in both the robot_csv node and simulation to visualize what was happening. A summary of how the problem was solved is as follows:

  1. Check if difference between previous frame and current frames positions are within 0.1 rad/100ms (5.73 deg/100 ms). (1 rad/sec = 0.1 rad/100ms)
  2. If difference exceeds limit, divide difference by limit to get number of added frames: difference / 5.73
  3. Insert frames between data points smoothly transitioning between positions

An example of the solution can be seen below.

Frame interpolation example for Simulator and CSV

The full working system can be seen here (Note: the developed system was only tested in simulation and not on physical robot):