UC Berkeley Robotics / Vision / PINN / Webots
This project studies a complete simulated pick-and-place pipeline for a UR5e manipulator: a Webots camera produces object observations, a trained visual regressor estimates the target location, and a physics-informed neural network predicts the six joint angles needed to reach the object.
The core engineering question is whether a neural IK solver can be made useful for robotics by constraining it with forward kinematics, instead of treating inverse kinematics as a purely black-box regression problem.
I redesigned this page as a technical case study: hypothesis, data, model, physics constraint, benchmark evidence, and limitations, using real project captures and data-driven figures.
Problem
A UR5e pose can correspond to several joint configurations. A naive neural regressor can average incompatible branches and output a joint vector that is numerically smooth but geometrically wrong.
Hypothesis
By passing predicted joints back through a differentiable FK model, the network is trained to minimize not only joint error, but also end-effector position error in Cartesian space.
Contribution
The work connects dataset generation, visual target estimation, neural IK training, PyTorch physics losses, and a Webots controller benchmarked against IKPY.
The vision part is based on rendered Webots observations and pixel labels. The published artefacts in the project folder show a VGG16-based visual regressor trained to predict object coordinates from images.
Dataset evidence
The dataset contains RGB frames and a CSV file with \((x_{pixel}, y_{pixel})\) targets. The model code loads the image tensor as \(1000 \times 256 \times 256 \times 3\), normalizes it, and splits it into 800 training images and 200 validation images.
Vision model
The project notebook uses a frozen VGG16 image backbone and replaces the classification head with regression layers. The output is not a class label; it is the object location in the camera image.
The PINN maps a Cartesian target to a 6-DOF joint vector. The scientific part is the loss: the predicted joint vector is evaluated through the UR5e forward-kinematics equations during training.
Camera or simulation gives the desired end-effector position.
x_target = (X, Y, Z)
The MLP predicts the six UR5e joint angles.
q_pred = fθ(x_target)
PyTorch FK reconstructs where the predicted joints actually place the tool.
x_hat = FK(q_pred)
The residual penalizes Cartesian geometry error.
||x_hat - x_target||²
PINN objective
The implementation has two training routes: a 6-DOF dataset generated through IK sampling, and a “true physics” variant that computes the FK residual directly in PyTorch. Both keep the same principle: the joint prediction must be numerically close to data and physically coherent in Cartesian space.
6-DOF PINN route
The training script samples 120,000 candidate Cartesian targets in the UR5e workspace, keeps valid IK solutions, and trains a \(3 \rightarrow 6\) MLP with 256-neuron hidden layers, SiLU activations, batch size 256, and 60 epochs.
True physics route
A second script generates 25,000 Webots table targets and trains a 512-hidden-unit model with a differentiable UR5 FK module. It reports physics error in millimetres, joint-angle MSE, and learning-rate updates during training.
The figures below are intentionally simple: one curve from an archived training log and one timing chart from the Webots comparison. They make the page more scientific without pretending that every experiment was already packaged like a final paper.
Training trace
The archived prototype log shows loss and mean FK error decreasing across 30 epochs. The later 6-DOF pipeline is more rigorous, because the code explicitly weights the physics loss and validates FK error on held-out samples.
Webots benchmark
The comparative world places the neural solver beside an IKPY analytic baseline. The Webots HUD logs sub-millisecond compute times while the robot follows the target. This is the practical robotics test: not just “does the network train?”, but “can it sit inside the controller loop?”.
These are the concrete pieces that make the project reproducible and credible as engineering work rather than a decorative AI demo.
Generated camera observations and object-center coordinates for the learning pipeline.
Transfer-learning model mapping \(256\times256\) RGB images to two pixel coordinates.
MLP trained on valid IK samples with a strong forward-kinematics consistency term.
Differentiable FK converts predicted joints back to Cartesian tool position.
Controller executes approach, grasp, lift, and placement trajectories.
Analytic IK baseline kept in the loop to compare speed and qualitative behavior.
What this page now emphasizes
The project reads as a robotics study: it shows where the data comes from, how the vision model was trained, why the PINN is physically constrained, what equations define the loss, and how the solver is exercised in Webots against IKPY.