Gesture Recognition & Assistive Robot

Project Overview

This pilot study introduces a vision-based assistive robotic system that merges real-time gesture recognition with preliminary cancer detection to support oncology workflows in resource-limited environments. A ResNet50-based model interprets hand gestures, while a second module classifies histopathology images for malignancy detection. Both tasks share a unified CNN backbone to minimize data requirements.

An Arduino-controlled robotic arm adjusts a medical display in response to gesture commands or diagnostic alerts, and the system sends real-time notifications to clinicians.

Technical Summary

• Gesture recognition trained on Jester 20BN + Kaggle datasets
• Histopathology classifier trained on the Breast Histopathology Images dataset
• Shared ResNet50 backbone reduces parameters by 15%
• Real-time robotic actuation controlled via Arduino
• Notification system with <1 second latency

Experimental Setup

Experiments were conducted in a simulated laboratory environment with 10 volunteer participants across 100 trials. The system processed:

• 277,524 histopathology image patches (4× magnification)
• Real-time gesture input captured through bedside cameras
• Robotic arm motions including tilt, zoom, and pan

Results & Performance

• Gesture recognition accuracy: 85%
• Malignancy classification accuracy: 94%
• ROC-AUC for cancer detection: 0.98
• Gesture F1-score: 0.83
• Precision/Recall (malignant): 0.93 / 0.95
• Robotic actuation success rate: 90%
• Notification delivery success: 95% with mean latency of <1s

Data augmentation improved robustness, though low-light conditions reduced accuracy by 5–10%.

Key Contributions

• A unified ResNet-based CNN supporting both gesture recognition and cancer detection
• Gesture-controlled real-time HRI with physical robotic actuation
• Integrated diagnostic feedback + clinician alerts
• Feasibility benchmarks demonstrating operability in resource-constrained scenarios

Conclusion

This work provides a proof-of-concept for an AI-augmented assistive robotic system targeting oncology support. The pilot demonstrates strong technical feasibility with competitive performance across gesture recognition and cancer detection components.

Future Work

• IRB-approved clinical trials with oncology specialists
• Multi-modal interaction (gesture + voice)
• Diverse multi-ethnic histopathology datasets
• Deployment in rural tele-oncology hubs

Technologies Used

PyTorch ResNet50 Computer Vision Robotics Arduino CNN