Tanmay Shankar

tanmayshankar [at] cmu [dot] edu

I'm a Research Scientist at The AI Institute! I recently finished my Ph.D. at the Robotics Institute at Carnegie Mellon University, where I work with Jean Oh.

My Ph.D. research aimed to learn and translate temporal abstractions of behaviors (such as skills) across humans and robots. Here's a recording of my recent proposal talk for details, and here's a description of my research interests and philosophy!

I'm interested in solving the human-to-robot imitation learning problem, particularly by building temporal abstractions of behavior across both humans and robots. Much of my work adopts a representation learning perspective to this problem, borrowing ideas from unsupervised learning, machine translation, and probalistic inference. I strongly believe in such interdisciplinary research; for example, my past work has made connections between cross-domain imtation learning and unsupervised machine translation, between value iteration and neural network architectural components, and more.

I believe much of my research is not only applicable to robots, but also to dextrous prosthetic hands. I am passionate about how I can explore prosthetics as an application domain. This stems from a broader interest in the potential for assistive and rehabilitational robotics, an area I have been passionate about since my undergrad.

Before my Ph.D., I was a research engineer at Facebook AI Research (FAIR), Pittsburgh for 2 years, where I worked on unsupervised skill learning for robots with Shubham Tulsiani and Abhinav Gupta. Before FAIR, I also did my Masters in Robotics from the Robotics Institute, working on differentiable imitation and reinforcement learning with Kris Kitani and Katharina Muelling. Here's more of my academic history.

In addition to working full time at FAIR, I returned to FAIR Pittsburgh during a summer of my Ph.D., working with Yixin Lin, Aravind Rajeswaran, Vikash Kumar, and Stuart Anderson on translating skills across humans and robots. Before my MS, I did my undergrad at IIT Guwahati, where I worked on reinforcement learning networks, with Prithwijit Guha and S. K. Dwivedy. Before I worked on robot learning, I used to work on assistive technology - an area I'm also passionate about. During my undergrad, I also spent summers working with Mykel Kochenderfer at the Stanford Intelligent Systems Lab, and with Howie Choset at the Biorobotics Lab at CMU.

CV  /  Google Scholar  /  Github  /  LinkedIn


Updates

[Nov '24]  

I started as a full-time Research Scientist at the AI Institute!

[Aug '24]  

I succesfully defended my thesis on August 29th! Here's a recording of my talk!

[Jul '24]  

Our paper on TransAct was accepted to IROS 2024 as an Oral! Check out the paper here! and our results here!

[Mar '24]  

Check out our real robot results on translating agent-environment interactions here!

[Mar '24]  

Submitted our work on translating agent-environment interaction abstractions to IROS 2024!

[Oct '23]  

I'm collaborating with the New Dexterity group from the University of Auckland on a new project!

[Dec '22]  

Presented my work on learning agent-environment interaction abstractions at the workshop on aligning human-robot representations at CoRL 2022.

[Nov '22]  

Successfully passed my Ph.D. thesis proposal! Here's a recording of my talk!


 

Robot Learning Research

Translating Agent-Environment Interactions across Humans and Robots
 
T. Shankar, C. Chawla, A. Hassan, J. Oh
 
Accepted at Intelligent Conference on Intelligent Robots and Systems, IROS 2024 (Oral)
 
Paper Website / Code / Video
 

Building on my previous skill learning work (ICML 2020) and translation work (ICML 2022), I developed TransAct, a framework to first learn abstract representations of agent-environment interactions, and then translate interactions with similar environmental effects across humans and robots. TransAct enabled zero-shot, in-domain transfer of complex, compositional task demonstrations from humans to robots.

Translating EMG Control Signals to Dextrous Robot and Prosthetic Hands
 
T. Shankar, B. Guan, R. Vilela, M. Liarkopis, J. Oh
 
In Preparation
 
 

Inspired by the success of my work translating skills across human and robot arms, along with collaborators from the University of Auckland, I'm exploring whether we can apply equivalent strategies to translating EMG signals to control dextrous robot and prosthetic hands.

Spline-FRIDA: Spline-FRIDA: Enhancing Robot Painting with Human Brushstroke Trajectories
 
L. Chen, L. Coleman, P. Schaldenbrand, T. Shankar, J. Oh
 
To be Submitted to Robotics and Automation Letters, RA+L 2024
 
 

Inspired by my previous work on representation learning for skill learning, and Peter Schaldenbrand's prior work on FRIDA the robot painter, together with Lawrence Chen, we are exploring whether building learnt representations of paint strokes would facilitate learning new types of paint strokes beyond ones the robot is preprogrammed with, to improve FRIDA's artistic expression.

Learning Abstract Representations of Agent Environment Interactions
 
T. Shankar, J. Oh
 
Aligning Human and Robot Representations Workshop, Conference on Robot Learning, December 2022
 
Paper / Website / Code / Video
 

Inspired by the success of my work learning representations of robot skills, I'm exploring whether we can apply equivalent machinery to learning temporal abstractions of environment state. In particular, I hope to learn representations of patterns of motion of objects in the environment, or patterns of change of state.

Translating Robot Skills: Learning Unsupervised Skill Correspondences across Humans and Robots
 
T. Shankar, Y. Lin, A. Rajeswaran, V. Kumar, S. Anderson, and J. Oh
 
International Conference on Machine Learning, July 2022
 
Paper / Talk / Website / Code
 

We developed an unsupervised approach to learn correspondences between skills across humans and various morphologically different robots, taking inspiration from unsupervised machine translation. Our approach is able to learn semantically meaningful orrespondences between skills across multiple robot-robot and human-robot domain pairs, despite being completely unsupervised.

Translating Dextrous Manipulation Skills across Human and Robot Hands
 
T. Shankar, A. Hassan, J. Oh
 
In Preparation
 
Code
 

Inspired by the success of my work translating skills across human and robot arms, I'm exploring whether we can apply equivalent strategies to translating dextrous manipulation skills across human and robot hands.

Learning Robot Skills with Temporal Variational Inference
 
T. Shankar, A. Gupta
 
International Conference on Machine Learning, July 2020
 
Paper / Talk / Website / Code
 

We presented an unsupervised approach to learn robot skills from demonstrations. We formulated a temporal variational inference, to learn robot skills from demonstrations in an entirely unsupervised manner, while also affording a learnt representation space of skills across a variety of robot and human characters.

Discovering Motor Programs by Recomposing Demonstrations
 
T. Shankar, S. Tulsiani, L. Pinto, A. Gupta
 
International Conference on Learning Representations, April 2020
 
Paper / Talk / Website / Code
 
Learning robot skills from demonstrations using a temporal alignment loss to recompose demonstrations from skills.
Learning Neural Parsers with Deterministic Differentiable Imitation Learning
 
T. Shankar, N. Rhinehart, K. Muelling, K. Kitani
 
Conference on Robot Learning, CoRL, October 2018
 
Paper / Talk / Code
 
Learning image parsers by imitating ID3 style decision tree oracles, using differentiable variants of imitation learning.
Reinforcement Learning via Recurrent Convolutional Neural Networks
 
T. Shankar, S. K. Dwivedy, P. Guha
 
International Conference on Pattern Recognition, ICPR, Cancun 2016
 
Paper / Talk / Code
 
Representing classical computations in Markov Decision Processes within architectures of Recurrent Convolutional Neural Networks.

 


Assistive Technology Research

A Hybrid Assistive Wheelchair Exoskeleton
 
T. Shankar, S. K. Dwivedy,
 
International Convention on Rehabilitation Engineering and Assistive Technology, i-CREATe 2015
 
Paper / Talk
 
Designing and prototyping a hybrid wheelchair exoskeleton for assisted mobility.
Development of an Assistive Stereo Vision System
 
T. Shankar, A. Biswas, S. K. Dwivedy,
 
International Convention on Rehabilitation Engineering and Assistive Technology, i-CREATe 2015
 
Paper / Talk
 
Designing and prototyping an assistive vision system for blind individuals.

Other Projects

Hierarchical Reinforcement Learning for Sequencing Behaviors
 
H. Salman, P. Singhal, T. Shankar, et. al.
 
Course Project, Deep Learning, CMU
 
Paper Video 1 Video 2
 
Using hierarchical reinforcement learning to sequence predefined primitives.
Learn Vector Policy Fields for Continuous Control
 
T. Shankar,
 
Course Project, Deep Reinforcement Learning, CMU
 
Paper / Video
 
Exploring how reinforcement learning networks can be applied to continuous quadrotor control.
Cooperative Vision based Collision Avoidance for Unmanned Aircraft
 
E. Muller, T. Shankar, M. Kochenderfer
 
Internship Project, Stanford Intelligent Systems Lab
 
Paper / Video
 
Applying visual SLAM to quadrotors, to enable them with cooperative collision avoidance.
Localization for an Interior Wing Assembly Mobile Platform
 
T. Shankar, L. Li, H. Choset,
 
Summer Internship Project, CMU, 2014
 
Video
 
Using AR Tag based localization for an interior wing assembly platform.

Last updated: Jan 2023

Imitation is the highest form of flattery