One challenge towards autonomous manipulation is to cope with uncertainty and noise in the sensory-motor system of the robot and the environment. We argue that such a system requires to close feedback control loops in novel ways, using predictive models and leveraging previous experiences.

Manipulation tasks often decompose into a sequence of skills, where each skill can be seen as a stereotypical movement, with respect to its goal frame. Assuming that movement dictates sensory feedback, sensory traces can be associated with the movement primitive (e.g. encoded as DMPs), forming Associative Skill Memories (ASMs) [].

ASMs allow to learn predictive models, which can be used to cope with uncertainty and noise in the perceptual system of the robots. ASMs have been successfully used to address several important issues towards an autonomous manipulation platform.

• Having a general skill library which allows to e.g. grasp an object robustly, the sequence of skills has to be determined based on the sensory feedback of the execution. [] showed how to incrementally determine a correct skill sequence despite perceptual errors using ASMs.
• Errors both in the perceptual and action system are inevitable. Predictive models learned from ASMs allow to predict failures in an online fashion [].
• For robust manipulation errors have to be detected online and possible successor skills selected in real time. In [] this problem is formulated as a data-driven online decision making problem, integration visual, acustic, and haptic sensors, as well as the system state to make real time decisions using the sensory information provided by the ASMs.

Interesting open research questions related to ASMs and manipulations are: Automatic skill decomposition; Learning good representation from unsupervised sensory information; Better failure incorporation.

[]

[]