Move 1
Move 2
Move 3
Move 4

¹Robots are used in human environments today more than ever before. ²Examples of such robots include simple vacuum cleaning or lawn mowing robots. ³Although robots have become increasingly popular, they are still mainly used in industrial settings as welders and assemblers. ⁴The reason for robots excelling in industrial settings is that the environment is for the most part static and that the task at hand is known a priori. ⁵For this reason, the robot can be preprogrammed to precisely follow specific trajectories.

Move 1
Move 2
Move 3
Move 4

⁶However, in a constantly changing environment, such as in our homes, or with complex tasks, such as playing ping pong [61], the robot cannot be preprogrammed as before.

Move 1
Move 2
Move 3
Move 4

⁷Thus, for a robot to learn in these situations, one possible and frequently used solution is to learn from a human demonstrator. ⁸The concept of having a human teach the robot by interacting with it is known as Learning from Demonstration (LfD), as well as Programming by Demonstration (PbD) or imitation learning. ⁹LfD has enabled robots to carry out complex tasks, such as playing the ball-in-a-cup game [50], playing ping pong [61], as well as many other tasks [10,63]. ¹⁰For robots to acquire a movement, they first and foremost must record the demonstrated movement by using either their internal sensors or an external monitoring system. ¹¹Then, the responsibility is shifted over to the robot, which in turn is expected to learn a reasonable representation of the movement from the recorded information. ¹²This learning is achieved through the guidance of a learning algorithm, which must be able to produce a representation of the movement as well as cope with perturbations without risking failure.

Move 1
Move 2
Move 3
Move 4

¹³LfD has attracted significant attention in recent years, with the number of papers written on the subject serving as a significant indicator [5,60,64]. ¹⁴Several reasons have been identified for this increased interest [3,53]. ¹⁵The first reason is the reduced learning time compared to traditional approaches that require a human to program the robot off-line. ¹⁶Another reason is that by demonstrating the movement to the robot, the user can also predict the behaviour in advance before the robot executes the learned skill, as this should follow the demonstrated movement quite accurately. ¹⁷This, in turn, increases safety and is exceptionally important when integrating robots into human environments. ¹⁸Finally, because the acquired movement is in line with the human workspace, critical parts of the movement can be modelled very accurately.

Move 1
Move 2
Move 3
Move 4

¹⁹Despite all the advantages of LfD, problems still remain concerning how to improve and generalize the learned skill. ²⁰For example, learning from a demonstration has been shown to not be enough to master complex tasks, such as flipping a pancake [52] or playing the ball-in-a-cup game [50]. ²¹Hence, to really master a learned skill requires subsequent practice.

Move 1
Move 2
Move 3
Move 4

²²A method for improving a skill through practice has already been implemented in robotics and is known as Reinforcement Learning (RL), in which the robot tries to improve the learned movement by interacting with the environment and receiving responses in the form of rewards [48]. ²³Based on these rewards, the learned movement is adapted to increase future rewards. ²⁴This resembles the way that humans acquire a new skill: first, we imitate the skill performed by another person, and then we subsequently practise the skill in order to fine-tune the movement until we can master it.

Move 1
Move 2
Move 3
Move 4

²⁵Although reinforcement learning has become the standard approach for improving a taught skill, one problem still remains and that is how to safely explore new trajectories.

Move 1
Move 2
Move 3
Move 4

²⁶To explore new trajectories, noise is typically added to the learned representation of the skill, resulting in a slightly new trajectory.

Move 1
Move 2
Move 3
Move 4

²⁷However, generating the noise is not a trivial task, and problems, such as the magnitude of the generated noise and whether the noise should be correlated, still remain an open question.

Move 1
Move 2
Move 3
Move 4

²⁸Consequently, various solutions have been proposed for generating noise. ²⁹For example, in [50] and [49], the generated noise is dependent on the learned model.

Move 1
Move 2
Move 3
Move 4

³⁰Intuitively, this makes sense; however, it does not take into consideration the constraints of the executing system, which in both cases is a robotic arm.

Move 1
Move 2
Move 3
Move 4

³¹Although recent studies [44] have generated noise focusing on the constraints of the actual system, no studies have yet attempted to apply this to improving a learned model represented as a Dynamic Movement Primitives (DMP).

Move 1
Move 2
Move 3
Move 4

³²Hence, this thesis will develop a method to safely explore new trajectories for a skill modelled as a DMP.

Move 1
Move 2
Move 3
Move 4

³³To accomplish this objective, the thesis must carry out the following tasks:

• model the skill as a DMP,
• select an RL algorithm for improving the modelled skill,
• evaluate the safety of the exploration method.

³⁴To test the effectiveness of the new approach for generating safe trajectories works, a challenging enough real-world task is needed for experimentation. ³⁵For this purpose, the thesis will use the ball-in-a-cup game, as this game can be seen as a benchmark problem in robotics [50]. ³⁶The game consists of a ball attached to a string, which is then attached to the bottom of a cup. ³⁷The objective of the game is to get the ball into the cup. ³⁸This is achieved by inducing a movement on the ball that quickly moves the cup back and forth and then pulls it up and moves the cup under the ball. ³⁹Although the game is simple by nature, it is not that easy to learn, as it requires fast, precise movement, and even small changes in the movement can have a drastic impact on the trajectory of the ball. ⁴⁰The ball-in-a-cup game is of interest, as a robot cannot get the ball into the cup by merely executing the movement produced by the initially learned model, thus requiring subsequent RL to successfully master the skill.

Move 1
Move 2
Move 3
Move 4

⁴¹The rest of this thesis is organized as follows. ⁴²Chapter 2 introduces and compares several state-of-the-art LfD approaches. ⁴³Based on the results, DMP is chosen to model movements. ⁴⁴Chapter 3, in turn, is devoted to explaining DMP in more detail. ⁴⁵As reinforcement learning is used to improve the learned movement, Chapter 4 is dedicated to explaining RL. ⁴⁶The chapter first explains the theory behind RL and then introduces state-of-the-art algorithms. ⁴⁷Chapter 5 introduces the testbed, the hardware, as well as already implemented and newly developed software for supporting the experimental part of this thesis. ⁴⁸The experiments and results are presented in Chapter 6. ⁴⁹These are also critically discussed in the same chapter. ⁵⁰Finally, conclusions and suggestions for future work are presented in Chapter 7.