F A C U L T Y O F S C I E N C E U N I V E R S I T Y O F C O P E N H A G E N PhD thesis Morten Pol Engell-Nørregård Interactive Modelling and Simulation of Human Motion Academic advisors: Kenny Erleben Kim Steenstrup Pedersen Submitted: june/01/2012 I’ve got it! But it only works for spherical chickens in a vacuum. III Prologue Why would I start something as serious as my PhD thesis with the punchline from a joke. Well first because it succinctly describes one of the major difficulties of physical modelling, the need for simplification. Secondly but equally important because, as the danish poet Piet Hein put it: Den, som kun tar spøg for spøg og alvor kun alvorligt, han og hun har faktisk fattet begge dele dårligt. Which can be translated as Taking fun as simply fun and earnestness in earnest shows how thoroughly thou none of the two discernest. For me, what this means is that as important as it may be to acquire new knowledge, the motivation and drive will always be the enjoyment of the process. I am glad to say, that even though it may have been hard at times, I have thoroughly enjoyed myself in the 3 years I have spent as a PhD student at DIKU. The opportunity to pursue the goals that I set up myself is something that makes me feel very privileged. Returning to the Punchline, I managed to find the time to see several episodes of the series Big bang theory which tells the story of a bunch of friends who also happens to be scientists. One episode contains a joke, told by one of the characters, about a farmer who has a problem with sick chickens. The Farmer asks a friend who is a physicist to help him and the friend starts scribbling maniacally in his notebook until finally he exclaims "I’ve got it! But it only works for spherical chickens in a vacuum." This exemplifies the other part of the verse above. Even though something is Funny it can contain valuable information. In this case, the people who usually find this joke hilarious are people working in science, since they get the more subtle implication that, when modelling real world phenomena we are always forced to simplify, and the choice of simplification is a very important one indeed. My work has been full of situations where, I have had to choose which part of the real human motion I wanted to include in my models, and which to exclude. In this process, the applicability of the IV end result has always been important to me. Whether I have had Graphics applications or bio-mechanical models in mind. Therefore I hope, when you have read this text, you will agree with me that this work is not only suitable for Spherical chickens in a vacuum. V Acknowledgement When I started at the university some years ago I came from a background as an artisan, a goldsmith to be concise. My highschool math was all but forgotten and my programming experience amounted to my experience as a kid, copying basic programs from books unto my commodore 64. I owe great thanks to the many people who, have helped me attain the knowledge and skills, which made it possible for me to get this far. Also all the people who have supported me on a more personal level. I want to start by thanking my many mentors. Kenny Erleben, my primary supervisor for always sharing the enthusiasm for my weird ideas and always beating me to it, with even wilder ideas. Always helping me when the math made my head hurt, and making my head hurt with math when it didn’t. Being not only a good supervisor but a good friend. Kim Steenstrup Pedersen my co supervisor for teaching me that there are other research fields and other ways of solving the problem, than the physically based simulations. Knud Henriksen for initially sparking my interest for graphics, and Mathias Teschner for taking good care of me during my stay in Freiburg. I would also like to take the opportunity to thank all my colleagues at the e-science institute. They are legion and I could not possibly mention them all but I must mention at least Søren Hauberg and Sarah Niebe with whom I have wasted more time than I care to think about, exchanging irrelevant information or plain out gibberish. All my office mates have partaken in this (whether they liked it or not) but none more than these two. During my studies I have used several open-source tools to help me visualize my work. I would like to acknowledge the great work of the teams behind: Gimp, Inkscape and most of all the Blender team. You are doing a great job out there. Even Though it may not always seem like it, there is a life outside the PhD project. Without the love and support of my family and friends I would have never had the energy to get through this process. In particular I want to thank my wife and kids who bore the brunt of my absence and absent mindedness, when the going got tough. Finally I want to thank the committee for taking time from their busy schedules to read and give feedback to my work Thank you all! VI Notation Here follows a short, incomplete list of the notation used in this Thesis. Most of the notation is explained as it is introduced but I felt it was justified to mention some of the most common notations which may be used differently elsewhere. x is the vector x M is the matrix M || ∗ || is the euclidean norm ∇ is used to denote the gradient usually in the form ∇f L is the Lagrangian function J denotes the Jacobian matrix H denotes the Hessian matrix ẋ In relation to governing equations of motion, we sometimes use the physicist way of writing the time derivative. A note on visual content With a few exceptions all figures and 3d models are made by me in the course of my studies. I have used Blender [1] for most of the 3d work and inkscape and gimp for 2D images. This was done mainly to have access to content without having to worry about licensing. I have however used a few things that are not my own work. Of note, is the lizard used in the spline activation (see Figure 27) which I found on the web. It is made by Kevin Hayes and kindly released under a creative commons license . Also the female in Figure 1 is a mesh from the Ogre framework [2] In relation to my publications I have made media content in the form of videoclips. Since My topic is Motion I think some of the arguments are best described using these videos. They are available at http://iphys.wordpress.com/. VII Summary This PhD thesis concerns itself with modelling and simulation of human motion. The research subjects in this thesis has at least two things in common. First they are concerned with Human Motion. Even though the models may be used for other things as well, the main focus is on modelling the human body. Second they are all concerned with simulation as a tool to synthesize motion and thus, get animations. This is an important point since it means we are not only creating tools for animators to make fun and interesting animations, but also models for simulation of realistic motion. As the project progressed the focus has shifted from purely graphics oriented, to something which may be at least as interesting for the biomechanics community. Scientific contributions The main scientific contributions of this work are: • An efficient method for solving interactive constrained inverse kinematics problems , using an optimization based approach. The method is usable for motion planning of complex articulated mechanisms, with a large degree of interdependency between different parts, such as a human body. • A general and fast joint constraint model. The joint constraint model is well suited for modelling joints with highly non-convex limits and multiple degrees of freedom. Even though this constraint model may have many other uses we believe it is very well suited for the modelling of human joints which exhibit both non-convexity and multiple degrees of freedom • A general and versatile model for activating soft bodies. The model may be used as an animation tool but would be equally well suited for simulation of human muscles since it adheres to basic physical principles. Further, it can be used with any softbody simulation method such as finite elements or mass spring systems. • A control method for deformable bodies based on the space time optimization. the method may be used to control the contraction of muscles in a muscle simulation. VIII Dansk resumé Denne ph.d.-afhandling beskæftiger sig med modellering og simulation af menneskelig bevægelse. Emnerne i denne afhandling har mindst to ting til fælles. For det første beskæftiger de sig med menneskelig bevægelse. Selv om de udviklede modeller også kan benyttes til andre ting,er det primære fokus på at modellere den menneskelige krop. For det andet, beskæftiger de sig alle med simulering som et redskab til at syntetisere bevægelse og dermed skabe animationer. Dette er en vigtigt pointe, da det betyder, at vi ikke kun skaber værktøjer til animatorer, som de kan bruge til at lave sjove og spændende animationer, men også modeller til simulering af realistiske bevægelser. I løbet af projektet er fokus flyttet fra at være rent grafik orienteret , til noget som er mindst lige så interessant for biomekanikere. Videnskabelige bidrag De vigtigste videnskabelige bidrag af dette arbejde er: • En effektiv metode til at løse interaktive, inverse kinematik problemer med ledbegrænsninger, ved hjælp af en optimerings tilgang. Metoden er anvendelig til bevægelses planlægning for komplekse ledmekanismer, med en høj grad af indbyrdes afhængighed mellem de forskellige dele, såsom et menneskes krop. • En generel og hurtig led-begrænsning model. Denne led-begrænsning model er velegnet til modellering af led med særdeles ikke-konvekse grænser og flere frihedsgrader. Selv om denne led-begrænsnings model kan have mange andre anvendelser,mener vi, den er meget velegnet til modellering af menneskers led, der udviser både ikke-konveksitet og flere frihedsgrader • En generel og alsidig model for aktivering af bløde legemer. Modellen kan anvendes som et animations værktøj, men er lige så velegnet til simulering af menneskelige muskler, da den opfylder de grundlæggende fysiske principper. Endvidere kan den anvendes med enhver softbody simuleringsmodel som finite elements eller mass spring systemer. • En kontrol metode til deformerbare legemer baseret på rum tids optimering. fremgangsmåden kan anvendes til at styre sammentrækning af muskler i en muskel simulering. IX Contents I Inverse Kinematics 6 1 Constrained Inverse Kinematics 1.1 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . 7 9 11 2 The Traditional Approach 12 3 A Non-Linear Optimization Approach 16 4 Projected Gradient 4.1 Computing the Gradient . . . . . . . . . . . . . . . . . . . . 4.2 Finding a Step-Length . . . . . . . . . . . . . . . . . . . . . 4.3 Projected Armijo Back-Tracking Line-Search . . . . . . . . . 18 18 19 20 5 A Geometric Approach to the Differential 5.1 A Ball and Socket Joint . . . . . . . . . . . . . . . . . . . . . 21 24 6 Performance of Projected Back-Tracking Line-Search 6.1 Comparisons . . . . . . . . . . . . . . . . . . . . . 6.2 Test Framework . . . . . . . . . . . . . . . . . . . . 6.3 Visual Quality . . . . . . . . . . . . . . . . . . . . . 6.4 Robustness . . . . . . . . . . . . . . . . . . . . . . 6.5 Generality . . . . . . . . . . . . . . . . . . . . . . . 26 26 27 28 29 29 . . . . . . . . . . . . . . . . . . . . . . . . . 7 Alternatives to Gradient descent 29 8 An Application in motion tracking 8.1 Motion Estimation . . . . . . . . . . . . . . . . . . . . . . . 29 30 9 Visual Motion Estimation 9.1 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 32 10 Discussion and Conclusion 33 II 35 Joint Constraint Models 11 Joint constraints using signed distance fields 1 36 12 Introduction 36 13 Related Work 37 14 The Signed Distance-Field Model 14.1 Building the Signed Distance-Map . . . . . . . . . . . . . . . 14.2 Adaptive Sampling of Motion . . . . . . . . . . . . . . . . . 14.3 The Signed Distance Property . . . . . . . . . . . . . . . . . 14.4 Sculpted Signed Distance-Field Joint-Constraints . . . . . . 14.5 Applying Signed Distance-Field Joint-Constraints for Inverse Kinematics . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 41 43 43 44 15 Results and Discussion 15.1 Building the signed distance-maps . . . . . . . . . 15.2 Parametrizing the Distance-Field Joint-Constraints 15.3 Constant Time Performance . . . . . . . . . . . . . 15.4 Increased Modelling Accuracy . . . . . . . . . . . . 46 46 46 47 47 . . . . . . . . . . . . . . . . . . . . 45 16 Conclusion 47 III 50 Activation splines 17 Activation splines 51 18 Introduction 51 19 Related Work 52 20 The Activation Spline Method 20.1 Discretization of the Activation Spline . 20.2 Embedding the Activation Spline . . . . 20.3 Computation of Activation Spline Forces 20.4 The Force Equivalence Principle . . . . . 20.5 Contact Forces . . . . . . . . . . . . . . . . . . . 56 58 58 59 60 64 21 Validation 21.1 Performance measurements . . . . . . . . . . . . . . . . . . 65 67 22 Conclusion and Future Work 68 IV 70 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Optimal control using space time optimization 2 . . . . . 23 Introduction to Space Time Optimization 71 24 Previous Work 71 25 The Space Time Constraint Problem 25.1 The Adjoint Method . . . . . . . . . . . . . . . . . 25.2 Computing the Gradient using the Adjoint Method 25.3 Exploiting the Sparse Block Matrix Patterns . . . . 25.4 Deformable Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 74 75 76 78 26 A Simple 2D Particle Example 80 27 A 2 Particle Example in 2D 80 28 Results 82 29 Conclusion on space time optimization 83 V 84 Concluding remarks 30 Conclusion 85 31 Future Work 85 32 Epilogue 87 3 Introduction This Thesis describes the work I have done over the past 3 years on Modelling and simulating human motion. With such a broad area it is necessary to impose some restrictions on the subject. I have chosen to describe human motion as two steps, the planning phase and the execution phase. In the planning phase, the wanted motion is chosen from the space of possible motions. Essentially this means that a number of goals are converted into a pose. A simple example is taking a step. If a person wants to take a step, we can pose the problem as moving one foot from position a to position b while clearing the ground This way of deciding a motion based on position and/or orientation of a subset of the joints is an intuitive and easy way of specifying motion which appeals to the way humans think about motion. This can be seen by the fact that this is the way animators and robot controllers work. In the execution phase the actual forward dynamics simulation of the planned motion should be performed. here I have chosen to focus on the motor of human motion namely the Muscles. I have done this through the development of a deformable spline activation model suitable for simulation of simple muscles and with the added advantage of the passive deformable mesh which is of interest both from a visualization point of view and for collision detection the strength of the method is its ability to be combined with the deformable mesh of choice for a given simulation . Finally I have recently started exploring the possibilities for using the motion planning to control the simulation that is use the poses calculated using the inverse kinematics and joint constraints to drive the motion of the dynamically simulated mesh. This is still work in progress but I have chosen to include it since it represents the connection between the two other parts of the project and since it is also the most recent research activity I have 4 undertaken. The Research presented in this thesis is to a large extend identical to the results presented in my published papers, but I have taken the opportunity to elaborate on some of the subjects which had to be left out or shortened in the publications due to page restrictions. I have rewritten and reformatted the papers to fit into the context of this thesis.That is, I present the research as one text combined from the papers, but as much as possible, I have kept the original thoughts and arguments unchanged. The Papers, on which this thesis is based, all have co-authors, and I felt that it would be most honest to keep the plural form in this text. 5 Part I Inverse Kinematics In the first part of my PhD project I worked with optimization methods in inverse kinematics. This resulted in the following publications: • ’A Projected Back-tracking Line-search for Constrained Interactive Inverse Kinematics’ [3] . Published in Computers and Graphics. Together with Kenny Erleben. • ’Interactive inverse kinematics for human motion estimation’ published at Vriphys 09: 6th Workshop on Virtual reality Interactions and Physical Simulations [4] . This work was done together with Søren Hauberg ,Jerome Lapuyade , Kenny Erleben and Kim Steenstrup Pedersen. • Three Dimensional Monocular Human Motion Analysis in End-Effector Space. Published in Lecture Notes in Computer Science. This work was done together with Søren Hauberg ,Jerome Lapuyade , Kenny Erleben and Kim Steenstrup Pedersen. The following is a description of this work, based on [3]. with a description of the results of the application described in [4]. 6 Figure 1: Inverse kinematics solved figures, the method is general and poses any conceivable figure. The developed method was integrated in an Ogre demo. 1 Constrained Inverse Kinematics Inverse kinematics is the problem of manipulating the pose of an articulated figure in order to achieve a desired goal disregarding inertia and forces. One can approach the problem as a non-linear optimization problem or as non-linear equation solving. The former approach is superior in its generality and ability to generate realistic poses, whereas the latter approach is recognized for its low iteration cost. Therefore, many prefer equation solving over optimization for interactive applications. In the following, we present a projected gradient method for solving the inverse kinematics problem interactively, which exhibit good performance and precision. The method is compared to existing work in terms of visual quality and accuracy. Our method shows good convergence properties and deals with joint constraints in a simple and elegant manner. Our main contribution lies in the explicit incorporation of joint limits in an interactive solver. This makes it possible to compute the pose in each frame without the discontinuities exhibited by existing key frame animation techniques. In this section we have limited ourselves to using a box-limit joint constraint model. This is done because the focus in this section is the Inverse kinematics solver and not the joint limits. The method works with any projection based joint limit method as we will demonstrate later. Inverse kinematics is used for a wide range of applications such as robot simulation or motion planning, creation of digital content for movies or commercials, or for synthesizing motion in interactive applications such as computer games and other types of virtual worlds. In Figure 1 I have illustrated animations created with our own interactive inverse kinematics 7 method. Inverse kinematics is a standard tool in many applications like Maya, 3D Studio Max [5] or Blender [1]. Recently inverse kinematics have been employed as a dimensionality reduction tool in a tracker of human motion [6]. In short, it is a well-known and wide-spread technique, and better numerical methods will therefore be valuable to a large community. A generally applicable method should be easy to use and thus minimize the number and complexity of user-defined parameters, while giving as realistic a pose as possible. Furthermore speed is essential, whether the method is used interactively to pose a figure, or to simulate movement in a virtual world. Most applications have chosen one of two avenues. Either, they are specialized closed form solutions for specific low-dimensional manipulators, the approach often taken in Robotics, or they are general type methods. Even though inverse kinematics has been around for quite some time, there seems to be very little work done in exploring methods which can bridge the gap between the two extremes, perhaps because the animation industry has not felt the need for further improving their methods, and the design of robots have followed the same line of thought. However, with the development of more and more humanoid robots and the move towards more physics based animations in media, the need for interactive general purpose inverse kinematics methods is again topical. Our focus is on the underlying method of solving the inverse kinematics problem. This can be extended to handle more user control, but that is not our interest. We focus on a general, interactive method which includes joint limits. To state the problem more formally: Given a serial mechanism, we can set up a coordinate transformation from one joint frame to the next. Thus, we can find one transformation that takes a point specified in the frame of the end-effector into the root frame of the mechanism. We write it in general as y = F(θ). (1) We can change the values of the joint parameter θ and gain explicit control over the position and orientation of the end-effector, y. This is commonly known as forward kinematics. Given a desired goal position, g, one seeks the value of θ such that θ = F−1 (g). (2) This is known as inverse kinematics and it is the problem we address in this part of the thesis. 8 1.1 Related Work Inverse kinematics was introduced in robotics [7] and to the graphics community early on [8]. In robotics, the problem is usually phrased as a dynamic system which may lead to different schemes [9]. An overview of numerical methods used in computer graphics can be found in [10]. Inverse Kinematics methods In [11] a new mathematical formalism is presented for solving inverse kinematics as a constrained non-linear optimization problem. This work allows for general types of constraints. Inverse kinematics is known to suffer from problems with redundancies and singularities [12]. In robotics, redundancy problems have been addressed by adding more constraints [13]. In animation the redundancy is most often handled by using the spatial temporal coherency of consecutive solutions. Thus, the correct solution closest to the previous solution is chosen. This is also true in our system (see e.g. Figure 2). Figure 2: An example of a mathematically correct solution which deviates from the reference due to redundancy. The Ghosted overlay with the grey skeleton is the projected line-search inverse kinematics solution. Notice the large difference on the wrist and elbow and the small but noticeable difference on the shoulder joint. A positional goal was used in this example. In [14], three methods are revisited and evaluated for computer game usage: an algebraic method, Cyclic-Coordinate-Descent [15], and a NewtonRaphson method inspired by [11]. The Newton-based method is used for complex manipulation and claimed to give the most realistic looking poses, but it is the slowest. However, joint limits were not dealt with in this work. Other formulations have been investigated for instance in [16] the problem 9 is solved using linear programming. In [17] a mesh based inverse kinematics method was presented. This relies on example poses to manipulate the mesh directly and does not handle joint limits. Furthermore the mesh based approach runs with only 1 frame per second for even moderately sized meshes. In robotics closed-form methods are popular [18]. Closed-form methods often result in algebraic systems that can be solved very fast and reliably, but these methods are highly specialized for a specific low-dimensional manipulator (up to 7 degrees of freedom). In Computer Graphics which is our main focus in this work, even a single arm will usually have more than 7 degrees of freedom. In the simplified male human skeleton from Figure 1 and 3 this is 11. The shoulder has 5 degrees of freedom (2 in the collar bone 3 in the shoulder joint) 1 in the elbow , 3 in the wrist and 2 in the hand. Thus, general purpose methods capable of dealing with many degrees of freedom are desired. Figure 3: A running animation made using the projected line-search optimization approach. The presented method supports interactive editing of animated characters. In [19], inverse kinematics is combined with other techniques and a sequential quadratic programming problem is solved. The running times are in minutes which prohibits interactive usage. Motion synthesis using space-time optimization and machine learning has also been tried [20, 21] although running times are not yet within the grasp of the real-time domain. An example of a widely used general method is the Blender Software [1] which uses a Jacobian Inverse scheme with the pseudo inverse being computed using SVD. To sum up, there is still a need for fast general purpose methods for posing characters with direct manipulation. Joint Limits Often constraints such as joint limits are omitted [14] or dealt with as a post-processing step [15]. The added value of handling joint 10 limits are shown clearly in Figure 5. Several approaches for handling joint limits has been proposed. Joint sinus curves was used in [22] to describe the boundaries of the feasible motion space of the joints. If a joint exceeds its boundary then it is projected back to the boundary and kept fixed at the boundary until the goal position is changed. In [23], quaternion boundary fields were created, and a bisection algorithm was used to back-project infeasible joint positions onto the closest-point on the quaternion boundary field. In [24] a back-projection is used after the joint-parameters have been updated. In [25], joint reach cones are introduced and later refined in [26] to handle moving center of rotations. Here back-projection of infeasible joints is also used. In [11] they keep track of currently active joint limits and modify their scheme in such way that joint limits will not be violated. An example of a method currently used in Commodity software, is the Jacobian Inverse method used in the Blender software, which uses a projection to move the solution unto a feasible region. This projection is performed as a separate step after an unconstrained solution has been found. It has not been possible to get information regarding the methods used in other major 3D systems, but their performance and quality are comparable. Thus, Blender has been chosen for the explicit comparisons in this work. 1.2 Contribution Figure 4: The Inverse kinematics system in Maya exhibiting the typical flipping behavior of systems which project the joint limits after solving the system. The skeleton has 16 degrees of freedom in total and 2 positional end effector goals. Joint limits are shown as green cones. One goal (shown by yellow arrow) was moved vertically to force the system to move along the joint limits. The flip is clearly seen from the third to the fourth picture. In previous work, limits are dealt with as a post-processing step that simply back-projects infeasible iterates to a feasible iterate or an active set 11 approach is used. These approaches either disregard a sufficient improvement in the solution or result in computationally expensive book-keeping. Our major contribution is a line-search method that guarantees an improvement of the solution, and robust handling of joint limits. Our experience and the literature seem to indicate that constrained nonlinear optimization is not the favourite choice for interactive applications. This is a shame since such a formalism offers more realistic motion and general constraints. This has motivated our work. We believe the task lies in creating a simple and elegant numerical method well suited for the purpose of inverse kinematics and which is easily implemented by programmers in industry. In this work, we will present and evaluate a numerical approach for solving the interactive inverse kinematics problem as a constrained nonlinear optimization problem. We will demonstrate how our numerical approach can be used interactively and can deal with joint limits. Our method does not exhibit the flipping behavior of methods which solve the unconstrained inverse kinematics problem before projecting the solution unto the feasible set. It avoids this without having to resort to such pre or post processing as e.g. the preferred pose of Maya. in fig 4 an example of the unwanted flipping behavior exhibited in Maya for a constrained solution with two positional goals and joint limits are shown. Our focus has not been on developing a finished interactive animation suite but rather to present a method which can be used by others in their systems. We have done this by making all code available in the OpenTissue library [27]. From this the implementation details can be seen and the method may be included in any project. Explicit comparison is performed with the inverse kinematics solver found in the Blender software package. Blender has been chosen because it is fully comparable in functionality with the major 3D animation softwares Maya and 3D Max. Furthermore, Blender is open–source. Thus, we could compare the timings of the IK-solvers directly without e.g. render-time disturbing the measurements. 2 The Traditional Approach In the following we will describe the traditional approach for solving the inverse kinematics problem. In the context of human modelling, a skeleton is often modelled as a collection of rigid bodies connected by rotational joints of 1–3 degrees of freedom. An example is shown in Figure 6. All joints are constrained in 12 Figure 5: Joint limits shown by posing a leg in its extreme positions using positional goals. The goal position of the end-effector is shown as a small orange ball. Notice in pose 3 that the goal would be reachable if no joint limits were present, and that both pose 4 and 5 would be different without joint limits. Figure 6: An illustration of the kinematic model. End–effector positions are shown as green dots, while the desired positions (goals) are shown as red dots. their rotation, as exemplified by joint i in Figure 6 with li and ui showing the limits of the angle θi . To compute the position and orientation of a joint in space, we perform a transformation of the bone relative to its parent joint. The transformation consists of a rotation and a translation corresponding to the shape and orientation of the joint, relative to its parent. These transformations are then nested to create chains of joints. Each chain ends in an end–effector, which can be regarded as the handle for controlling the chain. Thus, the full transformation of a joint from local space to global space can be performed. Initially, we know the value of the joint parameters, θk , and a desired goal state for the end-effector, g. The corresponding initial state of the 13 end-effector is given by yk = F(θk ). (3) θ = θk + ∆θk , (4) Writing where ∆θk is the change in joint parameter values. Our task is now to compute ∆θk such that g = F(θk + ∆θk ). (5) We perform a Taylor series expansion of the right-hand side g = F(θk ) + ∂F(θk ) k ∆θ + O(� ∆θk �2 ). ∂θ (6) We introduce the notation J(θk ) = ∂F(θk ) . ∂θ (7) and call the matrix J the Jacobian. Next, we ignore the remainder term of the Taylor series expansion, to obtain the approximation g ≈ F(θk ) + J(θk )∆θk . (8) Recall that y = F(θk ) and for the moment assume that J(θk ) is invertible. Then we can isolate the unknowns of our problem ∆θk = J(θk )−1 (g − yk ). (9) This is a linear model for taking us as close to g as possible with a linear step. Thus, we may not get to g in one step. To solve this we will keep on taking more steps until we get sufficiently close. That is, we compute θk+1 = θk + ∆θk and repeat the above steps with k replaced by k + 1. This results in a non-linear Newton method. The important thing to notice is that the method only needs to know how to evaluate the function-value, F(θ), and the Jacobian J(θ). From [28] we know that if F is continuously differentiable and the Newton sub-system is solved with sufficient accuracy then the non-linear Newton method will have quadratic convergence. We also know, that we are guaranteed to find a solution to the vector equation g = F(θ) given the initial iterate θ1 is sufficiently close to the solution. If the initial iterate is not sufficiently close then we may only get linear convergence. In practice, J is rarely invertible. To overcome these problems one may use the Moore–Penrose pseudo inverse in which case the method is known 14 as the Jacobian Inverse method [8]. The pseudo inverse update is the solution of the least squares problem of minimizing 12 � J∆θk − (g − F(θk )) �2 . The residual function, r, can be written as r(∆θ) = yk+1 − g = F(θk + ∆θ) − g. (10) Taking a first-order approximation yields r(∆θ) ≈ J∆θ − (g − F(θk )). (11) Using this linear residual model we wish to minimize � J∆θ − (g − F(θk )) � or equivalently f (∆θ) = 1 � J∆θ − (g − F(θk )) �2 . 2 (12) From the first-order optimality conditions we have the minimizer ∇f (∆θ∗ ) = JT J∆θ∗ − JT (g − F(θk )) = 0. (13) Setting ∆θk to be the minimizer and re-arranging terms while assuming full column-rank of J, we have ∆θk = (JT J)−1 JT (g − yk ). (14) Thus, the pseudo-inverse is a Gauss-Newton type of method that yield the solution of a least square problem. A major draw-back of the pseudo-inverse method is the discontinuity of the pseudo-inverse near a singularity [12]. A damped least square (Levenberg-Marquardt) type method can be used to overcome this problem. That is, one seek to minimize 12 � J∆θ − (g − F(θk )) �2 +λ2 � ∆θ �2 , where λ > 0 is a regularization/damping parameter. Performing a similar derivation as above results in the update formula, ∆θk = (JT J + λ2 I)−1 JT (g − yk ). (15) However, one must deal with the problem of selecting a regularization value. Actually, Levenberg-Marquardt is using JT J as the Hessian approximation and can be understood as a modified Newton method combined with a trust region [28]. Notice that in most work on inverse kinematics only a single Gauss-Newton or Levenberg-Marquardt iteration is taken to solve the Newton sub-system. In some cases, the inverse Jacobian can be approximated by the transpose, J−1 ≈ JT , this variant of the method is known as the Jacobian Transpose [15]. The Jacobian Transpose method has linear convergence to the unconstrained minimizer. 15 Figure 7: Example of the visual quality showing 11 key-frames in an animation of a jump. The animation was edited manually using a simple editing suite made in connection with this work. One may also use singular value decomposition to obtain a minimum norm solution. Singular value decomposition has the benefit that one can deal with the singularities and ill-conditioning arising from the loss of freedom [12], while it retains the ability to handle secondary goals. The open source software package Blender [1] uses a singular value decomposition based Jacobian Inverse method, which deals with joint limits by projection of the final solution unto the feasible region. If one uses a matrix splitting method [29] for solving the Newton equation then one would obtain the equivalent of the Cyclic-Coordinate-Descent method [15]. The iteration cost of this method is very low. However, it has poor convergence rate. All of the above variants suffer from the following two drawbacks. First their extension to deal with joint limits is not an explicit part of the mathematical model and can be described as applying a back-projection of nonfeasible iterates disregarding the optimality of the projected iterate. Second, the Newton sub-system is not well-posed and approximate solutions are needed in one way or the other. This often results in poor convergence rates and maybe even divergence. Taking a non-linear optimization approach to the inverse kinematics problems allows one to model joint limits in the underlying mathematical model of the problem, and the numerical problem of singularities of the Jacobian is avoided. 3 A Non-Linear Optimization Approach We use a squared weighted norm formulation measuring the distance between the goal positions and the end-effector positions. This formulation is similar to [11], and like them we can support numerous goal types including both position and orientation goals. The main difference being that 16 we have agglomerated all goals and introduced a general weighting matrix instead of dealing with K square weighted summation terms. This formulation is well suited for calculation of the solution to a global kinematics problem with multiple end-effectors, because it takes into account the interdependency between various branches. The Jumping motion in Figure 7 is an example of such an animation with multiple dependencies. Given a branched mechanism containing K kinematic chains, where each chain has exactly one end-effector. We agglomerate the K end-effector functions into one function y1 F1 (θ) .. .. . . y = yj = Fj (θ) = F(θ), (16) . . .. .. yK FK (θ) where yj is the world coordinate position of the j th end-effector, and Fj (θ) is the end-effector function corresponding to the j th kinematic chain. Using the agglomerated end-effector function, we create the objective function f (θ) = (g − F(θ))T W(g − F(θ)), (17) where W is a symmetric positive definite and possible diagonal matrix and � � T T g = g1T · · · gK is the agglomerated vector of end-effector goals. The optimization problem is θ∗ = min f (θ) (18) θ subject to the linear box-constraints θ≥l θ ≤ u, (19a) (19b) which models the minimum and maximum joint parameter values. Here l is a vector containing the minimum joint limits and u is a vector of the maximum joints limits. This implies l ≤ u at all times. If F is sufficiently smooth and θk → θ∗ as k → ∞ then F behaves almost as a quadratic function at θ∗ . This intuition suggest that when we get close to a solution the formulation behaves as a convex quadratic minimization problem. Further, by design all constraints are linear functions defining a convex feasible region. Thus, a simple constraint qualification for the first-order necessary Karush-Kuhn-Tucker optimality conditions is 17 always fulfilled [28]. This would imply that a Newton method would be the method best suited for solving this problem. Unfortunately this approach is infeasible, since the interactivity requirements of the system prohibits the costly computation of a Hessian, and even Quasi-Newton methods such as the Broyden - Fletcher - Goldfarb Shanno (BFGS) method [28] would still be costly compared to methods relying solely on the first order information. Originally, [11] combined a Quasi-Newton method with the active set idea used in the projected gradient method of [30, 31]. The idea can be stated as modifying the Newton direction by applying a modification to the current Hessian approximation such that the resulting Newton search direction obtained from the Newton sub-system is kept inside the feasible region. Of course then one must search for blocking constraints when performing a line-search, and furthermore, one must update the set of active constraints and conduct the corresponding projections on the Hessian approximation. The approach of Zhao et al. does not exploit the fact that the above formulation has a convex feasible region. In fact the feasible region is a boxed feasible region. This suggest that all the book-keeping and modifications of the Hessian matrix can be omitted if the active set idea is replaced by a projected line-search. Unfortunately preliminary testing showed us that projection of a Quasi-Newton direction was not a feasible approach, since the projected direction can no longer be guaranteed to give a reduction in the objective function. To be able to guarantee this reduction a method which is more perpendicular to the iso-contour of the objective function must be chosen. 4 Projected Gradient The simplest idea for a projected line-search method is to use the gradient descent as a basis. This method is called Gradient projection or the projected gradient method. It is very well suited for nonlinear optimization with box constraints and is robust. In fact, it is not even necessary to assume feasibility of the previous iterate to ensure feasibility of the current iterate, since the method relies on projection. The unprojected search direction of the gradient descent is orthogonal to the iso-contour of the objective function. Thus, we can guarantee that the projected search direction will always be a descent direction for a sufficiently small step-length. 18 4.1 Computing the Gradient From (17) we have f (θ) = (g − F(θ))T W(g − F(θ)). (20) The differential can be computed as follows df = d(g − F(θ))T W(g − F(θ)) + (g − F(θ))T Wd(g − F(θ)), (21) which reduces to df = 2(g − F(θ))T Wd(g − F(θ)), (22) where ∂F(θ) dθ, ∂θ = −Jdθ. (23a) d(g − F(θ)) = − (23b) Which means From this we have (24) df = −2(g − F(θ))T WJdθ. df = −2(g − F(θ))T WJ, dθ and the gradient can now be written ∇f = (25) df T = −2JT W(g − F(θ)). dθ (26) How to compute the Jacobian is treated in 5. 4.2 Finding a Step-Length Given this search direction we can update our parameter vector θ by (27) θk+1 = θk − τ ∇f, where τ is some scalar. If τ is a constant or given by a formula this is equivalent to solving the Jacobian Transpose method as can be seen from (26). Several values of τ have been tried for the Jacobian Transpose method. In [10], they compute the step-length according to, eT z τ= T z z � � where e = g − yk 19 and z = JJT e. (28) Whereas in [32] they compute the step-length such that � � � I − JJ+ τ e �≤ ε (29) where ε > 0 is a user-specified constant. The rationale behind both approaches is to measure the deviation from the linear approximation. If the deviation is too big then the step-length is reduced. In Robotics, rate control is used [33, 9] in which � ∆θ � is clamped to a specified maximum, and only a single iteration is used. Others [34] clamp the goal displacement if it exceeds a threshold. Due to linearity down-scaling, the goal-displacement is equivalent to using a smaller step-length. A non linear optimization method uses some dynamic scheme to find a suitable step-length in each iteration. Usually, a simple inexact line-search is used, such as the Armijo back-tracking. As our approach needs to satisfy the constraints we need a modification of this approach as we will describe next. 4.3 Projected Armijo Back-Tracking Line-Search An important part of a projected method is the projected line-search since it is here that the actual projection and thus the constraining of the solution is done. Numerous ways of performing inexact line-searches exist and most of them could be combined with projection. We have chosen the Armijo back-tracking approach because of its beneficial properties with regard to speed and robustness, and because it guarantees good improvements in the objective function when such is possible. We can think of f (θ) as being a function of the step-length parameter, τ , thus we may write f (τ ) = f (θ + τ ∆θ) (30) A first order Taylor approximation around τ = 0 yields f (τ ) ≈ f (0) + f � (0)τ (31) The sufficient decrease condition, the Armijo condition [28], is f (τ ) < f (0) + αf � (0)τ (32) for some α ∈ [0..1]. Observe that f� = d f (θ + τ ∆θ) = ∇f (θ)T ∆θ dτ 20 (33) This is nothing more than the directional derivative of f taken at θ and in the direction of ∆θ. The idea is now to perform an iterative step reduction by setting τ 1 = 1 and verify the above test. If the test fails one updates the step-length as τ k+1 = βτ k (34) where α ≤ β < 1 is the step-reduction parameter. Performing back-tracking ensures the longest possible step is taken. Therefore there is no need for a curvature condition to avoid unnecessarily small steps. We can now rephrase the test as follows � � f (θ + τ k ∆θ) < f (θ) + α∇f (θ)T ∆θ τ k (35) This is the Armijo test used in an un-projected line-search. If a projected line-search is done, then we can think of θ as a function of τ k so we write θ̂(τ k ) = θ + ∆θτ k , (36) moving some terms around results in ∆θτ k = θ̂(τ k ) − θ. (37) Using this in the original Armijo condition we have f (θ̂(τ k )) < f (θ) + α∇f (θ)T (θ̂(τ k ) − θ) (38) Next we will keep θ̂(τ k ) feasible by doing a projection onto the feasible region f (P (θ̂(τ k ))) < f (θ) + α∇f (θ)T (P (θ̂(τ k )) − θ) (39) where P is a projection operator, for instance it could be P (θ̂(τ k )) = max(min(θ̂(τ k ), u), l) (40) where the comparison are element–wise. The vectors l and u would be constant lower and upper bounds for θ. This ensures that even if a previous iterate was infeasible then the current iterate will be feasible. Given this projected line-search it is possible to perform a fast and robust computation of the pose. 5 A Geometric Approach to the Differential To give the reader a more full picture of the method we have included details of the Jacobian computation. The derivations in this section are 21 based on previous work such as [11], but describes this in more detail for the multiple end-effector case. Without loss of generality, we will choose homogeneous coordinates to develop the theory in the following. Given a chain with n links, we have 0 T1 , . . . , n−1 Tn homogeneous coordinate matrices. We will assume that the ith joint depends on the parameters θi . That is i−1 Ti can be thought of as a function of θi . We will specify the tool held by the end-effector and the goal placement of the tool by the agglomerated vectors p pgoal [y]n = î , [g]0 = îgoal ∈ R3 × S 2 × S 2 (41) ĵ n ĵgoal 0 where p is the position while î and ĵ are unit vectors specifying the orientation. The bracket notation [·]i is used to make it explicit that vectors are expressed in the coordinates of the ith joint frame. Using homogeneous coordinates we can write the instantaneous position of the tool as 0 p p Tn 0 0 0 Tn 0 î = F(θ), y = î = 0 (42) 0 0 0 Tn ĵ 0 ĵ n where 0 Tn = 0 T1 · · · n−1 Tn . Often one would use the practical choices 0 1 0 0 0 1 [p]n = (43) 0 , [î]n = 0 , and [ĵ]n = 0 1 0 0 which could greatly simplify all expressions. However, in the following we will keep things general. Let us investigate the differential of the endeffector function, dθ1 .. . � � ∂F(θ) dF = dθ = J1 · · · Ji · · · Jn dθi (44) . � ∂θ �� � .. J dθn where J is the Jacobian. The above equation tell us what the differential change of the end-effector would be if we induced some differential change in the joint parameters. This opens up for an intuitive way of computing J. 22 We observe that Ji dθi describes how the end-effector position is influenced by a change in θi . Without loss of generality, we will only focus on the position term. The remaining terms follows in a similar fashion. We have � � � � ∂ p ∂(i−1 Ti ) i p 0 i−2 n−1 dθi = T1 · · · Ti−1 Ti+1 · · · Tn dθ (45a) � �� � ∂θi � �� � 1 n i ∂θi 1 0 0T i−1 � � ∂(i−1 Ti ) p 0 = Ti−1 dθ , 1 i i ∂θi iT n (45b) Assume that the ith joint is a revolute joint with the unit joint axis [ui ]i−1 = � �T xi yi zi specified as a constant vector in the i − 1th frame. One can show � × � ∂(i−1 Ti ) U i Ri 0 = (46) 0T 0 ∂θi where Ri is the 3-by-3 upper part of i−1 Ti � � Ri t i i−1 Ti = T 0 1 where ti is the translational part and 0 yi −zi 0 xi U× i = −yi zi −xi 0 (47) (48) is the skew-symmetric cross-product matrix. That is [ui ]i−1 × p = U× i p for some p-vector. This means we have � � � × �� � ∂ p U i Ri 0 p 0 dθi = Ti−1 dθ , (49a) 0T 0 1 i i ∂θi 1 0 � × � Ui Ri [p]i 0 = Ti−1 dθi , (49b) 0 Notice that Ri [p]i = [p]i−1 − [ti ]i−1 , so � � � � ∂ p [ui ]i−1 × ([p]i−1 − [ti ]i−1 ) 0 dθi = Ti−1 dθi , 0 ∂θi 1 0 � � [ui ]0 × ([p]0 − [ti ]0 ) = dθi , 0 23 (50a) (50b) Using similar derivations for i and j terms allow us to conclude that we can obtain the ith column of the Jacobian corresponding to a revolute joint by [ui ]0 × ([p]0 − [ti ]0 ) [ui ]0 × [î]0 Ji = (51) [ui ]0 × [ĵ]0 If the ith joint was a prismatic joint with sliding along the joint axis given by the unit vector [ui ]i−1 then one would have � � ∂(i−1 Ti ) 0 [ui ]i−1 = T (52) 0 0 ∂θi from this we have � � � � � � ∂ p 0 [ui ]i−1 i p 0 dθi = Ti−1 T Tn dθ , 0 0 1 n i ∂θi 1 0 � � u 0 = Ti−1 i dθ , 0 i−1 i � � u = i dθi , 0 0 (53a) (53b) (53c) Note that the sub-parts corresponding to orientation, î and ĵ, are zero. � � � � � � ∂ î 0 [ui ]i−1 i î 0 = Ti−1 T Tn dθi , (54a) 0 0 0 0 ∂θi 0 n � �� � 0 [u ] î i i−1 = 0 Ti−1 T dθ (54b) 0 0 0 i i (54c) =0 similar for the ĵ-term. Thus for the case of the prismatic joint we have [ui ]0 Ji = 0 (55) 0 The extension to more than one translation axis is trivial. As seen from all the above derivations the “effect” of the tool is computed from the geometry expressed in the world coordinate system, and we can conclude that the computation of the Jacobian is totally independent of what type of coordinate representation one uses. We exploit this to use a quaternion representation rather than a homogeneous coordinate 24 representation. Since spherical linear interpolation quaternions are better suited for interpolation of the in-between key-frames and superior skinning techniques exist based on quaternions as well [35]. Thus, one can avoid conversion between quaternions and matrices completely. Further the computational effort in computing the absolute transformations of each joint is less expensive using a quaternion representation than using a matrix representation. 5.1 A Ball and Socket Joint In the above, we only dealt with having one rotation axis and multiple translation axes. Thus, a natural question is what to do with multiple rotation axes? Traditionally, this has been handled by creating imaginary multiple joints. Thus a joint with rotation around three axes can be modeled as three revolute joints placed on top of each other. In the following, we will derive equations for dealing with such a joint in a canonical way. Euler angles are a popular choice as parameterization in motion capture and animation formats, therefore we will use an Euler angle parameterization of a ball and socket joint. Inspired by the robotics community [7, 36], we will adopt the ZY Z Euler angle convention. This means that if the ith joint is a ball-and-socket joint then it will be parametrized by the angles θi , φi , and ψi . Further, the relative transformation is given as � � RZ (φi )RY (ψi )RZ (θi ) ti i−1 Ti = , (56) 0T 0 where RZ and RY are rotations around the z and y axes of the (i − 1)th joint frame. Trivially we have � × � ∂(i−1 Ti ) (Z RZ (φi )) RY (ψi )RZ (θi ) 0 = (57a) 0T 0 ∂φi � � ∂(i−1 Ti ) RZ (φi ) (Y× RY (ψi )) RZ (θi ) 0 = (57b) 0T 0 ∂ψi � � ∂(i−1 Ti ) RZ (φi )RY (ψi ) (Z× RZ (θi )) 0 = (57c) 0T 0 ∂θi where 0 × Y p = 1 × p and 0 25 0 × Z p = 0 × p 1 (58) for some vector p. Next we define the three vectors 0 [ui ]i−1 = 0 1 i−1 0 [vi ]i−1 = RZ (φi ) 1 0 i−1 0 [wi ]i−1 = RZ (φi )RY (ψi ) 0 1 i−1 (59a) (59b) (59c) and by straightforward computation we have, � � ∂[p]0 dφi = 0 Ti−1 [ui ]i−1 × ([p]0 − [ti ]0 ) dφi , ∂φi � � ∂[p]0 dψi = 0 Ti−1 [vi ]i−1 × ([p]0 − [ti ]0 ) dψi , ∂ψi � � ∂[p]0 dθi = 0 Ti−1 [wi ]i−1 × ([p]0 − [ti ]0 ) dθi , ∂θi (60a) (60b) (60c) now renaming the vector ([p]0 − [ti ]0 ) r0 = ([p]0 − [ti ]0 ) we get the Jacobian entry [ui ]0 × r0 [vi ]0 × r0 [wi ]0 × r0 Ji = [ui ]0 × [î]0 [vi ]0 × [î]0 [wi ]0 × [î]0 [ui ]0 × [ĵ]0 [vi ]0 × [ĵ]0 [wi ]0 × [ĵ]0 6 (61) (62) Performance of Projected Back-Tracking LineSearch The projected line-search developed in this work has been tested with two different methods: the Jacobian Transpose method, which is the same as a Steepest Descent method using a fixed step-length, and a projected linesearch method with the projected Armijo back-tracking line-search we described in Section 4.3. We have compared our results with the results from the SVD based Jacobian Inverse method used in the Blender Software package. 26 Figure 8: An example of what may happen if the step-length of the Jacobian Transpose method is chosen too high. The motion flips between the green and the red pose, superimposed on the desired result. Step-length in this example was set at 0.05. This example used 5 positional goals and joint limits. 6.1 Comparisons Blender was chosen as an example of a widely used software package which lives up to current performance and quality demands. The reason for choosing Blender instead of e.g. Maya or 3D Max was, that it is possible to perform actual measurements on Blender due to it being open source. Since the functionality, quality and performance of Blenders inverse kinematics system is very similar to the other solutions (see e.g. Figure 4 or the so called cg survey by [37]), we chose only this one. Our reference implementation of the Jacobian Transpose method corresponds to the one used by [15] where a fixed sufficiently small step-length is used. Experiments were done with the Jacobian Transpose method varying the fixed step-length. These experiments showed that a step-length of more than 0.005 would make the Jacobian Transpose method diverge even if the method were given an initial iterate comparatively close to a solution. Step-length below this limit slowed down the method without discernible improvement to the quality. Therefore, in all our results we used Jacobian 27 Transpose method with a fixed step-length of 0.005. An example of the divergence exhibited with too large step-length is shown in Figure 8. (a) Animation using 28 frames per (b) Animation using 7 frames per cycle cycle (c) Animation using 5 frames per cycle Figure 9: The difference between the three sampling settings of motion capture data. Notice that spatial-temporal coherence is decreased from a to c. 6.2 Test Framework The methods were tested by running a number of repeated tests under different conditions. The test scenario comprised a fixed setup using a motion captured animation of a person doing a gymnastics exercise. A 28 time lapse of the animation showing the exercise can be seen in Figure 13. The positions of the hands, feet, pelvis and head of the motion capture animation were used as goals for the inverse kinematics skeleton. The orientation of the goals was not used in these tests. The goals were updated each frame. The skeleton consists of 71 degrees of freedom, and joint limit constraints on all joints, giving a total of 142 constraints (71 upper and 71 lower). Since interactive performance was a major factor in the evaluation we chose to compare the quality of the method with a fixed time slot at their disposal giving each a maximum number of iterations which would make it converge in approximately 0.015 seconds or less, corresponding to approximately 66 frames / seconds. We varied how far the goal-positions were placed from the end-effector positions. This was done by sampling the motion captured data with varying intervals. The settings used were 0.05, 0.7, and 1.0 seconds. Figure 9 shows three consecutive frames of the reference animation, using the three settings. The test cases were run on an Intel®dual core 1.66 GHz architecture with 1 GB memory utilizing only one core. An absolute tolerance of 0.05 units was set. The tolerance was chosen to be small enough not to interfere with the visual quality of the animation. 0.05 units is approximately 1 centimeter if the skeleton corresponds to a person that is approximately 1.80 meters high. The function value in Figure 10, 11, and 12 are the sums of all end-effector squared errors. 6.3 Visual Quality The visual quality is graded by the closeness to the reference animation as well as the smoothness of the animation. In both cases the gradient projection gives superior results to what is being computed by the SVD based Jacobian Inverse method. This can be seen from the plots in Figure 10, 11 and 12.The figures show the Median value as a red line. The edges of the blue boxes denote the 25th and 75th percentiles, the whiskers denote the boundary of the inliers of the data set. Outliers are shown individually as red crosses. The smoothness of the animation and the handling of joint limits and orientation/position of intermediate joints can be seen from the frames in Figure 13 notice frames 7 and 9 where the animation made by the reference method clearly jitters. 29 Figure 10: Plot of the distribution corresponding to the animation shown in Figure 9(a). Red crosses denote outliers in the data set. Notice that the Projected line-search has much lower Median as well as a more compact distribution. Figure 11: Plot of the distribution corresponding to the animation shown in Figure 9(b). Red crosses denote outliers in the data set. 30 Figure 12: Plot of the distribution corresponding to the animation shown in Figure 9(c). Notice that the Jacobian Transpose method is beginning to have problems in this case due to the lack of coherence. Some significant outliers with values of around 80 has been omitted from the Jacobian Transpose method plot to make comparison possible. 6.4 Robustness Since the method guarantees a reduction in the objective function, the method is very robust. No restrictions on previous poses are necessary. The robustness can be seen from the tight distributions of the values in Figure 10, 11 and 12. 6.5 Generality The method can be used with any conceivable skeleton. Examples of skeletons we have tested are shown in Figure 1. The method is easily integrable in other systems. The method is implemented in the free open-source metalibrary OpenTissue [27], and can be included in any source code under the Zlib license [38]. As shown in Figure 1 the method has been included in an ogre demo by a third party. 7 Alternatives to Gradient descent Gradient descent is not the best performer in the family of optimization methods. For general constrained problems a quasi-newton approach is usually a much better choice. Even with the higher iteration cost and the 31 Figure 13: Selected frames from the animation used in testing the methods, each frame shows from left to right: the Blender version, the Motion Capture reference, the Projected line-search method. Notice that the Blender method jitters from frame to frame and that it differs significantly from the reference (notice especially frame 7 and 9). necessity to keep track of the active set of constraints, better performance can usually be achieved. For the Inverse kinematics problem in an interactive setting it is often not as important to get an exact solution and when we add to this that the projection operation cannot be combined with the quasi newton update step, it becomes infeasible to use the active set method. In this PhD project we experimented with several method and as it can be seen from our earlier work [39] the projection operator can be combined with a conjugate gradient method, with good results. The projection operator was also combined with a dogleg method in [40] The performance of these alternatives are not handled in this work. The interested reader is refered to these texts. 32 8 An Application in motion tracking In the following I will describe an application of the Inverse kinematics system. The system was used as a dimensionality reduction in a particle filter tracker developed by Søren Hauberg in conjunction with his Phd work [41]. The description here is an extract of the one presented in [4] here I have concentrated on presenting the results. 8.1 Motion Estimation as a dimensionality reduction for monocular human motion estimation. The inverse kinematics solver deals efficiently and robustly with box constraints and does not suffer from shaking artifacts. The presented motion estimation system uses a single camera to estimate the motion of a human. The results show that inverse kinematics can significantly speed up the estimation process, while retaining a quality comparable to a full pose motion estimation system. Our novelty lies primarily in use of inverse kinematics to significantly speed up the particle filtering. It should be stressed that the observation part of the system has not been our focus, and as such is described only from a sense of completeness. With our approach it is possible to construct a robust and computationally efficient system for human motion estimation. Inverse kinematics has found widespread use as an intuitive posing system for articulated figures in Computer Graphics [14] and for motion planning in Robotics [42]. We propose a novel use of inverse kinematics as a means to reduce the dimensionality of a particle filter based tracking algorithm. Preliminary results show that this approach significantly reduces the time demands compared to existing approaches with comparable results. The method may make it possible to perform visual tracking of general human motion in an interactive way. Three dimensional human motion analysis is the process of estimating the configuration of body parts over time from sensor input [43]. Traditionally motion capture equipment has been used to track this motion. In motion capture, markers are attached to the body and then tracked in 3 dimensions. Usually this requires multiple tracking devices so motion capture is most often performed in pre–calibrated laboratory settings. Our long term goal is to use human motion analysis as part of a physiotherapeutic rehabilitation system. In the system, the motion of a patient is tracked and analyzed during exercise sessions performed both at the clinic and at the patient’s home. This application rules out the traditional mo33 tion capture approach, and results in the need for a simpler system with fewer cameras. These limitations make it necessary to formulate a better model to compensate for the lack of information from the image data. Our approach is to use inverse kinematics as a way to impose this information. Our approach utilizes the fact that the pose of a skeleton can be deduced from the end–effector positions. While a full human skeleton may have more than 100 degrees of freedom, the end–effectors space of the same articulated figure can have e.g. only 15 degrees of freedom (positional parameters of head, hands and feet). This dimensionality reduction accounts for a significant speedup in the computational demands of the system, compared to analyzing the motion in the full pose configuration space of the skeleton. Estimation of human motion is an inherently high dimensional problem, since human motion is both diverse and has many degrees of freedom. The traditional approach to reducing the dimensionality has been to utilize manifold learning, i.e. to try and restrain the motions in a subspace of the full space. This approach is used in [44, 45, 46], but seems to be most suited for a constrained set of motions like walking or golf–swinging. We want to be able to process a larger set of motions, and thus need some other means of reducing the dimensionality. Inverse kinematics has been used in motion estimation before [47], but not as a dimensionality reduction tool. Our work differs in that it utilizes the posing abilities of the inverse kinematics system to infer the pose of the remaining joints. Thus, the estimation can be performed on the end–effector joints only. Our inverse kinematics system is based on the robust system described in previous sections. The motion estimation system used is the one described in [6]. While [6] concentrated on the motion estimation, this work focuses on the interactive inverse kinematics solver. 9 Visual Motion Estimation Visual motion estimation is the process of inferring the motion of a moving object from a sequence of images. In this work we wish to infer the 3 dimensional pose of a human moving in front of a camera. In this Thesis the Visual motion estimation will not be described in detail since this part of the work must be attributed mainly to Søren Hauberg. The interested reader is referred to the papers [6] and [4] where a detailed description of the motion tracking can be found. In this workk we will 34 Figure 14: Traditional motion estimation in full pose space using 100 particles. Notice that the superimposed figure looses track of the person in the image. concentrate on the effects of introducing the inverse kinematics solver as a dimensionality reduction tool. and thus we present the results of our tests. 9.1 Results To verify the quality and to measure the time improvement we performed some simple tests. The tests consisted of estimating the motion of a person sitting on a chair moving his arms about. The resulting skeleton had 3 end– effectors , the hands and the head. The skeleton was fixed at the hip and the legs were not modelled. The method can handle a full skeleton but for these tests we chose a simple skeleton. The original video clip used was app. 45 seconds long, at 15 fps. The purpose of the tests was to compare the time expenditure and quality of our method to a traditional method. We performed tests and timing of the system, using a Lenovo T400 Thinkpad� laptop with an Intel �core™2 duo 2.40 Ghz. 35 Figure 15: Traditional motion estimation in full pose space using 5000 particles. The results of this motion analysis are satisfactory but the computation took more than 10 hours. 36 Figure 16: Motion estimation using inverse kinematics with 25 particles. As it can be seen the results are very similar to Figure 15. However the time spent was only 5 minutes. 37 Figure 14, 15 and 16 show selected frames from an image sequence with the pose estimation results superimposed. The images in each of the figures correspond to the 32nd , 94th , 126th and the 196th frame of the sequence. A video version of the test results are available at: http://humim.org/vriphys2009/ The first test was a traditional full pose motion estimation without inverse kinematics using 100 particles. Figure 14 shows the result. Here the system quickly looses track of one arm and produces a large amount of shaking in the motion estimation. The computations took approximately five minutes on the test hardware. We then increased the number of particles to 5000, which resulted in a successful motion estimation with only little shaking. Unfortunately, this required more than 10 hours of computation time. The results of this test can be seen in Figure 15. The final experiment was run using the inverse kinematics system for pose calculation, making it possible to track in only 9 dimensions. Only 25 particles were used and the running time was approximately 5 minutes. This is a speedup factor of approximately 120 compared to the 5000 particle run and a comparable quality, while the 100 particle run is comparable in time spent, but in this case, the quality of the inverse kinematics tracker is much better. The results show that motion estimation in end–effector space is possible and that large speedups can be achieved using this approach. Our long term goal is to create a motion estimation system for use in a physiotherapeutic rehabilitation program and here it is essential to have real–time performance, in order to provide feedback to both patients and therapists. Estimation in end–effector space makes this requirement more feasible compared to estimation in full–pose space. Using only a single camera gives no depth information which means that the system has difficulty in placing the goals correctly in this direction. This problem might be solved by using a camera type which can infer some depth information such as a stereo camera or a time of flight camera. Weighting the depth parameter of the goals with a small weight might also help since this would give the inverse kinematics solver more freedom in the placement with regards to the depth. It could however, also result in poses which would fit badly with the visual data so it could not stand alone. 10 Discussion and Conclusion The non-linear optimization approach has shown beneficial behavior as a strong mathematical formulation of the inverse kinematics problem. We 38 have presented an efficient and robust numerical approach for solving the problem. We believe our approach is simple and elegant and easy to implement, even for non-specialists in numerical optimization. Further, we have demonstrated that our approach is sufficiently exact and responsive for interactive manipulation of multiple end-effectors while handling jointlimits. We have shown the connection between using a Projected gradient method for the non-linear optimization and the traditional Jacobian Transpose method. Further, we showed how the Jacobian Transpose method could be improved by using our projected line-search method. The shaky and jittery motion often reported near singular configurations can be avoided completely when using an optimization approach, since the variable step-length cancels out the adverse effect of the high angular velocity. It is thus only a matter of a sufficiently exact line-search and a sufficiently low minimum step-length. We have demonstrated how the inverse kinematics solver can be used in an articulated tracker application with a significant speedup as the result. In our opinion, we have only touched upon the subject of seeking the best suited mathematical formulation for interactive inverse kinematics and matching numerical methods. There are many possible avenues for further work. It could be interesting to reformulate the first order necessary optimality conditions into a complementarity problem and further reformulate this as a possible smooth or non-smooth non-linear equation solving problem. This would in a sense take us back to the equation solving approaches but with the difference that constraints are dealt with implicitly without the need for projections at all. Quasi-Newton and Steepest Descent methods are but a few out of many methods for solving the constrained minimization problem we have stated. Trust region methods or hybrids may be other interesting methods to adapt to the formulation given. 39 Part II Joint Constraint Models Box constrained joint limits are excellent for modelling the joint constraints of e.g. robots, but it is not a very exact reconstruction of actual joint limits in the human body. Therefore, my research began to concentrate on finding a good alternative, that would still be compatible with the Inverse Kinematics method described in the previous part, as well as being usable for other methods. This led to an investigation into whether such a model existed. As it turned out there was no method for describing joint limits which could supply a constant time projection operator, thus, we decided to invent our own. This research resulted in the following publications • ’ A Joint-Constraint Model for Human Joints using Signed DistanceFields. Special issue of Multibody System Dynamics [48]. • Local Joint-Limits using Distance Field Cones in Euler Angle Space. Computer graphics international CGI 2010 [49]. • Distance-Field Based Joint-Limits for Biomechanic Joint Models. Euromech Colloquium 511 - Biomechanics of Human Motion. All made in collaboration with Sarah Niebe and Kenny Erleben. The topics of these 3 publications are the development and presentation of the signed distance field joint constraint model. The application moved towards biomechanics and clinical motion analysis. This Is a very nice example of the way a method developed initially for a purely graphics oriented application can be used in e.g. biomechanical work. 40 11 Joint constraints using signed distance fields We present a local joint-constraint model for a single joint which is based on distance fields. Our model is fast, general and well suited for modelling human joints. In this work, we take a geometric approach and model the geometry of the boundary of the feasible region, i.e. the boundary of all allowed poses. A region of feasible posed can be build by embedding motion captured data points in a signed distance field. The only assumption is that the feasible poses form a single connected set of angular values. We show how signed distance fields can be used to generate fast and general joint-constraint models for kinematic figures. Our model is compared to existing joint-constraint models, both in terms of generality and computational cost. The presented method supports joint-constraints of up to three degrees of freedom and works well with sampled motion data. Our model can be extended to handle inter-joint dependencies, or joints with more than three degrees of freedom. The resolution of the joint-constraints can be tweaked individually for each degree of freedom, which can be used to optimize memory usage. We perform a comparative study of the key-properties of various joint-constraint models, as well as a performance study of our model compared to the fastest alternative, the box limit model. The study is performed on the shoulder joint, using a motion captured jumping motion as reference. 12 Introduction When simulating articulated figures, one needs a model that describes the range of motion of the individual joints. An example of such a model is the inverse kinematics (IK) skeleton which is a hierarchy of bones where each bone is connected to a parent bone by a joint [11]. The relative coordinate transforms between connected bones are given by a set of joint parameters. The focus of this work is the presentation of a novel accurate model for representing legal values of these joint parameters based on experimental kinematic data. From a mathematical viewpoint, an IK skeleton is a hierarchy of homogeneous coordinate transformations, where each bone corresponds to a homogeneous transformation. The different joints of the skeleton require varying numbers of parameters for representing a given pose. For instance, the elbow joint only needs a single parameter describing the angle between the upper and lower arm. Joints such as the shoulder or hip joints, have a 41 higher degree of freedom (DOF) and thus require more angle parameters. One could even use a translational joint parameter to model the sliding of the scapula on the thorax in the shoulder complex. These joint parameters are not unbounded, they are constrained by a highly non-convex, continuous connected and closed subset of the parameter space. We use the term feasible pose when all the joint parameters of a given pose are in the legal parameter space. Joint-constraints describe the boundaries of the feasible region of poses. Figure 12 illustrates the geometric complexity of the feasible regions we face within our application perspective. It shows joint-constraints sampled from several motions, found in the Carnegie Mellon university database of motion capture recordings [50]. Figure 17: Examples of joint-constraints for different joint types and different motions. Each row corresponds to one joint type and each column to one motion. Observe the complex geometry of the feasible region. The method we present is comparable to the method presented in [23]. However, the use of Euler angles and distance-fields results in a constant time projection operator, making our method a faster, more accurate and attractive approach. Our model, the signed distance-field joint-constraint model, is generalized, supports highly nonconvex joint-constraints, has a simple geometric interpretation and shows no performance problems that would prohibit use in interactive applications. 42 The work presented here is highly motivated by tracking human poses using computer vision and machine learning based methods [6]. Within this application perspective, there is a need for a joint-constraint model that is computationally fast and gives an accurate description of the feasible region of a single specific human motion. An early version of this research, with an animation focus, was presented in [51]. 13 Related Work Several researchers have investigated the range of motion of joints [52, 53], and numerous models exist e.g [23, 22, 25]. The box-limit model is the prevalent model, used in motion file formats such as Acclaim’s asf/amc [54]. Animation tools such as Maya [5] and Blender [1] also use the boxlimit model. With box-limits, the individual parameters are bounded within a minimum and maximum of allowed values, thus the feasible region forms a product of independent intervals. As this is a linear model with a convex feasible region, the box-limit model is easy to combine with optimization methods []. However, there are two major drawbacks of box-limits: They tend to result in a loose fitting boundary and they fail to include interdependencies between joint parameters. In practice, these short-comings are handled by tweaking the box-limits for specific motions, thus for a running motion one set of values would be used whereas, for a jumping motion, another set of joint parameters would be used. Despite these shortcomings, the box-limit model is widely used. As shown in previous work, box-limits can be determined automatically from motion capture data [6, 55]. The alternatives to the box-limit model offer more descriptive models, at the cost of increased complexity. The signed distance-field model, presented in this work, takes a geometric approach. For each joint, we model the geometry of the boundary of the feasible region of the joint parameters. Our model is local, in the sense that we only model joint parameter dependencies for each individual joint. The model requires a two-manifold feasible region, so it can be embedded as the zero level-set of a signed distancefield, allowing us to handle any nonconvex joint-constraint regions. In this work, we have used the shoulder joint as the primary test case because it exhibits some of the more complex behaviour of joint-constraints and thereby stresses the joint-constraint model used. The legal parameter space of the human shoulder joint is bounded by a nonconvex nonlinear joint-constraint [23, 22]. The authors of [22] use a sinus-cone model from [56], a human shoulder is modelled by a hierarchical IK skeleton. The scapula-thoracic joint is 43 modelled by breaking the closed chain and using the scapula as an endeffector constrained to the surface of an ellipsoidal thorax. The sinus-cone model is more general than the box-limit model. A reach-cone model based on an idea from Korein [57] is presented by Wilhelms and Gelder [25]. This is extended in [26] where a general joint component framework is described. In [58], a spline based implicit joint model is suggested for multibody dynamics. Due to the implicit nature of the model, the geometry of the boundary of the feasible region can be modelled as box-limits in the spline parameter space, interdependency of parameters is omitted. Shoulder joint-constraints are modelled in [23] using quaternion field boundaries. The orientation of the shoulder joint is sampled from motion capture data in quaternion space. Radial basis-functions are used to reconstruct an implicit surface representation of the boundary of the feasible quaternions. Our approach has similarities with this method, as it uses an implicit surface and supports general nonconvex joint-constraints. However, our projection operator is superior as it is a constant time operation, i.e. it is not an iterative procedure. The approach of [23] is further developed in [59] where a hierarchical model of joints is presented. It seems that this approach can only handle a single parent hierarchy as there is no information about deeper hierarchies. In both papers, the main focus is on the machine learning part of building the joint-constraints, while our focus is on the joint-constraint model itself. Table 1 is a summary of a comparison study between key-properties of the above mentioned methods and our signed distance-field (SDF) jointconstraint model. All the compared models can essentially be seen as geometric models of the boundary of the feasible region. Their differences lie mostly in their choice of geometric model for representing the boundary and in the actual computational representation. Finally, there are some differences in the technicalities of how the back-projection operator and feasibility tests are supported. As Table 1 shows, the SDF model offers more generality than its alternatives, while supporting constant time operations. Projection operators fall in two categories: constant time operation or iterative search schemes. One major feature of our SDF model is that the projection is a constant time operation. The only other model that can offer this, is the box-limit model. On the other hand, the memory usage can be high. In our work, each joint needs I × J × K cells of a map, storing a regular sampling of the distance-field, where I, J, K denote the resolution of the map along the three axes. Adaptive distance-maps could be used in place of a regular sampling. However, our results show that in most practical cases, coarse maps can be used and so, the memory usage can be 44 Model Model Representation Data Representation Projection Operator Box Limits Explicit Vector of intervals Constant time projection Sinus Cones Implicit Implicit function Root-search problem, Iterative Newton method Reach Cones Explicit Set of connected tetrahedra Linear search for closest tetrahedron Spline Joints Para-metric B-splines Not available Quaternion Boundary Fields Implicit Radial basis functions (RBF) Root-search problem, Iterative Newton method Signed Distance-Field Implicit Signed distance-field Constant gradient projection Model Feasibility Test Operator Model Capabilities Box Limits Constant time verification of enclosing interval Convex/ Boxed Sinus Cones Evaluation of closed form solution Convex/ Ellipsoid Reach Cones Linear search for enclosing tetrahedron Nonconvex, but no holes Spline Joints Infeasible poses are not allowed General nonconvex Quaternion Boundary Fields Global support of RBF convolution of all samples General nonconvex, difficulties with holes Signed Distance-Field Constant time lookup of distance value General nonconvex Table 1: A comparison study of key-properties of various joint-constraint models. kept at an acceptable level. The predominant trend in previous work in this research area is to con45 sider only local joint-constraints models, the exception being [26] where the global dependency of joint parameters is considered in the case of forward kinematics. In this respect, our work is no different, our model is also a local model. However, we do consider the full dependency between joint parameters within a single joint. Neither box-limits, sinus-cones nor reach-cones offer this. One final aspect of joint-constraints, on which we will elaborate below, is the ease with which one can set up the model. In machine learning, this is termed model selection and could be approached as a nonlinear regression problem [28]. Some models can employ sampled feasible poses for setting up the joint-constraints, the SDF model is one such model. Quaternion boundary fields share similarities with our approach in this regard, although their coordinate basis (imaginary part of a quaternion) is nontrivial to work with. To our knowledge, no prior work addresses model selection of sinus-cones or reach-cones. The presented SDF model offers full modelling generality of local jointconstraints with constant time operations and easy model selection. This makes the presented model a novel method for obtaining more accurate joint-constraints, highly suitable for motion simulation of articulated figures. 14 The Signed Distance-Field Model Even though the model presented here is a local model, it could be extended to cover inter-joint dependencies. Our base assumption is that the boundary of the feasible parameter space of a joint forms a single connected component. This implies that feasible joint motion is a connected subset. Thus, the test for feasibility is reduced to a simple inside/outside test. Thus, in case of an infeasible configuration, the point is projected unto the closest feasible point. We use Z-Y-Z Euler angles as basis, where the orientation is specified by the � �T angles p = φ ψ θ . This allows us to work in a 3D space rather than 4D or 9D as would be the case for quaternions and homogeneous coordinates. By modelling the motion range geometrically, we have a broader basis of well-known geometrical representations to choose from, e.g. polygonal meshes, tetrahedral meshes etc. As performance is of great importance, the geometric representation must support two constant time operations: Verification of a feasible joint pose and projection of infeasible joint poses back onto the boundary of the feasible region. Distance-fields are known to offer both these qualities, but 46 at the cost of extra memory usage. A distance-field is an implicit representation of the geometry, defined by the function Φ : R3 �→ R where | ∇Φ(p) |= 1 everywhere and Φ(p) = 0 for all p corresponding to the geometry. 14.1 Building the Signed Distance-Map In the following we describe how we compute the discrete signed distancemap from the continuous signed distance-field. In principle, any signed distance-field algorithm could be used [60]. For this work, we used a simple brute-force approach. Because the signed distance-field rigging is done as a preprocessing step, the cost of generating the signed distance-map is of minor importance. In fact, we did not even bother storing the preprocessing for our tests. Our experience shows that a fairly coarse resolution is sufficient, as the set of motions we study only requires a low number of temporal samples. Running this process off-line means that generating the distance-map is not a bottleneck, as might have been suspected. Table 2 in section 15.1 lists runtime statistics for generating the signed distance-maps of a jumping motion. Although we use a simple brute-force method in constructing the distancemap, we will – for completeness of presentation – give a full detail description. Initially, we create a regular grid of a user specified resolution. The grid is located in space such that the minimum and maximum corner points of the grid is within the angle interval bounds of the Euler angles: φ 0, 2π ψ ∈ 0, 2π . (63) θ 0, 2π Once the grid has been created all distance values stored at the grid nodes are initially set to ∞. Φ(p) = ∞ ∀p ∈ N , (64) where N is the set of all grid nodal positions in the map. 1. Next we sample the joint motion from some exemplar based motion,{q i }i=N i=1 , for instance some motion capture data. 2. For each time sample q i we extract the Euler angles of the present joint i φ i p = ψi ← qi. (65) i θ 47 The Euler angles are collected in a chronologically ordered list {pi }i=N i=1 . 3. For each node of the grid, we compute the distance to the closest sample point. Running through the entire list of samples we check the distance to the current sample. If the newly computed distance is less than the distance value currently stored in the corner node then we replace the stored value with the new value. This gives the following selection operation: � � Φ(p) ← min Φ(p), � pi − p � ∀p ∈ N (66) i=1..N The temporal sampling of the motion might be too coarse, thereby creating multiple disjoint regions in the final signed distance-map. There are at least two ways of dealing with the sampling problem. One may adopt a naive approach: Detect if the problem occurs and then redo the entire motion sampling at a finer resolution. This could be done by detecting how many connected components we have in the final map. However, a more intelligent approach is to use an adaptive sampling: Given a grid resolution we can adaptively modify the motion capture sampling rate to ensure that the distance between any two consecutive motion samples is never larger than half the maximum grid spacing. We use the adaptive motion sampling strategy in our implementation. 14.2 Adaptive Sampling of Motion We use a chronologically ordered list of our samples to interpolate between neighbouring samples. Coarsely sampled regions are subsampled using spherical linear interpolation( slerp [61]) in angle space. We can subsample the region between two neighbouring samples as densely as necessary, to ensure that the distance between the new samples are never greater than half the grid resolution. The resulting quaternions are then converted into Euler angles and added to the list between the existing samples. Figure 18 summarizes the adaptive sampling algorithm. If disjoint motion is encountered, some interpolation scheme must be established to connect the manifolds of the unconnected motions. For testing purposes, we simply interpolate between the last frame of one animation with the first frame of the second animation. For the test sets used, this simple approach works well. It should be noted that this would not generally be sufficient. It should be ensured that the motion samples are not disjoint, meaning the motions should share at least one common point. 48 n=0 t=0 pn ← q n ← current sample at time t for n = 1 to end q n+1 ← next sample at time t + ∆t while distance(q n , q n + 1) > grid spacing/2 ∆t ← ∆t/2 q n+1 ← next sample at time t + ∆t end if ( distance to small) increase ∆t n←n+1 end Figure 18: Adaptive motion sample algorithm. Observe that he ∆t variable can be seen as a kind of trust region radius. 14.3 The Signed Distance Property We know that all motion samples are feasible poses and we can assume with some certainty that not all feasible poses are represented. Further, we know that some poses are in the interior of the region and some are on the boundary. In some cases we want to treat all samples as if they were lying on the boundary. This would be the case when we have a very restricted motion from which we have built our signed distance-field, and we want to use these to make it possible to mimic the specific motion from which the samples have been taken. Thus, one may wish to add an additional pass to the distance-map generation algorithm. We want to implicitly add the interior void to our representation. The above part of the algorithm only represents the samples as the feasible position and has no notion of what the interior is. To add such a notion one may convert the distancemap into a signed distance-map. The idea is to place a positive sign on some outside border cell of the grid. Then one simply performs a region filling operation where all neighbouring cells not crossing the zero levelset is given a plus sign. In the end, all unassigned cells must be interior and are given a minus sign. One flaw in this approach is that the region must be a closed manifold, otherwise it will be hollowed out and there will be no interior region assigned. This can be helped by either ensuring that the motion is sufficiently dense sampled or by using an alternative method for determining boundary cells. Ensuring the density of the samples has shown to be difficult [23], and for this application the extra work is not 49 needed. The single motion sampling represents a very limited area. In this setting the above mentioned subsampling is sufficient to ensure the signed property of the map. For sculpted joint-constraints on the other hand, the signed distance-map property must be handled as we will show in the next section. 14.4 Sculpted Signed Distance-Field Joint-Constraints It may be difficult to obtain enough motion samples to automatically generate signed distance-fields. Also, one may wish for the option to have specific control over the motion and directly model the behaviour. Using a geometric representation, such as distance-fields, allows for an easy and direct approach for modelling signed distance-fields, for instance by using sculpting tools [62]. As a proof-of-concept to support our claim we used a simple setup exporting the surface of a signed distance-field as a polygonal mesh and then used the open source 3D modelling program Blender [1] to tightly fit a two-manifold to the data, using an enclosing sphere and the shrink-wrap modifier. An example of a feasible region obtained in this way is shown in Figure 19(b). An advantage of this approach is that a nonconvex manifold can be easily obtained and is easily modifiable, either by directly manipulating the mesh or by manipulation through some intuitive interface such as modifying the feasible region as a polygon on a sphere. Figure 20 shows the result of applying the sculpted signed distance-fields. The shrink wrap method used in this example is a way of filling out the empty parts of the feasible region. For completeness, we will give a brief description of the way this modifier works. For a full description we refer to Blender documentation [1]. 1. Choose an enclosing two-manifold object e.g. a sphere. 2. Project all points of the enclosing object inward onto the surface of the target object. 3. Generate a new signed distance-field from this object. Note that the original shape of the two-manifold object has a significant impact on the quality of the resulting joint-constraints. We have chosen this approach as an easy-to-use and intuitive approach, to show the possibilities of signed distance-field joint-constraints. Other more elaborate techniques for constructing a surface from a point cloud, such as the one described 50 (a) Sampled joint-constraints. (b) Sculpted joint-constraints using shrink-wrap. (c) Sculpted joint-constraints shown in Cartesian space. Figure 19: An example of refining the sampled joint-constraints of the left shoulder using an enclosing sphere and a simple shrink-wrap modifier from Blender [1]. in [23], may give more correct results. Either technique could be used together with our method. 14.5 Applying Signed Distance-Field Joint-Constraints for Inverse Kinematics We use an inverse kinematic (IK) modelling approach similar to [11], where the IK problem is formulated as an optimization problem which is solved using an iterative line-search method. The feasibility of a given IK iterate, p, is determined by testing the corresponding Euler angle distance value in the signed distance-fields: Φ(p) ≤ ε ⇒ p is feasible. 51 (67) The parameter ε is a threshold value to counter round-off errors. The impact of this parameter is analysed in our results section. The signed distance-field lookup is performed in constant time by indexing the surrounding grid points and performing a tri-linear interpolation at the lookup position. Infeasible poses are projected back onto the feasible region, by moving in the opposite direction of the distance-map gradient. The gradient is computed as a central finite difference approximation: Φ −Φ i+1,j,k ∇Φi,j,k = i−1,j,k Φi,j+1,k2∆i −Φi,j−1,k , 2∆j Φi,j,k+1 −Φi,j,k−1 2∆k (68) where ∆i,∆j,∆k denote the grid spacing along each coordinate axis. The gradients at the grid nodes surrounding p are interpolated using a tri-linear interpolation on each component of the gradient. The projection of the infeasible p is then: p ← p − Φ(p)∇Φ(p). (69) The central difference approximation of the gradient may cause numerical dissipation in the computation of Φ(p) and ∇Φ(p). To alleviate this, the projection (69) can be applied twice, this does not make the procedure iterative. It is merely an implementation safeguard against numerical dissipation. Due to the distance-map properties, no more than two iterations is needed, this gives a constant time operation. 15 Results and Discussion We have chosen to verify and validate the signed distance-fields in the context of IK. We use the box-limit model as a base of reference, mostly due to its widespread use in interactive application and because it is the model currently used in human motion tracking. We have excluded the alternative methods from the tests, since none of them live up to both the time and modelling demands of our application. We focus on performance and accuracy. 15.1 Building the signed distance-maps In our tests, a brute-force approach was used for building the distancemaps from motion samples. Table 2 shows the timings for generating the signed distance-maps for the joint-constraints of a single jumping motion. 52 Figure 20: Comparison study of animation quality when using box-limits versus signed distance-fields. The figure on the left shows signed distancefields, the figure on the right shows box-limits. The figure in the center is the motion capture reference. Grid Resolution 8×8×8 16 × 16 × 16 32 × 32 × 32 # Motion Samples 200 200 200 Storage Requirement 4 kB 32,7 kB 262 kB Computing Time (secs.) 0.21 1.34 8.30 Table 2: Timings of brute-force approach for distance-map generation for 30 joints in a jumping motion sample. Observe that even for detailed maps, the processing time is acceptable as this is a preprocessing step. 15.2 Parametrizing the Distance-Field Joint-Constraints The signed distance-field model is dependent on the user specified grid resolution I × J × K and the ε parameter. The parameters are orthogonal in the sense that grid resolution mostly influences the signed distance-field generation, while the feasibility threshold parameter is a run-time only parameter. We have tested different grid resolutions and it turned out that for our single motion sampled signed distance-fields there seemed to be a threshold around a resolution of 16×16×16. For finer resolutions, the animations ran smoothly and with acceptable quality. For coarser resolutions, the generated motions were jagged and tended to get stuck. For sculpted signed distance-fields, which tend to be more connected (see Figure 19(b)), the resolution could be set much lower. Resolutions of 8 × 8 × 8 or lower were acceptable in this case. For the single motion case, the ε parameter needs to be large enough to ensure the existence of a solution for all poses, yet small enough to counter 53 loose constraints. We observed that for values above 0.2 radians, the constraints are too loosely fitted, and for values below 0.2 radians the motion is jittery and the IK solver tends to stall. 15.3 Constant Time Performance Figure 21: Computation times of box-limits versus signed distance-fields when used in inverse kinematics. The measurements are ordered by value to make comparison easier. Notice that the box-limits are generally faster but cannot guarantee the same upper bound on computation time as the signed distance-fields. We have measured the computational time in an application, where the end-effectors of an IK solver are driven by the corresponding end-effector positions in a motion capture example. The measurements are performed on the IK solution. Figure 21 shows a plot of our measurements. Our experiments show that the signed distance-fields are slower than the box-limits. This was to be expected. We can also see that the worst case performance of the signed distance-fields is much better than worst case for the box-limits. In the worst case the signed distance-fields still achieves approximately 20 fps, while the box-limits only reaches approximately 10 fps. 15.4 Increased Modelling Accuracy Using the more constrained signed distance-field model, we expect a gain in accuracy as this should reduce the redundancy of the simulation. The test uses an elbow joint, this joint is a child joint of the shoulder which has a wide rang of motion. We measured the deviation of the joint, using both 54 (a) Plot of error compared to motion capture. (b) Accumulated error over time. Figure 22: Plots showing the difference between the right elbow ’s global position using the different joint-constraints. The more restrictive jointconstraints cancel out the effect of the redundancy of the joint, thus getting a solution closer to the motion capture reference. 1 unit on the axis corresponds roughly to 20 cm in real world measurements. sampled, sculpted, and box-limits, shown in Figure 22(a). As expected, the results show that the box-limits does not constrain the position very much. The box-limits result in an error which, transformed into real world measures, corresponds to an error of 20 cm. The simulated motion is shown in Figure 20. 16 Conclusion We have presented a novel joint-constraint model using signed distancefields. The fitting of our model, although loose on account of the threshold, is 55 still the tightest fitting model, compared with the other presented models as seen in Figure 1. The model supports general non-convex joint-constraints of 3 degrees of freedom and works well with sampled motion data. The memory usage is cubic in the resolution of the mesh. However, in most cases it is possible to get by with a comparably low resolution, in which case the memory usage is acceptable. The assumption of locality has shown to be insufficient, e.g. the orientation of the hip joint indeed has an effect on the joint-constraints of the knee. Therefore, an interesting venue for further research would be to extend our model to handle more than 3 degrees of freedom. 56 Part III Activation splines The Joint Limits conclude the work I have done with motion planning. Now we move on to something even more interesting. How to actuate a body in a dynamics simulation, to actually do the motions, we have planned in the previous chapters. The first thing you need, when getting something to move, is a motor. The human body use muscles for actuation, so the development of a muscle model is the logical next step. To develop a muscle model, it is necessary to decide on the application of the model. Software packages, such as Maya, has inbuilt methods for animating muscles, but what I wanted was something more similar to the models used in bio-mechanics something that would have the possibility to replace the existing methods in software such as anybody but would still retain the interactive nature of simple linear actuators. The Idea was to let the actuator remain a one dimensional spline but embed it into a deformable body of arbitrary complexity. This way we get something that is potentially as complex or simple as is needed. The Work presented in this part is done in collaboration with Kenny Erleben. It is not yet published but a paper presenting this method has been submitted to the SCA 2012 Symposium on computer animation. The following is an extended and modified version of that work. 58 17 Activation splines Figure 23: A man flexing his arm. The simulation was made using 5 spline muscles in the shoulder and arm. We present a method for simulating the active contraction of deformable models, usable for animation of characters and soft deformable objects. We propose to embed splines into a deformable model to control deformation patterns. The splines model a type of contraction force that is transferred to the ambient space of the deformable model. We present a novel physical principle as the governing equation for the coupling between the low dimensional 1D activation force model and the higher dimensional 2D/3D deformable model. The coupling model is robust towards artistic force splines that might overlap and share their region of influence in the deformable model. Our activation force method works well with both massspring systems, finite volume, and finite element simulations. The contraction method we propose for our splines is shown through experiments to be robust and allows for rapid and responsive control of the deformations. Our activation splines are easy to set up and can be used for physics based animation of deformable models such as snake motion and locomotion of characters. Our approach generalises easily to both 2D and 3D simulations and is applicable in physics based games or animations due to its simplicity and low computational cost. Figure 24: A cartoon stool rigged with an activation spline in each leg. A simple impulse activation signal controlling the contraction of the leg activation splines produces a gait for the locomotion of the stool. 59 Figure 25: Our system is used to make a snake crawl on the ground. The snake contains a left and right activation spline running from head to tail. Two cosine functions with a phase shift control the contraction that activates the spline. A simple anisotropic Coulomb friction model is used to model friction between the snake’s skin and the ground. 18 Introduction Making deformable models move by themselves in computer animation, as though they have internal muscles, is a time consuming process both in terms of rigging and computational cost. Existing approaches work through external dynamic constraints acting between bodies like creating point-tonail constraints on the fly. Although an animator may control this, the approach uses an external fictitious view to create the actuation rather than an internal more physically correct view. Other approaches use accurate muscle models with a micro scale view of muscle fibres. These models are hard to come by as only specialists know how to create the correct fibre data and the models require a substantial computational effort. Thus, we believe such models are not easily adaptable by animators or feasible in an interactive application context. We present a method which allows animators to model activation forces with splines embedded in the passive mesh. Each spline affects a region of the mesh. The splines are intuitive, as they can be understood as tendency lines of the true continuum field of activation forces. Thus, our spline model reduces time consumption in rigging compared to a muscle based model. Existing physically realistic methods are computationally heavy, taking hours of wall clock time to simulate seconds. Our method is conceptually closer to the traditional line-based models such as Hill [63] and Zajac [64] . However, our approach naturally overcomes the difficulties in having to model via points to ensure proper physical correct bending moments. Our work allows for a natural coupling between the activation forces and the resulting deformations. This is ignored in most line-based modelling. Our spline model bridges the gap between the physically accurate, but computationally heavy, methods, and the classical biomechanics 60 models which are computationally efficient but typically lack deformation and collision support. Our novel contributions are, • A novel force equivalence principle for transferring coarse low 1 dimensional representation of activation forces to a fine high 2/3 dimensional representation of the activation force field. • A generalization of line-based force models to arbitrary smooth curved lines. • An orthogonal approach to extend line based force models to include a physical realistic passive deformation of the deformable models. • A unique robust and responsive method for contraction, which works by altering the rest shape of a spline, rather than tweaking physical parameters such as stiffness. We present parameter studies, demonstrating that our method is not sensitive to specific modelling of the splines and that complicated deformations are possible with the novel contraction parameter. Thus, our approach is suitable for artistic created models as it allows for overlapping splines and can transfer activation forces to all regions of the deformable model . Several animation examples show that the method produces robust animations, resulting in interesting motion with little rigging or animation time. Figure 26: An animation of a biceps muscle bending an arm. The simulation does simple collision handling and uses gravity to pull the arm down after flexing. Our method was not developed for biomechanics. However, as the example shows it can capture some of the real world muscle effects. 61 19 Related Work Early work in computer graphics focused on simple kinematic deformations [65] or cage-based deformations such as free form deformation (FFD) [66]. Later, energy constraints were introduced for adding constraints between parametrized models [67] and generalized [68]. A more physical approach used the finite element method (FEM) for simulation of deformable models and added constraints using Lagrange multiplier method [69]. Some of these early ideas have been optimized by precomputing of principal components [70] to build responsive dynamic textures [71]. There have been kinematic approaches to make a connection between deformation and muscle motion. An example of this is the approach by Aubel and Thalmann [72] in which a visualization mesh is wrapped around a spline and volume preservation is approximated, or Lee and Ashraf [73] which makes passive deformation of simple fusiform muscles. None of these kinematic or animation approaches handle the activation of the deformable models which is our main focus in this work. Activation of mass spring systems have been investigated previously in computer graphics where a mass-spring lattice/cage mesh is used for both activation forces, the deformable model, and handling of collisions [74, 75]. The cage ideas have been extended to use hierarchical FFD as the lattice [76] . These are all best classified as Eulerian approaches to the activation forces whereas ours is a Lagrangian approach. We embed lower dimensional 1D spline polylines into the 2D/3D deformable model rather than caging the deformable model. For the coupling between the two different physical systems we apply a physical principle to tie together the deformable model and the actuation model. No such physical principle is used in previous work[74, 77, 75, 76] as their cage holds both the deformable model and collision/contact response. Our contribution is based on a physical principle for coupling our activation model with the deformable model – no previous work is as far as we know based on a physical principle for this. Our approach allows the environment to interact directly with the deformable model rather than with a cage. Our approach also allow for any deformable model being massspring systems, finite element/volume methods or to be used together with our activation force model. Tu, Grzeszczuk and Terzopoulos [78, 75] use mass spring systems in the same manner as [74]. The focus in this work is on learning motion patterns for these types of mass spring models the focus on our work is on the actual activation force model. These authors vary their activation force springs using a linear interpolation between a prescribed fully contracted rest length 62 Figure 27: A children toy rubber lizard walking using 6 activation splines. Two are used to flex the spine and one is placed in each leg. The activation is controlled by phase shifted cosine functions. (3D model by Kevin Hays.) and the initial relaxed rest length in contrast we use a simpler linear scaling of the initial rest length. This allow our activation forces to contract to zero length if needed or even expand far beyond the initial rest length. Our choice seems more reasonable for purpose of general animation. Space-time optimization problems use a force field applied on the whole 2D/3D deformable model to actuate the deformable model and make it move in some desired manner [79, 80]. This means that any external force can be applied to a model to make it move into some desired shape. This is an unnatural activation force. Our approach models activation forces much more similar to the approaches taken for articulated figures where activation forces live in the joint space of the model [81, 19]. Hence for deformable models the control parameter vector scales linear with the number of vertices in the deformable model mesh. In our case we have as little as a single parameter for a whole embedded spline. Hence our control parameter vector scales linear in the number of splines we apply. Space-time optimization problems are not the focus of this work but we speculate that these could benefit from a low dimensional activation force model such as ours. Simulation of muscles has been researched in computer graphics, computational physics and engineering, and biomechanics communities for years. Viceconti et al [82] published a review of the current state of the art of modelling the musculoskeletal apparatus. They state that most full body simulators use linear muscle-actuators, ideal joints and infinitely rigid bones, and stress that this is not an optimal modelling, since it neglect the factors of muscle interaction and non-linearity of the force splines. In the review by Pandy and Andriacchi [83], the function of human muscles is extensively reviewed and several ways of measuring muscle activation are compared to current models. Computational modelling is described as a valuable tool in the analysis and explanation of human joint and muscle function. Most approaches use a FEM method [84, 85, 86] or a finite volume method (FVM) [87, 88] to model the muscle and use a three dimensional 63 vector field to represent the fiber direction. The fiber direction may be modeled from real life observations as a constant vector field. All of these approaches are off-line methods. In comparison, our approach simplifies the modelling of the direction of the activation force field in exchange for less computational complexity. Our method can handle branches of the deformable models and multiple force directions. More accurate muscle models in computer graphics includes Teran et al [87] which use B-spline solids to represent the fibre direction. Their approach is further developed and applied to a 3D muscle test case of the arm and shoulder complex in [88]. Their fibre representation is mixed with the constitutive equations for the muscle tissue. In contrast our work completely separates the activation force model from the constitutive equation of the deformable material. Hence, we do not need to worry about nonlinear elasticity, plasticity or volume preservation as these properties are determined from the deformable model simulator. Lastly, our work use 1D spline curves rather than spline solids that must cover the whole deformable model. This makes for a much more intuitive interface. In principle our model can be perceived as a graceful degradation of the spline solid. In the limit of adding infinite many 1D splines our model would become a spline solid. As our 1D embedded splines do not need to cover the whole region of the deformable model they are much more easy to rig. One major benefit of the 1D splines over the spline solids is the ability to handle multiple directions within the same tissue. The spline solid approach lacks this ability completely. Porcher-Nedel and Thalmann[89] used a polyline representing the action line of a muscle. The muscle surface is modelled using a mass spring system. The idea is refined by Aubel & Thalmann[72]. The polyline is moved using either pre defined behavior or a 1D damped mass spring system. The mass spring system is not unlike our activation spline. However, our splines are not restricted to follow the line of action but can be embedded anywhere in the deformable model, Thus providing much more generality and flexibility for an animator. Aubel and Thalmann use attractive and repulsive force fields to constrain the polyline and pre-defined behaviors – pinned guide points – for dealing with attachments and proper wrap around of the line of action around joints. In contrast our approach does not rely on force fields to deal with constraints or guide points instead we use the physical simulation of the deformable model to deal implicitly with all these issues. Thus, our approach requires much less modelling input and simpler rigging. During simulation Aubel and Thalmann updates local coordinate frames at points on their polyline then the muscle surface vertices are kinemati64 cally coupled to the motion of the polyline. In our work the coupling is fully dynamic and based on a physical equivalence principle. Further, in our work we address how to choose the mass spring parameters to yield rapid response and we introduce a control parameter for controling the dynamic behavior of our activation spline. Sueda et al. [90] describe a framework for modelling muscle and specifically tendon movement in hand animation, originally presented in [91]. It has some similarity to our approach.The fact that it uses a spline along which it activates and transfers force to the bones is similar. However, it does not have the two way coupling between the active spline and the deformable mesh. In teh work by Pandy and Garner [92], the focus is on modelling the muscle path, that is the path spanned by the centroid line of the muscle. In this way, it is similar to the piecewise linear Hill type muscles, but it extends the linear segments with curved segments. Their approach cannot handle muscles with fibres, not following the centroid line. The work by Pandy and Garner did not consider deformation – their work allowed classical line models to wrap around bones using via points – their focus is the obstacle problem for classical usage of 1D muscle models . Our work makes via points unnecessary as the wrap around effects would be handled implicitly by the deformable model simulation (contact forces between deformable model and rigid body model). We extend these existing approaches by presenting interactive behaviour, general non-muscle shapes, such as a stool, and intuitive rigging of the muscle fibre direction allowing animators to e.g. paint activation splines into a deformable model. 20 The Activation Spline Method We present a system consisting of a passive deformable model represented by a volume mesh and one or more splines, used to represent the activation forces in the deformable model. On a coarse level, our approach can be described as follows • As a pre-processing step, activation splines are discretized and embedded into the deformable model such that the splines will follow the motion of the deformable model once simulation starts. • During run time, our simulation loop’s first step is to compute activation forces on the activation splines. This can be viewed as a coarse low dimensional representation of the true activation force field in the 65 deformable model. A novel approach is taken to control the activation forces, where we change the rest shape of the activation splines rather than their material parameters such as elastic stiffness. • We formulate a novel force equivalence principle as the governing equation for the force transfer between spline and volume mesh. This allow us to transfer the coarse activation forces to the entire influence region in the deformable model. • We feed the whole volume mesh activation force field to a deformable model simulator as an external force field. This allows us to use any deformable model simulator as a black box simulator. Note that it is only an “implementation” trick to apply the activation forces as external forces. In reality the activation forces are internal forces in the whole model – Hence, our activation force model is a Lagrangian model. In principle this can be viewed as an interleaved simulation approach [93, 77]. • Having obtained a new state for the deformable model we update the spline position before we initiate the next step of our simulation loop. The simulation loop is sketched out in Figure 28. To ease notation, and without loss of generality, in the following, we will describe our method using a single spline. We assume that an animator or modeller have created a spline with K control points and any point on the spline can be found using p(s) = K � Nk (s)gk (70) k=1 where p ∈ RD with D = 2, 3 and Nk : R �→ R+ is the k th global basis function of the spline. The vector gk ∈ RD is the corresponding control point and s ∈ [0..L] is the spline parameter. The spline is assumed to be inside the volume mesh having V vertices, where xj ∈ RD is the coordinates of the j th vertex. The governing equation of motion for any deformable model can be written abstractly as M d2 x dx +C + k(x − x0 ) = Fext 2 dt dt (71) where M is a mass matrix, C is a damping matrix and k is the elastic forces that depends on the current displacement field x − x0 . For linear elastic 66 (a) (b) (c) (d) Figure 28: Illustration of our pre-processing phase and simulation loop. As a pre-processing step (a) we bind a spline to a volume mesh. During the simulation loop we first compute spline forces (b), afterwards we transfer the forces to the volume mesh using neighbourhoods (c). Finally we update the deformable model and move the embedded spline (d). materials there is k = K(x − x0 ) where K is the stiffness matrix. The vectors x and x0 are the concatenation of the current and initial (rest) mesh vertex positions respectively. Thus, x, x0 , k ∈ RDV and M, C, K ∈ RDV ×DV . Fext ∈ RDV is the concatenation of external forces. In our simulations, we mostly use gravity which in 3D means Fext,j = (0, 0, −g)T , ∀j = 1..V (72) where g is the gravitational acceleration. We have simple ground reaction forces incorporated which work by inserting a spring force, but without loss of generality we do not describe these here. During a simulation, k is the elastic passive forces coming from the deformable model itself. The activation spline forces will be coupled to the model by adding a right hand side force term, Fact ∈ RDV . We will later add a contact force term to the right hand side, Fcon ∈ RDV . The resulting equations of motion to be time integrated is written as M d2 x dx +C + k(x − x0 ) = Fext + Fact + Fcon 2 dt dt (73) In the following, we will detail the steps of the simulation loop and how the forces in the governing equation can be computed. 67 20.1 Discretization of the Activation Spline For simplicity we choose to discretize a spline into a sequence of N discrete spline points pi and corresponding line segments li between two consecutive discrete spline points i + 1 and i. We label the discrete spline points by p1 , p2 , p3 ,. . . , pN where pi ∈ RD for all i. The spline has N − 1 line segments and the length of the ith segment is given by li =� pi+1 − pi � (74) For simplicity, we discretize the spline equidistant in parameter space. Thus if s = [0..L] then � pi = Nk (si )gk for i = 1..N (75) k where Nk : R �→ R+ is the k th global basis function of the spline and gk ∈ RD is the corresponding control point and si = (i−1)L . For our test examples N −1 this has been sufficient. One could have taken curvature of the spline into account to create a more adaptive discrete version of the spline. 20.2 Embedding the Activation Spline We embed the discrete spline into a volume mesh by binding the spline points to a volume mesh using mesh coupling. We use barycentric coordinates for an embedding tetrahedron or bi-/tri-linear interpolation for a square or cube mesh [94, 95]. Given a volume mesh with vertex positions xj ∈ RD for j = 1 to V , we write � pi = w j xj (76) j∈C(pi ) where C(pi ) is the vertex index set of the mesh cell (triangle, tetrahedron, square, or cube) that contains the point pi . The values wj ∈ R are the interpolation weights. At initialization, after an animator has defined the spline and its influence region, the spline is discretized. Then, all wj ’s for each spline point can be precomputed and stored as attributes for each spline point. During run time the above equation can be used to re-compute and update the current position of the discrete spline point. When pi lies on the border between two elements, simply pick the element at random. If pi is outside the mesh then pick the “closest” element. Barycentric coordinates may be used even when pi is outside the element. For squares and cubes similar ideas can be used with little modification. 68 20.3 Computation of Activation Spline Forces We compute the spline forces at the discrete spline points. This is done by creating a linear damped spring for each line segment of a spline. Let ei ∈ RD be the unit direction vector from xi to xi+1 then the spring force on xi+1 is aii+1 = −ki (li − αi l0i ) ei − ci ei eTi (vi+1 − vi ) (77) and on xi we have from Newton’s third law of motion aii = −aii+1 (78) where the spline point velocity is given by vi = dtd pi for all i. Further, ki , ci ∈ R+ are the stiffness and damping coefficients respectively. Here li is the current length and l0i is the initial rest length pre-computed during the binding process both given by (74). Note that ei = pi+1li−pi . We have extended the standard spring damper law with the parameter αi ∈ R+ . This is our contraction parameter which we use to control contraction and extension of the spline. When 0 < αi < 1 then the ith segment of our spline contracts. If α > 1, we have an extension and for α = 1 the activation spline is passive. We can now compute the spline force at the ith spline point as fi = ai−1 + aii (79) i If a spline force is wanted for an arbitrary s-value the spline basis functions are used to interpolate the wanted value. In our examples, we control the α-values. In some of our test cases, we use a single α-value for an entire activation spline as in the example of the stool legs shown in Figure 24. In other cases, we use a functional expression to control the contraction of each segment as in the case of the snake in Figure 25. Our initial experiments indicated that we obtained much better controllability of the activation splines by manipulating the αparameter rather than trying to change ki and ci on the fly. This makes good sense as varying ki and ci on the fly would change the overall material. This would subsequently affect the time integration method, possibly resulting in a stiff system, which is unsuited for interactive simulations. We choose the spring and damping coefficients to create a critically damped spring. This is motivated by behaviour of real human tissue. We follow the approach by Barzel and Barr [68] and use 1 2 and ci = (80) 2 τ τ where τ is the characteristic time. In Erleben et. al[96] it was shown that if the deviation of the spring length from rest length should be within a ki = 69 fraction 0 < ε � 1 after the frame time, characteristic time intervals must be used. Thus, τ should be given by τ = ∆t . We apply this formula to n control how fast the spline forces should be able to contract to a given deviation within a frame time-step. Our experiments showed that a value of � < 0.01 gave visually satisfying results. From our 2D experiments using mass-spring systems for our deformable models, we observed that a spline should be at least twice as stiff as the embedding deformable model to be able to provide a rapid response in a controllable manner. 20.4 The Force Equivalence Principle We will introduce our novel force equivalence principle between a force on a spline point p(s) and the volume integral of the activation force F(x) in a neighbourhood N ⊂ RD around the spline point, � 1 f (s) = F(x) dV. (81) VN (p(s)) x∈N (p(s)) where VN (p(s)) is the total volume of the neighbourhood N (p(s)). Thus, f (s)can be interpreted as the average force over the entire neighbourhood. We will use this single principle to create a discrete mapping between activation forces at discrete spline points and discrete vertices in the ambient mesh where the spline is embedded. This single equation forms the main contribution of our model. We discretize the spline parameter using our discrete spline points � 1 fi = F(x) dV. for i = 1..N (82) VN (pi ) x∈N (pi ) We then define the discrete counterpart Ni of N (pi ). For our FEM simulations, like the muscle animation in Figure 26, we have added all vertex indices in the mesh to Ni , where the corresponding vertex is closer to pi than any other pm for m �= i. When computing the distance between a spline point pi and a mesh vertex xj care must be taken to respect the boundaries of the deformable model. The shortest path inside the volume of the deformable model should be chosen. This is important in order to avoid an activation spline in one tentacle of an object affecting a different tentacle of the object. Our method is robust towards the neighbourhood definition and allows for overlapping neighbourhoods. Neighbourhoods could even be defined by animators using a painting tool such as the ones used for skinning. For the 2D simulations using a square, 70 (a) (b) (c) Figure 29: Illustration of different ways to define the neighbourhood set for different types of meshes.(a) A closest point set, based on geodesic distances inside the volume mesh. (b) A simpler grid based definition. (c) A region defined by a manifold mesh, a distance measure or a function. mesh we define neighbourhoods to be the four enclosing mesh nodes. For the 3D simulations using cube meshes, we use the eight enclosing nodes. The different neighbourhood definitions are illustrated in Figure 29. Using the discrete vertex index sets Ni and the mid-point rule approximation, we can rewrite the integral of (82) as a summation over the discrete volume elements 1 � fi = Fj ∆Vj , for i = 1..N (83) VNi j∈N i ∆Vj is the volume associated with the j th vertex of the volume mesh. Given a lumped mass matrix M, we get Mjj = mj ID×D , where ID×D is the Dby-D identity matrix and mj is the mass of the j th vertex, Mjk = 0 for j �= k. For a material with � a homogeneous constant mass density, ρ, then m ∆Vj = ρj and VNi = ρ1 j∈Ni mj . For lumped mass matrices, the following is equivalent to (83) � j∈N mj Fj fi = � i , (84) j∈Ni mj In all our examples we used (84). We can state the problem as a system of 71 equations f1 F1 .. .. . = A . . fN � �� � (85) FV � �� � y b Each block of A contains the mass ratio coefficients defined by (84). A sub block of A is given by � mj � ID×D If j ∈ Ni , j∈Ni mj Aij = (86) 0 Otherwise Once we have solved for the unknown y, we have Fact = y. In general, A will not be a square matrix. This depends on the resolution of the discrete spline and the resolution of the volume mesh. In most cases, A will have more columns than rows. We have Ni �= Nj when i �= j, this means A can be assumed to have full row rank. If all index sets Ni are disjoint, meaning Ni ∩ Nj = ∅ for all i �= j, the matrix system Ay = b can be broken down into N smaller independent systems. These can be solved in a naive parallel manner. This is the case for the walking stool example. In general, we have an under-constrained system and we solve this by adding the constraint that we want to minimize the norm of Fact 1 F∗act = arg min FT F F 2 s.t. AF − b = 0 (87) The first order optimality (KKT) conditions result in the saddle point problem � �� ∗ � � � IV D×V D AT Fact 0 = , (88) A 0 −λ∗ b � �� � KKT where λ is the Lagrange multiplier for the solution and IV D×V D is a V D dimensional identity matrix. The KKT-matrix is a square symmetric indefinite matrix of full rank. Using a Schur complement method, the solution is found to be � �−1 F∗act = AT AAT b. (89) � �� � ∗ A† The pseudo-inverse matrix, A† , can be pre-computed during initialization and allows for a fast run-time solution. However, the pre-computation can be even more effective, since (88) may be solved using an iterative solver such as preconditioned conjugate 72 gradient method (PCG). This is very fast as A is usually very sparse. This method scales linearly in the number of mesh vertices, both in memory usage and in computational cost. The drawback is that the approximate solution will not solve (85) exactly. This can cause visible ghost forces if the solution is not sufficiently accurate. Rather than using a direct method for minimizing (87), a gradient descent method can be used. We note that the gradient of the Lagrangian function L is ∇F L = F − A T λ ∇λ L = b − AF (90a) (90b) λk+1 = λk + ∇λ Lk = λk + b − AFk (91a) We need to maximize wrt. λ and minimize wrt. F. This leads to the iterative updates Fk+1 = Fk − ∇F Lk+1 = AT λk+1 (91b) λ1 = b (92a) F1 = A T b (92b) Letting λ0 = 0 and F0 = 0, the first iterate yields If AAT ≈ I little improvement can be gained from more iterations. Thus, we have the approximate solution (93) F∗act = AT b. It is worthwhile noting that using this approach, even if we do not get an accurate solution during the first integration step, the method will converge to the optimal solution over time. For our 2D and 3D simulations, using square and cube meshes, we have applied this gradient solver technique with great success. Note that the condition AAT ≈ I bares similarity with the Galerkin condition for algebraic multigrid methods [97]. The matrices A and AT can be viewed as restriction and prolongation operators in a multigrid method for computing activation forces. If more than one activation spline is used, we obtain a linear system for each spline. We have to solve b1 A1 F1 .. .. .. (94) . = . . S S b A FV � �� � � �� � � �� � b� A� 73 y where we have S splines and each bi and Ai are defined as in (85). Observe that we still have the same number of unknowns, but our system now has more rows than previously. In principle, if we add enough splines, the linear system will become over-constrained. However, for all the examples using multiple splines, like our muscle simulation in Figure 26, the system is under-constrained and we apply all of the above solution techniques. It is known, from real muscles, that their activation cannot move the center of mass of an isolated system, the total activation force of a muscle must sum to zero. This is obviously true for a single deformable object floating in space. This is also true for more complex systems like an astronaut in outer space. It is only through interaction with the � � environment that objects move. In our model this means i fi = 0 and Vj=1 Fj = 0. Furthermore, one should guarantee that the total torque of the�splines equals �V the total torque of the mesh activation force field. That is, i ri × fi = j=1 rj × Fj where rj is vector from some origin to pi . Similarly, rj is a vector from that same origin to xj . By construction, our spline model fulfils these criteria. If some other activation force model is used, it should be ensured that these criteria are fulfilled. All the criteria can be expressed as linear constraints for (87). Thus, our method can easily be extended to other activation force models. 20.5 Contact Forces Our method generalizes to any collision handling strategy. For our proof-ofconcept simulations however, we have chosen a simple penalty force simulator. We have a single deformable model that can collide with a ground plane. The contact force for any mesh vertex above the plane is zero. If a vertex is below the plane, we apply a penalty force in the normal direction of the plane, proportional to the penetration distance. Let n ∈ RD be the unit outer normal for the plane and let q ∈ RD be an arbitrary point on the plane. The velocity of the j th vertex is given by uj = dtd xj .The penetration depth is then computed as dj = (xj − q) · n The normal force is nj = � 0 −kdj − cnnT uj if dj ≥ 0 if dj < 0 (95) (96) where k and c are the stiffness and damping coefficients, which we set using the same principles as in Section 20.3. If there is a non-zero tangential 74 velocity (97) uT,j = uj − nnT uj we add a friction force according to Coulombs law of friction, opposing the sliding direction uT,j tj = −µ � nj � (98) � uT,j � where µ > 0 is the scalar coefficient of friction. In some cases (like in our snake simulation) we change the value of µ depending on the direction of uT,j . The total contact force on the j th vertex is (99) Fcon,j = nj + tj 21 Validation 23 22 Node displacement 21 20 19 ratio = 1.0 ratio = 1.25 ratio = 1.5 ratio = 1.75 ratio = 2.0 ratio = 2.25 18 17 16 15 14 0 1000 2000 3000 4000 5000 6000 Simulation steps Figure 30: An example of the effect of changing the ratio between active and passive stiffness constants. The passive stiffness is kept constant and the active stiffness is changed. To validate the method, we have implemented a proof-of-concept simulator in Matlab for both 2D and 3D. This system uses simple mass spring systems for the passive mesh, where spline points are assigned to cells in the mesh as shown in Figure 29(b). This makes it easy to assign splines and 75 setup simulations. We also implemented a C++ version of a co-rotational linear FEM based on tetrahedral meshes. This version supports both the assignment strategies shown in Figure 29(a) and 29(c). 0.3 Displacement of mesh nodes in x direction 0.25 0.2 0.15 0.1 0.05 0 −0.05 −0.1 −0.15 −0.2 0 20 40 60 80 100 Frames Figure 31: A plot showing the near critically damped response, given contraction of two overlapping splines at different times during a simulation. The plot shows the displacement of nodes in a mesh over time, during a contraction sequence similar to the one shown on the left in Figure 33. We performed extensive parameter studies to investigate the effect of changing the damping and spring stiffness ratios between activation force and passive response. We have included a few of the results. Figure 30 shows a study of the effect of changing the ratio between passive and active stiffness. The passive stiffness is kept constant, while the active stiffness is changed. The simulation mesh is first contracted actively at simulation step 1000 and then released at step 3000. It can be seen that it is important to find a ratio which gives a rapid and effective contraction but does not cause too long oscillations after release. We have used the ratio 2:1 for all our tests. Note that the oscillations are not present in the final simulation as explained in Section 20.3, and shown in Figure 31. A number of simple tests have been performed to validate the correctness of the activation spline model as well as the generality of the activation 76 Figure 32: A time lapse of a 2D table walking, by pushing of from the ground and elongating or contracting its middle part. force spline model. These have been designed to address both varying and multiple activation force directions, in both 2D and 3D. Figure 33: Test case for overlapping splines. The region of the first spline is shown in Yellow, The region of the second in blue, while the overlap is colored green.Notice that even though both bodies undergo rapid deformation no linear or angular momentum is induced. Figure 31 shows a plot of the displacement of the mesh nodes in the x direction frame by frame. Even though the mass spring system imposes secondary oscillations, the response when the contractions are initiated in frames 20 and 60 is close to instantaneous and oscillations are minimal. Likewise, when the spline is relaxed in frames 40 and 80. Figure 33 shows one such test case, where two cylinders is fitted with two overlapping splines and activating one after the other. One is overlapping in the direction of contraction the other orthogonal to this. The colors show the region of influence for each spline. yellow for the first spline blue for the other. The green region is the overlapping region. First the blue region is contracted, then the yellow and finally both are contracted. the leftmost image show the relaxed state. Figure 34 show the effect of larger influence regions. In this test the the three cylinders where fitted with identical splines but each was given a 77 different size of influence region. the red one was given an influence radius of 0.2 the green 0.4 and the blue 0.6. This meant that for the green and blue cylinders there was an overlap of the influence of the points in the spline. this turns out not to be a problem since it smooths out the force influence of individual spline points. The fundamental constraints are still upheld and notice in particular that the overall contraction is the same for all three cylinders. This makes sense since the splines are identical and thus, induce the same force on the mesh. Figure 34: Three different settings for the size of influence region the red cylinder has the smallest influence region while the blue has the largest.Notice how the larger influence region smooths out the contraction as spline points share mesh nodes. Furthermore, we have designed a number of simulations to test the capabilities of the system. In 2D, we have constructed a walking table (Figure 32). In this case, the deformable model was propelled forward using the activation splines. The table comprised two splines for pushing the legs of from the ground and one spline to elongate and contract the “back”. As it turned out the resulting locomotion was surprisingly effective, In 3D, we designed a number of simulations. To show the ability to make simple animations, a cartoon stool was rigged with activation splines in the legs. By simple impulse contractions, the stool was made to walk along the ground as shown in Figure 24. This locomotion was very similar to the one in the 2D case but since we now had four legs we could use these instead of having to contract the seat of the stool. This was animated using our matlab version of the system. Figure 26 shows a simple muscle animation made with the C++ framework. The simulation consists of simple collision handling and two way constraints between deformable bodies and rigid bodies. It is possible to flex the arm and let it fall back to a stretched position under gravity. A snake crawling along the ground was made using one spline in the left side from head to tail and one in the right side. The splines were activated using a phase shifted cosine function. This resulted in a surprisingly realistic behaviour for such a simple setup. Figure 25 shows a frame from 78 the animation. Anisotropic friction was used to simulate the properties of snake skin. This animation was further developed to show the control it is possible to gain over the meshes. We let three snakes crawl and deform to spell three letters, Figure 35 shows the result of this animation. Figure 35: A time lapse of snakes spelling “SCA”. Even though the animation is based on a physical simulation, explicit control of the poses of the bodies can be obtained. The visualization of the 3D simulation meshes has been done using embedded surface meshes which are then exported and rendered separately Using the Blender rendering engine. The 2D versions show the simulation mesh directly. Figure 24 and Figure 27 shows examples of animations where unlikely objects are articulated using the spline activation. Figure 27 shows a toy rubber lizard actuated using 6 splines. the actuation makes the toy come to life and waggle its way forward. The rubbery secondary effects of the passive mesh are a biproduct of the fact that this is a deformable model simulation and not a scripted animation. Similarly the walking stool in Figure 24 shows how an inanimate object can be brought to life. even though a very simple impulse activation is used the stool actually moves along the ground in a controllable way. Our final test case is a human bending his arm. For this test we needed several new features. In contrast to rubber toys, humans have a skeleton which constrain the deformation of the soft parts. This was included by manipulating the stiffness of individual tetrahedrons in the simulation mesh. A skeleton was modelled and embedded into the mesh, where the skeleton mesh and the simulation mesh intersected the stiffness of the affected tetrahedra was increased 250 times. Figure 36(a) show the skeleton used. We used a comparatively fine grained mesh for this simulation, comprising 6902 tetrahedral elements. The figure was fixed at the pelvis so as not to fall, since we have no balance control in this simulation. Three muscles, the biceps the deltoid and the trapezius was activated to bent, raise and lower the arm. It proved slightly more difficult to control this particular simulation due to the rather thin limbs of humans. However even without a control parameter the figure flexes and releases his arm. Figure 23 shows 79 (a) (b) (c) Figure 36: (a) shows the skeleton used for the human simulation. The stiffness of the tetrahedra which comprised the skeleton was set 250 times higher than the surrounding mesh. (b) shows the tetrahedral mesh used in the simulation. (c) shows the influence region of the five muscles overlaid on the visualization mesh. 80 four frames from this simulation. In Figure 37 the noticeable bulging of the biceps muscle under activation is clear. Although no user interface was developed for the system, all simulations were set up in a matter of a few hours, at worst. This included creating and rigging the simulation mesh. Figure 37: The bulging of the biceps muscle under activation is clearly seen in this animation 21.1 Performance measurements To gauge the speed of the system we performed several tests with different mesh resolutions, using the c++ framework. We used a tetrahedral mesh cýlinder with mesh resolutions varying from 128 to 4011 elements. The cylinder was contracted for 1000 frames and the median values as well as the 1st and 3rd quartiles was plotted for each resolution. The results can be seen in Figure 38 As can be seen the time spent was very stable and didnt exceed 60 ms/f r. for any of the test. This timing does not tell so much about our method as it tells about the FEM solver. The timings shown are for a full simulation pass including visualisation. The actual spline activation is app. 0.5 − 1.0 percent of the full simulation time and ranges from 0.02 to 0.3 ms for the shown mesh resolutions. Figure 39 show the time spent on the activation, which is much less and never exceeds 0.4ms/f r for the tested meshes. Further we measured the performance when changing the number of splines in the mesh. we made measurements for 1-40 splines with an interval of 5. Figure 40 show the results of these measurements. Since the activation matrix can be precomputed and the number of splines are usually much lower than the number of mesh nodes, the number of splines has negligible effect on the overall time spent. As can be seen from all these timings the systems runs interactively for even moderately large tetrahedral meshes in our c++ implementation. 81 Execution time in ms for 1 frame 60 50 Upper Quartile Median value Lower quartile 40 30 20 10 0 0 1000 2000 3000 4000 Number of tetrahedral elements in mesh Figure 38: The timings for contraction of a single tetrahedral mesh cylinder with varying mesh resolution. a single spline was fitted to the mesh and contracted for 1000 frames. The plot shows the median and upper and lower quartiles for different mesh resolutions Further refinements might give even better speed, but since the activation spline method takes up such a comparatively small amount of the time we have not investigated this further. It would seem that With our method the limiting factor is always going to be the passive mesh simulation and possibly the collision detection algorithm. It would seem that the time is better spent optimizing these than tweaking the already fast activation method. 22 Conclusion and Future Work We have presented a novel approach for describing and modelling the activation forces of deformable models, based on 1D splines. We have demonstrated the method in several scenarios, both 2D and 3D, to show how it is possible to make animations using the actuation splines. No explicit control scheme is present, but it is still possible to explicitly control posing due to the straight forward and intuitive relation between the spline and the mesh. 82 Execution time in ms for 1 activation pass 0.5 Upper Quartile Median value Lower quartile 0.4 0.3 0.2 0.1 0 0 1000 2000 3000 4000 Number of tetrahedral elements in mesh Figure 39: The timings for activation part of simulation. The plot shows the median and upper and lower quartiles for different mesh resolutions. Notice that the time never exceeds 0.4ms/f r As we have demonstrated, our method presents an attractive compromise between precision and realism on the one hand, and speed and generality on the other. Activation splines are an intuitive way of extending the simple line-actuator to a deformable contraction spline. These follow the deformation of the surrounding mesh, and are influenced by collisions and attachments to other bodies. A number of promising avenues for extending the presented ideas are obvious. Regarding the muscle simulation case, the current version uses a hand modelled skeleton. It would be interesting to use real scanned data and actual measured parameters to further refine the model and show its applicability outside the animation world. Control using key framing, or similar techniques, would make the system easier to use for animators. We speculate that short horizon space-time optimization or inverse dynamics schemes might be possible ideas for this. Our work focused on allowing animators artistic freedom to create activation splines in. However, to assist animators one could consider creating tools that could automate some of the spline creation work. We speculate that medial surface representations or curve skeletons could be used to 83 Calculation of activationforce. ms for 1 frame 0.1 0.098 0.096 0.094 0.092 0.09 0.088 0.086 0.084 0.082 0.08 0 10 20 30 40 50 Number of active splines in mesh Figure 40: The timings for running simulations with varying number of splines. As it can be seen the number of splines have very little effect on the simulation time. seed 1D splines [98]. Even with our simple interface it was comparatively straight forward to rig the activation splines and their influence regions but with more time this could be refined considerably, making the activation splines a valuable tool for animation and simulation. 84 Part IV Optimal control using space time optimization Control of synthesized motion is something I have not yet touched upon. Well that may not be entirely true, the motion planning in itself, is an example of a simple form of control. It does not take forces into account though. Therefore, some other scheme is needed to get from the purely kinematic motion to a dynamic simulation. In short we need something that can calculate the forces from a kinematically driven motion. If we where talking about rigid body dynamics, a simple inverse dynamics scheme might suffice. Using that, we could plan the motion, with the inverse kinematics method from section 3. Then, we could calculate the necessary forces, using an off the shelf inverse dynamics solver such as the one used in the Anybody system [99] or in opensim [100]. When it comes to controlling the motion of deformable objects however, there is no such easy road. Equally important is the fact that inverse dynamics only works if there is a pose available in every time step of the simulation, which makes optimizing motions hard. We wanted something that could both, handle deformable objects, and calculate the forces necessary, for a given motion, with only keyframes as the input. Space Time optimization does exactly that. The following section describes our first ideas and our results for simple spring connected particles. The theory extends easily to deal with more complex structures and our first promising results makes us believe this is indeed a feasible way of solving the problem of optimal control for deformable objects. The research in this last part of my thesis is not published yet. The plan is to publish this as soon as we have sufficient results to be sure the method works robustly. Our initial results are good enough for us to believe this is the case. 85 23 Introduction to Space Time Optimization In the following we will introduce the Adjoint method for space time optimization of deformable bodies. The space time optimization problem is, by design, stated as an optimization problem and hence our challenge is that of finding a computational cheap way of computing the gradient of the objective function. For this we will use the Adjoint method. This makes it possible for us to reduce the number of unknowns from n2 to n. Further we will take advantage of the sparse block structure of the resulting equations to yield an O(n) algorithm for computing the gradient. We present our modification of the adjoint method for solving the space time optimization problem for deformable bodies the method may be extended to spline activated softbodies as the ones described in the previous part, by constraining the possible forces to the subspace given by the force splines, and incorporating the activation mapping in the calculation of the active component of the gradient. 24 Previous Work In this section we will give a short introduction to space time optimization. this is by no means a full description of the many methods developed for control of articulated mechanisms. we only seek to give the reader a background for appreciating the following. In 1988 Witkin & Kass [81] introduced space–time constraints to the graphics community for articulated rigid bodies. Here motion control was solved as an optimization problem. The gradient of their objective function was found using automatic numerical differentiation. Witkin & Kass never experienced problems with discontinuities as their simple examples allowed for analytical solutions (they never used the Adjoint method) and contact was modelled as explicit constraints. Later, McNamara et. al. [101] applied the Adjoint method for optimal control of a fluid solver. McNamara et. al. did have discontinuities when dealing with their level set surface and did report some problems from the collision of a clay blob. Wojtan et. al. [79] used the Adjoint method for particle systems and cloth. They did report that the collisions could be a problem but showed examples that handled the cases. The Adjoint method was applied for deformable models in [80]. The authors used per vertex control forces. However, they applied a reduced coordinate method to the deformable models and they created a technique for obtaining a good starting iterate. The work goes into detail 86 on how to create a reduction basis that actually contain the deformation modes of the key frames. Collisions and impact are dealt with using bilateral contraints this is similar to what Witkin & Kass did [81]. 25 The Space Time Constraint Problem We will now explain how the Adjoint problem is related to the space time constraint problem. Assume we are given a state vector of some system q ∈ Rm . For now we will abstract over the details of the specific system and keep things on a general abstract level. Later we will explain explicitly how our state vector is defined. Using some forward simulation routine we advance the state from some time t to the time t + ∆t. We write this abstractly as qt+∆t = Ft (qt , αt ) (100) where αt ∈ Rs is the value of the control parameter vector for the tth simulation step. The function F(·, ·) is the actual forward simulation that is being run. However, for all purposes we can just think of this as a function that takes a given input state and input control parameter vector and produces a resulting final output state. If we are given the initial state of the system q0 as initial conditions and then after N simulation steps we will have computed qN . Introducing matrix vector notation we may stack all the elements from (100) and write one system of equations as qN FN −1 (qN −1 , αN −1 ) .. .. . . = (101) 2 1 1 1 q F (q , α ) q1 F0 (q0 , α0 ) � �� � � �� � Q F(Q,a) where Q ∈ Rn with n = N m, F : Rn × Rk �→ Rn with k = N s and we have defined the stacked control parameter vector a ∈ Rk as a= �� α � N −1 T 0 T . . . (α ) �T (102) Definition 25.1 The simplest form of the space-time constraint problem can now be stated as the problem of finding a solution a∗ for the control parameter vector a such that, given the initial state q0 , after N simulation steps, a∗ will produce a motion such that, a given wanted final state qgoal is reached. 87 In principle there may be many solutions for this problem hence to make the problem solvable we write the problem mathematically as a nonlinear optimization problem a∗ = arg min φ(Q, a) (103) a such that Q − F(Q, a) = 0 and where φ(Q, a) ≡ 1 µ � a �2 + � qN − qgoal �2 2 2 (104) where µ ∈ R+ is a user-specified penalty parameter. This is a penalty method that seeks to minimize the control effort given by the first term, subject to the penalty constraint (the second term) of meeting the prescribed goal state after N simulation steps. The parameter µ can be used as a tradeoff parameter between minimizing the effort or reaching the goal state. To effectively solve such a minimization problem we are interested in being able to compute the gradient dφ . This is the computational problem da we need to solve. The specific choice of objective function (104) is based on current practice in computer graphics [81, 101, 79, 80]. In principle one can alter and change the objective function depending on specific needs.It may seem strange that the penalty term is associated with the state term and not the control parameter term. The answer is obvious as the goal state may be a physically unrealisable state due to the fact that it can be a user specified input(e.g. a pose specified by an artist). It is therefore quite possible that we are in the situation where no solution for a exist. A penalty formulation allow us to solve the problem even in this case. One generalization of the simple space time optimization problem may be to introduce multiple goal states along the motion path (also known as key frames). This would change the objective function into N 1 µ� φG (Q, a) ≡ � a �2 + wi � qi − qgoal,i �2 2 2 i=1 (105) Here we have introduced the weights wi ∈ [0..1]. These can be used to control the importance of the individual key frames. In particular using wN = 1 and wi = 0 for all i �= N we recover the simple form in (104). Similar optimization formulations exist in other fields such as image processing. Here people often use 1-norms and a kind of regularization of the a-term that prevents it from changing too much from frame to frame. 88 This leads to the following formulation. φI (Q, a) ≡� a �21 k � β � �� k + a − ak−1 � 2 j=2 N µ� + wi � qi − qgoal,i �2 2 i=1 (106) The 1-norm may be a numerical advantage as it often favours sparse solutions. The rate limited constraint can be controlled through the weight parameter β ∈ R+ . The rate limitation has some physical interpretation imagine a representing the power input for an engine, or the activation of a muscle. In real life both muscles and motors have a certain maximal activation speed. For the muscle, this is the time for the nerve signals to reach the muscle fibres, and the chemical process to take place. In the motor it is the time needed to accelerate. From an optimization viewpoint the term is attractive as it couples the components of a. Intuitively speaking the extra term reduces the possible search space of a making the optimization problem “easier” to solve. 25.1 The Adjoint Method We will start by introducing the Adjoint problem on an abstract level. We want to compute the value of gT B for a known g ∈ Rn and an unknown B ∈ Rn×k such that the relation AB = C always hold for a given known A, ∈ Rn×n and C ∈ Rn×k . The difficulty here is that B is a matrix of n × k unknowns. We write the problem as gT B such that AB = C (107) The Adjoint problem is then given by rT C such that AT r = g (108) where r ∈ Rn is the unknown Adjoint vector. Observe that � �T rT C = rT AB = AT r B = gT B (109) Thus, rather than solving for B and compute gT B we can solve for r,reducing the number of unknowns from n2 to n unknowns – and compute rT C instead. 89 25.2 Computing the Gradient using the Adjoint Method From the chain rule we get that dφ ∂φ dQ ∂φ = + da ∂Q da ∂a (110) ∂φ dQ The first term ∂Q is the tricky part as the part dQ is very difficult to da da compute analytically for large complex systems. The second term ∂φ can ∂a usually be obtained analytically with little difficulty and as such pose no real difficulty. For our specific objective function (104) we find that ∂φ =a ∂a (111) We know Q − F(Q, a) = 0 always hold for all values of Q and a. Thus, the gradient must always be 0 This can be rewritten as d (Q − F(Q, a)) = 0 da (112) � (113) ∂F I− ∂Q � dQ ∂F = da ∂a The problem of computing the gradient can now be stated as the problem of computing dφ ∂φ dQ ∂φ = + (114) da ∂Q da ∂a such that � � ∂F dQ ∂F I− = (115) ∂Q da ∂a It seems we have obtained nothing but making the problem more complicated but Defining: ∂φ ∂Q � � ∂F A = I− ∂Q dQ B = da ∂F C = ∂a gT = 90 (116) (117) (118) (119) (120) we have dφ ∂φ = gT B + such that AB = C (121) da ∂a From this we immediately recognize that we can apply the Adjoint method and instead solve the Adjoint problem of computing rT C such that AT r = g which is ∂F r ∂a T such that � ∂F I− ∂Q Afterwards we may compute the gradient as �T r= (122) ∂φ T . ∂Q dφ ∂F ∂φ = rT + da ∂a ∂a (123) (124) We may now summarize the computational method for computing the gradient Step 1: Calculate and assemble ∂F ∂F ∂φ , , , ∂a ∂Q ∂Q and ∂φ . ∂a Step 2: From the right hand side of (123) solve for r such that r= ∂F T ∂φ T r+ ∂Q ∂Q (125) Step 3: Finally compute the gradient dφ ∂F ∂φ = rT + da ∂a ∂a (126) This is the general recipe which we may now apply some optimization for where we take advantage of sparsity patterns. Note that r depends on itself, but bear with us a moment then we will explain how this can be. 25.3 Exploiting the Sparse Block Matrix Patterns For our specific choice of (104) we find that � � �T ∂φ µ qN − qgoal if i = N = i T ∂q 0 otherwise 91 (127) Thus � � ∂φ ∂φ ∂φ ∂φ · · · = ∂q N N −1 1 ∂q ∂q ∂Q � � � = µ qN − qgoal T 0T · · · 0 T � (128) and ∂φ = a. Further, we observe from (101) that the partial derivative of ∂a the forward simulation with respect to the state has the block structure N −1 0 ∂F 0 ... 0 0 ∂qN −1 ∂FN −2 0 0 ... 0 0 ∂qN −2 ∂F . .. .. .. .. = .. (129) . . . . ∂Q 1 0 0 0 . . . 0 ∂F ∂q1 0 0 0 ... 0 0 Notice that only the first upper band blocks are non-zero. The partial derivative with respect to the control parameters give the block diagonal structure ∂FN −1 0 ... 0 0 ∂αN −1 ∂FN −2 0 ... 0 0 ∂αN −2 ∂F . .. .. .. = .. (130) . . . ∂a 1 0 0 . . . ∂F1 0 ∂α 0 0 ... ∂F0 ∂α0 0 ∂F The very sparse block structure of ∂Q and ∂F allow us to solve (125) very ∂a efficiently. Defining the block notation of the Adjoint vector r= �� rN �T ··· (r1 ) T �T (131) with ri ∈ Rm for all i = [1..N ]. We compute the solution of (125) by initially ∂φ T setting rN = ∂q and then incrementally compute N ri−1 = ∂F T i ∂φ T r + ∂qi ∂qi−1 for i = N − 1 to 2 (132) The final ingredients we are missing in our framework is closed form solui i tions for computing ∂F and ∂F . Here we must consider the specific case ∂qi ∂αi of the deformable model simulation. 92 25.4 Deformable Models After appropriate spatial discretization (using for instance the finite element method ) the governing equations of a deformable model is given by the ordinary differential system of equations Mu̇ + Cu + k(x − x0 ) = fext + f (α) ẋ = u (133a) (133b) where M ∈ R3V ×3V is the mass matrix and V is the number of vertices in the spatial computational mesh, C3V ×3V is the damping matrix, x ∈ R3V is the spatial coordinate vector and u ∈ R3V is the spatial velocity vector, x0 = x(0) is the initial state of the deformable model. Hence x − x0 is the current displacement field and k(x − x0 ) : R3V �→ R3V is the elastic force function that computes the spatial elastic forces given the current displacement field as argument. Further, the system is subject to external forces such as gravity etc. given by fext ∈ R3V . The last term f (α) : Rs �→ R3V is the control forces. These are a function of the control parameters α ∈ Rs . Time discretization of the ordinary differential equations yields the discrete velocity update � � ut+1 = I − ∆tM−1 C ut � � (134) + ∆tM−1 fext + f (αt ) − k(xt − x0 ) and the position update xt+1 = xt + ∆tut+1 . (135) From the velocity and position updates, (134) and (135), we have that the partial derivatives of the forward simulation function are given by ∂xt+1 ∂xt ∂xt+1 ∂ut ∂ut+1 ∂xt ∂ut+1 ∂ut (136a) =I = ∆t ∂ut+1 ∂ut (136b) = ∆tM−1 K (136c) = I + ∆tM−1 C (136d) and ∂xt+1 =0 ∂αt ∂ut+1 ∂f = ∆tM−1 t t ∂α ∂α 93 (137a) (137b) where K = ∂k is the stiffness tangent matrix. We Define the state space ∂x vector qt ∈ Rm with m = 6V as � t� x t q = t (138) u This results in the partial derivatives � � ∂Fi I ∆tI + ∆t2 M−1 C = ∆tM−1 K (I + ∆tM−1 C) ∂qi � � ∂f ∂Fi ∆t2 M−1 ∂α t = ∂f ∆tM−1 ∂α ∂αi t (139a) (139b) We observe from this, that we really only need to compute the tangent ∂f stiffness matrix K and the derivative of the control force ∂α . Notice that for linear elasticity the tangent stiffness matrix is constant and needs not be re-computed. However, for nonlinear elasticity the stiffness matrix would depend on the current displacement field and must be re-computed for each simulation step. This requires us to store all the states q0 ,q1 , . . ., qN . We may compute the tangent stiffness matrix for general hyper elastic materials using a closed form solution (See Erleben [102, 103, 104, 105, 106]). The control force can be quite complicated. However, the most simple model would simply be a nodal force. Hence f (α) = α (140) ∂f and one would find ∂α = I. For the muscle splines described in the previous part we could replace f (α) in (140) with the spline activation and force distribution function. Observe that to compute φ, the forward simulation is performed and q1 ,. . .,qN are stored. Once this is done it is straightforward to evaluate the value of the objective function. In order to compute the gradient of the objective function, compute the Adjoint vectors rN , . . .,r1 . Notice these are computed in reverse order of the forward simulation. For the ith step of the Adjoint problem we need the qi state to calculate the partial derivatives. This is a potential drawback, as we need sufficient storage, to store the state for all simulation steps. Another potential drawback is the costly computation of the tangent stiffness matrix. Finally, from previous work it is well known that the Adjoint method has difficulties dealing with discontinuities in the state function. This, obviously occurs during collisions and contacts. Some scheme is needed to handle this if the method is to be effective beyond short horizon planning. 94 26 A Simple 2D Particle Example Consider the equation of motion of a single 2D particle in a gravitational field mr̈ = fext + f (α) (141) where m is the particle mass and r = (x, y)T is the particle position, fext = (0, −mg) is the gravitational force with the gravitational acceleration given by g ≈ 9.82 [m/s2 ]. The control force is given by f (α) = (αx , αy ). Introducing the particle velocity ṙ = v and using the usual time-discretization we obtain the velocity and update rules (142a) rt+1 = rt + ∆tvt+1 , � ∆t � vt+1 = vt + fext + αt , m (142b) where ∆t is the time step size. This defines our forward simulation function Ft (qt , αt ) with the state given by � t� r t q = t . (143) v We may now compute the partial derivatives ∂Ft = ∂qt � ∂rt+1 ∂rt ∂vt+1 ∂rt ∂rt+1 ∂vt ∂vt+1 ∂vt � ∂Ft ∂qt and � ∂Ft ∂αt I ∆tI2×2 = 2×2 0 I2×2 � as (144) and ∂Ft = ∂αt � ∂rt+1 � ∂αt ∂vt+1 ∂αt = � ∆t2 I m 2×2 ∆t I m 2×2 � (145) It may be instructive to compare this to the general deformable model formulas using C = K = 0 and M = m 27 A 2 Particle Example in 2D Building on the example from the previous section, we extend our system to have two particles label A and B. Now let us consider adding a single spring between the two particles. this simple system is the foundation of more complicated systems since every mass spring system can be disassembled into this simple relation. 95 Defining e = rA − rB then the spring force on particle A is � � e eeT kA = −k (� e � −l0 ) −c (vA − vB ) �e� � e �2 (146) The force on particle B is by symmetry kB = −kA . By the product rule we have � � ∂kA e ∂ (� e �) = −k ∂rA �e� ∂rA � � e ∂ �e� − k (� e � −l0 ) ∂rA � � e ∂ �e� T e (vA − vB ) −c ∂rA �e� � T � e ∂ �e� e T −c (vA − vB ) �e� ∂rA Defining the auxiliary variables ∂ (� e �) eT = ∂r �e� � A� e ∂ �e� 1 eeT H= = I2×2 − ∂rA �e� � e �3 hT = We may rewrite the equation � ∂kA e � T =− kh + c (vA − vB )T H ∂rA �e� � � eT (vA − vB ) − k (� e � −l0 ) + c H �e� A A and ∂k = − ∂k . Now the stiffness matrix for our 2 particle mass spring ∂rB ∂rA system would be � � K= ∂kA ∂rA ∂kB ∂rA 96 ∂kA ∂rB ∂kB ∂rB The damping terms are ∂kA eeT = −c ∂vA � e �2 and ∂kA ∂vB A = − ∂k so the damping matrix is ∂vA C= � ∂kA ∂vA ∂kB ∂vA ∂kA ∂vB ∂kB ∂vB � and the mass matrix is � mA I 0 C= 0 mB I � where mA and mB are the respective particle masses. Observe that if one extends to a mass spring system with several particles and several springs. Then one would assemble a global stiffness matrix and damping matrix by looping over all springs and compute the above local spring element matrices and accumulate their values into a system global matrix. 28 Results Since this research is still very much ongoing, we have less results to show that was the case for the other parts of this thesis. We have however made a simple setup with 2 particles connected by a spring as described in section 27. To test the method we shoot each of the particles in a given direction, and let them travel for a fixed number of timesteps under the influence of a given force and the spring force from their connection. For each of the time-steps we measure the position and velocity of each of the particles we then make a perturbed solution which we use as a starting guess for the optimization. Our objective function is the least squares difference of velocity and position for every fourth frame. We then run the optimization routine with the expectation of getting a solution which is close to the original. We use the inbuilt matlab solver fminunc, with the gradient which we have calculated using the results from above. The optimization took around 10-15 seconds in most cases. Figure 41 show the results of our test. As it can be seen the result is almost identical to the ground truth. 97 Trajectory 2.8 Optimized A Optimized B Initial Guess A Initial Guess B Ground Truth A Ground Truth B 2.6 2.4 2.2 y position 2 1.8 1.6 1.4 1.2 1 0.8 1 1.5 2 2.5 3 3.5 4 4.5 x Position Figure 41: The result of testing the adjoint method on two particles connected with a spring. 29 Conclusion on space time optimization A lot of work still remains for this method to be feasible but the preliminary result are very promising. We have shown that we can set up the relation of the passive deformable mesh and shown a simple 2D example to verify the correctness of the solution. We have sketched the modification necessary, to incorporate active contraction of the mesh along a fibre direction. What we are still missing is the actual implementation of the general method in 3D for larger, more complex systems and the process of validating the method based on the results 98 Part V Concluding remarks This concludes my research work for this PhD project in the following, I will try to wrap up and conclude on the project as a whole. Three years is a long time and much have happened. During these years I have learned many things I didn’t even know was possible (some of them weren’t when I started out.) Most of my research have been published but more ideas keep popping up, so if possible, I expect to keep doing this for the next many years. 99 30 Conclusion Initially I set out to do Interactive Human motion. Our common goal in the HUMIM group, was a system which could help people in physiotherapeutic rehabilitation. As exciting a goal as this was it did not fully encompass what I wanted to do. My interests are in the simulation of human motion, not in vision and machine learning, even though these subject are interesting in their own right. My goal was to be able to simulate motion of humans but even when I started I knew that to make a full simulation system from scratch in three years would be an impossible task. Even more so, as I had to learn (and to some extend invent) all the theory behind it first. My more realistic goal was the following • develop a robust and intuitive inverse kinematics system for motion planning and kinematic analysis. • investigate methods and develop a model for joint constraints which could describe the highly non convex form of human joint limits. • develop a muscle model which included deformation without loosing the interactive quality. • investigate ideas for controlling the motion of complex deformable bodies in space and time. I believe I have succeeded in at least coming up with some solutions for each of these points. I believe that the methods and models that we have developed are usable and applicable and I hope that People will use them, not just as citations in their own papers but as actual tools in their projects, be that research, industrial, artistic or something completely different. 31 Future Work Human motion simulation is a huge area. I have so far only scratched the surface of the many possibilities for contributing to this exciting field of research. With the knowledge I have acquired through my three years of PhD studies, I expect to be able to continue contributing to the Human Motion simulation research. Some of the topics I find most appealing in the light of my previous work are: Developing the control scheme further so it becomes possible to drive the spline activation using kinematic data. 100 Applying the spline activation in a biomechanical muscle model thus, hopefully making better analysis of muscle activation analysis possible. Combine all the techniques I have developed in a motion analysis framework. Such a framework might be able to do synthesized or optimized motion from temporally sparse samples, instead of relying, as most current tools do, on full motion captured kinematic data 101 32 Epilogue Life is just a party, and parties weren’t meant to last! Prince All things must have an end. And so this is it, for my PhD studies. This Thesis effectively concludes three exciting years of research. I can honestly say that I have had fun all the way. I feel privileged to have been able to follow my whims and ideas and to explore what I found interesting. The only Problem with this is, I want more. I cannot now imagine a life where I do not have the opportunity to investigate ideas, and invent solutions for problems I encounter and find interesting. I am convinced that I will continue doing research , either in Academia if the opportunity presents itself, in an R & D department in a private company, or just as something i do on the side. Conviction got me this far, i am sure it will get me further. It is My hope that my collaborators and colleagues are prepared to indulge me in the future as they have done until now. with this I conclude. 102 References [1] Blender. The blender foundation. web page. http://www.blender.org. [2] the Ogre team. Ogre3d. web page. http://www.ogre3d.org. [3] Morten Engell-Nørregård and Kenny Erleben. A projected back-tracking line-search for constrained interactive inverse kinematics. Computers & Graphics, 35(2):288 – 298, 2011. [4] M. Engell-Nørregård, S. Hauberg, J. Lapuyade, K. Erleben, and K.S. Pedersen. Interactive inverse kinematics for human motion estimation. [5] Autodesk. Autodesk. web page. http://usa.autodesk.com. [6] Søren Hauberg, Jerome Lapuyade, Morten Engell-Nørregård, Kenny Erleben, and Kim Steenstrup Pedersen. Three dimensional monocular human motion analysis in end-effector space. In Daniel Cremers et al., editors, Energy Minimization Methods in Computer Vision and Pattern Recognition, Lecture Notes in Computer Science, pages 235–248. Springer, August 2009. [7] John J. Craig. Introduction to Robotics: Mechanics and Control. AddisonWesley Longman Publishing Co., Inc., Boston, MA, USA, 1989. [8] Michael Girard and A. A. Maciejewski. Computational modeling for the computer animation of legged figures. In SIGGRAPH ’85: Proceedings of the 12th annual conference on Computer graphics and interactive techniques, pages 263–270, New York, NY, USA, 1985. ACM Press. [9] T. C. Hsia and Z. Y. Guo. New inverse kinematic algorithms for redundant robots. Journal of Robotic Systems, 8(1):117–132, 1991. [10] Samuel R. Buss. Introduction to inverse kinematics with jacobian transpose, pseudoinverse and damped least squares methods. Unpublished survey, april 2004. [11] Jianmin Zhao and Norman I. Badler. Inverse kinematics positioning using nonlinear programming for highly articulated figures. ACM Trans. Graph., 13(4):313–336, 1994. [12] Anthony A. Maciejewski. Motion simulation: Dealing with the illconditioned equations of motion for articulated figures. IEEE Comput. Graph. Appl., 10(3):63–71, 1990. [13] Pasquale Chiacchio and Bruno Siciliano. A closed-loop jacobian transpose scheme for solving the inverse kinematics of nonredundant and redundant wrists. Journal of Robotic Systems, 6(5):601–630, 1989. 103 [14] Martin Fedor. Application of inverse kinematics for skeleton manipulation in real-time. In SCCG ’03: Proceedings of the 19th spring conference on Computer graphics, pages 203–212, New York, NY, USA, 2003. ACM Press. [15] Chris Wellman. Inverse kinematics and geometric constraints for articulated figure manipulation. Master’s thesis, Simon Fraser University, 1993. [16] Edmond S. L. Ho, Taku Komura, and Rynson W. H. Lau. Computing inverse kinematics with linear programming. In VRST ’05: Proceedings of the ACM symposium on Virtual reality software and technology, pages 163–166, New York, NY, USA, 2005. ACM. [17] Robert W. Sumner, Matthias Zwicker, Craig Gotsman, and Jovan Popović. Mesh-based inverse kinematics. ACM Trans. Graph., 24(3):488–495, 2005. [18] Hadi Moradi and Sukhan Lee. Joint limit analysis and elbow movement minimization for redundant manipulators using closed form method. In DeShuang Huang, Xiao-Ping Zhang, and Guang-Bin Huang, editors, Advances in Intelligent Computing, International Conference on Intelligent Computing, ICIC 2005, Hefei, China, August 23-26, 2005, Proceedings, Part II, volume 3645 of Lecture Notes in Computer Science, pages 423–432. Springer, 2005. [19] Alla Safonova, Jessica K. Hodgins, and Nancy S. Pollard. Synthesizing physically realistic human motion in low-dimensional, behavior-specific spaces. ACM Trans. Graph., 23(3):514–521, 2004. [20] Anthony C. Fang and Nancy S. Pollard. Efficient synthesis of physically valid human motion. ACM Transactions on Graphics (TOG), 22(3):417–426, 2003. [21] C. Karen Liu, Aaron Hertzmann, and Zoran Popović. Learning physics-based motion style with nonlinear inverse optimization. ACM Trans. Graph., 24(3):1071–1081, 2005. [22] Walter Maurel and Daniel Thalmann. Human shoulder modeling including scapulo-thoracic constraint and joint sinus cones. Computers & Graphics, 24(2):203–218, 2000. [23] Lorna Herda, Raquel Urtasun, Pascal Fua, and Andrew J. Hanson. Automatic determination of shoulder joint limits using quaternion field boundaries. I. J. Robotic Res., 22(6):419–438, 2003. [24] Michael Meredith and Steve Maddock. Using a half-jacobian for real-time inverse kinematics. In Proceedings of The 5th International Conference on Computer Games: Artificial Intelligence, Design and Education (CGADIDE, pages 81–88, Reading, United Kingdom, November 2004. 104 [25] Jane Wilhelms and Allen Van Gelder. Fast and easy reach-cone joint limits. J. Graph. Tools, 6(2):27–41, 2001. [26] Wei Shao and Victor Ng-Thow-Hing. A general joint component framework for realistic articulation in human characters. In I3D ’03: Proceedings of the 2003 symposium on Interactive 3D graphics, pages 11–18, New York, NY, USA, 2003. ACM. [27] OpenTissueBoard. Opentissue. http://www.opentissue.org. web page, 2010. [28] Jorge Nocedal and Stephen J. Wright. Numerical optimization. Springer Series in Operations Research. Springer-Verlag, New York, 1999. [29] Yousef Saad. Iterative Methods for Sparse Linear Systems, 2nd edition. SIAM, Philadelpha, PA, 2003. [30] J. B. Rosen. The gradient projection method for nonlinear programming. part i. linear constraints. Journal of the Society for Industrial and Applied Mathematics, 8(1):181–217, 1960. [31] J. B. Rosen. The gradient projection method for nonlinear programming. part ii. nonlinear constraints. Journal of the Society for Industrial and Applied Mathematics, 9(4):514–532, 1961. [32] Michael Meredith and Steve Maddock. Adapting motion capture data using weighted real-time inverse kinematics. Comput. Entertain., 3(1):5–5, 2005. [33] Se-Young Oh, David Orin, and Michael Bach. An inverse kinematic solution for kinematically redundant robot manipulators. Journal of Robotic Systems, 1(3):235–249, 1984. [34] Samuel R. Buss and Jin-Su Kim. Selectively damped least squares for inverse kinematics. journal of graphics tools, 10(3):37–49, 2005. [35] Ladislav Kavan and Jiri Zara. Spherical blend skinning: A real-time deformation of articulated models. In 2005 ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, pages 9–16. ACM Press, April 2005. [36] Matthew Mason. Mechanics of Robotic Manipulation. MIT Press, Cambridge, Massachusetts, London, England, August 2001. Intelligent Robotics and Autonomous Agents Series, ISBN 0-262-13396-2. [37] CGgenie. Cg genie. web page, 2010. http://cgenie.com/. [38] Zlib. Zlib-libpng license. web http://opensource.org/licenses/zlib-license.php. 105 page, 2010. [39] Morten Engell-Nørregård and Kenny Erleben. A Projected Non-linear Conjugate Gradient Method for Interactive Inverse Kinematics. In MATHMOD 2009 - 6th Vienna International Conference on Mathematical Modelling, 2009. [40] Morten Engell-Nørregård. A comparative analysis of selected optimization solvers, 2008. [41] Søren Hauberg. Spatial Models of Human Motion. PhD thesis, 2011. [42] Luis Sentis and Oussama Khatib. A whole-body control framework for humanoids operating in human environments. In ICRA, pages 2641–2648, 2006. [43] Ronald Poppe. Vision-based human motion analysis: An overview. Computer Vision Image Understing, 108(1-2):4–18, 2007. [44] J. M. Wang, D. J. Fleet, and A. Hertzmann. Gaussian Process Dynamical Models for Human Motion. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 30(2):283–298, 2008. [45] Raquel Urtasun, David J. Fleet, and Pascal Fua. 3D People Tracking with Gaussian Process Dynamical Models. In CVPR ’06: Proceedings of the 2006 IEEE Computer Society , Conference on Computer Vision and Pattern Recognition, pages 238–245, Washington, DC, USA, 2006. IEEE Computer Society. [46] Zhengdong Lu, Miguel Carreira-Perpinan, and Cristian Sminchisescu. People Tracking with the Laplacian Eigenmaps Latent Variable Model. In J.C. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems 20, pages 1705–1712. MIT Press, Cambridge, MA, 2008. [47] Cristian Sminchisescu and Bill Triggs. Kinematic Jump Processes for Monocular 3D Human Tracking. In In IEEE International Conference on Computer Vision and Pattern Recognition, pages 69–76, 2003. [48] M. Engell-Nørregård, S. Niebe, and K. Erleben. A joint-constraint model for human joints using signed distance-fields. Multibody System Dynamics, pages 1–13, 2012. [49] M. Engell-Nørregård, S. Niebe, and K. Erleben. Local joint–limits using distance field cones in euler angle space. In Computer graphics international, 2010. [50] CMU Graphics Lab Motion Capture Database, http://mocap.cs.cmu.edu/, 2009. Carnegie Mellon University. 106 [51] Morten Engell-Nørregård, Sarah Niebe, and Kenny Erleben. Local Joint– Limits using Distance Field Cones in Euler Angle Space. [52] Joo Kim, Jingzhou Yang, and Karim Abdel-Malek. A novel formulation for determining joint constraint loads during optimal dynamic motion of redundant manipulators in dh representation. Multibody System Dynamics, 19:427–451, 2008. 10.1007/s11044-007-9100-4. [53] Hiroyuki Sugiyama and Hiroki Yamashita. Spatial joint constraints for the absolute nodal coordinate formulation using the non-generalized intermediate coordinates. Multibody System Dynamics, 26:15–36, 2011. 10.1007/s11044-010-9236-5. [54] Darwin 3d. Acclaim fileformat. http://www.darwin3d.com/gamedev/. web page. [55] Morten Engell-Nørregård and Kenny Erleben. Estimation of Joint types and Joint Limits from Motion capture data. In WSCG 2009: 17-th International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision, 2009. [56] A. E. Engin and S. T. Tümer. Three-dimensional kinematic modelling of the human shoulder complex—part i: Physical model and determination of joint sinus cones. Journal of Biomechanical Engineering, 111(2):107–112, 1989. [57] James U. Korein. A Geometric Investigation of reach. MIT Press Cambridge, 1985. [58] Sung-Hee Lee and Demetri Terzopoulos. Spline joints for multibody dynamics. ACM Trans. Graph., 27(3):1–8, 2008. [59] Lorna Herda, Raquel Urtasun, and Pascal Fua. Hierarchical implicit surface joint limits for human body tracking. Computer Vision and Image Understanding, 99(2):189–209, 2005. [60] A. Bærentzen. Robust generation of signed distance fields from triangle meshes. In D. Fellner, T. Moller, and S. Spencer, editors, Fourth International Workshop on Volume Graphics, 2005., pages 167–239, jun 2005. [61] Erik Dam, Martin Koch, and Martin Lillholm. Quaternions, interpolation and animation. Technical report, University of Copenhagen, 1998. [62] Jakob Andreas Bærentzen and Niels Jorgen Christensen. Interactive modelling of shapes using the level-set method. International Journal of Shaping Modeling, 8(2):79–97, 2002. 107 [63] A. V. Hill. The heat of shortening and the dynamic constants of muscle. Proceedings of the Royal Society of London. Series B, Biological Sciences, 126(843):pp. 136–195, 1938. [64] FE Zajac, EL Topp, and PJ Stevenson. A dimensionless musculotendon model. Proceedings IEEE Engineering in Medicine and Biology, pages 26–31, 1986. [65] Alan H. Barr. Global and local deformations of solid primitives. SIGGRAPH Comput. Graph., 18:21–30, January 1984. [66] Thomas W. Sederberg and Scott R. Parry. Free-form deformation of solid geometric models. SIGGRAPH Comput. Graph., 20:151–160, August 1986. [67] Andrew Witkin, Kurt Fleischer, and Alan Barr. Energy constraints on parameterized models. SIGGRAPH Comput. Graph., 21:225–232, August 1987. [68] Ronen Barzel and Alan H. Barr. A modeling system based on dynamic constraints. SIGGRAPH Comput. Graph., 22:179–188, June 1988. [69] Dimitri Metaxas and Demetri Terzopoulos. Dynamic deformation of solid primitives with constraints. SIGGRAPH Comput. Graph., 26:309–312, July 1992. [70] A. Pentland and J. Williams. Good vibrations: modal dynamics for graphics and animation. SIGGRAPH Comput. Graph., 23:207–214, July 1989. [71] Doug L. James and Dinesh K. Pai. Dyrt: dynamic response textures for real time deformation simulation with graphics hardware. ACM Trans. Graph., 21:582–585, July 2002. [72] A. Aubel and D. Thalmann. Efficient muscle shape deformation. In Deformable avatars: IFIP TC5/WG5. 10 DEFORM’2000 Workshop, November 29-30, 2000, Geneva, Switzerland and AVATARS’2000 Workshop, November 30-December 1, 2000, Lausanne, Switzerland, page 132. Springer Netherlands, 2001. [73] K.S. Lee and G. Ashraf. Simplified Muscle Dynamics for Appealing RealTime Skin Deformation. In International Conference on Computer Graphics and Virtual Reality, 2007. [74] Gavin S. P. Miller. The motion dynamics of snakes and worms. SIGGRAPH Comput. Graph., 22(4):169–173, June 1988. [75] Radek Grzeszczuk and Demetri Terzopoulos. Automated learning of muscle-actuated locomotion through control abstraction. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, SIGGRAPH ’95, pages 63–70, New York, NY, USA, 1995. ACM. 108 [76] Petros Faloutsos, Michiel van de Panne, and Demetri Terzopoulos. Dynamic free-form deformations for animation synthesis. IEEE Transactions on Visualization and Computer Graphics, 3(3):201–214, July 1997. [77] J. E. Chadwick, D. R. Haumann, and R. E. Parent. Layered construction for deformable animated characters. SIGGRAPH Comput. Graph., 23:243–252, July 1989. [78] Xiaoyuan Tu and Demetri Terzopoulos. Artificial fishes: physics, locomotion, perception, behavior. In Proceedings of the 21st annual conference on Computer graphics and interactive techniques, SIGGRAPH ’94, pages 43–50, New York, NY, USA, 1994. ACM. [79] Chris Wojtan, Peter J. Mucha, and Greg Turk. Keyframe control of complex particle systems using the adjoint method. In Proceedings of the 2006 ACM SIGGRAPH/Eurographics symposium on Computer animation, SCA ’06, pages 15–23, Aire-la-Ville, Switzerland, Switzerland, 2006. Eurographics Association. [80] Jernej Barbič, Marco da Silva, and Jovan Popović. Deformable object animation using reduced optimal control. ACM Trans. Graph., 28(3):53:1– 53:9, July 2009. [81] Andrew Witkin and Michael Kass. Spacetime constraints. SIGGRAPH Comput. Graph., 22:159–168, June 1988. [82] M. Viceconti, D. Testi, F. Taddei, S. Martelli, G. J. Clapworthy, and S. V. Jan. Biomechanics modeling of the musculoskeletal apparatus: Status and key issues. Proceedings of the Ieee, 94(4):725–739, April 2006. [83] Marcus G. Pandy and Thomas P. Andriacchi. Muscle and joint function in human locomotion. Annual Review of Biomedical Engineering, 12(1):401– 433, 2010. [84] Qing hong Zhu, Yan Chen, and Arie E. Kaufman. Real-time biomechanically-based muscle volume deformation using fem. Comput. Graph. Forum, 17(3):275–284, 1998. [85] C W J Oomens, M Maenhout, C H van Oijen, M R Drost, and F P Baaijens. Finite element modelling of contracting skeletal muscle. Philosophical Transactions. Royal Society B, 358(1437), 2003. [86] Xianlian Zhou and Jia Lu. Nurbs-based galerkin method and application to skeletal muscle modeling. In Proceedings of the 2005 ACM symposium on Solid and physical modeling, SPM ’05, pages 71–78, New York, NY, USA, 2005. ACM. 109 [87] J. Teran, S. Blemker, V. Ng Thow Hing, and R. Fedkiw. Finite volume methods for the simulation of skeletal muscle. In Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, SCA ’03, pages 68–74, Aire-la-Ville, Switzerland, Switzerland, 2003. Eurographics Association. [88] Joseph Teran, Eftychios Sifakis, Silvia S. Blemker, Victor Ng-Thow-Hing, Cynthia Lau, and Ronald Fedkiw. Creating and simulating skeletal muscle from the visible human data set. IEEE Transactions on Visualization and Computer Graphics, 11:317–328, 2005. [89] L. P. Nedel and D. Thalmann. Real time muscle deformations using massspring systems. In Proceedings of the Computer Graphics International 1998, CGI ’98, pages 156–, Washington, DC, USA, 1998. IEEE Computer Society. [90] Shinjiro Sueda, Andrew Kaufman, and Dinesh K. Pai. Musculotendon simulation for hand animation. ACM Trans. Graph. (Proc. SIGGRAPH), 27(3), 2008. [91] Dinesh K. Pai, Shinjiro Sueda, and Qi Wei. Fast physically based musculoskeletal simulation. In ACM SIGGRAPH 2005 Sketches, SIGGRAPH ’05, New York, NY, USA, 2005. ACM. [92] Brian A. Garner and Marcus G. Pandy. The obstacle-set method for representing muscle paths in musculoskeletal models. "Computer Methods in Biomechanics and Biomedical Engineering", 3, 2000. [93] D. Baraff and A. Witkin. Partitioned dynamics. Technical Report CMU-RITR-97-33, Robotics Institute, Carnegie Mellon University, 1997. [94] Matthias Müller and Markus Gross. Interactive virtual materials. In Proceedings of Graphics Interface 2004, GI ’04, pages 239–246, School of Computer Science, University of Waterloo, Waterloo, Ontario, Canada, 2004. Canadian Human-Computer Communications Society. [95] Matthias Teschner, Bruno Heidelberger, Matthias Muller, and Markus Gross. A versatile and robust model for geometrically complex deformable solids. In Proceedings of the Computer Graphics International, pages 312–319, Washington, DC, USA, 2004. IEEE Computer Society. [96] Kenny Erleben, Jon Sporring, Knud Henriksen, and Henrik Dohlman. Physics-based Animation (Graphics Series). Charles River Media, Inc., Rockland, MA, USA, 2005. [97] W.L. Briggs, V.E. Henson, and S.F. McCormick. A multigrid tutorial. Miscellaneous Bks. Society for Industrial and Applied Mathematics, 2000. 110 [98] B. Gilles and N. Magnenat-Thalmann. Musculoskeletal mri segmentation using multi-resolution simplex meshes with medial representations. Medical Image Anaylsis, pages 291–302, June 2010. [99] J. Rasmussen, M. Damsgaard, E. Surma, S.T. Christensen, M. de Zee, and V. Vondrak. Anybody-a software system for ergonomic optimization. In Fifth World Congress on Structural and Multidisciplinary Optimization, 2003, 2003. [100] S.L. Delp, F.C. Anderson, A.S. Arnold, P. Loan, A. Habib, C.T. John, E. Guendelman, and D.G. Thelen. Opensim: open-source software to create and analyze dynamic simulations of movement. Biomedical Engineering, IEEE Transactions on, 54(11):1940–1950, 2007. [101] Antoine McNamara, Adrien Treuille, Zoran Popović, and Jos Stam. Fluid control using the adjoint method. ACM Trans. Graph., 23:449–456, August 2004. [102] Kenny Erleben. Hyper elasticity for small and large deformations based on a lagrangian formulation. Technical Report 2011/8, Department of Computer Science, University of Copenhagen, Denmark, December 2011. [103] Kenny Erleben. The tangent stiffness matrix for for the implicit nonlinear finite element method. Technical Report 2011/9, Department of Computer Science, University of Copenhagen, Denmark, December 2011. [104] Kenny Erleben. Preliminaries on tensor notation and continuum mechanics. Technical Report 2011/7, Department of Computer Science, University of Copenhagen, Denmark, December 2011. [105] Kenny Erleben. Implementation tricks and tips for finite element modeling of hyper elastic materials. Technical Report 2011/10, Department of Computer Science, University of Copenhagen, Denmark, December 2011. [106] Kenny Erleben. hyper-sim. Published online at code.google.com/p/hypersim/, November 2011. Open source project for simulation methods for hyper elastic materials in physics-based animation. 111

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertisement